view org/body.org @ 472:516a029e0be9

complete first draft of hearing.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 18:14:04 -0400
parents da311eefbb09
children
line wrap: on
line source
1 #+title: Building a Body
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulating a body (movement, touch, proprioception) in jMonkeyEngine3.
5 #+SETUPFILE: ../../aurellem/org/setup.org
6 #+INCLUDE: ../../aurellem/org/level-0.org
8 * Design Constraints
10 I use [[www.blender.org/][blender]] to design bodies. The design of the bodies is
11 determined by the requirements of the AI that will use them. The
12 bodies must be easy for an AI to sense and control, and they must be
13 relatively simple for jMonkeyEngine to compute.
15 # I'm a secret test! :P
16 ** Bag of Bones
18 How to create such a body? One option I ultimately rejected is to use
19 blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea would have been to define a mesh
20 which describes the creature's entire body. To this you add an
21 skeleton which deforms this mesh. This technique is used extensively
22 to model humans and create realistic animations. It is hard to use for
23 my purposes because it is difficult to update the creature's Physics
24 Collision Mesh in tandem with its Geometric Mesh under the influence
25 of the armature. Without this the creature will not be able to grab
26 things in its environment, and it won't be able to tell where its
27 physical body is by using its eyes. Also, armatures do not specify any
28 rotational limits for a joint, making it hard to model elbows,
29 shoulders, etc.
31 ** EVE
33 Instead of using the human-like "deformable bag of bones" approach, I
34 decided to base my body plans on the robot EVE from the movie wall-E.
36 #+caption: EVE from the movie WALL-E. This body plan turns out to
37 #+caption: be much better suited to my purposes than a more
38 #+caption: human-like one.
39 [[../images/Eve.jpg]]
41 EVE's body is composed of several rigid components that are held
42 together by invisible joint constraints. This is what I mean by
43 "eve-like". The main reason that I use eve-style bodies is so that
44 there will be correspondence between the AI's vision and the physical
45 presence of its body. Each individual section is simulated by a
46 separate rigid body that corresponds exactly with its visual
47 representation and does not change. Sections are connected by
48 invisible joints that are well supported in jMonkeyEngine. Bullet, the
49 physics backend for jMonkeyEngine, can efficiently simulate hundreds
50 of rigid bodies connected by joints. Sections do not have to stay as
51 one piece forever; they can be dynamically replaced with multiple
52 sections to simulate splitting in two. This could be used to simulate
53 retractable claws or EVE's hands, which are able to coalesce into one
54 object in the movie.
56 * Solidifying the Body
58 Here is a hand designed eve-style in blender.
60 #+attr_html: width="755"
61 [[../images/hand-screenshot0.png]]
63 If we load it directly into jMonkeyEngine, we get this:
65 #+name: test-1
66 #+begin_src clojure
67 (def hand-path "Models/test-creature/hand.blend")
69 (defn hand [] (load-blender-model hand-path))
71 (defn setup [world]
72 (let [cam (.getCamera world)]
73 (println-repl cam)
74 (.setLocation
75 cam (Vector3f.
76 -6.9015837, 8.644911, 5.6043186))
77 (.setRotation
78 cam
79 (Quaternion.
80 0.14046453, 0.85894054, -0.34301838, 0.3533118)))
81 (light-up-everything world)
82 (.setTimer world (RatchetTimer. 60))
83 world)
85 (defn test-hand-1
86 ([] (test-hand-1 false))
87 ([record?]
88 (world (hand)
89 standard-debug-controls
90 (fn [world]
91 (if record?
92 (Capture/captureVideo
93 world
94 (File. "/home/r/proj/cortex/render/body/1")))
95 (setup world)) no-op)))
96 #+end_src
99 #+begin_src clojure :results silent
100 (.start (cortex.test.body/test-one))
101 #+end_src
103 #+begin_html
104 <div class="figure">
105 <center>
106 <video controls="controls" width="640">
107 <source src="../video/ghost-hand.ogg" type="video/ogg"
108 preload="none" poster="../images/aurellem-1280x480.png" />
109 </video>
110 <br> <a href="http://youtu.be/9LZpwTIhjzE"> YouTube </a>
111 </center>
112 <p>The hand model directly loaded from blender. It has no physical
113 presence in the simulation. </p>
114 </div>
115 #+end_html
117 You will notice that the hand has no physical presence -- it's a
118 hologram through which everything passes. Therefore, the first thing
119 to do is to make it solid. Blender has physics simulation on par with
120 jMonkeyEngine (they both use bullet as their physics backend), but it
121 can be difficult to translate between the two systems, so for now I
122 specify the mass of each object as meta-data in blender and construct
123 the physics shape based on the mesh in jMonkeyEngine.
125 #+name: body-1
126 #+begin_src clojure
127 (defn physical!
128 "Iterate through the nodes in creature and make them real physical
129 objects in the simulation."
130 [#^Node creature]
131 (dorun
132 (map
133 (fn [geom]
134 (let [physics-control
135 (RigidBodyControl.
136 (HullCollisionShape.
137 (.getMesh geom))
138 (if-let [mass (meta-data geom "mass")]
139 (do
140 ;;(println-repl
141 ;; "setting" (.getName geom) "mass to" (float mass))
142 (float mass))
143 (float 1)))]
144 (.addControl geom physics-control)))
145 (filter #(isa? (class %) Geometry )
146 (node-seq creature)))))
147 #+end_src
149 =physical!= iterates through a creature's node structure, creating
150 CollisionShapes for each geometry with the mass specified in that
151 geometry's meta-data.
153 #+name: test-2
154 #+begin_src clojure
155 (in-ns 'cortex.test.body)
157 (def gravity-control
158 {"key-g" (fn [world _]
159 (set-gravity world (Vector3f. 0 -9.81 0)))
160 "key-u" (fn [world _] (set-gravity world Vector3f/ZERO))})
162 (defn floor []
163 (box 10 3 10 :position (Vector3f. 0 -10 0)
164 :color ColorRGBA/Gray :mass 0))
166 (defn test-hand-2
167 ([] (test-hand-2 false))
168 ([record?]
169 (world
170 (nodify
171 [(doto (hand)
172 (physical!))
173 (floor)])
174 (merge standard-debug-controls gravity-control)
175 (fn [world]
176 (if record?
177 (Capture/captureVideo
178 world (File. "/home/r/proj/cortex/render/body/2")))
179 (set-gravity world Vector3f/ZERO)
180 (setup world))
181 no-op)))
182 #+end_src
184 #+results: test-2
185 : #'cortex.test.body/test-hand-2
187 #+begin_html
188 <div class="figure">
189 <center>
190 <video controls="controls" width="640">
191 <source src="../video/crumbly-hand.ogg" type="video/ogg"
192 preload="none" poster="../images/aurellem-1280x480.png" />
193 </video>
194 <br> <a href="http://youtu.be/GEA1SACwpPg"> YouTube </a>
195 </center>
196 <p>The hand now has a physical presence, but there is nothing to hold
197 it together.</p>
198 </div>
199 #+end_html
201 Now that's some progress.
203 * Joints
205 Obviously, an AI is not going to be doing much while lying in pieces
206 on the floor. So, the next step to making a proper body is to connect
207 those pieces together with joints. jMonkeyEngine has a large array of
208 joints available via bullet, such as Point2Point, Cone, Hinge, and a
209 generic Six Degree of Freedom joint, with or without spring
210 restitution.
212 Although it should be possible to specify the joints using blender's
213 physics system, and then automatically import them with jMonkeyEngine,
214 the support isn't there yet, and there are a few problems with bullet
215 itself that need to be solved before it can happen.
217 So, I will use the same system for specifying joints as I will do for
218 some senses. Each joint is specified by an empty node whose parent
219 has the name "joints". Their orientation and meta-data determine what
220 joint is created.
222 #+attr_html: width="755"
223 #+caption: Joints hack in blender. Each empty node here will be transformed into a joint in jMonkeyEngine
224 [[../images/hand-screenshot1.png]]
226 The empty node in the upper right, highlighted in yellow, is the
227 parent node of all the empties which represent joints. The following
228 functions must do three things to translate these into real joints:
230 - Find the children of the "joints" node.
231 - Determine the two spatials the joint is meant to connect.
232 - Create the joint based on the meta-data of the empty node.
234 ** Finding the Joints
236 The higher order function =sense-nodes= from =cortex.sense= simplifies
237 the first task.
239 #+name: joints-2
240 #+begin_src clojure
241 (def
242 ^{:doc "Return the children of the creature's \"joints\" node."
243 :arglists '([creature])}
244 joints
245 (sense-nodes "joints"))
246 #+end_src
248 ** Joint Targets and Orientation
250 This technique for finding a joint's targets is very similar to
251 =cortex.sense/closest-node=. A small cube, centered around the
252 empty-node, grows exponentially until it intersects two /physical/
253 objects. The objects are ordered according to the joint's rotation,
254 with the first one being the object that has more negative coordinates
255 in the joint's reference frame. Since the objects must be physical,
256 the empty-node itself escapes detection. Because the objects must be
257 physical, =joint-targets= must be called /after/ =physical!= is
258 called.
260 #+name: joints-3
261 #+begin_src clojure
262 (defn joint-targets
263 "Return the two closest two objects to the joint object, ordered
264 from bottom to top according to the joint's rotation."
265 [#^Node parts #^Node joint]
266 (loop [radius (float 0.01)]
267 (let [results (CollisionResults.)]
268 (.collideWith
269 parts
270 (BoundingBox. (.getWorldTranslation joint)
271 radius radius radius) results)
272 (let [targets
273 (distinct
274 (map #(.getGeometry %) results))]
275 (if (>= (count targets) 2)
276 (sort-by
277 #(let [joint-ref-frame-position
278 (jme-to-blender
279 (.mult
280 (.inverse (.getWorldRotation joint))
281 (.subtract (.getWorldTranslation %)
282 (.getWorldTranslation joint))))]
283 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
284 (take 2 targets))
285 (recur (float (* radius 2))))))))
286 #+end_src
288 ** Generating Joints
290 This section of code iterates through all the different ways of
291 specifying joints using blender meta-data and converts each one to the
292 appropriate jMonkeyEngine joint.
294 #+name: joints-4
295 #+begin_src clojure
296 (defmulti joint-dispatch
297 "Translate blender pseudo-joints into real JME joints."
298 (fn [constraints & _]
299 (:type constraints)))
301 (defmethod joint-dispatch :point
302 [constraints control-a control-b pivot-a pivot-b rotation]
303 ;;(println-repl "creating POINT2POINT joint")
304 ;; bullet's point2point joints are BROKEN, so we must use the
305 ;; generic 6DOF joint instead of an actual Point2Point joint!
307 ;; should be able to do this:
308 (comment
309 (Point2PointJoint.
310 control-a
311 control-b
312 pivot-a
313 pivot-b))
315 ;; but instead we must do this:
316 ;;(println-repl "substituting 6DOF joint for POINT2POINT joint!")
317 (doto
318 (SixDofJoint.
319 control-a
320 control-b
321 pivot-a
322 pivot-b
323 false)
324 (.setLinearLowerLimit Vector3f/ZERO)
325 (.setLinearUpperLimit Vector3f/ZERO)))
327 (defmethod joint-dispatch :hinge
328 [constraints control-a control-b pivot-a pivot-b rotation]
329 ;;(println-repl "creating HINGE joint")
330 (let [axis
331 (if-let
332 [axis (:axis constraints)]
333 axis
334 Vector3f/UNIT_X)
335 [limit-1 limit-2] (:limit constraints)
336 hinge-axis
337 (.mult
338 rotation
339 (blender-to-jme axis))]
340 (doto
341 (HingeJoint.
342 control-a
343 control-b
344 pivot-a
345 pivot-b
346 hinge-axis
347 hinge-axis)
348 (.setLimit limit-1 limit-2))))
350 (defmethod joint-dispatch :cone
351 [constraints control-a control-b pivot-a pivot-b rotation]
352 (let [limit-xz (:limit-xz constraints)
353 limit-xy (:limit-xy constraints)
354 twist (:twist constraints)]
356 ;;(println-repl "creating CONE joint")
357 ;;(println-repl rotation)
358 ;;(println-repl
359 ;; "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0)))
360 ;;(println-repl
361 ;; "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0)))
362 ;;(println-repl
363 ;; "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1)))
364 (doto
365 (ConeJoint.
366 control-a
367 control-b
368 pivot-a
369 pivot-b
370 rotation
371 rotation)
372 (.setLimit (float limit-xz)
373 (float limit-xy)
374 (float twist)))))
376 (defn connect
377 "Create a joint between 'obj-a and 'obj-b at the location of
378 'joint. The type of joint is determined by the metadata on 'joint.
380 Here are some examples:
381 {:type :point}
382 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
383 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
385 {:type :cone :limit-xz 0]
386 :limit-xy 0]
387 :twist 0]} (use XZY rotation mode in blender!)"
388 [#^Node obj-a #^Node obj-b #^Node joint]
389 (let [control-a (.getControl obj-a RigidBodyControl)
390 control-b (.getControl obj-b RigidBodyControl)
391 joint-center (.getWorldTranslation joint)
392 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
393 pivot-a (world-to-local obj-a joint-center)
394 pivot-b (world-to-local obj-b joint-center)]
396 (if-let [constraints
397 (map-vals
398 eval
399 (read-string
400 (meta-data joint "joint")))]
401 ;; A side-effect of creating a joint registers
402 ;; it with both physics objects which in turn
403 ;; will register the joint with the physics system
404 ;; when the simulation is started.
405 (do
406 ;;(println-repl "creating joint between"
407 ;; (.getName obj-a) "and" (.getName obj-b))
408 (joint-dispatch constraints
409 control-a control-b
410 pivot-a pivot-b
411 joint-rotation))
412 ;;(println-repl "could not find joint meta-data!")
413 )))
414 #+end_src
416 Creating joints is now a matter of applying =connect= to each joint
417 node.
419 #+name: joints-5
420 #+begin_src clojure
421 (defn joints!
422 "Connect the solid parts of the creature with physical joints. The
423 joints are taken from the \"joints\" node in the creature."
424 [#^Node creature]
425 (dorun
426 (map
427 (fn [joint]
428 (let [[obj-a obj-b] (joint-targets creature joint)]
429 (connect obj-a obj-b joint)))
430 (joints creature))))
431 #+end_src
433 ** Round 3
435 Now we can test the hand in all its glory.
437 #+name: test-3
438 #+begin_src clojure
439 (in-ns 'cortex.test.body)
441 (def debug-control
442 {"key-h" (fn [world val]
443 (if val (enable-debug world)))})
445 (defn test-hand-3
446 ([] (test-hand-3 false))
447 ([record?]
448 (world
449 (nodify
450 [(doto (hand)
451 (physical!)
452 (joints!))
453 (floor)])
454 (merge standard-debug-controls debug-control
455 gravity-control)
456 (comp
457 #(Capture/captureVideo
458 % (File. "/home/r/proj/cortex/render/body/3"))
459 #(do (set-gravity % Vector3f/ZERO) %)
460 setup)
461 no-op)))
462 #+end_src
464 =physical!= makes the hand solid, then =joints!= connects each
465 piece together.
467 #+begin_html
468 <div class="figure">
469 <center>
470 <video controls="controls" width="640">
471 <source src="../video/full-hand.ogg" type="video/ogg"
472 preload="none" poster="../images/aurellem-1280x480.png" />
473 </video>
474 <br> <a href="http://youtu.be/4affLfwSPP4"> YouTube </a>
475 </center>
476 <p>Now the hand is physical and has joints.</p>
477 </div>
478 #+end_html
480 The joints are visualized as green connections between each segment
481 for debug purposes. You can see that they correspond to the empty
482 nodes in the blender file.
484 * Wrap-Up!
486 It is convenient to combine =physical!= and =joints!= into one
487 function that completely creates the creature's physical body.
489 #+name: joints-6
490 #+begin_src clojure
491 (defn body!
492 "Endow the creature with a physical body connected with joints. The
493 particulars of the joints and the masses of each body part are
494 determined in blender."
495 [#^Node creature]
496 (physical! creature)
497 (joints! creature))
498 #+end_src
500 * The Worm
502 Going forward, I will use a model that is less complicated than the
503 hand. It has two segments and one joint, and I call it the worm. All
504 of the senses described in the following posts will be applied to this
505 worm.
507 #+name: test-4
508 #+begin_src clojure
509 (in-ns 'cortex.test.body)
511 (defn worm []
512 (load-blender-model
513 "Models/test-creature/worm.blend"))
515 (defn test-worm
517 "Testing physical bodies:
518 You should see the the worm fall onto a table. You can fire
519 physical balls at it and the worm should move upon being struck.
521 Keys:
522 <space> : fire cannon ball."
524 ([] (test-worm false))
525 ([record?]
526 (let [timer (RatchetTimer. 60)]
527 (world
528 (nodify
529 [(doto (worm)
530 (body!))
531 (floor)])
532 (merge standard-debug-controls debug-control)
533 #(do
534 (speed-up %)
535 (light-up-everything %)
536 (.setTimer % timer)
537 (cortex.util/display-dilated-time % timer)
538 (if record?
539 (Capture/captureVideo
540 % (File. "/home/r/proj/cortex/render/body/4"))))
541 no-op))))
542 #+end_src
544 #+results: test-4
545 : #'cortex.test.body/test-worm
547 #+begin_html
548 <div class="figure">
549 <center>
550 <video controls="controls" width="640">
551 <source src="../video/worm-1.ogg" type="video/ogg"
552 preload="none" poster="../images/aurellem-1280x480.png" />
553 </video>
554 <br> <a href="http://youtu.be/rFVXI0T3iSE"> YouTube </a>
555 </center>
556 <p>This worm model will be the platform onto which future senses will
557 be grafted.</p>
558 </div>
559 #+end_html
561 * Headers
562 #+name: body-header
563 #+begin_src clojure
564 (ns cortex.body
565 "Assemble a physical creature using the definitions found in a
566 specially prepared blender file. Creates rigid bodies and joints so
567 that a creature can have a physical presence in the simulation."
568 {:author "Robert McIntyre"}
569 (:use (cortex world util sense))
570 (:import
571 (com.jme3.math Vector3f Quaternion Vector2f Matrix3f)
572 (com.jme3.bullet.joints
573 SixDofJoint Point2PointJoint HingeJoint ConeJoint)
574 com.jme3.bullet.control.RigidBodyControl
575 com.jme3.collision.CollisionResults
576 com.jme3.bounding.BoundingBox
577 com.jme3.scene.Node
578 com.jme3.scene.Geometry
579 com.jme3.bullet.collision.shapes.HullCollisionShape))
580 #+end_src
582 #+name: test-header
583 #+begin_src clojure
584 (ns cortex.test.body
585 (:use (cortex world util body))
586 (:import
587 (com.aurellem.capture Capture RatchetTimer IsoTimer)
588 (com.jme3.math Quaternion Vector3f ColorRGBA)
589 java.io.File))
590 #+end_src
592 #+results: test-header
593 : java.io.File
595 * Source
596 - [[../src/cortex/body.clj][cortex.body]]
597 - [[../src/cortex/test/body.clj][cortex.test.body]]
598 - [[../assets/Models/test-creature/hand.blend][hand.blend]]
599 - [[../assets/Models/test-creature/palm.png][UV-map-1]]
600 - [[../assets/Models/test-creature/worm.blend][worm.blend]]
601 - [[../assets/Models/test-creature/retina-small.png][UV-map-1]]
602 - [[../assets/Models/test-creature/tip.png][UV-map-2]]
603 #+html: <ul> <li> <a href="../org/body.org">This org file</a> </li> </ul>
604 - [[http://hg.bortreb.com ][source-repository]]
606 * Next
607 The body I have made here exists without any senses or effectors. In
608 the [[./vision.org][next post]], I'll give the creature eyes.
610 * COMMENT Generate Source
611 #+begin_src clojure :tangle ../src/cortex/body.clj
612 <<body-header>>
613 <<body-1>>
614 <<joints-2>>
615 <<joints-3>>
616 <<joints-4>>
617 <<joints-5>>
618 <<joints-6>>
619 #+end_src
621 #+begin_src clojure :tangle ../src/cortex/test/body.clj
622 <<test-header>>
623 <<test-1>>
624 <<test-2>>
625 <<test-3>>
626 <<test-4>>
627 #+end_src