view org/body.org @ 311:7f3b77238045

re-encoded hearing video with latest ffmpeg to fix broken keyframes
author Robert McIntyre <rlm@mit.edu>
date Tue, 21 Feb 2012 14:56:33 -0700
parents 5d448182c807
children b46ccfd128a2
line wrap: on
line source
1 #+title: Building a Body
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulating a body (movement, touch, proprioception) in jMonkeyEngine3.
5 #+SETUPFILE: ../../aurellem/org/setup.org
6 #+INCLUDE: ../../aurellem/org/level-0.org
8 * Design Constraints
10 I use [[www.blender.org/][blender]] to design bodies. The design of the bodies is
11 determined by the requirements of the AI that will use them. The
12 bodies must be easy for an AI to sense and control, and they must be
13 relatively simple for jMonkeyEngine to compute.
15 # I'm a secret test! :P
16 ** Bag of Bones
18 How to create such a body? One option I ultimately rejected is to use
19 blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea would have been to define a mesh
20 which describes the creature's entire body. To this you add an
21 skeleton which deforms this mesh. This technique is used extensively
22 to model humans and create realistic animations. It is hard to use for
23 my purposes because it is difficult to update the creature's Physics
24 Collision Mesh in tandem with its Geometric Mesh under the influence
25 of the armature. Without this the creature will not be able to grab
26 things in its environment, and it won't be able to tell where its
27 physical body is by using its eyes. Also, armatures do not specify
28 any rotational limits for a joint, making it hard to model elbows,
29 shoulders, etc.
31 ** EVE
33 Instead of using the human-like "deformable bag of bones" approach, I
34 decided to base my body plans on the robot EVE from the movie wall-E.
36 #+caption: EVE from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one.
37 [[../images/Eve.jpg]]
39 EVE's body is composed of several rigid components that are held
40 together by invisible joint constraints. This is what I mean by
41 "eve-like". The main reason that I use eve-style bodies is so that
42 there will be correspondence between the AI's vision and the physical
43 presence of its body. Each individual section is simulated by a
44 separate rigid body that corresponds exactly with its visual
45 representation and does not change. Sections are connected by
46 invisible joints that are well supported in jMonkeyEngine. Bullet, the
47 physics backend for jMonkeyEngine, can efficiently simulate hundreds
48 of rigid bodies connected by joints. Sections do not have to stay as
49 one piece forever; they can be dynamically replaced with multiple
50 sections to simulate splitting in two. This could be used to simulate
51 retractable claws or EVE's hands, which are able to coalesce into one
52 object in the movie.
54 * Solidifying the Body
56 Here is a hand designed eve-style in blender.
58 #+attr_html: width="755"
59 [[../images/hand-screenshot0.png]]
61 If we load it directly into jMonkeyEngine, we get this:
63 #+name: test-1
64 #+begin_src clojure
65 (def hand-path "Models/test-creature/hand.blend")
67 (defn hand [] (load-blender-model hand-path))
69 (defn setup [world]
70 (let [cam (.getCamera world)]
71 (println-repl cam)
72 (.setLocation
73 cam (Vector3f.
74 -6.9015837, 8.644911, 5.6043186))
75 (.setRotation
76 cam
77 (Quaternion.
78 0.14046453, 0.85894054, -0.34301838, 0.3533118)))
79 (light-up-everything world)
80 (.setTimer world (RatchetTimer. 60))
81 world)
83 (defn test-hand-1
84 ([] (test-hand-1 false))
85 ([record?]
86 (world (hand)
87 standard-debug-controls
88 (fn [world]
89 (if record?
90 (Capture/captureVideo
91 world
92 (File. "/home/r/proj/cortex/render/body/1")))
93 (setup world)) no-op)))
94 #+end_src
97 #+begin_src clojure :results silent
98 (.start (cortex.test.body/test-one))
99 #+end_src
101 #+begin_html
102 <div class="figure">
103 <center>
104 <video controls="controls" width="640">
105 <source src="../video/ghost-hand.ogg" type="video/ogg"
106 preload="none" poster="../images/aurellem-1280x480.png" />
107 </video>
108 <br> <a href="http://youtu.be/9LZpwTIhjzE"> YouTube </a>
109 </center>
110 <p>The hand model directly loaded from blender. It has no physical
111 presence in the simulation. </p>
112 </div>
113 #+end_html
115 You will notice that the hand has no physical presence -- it's a
116 hologram through which everything passes. Therefore, the first thing
117 to do is to make it solid. Blender has physics simulation on par with
118 jMonkeyEngine (they both use bullet as their physics backend), but it
119 can be difficult to translate between the two systems, so for now I
120 specify the mass of each object as meta-data in blender and construct
121 the physics shape based on the mesh in jMonkeyEngine.
123 #+name: body-1
124 #+begin_src clojure
125 (defn physical!
126 "Iterate through the nodes in creature and make them real physical
127 objects in the simulation."
128 [#^Node creature]
129 (dorun
130 (map
131 (fn [geom]
132 (let [physics-control
133 (RigidBodyControl.
134 (HullCollisionShape.
135 (.getMesh geom))
136 (if-let [mass (meta-data geom "mass")]
137 (do
138 (println-repl
139 "setting" (.getName geom) "mass to" (float mass))
140 (float mass))
141 (float 1)))]
142 (.addControl geom physics-control)))
143 (filter #(isa? (class %) Geometry )
144 (node-seq creature)))))
145 #+end_src
147 =physical!)= iterates through a creature's node structure, creating
148 CollisionShapes for each geometry with the mass specified in that
149 geometry's meta-data.
151 #+name: test-2
152 #+begin_src clojure
153 (in-ns 'cortex.test.body)
155 (def gravity-control
156 {"key-g" (fn [world _]
157 (set-gravity world (Vector3f. 0 -9.81 0)))
158 "key-u" (fn [world _] (set-gravity world Vector3f/ZERO))})
161 (defn floor []
162 (box 10 3 10 :position (Vector3f. 0 -10 0)
163 :color ColorRGBA/Gray :mass 0))
165 (defn test-hand-2
166 ([] (test-hand-2 false))
167 ([record?]
168 (world
169 (nodify
170 [(doto (hand)
171 (physical!))
172 (floor)])
173 (merge standard-debug-controls gravity-control)
174 (fn [world]
175 (if record?
176 (Capture/captureVideo
177 world (File. "/home/r/proj/cortex/render/body/2")))
178 (set-gravity world Vector3f/ZERO)
179 (setup world))
180 no-op)))
181 #+end_src
183 #+begin_html
184 <div class="figure">
185 <center>
186 <video controls="controls" width="640">
187 <source src="../video/crumbly-hand.ogg" type="video/ogg"
188 preload="none" poster="../images/aurellem-1280x480.png" />
189 </video>
190 <br> <a href="http://youtu.be/GEA1SACwpPg"> YouTube </a>
191 </center>
192 <p>The hand now has a physical presence, but there is nothing to hold
193 it together.</p>
194 </div>
195 #+end_html
197 Now that's some progress.
199 * Joints
201 Obviously, an AI is not going to be doing much while lying in pieces
202 on the floor. So, the next step to making a proper body is to connect
203 those pieces together with joints. jMonkeyEngine has a large array of
204 joints available via bullet, such as Point2Point, Cone, Hinge, and a
205 generic Six Degree of Freedom joint, with or without spring
206 restitution.
208 Although it should be possible to specify the joints using blender's
209 physics system, and then automatically import them with jMonkeyEngine,
210 the support isn't there yet, and there are a few problems with bullet
211 itself that need to be solved before it can happen.
213 So, I will use the same system for specifying joints as I will do for
214 some senses. Each joint is specified by an empty node whose parent
215 has the name "joints". Their orientation and meta-data determine what
216 joint is created.
218 #+attr_html: width="755"
219 #+caption: Joints hack in blender. Each empty node here will be transformed into a joint in jMonkeyEngine
220 [[../images/hand-screenshot1.png]]
222 The empty node in the upper right, highlighted in yellow, is the
223 parent node of all the empties which represent joints. The following
224 functions must do three things to translate these into real joints:
226 - Find the children of the "joints" node.
227 - Determine the two spatials the joint it meant to connect.
228 - Create the joint based on the meta-data of the empty node.
230 ** Finding the Joints
232 The higher order function =sense-nodes= from =cortex.sense= simplifies
233 the first task.
235 #+name: joints-2
236 #+begin_src clojure
237 (defvar
238 ^{:arglists '([creature])}
239 joints
240 (sense-nodes "joints")
241 "Return the children of the creature's \"joints\" node.")
242 #+end_src
244 ** Joint Targets and Orientation
246 This technique for finding a joint's targets is very similar to
247 =cortex.sense/closest-node=. A small cube, centered around the
248 empty-node, grows exponentially until it intersects two /physical/
249 objects. The objects are ordered according to the joint's rotation,
250 with the first one being the object that has more negative coordinates
251 in the joint's reference frame. Since the objects must be physical,
252 the empty-node itself escapes detection. Because the objects must be
253 physical, =joint-targets= must be called /after/ =physical!= is
254 called.
256 #+name: joints-3
257 #+begin_src clojure
258 (defn joint-targets
259 "Return the two closest two objects to the joint object, ordered
260 from bottom to top according to the joint's rotation."
261 [#^Node parts #^Node joint]
262 (loop [radius (float 0.01)]
263 (let [results (CollisionResults.)]
264 (.collideWith
265 parts
266 (BoundingBox. (.getWorldTranslation joint)
267 radius radius radius) results)
268 (let [targets
269 (distinct
270 (map #(.getGeometry %) results))]
271 (if (>= (count targets) 2)
272 (sort-by
273 #(let [joint-ref-frame-position
274 (jme-to-blender
275 (.mult
276 (.inverse (.getWorldRotation joint))
277 (.subtract (.getWorldTranslation %)
278 (.getWorldTranslation joint))))]
279 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
280 (take 2 targets))
281 (recur (float (* radius 2))))))))
282 #+end_src
284 ** Generating Joints
286 This section of code iterates through all the different ways of
287 specifying joints using blender meta-data and converts each one to the
288 appropriate jMonkeyEngine joint.
290 #+name: joints-4
291 #+begin_src clojure
292 (defmulti joint-dispatch
293 "Translate blender pseudo-joints into real JME joints."
294 (fn [constraints & _]
295 (:type constraints)))
297 (defmethod joint-dispatch :point
298 [constraints control-a control-b pivot-a pivot-b rotation]
299 (println-repl "creating POINT2POINT joint")
300 ;; bullet's point2point joints are BROKEN, so we must use the
301 ;; generic 6DOF joint instead of an actual Point2Point joint!
303 ;; should be able to do this:
304 (comment
305 (Point2PointJoint.
306 control-a
307 control-b
308 pivot-a
309 pivot-b))
311 ;; but instead we must do this:
312 (println-repl "substituting 6DOF joint for POINT2POINT joint!")
313 (doto
314 (SixDofJoint.
315 control-a
316 control-b
317 pivot-a
318 pivot-b
319 false)
320 (.setLinearLowerLimit Vector3f/ZERO)
321 (.setLinearUpperLimit Vector3f/ZERO)))
323 (defmethod joint-dispatch :hinge
324 [constraints control-a control-b pivot-a pivot-b rotation]
325 (println-repl "creating HINGE joint")
326 (let [axis
327 (if-let
328 [axis (:axis constraints)]
329 axis
330 Vector3f/UNIT_X)
331 [limit-1 limit-2] (:limit constraints)
332 hinge-axis
333 (.mult
334 rotation
335 (blender-to-jme axis))]
336 (doto
337 (HingeJoint.
338 control-a
339 control-b
340 pivot-a
341 pivot-b
342 hinge-axis
343 hinge-axis)
344 (.setLimit limit-1 limit-2))))
346 (defmethod joint-dispatch :cone
347 [constraints control-a control-b pivot-a pivot-b rotation]
348 (let [limit-xz (:limit-xz constraints)
349 limit-xy (:limit-xy constraints)
350 twist (:twist constraints)]
352 (println-repl "creating CONE joint")
353 (println-repl rotation)
354 (println-repl
355 "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0)))
356 (println-repl
357 "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0)))
358 (println-repl
359 "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1)))
360 (doto
361 (ConeJoint.
362 control-a
363 control-b
364 pivot-a
365 pivot-b
366 rotation
367 rotation)
368 (.setLimit (float limit-xz)
369 (float limit-xy)
370 (float twist)))))
372 (defn connect
373 "Create a joint between 'obj-a and 'obj-b at the location of
374 'joint. The type of joint is determined by the metadata on 'joint.
376 Here are some examples:
377 {:type :point}
378 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
379 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
381 {:type :cone :limit-xz 0]
382 :limit-xy 0]
383 :twist 0]} (use XZY rotation mode in blender!)"
384 [#^Node obj-a #^Node obj-b #^Node joint]
385 (let [control-a (.getControl obj-a RigidBodyControl)
386 control-b (.getControl obj-b RigidBodyControl)
387 joint-center (.getWorldTranslation joint)
388 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
389 pivot-a (world-to-local obj-a joint-center)
390 pivot-b (world-to-local obj-b joint-center)]
392 (if-let [constraints
393 (map-vals
394 eval
395 (read-string
396 (meta-data joint "joint")))]
397 ;; A side-effect of creating a joint registers
398 ;; it with both physics objects which in turn
399 ;; will register the joint with the physics system
400 ;; when the simulation is started.
401 (do
402 (println-repl "creating joint between"
403 (.getName obj-a) "and" (.getName obj-b))
404 (joint-dispatch constraints
405 control-a control-b
406 pivot-a pivot-b
407 joint-rotation))
408 (println-repl "could not find joint meta-data!"))))
409 #+end_src
411 Creating joints is now a matter of applying =connect= to each joint
412 node.
414 #+name: joints-5
415 #+begin_src clojure
416 (defn joints!
417 "Connect the solid parts of the creature with physical joints. The
418 joints are taken from the \"joints\" node in the creature."
419 [#^Node creature]
420 (dorun
421 (map
422 (fn [joint]
423 (let [[obj-a obj-b] (joint-targets creature joint)]
424 (connect obj-a obj-b joint)))
425 (joints creature))))
426 #+end_src
428 ** Round 3
430 Now we can test the hand in all its glory.
432 #+name: test-3
433 #+begin_src clojure
434 (in-ns 'cortex.test.body)
436 (def debug-control
437 {"key-h" (fn [world val]
438 (if val (enable-debug world)))})
440 (defn test-hand-3
441 ([] (test-hand-3 false))
442 ([record?]
443 (world
444 (nodify
445 [(doto (hand)
446 (physical!)
447 (joints!))
448 (floor)])
449 (merge standard-debug-controls debug-control
450 gravity-control)
451 (comp
452 #(Capture/captureVideo
453 % (File. "/home/r/proj/cortex/render/body/3"))
454 #(do (set-gravity % Vector3f/ZERO) %)
455 setup)
456 no-op)))
457 #+end_src
459 =physical!= makes the hand solid, then =joints!= connects each
460 piece together.
462 #+begin_html
463 <div class="figure">
464 <center>
465 <video controls="controls" width="640">
466 <source src="../video/full-hand.ogg" type="video/ogg"
467 preload="none" poster="../images/aurellem-1280x480.png" />
468 </video>
469 <br> <a href="http://youtu.be/4affLfwSPP4"> YouTube </a>
470 </center>
471 <p>Now the hand is physical and has joints.</p>
472 </div>
473 #+end_html
475 The joints are visualized as green connections between each segment
476 for debug purposes. You can see that they correspond to the empty
477 nodes in the blender file.
479 * Wrap-Up!
481 It is convenient to combine =physical!= and =joints!= into one
482 function that completely creates the creature's physical body.
484 #+name: joints-6
485 #+begin_src clojure
486 (defn body!
487 "Endow the creature with a physical body connected with joints. The
488 particulars of the joints and the masses of each body part are
489 determined in blender."
490 [#^Node creature]
491 (physical! creature)
492 (joints! creature))
493 #+end_src
495 * The Worm
497 Going forward, I will use a model that is less complicated than the
498 hand. It has two segments and one joint, and I call it the worm. All
499 of the senses described in the following posts will be applied to this
500 worm.
502 #+name: test-4
503 #+begin_src clojure
504 (in-ns 'cortex.test.body)
506 (defn worm []
507 (load-blender-model
508 "Models/test-creature/worm.blend"))
510 (defn test-worm
511 ([] (test-worm false))
512 ([record?]
513 (let [timer (RatchetTimer. 60)]
514 (world
515 (nodify
516 [(doto (worm)
517 (body!))
518 (floor)])
519 (merge standard-debug-controls debug-control)
520 #(do
521 (speed-up %)
522 (light-up-everything %)
523 (.setTimer % timer)
524 (cortex.util/display-dialated-time % timer)
525 (if record?
526 (Capture/captureVideo
527 % (File. "/home/r/proj/cortex/render/body/4"))))
528 no-op))))
529 #+end_src
531 #+begin_html
532 <div class="figure">
533 <center>
534 <video controls="controls" width="640">
535 <source src="../video/worm-1.ogg" type="video/ogg"
536 preload="none" poster="../images/aurellem-1280x480.png" />
537 </video>
538 <br> <a href="http://youtu.be/rFVXI0T3iSE"> YouTube </a>
539 </center>
540 <p>This worm model will be the platform onto which future senses will
541 be grafted.</p>
542 </div>
543 #+end_html
545 * Headers
546 #+name: body-header
547 #+begin_src clojure
548 (ns cortex.body
549 "Assemble a physical creature using the definitions found in a
550 specially prepared blender file. Creates rigid bodies and joints so
551 that a creature can have a physical presence in the simulation."
552 {:author "Robert McIntyre"}
553 (:use (cortex world util sense))
554 (:use clojure.contrib.def)
555 (:import
556 (com.jme3.math Vector3f Quaternion Vector2f Matrix3f)
557 (com.jme3.bullet.joints
558 SixDofJoint Point2PointJoint HingeJoint ConeJoint)
559 com.jme3.bullet.control.RigidBodyControl
560 com.jme3.collision.CollisionResults
561 com.jme3.bounding.BoundingBox
562 com.jme3.scene.Node
563 com.jme3.scene.Geometry
564 com.jme3.bullet.collision.shapes.HullCollisionShape))
565 #+end_src
567 #+name: test-header
568 #+begin_src clojure
569 (ns cortex.test.body
570 (:use (cortex world util body))
571 (:import
572 (com.aurellem.capture Capture RatchetTimer)
573 (com.jme3.math Quaternion Vector3f ColorRGBA)
574 java.io.File))
575 #+end_src
577 * Source
578 - [[../src/cortex/body.clj][cortex.body]]
579 - [[../src/cortex/test/body.clj][cortex.test.body]]
580 - [[../assets/Models/test-creature/hand.blend][hand.blend]]
581 - [[../assets/Models/test-creature/palm.png][UV-map-1]]
582 - [[../assets/Models/test-creature/worm.blend][worm.blend]]
583 - [[../assets/Models/test-creature/retina-small.png][UV-map-1]]
584 - [[../assets/Models/test-creature/tip.png][UV-map-2]]
585 #+html: <ul> <li> <a href="../org/body.org">This org file</a> </li> </ul>
586 - [[http://hg.bortreb.com ][source-repository]]
588 * Next
589 The body I have made here exists without any senses or effectors. In
590 the [[./vision.org][next post]], I'll give the creature eyes.
592 * COMMENT Generate Source
593 #+begin_src clojure :tangle ../src/cortex/body.clj
594 <<body-header>>
595 <<body-1>>
596 <<joints-2>>
597 <<joints-3>>
598 <<joints-4>>
599 <<joints-5>>
600 <<joints-6>>
601 #+end_src
603 #+begin_src clojure :tangle ../src/cortex/test/body.clj
604 <<test-header>>
605 <<test-1>>
606 <<test-2>>
607 <<test-3>>
608 <<test-4>>
609 #+end_src