view org/body.org @ 208:30bd811c5dcb

remove blender clutter from screenshots
author Robert McIntyre <rlm@mit.edu>
date Thu, 09 Feb 2012 04:36:31 -0700
parents bb3b75bf1664
children e49c690d2bcf
line wrap: on
line source
1 #+title: Building a Body
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulating a body (movement, touch, propioception) in jMonkeyEngine3.
5 #+SETUPFILE: ../../aurellem/org/setup.org
6 #+INCLUDE: ../../aurellem/org/level-0.org
8 * Design Constraints
10 I use [[www.blender.org/][blender]] to design bodies. The design of the bodies is
11 determined by the requirements of the AI that will use them. The
12 bodies must be easy for an AI to sense and control, and they must be
13 relatively simple for jMonkeyEngine to compute.
15 ** Bag of Bones
17 How to create such a body? One option I ultimately rejected is to use
18 blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea would have been to define a mesh
19 which describes the creature's entire body. To this you add an
20 (skeleton) which deforms this mesh. This technique is used extensively
21 to model humans and create realistic animations. It is hard to use for
22 my purposes because it is difficult to update the creature's Physics
23 Collision Mesh in tandem with its Geometric Mesh under the influence
24 of the armature. Withouth this the creature will not be able to grab
25 things in its environment, and it won't be able to tell where its
26 physical body is by using its eyes. Also, armatures do not specify
27 any rotational limits for a joint, making it hard to model elbows,
28 shoulders, etc.
30 ** EVE
32 Instead of using the human-like "deformable bag of bones" approach, I
33 decided to base my body plans on the robot EVE from the movie wall-E.
35 #+caption: EVE from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one.
36 [[../images/Eve.jpg]]
38 EVE's body is composed of several rigid components that are held
39 together by invisible joint constraints. This is what I mean by
40 "eve-like". The main reason that I use eve-style bodies is so that
41 there will be correspondence between the AI's vision and the physical
42 presence of its body. Each individual section is simulated by a
43 separate rigid body that corresponds exactly with its visual
44 representation and does not change. Sections are connected by
45 invisible joints that are well supported in jMonkyeEngine. Bullet, the
46 physics backend for jMonkeyEngine, can efficiently simulate hundreds
47 of rigid bodies connected by joints. Sections do not have to stay as
48 one piece forever; they can be dynamically replaced with multiple
49 sections to simulate splitting in two. This could be used to simulate
50 retractable claws or EVE's hands, which could coalece into one object
51 in the movie.
53 * Solidifying the Body
55 Here is a hand designed eve-style in blender.
57 #+attr_html: width="755"
58 [[../images/hand-screenshot0.png]]
60 If we load it directly into jMonkeyEngine, we get this:
62 #+name: test-1
63 #+begin_src clojure
64 (def hand-path "Models/test-creature/hand.blend")
66 (defn hand [] (load-blender-model hand-path))
68 (defn setup [world]
69 (let [cam (.getCamera world)]
70 (println-repl cam)
71 (.setLocation
72 cam (Vector3f.
73 -6.9015837, 8.644911, 5.6043186))
74 (.setRotation
75 cam
76 (Quaternion.
77 0.14046453, 0.85894054, -0.34301838, 0.3533118)))
78 (light-up-everything world)
79 (.setTimer world (RatchetTimer. 60))
80 world)
82 (defn test-one []
83 (world (hand)
84 standard-debug-controls
85 (comp
86 #(Capture/captureVideo
87 % (File. "/home/r/proj/cortex/render/body/1"))
88 setup)
89 no-op))
90 #+end_src
93 #+begin_src clojure :results silent
94 (.start (cortex.test.body/test-one))
95 #+end_src
97 #+begin_html
98 <div class="figure">
99 <center>
100 <video controls="controls" width="640">
101 <source src="../video/ghost-hand.ogg" type="video/ogg"
102 preload="none" poster="../images/aurellem-1280x480.png" />
103 </video>
104 </center>
105 <p>The hand model directly loaded from blender. It has no physical
106 presense in the simulation. </p>
107 </div>
108 #+end_html
110 You will notice that the hand has no physical presence -- it's a
111 hologram through which everything passes. Therefore, the first thing
112 to do is to make it solid. Blender has physics simulation on par with
113 jMonkeyEngine (they both use bullet as their physics backend), but it
114 can be difficult to translate between the two systems, so for now I
115 specify the mass of each object in blender and construct the physics
116 shape based on the mesh in jMonkeyEngine.
118 #+name: body-1
119 #+begin_src clojure
120 (defn physical!
121 "Iterate through the nodes in creature and make them real physical
122 objects in the simulation."
123 [#^Node creature]
124 (dorun
125 (map
126 (fn [geom]
127 (let [physics-control
128 (RigidBodyControl.
129 (HullCollisionShape.
130 (.getMesh geom))
131 (if-let [mass (meta-data geom "mass")]
132 (do
133 (println-repl
134 "setting" (.getName geom) "mass to" (float mass))
135 (float mass))
136 (float 1)))]
137 (.addControl geom physics-control)))
138 (filter #(isa? (class %) Geometry )
139 (node-seq creature)))))
140 #+end_src
142 =(physical!)= iterates through a creature's node structure, creating
143 CollisionShapes for each geometry with the mass specified in that
144 geometry's meta-data.
146 #+name: test-2
147 #+begin_src clojure
148 (in-ns 'cortex.test.body)
150 (def normal-gravity
151 {"key-g" (fn [world _]
152 (set-gravity world (Vector3f. 0 -9.81 0)))})
154 (defn floor []
155 (box 10 3 10 :position (Vector3f. 0 -10 0)
156 :color ColorRGBA/Gray :mass 0))
158 (defn test-two []
159 (world (nodify
160 [(doto (hand)
161 (physical!))
162 (floor)])
163 (merge standard-debug-controls normal-gravity)
164 (comp
165 #(Capture/captureVideo
166 % (File. "/home/r/proj/cortex/render/body/2"))
167 #(do (set-gravity % Vector3f/ZERO) %)
168 setup)
169 no-op))
170 #+end_src
172 #+begin_html
173 <div class="figure">
174 <center>
175 <video controls="controls" width="640">
176 <source src="../video/crumbly-hand.ogg" type="video/ogg"
177 preload="none" poster="../images/aurellem-1280x480.png" />
178 </video>
179 </center>
180 <p>The hand now has a physical presence, but there is nothing to hold
181 it together.</p>
182 </div>
183 #+end_html
185 Now that's some progress.
187 * Joints
189 Obviously, an AI is not going to be doing much just lying in pieces on
190 the floor. So, the next step to making a proper body is to connect
191 those pieces together with joints. jMonkeyEngine has a large array of
192 joints available via bullet, such as Point2Point, Cone, Hinge, and a
193 generic Six Degree of Freedom joint, with or without spring
194 restitution.
196 Although it should be possible to specify the joints using blender's
197 physics system, and then automatically import them with jMonkeyEngine,
198 the support isn't there yet, and there are a few problems with bullet
199 itself that need to be solved before it can happen.
201 So, I will use the same system for specifying joints as I will do for
202 some senses. Each joint is specified by an empty node whose parent
203 has the name "joints". Their orientation and meta-data determine what
204 joint is created.
206 #+attr_html: width="755"
207 #+caption: joints hack in blender. Each empty node here will be transformed into a joint in jMonkeyEngine
208 [[../images/hand-screenshot1.png]]
210 The empty node in the upper right, highlighted in yellow, is the
211 parent node of all the emptys which represent joints. The following
212 functions must do three things to translate these into real joints:
214 - Find the children of the "joints" node.
215 - Determine the two spatials the joint it meant to connect.
216 - Create the joint based on the meta-data of the empty node.
218 ** Finding the Joints
219 #+name: joints-2
220 #+begin_src clojure
221 (defvar
222 ^{:arglists '([creature])}
223 joints
224 (sense-nodes "joints")
225 "Return the children of the creature's \"joints\" node.")
226 #+end_src
228 The higher order function =(sense-nodes)= from cortex.sense makes our
229 first task very easy.
231 ** Joint Targets and Orientation
233 This technique for finding a joint's targets is very similiar to
234 =(cortex.sense/closest-node)=. A small cube, centered around the
235 empty-node, grows exponentially until it intersects two /physical/
236 objects. The objects are ordered according to the joint's rotation,
237 with the first one being the object that has more negative coordinates
238 in the joint's reference frame. Since the objects must be physical,
239 the empty-node itself escapes detection. Because the objects must be
240 physical, =(joint-targets)= must be called /after/ =(physical!)= is
241 called.
243 #+name: joints-3
244 #+begin_src clojure
245 (defn joint-targets
246 "Return the two closest two objects to the joint object, ordered
247 from bottom to top according to the joint's rotation."
248 [#^Node parts #^Node joint]
249 (loop [radius (float 0.01)]
250 (let [results (CollisionResults.)]
251 (.collideWith
252 parts
253 (BoundingBox. (.getWorldTranslation joint)
254 radius radius radius)
255 results)
256 (let [targets
257 (distinct
258 (map #(.getGeometry %) results))]
259 (if (>= (count targets) 2)
260 (sort-by
261 #(let [v
262 (jme-to-blender
263 (.mult
264 (.inverse (.getWorldRotation joint))
265 (.subtract (.getWorldTranslation %)
266 (.getWorldTranslation joint))))]
267 (println-repl (.getName %) ":" v)
268 (.dot (Vector3f. 1 1 1)
269 v))
270 (take 2 targets))
271 (recur (float (* radius 2))))))))
272 #+end_src
274 ** Generating Joints
276 This long chunk of code iterates through all the different ways of
277 specifying joints using blender meta-data and converts each one to the
278 appropriate jMonkyeEngine joint.
280 #+name: joints-4
281 #+begin_src clojure
282 (defmulti joint-dispatch
283 "Translate blender pseudo-joints into real JME joints."
284 (fn [constraints & _]
285 (:type constraints)))
287 (defmethod joint-dispatch :point
288 [constraints control-a control-b pivot-a pivot-b rotation]
289 (println-repl "creating POINT2POINT joint")
290 ;; bullet's point2point joints are BROKEN, so we must use the
291 ;; generic 6DOF joint instead of an actual Point2Point joint!
293 ;; should be able to do this:
294 (comment
295 (Point2PointJoint.
296 control-a
297 control-b
298 pivot-a
299 pivot-b))
301 ;; but instead we must do this:
302 (println-repl "substuting 6DOF joint for POINT2POINT joint!")
303 (doto
304 (SixDofJoint.
305 control-a
306 control-b
307 pivot-a
308 pivot-b
309 false)
310 (.setLinearLowerLimit Vector3f/ZERO)
311 (.setLinearUpperLimit Vector3f/ZERO)))
313 (defmethod joint-dispatch :hinge
314 [constraints control-a control-b pivot-a pivot-b rotation]
315 (println-repl "creating HINGE joint")
316 (let [axis
317 (if-let
318 [axis (:axis constraints)]
319 axis
320 Vector3f/UNIT_X)
321 [limit-1 limit-2] (:limit constraints)
322 hinge-axis
323 (.mult
324 rotation
325 (blender-to-jme axis))]
326 (doto
327 (HingeJoint.
328 control-a
329 control-b
330 pivot-a
331 pivot-b
332 hinge-axis
333 hinge-axis)
334 (.setLimit limit-1 limit-2))))
336 (defmethod joint-dispatch :cone
337 [constraints control-a control-b pivot-a pivot-b rotation]
338 (let [limit-xz (:limit-xz constraints)
339 limit-xy (:limit-xy constraints)
340 twist (:twist constraints)]
342 (println-repl "creating CONE joint")
343 (println-repl rotation)
344 (println-repl
345 "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0)))
346 (println-repl
347 "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0)))
348 (println-repl
349 "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1)))
350 (doto
351 (ConeJoint.
352 control-a
353 control-b
354 pivot-a
355 pivot-b
356 rotation
357 rotation)
358 (.setLimit (float limit-xz)
359 (float limit-xy)
360 (float twist)))))
362 (defn connect
363 "Create a joint between 'obj-a and 'obj-b at the location of
364 'joint. The type of joint is determined by the metadata on 'joint.
366 Here are some examples:
367 {:type :point}
368 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
369 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
371 {:type :cone :limit-xz 0]
372 :limit-xy 0]
373 :twist 0]} (use XZY rotation mode in blender!)"
374 [#^Node obj-a #^Node obj-b #^Node joint]
375 (let [control-a (.getControl obj-a RigidBodyControl)
376 control-b (.getControl obj-b RigidBodyControl)
377 joint-center (.getWorldTranslation joint)
378 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
379 pivot-a (world-to-local obj-a joint-center)
380 pivot-b (world-to-local obj-b joint-center)]
382 (if-let [constraints
383 (map-vals
384 eval
385 (read-string
386 (meta-data joint "joint")))]
387 ;; A side-effect of creating a joint registers
388 ;; it with both physics objects which in turn
389 ;; will register the joint with the physics system
390 ;; when the simulation is started.
391 (do
392 (println-repl "creating joint between"
393 (.getName obj-a) "and" (.getName obj-b))
394 (joint-dispatch constraints
395 control-a control-b
396 pivot-a pivot-b
397 joint-rotation))
398 (println-repl "could not find joint meta-data!"))))
399 #+end_src
401 Creating joints is now a matter applying =(connect)= to each joint
402 node.
404 #+name: joints-5
405 #+begin_src clojure
406 (defn joints!
407 "Connect the solid parts of the creature with physical joints. The
408 joints are taken from the \"joints\" node in the creature."
409 [#^Node creature]
410 (dorun
411 (map
412 (fn [joint]
413 (let [[obj-a obj-b] (joint-targets creature joint)]
414 (connect obj-a obj-b joint)))
415 (joints creature))))
416 #+end_src
419 ** Round 3
421 Now we can test the hand in all its glory.
423 #+name: test-3
424 #+begin_src clojure
425 (in-ns 'cortex.test.body)
427 (def debug-control
428 {"key-h" (fn [world val]
429 (if val (enable-debug world)))
430 "key-u" (fn [world _] (set-gravity world Vector3f/ZERO))})
432 (defn test-three []
433 (world (nodify
434 [(doto (hand)
435 (physical!)
436 (joints!))
437 (floor)])
438 (merge standard-debug-controls debug-control
439 normal-gravity)
440 (comp
441 #(Capture/captureVideo
442 % (File. "/home/r/proj/cortex/render/body/3"))
443 #(do (set-gravity % Vector3f/ZERO) %)
444 setup)
445 no-op))
446 #+end_src
448 =(physical!)= makes the hand solid, then =(joints!)= connects each
449 piece together.
451 #+begin_html
452 <div class="figure">
453 <center>
454 <video controls="controls" width="640">
455 <source src="../video/full-hand.ogg" type="video/ogg"
456 preload="none" poster="../images/aurellem-1280x480.png" />
457 </video>
458 </center>
459 <p>Now the hand is physical and has joints.</p>
460 </div>
461 #+end_html
463 The joints are visualized as green connections between each segment
464 for debug purposes. You can see that they correspond to the empty
465 nodes in the blender file.
467 * Wrap-Up!
469 It is convienent to combine =(physical!)= and =(joints!)= into one
470 function that completely creates the creature's physical body.
472 #+name: joints-6
473 #+begin_src clojure
474 (defn body!
475 "Endow the creature with a physical body connected with joints. The
476 particulars of the joints and the masses of each pody part are
477 determined in blender."
478 [#^Node creature]
479 (physical! creature)
480 (joints! creature))
481 #+end_src
483 * The Worm
485 Going forward, I will use a model that is less complicated than the
486 hand. It has two segments and one joint, and I call it the worm. All
487 of the senses described in the following posts will be applied to this
488 worm.
490 #+name: test-4
491 #+begin_src clojure
492 (in-ns 'cortex.test.body)
494 (defn worm-1 []
495 (let [timer (RatchetTimer. 60)]
496 (world
497 (nodify
498 [(doto
499 (load-blender-model
500 "Models/test-creature/worm.blend")
501 (body!))
502 (floor)])
503 (merge standard-debug-controls debug-control)
504 #(do
505 (speed-up %)
506 (light-up-everything %)
507 (.setTimer % timer)
508 (cortex.util/display-dialated-time % timer)
509 (Capture/captureVideo
510 % (File. "/home/r/proj/cortex/render/body/4")))
511 no-op)))
512 #+end_src
514 #+begin_html
515 <div class="figure">
516 <center>
517 <video controls="controls" width="640">
518 <source src="../video/worm-1.ogg" type="video/ogg"
519 preload="none" poster="../images/aurellem-1280x480.png" />
520 </video>
521 </center>
522 <p>This worm model will be the platform onto which future senses will
523 be grafted.</p>
524 </div>
525 #+end_html
527 * Bookkeeping
529 Headers; here for completeness.
531 #+name: body-header
532 #+begin_src clojure
533 (ns cortex.body
534 "Assemble a physical creature using the definitions found in a
535 specially prepared blender file. Creates rigid bodies and joints so
536 that a creature can have a physical presense in the simulation."
537 {:author "Robert McIntyre"}
538 (:use (cortex world util sense))
539 (:use clojure.contrib.def)
540 (:import
541 (com.jme3.math Vector3f Quaternion Vector2f Matrix3f)
542 (com.jme3.bullet.joints
543 SixDofJoint Point2PointJoint HingeJoint ConeJoint)
544 com.jme3.bullet.control.RigidBodyControl
545 com.jme3.collision.CollisionResults
546 com.jme3.bounding.BoundingBox
547 com.jme3.scene.Node
548 com.jme3.scene.Geometry
549 com.jme3.bullet.collision.shapes.HullCollisionShape))
550 #+end_src
552 #+name: test-header
553 #+begin_src clojure
554 (ns cortex.test.body
555 (:use (cortex world util body))
556 (:import
557 (com.aurellem.capture Capture RatchetTimer)
558 (com.jme3.math Quaternion Vector3f ColorRGBA)
559 java.io.File))
560 #+end_src
562 * Source
563 - [[../src/cortex/body.clj][cortex.body]]
564 - [[../src/cortex/test/body.clj][cortex.test.body]]
565 - [[../assets/Models/test-creature/hand.blend][hand.blend]]
566 - [[../assets/Models/test-creature/worm.blend][worm.blend]]
567 - [[../assets/Models/test-creature/retina-small.png][UV-map-1]]
568 - [[../assets/Models/test-creature/tip.png][UV-map-2]]
570 * COMMENT Generate Source
571 #+begin_src clojure :tangle ../src/cortex/body.clj
572 <<body-header>>
573 <<body-1>>
574 <<joints-2>>
575 <<joints-3>>
576 <<joints-4>>
577 <<joints-5>>
578 <<joints-6>>
579 #+end_src
581 #+begin_src clojure :tangle ../src/cortex/test/body.clj
582 <<test-header>>
583 <<test-1>>
584 <<test-2>>
585 <<test-3>>
586 <<test-4>>
587 #+end_src