rlm@202: #+title: Building a Body
rlm@0: #+author: Robert McIntyre
rlm@0: #+email: rlm@mit.edu
rlm@4: #+description: Simulating a body (movement, touch, propioception) in jMonkeyEngine3.
rlm@4: #+SETUPFILE: ../../aurellem/org/setup.org
rlm@4: #+INCLUDE: ../../aurellem/org/level-0.org
rlm@4:
rlm@202: * Design Constraints
rlm@202:
rlm@202: I use [[www.blender.org/][blender]] to design bodies. The design of the bodies is
rlm@202: determined by the requirements of the AI that will use them. The
rlm@202: bodies must be easy for an AI to sense and control, and they must be
rlm@202: relatively simple for jMonkeyEngine to compute.
rlm@202:
rlm@202: ** Bag of Bones
rlm@202:
rlm@202: How to create such a body? One option I ultimately rejected is to use
rlm@202: blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea would have been to define a mesh
rlm@202: which describes the creature's entire body. To this you add an
rlm@202: (skeleton) which deforms this mesh. This technique is used extensively
rlm@202: to model humans and create realistic animations. It is hard to use for
rlm@202: my purposes because it is difficult to update the creature's Physics
rlm@202: Collision Mesh in tandem with its Geometric Mesh under the influence
rlm@202: of the armature. Withouth this the creature will not be able to grab
rlm@202: things in its environment, and it won't be able to tell where its
rlm@202: physical body is by using its eyes. Also, armatures do not specify
rlm@202: any rotational limits for a joint, making it hard to model elbows,
rlm@202: shoulders, etc.
rlm@202:
rlm@202: ** EVE
rlm@202:
rlm@202: Instead of using the human-like "deformable bag of bones" approach, I
rlm@202: decided to base my body plans on the robot EVE from the movie wall-E.
rlm@202:
rlm@202: #+caption: EVE from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one.
rlm@202: [[../images/Eve.jpg]]
rlm@202:
rlm@204: EVE's body is composed of several rigid components that are held
rlm@204: together by invisible joint constraints. This is what I mean by
rlm@204: "eve-like". The main reason that I use eve-style bodies is so that
rlm@204: there will be correspondence between the AI's vision and the physical
rlm@204: presence of its body. Each individual section is simulated by a
rlm@204: separate rigid body that corresponds exactly with its visual
rlm@204: representation and does not change. Sections are connected by
rlm@204: invisible joints that are well supported in jMonkyeEngine. Bullet, the
rlm@204: physics backend for jMonkeyEngine, can efficiently simulate hundreds
rlm@204: of rigid bodies connected by joints. Sections do not have to stay as
rlm@204: one piece forever; they can be dynamically replaced with multiple
rlm@204: sections to simulate splitting in two. This could be used to simulate
rlm@209: retractable claws or EVE's hands, which are able to coalece into one
rlm@209: object in the movie.
rlm@202:
rlm@202: * Solidifying the Body
rlm@202:
rlm@202: Here is a hand designed eve-style in blender.
rlm@202:
rlm@203: #+attr_html: width="755"
rlm@202: [[../images/hand-screenshot0.png]]
rlm@202:
rlm@202: If we load it directly into jMonkeyEngine, we get this:
rlm@202:
rlm@205: #+name: test-1
rlm@202: #+begin_src clojure
rlm@202: (def hand-path "Models/test-creature/hand.blend")
rlm@202:
rlm@202: (defn hand [] (load-blender-model hand-path))
rlm@202:
rlm@202: (defn setup [world]
rlm@202: (let [cam (.getCamera world)]
rlm@202: (println-repl cam)
rlm@202: (.setLocation
rlm@202: cam (Vector3f.
rlm@202: -6.9015837, 8.644911, 5.6043186))
rlm@202: (.setRotation
rlm@202: cam
rlm@202: (Quaternion.
rlm@202: 0.14046453, 0.85894054, -0.34301838, 0.3533118)))
rlm@202: (light-up-everything world)
rlm@202: (.setTimer world (RatchetTimer. 60))
rlm@202: world)
rlm@202:
rlm@202: (defn test-one []
rlm@202: (world (hand)
rlm@202: standard-debug-controls
rlm@202: (comp
rlm@202: #(Capture/captureVideo
rlm@202: % (File. "/home/r/proj/cortex/render/body/1"))
rlm@202: setup)
rlm@202: no-op))
rlm@202: #+end_src
rlm@202:
rlm@202:
rlm@202: #+begin_src clojure :results silent
rlm@202: (.start (cortex.test.body/test-one))
rlm@202: #+end_src
rlm@202:
rlm@202: #+begin_html
rlm@203:
rlm@202: #+end_html
rlm@202:
rlm@202: You will notice that the hand has no physical presence -- it's a
rlm@204: hologram through which everything passes. Therefore, the first thing
rlm@202: to do is to make it solid. Blender has physics simulation on par with
rlm@202: jMonkeyEngine (they both use bullet as their physics backend), but it
rlm@202: can be difficult to translate between the two systems, so for now I
rlm@209: specify the mass of each object as meta-data in blender and construct
rlm@209: the physics shape based on the mesh in jMonkeyEngine.
rlm@202:
rlm@203: #+name: body-1
rlm@202: #+begin_src clojure
rlm@202: (defn physical!
rlm@202: "Iterate through the nodes in creature and make them real physical
rlm@202: objects in the simulation."
rlm@202: [#^Node creature]
rlm@202: (dorun
rlm@202: (map
rlm@202: (fn [geom]
rlm@202: (let [physics-control
rlm@202: (RigidBodyControl.
rlm@202: (HullCollisionShape.
rlm@202: (.getMesh geom))
rlm@202: (if-let [mass (meta-data geom "mass")]
rlm@202: (do
rlm@202: (println-repl
rlm@202: "setting" (.getName geom) "mass to" (float mass))
rlm@202: (float mass))
rlm@202: (float 1)))]
rlm@202: (.addControl geom physics-control)))
rlm@202: (filter #(isa? (class %) Geometry )
rlm@202: (node-seq creature)))))
rlm@202: #+end_src
rlm@202:
rlm@202: =(physical!)= iterates through a creature's node structure, creating
rlm@202: CollisionShapes for each geometry with the mass specified in that
rlm@202: geometry's meta-data.
rlm@202:
rlm@205: #+name: test-2
rlm@0: #+begin_src clojure
rlm@202: (in-ns 'cortex.test.body)
rlm@160:
rlm@209: (def gravity-control
rlm@202: {"key-g" (fn [world _]
rlm@209: (set-gravity world (Vector3f. 0 -9.81 0)))
rlm@209: "key-u" (fn [world _] (set-gravity world Vector3f/ZERO))})
rlm@209:
rlm@202:
rlm@202: (defn floor []
rlm@202: (box 10 3 10 :position (Vector3f. 0 -10 0)
rlm@202: :color ColorRGBA/Gray :mass 0))
rlm@202:
rlm@202: (defn test-two []
rlm@202: (world (nodify
rlm@202: [(doto (hand)
rlm@202: (physical!))
rlm@202: (floor)])
rlm@209: (merge standard-debug-controls gravity-control)
rlm@202: (comp
rlm@202: #(Capture/captureVideo
rlm@202: % (File. "/home/r/proj/cortex/render/body/2"))
rlm@202: #(do (set-gravity % Vector3f/ZERO) %)
rlm@202: setup)
rlm@202: no-op))
rlm@202: #+end_src
rlm@202:
rlm@202: #+begin_html
rlm@203:
rlm@202: #+end_html
rlm@202:
rlm@202: Now that's some progress.
rlm@202:
rlm@202: * Joints
rlm@202:
rlm@209: Obviously, an AI is not going to be doing much while lying in pieces
rlm@209: on the floor. So, the next step to making a proper body is to connect
rlm@202: those pieces together with joints. jMonkeyEngine has a large array of
rlm@202: joints available via bullet, such as Point2Point, Cone, Hinge, and a
rlm@202: generic Six Degree of Freedom joint, with or without spring
rlm@202: restitution.
rlm@202:
rlm@202: Although it should be possible to specify the joints using blender's
rlm@202: physics system, and then automatically import them with jMonkeyEngine,
rlm@202: the support isn't there yet, and there are a few problems with bullet
rlm@202: itself that need to be solved before it can happen.
rlm@202:
rlm@202: So, I will use the same system for specifying joints as I will do for
rlm@202: some senses. Each joint is specified by an empty node whose parent
rlm@202: has the name "joints". Their orientation and meta-data determine what
rlm@202: joint is created.
rlm@202:
rlm@203: #+attr_html: width="755"
rlm@209: #+caption: Joints hack in blender. Each empty node here will be transformed into a joint in jMonkeyEngine
rlm@202: [[../images/hand-screenshot1.png]]
rlm@202:
rlm@203: The empty node in the upper right, highlighted in yellow, is the
rlm@203: parent node of all the emptys which represent joints. The following
rlm@203: functions must do three things to translate these into real joints:
rlm@202:
rlm@203: - Find the children of the "joints" node.
rlm@203: - Determine the two spatials the joint it meant to connect.
rlm@203: - Create the joint based on the meta-data of the empty node.
rlm@202:
rlm@203: ** Finding the Joints
rlm@209:
rlm@209: The higher order function =(sense-nodes)= from =cortex.sense= simplifies
rlm@209: the first task.
rlm@209:
rlm@203: #+name: joints-2
rlm@203: #+begin_src clojure
rlm@203: (defvar
rlm@203: ^{:arglists '([creature])}
rlm@203: joints
rlm@203: (sense-nodes "joints")
rlm@203: "Return the children of the creature's \"joints\" node.")
rlm@203: #+end_src
rlm@202:
rlm@203:
rlm@203: ** Joint Targets and Orientation
rlm@203:
rlm@203: This technique for finding a joint's targets is very similiar to
rlm@203: =(cortex.sense/closest-node)=. A small cube, centered around the
rlm@203: empty-node, grows exponentially until it intersects two /physical/
rlm@203: objects. The objects are ordered according to the joint's rotation,
rlm@203: with the first one being the object that has more negative coordinates
rlm@203: in the joint's reference frame. Since the objects must be physical,
rlm@203: the empty-node itself escapes detection. Because the objects must be
rlm@203: physical, =(joint-targets)= must be called /after/ =(physical!)= is
rlm@203: called.
rlm@203:
rlm@203: #+name: joints-3
rlm@202: #+begin_src clojure
rlm@135: (defn joint-targets
rlm@135: "Return the two closest two objects to the joint object, ordered
rlm@135: from bottom to top according to the joint's rotation."
rlm@135: [#^Node parts #^Node joint]
rlm@135: (loop [radius (float 0.01)]
rlm@135: (let [results (CollisionResults.)]
rlm@135: (.collideWith
rlm@135: parts
rlm@135: (BoundingBox. (.getWorldTranslation joint)
rlm@209: radius radius radius) results)
rlm@135: (let [targets
rlm@135: (distinct
rlm@135: (map #(.getGeometry %) results))]
rlm@135: (if (>= (count targets) 2)
rlm@135: (sort-by
rlm@209: #(let [joint-ref-frame-position
rlm@135: (jme-to-blender
rlm@135: (.mult
rlm@135: (.inverse (.getWorldRotation joint))
rlm@135: (.subtract (.getWorldTranslation %)
rlm@135: (.getWorldTranslation joint))))]
rlm@209: (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
rlm@135: (take 2 targets))
rlm@135: (recur (float (* radius 2))))))))
rlm@203: #+end_src
rlm@135:
rlm@203: ** Generating Joints
rlm@203:
rlm@209: This section of code iterates through all the different ways of
rlm@203: specifying joints using blender meta-data and converts each one to the
rlm@203: appropriate jMonkyeEngine joint.
rlm@203:
rlm@203: #+name: joints-4
rlm@203: #+begin_src clojure
rlm@160: (defmulti joint-dispatch
rlm@160: "Translate blender pseudo-joints into real JME joints."
rlm@160: (fn [constraints & _]
rlm@160: (:type constraints)))
rlm@141:
rlm@160: (defmethod joint-dispatch :point
rlm@160: [constraints control-a control-b pivot-a pivot-b rotation]
rlm@160: (println-repl "creating POINT2POINT joint")
rlm@160: ;; bullet's point2point joints are BROKEN, so we must use the
rlm@160: ;; generic 6DOF joint instead of an actual Point2Point joint!
rlm@141:
rlm@160: ;; should be able to do this:
rlm@160: (comment
rlm@160: (Point2PointJoint.
rlm@160: control-a
rlm@160: control-b
rlm@160: pivot-a
rlm@160: pivot-b))
rlm@141:
rlm@160: ;; but instead we must do this:
rlm@160: (println-repl "substuting 6DOF joint for POINT2POINT joint!")
rlm@160: (doto
rlm@160: (SixDofJoint.
rlm@160: control-a
rlm@160: control-b
rlm@160: pivot-a
rlm@160: pivot-b
rlm@160: false)
rlm@160: (.setLinearLowerLimit Vector3f/ZERO)
rlm@203: (.setLinearUpperLimit Vector3f/ZERO)))
rlm@160:
rlm@160: (defmethod joint-dispatch :hinge
rlm@160: [constraints control-a control-b pivot-a pivot-b rotation]
rlm@160: (println-repl "creating HINGE joint")
rlm@160: (let [axis
rlm@160: (if-let
rlm@160: [axis (:axis constraints)]
rlm@160: axis
rlm@160: Vector3f/UNIT_X)
rlm@160: [limit-1 limit-2] (:limit constraints)
rlm@160: hinge-axis
rlm@160: (.mult
rlm@160: rotation
rlm@160: (blender-to-jme axis))]
rlm@160: (doto
rlm@160: (HingeJoint.
rlm@160: control-a
rlm@160: control-b
rlm@160: pivot-a
rlm@160: pivot-b
rlm@160: hinge-axis
rlm@160: hinge-axis)
rlm@160: (.setLimit limit-1 limit-2))))
rlm@160:
rlm@160: (defmethod joint-dispatch :cone
rlm@160: [constraints control-a control-b pivot-a pivot-b rotation]
rlm@160: (let [limit-xz (:limit-xz constraints)
rlm@160: limit-xy (:limit-xy constraints)
rlm@160: twist (:twist constraints)]
rlm@160:
rlm@160: (println-repl "creating CONE joint")
rlm@160: (println-repl rotation)
rlm@160: (println-repl
rlm@160: "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0)))
rlm@160: (println-repl
rlm@160: "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0)))
rlm@160: (println-repl
rlm@160: "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1)))
rlm@160: (doto
rlm@160: (ConeJoint.
rlm@160: control-a
rlm@160: control-b
rlm@160: pivot-a
rlm@160: pivot-b
rlm@160: rotation
rlm@160: rotation)
rlm@160: (.setLimit (float limit-xz)
rlm@160: (float limit-xy)
rlm@160: (float twist)))))
rlm@160:
rlm@160: (defn connect
rlm@175: "Create a joint between 'obj-a and 'obj-b at the location of
rlm@175: 'joint. The type of joint is determined by the metadata on 'joint.
rlm@175:
rlm@175: Here are some examples:
rlm@160: {:type :point}
rlm@160: {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
rlm@160: (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@160:
rlm@160: {:type :cone :limit-xz 0]
rlm@160: :limit-xy 0]
rlm@160: :twist 0]} (use XZY rotation mode in blender!)"
rlm@160: [#^Node obj-a #^Node obj-b #^Node joint]
rlm@160: (let [control-a (.getControl obj-a RigidBodyControl)
rlm@160: control-b (.getControl obj-b RigidBodyControl)
rlm@160: joint-center (.getWorldTranslation joint)
rlm@160: joint-rotation (.toRotationMatrix (.getWorldRotation joint))
rlm@160: pivot-a (world-to-local obj-a joint-center)
rlm@160: pivot-b (world-to-local obj-b joint-center)]
rlm@160:
rlm@160: (if-let [constraints
rlm@160: (map-vals
rlm@160: eval
rlm@160: (read-string
rlm@160: (meta-data joint "joint")))]
rlm@160: ;; A side-effect of creating a joint registers
rlm@160: ;; it with both physics objects which in turn
rlm@160: ;; will register the joint with the physics system
rlm@160: ;; when the simulation is started.
rlm@160: (do
rlm@160: (println-repl "creating joint between"
rlm@160: (.getName obj-a) "and" (.getName obj-b))
rlm@160: (joint-dispatch constraints
rlm@160: control-a control-b
rlm@160: pivot-a pivot-b
rlm@160: joint-rotation))
rlm@160: (println-repl "could not find joint meta-data!"))))
rlm@203: #+end_src
rlm@160:
rlm@209: Creating joints is now a matter of applying =(connect)= to each joint
rlm@203: node.
rlm@160:
rlm@205: #+name: joints-5
rlm@203: #+begin_src clojure
rlm@175: (defn joints!
rlm@175: "Connect the solid parts of the creature with physical joints. The
rlm@175: joints are taken from the \"joints\" node in the creature."
rlm@175: [#^Node creature]
rlm@160: (dorun
rlm@160: (map
rlm@160: (fn [joint]
rlm@175: (let [[obj-a obj-b] (joint-targets creature joint)]
rlm@160: (connect obj-a obj-b joint)))
rlm@175: (joints creature))))
rlm@203: #+end_src
rlm@160:
rlm@203:
rlm@203: ** Round 3
rlm@203:
rlm@203: Now we can test the hand in all its glory.
rlm@203:
rlm@205: #+name: test-3
rlm@203: #+begin_src clojure
rlm@203: (in-ns 'cortex.test.body)
rlm@203:
rlm@203: (def debug-control
rlm@203: {"key-h" (fn [world val]
rlm@209: (if val (enable-debug world)))})
rlm@203:
rlm@203: (defn test-three []
rlm@203: (world (nodify
rlm@203: [(doto (hand)
rlm@205: (physical!)
rlm@205: (joints!))
rlm@203: (floor)])
rlm@203: (merge standard-debug-controls debug-control
rlm@209: gravity-control)
rlm@203: (comp
rlm@203: #(Capture/captureVideo
rlm@203: % (File. "/home/r/proj/cortex/render/body/3"))
rlm@203: #(do (set-gravity % Vector3f/ZERO) %)
rlm@203: setup)
rlm@203: no-op))
rlm@203: #+end_src
rlm@203:
rlm@203: =(physical!)= makes the hand solid, then =(joints!)= connects each
rlm@203: piece together.
rlm@203:
rlm@203: #+begin_html
rlm@203:
rlm@203: #+end_html
rlm@203:
rlm@203: The joints are visualized as green connections between each segment
rlm@203: for debug purposes. You can see that they correspond to the empty
rlm@203: nodes in the blender file.
rlm@203:
rlm@203: * Wrap-Up!
rlm@203:
rlm@203: It is convienent to combine =(physical!)= and =(joints!)= into one
rlm@203: function that completely creates the creature's physical body.
rlm@203:
rlm@205: #+name: joints-6
rlm@203: #+begin_src clojure
rlm@175: (defn body!
rlm@175: "Endow the creature with a physical body connected with joints. The
rlm@175: particulars of the joints and the masses of each pody part are
rlm@175: determined in blender."
rlm@175: [#^Node creature]
rlm@175: (physical! creature)
rlm@175: (joints! creature))
rlm@64: #+end_src
rlm@63:
rlm@205: * The Worm
rlm@205:
rlm@205: Going forward, I will use a model that is less complicated than the
rlm@205: hand. It has two segments and one joint, and I call it the worm. All
rlm@205: of the senses described in the following posts will be applied to this
rlm@205: worm.
rlm@205:
rlm@205: #+name: test-4
rlm@205: #+begin_src clojure
rlm@205: (in-ns 'cortex.test.body)
rlm@205:
rlm@205: (defn worm-1 []
rlm@205: (let [timer (RatchetTimer. 60)]
rlm@205: (world
rlm@205: (nodify
rlm@205: [(doto
rlm@205: (load-blender-model
rlm@205: "Models/test-creature/worm.blend")
rlm@205: (body!))
rlm@205: (floor)])
rlm@205: (merge standard-debug-controls debug-control)
rlm@205: #(do
rlm@205: (speed-up %)
rlm@205: (light-up-everything %)
rlm@205: (.setTimer % timer)
rlm@205: (cortex.util/display-dialated-time % timer)
rlm@205: (Capture/captureVideo
rlm@205: % (File. "/home/r/proj/cortex/render/body/4")))
rlm@205: no-op)))
rlm@205: #+end_src
rlm@205:
rlm@205: #+begin_html
rlm@205:
rlm@205: #+end_html
rlm@205:
rlm@209: * Headers
rlm@205: #+name: body-header
rlm@202: #+begin_src clojure
rlm@202: (ns cortex.body
rlm@202: "Assemble a physical creature using the definitions found in a
rlm@202: specially prepared blender file. Creates rigid bodies and joints so
rlm@202: that a creature can have a physical presense in the simulation."
rlm@202: {:author "Robert McIntyre"}
rlm@202: (:use (cortex world util sense))
rlm@202: (:use clojure.contrib.def)
rlm@202: (:import
rlm@202: (com.jme3.math Vector3f Quaternion Vector2f Matrix3f)
rlm@202: (com.jme3.bullet.joints
rlm@202: SixDofJoint Point2PointJoint HingeJoint ConeJoint)
rlm@202: com.jme3.bullet.control.RigidBodyControl
rlm@202: com.jme3.collision.CollisionResults
rlm@202: com.jme3.bounding.BoundingBox
rlm@202: com.jme3.scene.Node
rlm@202: com.jme3.scene.Geometry
rlm@202: com.jme3.bullet.collision.shapes.HullCollisionShape))
rlm@202: #+end_src
rlm@133:
rlm@205: #+name: test-header
rlm@205: #+begin_src clojure
rlm@205: (ns cortex.test.body
rlm@205: (:use (cortex world util body))
rlm@205: (:import
rlm@205: (com.aurellem.capture Capture RatchetTimer)
rlm@205: (com.jme3.math Quaternion Vector3f ColorRGBA)
rlm@205: java.io.File))
rlm@205: #+end_src
rlm@205:
rlm@202: * Source
rlm@207: - [[../src/cortex/body.clj][cortex.body]]
rlm@207: - [[../src/cortex/test/body.clj][cortex.test.body]]
rlm@207: - [[../assets/Models/test-creature/hand.blend][hand.blend]]
rlm@209: - [[../assets/Models/test-creature/palm.png][UV-map-1]]
rlm@207: - [[../assets/Models/test-creature/worm.blend][worm.blend]]
rlm@207: - [[../assets/Models/test-creature/retina-small.png][UV-map-1]]
rlm@207: - [[../assets/Models/test-creature/tip.png][UV-map-2]]
rlm@63:
rlm@206: * COMMENT Generate Source
rlm@44: #+begin_src clojure :tangle ../src/cortex/body.clj
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@0: #+end_src
rlm@64:
rlm@69: #+begin_src clojure :tangle ../src/cortex/test/body.clj
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@205: <>
rlm@64: #+end_src
rlm@64:
rlm@64:
rlm@0:
rlm@206: