rlm@202: #+title: Building a Body rlm@0: #+author: Robert McIntyre rlm@0: #+email: rlm@mit.edu rlm@4: #+description: Simulating a body (movement, touch, propioception) in jMonkeyEngine3. rlm@4: #+SETUPFILE: ../../aurellem/org/setup.org rlm@4: #+INCLUDE: ../../aurellem/org/level-0.org rlm@4: rlm@202: * Design Constraints rlm@202: rlm@202: I use [[www.blender.org/][blender]] to design bodies. The design of the bodies is rlm@202: determined by the requirements of the AI that will use them. The rlm@202: bodies must be easy for an AI to sense and control, and they must be rlm@202: relatively simple for jMonkeyEngine to compute. rlm@202: rlm@202: ** Bag of Bones rlm@202: rlm@202: How to create such a body? One option I ultimately rejected is to use rlm@202: blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea would have been to define a mesh rlm@202: which describes the creature's entire body. To this you add an rlm@202: (skeleton) which deforms this mesh. This technique is used extensively rlm@202: to model humans and create realistic animations. It is hard to use for rlm@202: my purposes because it is difficult to update the creature's Physics rlm@202: Collision Mesh in tandem with its Geometric Mesh under the influence rlm@202: of the armature. Withouth this the creature will not be able to grab rlm@202: things in its environment, and it won't be able to tell where its rlm@202: physical body is by using its eyes. Also, armatures do not specify rlm@202: any rotational limits for a joint, making it hard to model elbows, rlm@202: shoulders, etc. rlm@202: rlm@202: ** EVE rlm@202: rlm@202: Instead of using the human-like "deformable bag of bones" approach, I rlm@202: decided to base my body plans on the robot EVE from the movie wall-E. rlm@202: rlm@202: #+caption: EVE from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one. rlm@202: [[../images/Eve.jpg]] rlm@202: rlm@204: EVE's body is composed of several rigid components that are held rlm@204: together by invisible joint constraints. This is what I mean by rlm@204: "eve-like". The main reason that I use eve-style bodies is so that rlm@204: there will be correspondence between the AI's vision and the physical rlm@204: presence of its body. Each individual section is simulated by a rlm@204: separate rigid body that corresponds exactly with its visual rlm@204: representation and does not change. Sections are connected by rlm@204: invisible joints that are well supported in jMonkyeEngine. Bullet, the rlm@204: physics backend for jMonkeyEngine, can efficiently simulate hundreds rlm@204: of rigid bodies connected by joints. Sections do not have to stay as rlm@204: one piece forever; they can be dynamically replaced with multiple rlm@204: sections to simulate splitting in two. This could be used to simulate rlm@209: retractable claws or EVE's hands, which are able to coalece into one rlm@209: object in the movie. rlm@202: rlm@202: * Solidifying the Body rlm@202: rlm@202: Here is a hand designed eve-style in blender. rlm@202: rlm@203: #+attr_html: width="755" rlm@202: [[../images/hand-screenshot0.png]] rlm@202: rlm@202: If we load it directly into jMonkeyEngine, we get this: rlm@202: rlm@205: #+name: test-1 rlm@202: #+begin_src clojure rlm@202: (def hand-path "Models/test-creature/hand.blend") rlm@202: rlm@202: (defn hand [] (load-blender-model hand-path)) rlm@202: rlm@202: (defn setup [world] rlm@202: (let [cam (.getCamera world)] rlm@202: (println-repl cam) rlm@202: (.setLocation rlm@202: cam (Vector3f. rlm@202: -6.9015837, 8.644911, 5.6043186)) rlm@202: (.setRotation rlm@202: cam rlm@202: (Quaternion. rlm@202: 0.14046453, 0.85894054, -0.34301838, 0.3533118))) rlm@202: (light-up-everything world) rlm@202: (.setTimer world (RatchetTimer. 60)) rlm@202: world) rlm@202: rlm@202: (defn test-one [] rlm@202: (world (hand) rlm@202: standard-debug-controls rlm@202: (comp rlm@202: #(Capture/captureVideo rlm@202: % (File. "/home/r/proj/cortex/render/body/1")) rlm@202: setup) rlm@202: no-op)) rlm@202: #+end_src rlm@202: rlm@202: rlm@202: #+begin_src clojure :results silent rlm@202: (.start (cortex.test.body/test-one)) rlm@202: #+end_src rlm@202: rlm@202: #+begin_html rlm@203:
rlm@203:
rlm@203: rlm@203:
rlm@203:

The hand model directly loaded from blender. It has no physical rlm@203: presense in the simulation.

rlm@203:
rlm@202: #+end_html rlm@202: rlm@202: You will notice that the hand has no physical presence -- it's a rlm@204: hologram through which everything passes. Therefore, the first thing rlm@202: to do is to make it solid. Blender has physics simulation on par with rlm@202: jMonkeyEngine (they both use bullet as their physics backend), but it rlm@202: can be difficult to translate between the two systems, so for now I rlm@209: specify the mass of each object as meta-data in blender and construct rlm@209: the physics shape based on the mesh in jMonkeyEngine. rlm@202: rlm@203: #+name: body-1 rlm@202: #+begin_src clojure rlm@202: (defn physical! rlm@202: "Iterate through the nodes in creature and make them real physical rlm@202: objects in the simulation." rlm@202: [#^Node creature] rlm@202: (dorun rlm@202: (map rlm@202: (fn [geom] rlm@202: (let [physics-control rlm@202: (RigidBodyControl. rlm@202: (HullCollisionShape. rlm@202: (.getMesh geom)) rlm@202: (if-let [mass (meta-data geom "mass")] rlm@202: (do rlm@202: (println-repl rlm@202: "setting" (.getName geom) "mass to" (float mass)) rlm@202: (float mass)) rlm@202: (float 1)))] rlm@202: (.addControl geom physics-control))) rlm@202: (filter #(isa? (class %) Geometry ) rlm@202: (node-seq creature))))) rlm@202: #+end_src rlm@202: rlm@202: =(physical!)= iterates through a creature's node structure, creating rlm@202: CollisionShapes for each geometry with the mass specified in that rlm@202: geometry's meta-data. rlm@202: rlm@205: #+name: test-2 rlm@0: #+begin_src clojure rlm@202: (in-ns 'cortex.test.body) rlm@160: rlm@209: (def gravity-control rlm@202: {"key-g" (fn [world _] rlm@209: (set-gravity world (Vector3f. 0 -9.81 0))) rlm@209: "key-u" (fn [world _] (set-gravity world Vector3f/ZERO))}) rlm@209: rlm@202: rlm@202: (defn floor [] rlm@202: (box 10 3 10 :position (Vector3f. 0 -10 0) rlm@202: :color ColorRGBA/Gray :mass 0)) rlm@202: rlm@202: (defn test-two [] rlm@202: (world (nodify rlm@202: [(doto (hand) rlm@202: (physical!)) rlm@202: (floor)]) rlm@209: (merge standard-debug-controls gravity-control) rlm@202: (comp rlm@202: #(Capture/captureVideo rlm@202: % (File. "/home/r/proj/cortex/render/body/2")) rlm@202: #(do (set-gravity % Vector3f/ZERO) %) rlm@202: setup) rlm@202: no-op)) rlm@202: #+end_src rlm@202: rlm@202: #+begin_html rlm@203:
rlm@203:
rlm@203: rlm@203:
rlm@203:

The hand now has a physical presence, but there is nothing to hold rlm@203: it together.

rlm@203:
rlm@202: #+end_html rlm@202: rlm@202: Now that's some progress. rlm@202: rlm@202: * Joints rlm@202: rlm@209: Obviously, an AI is not going to be doing much while lying in pieces rlm@209: on the floor. So, the next step to making a proper body is to connect rlm@202: those pieces together with joints. jMonkeyEngine has a large array of rlm@202: joints available via bullet, such as Point2Point, Cone, Hinge, and a rlm@202: generic Six Degree of Freedom joint, with or without spring rlm@202: restitution. rlm@202: rlm@202: Although it should be possible to specify the joints using blender's rlm@202: physics system, and then automatically import them with jMonkeyEngine, rlm@202: the support isn't there yet, and there are a few problems with bullet rlm@202: itself that need to be solved before it can happen. rlm@202: rlm@202: So, I will use the same system for specifying joints as I will do for rlm@202: some senses. Each joint is specified by an empty node whose parent rlm@202: has the name "joints". Their orientation and meta-data determine what rlm@202: joint is created. rlm@202: rlm@203: #+attr_html: width="755" rlm@209: #+caption: Joints hack in blender. Each empty node here will be transformed into a joint in jMonkeyEngine rlm@202: [[../images/hand-screenshot1.png]] rlm@202: rlm@203: The empty node in the upper right, highlighted in yellow, is the rlm@203: parent node of all the emptys which represent joints. The following rlm@203: functions must do three things to translate these into real joints: rlm@202: rlm@203: - Find the children of the "joints" node. rlm@203: - Determine the two spatials the joint it meant to connect. rlm@203: - Create the joint based on the meta-data of the empty node. rlm@202: rlm@203: ** Finding the Joints rlm@209: rlm@209: The higher order function =(sense-nodes)= from =cortex.sense= simplifies rlm@209: the first task. rlm@209: rlm@203: #+name: joints-2 rlm@203: #+begin_src clojure rlm@203: (defvar rlm@203: ^{:arglists '([creature])} rlm@203: joints rlm@203: (sense-nodes "joints") rlm@203: "Return the children of the creature's \"joints\" node.") rlm@203: #+end_src rlm@202: rlm@203: rlm@203: ** Joint Targets and Orientation rlm@203: rlm@203: This technique for finding a joint's targets is very similiar to rlm@203: =(cortex.sense/closest-node)=. A small cube, centered around the rlm@203: empty-node, grows exponentially until it intersects two /physical/ rlm@203: objects. The objects are ordered according to the joint's rotation, rlm@203: with the first one being the object that has more negative coordinates rlm@203: in the joint's reference frame. Since the objects must be physical, rlm@203: the empty-node itself escapes detection. Because the objects must be rlm@203: physical, =(joint-targets)= must be called /after/ =(physical!)= is rlm@203: called. rlm@203: rlm@203: #+name: joints-3 rlm@202: #+begin_src clojure rlm@135: (defn joint-targets rlm@135: "Return the two closest two objects to the joint object, ordered rlm@135: from bottom to top according to the joint's rotation." rlm@135: [#^Node parts #^Node joint] rlm@135: (loop [radius (float 0.01)] rlm@135: (let [results (CollisionResults.)] rlm@135: (.collideWith rlm@135: parts rlm@135: (BoundingBox. (.getWorldTranslation joint) rlm@209: radius radius radius) results) rlm@135: (let [targets rlm@135: (distinct rlm@135: (map #(.getGeometry %) results))] rlm@135: (if (>= (count targets) 2) rlm@135: (sort-by rlm@209: #(let [joint-ref-frame-position rlm@135: (jme-to-blender rlm@135: (.mult rlm@135: (.inverse (.getWorldRotation joint)) rlm@135: (.subtract (.getWorldTranslation %) rlm@135: (.getWorldTranslation joint))))] rlm@209: (.dot (Vector3f. 1 1 1) joint-ref-frame-position)) rlm@135: (take 2 targets)) rlm@135: (recur (float (* radius 2)))))))) rlm@203: #+end_src rlm@135: rlm@203: ** Generating Joints rlm@203: rlm@209: This section of code iterates through all the different ways of rlm@203: specifying joints using blender meta-data and converts each one to the rlm@203: appropriate jMonkyeEngine joint. rlm@203: rlm@203: #+name: joints-4 rlm@203: #+begin_src clojure rlm@160: (defmulti joint-dispatch rlm@160: "Translate blender pseudo-joints into real JME joints." rlm@160: (fn [constraints & _] rlm@160: (:type constraints))) rlm@141: rlm@160: (defmethod joint-dispatch :point rlm@160: [constraints control-a control-b pivot-a pivot-b rotation] rlm@160: (println-repl "creating POINT2POINT joint") rlm@160: ;; bullet's point2point joints are BROKEN, so we must use the rlm@160: ;; generic 6DOF joint instead of an actual Point2Point joint! rlm@141: rlm@160: ;; should be able to do this: rlm@160: (comment rlm@160: (Point2PointJoint. rlm@160: control-a rlm@160: control-b rlm@160: pivot-a rlm@160: pivot-b)) rlm@141: rlm@160: ;; but instead we must do this: rlm@160: (println-repl "substuting 6DOF joint for POINT2POINT joint!") rlm@160: (doto rlm@160: (SixDofJoint. rlm@160: control-a rlm@160: control-b rlm@160: pivot-a rlm@160: pivot-b rlm@160: false) rlm@160: (.setLinearLowerLimit Vector3f/ZERO) rlm@203: (.setLinearUpperLimit Vector3f/ZERO))) rlm@160: rlm@160: (defmethod joint-dispatch :hinge rlm@160: [constraints control-a control-b pivot-a pivot-b rotation] rlm@160: (println-repl "creating HINGE joint") rlm@160: (let [axis rlm@160: (if-let rlm@160: [axis (:axis constraints)] rlm@160: axis rlm@160: Vector3f/UNIT_X) rlm@160: [limit-1 limit-2] (:limit constraints) rlm@160: hinge-axis rlm@160: (.mult rlm@160: rotation rlm@160: (blender-to-jme axis))] rlm@160: (doto rlm@160: (HingeJoint. rlm@160: control-a rlm@160: control-b rlm@160: pivot-a rlm@160: pivot-b rlm@160: hinge-axis rlm@160: hinge-axis) rlm@160: (.setLimit limit-1 limit-2)))) rlm@160: rlm@160: (defmethod joint-dispatch :cone rlm@160: [constraints control-a control-b pivot-a pivot-b rotation] rlm@160: (let [limit-xz (:limit-xz constraints) rlm@160: limit-xy (:limit-xy constraints) rlm@160: twist (:twist constraints)] rlm@160: rlm@160: (println-repl "creating CONE joint") rlm@160: (println-repl rotation) rlm@160: (println-repl rlm@160: "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0))) rlm@160: (println-repl rlm@160: "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0))) rlm@160: (println-repl rlm@160: "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1))) rlm@160: (doto rlm@160: (ConeJoint. rlm@160: control-a rlm@160: control-b rlm@160: pivot-a rlm@160: pivot-b rlm@160: rotation rlm@160: rotation) rlm@160: (.setLimit (float limit-xz) rlm@160: (float limit-xy) rlm@160: (float twist))))) rlm@160: rlm@160: (defn connect rlm@175: "Create a joint between 'obj-a and 'obj-b at the location of rlm@175: 'joint. The type of joint is determined by the metadata on 'joint. rlm@175: rlm@175: Here are some examples: rlm@160: {:type :point} rlm@160: {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)} rlm@160: (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints) rlm@160: rlm@160: {:type :cone :limit-xz 0] rlm@160: :limit-xy 0] rlm@160: :twist 0]} (use XZY rotation mode in blender!)" rlm@160: [#^Node obj-a #^Node obj-b #^Node joint] rlm@160: (let [control-a (.getControl obj-a RigidBodyControl) rlm@160: control-b (.getControl obj-b RigidBodyControl) rlm@160: joint-center (.getWorldTranslation joint) rlm@160: joint-rotation (.toRotationMatrix (.getWorldRotation joint)) rlm@160: pivot-a (world-to-local obj-a joint-center) rlm@160: pivot-b (world-to-local obj-b joint-center)] rlm@160: rlm@160: (if-let [constraints rlm@160: (map-vals rlm@160: eval rlm@160: (read-string rlm@160: (meta-data joint "joint")))] rlm@160: ;; A side-effect of creating a joint registers rlm@160: ;; it with both physics objects which in turn rlm@160: ;; will register the joint with the physics system rlm@160: ;; when the simulation is started. rlm@160: (do rlm@160: (println-repl "creating joint between" rlm@160: (.getName obj-a) "and" (.getName obj-b)) rlm@160: (joint-dispatch constraints rlm@160: control-a control-b rlm@160: pivot-a pivot-b rlm@160: joint-rotation)) rlm@160: (println-repl "could not find joint meta-data!")))) rlm@203: #+end_src rlm@160: rlm@209: Creating joints is now a matter of applying =(connect)= to each joint rlm@203: node. rlm@160: rlm@205: #+name: joints-5 rlm@203: #+begin_src clojure rlm@175: (defn joints! rlm@175: "Connect the solid parts of the creature with physical joints. The rlm@175: joints are taken from the \"joints\" node in the creature." rlm@175: [#^Node creature] rlm@160: (dorun rlm@160: (map rlm@160: (fn [joint] rlm@175: (let [[obj-a obj-b] (joint-targets creature joint)] rlm@160: (connect obj-a obj-b joint))) rlm@175: (joints creature)))) rlm@203: #+end_src rlm@160: rlm@203: rlm@203: ** Round 3 rlm@203: rlm@203: Now we can test the hand in all its glory. rlm@203: rlm@205: #+name: test-3 rlm@203: #+begin_src clojure rlm@203: (in-ns 'cortex.test.body) rlm@203: rlm@203: (def debug-control rlm@203: {"key-h" (fn [world val] rlm@209: (if val (enable-debug world)))}) rlm@203: rlm@203: (defn test-three [] rlm@203: (world (nodify rlm@203: [(doto (hand) rlm@205: (physical!) rlm@205: (joints!)) rlm@203: (floor)]) rlm@203: (merge standard-debug-controls debug-control rlm@209: gravity-control) rlm@203: (comp rlm@203: #(Capture/captureVideo rlm@203: % (File. "/home/r/proj/cortex/render/body/3")) rlm@203: #(do (set-gravity % Vector3f/ZERO) %) rlm@203: setup) rlm@203: no-op)) rlm@203: #+end_src rlm@203: rlm@203: =(physical!)= makes the hand solid, then =(joints!)= connects each rlm@203: piece together. rlm@203: rlm@203: #+begin_html rlm@203:
rlm@203:
rlm@203: rlm@203:
rlm@203:

Now the hand is physical and has joints.

rlm@203:
rlm@203: #+end_html rlm@203: rlm@203: The joints are visualized as green connections between each segment rlm@203: for debug purposes. You can see that they correspond to the empty rlm@203: nodes in the blender file. rlm@203: rlm@203: * Wrap-Up! rlm@203: rlm@203: It is convienent to combine =(physical!)= and =(joints!)= into one rlm@203: function that completely creates the creature's physical body. rlm@203: rlm@205: #+name: joints-6 rlm@203: #+begin_src clojure rlm@175: (defn body! rlm@175: "Endow the creature with a physical body connected with joints. The rlm@175: particulars of the joints and the masses of each pody part are rlm@175: determined in blender." rlm@175: [#^Node creature] rlm@175: (physical! creature) rlm@175: (joints! creature)) rlm@64: #+end_src rlm@63: rlm@205: * The Worm rlm@205: rlm@205: Going forward, I will use a model that is less complicated than the rlm@205: hand. It has two segments and one joint, and I call it the worm. All rlm@205: of the senses described in the following posts will be applied to this rlm@205: worm. rlm@205: rlm@205: #+name: test-4 rlm@205: #+begin_src clojure rlm@205: (in-ns 'cortex.test.body) rlm@205: rlm@205: (defn worm-1 [] rlm@205: (let [timer (RatchetTimer. 60)] rlm@205: (world rlm@205: (nodify rlm@205: [(doto rlm@205: (load-blender-model rlm@205: "Models/test-creature/worm.blend") rlm@205: (body!)) rlm@205: (floor)]) rlm@205: (merge standard-debug-controls debug-control) rlm@205: #(do rlm@205: (speed-up %) rlm@205: (light-up-everything %) rlm@205: (.setTimer % timer) rlm@205: (cortex.util/display-dialated-time % timer) rlm@205: (Capture/captureVideo rlm@205: % (File. "/home/r/proj/cortex/render/body/4"))) rlm@205: no-op))) rlm@205: #+end_src rlm@205: rlm@205: #+begin_html rlm@205:
rlm@205:
rlm@205: rlm@205:
rlm@205:

This worm model will be the platform onto which future senses will rlm@205: be grafted.

rlm@205:
rlm@205: #+end_html rlm@205: rlm@209: * Headers rlm@205: #+name: body-header rlm@202: #+begin_src clojure rlm@202: (ns cortex.body rlm@202: "Assemble a physical creature using the definitions found in a rlm@202: specially prepared blender file. Creates rigid bodies and joints so rlm@202: that a creature can have a physical presense in the simulation." rlm@202: {:author "Robert McIntyre"} rlm@202: (:use (cortex world util sense)) rlm@202: (:use clojure.contrib.def) rlm@202: (:import rlm@202: (com.jme3.math Vector3f Quaternion Vector2f Matrix3f) rlm@202: (com.jme3.bullet.joints rlm@202: SixDofJoint Point2PointJoint HingeJoint ConeJoint) rlm@202: com.jme3.bullet.control.RigidBodyControl rlm@202: com.jme3.collision.CollisionResults rlm@202: com.jme3.bounding.BoundingBox rlm@202: com.jme3.scene.Node rlm@202: com.jme3.scene.Geometry rlm@202: com.jme3.bullet.collision.shapes.HullCollisionShape)) rlm@202: #+end_src rlm@133: rlm@205: #+name: test-header rlm@205: #+begin_src clojure rlm@205: (ns cortex.test.body rlm@205: (:use (cortex world util body)) rlm@205: (:import rlm@205: (com.aurellem.capture Capture RatchetTimer) rlm@205: (com.jme3.math Quaternion Vector3f ColorRGBA) rlm@205: java.io.File)) rlm@205: #+end_src rlm@205: rlm@202: * Source rlm@210: rlm@207: - [[../src/cortex/body.clj][cortex.body]] rlm@207: - [[../src/cortex/test/body.clj][cortex.test.body]] rlm@207: - [[../assets/Models/test-creature/hand.blend][hand.blend]] rlm@209: - [[../assets/Models/test-creature/palm.png][UV-map-1]] rlm@207: - [[../assets/Models/test-creature/worm.blend][worm.blend]] rlm@207: - [[../assets/Models/test-creature/retina-small.png][UV-map-1]] rlm@207: - [[../assets/Models/test-creature/tip.png][UV-map-2]] rlm@211: #+html: rlm@211: rlm@211: * Next rlm@211: The body I have made here exists without any senses or effectors. In rlm@211: the [[./vision.org][next post]], I'll give the creature eyes. rlm@63: rlm@206: * COMMENT Generate Source rlm@44: #+begin_src clojure :tangle ../src/cortex/body.clj rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@0: #+end_src rlm@64: rlm@69: #+begin_src clojure :tangle ../src/cortex/test/body.clj rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@205: <> rlm@64: #+end_src rlm@64: rlm@64: rlm@0: rlm@206: