view org/proprioception.org @ 283:23aadf376e9d

upgraded to latest jMonkeyEngine, streamlined tests
author Robert McIntyre <rlm@mit.edu>
date Wed, 15 Feb 2012 16:56:01 -0700
parents c39b8b29a79e
children 7e7f8d6d9ec5
line wrap: on
line source
1 #+title: The Sense of Proprioception
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: proprioception for simulated creatures
5 #+keywords: simulation, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Proprioception
11 Close your eyes, and touch your nose with your right index finger. How
12 did you do it? You could not see your hand, and neither your hand nor
13 your nose could use the sense of touch to guide the path of your hand.
14 There are no sound cues, and Taste and Smell certainly don't provide
15 any help. You know where your hand is without your other senses
16 because of Proprioception.
18 Humans can sometimes loose this sense through viral infections or
19 damage to the spinal cord or brain, and when they do, they loose the
20 ability to control their own bodies without looking directly at the
21 parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a Hat]],
22 a woman named Christina looses this sense and has to learn how to move
23 by carefully watching her arms and legs. She describes proprioception
24 as the "eyes of the body, the way the body sees itself".
26 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
27 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
28 positions of each pody part by monitoring muscle strain and length.
30 It's clear that this is a vital sense for fulid, graceful
31 movement. It's also particurally easy to implement in jMonkeyEngine.
33 My simulated proprioception calculates the relative angles of each
34 joint from the rest position defined in the blender file. This
35 simulates the muscle-spindles and joint capsules. I will deal with
36 Golgi tendon organs, which calculate muscle strain, in the [[./movement.org][next post]].
38 * Helper Functions
40 =absolute-angle= calculates the angle between two vectors, relative to a
41 third axis vector. This angle is the number of radians you have to
42 move counterclockwise around the axis vector to get from the first to
43 the second vector. It is not commutative like a normal dot-product
44 angle is.
46 #+name: helpers
47 #+begin_src clojure
48 (in-ns 'cortex.proprioception)
50 (defn right-handed?
51 "true iff the three vectors form a right handed coordinate
52 system. The three vectors do not have to be normalized or
53 orthogonal."
54 [vec1 vec2 vec3]
55 (< 0 (.dot (.cross vec1 vec2) vec3)))
57 (defn absolute-angle
58 "The angle between 'vec1 and 'vec2 around 'axis. In the range
59 [0 (* 2 Math/PI)]."
60 [vec1 vec2 axis]
61 (let [angle (.angleBetween vec1 vec2)]
62 (if (right-handed? vec1 vec2 axis)
63 angle (- (* 2 Math/PI) angle))))
64 #+end_src
66 #+begin_src clojure :exports both
67 (in-ns 'cortex.proprioception)
68 (absolute-angle Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
69 #+end_src
71 #+results:
72 : 1.5707964
74 #+begin_src clojure :exports both
75 (in-ns 'cortex.proprioception)
76 (absolute-angle
77 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)
78 #+end_src
80 #+results:
81 : 4.7123889366733
83 * Proprioception Kernel
85 Given a joint, =proprioception-kernel= produces a function that
86 calculates the euler angles between the the objects the joint
87 connects.
89 #+name: proprioception
90 #+begin_src clojure
91 (defn proprioception-kernel
92 "Returns a function which returns proprioceptive sensory data when
93 called inside a running simulation."
94 [#^Node parts #^Node joint]
95 (let [[obj-a obj-b] (joint-targets parts joint)
96 joint-rot (.getWorldRotation joint)
97 x0 (.mult joint-rot Vector3f/UNIT_X)
98 y0 (.mult joint-rot Vector3f/UNIT_Y)
99 z0 (.mult joint-rot Vector3f/UNIT_Z)]
100 (fn []
101 (let [rot-a (.clone (.getWorldRotation obj-a))
102 rot-b (.clone (.getWorldRotation obj-b))
103 x (.mult rot-a x0)
104 y (.mult rot-a y0)
105 z (.mult rot-a z0)
107 X (.mult rot-b x0)
108 Y (.mult rot-b y0)
109 Z (.mult rot-b z0)
110 heading (Math/atan2 (.dot X z) (.dot X x))
111 pitch (Math/atan2 (.dot X y) (.dot X x))
113 ;; rotate x-vector back to origin
114 reverse
115 (doto (Quaternion.)
116 (.fromAngleAxis
117 (.angleBetween X x)
118 (let [cross (.normalize (.cross X x))]
119 (if (= 0 (.length cross)) y cross))))
120 roll (absolute-angle (.mult reverse Y) y x)]
121 [heading pitch roll]))))
123 (defn proprioception!
124 "Endow the creature with the sense of proprioception. Returns a
125 sequence of functions, one for each child of the \"joints\" node in
126 the creature, which each report proprioceptive information about
127 that joint."
128 [#^Node creature]
129 ;; extract the body's joints
130 (let [senses (map (partial proprioception-kernel creature)
131 (joints creature))]
132 (fn []
133 (map #(%) senses))))
134 #+end_src
137 =proprioception!= maps =proprioception-kernel= across all the
138 joints of the creature. It uses the same list of joints that
139 =cortex.body/joints= uses.
141 * Visualizing Proprioception
143 Proprioception has the lowest bandwidth of all the senses so far, and
144 it doesn't lend itself as readily to visual representation like
145 vision, hearing, or touch. This visualization code creates a "guage"
146 to view each of the three relative angles along a circle.
148 #+name: visualize
149 #+begin_src clojure
150 (in-ns 'cortex.proprioception)
152 (defn draw-sprite [image sprite x y color ]
153 (dorun
154 (for [[u v] sprite]
155 (.setRGB image (+ u x) (+ v y) color))))
157 (defn view-angle
158 "create a debug view of an angle"
159 [color]
160 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
161 previous (atom [25 25])
162 sprite [[0 0] [0 1]
163 [0 -1] [-1 0] [1 0]]]
164 (fn [angle]
165 (let [angle (float angle)]
166 (let [position
167 [(+ 25 (int (* 20 (Math/cos angle))))
168 (+ 25 (int (* -20 (Math/sin angle))))]]
169 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
170 (draw-sprite image sprite (position 0) (position 1) color)
171 (reset! previous position))
172 image))))
174 (defn proprioception-display-kernel
175 "Display proprioception angles in a BufferedImage"
176 [[h p r]]
177 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
178 previous-heading (atom [25 25])
179 previous-pitch (atom [25 25])
180 previous-roll (atom [25 25])
182 heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
183 pitch-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
184 roll-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
185 draw-angle
186 (fn [angle sprite previous color]
187 (let [angle (float angle)]
188 (let [position
189 [(+ 25 (int (* 20 (Math/cos angle))))
190 (+ 25 (int (* -20 (Math/sin angle))))]]
191 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
192 (draw-sprite image sprite (position 0) (position 1) color)
193 (reset! previous position))
194 image))]
195 (dorun (map draw-angle
196 [h p r]
197 [heading-sprite pitch-sprite roll-sprite]
198 [previous-heading previous-pitch previous-roll]
199 [0xFF0000 0x00FF00 0xFFFFFF]))
200 image))
202 (defn view-proprioception
203 "Creates a function which accepts a list of proprioceptive data and
204 display each element of the list to the screen as an image."
205 []
206 (view-sense proprioception-display-kernel))
207 #+end_src
209 * Proprioception Test
210 This test does not use the worm, but instead uses two bars, bound
211 together by a point2point joint. One bar is fixed, and I control the
212 other bar from the keyboard.
214 #+name: test-proprioception
215 #+begin_src clojure
216 (in-ns 'cortex.test.proprioception)
218 (defn test-proprioception
219 "Testing proprioception:
220 You should see two foating bars, and a printout of pitch, yaw, and
221 roll. Pressing key-r/key-t should move the blue bar up and down and
222 change only the value of pitch. key-f/key-g moves it side to side
223 and changes yaw. key-v/key-b will spin the blue segment clockwise
224 and counterclockwise, and only affect roll."
225 ([] (test-proprioception false))
226 ([record?]
227 (let [hand (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
228 :mass 0 :color ColorRGBA/Gray :name "hand")
229 finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
230 :mass 1
231 :color
232 (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1)
233 :name "finger")
234 joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
235 :position (Vector3f. 0 1.2 0)
236 :rotation (doto (Quaternion.)
237 (.fromAngleAxis
238 (/ Math/PI 2)
239 (Vector3f. 0 0 1)))
240 :physical? false)
241 creature (nodify [hand finger joint-node])
242 finger-control (.getControl finger RigidBodyControl)
243 hand-control (.getControl hand RigidBodyControl)
244 joint (joint-dispatch {:type :point} hand-control finger-control
245 (Vector3f. 0 1.2 0)
246 (Vector3f. 0 -1.2 0) nil)
248 root (nodify [creature])
249 prop (proprioception-kernel creature joint-node)
250 prop-view (view-proprioception)]
251 (.setCollisionGroup
252 (.getControl hand RigidBodyControl)
253 PhysicsCollisionObject/COLLISION_GROUP_NONE)
254 (apply
255 world
256 (with-movement
257 finger
258 ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
259 [1 1 10 10 10 10]
260 [root
261 standard-debug-controls
262 (fn [world]
263 (if record?
264 (Capture/captureVideo
265 world
266 (File. "/home/r/proj/cortex/render/proprio/main-view")))
267 (.setTimer world (com.aurellem.capture.RatchetTimer. 60))
268 (set-gravity world (Vector3f. 0 0 0))
269 (enable-debug world)
270 (light-up-everything world))
271 (fn [_ _]
272 (prop-view
273 (list (prop))
274 (if record?
275 (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))
276 #+end_src
278 #+results: test-proprioception
279 : #'cortex.test.proprioception/test-proprioception
281 * Video of Proprioception
283 #+begin_html
284 <div class="figure">
285 <center>
286 <video controls="controls" width="550">
287 <source src="../video/test-proprioception.ogg" type="video/ogg"
288 preload="none" poster="../images/aurellem-1280x480.png" />
289 </video>
290 </center>
291 <p>Proprioception in a simple creature. The proprioceptive readout is
292 in the upper left corner of the screen.</p>
293 </div>
294 #+end_html
296 ** Generating the Proprioception Video
297 #+name: magick6
298 #+begin_src clojure
299 (ns cortex.video.magick6
300 (:import java.io.File)
301 (:use clojure.contrib.shell-out))
303 (defn images [path]
304 (sort (rest (file-seq (File. path)))))
306 (def base "/home/r/proj/cortex/render/proprio/")
308 (defn pics [file]
309 (images (str base file)))
311 (defn combine-images []
312 (let [main-view (pics "main-view")
313 proprioception (pics "proprio/0")
314 targets (map
315 #(File. (str base "out/" (format "%07d.png" %)))
316 (range 0 (count main-view)))]
317 (dorun
318 (pmap
319 (comp
320 (fn [[ main-view proprioception target]]
321 (println target)
322 (sh "convert"
323 main-view
324 proprioception "-geometry" "+20+20" "-composite"
325 target))
326 (fn [& args] (map #(.getCanonicalPath %) args)))
327 main-view proprioception targets))))
328 #+end_src
330 #+begin_src sh :results silent
331 cd ~/proj/cortex/render/proprio
332 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \
333 test-proprioception.ogg
334 #+end_src
336 * Headers
337 #+name: proprioception-header
338 #+begin_src clojure
339 (ns cortex.proprioception
340 "Simulate the sense of proprioception (ability to detect the
341 relative positions of body parts with repsect to other body parts)
342 in jMonkeyEngine3. Reads specially prepared blender files to
343 automatically generate proprioceptive senses."
344 (:use (cortex world util sense body))
345 (:use clojure.contrib.def)
346 (:import com.jme3.scene.Node)
347 (:import java.awt.image.BufferedImage)
348 (:import (com.jme3.math Vector3f Quaternion)))
349 #+end_src
351 #+name: test-proprioception-header
352 #+begin_src clojure
353 (ns cortex.test.proprioception
354 (:import (com.aurellem.capture Capture RatchetTimer))
355 (:use (cortex util world proprioception body))
356 (:import java.io.File)
357 (:import com.jme3.bullet.control.RigidBodyControl)
358 (:import com.jme3.bullet.collision.PhysicsCollisionObject)
359 (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))
360 #+end_src
362 * Source Listing
363 - [[../src/cortex/proprioception.clj][cortex.proprioception]]
364 - [[../src/cortex/test/touch.clj][cortex.test.proprioception]]
365 - [[../src/cortex/video/magick6.clj][cortex.video.magick6]]
366 #+html: <ul> <li> <a href="../org/proprioception.org">This org file</a> </li> </ul>
367 - [[http://hg.bortreb.com ][source-repository]]
369 * Next
371 Next time, I'll give the Worm the power to [[./movement.org][move on it's own]].
374 * COMMENT generate source
375 #+begin_src clojure :tangle ../src/cortex/proprioception.clj
376 <<proprioception-header>>
377 <<helpers>>
378 <<proprioception>>
379 <<visualize>>
380 #+end_src
382 #+begin_src clojure :tangle ../src/cortex/test/proprioception.clj
383 <<test-proprioception-header>>
384 <<test-proprioception>>
385 #+end_src
387 #+begin_src clojure :tangle ../src/cortex/video/magick6.clj
388 <<magick6>>
389 #+end_src