view org/proprioception.org @ 312:e0e1561cbd22

re-encoded all videos and fixed all bad keyframes
author Robert McIntyre <rlm@mit.edu>
date Tue, 21 Feb 2012 15:22:15 -0700
parents 5d448182c807
children 2c7fbcbd5ebb
line wrap: on
line source
1 #+title: The Sense of Proprioception
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: proprioception for simulated creatures
5 #+keywords: simulation, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Proprioception
11 Close your eyes, and touch your nose with your right index finger. How
12 did you do it? You could not see your hand, and neither your hand nor
13 your nose could use the sense of touch to guide the path of your hand.
14 There are no sound cues, and Taste and Smell certainly don't provide
15 any help. You know where your hand is without your other senses
16 because of Proprioception.
18 Humans can sometimes loose this sense through viral infections or
19 damage to the spinal cord or brain, and when they do, they loose the
20 ability to control their own bodies without looking directly at the
21 parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a Hat]],
22 a woman named Christina looses this sense and has to learn how to move
23 by carefully watching her arms and legs. She describes proprioception
24 as the "eyes of the body, the way the body sees itself".
26 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
27 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
28 positions of each body part by monitoring muscle strain and length.
30 It's clear that this is a vital sense for fluid, graceful
31 movement. It's also particularly easy to implement in jMonkeyEngine.
33 My simulated proprioception calculates the relative angles of each
34 joint from the rest position defined in the blender file. This
35 simulates the muscle-spindles and joint capsules. I will deal with
36 Golgi tendon organs, which calculate muscle strain, in the [[./movement.org][next post]].
38 * Helper Functions
40 =absolute-angle= calculates the angle between two vectors, relative to a
41 third axis vector. This angle is the number of radians you have to
42 move counterclockwise around the axis vector to get from the first to
43 the second vector. It is not commutative like a normal dot-product
44 angle is.
46 #+name: helpers
47 #+begin_src clojure
48 (in-ns 'cortex.proprioception)
50 (defn right-handed?
51 "true iff the three vectors form a right handed coordinate
52 system. The three vectors do not have to be normalized or
53 orthogonal."
54 [vec1 vec2 vec3]
55 (< 0 (.dot (.cross vec1 vec2) vec3)))
57 (defn absolute-angle
58 "The angle between 'vec1 and 'vec2 around 'axis. In the range
59 [0 (* 2 Math/PI)]."
60 [vec1 vec2 axis]
61 (let [angle (.angleBetween vec1 vec2)]
62 (if (right-handed? vec1 vec2 axis)
63 angle (- (* 2 Math/PI) angle))))
64 #+end_src
66 #+begin_src clojure :exports both
67 (in-ns 'cortex.proprioception)
68 (absolute-angle Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
69 #+end_src
71 #+results:
72 : 1.5707964
74 #+begin_src clojure :exports both
75 (in-ns 'cortex.proprioception)
76 (absolute-angle
77 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)
78 #+end_src
80 #+results:
81 : 4.7123889366733
83 * Proprioception Kernel
85 Given a joint, =proprioception-kernel= produces a function that
86 calculates the Euler angles between the the objects the joint
87 connects.
89 #+name: proprioception
90 #+begin_src clojure
91 (defn proprioception-kernel
92 "Returns a function which returns proprioceptive sensory data when
93 called inside a running simulation."
94 [#^Node parts #^Node joint]
95 (let [[obj-a obj-b] (joint-targets parts joint)
96 joint-rot (.getWorldRotation joint)
97 x0 (.mult joint-rot Vector3f/UNIT_X)
98 y0 (.mult joint-rot Vector3f/UNIT_Y)
99 z0 (.mult joint-rot Vector3f/UNIT_Z)]
100 (fn []
101 (let [rot-a (.clone (.getWorldRotation obj-a))
102 rot-b (.clone (.getWorldRotation obj-b))
103 x (.mult rot-a x0)
104 y (.mult rot-a y0)
105 z (.mult rot-a z0)
107 X (.mult rot-b x0)
108 Y (.mult rot-b y0)
109 Z (.mult rot-b z0)
110 heading (Math/atan2 (.dot X z) (.dot X x))
111 pitch (Math/atan2 (.dot X y) (.dot X x))
113 ;; rotate x-vector back to origin
114 reverse
115 (doto (Quaternion.)
116 (.fromAngleAxis
117 (.angleBetween X x)
118 (let [cross (.normalize (.cross X x))]
119 (if (= 0 (.length cross)) y cross))))
120 roll (absolute-angle (.mult reverse Y) y x)]
121 [heading pitch roll]))))
123 (defn proprioception!
124 "Endow the creature with the sense of proprioception. Returns a
125 sequence of functions, one for each child of the \"joints\" node in
126 the creature, which each report proprioceptive information about
127 that joint."
128 [#^Node creature]
129 ;; extract the body's joints
130 (let [senses (map (partial proprioception-kernel creature)
131 (joints creature))]
132 (fn []
133 (map #(%) senses))))
134 #+end_src
137 =proprioception!= maps =proprioception-kernel= across all the
138 joints of the creature. It uses the same list of joints that
139 =cortex.body/joints= uses.
141 * Visualizing Proprioception
143 Proprioception has the lowest bandwidth of all the senses so far, and
144 it doesn't lend itself as readily to visual representation like
145 vision, hearing, or touch. This visualization code creates a "gauge"
146 to view each of the three relative angles along a circle.
148 #+name: visualize
149 #+begin_src clojure
150 (in-ns 'cortex.proprioception)
152 (defn draw-sprite [image sprite x y color ]
153 (dorun
154 (for [[u v] sprite]
155 (.setRGB image (+ u x) (+ v y) color))))
157 (defn view-angle
158 "create a debug view of an angle"
159 [color]
160 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
161 previous (atom [25 25])
162 sprite [[0 0] [0 1]
163 [0 -1] [-1 0] [1 0]]]
164 (fn [angle]
165 (let [angle (float angle)]
166 (let [position
167 [(+ 25 (int (* 20 (Math/cos angle))))
168 (+ 25 (int (* -20 (Math/sin angle))))]]
169 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
170 (draw-sprite image sprite (position 0) (position 1) color)
171 (reset! previous position))
172 image))))
174 (defn proprioception-display-kernel
175 "Display proprioception angles in a BufferedImage"
176 [[h p r]]
177 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
178 previous-heading (atom [25 25])
179 previous-pitch (atom [25 25])
180 previous-roll (atom [25 25])
182 heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
183 pitch-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
184 roll-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
185 draw-angle
186 (fn [angle sprite previous color]
187 (let [angle (float angle)]
188 (let [position
189 [(+ 25 (int (* 20 (Math/cos angle))))
190 (+ 25 (int (* -20 (Math/sin angle))))]]
191 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
192 (draw-sprite image sprite (position 0) (position 1) color)
193 (reset! previous position))
194 image))]
195 (dorun (map draw-angle
196 [h p r]
197 [heading-sprite pitch-sprite roll-sprite]
198 [previous-heading previous-pitch previous-roll]
199 [0xFF0000 0x00FF00 0xFFFFFF]))
200 image))
202 (defn view-proprioception
203 "Creates a function which accepts a list of proprioceptive data and
204 display each element of the list to the screen as an image."
205 []
206 (view-sense proprioception-display-kernel))
207 #+end_src
209 * Proprioception Test
210 This test does not use the worm, but instead uses two bars, bound
211 together by a point2point joint. One bar is fixed, and I control the
212 other bar from the keyboard.
214 #+name: test-proprioception
215 #+begin_src clojure
216 (in-ns 'cortex.test.proprioception)
218 (defn test-proprioception
219 "Testing proprioception:
220 You should see two floating bars, and a printout of pitch, yaw, and
221 roll. Pressing key-r/key-t should move the blue bar up and down and
222 change only the value of pitch. key-f/key-g moves it side to side
223 and changes yaw. key-v/key-b will spin the blue segment clockwise
224 and counterclockwise, and only affect roll."
225 ([] (test-proprioception false))
226 ([record?]
227 (let [hand (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
228 :mass 0 :color ColorRGBA/Gray :name "hand")
229 finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
230 :mass 1
231 :color
232 (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1)
233 :name "finger")
234 joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
235 :position (Vector3f. 0 1.2 0)
236 :rotation (doto (Quaternion.)
237 (.fromAngleAxis
238 (/ Math/PI 2)
239 (Vector3f. 0 0 1)))
240 :physical? false)
241 creature (nodify [hand finger joint-node])
242 finger-control (.getControl finger RigidBodyControl)
243 hand-control (.getControl hand RigidBodyControl)
244 joint (joint-dispatch {:type :point} hand-control finger-control
245 (Vector3f. 0 1.2 0)
246 (Vector3f. 0 -1.2 0) nil)
248 root (nodify [creature])
249 prop (proprioception-kernel creature joint-node)
250 prop-view (view-proprioception)]
251 (.setCollisionGroup
252 (.getControl hand RigidBodyControl)
253 PhysicsCollisionObject/COLLISION_GROUP_NONE)
254 (apply
255 world
256 (with-movement
257 finger
258 ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
259 [1 1 10 10 10 10]
260 [root
261 standard-debug-controls
262 (fn [world]
263 (if record?
264 (Capture/captureVideo
265 world
266 (File. "/home/r/proj/cortex/render/proprio/main-view")))
267 (.setTimer world (com.aurellem.capture.RatchetTimer. 60))
268 (set-gravity world (Vector3f. 0 0 0))
269 (enable-debug world)
270 (light-up-everything world))
271 (fn [_ _]
272 (prop-view
273 (list (prop))
274 (if record?
275 (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))
276 #+end_src
278 #+results: test-proprioception
279 : #'cortex.test.proprioception/test-proprioception
281 * Video of Proprioception
283 #+begin_html
284 <div class="figure">
285 <center>
286 <video controls="controls" width="550">
287 <source src="../video/test-proprioception.ogg" type="video/ogg"
288 preload="none" poster="../images/aurellem-1280x480.png" />
289 </video>
290 <br> <a href="http://youtu.be/JjdDmyM8b0w"> YouTube </a>
291 </center>
292 <p>Proprioception in a simple creature. The proprioceptive readout is
293 in the upper left corner of the screen.</p>
294 </div>
295 #+end_html
297 ** Generating the Proprioception Video
298 #+name: magick6
299 #+begin_src clojure
300 (ns cortex.video.magick6
301 (:import java.io.File)
302 (:use clojure.contrib.shell-out))
304 (defn images [path]
305 (sort (rest (file-seq (File. path)))))
307 (def base "/home/r/proj/cortex/render/proprio/")
309 (defn pics [file]
310 (images (str base file)))
312 (defn combine-images []
313 (let [main-view (pics "main-view")
314 proprioception (pics "proprio/0")
315 targets (map
316 #(File. (str base "out/" (format "%07d.png" %)))
317 (range 0 (count main-view)))]
318 (dorun
319 (pmap
320 (comp
321 (fn [[ main-view proprioception target]]
322 (println target)
323 (sh "convert"
324 main-view
325 proprioception "-geometry" "+20+20" "-composite"
326 target))
327 (fn [& args] (map #(.getCanonicalPath %) args)))
328 main-view proprioception targets))))
329 #+end_src
331 #+begin_src sh :results silent
332 cd ~/proj/cortex/render/proprio
333 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \
334 test-proprioception.ogg
335 #+end_src
337 * Headers
338 #+name: proprioception-header
339 #+begin_src clojure
340 (ns cortex.proprioception
341 "Simulate the sense of proprioception (ability to detect the
342 relative positions of body parts with respect to other body parts)
343 in jMonkeyEngine3. Reads specially prepared blender files to
344 automatically generate proprioceptive senses."
345 (:use (cortex world util sense body))
346 (:use clojure.contrib.def)
347 (:import com.jme3.scene.Node)
348 (:import java.awt.image.BufferedImage)
349 (:import (com.jme3.math Vector3f Quaternion)))
350 #+end_src
352 #+name: test-proprioception-header
353 #+begin_src clojure
354 (ns cortex.test.proprioception
355 (:import (com.aurellem.capture Capture RatchetTimer))
356 (:use (cortex util world proprioception body))
357 (:import java.io.File)
358 (:import com.jme3.bullet.control.RigidBodyControl)
359 (:import com.jme3.bullet.collision.PhysicsCollisionObject)
360 (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))
361 #+end_src
363 * Source Listing
364 - [[../src/cortex/proprioception.clj][cortex.proprioception]]
365 - [[../src/cortex/test/touch.clj][cortex.test.proprioception]]
366 - [[../src/cortex/video/magick6.clj][cortex.video.magick6]]
367 #+html: <ul> <li> <a href="../org/proprioception.org">This org file</a> </li> </ul>
368 - [[http://hg.bortreb.com ][source-repository]]
370 * Next
372 Next time, I'll give the Worm the power to [[./movement.org][move on it's own]].
375 * COMMENT generate source
376 #+begin_src clojure :tangle ../src/cortex/proprioception.clj
377 <<proprioception-header>>
378 <<helpers>>
379 <<proprioception>>
380 <<visualize>>
381 #+end_src
383 #+begin_src clojure :tangle ../src/cortex/test/proprioception.clj
384 <<test-proprioception-header>>
385 <<test-proprioception>>
386 #+end_src
388 #+begin_src clojure :tangle ../src/cortex/video/magick6.clj
389 <<magick6>>
390 #+end_src