view org/proprioception.org @ 338:d37ccb6c888f

determined that the laptop cannot support arbitray dimensions when creating cameras. will need to use desktop from here on out when doing actual simulations.
author Robert McIntyre <rlm@mit.edu>
date Fri, 20 Jul 2012 16:40:25 -0500
parents 702b5c78c2de
children 4f5a5d5f1613
line wrap: on
line source
1 #+title: The Sense of Proprioception
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: proprioception for simulated creatures
5 #+keywords: simulation, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Proprioception
11 Close your eyes, and touch your nose with your right index finger. How
12 did you do it? You could not see your hand, and neither your hand nor
13 your nose could use the sense of touch to guide the path of your hand.
14 There are no sound cues, and Taste and Smell certainly don't provide
15 any help. You know where your hand is without your other senses
16 because of Proprioception.
18 Humans can sometimes loose this sense through viral infections or
19 damage to the spinal cord or brain, and when they do, they loose the
20 ability to control their own bodies without looking directly at the
21 parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a Hat]],
22 a woman named Christina looses this sense and has to learn how to move
23 by carefully watching her arms and legs. She describes proprioception
24 as the "eyes of the body, the way the body sees itself".
26 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
27 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
28 positions of each body part by monitoring muscle strain and length.
30 It's clear that this is a vital sense for fluid, graceful
31 movement. It's also particularly easy to implement in jMonkeyEngine.
33 My simulated proprioception calculates the relative angles of each
34 joint from the rest position defined in the blender file. This
35 simulates the muscle-spindles and joint capsules. I will deal with
36 Golgi tendon organs, which calculate muscle strain, in the [[./movement.org][next post]].
38 * Helper Functions
40 =absolute-angle= calculates the angle between two vectors, relative to a
41 third axis vector. This angle is the number of radians you have to
42 move counterclockwise around the axis vector to get from the first to
43 the second vector. It is not commutative like a normal dot-product
44 angle is.
46 #+name: helpers
47 #+begin_src clojure
48 (in-ns 'cortex.proprioception)
50 (defn right-handed?
51 "true iff the three vectors form a right handed coordinate
52 system. The three vectors do not have to be normalized or
53 orthogonal."
54 [vec1 vec2 vec3]
55 (< 0 (.dot (.cross vec1 vec2) vec3)))
57 (defn absolute-angle
58 "The angle between 'vec1 and 'vec2 around 'axis. In the range
59 [0 (* 2 Math/PI)]."
60 [vec1 vec2 axis]
61 (let [angle (.angleBetween vec1 vec2)]
62 (if (right-handed? vec1 vec2 axis)
63 angle (- (* 2 Math/PI) angle))))
64 #+end_src
66 #+begin_src clojure :exports both
67 (in-ns 'cortex.proprioception)
68 (absolute-angle Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
69 #+end_src
71 #+results:
72 : 1.5707964
74 #+begin_src clojure :exports both
75 (in-ns 'cortex.proprioception)
76 (absolute-angle
77 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)
78 #+end_src
80 #+results:
81 : 4.7123889366733
83 * Proprioception Kernel
85 Given a joint, =proprioception-kernel= produces a function that
86 calculates the Euler angles between the the objects the joint
87 connects.
89 #+name: proprioception
90 #+begin_src clojure
91 (defn proprioception-kernel
92 "Returns a function which returns proprioceptive sensory data when
93 called inside a running simulation."
94 [#^Node parts #^Node joint]
95 (let [[obj-a obj-b] (joint-targets parts joint)
96 joint-rot (.getWorldRotation joint)
97 x0 (.mult joint-rot Vector3f/UNIT_X)
98 y0 (.mult joint-rot Vector3f/UNIT_Y)
99 z0 (.mult joint-rot Vector3f/UNIT_Z)]
100 (fn []
101 (let [rot-a (.clone (.getWorldRotation obj-a))
102 rot-b (.clone (.getWorldRotation obj-b))
103 x (.mult rot-a x0)
104 y (.mult rot-a y0)
105 z (.mult rot-a z0)
107 X (.mult rot-b x0)
108 Y (.mult rot-b y0)
109 Z (.mult rot-b z0)
110 heading (Math/atan2 (.dot X z) (.dot X x))
111 pitch (Math/atan2 (.dot X y) (.dot X x))
113 ;; rotate x-vector back to origin
114 reverse
115 (doto (Quaternion.)
116 (.fromAngleAxis
117 (.angleBetween X x)
118 (let [cross (.normalize (.cross X x))]
119 (if (= 0 (.length cross)) y cross))))
120 roll (absolute-angle (.mult reverse Y) y x)]
121 [heading pitch roll]))))
123 (defn proprioception!
124 "Endow the creature with the sense of proprioception. Returns a
125 sequence of functions, one for each child of the \"joints\" node in
126 the creature, which each report proprioceptive information about
127 that joint."
128 [#^Node creature]
129 ;; extract the body's joints
130 (let [senses (map (partial proprioception-kernel creature)
131 (joints creature))]
132 (fn []
133 (map #(%) senses))))
134 #+end_src
137 =proprioception!= maps =proprioception-kernel= across all the
138 joints of the creature. It uses the same list of joints that
139 =cortex.body/joints= uses.
141 * Visualizing Proprioception
143 Proprioception has the lowest bandwidth of all the senses so far, and
144 it doesn't lend itself as readily to visual representation like
145 vision, hearing, or touch. This visualization code creates a "gauge"
146 to view each of the three relative angles along a circle.
148 #+name: visualize
149 #+begin_src clojure
150 (in-ns 'cortex.proprioception)
152 (defn draw-sprite [image sprite x y color ]
153 (dorun
154 (for [[u v] sprite]
155 (.setRGB image (+ u x) (+ v y) color))))
157 (defn view-angle
158 "create a debug view of an angle"
159 [color]
160 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
161 previous (atom [25 25])
162 sprite [[0 0] [0 1]
163 [0 -1] [-1 0] [1 0]]]
164 (fn [angle]
165 (let [angle (float angle)]
166 (let [position
167 [(+ 25 (int (* 20 (Math/cos angle))))
168 (+ 25 (int (* -20 (Math/sin angle))))]]
169 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
170 (draw-sprite image sprite (position 0) (position 1) color)
171 (reset! previous position))
172 image))))
174 (defn proprioception-display-kernel
175 "Display proprioception angles in a BufferedImage"
176 [[h p r]]
177 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
178 previous-heading (atom [25 25])
179 previous-pitch (atom [25 25])
180 previous-roll (atom [25 25])
182 heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
183 pitch-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
184 roll-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
185 draw-angle
186 (fn [angle sprite previous color]
187 (let [angle (float angle)]
188 (let [position
189 [(+ 25 (int (* 20 (Math/cos angle))))
190 (+ 25 (int (* -20 (Math/sin angle))))]]
191 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
192 (draw-sprite image sprite (position 0) (position 1) color)
193 (reset! previous position))
194 image))]
195 (dorun (map draw-angle
196 [h p r]
197 [heading-sprite pitch-sprite roll-sprite]
198 [previous-heading previous-pitch previous-roll]
199 [0xFF0000 0x00FF00 0xFFFFFF]))
200 image))
202 (defn view-proprioception
203 "Creates a function which accepts a list of proprioceptive data and
204 display each element of the list to the screen as an image."
205 []
206 (view-sense proprioception-display-kernel))
207 #+end_src
209 * Proprioception Test
210 This test does not use the worm, but instead uses two bars, bound
211 together by a point2point joint. One bar is fixed, and I control the
212 other bar from the keyboard.
214 #+name: test-proprioception
215 #+begin_src clojure
216 (in-ns 'cortex.test.proprioception)
218 (defn test-proprioception
219 "Testing proprioception:
220 You should see two floating bars, and a display of pitch, yaw, and
221 roll. The white dot measures pitch (spin around the long axis), the
222 green dot measures yaw (in this case, rotation around a circle
223 perpendicular to your line of view), and the red dot measures
224 roll (rotation around a circle perlendicular to the the other two
225 circles).
227 Keys:
228 r : rotate along long axis
229 t : opposite direction of rotation as <r>
231 f : rotate in field of view
232 g : opposite direction of rotation as <f>
234 v : rotate in final direction
235 b : opposite direction of rotation as <v>"
237 ([] (test-proprioception false))
238 ([record?]
239 (let [hand (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
240 :mass 0 :color ColorRGBA/Gray :name "hand")
241 finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
242 :mass 1
243 :color
244 (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1)
245 :name "finger")
246 joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
247 :position (Vector3f. 0 1.2 0)
248 :rotation (doto (Quaternion.)
249 (.fromAngleAxis
250 (/ Math/PI 2)
251 (Vector3f. 0 0 1)))
252 :physical? false)
253 creature (nodify [hand finger joint-node])
254 finger-control (.getControl finger RigidBodyControl)
255 hand-control (.getControl hand RigidBodyControl)
256 joint (joint-dispatch {:type :point} hand-control finger-control
257 (Vector3f. 0 1.2 0)
258 (Vector3f. 0 -1.2 0) nil)
260 root (nodify [creature])
261 prop (proprioception-kernel creature joint-node)
262 prop-view (view-proprioception)]
263 (.setCollisionGroup
264 (.getControl hand RigidBodyControl)
265 PhysicsCollisionObject/COLLISION_GROUP_NONE)
266 (apply
267 world
268 (with-movement
269 finger
270 ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
271 [1 1 10 10 10 10]
272 [root
273 standard-debug-controls
274 (fn [world]
275 (if record?
276 (Capture/captureVideo
277 world
278 (File. "/home/r/proj/cortex/render/proprio/main-view")))
279 (.setTimer world (com.aurellem.capture.RatchetTimer. 60))
280 (set-gravity world (Vector3f. 0 0 0))
281 (enable-debug world)
282 (light-up-everything world))
283 (fn [_ _]
284 (prop-view
285 (list (prop))
286 (if record?
287 (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))
288 #+end_src
290 #+results: test-proprioception
291 : #'cortex.test.proprioception/test-proprioception
293 * Video of Proprioception
295 #+begin_html
296 <div class="figure">
297 <center>
298 <video controls="controls" width="550">
299 <source src="../video/test-proprioception.ogg" type="video/ogg"
300 preload="none" poster="../images/aurellem-1280x480.png" />
301 </video>
302 <br> <a href="http://youtu.be/JjdDmyM8b0w"> YouTube </a>
303 </center>
304 <p>Proprioception in a simple creature. The proprioceptive readout is
305 in the upper left corner of the screen.</p>
306 </div>
307 #+end_html
309 ** Generating the Proprioception Video
310 #+name: magick6
311 #+begin_src clojure
312 (ns cortex.video.magick6
313 (:import java.io.File)
314 (:use clojure.java.shell))
316 (defn images [path]
317 (sort (rest (file-seq (File. path)))))
319 (def base "/home/r/proj/cortex/render/proprio/")
321 (defn pics [file]
322 (images (str base file)))
324 (defn combine-images []
325 (let [main-view (pics "main-view")
326 proprioception (pics "proprio/0")
327 targets (map
328 #(File. (str base "out/" (format "%07d.png" %)))
329 (range 0 (count main-view)))]
330 (dorun
331 (pmap
332 (comp
333 (fn [[ main-view proprioception target]]
334 (println target)
335 (sh "convert"
336 main-view
337 proprioception "-geometry" "+20+20" "-composite"
338 target))
339 (fn [& args] (map #(.getCanonicalPath %) args)))
340 main-view proprioception targets))))
341 #+end_src
343 #+begin_src sh :results silent
344 cd ~/proj/cortex/render/proprio
345 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \
346 test-proprioception.ogg
347 #+end_src
349 * Headers
350 #+name: proprioception-header
351 #+begin_src clojure
352 (ns cortex.proprioception
353 "Simulate the sense of proprioception (ability to detect the
354 relative positions of body parts with respect to other body parts)
355 in jMonkeyEngine3. Reads specially prepared blender files to
356 automatically generate proprioceptive senses."
357 (:use (cortex world util sense body))
358 (:import com.jme3.scene.Node)
359 (:import java.awt.image.BufferedImage)
360 (:import (com.jme3.math Vector3f Quaternion)))
361 #+end_src
363 #+name: test-proprioception-header
364 #+begin_src clojure
365 (ns cortex.test.proprioception
366 (:import (com.aurellem.capture Capture RatchetTimer))
367 (:use (cortex util world proprioception body))
368 (:import java.io.File)
369 (:import com.jme3.bullet.control.RigidBodyControl)
370 (:import com.jme3.bullet.collision.PhysicsCollisionObject)
371 (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))
372 #+end_src
374 * Source Listing
375 - [[../src/cortex/proprioception.clj][cortex.proprioception]]
376 - [[../src/cortex/test/touch.clj][cortex.test.proprioception]]
377 - [[../src/cortex/video/magick6.clj][cortex.video.magick6]]
378 #+html: <ul> <li> <a href="../org/proprioception.org">This org file</a> </li> </ul>
379 - [[http://hg.bortreb.com ][source-repository]]
381 * Next
383 Next time, I'll give the Worm the power to [[./movement.org][move on it's own]].
386 * COMMENT generate source
387 #+begin_src clojure :tangle ../src/cortex/proprioception.clj
388 <<proprioception-header>>
389 <<helpers>>
390 <<proprioception>>
391 <<visualize>>
392 #+end_src
394 #+begin_src clojure :tangle ../src/cortex/test/proprioception.clj
395 <<test-proprioception-header>>
396 <<test-proprioception>>
397 #+end_src
399 #+begin_src clojure :tangle ../src/cortex/video/magick6.clj
400 <<magick6>>
401 #+end_src