view org/proprioception.org @ 470:3401053124b0

integrating vision into thesis.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 17:10:43 -0400
parents 5205535237fb
children
line wrap: on
line source
1 #+title: The Sense of Proprioception
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: proprioception for simulated creatures
5 #+keywords: simulation, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Proprioception
11 Close your eyes, and touch your nose with your right index finger. How
12 did you do it? You could not see your hand, and neither your hand nor
13 your nose could use the sense of touch to guide the path of your hand.
14 There are no sound cues, and Taste and Smell certainly don't provide
15 any help. You know where your hand is without your other senses
16 because of Proprioception.
18 Humans can sometimes loose this sense through viral infections or
19 damage to the spinal cord or brain, and when they do, they loose the
20 ability to control their own bodies without looking directly at the
21 parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a Hat]],
22 a woman named Christina looses this sense and has to learn how to move
23 by carefully watching her arms and legs. She describes proprioception
24 as the "eyes of the body, the way the body sees itself".
26 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
27 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
28 positions of each body part by monitoring muscle strain and length.
30 It's clear that this is a vital sense for fluid, graceful
31 movement. It's also particularly easy to implement in jMonkeyEngine.
33 My simulated proprioception calculates the relative angles of each
34 joint from the rest position defined in the blender file. This
35 simulates the muscle-spindles and joint capsules. I will deal with
36 Golgi tendon organs, which calculate muscle strain, in the [[./movement.org][next post]].
38 * Helper Functions
40 =absolute-angle= calculates the angle between two vectors, relative to a
41 third axis vector. This angle is the number of radians you have to
42 move counterclockwise around the axis vector to get from the first to
43 the second vector. It is not commutative like a normal dot-product
44 angle is.
46 #+name: helpers
47 #+begin_src clojure
48 (in-ns 'cortex.proprioception)
50 (defn right-handed?
51 "true iff the three vectors form a right handed coordinate
52 system. The three vectors do not have to be normalized or
53 orthogonal."
54 [vec1 vec2 vec3]
55 (pos? (.dot (.cross vec1 vec2) vec3)))
57 (defn absolute-angle
58 "The angle between 'vec1 and 'vec2 around 'axis. In the range
59 [0 (* 2 Math/PI)]."
60 [vec1 vec2 axis]
61 (let [angle (.angleBetween vec1 vec2)]
62 (if (right-handed? vec1 vec2 axis)
63 angle (- (* 2 Math/PI) angle))))
64 #+end_src
66 #+begin_src clojure :exports both
67 (in-ns 'cortex.proprioception)
68 (absolute-angle Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
69 #+end_src
71 #+results:
72 : 1.5707964
74 #+begin_src clojure :exports both
75 (in-ns 'cortex.proprioception)
76 (absolute-angle
77 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)
78 #+end_src
80 #+results:
81 : 4.7123889366733
83 * Proprioception Kernel
85 Given a joint, =proprioception-kernel= produces a function that
86 calculates the Euler angles between the the objects the joint
87 connects.
89 #+name: proprioception
90 #+begin_src clojure
91 (defn proprioception-kernel
92 "Returns a function which returns proprioceptive sensory data when
93 called inside a running simulation."
94 [#^Node parts #^Node joint]
95 (let [[obj-a obj-b] (joint-targets parts joint)
96 joint-rot (.getWorldRotation joint)
97 x0 (.mult joint-rot Vector3f/UNIT_X)
98 y0 (.mult joint-rot Vector3f/UNIT_Y)
99 z0 (.mult joint-rot Vector3f/UNIT_Z)]
100 (fn []
101 (let [rot-a (.clone (.getWorldRotation obj-a))
102 rot-b (.clone (.getWorldRotation obj-b))
103 x (.mult rot-a x0)
104 y (.mult rot-a y0)
105 z (.mult rot-a z0)
107 X (.mult rot-b x0)
108 Y (.mult rot-b y0)
109 Z (.mult rot-b z0)
110 heading (Math/atan2 (.dot X z) (.dot X x))
111 pitch (Math/atan2 (.dot X y) (.dot X x))
113 ;; rotate x-vector back to origin
114 reverse
115 (doto (Quaternion.)
116 (.fromAngleAxis
117 (.angleBetween X x)
118 (let [cross (.normalize (.cross X x))]
119 (if (= 0 (.length cross)) y cross))))
120 roll (absolute-angle (.mult reverse Y) y x)]
121 [heading pitch roll]))))
123 (defn proprioception!
124 "Endow the creature with the sense of proprioception. Returns a
125 sequence of functions, one for each child of the \"joints\" node in
126 the creature, which each report proprioceptive information about
127 that joint."
128 [#^Node creature]
129 ;; extract the body's joints
130 (let [senses (map (partial proprioception-kernel creature)
131 (joints creature))]
132 (fn []
133 (map #(%) senses))))
134 #+end_src
137 =proprioception!= maps =proprioception-kernel= across all the
138 joints of the creature. It uses the same list of joints that
139 =cortex.body/joints= uses.
141 * Visualizing Proprioception
143 Proprioception has the lowest bandwidth of all the senses so far, and
144 it doesn't lend itself as readily to visual representation like
145 vision, hearing, or touch. This visualization code creates a "gauge"
146 to view each of the three relative angles along a circle.
148 #+name: visualize
149 #+begin_src clojure
150 (in-ns 'cortex.proprioception)
152 (defn draw-sprite [image sprite x y color ]
153 (dorun
154 (for [[u v] sprite]
155 (.setRGB image (+ u x) (+ v y) color))))
157 (defn view-angle
158 "create a debug view of an angle"
159 [color]
160 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
161 previous (atom [25 25])
162 sprite [[0 0] [0 1]
163 [0 -1] [-1 0] [1 0]]]
164 (fn [angle]
165 (let [angle (float angle)]
166 (let [position
167 [(+ 25 (int (* 20 (Math/cos angle))))
168 (+ 25 (int (* -20 (Math/sin angle))))]]
169 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
170 (draw-sprite image sprite (position 0) (position 1) color)
171 (reset! previous position))
172 image))))
174 (defn proprioception-display-kernel
175 "Display proprioception angles in a BufferedImage"
176 [[h p r]]
177 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
178 previous-heading (atom [25 25])
179 previous-pitch (atom [25 25])
180 previous-roll (atom [25 25])
182 heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
183 pitch-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
184 roll-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
185 draw-angle
186 (fn [angle sprite previous color]
187 (let [angle (float angle)]
188 (let [position
189 [(+ 25 (int (* 20 (Math/cos angle))))
190 (+ 25 (int (* -20 (Math/sin angle))))]]
191 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
192 (draw-sprite image sprite (position 0) (position 1) color)
193 (reset! previous position))
194 image))]
195 (dorun (map draw-angle
196 [h p r]
197 [heading-sprite pitch-sprite roll-sprite]
198 [previous-heading previous-pitch previous-roll]
199 [0xFF0000 0x00FF00 0xFFFFFF]))
200 image))
202 (defn view-proprioception
203 "Creates a function which accepts a list of proprioceptive data and
204 display each element of the list to the screen as an image."
205 []
206 (view-sense proprioception-display-kernel))
207 #+end_src
209 * Proprioception Test
210 This test does not use the worm, but instead uses two bars, bound
211 together by a point2point joint. One bar is fixed, and I control the
212 other bar from the keyboard.
214 #+name: test-proprioception
215 #+begin_src clojure
216 (in-ns 'cortex.test.proprioception)
218 (defn test-proprioception
219 "Testing proprioception:
220 You should see two floating bars, and a display of pitch, yaw, and
221 roll. The white dot measures pitch (spin around the long axis), the
222 green dot measures yaw (in this case, rotation around a circle
223 perpendicular to your line of view), and the red dot measures
224 roll (rotation around a circle perlendicular to the the other two
225 circles).
227 Keys:
228 r : rotate along long axis
229 t : opposite direction of rotation as <r>
231 f : rotate in field of view
232 g : opposite direction of rotation as <f>
234 v : rotate in final direction
235 b : opposite direction of rotation as <v>"
237 ([] (test-proprioception false))
238 ([record?]
239 (let [hand (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
240 :mass 0 :color ColorRGBA/Gray :name "hand")
241 finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
242 :mass 1
243 :color
244 (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1)
245 :name "finger")
246 joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
247 :position (Vector3f. 0 1.2 0)
248 :rotation (doto (Quaternion.)
249 (.fromAngleAxis
250 (/ Math/PI 2)
251 (Vector3f. 0 0 1)))
252 :physical? false)
253 creature (nodify [hand finger joint-node])
254 finger-control (.getControl finger RigidBodyControl)
255 hand-control (.getControl hand RigidBodyControl)
256 joint (joint-dispatch {:type :point} hand-control finger-control
257 (Vector3f. 0 1.2 0)
258 (Vector3f. 0 -1.2 0) nil)
260 root (nodify [creature])
261 prop (proprioception-kernel creature joint-node)
262 prop-view (view-proprioception)]
263 (.setCollisionGroup
264 (.getControl hand RigidBodyControl)
265 PhysicsCollisionObject/COLLISION_GROUP_NONE)
266 (apply
267 world
268 (with-movement
269 finger
270 ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
271 [1 1 10 10 10 10]
272 [root
273 standard-debug-controls
274 (fn [world]
275 (let [timer (RatchetTimer. 60)]
276 (.setTimer world timer)
277 (display-dilated-time world timer))
278 (if record?
279 (Capture/captureVideo
280 world
281 (File. "/home/r/proj/cortex/render/proprio/main-view")))
282 (set-gravity world (Vector3f. 0 0 0))
283 (enable-debug world)
284 (light-up-everything world))
285 (fn [_ _]
286 (prop-view
287 (list (prop))
288 (if record?
289 (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))
290 #+end_src
292 #+results: test-proprioception
293 : #'cortex.test.proprioception/test-proprioception
295 * Video of Proprioception
297 #+begin_html
298 <div class="figure">
299 <center>
300 <video controls="controls" width="550">
301 <source src="../video/test-proprioception.ogg" type="video/ogg"
302 preload="none" poster="../images/aurellem-1280x480.png" />
303 </video>
304 <br> <a href="http://youtu.be/JjdDmyM8b0w"> YouTube </a>
305 </center>
306 <p>Proprioception in a simple creature. The proprioceptive readout is
307 in the upper left corner of the screen.</p>
308 </div>
309 #+end_html
311 ** Generating the Proprioception Video
312 #+name: magick6
313 #+begin_src clojure
314 (ns cortex.video.magick6
315 (:import java.io.File)
316 (:use clojure.java.shell))
318 (defn images [path]
319 (sort (rest (file-seq (File. path)))))
321 (def base "/home/r/proj/cortex/render/proprio/")
323 (defn pics [file]
324 (images (str base file)))
326 (defn combine-images []
327 (let [main-view (pics "main-view")
328 proprioception (pics "proprio/0")
329 targets (map
330 #(File. (str base "out/" (format "%07d.png" %)))
331 (range (count main-view)))]
332 (dorun
333 (pmap
334 (comp
335 (fn [[ main-view proprioception target]]
336 (println target)
337 (sh "convert"
338 main-view
339 proprioception "-geometry" "+20+20" "-composite"
340 target))
341 (fn [& args] (map #(.getCanonicalPath %) args)))
342 main-view proprioception targets))))
343 #+end_src
345 #+begin_src sh :results silent
346 cd ~/proj/cortex/render/proprio
347 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \
348 test-proprioception.ogg
349 #+end_src
351 * Headers
352 #+name: proprioception-header
353 #+begin_src clojure
354 (ns cortex.proprioception
355 "Simulate the sense of proprioception (ability to detect the
356 relative positions of body parts with respect to other body parts)
357 in jMonkeyEngine3. Reads specially prepared blender files to
358 automatically generate proprioceptive senses."
359 (:use (cortex world util sense body))
360 (:import com.jme3.scene.Node)
361 (:import java.awt.image.BufferedImage)
362 (:import (com.jme3.math Vector3f Quaternion)))
363 #+end_src
365 #+name: test-proprioception-header
366 #+begin_src clojure
367 (ns cortex.test.proprioception
368 (:import (com.aurellem.capture Capture RatchetTimer IsoTimer))
369 (:use (cortex util world proprioception body))
370 (:import java.io.File)
371 (:import com.jme3.bullet.control.RigidBodyControl)
372 (:import com.jme3.bullet.collision.PhysicsCollisionObject)
373 (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))
374 #+end_src
376 #+results: test-proprioception-header
377 : com.jme3.math.ColorRGBA
379 * Source Listing
380 - [[../src/cortex/proprioception.clj][cortex.proprioception]]
381 - [[../src/cortex/test/touch.clj][cortex.test.proprioception]]
382 - [[../src/cortex/video/magick6.clj][cortex.video.magick6]]
383 #+html: <ul> <li> <a href="../org/proprioception.org">This org file</a> </li> </ul>
384 - [[http://hg.bortreb.com ][source-repository]]
386 * Next
388 Next time, I'll give the Worm the power to [[./movement.org][move on its own]].
391 * COMMENT generate source
392 #+begin_src clojure :tangle ../src/cortex/proprioception.clj
393 <<proprioception-header>>
394 <<helpers>>
395 <<proprioception>>
396 <<visualize>>
397 #+end_src
399 #+begin_src clojure :tangle ../src/cortex/test/proprioception.clj
400 <<test-proprioception-header>>
401 <<test-proprioception>>
402 #+end_src
404 #+begin_src clojure :tangle ../src/cortex/video/magick6.clj
405 <<magick6>>
406 #+end_src