Mercurial > cortex

annotate org/proprioception.org @ 470:3401053124b0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
integrating vision into thesis.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 17:10:43 -0400
parents 5205535237fb
children
rev   line source
rlm@157 1 #+title: The Sense of Proprioception
rlm@157 2 #+author: Robert McIntyre
rlm@157 3 #+email: rlm@mit.edu
rlm@157 4 #+description: proprioception for simulated creatures
rlm@157 5 #+keywords: simulation, jMonkeyEngine3, clojure
rlm@157 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@157 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@157 8
rlm@257 9 * Proprioception
rlm@257 10
rlm@257 11 Close your eyes, and touch your nose with your right index finger. How
rlm@257 12 did you do it? You could not see your hand, and neither your hand nor
rlm@257 13 your nose could use the sense of touch to guide the path of your hand.
rlm@257 14 There are no sound cues, and Taste and Smell certainly don't provide
rlm@257 15 any help. You know where your hand is without your other senses
rlm@257 16 because of Proprioception.
rlm@257 17
rlm@257 18 Humans can sometimes loose this sense through viral infections or
rlm@257 19 damage to the spinal cord or brain, and when they do, they loose the
rlm@257 20 ability to control their own bodies without looking directly at the
rlm@257 21 parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a Hat]],
rlm@257 22 a woman named Christina looses this sense and has to learn how to move
rlm@257 23 by carefully watching her arms and legs. She describes proprioception
rlm@257 24 as the "eyes of the body, the way the body sees itself".
rlm@257 25
rlm@257 26 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
rlm@257 27 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
rlm@306 28 positions of each body part by monitoring muscle strain and length.
rlm@257 29
rlm@306 30 It's clear that this is a vital sense for fluid, graceful
rlm@306 31 movement. It's also particularly easy to implement in jMonkeyEngine.
rlm@257 32
rlm@257 33 My simulated proprioception calculates the relative angles of each
rlm@257 34 joint from the rest position defined in the blender file. This
rlm@257 35 simulates the muscle-spindles and joint capsules. I will deal with
rlm@257 36 Golgi tendon organs, which calculate muscle strain, in the [[./movement.org][next post]].
rlm@257 37
rlm@257 38 * Helper Functions
rlm@257 39
rlm@273 40 =absolute-angle= calculates the angle between two vectors, relative to a
rlm@259 41 third axis vector. This angle is the number of radians you have to
rlm@259 42 move counterclockwise around the axis vector to get from the first to
rlm@259 43 the second vector. It is not commutative like a normal dot-product
rlm@259 44 angle is.
rlm@259 45
rlm@257 46 #+name: helpers
rlm@157 47 #+begin_src clojure
rlm@257 48 (in-ns 'cortex.proprioception)
rlm@157 49
rlm@173 50 (defn right-handed?
rlm@173 51 "true iff the three vectors form a right handed coordinate
rlm@257 52 system. The three vectors do not have to be normalized or
rlm@257 53 orthogonal."
rlm@173 54 [vec1 vec2 vec3]
rlm@430 55 (pos? (.dot (.cross vec1 vec2) vec3)))
rlm@157 56
rlm@173 57 (defn absolute-angle
rlm@259 58 "The angle between 'vec1 and 'vec2 around 'axis. In the range
rlm@259 59 [0 (* 2 Math/PI)]."
rlm@173 60 [vec1 vec2 axis]
rlm@157 61 (let [angle (.angleBetween vec1 vec2)]
rlm@157 62 (if (right-handed? vec1 vec2 axis)
rlm@157 63 angle (- (* 2 Math/PI) angle))))
rlm@257 64 #+end_src
rlm@157 65
rlm@259 66 #+begin_src clojure :exports both
rlm@259 67 (in-ns 'cortex.proprioception)
rlm@259 68 (absolute-angle Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
rlm@259 69 #+end_src
rlm@259 70
rlm@259 71 #+results:
rlm@259 72 : 1.5707964
rlm@259 73
rlm@259 74 #+begin_src clojure :exports both
rlm@259 75 (in-ns 'cortex.proprioception)
rlm@259 76 (absolute-angle
rlm@259 77 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)
rlm@259 78 #+end_src
rlm@259 79
rlm@259 80 #+results:
rlm@259 81 : 4.7123889366733
rlm@259 82
rlm@258 83 * Proprioception Kernel
rlm@258 84
rlm@273 85 Given a joint, =proprioception-kernel= produces a function that
rlm@306 86 calculates the Euler angles between the the objects the joint
rlm@259 87 connects.
rlm@259 88
rlm@257 89 #+name: proprioception
rlm@257 90 #+begin_src clojure
rlm@257 91 (defn proprioception-kernel
rlm@173 92 "Returns a function which returns proprioceptive sensory data when
rlm@173 93 called inside a running simulation."
rlm@173 94 [#^Node parts #^Node joint]
rlm@157 95 (let [[obj-a obj-b] (joint-targets parts joint)
rlm@157 96 joint-rot (.getWorldRotation joint)
rlm@157 97 x0 (.mult joint-rot Vector3f/UNIT_X)
rlm@157 98 y0 (.mult joint-rot Vector3f/UNIT_Y)
rlm@157 99 z0 (.mult joint-rot Vector3f/UNIT_Z)]
rlm@157 100 (fn []
rlm@157 101 (let [rot-a (.clone (.getWorldRotation obj-a))
rlm@157 102 rot-b (.clone (.getWorldRotation obj-b))
rlm@157 103 x (.mult rot-a x0)
rlm@157 104 y (.mult rot-a y0)
rlm@157 105 z (.mult rot-a z0)
rlm@157 106
rlm@157 107 X (.mult rot-b x0)
rlm@157 108 Y (.mult rot-b y0)
rlm@157 109 Z (.mult rot-b z0)
rlm@157 110 heading (Math/atan2 (.dot X z) (.dot X x))
rlm@157 111 pitch (Math/atan2 (.dot X y) (.dot X x))
rlm@157 112
rlm@157 113 ;; rotate x-vector back to origin
rlm@157 114 reverse
rlm@157 115 (doto (Quaternion.)
rlm@157 116 (.fromAngleAxis
rlm@157 117 (.angleBetween X x)
rlm@157 118 (let [cross (.normalize (.cross X x))]
rlm@157 119 (if (= 0 (.length cross)) y cross))))
rlm@157 120 roll (absolute-angle (.mult reverse Y) y x)]
rlm@157 121 [heading pitch roll]))))
rlm@157 122
rlm@173 123 (defn proprioception!
rlm@173 124 "Endow the creature with the sense of proprioception. Returns a
rlm@173 125 sequence of functions, one for each child of the \"joints\" node in
rlm@173 126 the creature, which each report proprioceptive information about
rlm@173 127 that joint."
rlm@157 128 [#^Node creature]
rlm@157 129 ;; extract the body's joints
rlm@257 130 (let [senses (map (partial proprioception-kernel creature)
rlm@173 131 (joints creature))]
rlm@157 132 (fn []
rlm@157 133 (map #(%) senses))))
rlm@257 134 #+end_src
rlm@175 135
rlm@259 136
rlm@273 137 =proprioception!= maps =proprioception-kernel= across all the
rlm@259 138 joints of the creature. It uses the same list of joints that
rlm@273 139 =cortex.body/joints= uses.
rlm@259 140
rlm@258 141 * Visualizing Proprioception
rlm@258 142
rlm@259 143 Proprioception has the lowest bandwidth of all the senses so far, and
rlm@259 144 it doesn't lend itself as readily to visual representation like
rlm@306 145 vision, hearing, or touch. This visualization code creates a "gauge"
rlm@259 146 to view each of the three relative angles along a circle.
rlm@259 147
rlm@257 148 #+name: visualize
rlm@257 149 #+begin_src clojure
rlm@257 150 (in-ns 'cortex.proprioception)
rlm@175 151
rlm@175 152 (defn draw-sprite [image sprite x y color ]
rlm@175 153 (dorun
rlm@175 154 (for [[u v] sprite]
rlm@175 155 (.setRGB image (+ u x) (+ v y) color))))
rlm@175 156
rlm@175 157 (defn view-angle
rlm@175 158 "create a debug view of an angle"
rlm@175 159 [color]
rlm@175 160 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
rlm@175 161 previous (atom [25 25])
rlm@175 162 sprite [[0 0] [0 1]
rlm@175 163 [0 -1] [-1 0] [1 0]]]
rlm@175 164 (fn [angle]
rlm@175 165 (let [angle (float angle)]
rlm@175 166 (let [position
rlm@175 167 [(+ 25 (int (* 20 (Math/cos angle))))
rlm@175 168 (+ 25 (int (* -20 (Math/sin angle))))]]
rlm@175 169 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
rlm@175 170 (draw-sprite image sprite (position 0) (position 1) color)
rlm@175 171 (reset! previous position))
rlm@175 172 image))))
rlm@175 173
rlm@190 174 (defn proprioception-display-kernel
rlm@190 175 "Display proprioception angles in a BufferedImage"
rlm@190 176 [[h p r]]
rlm@190 177 (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
rlm@190 178 previous-heading (atom [25 25])
rlm@190 179 previous-pitch (atom [25 25])
rlm@190 180 previous-roll (atom [25 25])
rlm@190 181
rlm@190 182 heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
rlm@190 183 pitch-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
rlm@190 184 roll-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
rlm@190 185 draw-angle
rlm@190 186 (fn [angle sprite previous color]
rlm@190 187 (let [angle (float angle)]
rlm@190 188 (let [position
rlm@190 189 [(+ 25 (int (* 20 (Math/cos angle))))
rlm@190 190 (+ 25 (int (* -20 (Math/sin angle))))]]
rlm@190 191 (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
rlm@190 192 (draw-sprite image sprite (position 0) (position 1) color)
rlm@190 193 (reset! previous position))
rlm@190 194 image))]
rlm@190 195 (dorun (map draw-angle
rlm@190 196 [h p r]
rlm@190 197 [heading-sprite pitch-sprite roll-sprite]
rlm@190 198 [previous-heading previous-pitch previous-roll]
rlm@190 199 [0xFF0000 0x00FF00 0xFFFFFF]))
rlm@190 200 image))
rlm@190 201
rlm@190 202 (defn view-proprioception
rlm@190 203 "Creates a function which accepts a list of proprioceptive data and
rlm@190 204 display each element of the list to the screen as an image."
rlm@175 205 []
rlm@190 206 (view-sense proprioception-display-kernel))
rlm@257 207 #+end_src
rlm@175 208
rlm@259 209 * Proprioception Test
rlm@259 210 This test does not use the worm, but instead uses two bars, bound
rlm@259 211 together by a point2point joint. One bar is fixed, and I control the
rlm@259 212 other bar from the keyboard.
rlm@157 213
rlm@259 214 #+name: test-proprioception
rlm@206 215 #+begin_src clojure
rlm@259 216 (in-ns 'cortex.test.proprioception)
rlm@259 217
rlm@206 218 (defn test-proprioception
rlm@206 219 "Testing proprioception:
rlm@321 220 You should see two floating bars, and a display of pitch, yaw, and
rlm@321 221 roll. The white dot measures pitch (spin around the long axis), the
rlm@321 222 green dot measures yaw (in this case, rotation around a circle
rlm@321 223 perpendicular to your line of view), and the red dot measures
rlm@321 224 roll (rotation around a circle perlendicular to the the other two
rlm@321 225 circles).
rlm@321 226
rlm@321 227 Keys:
rlm@321 228 r : rotate along long axis
rlm@321 229 t : opposite direction of rotation as <r>
rlm@321 230
rlm@321 231 f : rotate in field of view
rlm@321 232 g : opposite direction of rotation as <f>
rlm@321 233
rlm@321 234 v : rotate in final direction
rlm@321 235 b : opposite direction of rotation as <v>"
rlm@321 236
rlm@259 237 ([] (test-proprioception false))
rlm@259 238 ([record?]
rlm@259 239 (let [hand (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
rlm@259 240 :mass 0 :color ColorRGBA/Gray :name "hand")
rlm@259 241 finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
rlm@259 242 :mass 1
rlm@259 243 :color
rlm@259 244 (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1)
rlm@259 245 :name "finger")
rlm@259 246 joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
rlm@259 247 :position (Vector3f. 0 1.2 0)
rlm@259 248 :rotation (doto (Quaternion.)
rlm@259 249 (.fromAngleAxis
rlm@259 250 (/ Math/PI 2)
rlm@259 251 (Vector3f. 0 0 1)))
rlm@259 252 :physical? false)
rlm@259 253 creature (nodify [hand finger joint-node])
rlm@259 254 finger-control (.getControl finger RigidBodyControl)
rlm@259 255 hand-control (.getControl hand RigidBodyControl)
rlm@259 256 joint (joint-dispatch {:type :point} hand-control finger-control
rlm@259 257 (Vector3f. 0 1.2 0)
rlm@259 258 (Vector3f. 0 -1.2 0) nil)
rlm@206 259
rlm@259 260 root (nodify [creature])
rlm@259 261 prop (proprioception-kernel creature joint-node)
rlm@259 262 prop-view (view-proprioception)]
rlm@259 263 (.setCollisionGroup
rlm@259 264 (.getControl hand RigidBodyControl)
rlm@259 265 PhysicsCollisionObject/COLLISION_GROUP_NONE)
rlm@259 266 (apply
rlm@259 267 world
rlm@259 268 (with-movement
rlm@259 269 finger
rlm@259 270 ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
rlm@259 271 [1 1 10 10 10 10]
rlm@259 272 [root
rlm@259 273 standard-debug-controls
rlm@259 274 (fn [world]
rlm@340 275 (let [timer (RatchetTimer. 60)]
rlm@340 276 (.setTimer world timer)
rlm@340 277 (display-dilated-time world timer))
rlm@259 278 (if record?
rlm@259 279 (Capture/captureVideo
rlm@259 280 world
rlm@259 281 (File. "/home/r/proj/cortex/render/proprio/main-view")))
rlm@259 282 (set-gravity world (Vector3f. 0 0 0))
rlm@259 283 (enable-debug world)
rlm@259 284 (light-up-everything world))
rlm@259 285 (fn [_ _]
rlm@259 286 (prop-view
rlm@259 287 (list (prop))
rlm@259 288 (if record?
rlm@259 289 (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))
rlm@259 290 #+end_src
rlm@206 291
rlm@259 292 #+results: test-proprioception
rlm@259 293 : #'cortex.test.proprioception/test-proprioception
rlm@259 294
rlm@259 295 * Video of Proprioception
rlm@259 296
rlm@259 297 #+begin_html
rlm@259 298 <div class="figure">
rlm@259 299 <center>
rlm@259 300 <video controls="controls" width="550">
rlm@259 301 <source src="../video/test-proprioception.ogg" type="video/ogg"
rlm@259 302 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@259 303 </video>
rlm@309 304 <br> <a href="http://youtu.be/JjdDmyM8b0w"> YouTube </a>
rlm@259 305 </center>
rlm@259 306 <p>Proprioception in a simple creature. The proprioceptive readout is
rlm@259 307 in the upper left corner of the screen.</p>
rlm@259 308 </div>
rlm@259 309 #+end_html
rlm@259 310
rlm@259 311 ** Generating the Proprioception Video
rlm@259 312 #+name: magick6
rlm@259 313 #+begin_src clojure
rlm@259 314 (ns cortex.video.magick6
rlm@259 315 (:import java.io.File)
rlm@316 316 (:use clojure.java.shell))
rlm@259 317
rlm@259 318 (defn images [path]
rlm@259 319 (sort (rest (file-seq (File. path)))))
rlm@259 320
rlm@259 321 (def base "/home/r/proj/cortex/render/proprio/")
rlm@259 322
rlm@259 323 (defn pics [file]
rlm@259 324 (images (str base file)))
rlm@259 325
rlm@259 326 (defn combine-images []
rlm@259 327 (let [main-view (pics "main-view")
rlm@259 328 proprioception (pics "proprio/0")
rlm@259 329 targets (map
rlm@259 330 #(File. (str base "out/" (format "%07d.png" %)))
rlm@430 331 (range (count main-view)))]
rlm@259 332 (dorun
rlm@259 333 (pmap
rlm@259 334 (comp
rlm@259 335 (fn [[ main-view proprioception target]]
rlm@259 336 (println target)
rlm@259 337 (sh "convert"
rlm@259 338 main-view
rlm@259 339 proprioception "-geometry" "+20+20" "-composite"
rlm@259 340 target))
rlm@259 341 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@259 342 main-view proprioception targets))))
rlm@259 343 #+end_src
rlm@259 344
rlm@259 345 #+begin_src sh :results silent
rlm@259 346 cd ~/proj/cortex/render/proprio
rlm@259 347 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \
rlm@259 348 test-proprioception.ogg
rlm@258 349 #+end_src
rlm@206 350
rlm@258 351 * Headers
rlm@258 352 #+name: proprioception-header
rlm@258 353 #+begin_src clojure
rlm@258 354 (ns cortex.proprioception
rlm@258 355 "Simulate the sense of proprioception (ability to detect the
rlm@306 356 relative positions of body parts with respect to other body parts)
rlm@258 357 in jMonkeyEngine3. Reads specially prepared blender files to
rlm@258 358 automatically generate proprioceptive senses."
rlm@258 359 (:use (cortex world util sense body))
rlm@258 360 (:import com.jme3.scene.Node)
rlm@258 361 (:import java.awt.image.BufferedImage)
rlm@258 362 (:import (com.jme3.math Vector3f Quaternion)))
rlm@206 363 #+end_src
rlm@206 364
rlm@259 365 #+name: test-proprioception-header
rlm@259 366 #+begin_src clojure
rlm@259 367 (ns cortex.test.proprioception
rlm@340 368 (:import (com.aurellem.capture Capture RatchetTimer IsoTimer))
rlm@283 369 (:use (cortex util world proprioception body))
rlm@283 370 (:import java.io.File)
rlm@283 371 (:import com.jme3.bullet.control.RigidBodyControl)
rlm@283 372 (:import com.jme3.bullet.collision.PhysicsCollisionObject)
rlm@283 373 (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))
rlm@259 374 #+end_src
rlm@259 375
rlm@340 376 #+results: test-proprioception-header
rlm@340 377 : com.jme3.math.ColorRGBA
rlm@340 378
rlm@259 379 * Source Listing
rlm@259 380 - [[../src/cortex/proprioception.clj][cortex.proprioception]]
rlm@259 381 - [[../src/cortex/test/touch.clj][cortex.test.proprioception]]
rlm@259 382 - [[../src/cortex/video/magick6.clj][cortex.video.magick6]]
rlm@259 383 #+html: <ul> <li> <a href="../org/proprioception.org">This org file</a> </li> </ul>
rlm@259 384 - [[http://hg.bortreb.com ][source-repository]]
rlm@259 385
rlm@259 386 * Next
rlm@259 387
rlm@430 388 Next time, I'll give the Worm the power to [[./movement.org][move on its own]].
rlm@259 389
rlm@206 390
rlm@157 391 * COMMENT generate source
rlm@157 392 #+begin_src clojure :tangle ../src/cortex/proprioception.clj
rlm@257 393 <<proprioception-header>>
rlm@257 394 <<helpers>>
rlm@157 395 <<proprioception>>
rlm@257 396 <<visualize>>
rlm@157 397 #+end_src
rlm@259 398
rlm@259 399 #+begin_src clojure :tangle ../src/cortex/test/proprioception.clj
rlm@259 400 <<test-proprioception-header>>
rlm@259 401 <<test-proprioception>>
rlm@259 402 #+end_src
rlm@259 403
rlm@259 404 #+begin_src clojure :tangle ../src/cortex/video/magick6.clj
rlm@259 405 <<magick6>>
rlm@259 406 #+end_src