view org/sense.org @ 336:70469ff8eb56

more Viewable stuff.
author Robert McIntyre <rlm@mit.edu>
date Fri, 20 Jul 2012 13:04:25 -0500
parents 702b5c78c2de
children d37ccb6c888f
line wrap: on
line source
1 #+title: Helper Functions / Motivations
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: sensory utilities
5 #+keywords: simulation, jMonkeyEngine3, clojure, simulated senses
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Blender Utilities
10 In blender, any object can be assigned an arbitrary number of key-value
11 pairs which are called "Custom Properties". These are accessible in
12 jMonkeyEngine when blender files are imported with the
13 =BlenderLoader=. =meta-data= extracts these properties.
15 #+name: blender-1
16 #+begin_src clojure
17 (defn meta-data
18 "Get the meta-data for a node created with blender."
19 [blender-node key]
20 (if-let [data (.getUserData blender-node "properties")]
21 (.findValue data key) nil))
22 #+end_src
24 Blender uses a different coordinate system than jMonkeyEngine so it
25 is useful to be able to convert between the two. These only come into
26 play when the meta-data of a node refers to a vector in the blender
27 coordinate system.
29 #+name: blender-2
30 #+begin_src clojure
31 (defn jme-to-blender
32 "Convert from JME coordinates to Blender coordinates"
33 [#^Vector3f in]
34 (Vector3f. (.getX in) (- (.getZ in)) (.getY in)))
36 (defn blender-to-jme
37 "Convert from Blender coordinates to JME coordinates"
38 [#^Vector3f in]
39 (Vector3f. (.getX in) (.getZ in) (- (.getY in))))
40 #+end_src
42 * Sense Topology
44 Human beings are three-dimensional objects, and the nerves that
45 transmit data from our various sense organs to our brain are
46 essentially one-dimensional. This leaves up to two dimensions in which
47 our sensory information may flow. For example, imagine your skin: it
48 is a two-dimensional surface around a three-dimensional object (your
49 body). It has discrete touch sensors embedded at various points, and
50 the density of these sensors corresponds to the sensitivity of that
51 region of skin. Each touch sensor connects to a nerve, all of which
52 eventually are bundled together as they travel up the spinal cord to
53 the brain. Intersect the spinal nerves with a guillotining plane and
54 you will see all of the sensory data of the skin revealed in a roughly
55 circular two-dimensional image which is the cross section of the
56 spinal cord. Points on this image that are close together in this
57 circle represent touch sensors that are /probably/ close together on
58 the skin, although there is of course some cutting and rearrangement
59 that has to be done to transfer the complicated surface of the skin
60 onto a two dimensional image.
62 Most human senses consist of many discrete sensors of various
63 properties distributed along a surface at various densities. For
64 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
65 disks, and Ruffini's endings, which detect pressure and vibration of
66 various intensities. For ears, it is the stereocilia distributed
67 along the basilar membrane inside the cochlea; each one is sensitive
68 to a slightly different frequency of sound. For eyes, it is rods
69 and cones distributed along the surface of the retina. In each case,
70 we can describe the sense with a surface and a distribution of sensors
71 along that surface.
73 ** UV-maps
75 Blender and jMonkeyEngine already have support for exactly this sort
76 of data structure because it is used to "skin" models for games. It is
77 called [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Textures/Mapping/UV][UV-mapping]]. The three-dimensional surface of a model is cut
78 and smooshed until it fits on a two-dimensional image. You paint
79 whatever you want on that image, and when the three-dimensional shape
80 is rendered in a game the smooshing and cutting us reversed and the
81 image appears on the three-dimensional object.
83 To make a sense, interpret the UV-image as describing the distribution
84 of that senses sensors. To get different types of sensors, you can
85 either use a different color for each type of sensor, or use multiple
86 UV-maps, each labeled with that sensor type. I generally use a white
87 pixel to mean the presence of a sensor and a black pixel to mean the
88 absence of a sensor, and use one UV-map for each sensor-type within a
89 given sense. The paths to the images are not stored as the actual
90 UV-map of the blender object but are instead referenced in the
91 meta-data of the node.
93 #+CAPTION: The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.
94 #+ATTR_HTML: width="300"
95 [[../images/finger-UV.png]]
97 #+CAPTION: Ventral side of the UV-mapped finger. Notice the density of touch sensors at the tip.
98 #+ATTR_HTML: width="300"
99 [[../images/finger-1.png]]
101 #+CAPTION: Side view of the UV-mapped finger.
102 #+ATTR_HTML: width="300"
103 [[../images/finger-2.png]]
105 #+CAPTION: Head on view of the finger. In both the head and side views you can see the divide where the touch-sensors transition from high density to low density.
106 #+ATTR_HTML: width="300"
107 [[../images/finger-3.png]]
109 The following code loads images and gets the locations of the white
110 pixels so that they can be used to create senses. =load-image= finds
111 images using jMonkeyEngine's asset-manager, so the image path is
112 expected to be relative to the =assets= directory. Thanks to Dylan
113 for the beautiful version of =filter-pixels=.
115 #+name: topology-1
116 #+begin_src clojure
117 (defn load-image
118 "Load an image as a BufferedImage using the asset-manager system."
119 [asset-relative-path]
120 (ImageToAwt/convert
121 (.getImage (.loadTexture (asset-manager) asset-relative-path))
122 false false 0))
124 (def white 0xFFFFFF)
126 (defn white? [rgb]
127 (= (bit-and white rgb) white))
129 (defn filter-pixels
130 "List the coordinates of all pixels matching pred, within the bounds
131 provided. If bounds are not specified then the entire image is
132 searched.
133 bounds -> [x0 y0 width height]"
134 {:author "Dylan Holmes"}
135 ([pred #^BufferedImage image]
136 (filter-pixels pred image [0 0 (.getWidth image) (.getHeight image)]))
137 ([pred #^BufferedImage image [x0 y0 width height]]
138 ((fn accumulate [x y matches]
139 (cond
140 (>= y (+ height y0)) matches
141 (>= x (+ width x0)) (recur 0 (inc y) matches)
142 (pred (.getRGB image x y))
143 (recur (inc x) y (conj matches [x y]))
144 :else (recur (inc x) y matches)))
145 x0 y0 [])))
147 (defn white-coordinates
148 "Coordinates of all the white pixels in a subset of the image."
149 ([#^BufferedImage image bounds]
150 (filter-pixels white? image bounds))
151 ([#^BufferedImage image]
152 (filter-pixels white? image)))
153 #+end_src
155 ** Topology
157 Information from the senses is transmitted to the brain via bundles of
158 axons, whether it be the optic nerve or the spinal cord. While these
159 bundles more or less preserve the overall topology of a sense's
160 two-dimensional surface, they do not preserve the precise euclidean
161 distances between every sensor. =collapse= is here to smoosh the
162 sensors described by a UV-map into a contiguous region that still
163 preserves the topology of the original sense.
165 #+name: topology-2
166 #+begin_src clojure
167 (in-ns 'cortex.sense)
169 (defn average [coll]
170 (/ (reduce + coll) (count coll)))
172 (defn- collapse-1d
173 "One dimensional helper for collapse."
174 [center line]
175 (let [length (count line)
176 num-above (count (filter (partial < center) line))
177 num-below (- length num-above)]
178 (range (- center num-below)
179 (+ center num-above))))
181 (defn collapse
182 "Take a sequence of pairs of integers and collapse them into a
183 contiguous bitmap with no \"holes\" or negative entries, as close to
184 the origin [0 0] as the shape permits. The order of the points is
185 preserved.
187 eg.
188 (collapse [[-5 5] [5 5] --> [[0 1] [1 1]
189 [-5 -5] [5 -5]]) --> [0 0] [1 0]]
191 (collapse [[-5 5] [-5 -5] --> [[0 1] [0 0]
192 [ 5 -5] [ 5 5]]) --> [1 0] [1 1]]"
193 [points]
194 (if (empty? points) []
195 (let
196 [num-points (count points)
197 center (vector
198 (int (average (map first points)))
199 (int (average (map first points))))
200 flattened
201 (reduce
202 concat
203 (map
204 (fn [column]
205 (map vector
206 (map first column)
207 (collapse-1d (second center)
208 (map second column))))
209 (partition-by first (sort-by first points))))
210 squeezed
211 (reduce
212 concat
213 (map
214 (fn [row]
215 (map vector
216 (collapse-1d (first center)
217 (map first row))
218 (map second row)))
219 (partition-by second (sort-by second flattened))))
220 relocated
221 (let [min-x (apply min (map first squeezed))
222 min-y (apply min (map second squeezed))]
223 (map (fn [[x y]]
224 [(- x min-x)
225 (- y min-y)])
226 squeezed))
227 point-correspondence
228 (zipmap (sort points) (sort relocated))
230 original-order
231 (vec (map point-correspondence points))]
232 original-order)))
233 #+end_src
234 * Viewing Sense Data
236 It's vital to /see/ the sense data to make sure that everything is
237 behaving as it should. =view-sense= and its helper, =view-image=
238 are here so that each sense can define its own way of turning
239 sense-data into pictures, while the actual rendering of said pictures
240 stays in one central place. =points->image= helps senses generate a
241 base image onto which they can overlay actual sense data.
243 #+name: view-senses
244 #+begin_src clojure
245 (in-ns 'cortex.sense)
247 (defn view-image
248 "Initializes a JPanel on which you may draw a BufferedImage.
249 Returns a function that accepts a BufferedImage and draws it to the
250 JPanel. If given a directory it will save the images as png files
251 starting at 0000000.png and incrementing from there."
252 ([#^File save]
253 (let [idx (atom -1)
254 image
255 (atom
256 (BufferedImage. 1 1 BufferedImage/TYPE_4BYTE_ABGR))
257 panel
258 (proxy [JPanel] []
259 (paint
260 [graphics]
261 (proxy-super paintComponent graphics)
262 (.drawImage graphics @image 0 0 nil)))
263 frame (JFrame. "Display Image")]
264 (SwingUtilities/invokeLater
265 (fn []
266 (doto frame
267 (-> (.getContentPane) (.add panel))
268 (.pack)
269 (.setLocationRelativeTo nil)
270 (.setResizable true)
271 (.setVisible true))))
272 (fn [#^BufferedImage i]
273 (reset! image i)
274 (.setSize frame (+ 8 (.getWidth i)) (+ 28 (.getHeight i)))
275 (.repaint panel 0 0 (.getWidth i) (.getHeight i))
276 (if save
277 (ImageIO/write
278 i "png"
279 (File. save (format "%07d.png" (swap! idx inc))))))))
280 ([] (view-image nil)))
282 (defn view-sense
283 "Take a kernel that produces a BufferedImage from some sense data
284 and return a function which takes a list of sense data, uses the
285 kernel to convert to images, and displays those images, each in
286 its own JFrame."
287 [sense-display-kernel]
288 (let [windows (atom [])]
289 (fn this
290 ([data]
291 (this data nil))
292 ([data save-to]
293 (if (> (count data) (count @windows))
294 (reset!
295 windows
296 (doall
297 (map
298 (fn [idx]
299 (if save-to
300 (let [dir (File. save-to (str idx))]
301 (.mkdir dir)
302 (view-image dir))
303 (view-image))) (range (count data))))))
304 (dorun
305 (map
306 (fn [display datum]
307 (display (sense-display-kernel datum)))
308 @windows data))))))
311 (defn points->image
312 "Take a collection of points and visualize it as a BufferedImage."
313 [points]
314 (if (empty? points)
315 (BufferedImage. 1 1 BufferedImage/TYPE_BYTE_BINARY)
316 (let [xs (vec (map first points))
317 ys (vec (map second points))
318 x0 (apply min xs)
319 y0 (apply min ys)
320 width (- (apply max xs) x0)
321 height (- (apply max ys) y0)
322 image (BufferedImage. (inc width) (inc height)
323 BufferedImage/TYPE_INT_RGB)]
324 (dorun
325 (for [x (range (.getWidth image))
326 y (range (.getHeight image))]
327 (.setRGB image x y 0xFF0000)))
328 (dorun
329 (for [index (range (count points))]
330 (.setRGB image (- (xs index) x0) (- (ys index) y0) -1)))
331 image)))
333 (defn gray
334 "Create a gray RGB pixel with R, G, and B set to num. num must be
335 between 0 and 255."
336 [num]
337 (+ num
338 (bit-shift-left num 8)
339 (bit-shift-left num 16)))
340 #+end_src
342 * Building a Sense from Nodes
343 My method for defining senses in blender is the following:
345 Senses like vision and hearing are localized to a single point
346 and follow a particular object around. For these:
348 - Create a single top-level empty node whose name is the name of the sense
349 - Add empty nodes which each contain meta-data relevant
350 to the sense, including a UV-map describing the number/distribution
351 of sensors if applicable.
352 - Make each empty-node the child of the top-level
353 node. =sense-nodes= below generates functions to find these children.
355 For touch, store the path to the UV-map which describes touch-sensors in the
356 meta-data of the object to which that map applies.
358 Each sense provides code that analyzes the Node structure of the
359 creature and creates sense-functions. They also modify the Node
360 structure if necessary.
362 Empty nodes created in blender have no appearance or physical presence
363 in jMonkeyEngine, but do appear in the scene graph. Empty nodes that
364 represent a sense which "follows" another geometry (like eyes and
365 ears) follow the closest physical object. =closest-node= finds this
366 closest object given the Creature and a particular empty node.
368 #+name: node-1
369 #+begin_src clojure
370 (defn sense-nodes
371 "For some senses there is a special empty blender node whose
372 children are considered markers for an instance of that sense. This
373 function generates functions to find those children, given the name
374 of the special parent node."
375 [parent-name]
376 (fn [#^Node creature]
377 (if-let [sense-node (.getChild creature parent-name)]
378 (seq (.getChildren sense-node))
379 (do ;;(println-repl "could not find" parent-name "node")
380 []))))
382 (defn closest-node
383 "Return the physical node in creature which is closest to the given
384 node."
385 [#^Node creature #^Node empty]
386 (loop [radius (float 0.01)]
387 (let [results (CollisionResults.)]
388 (.collideWith
389 creature
390 (BoundingBox. (.getWorldTranslation empty)
391 radius radius radius)
392 results)
393 (if-let [target (first results)]
394 (.getGeometry target)
395 (recur (float (* 2 radius)))))))
397 (defn world-to-local
398 "Convert the world coordinates into coordinates relative to the
399 object (i.e. local coordinates), taking into account the rotation
400 of object."
401 [#^Spatial object world-coordinate]
402 (.worldToLocal object world-coordinate nil))
404 (defn local-to-world
405 "Convert the local coordinates into world relative coordinates"
406 [#^Spatial object local-coordinate]
407 (.localToWorld object local-coordinate nil))
408 #+end_src
410 ** Sense Binding
412 =bind-sense= binds either a Camera or a Listener object to any
413 object so that they will follow that object no matter how it
414 moves. It is used to create both eyes and ears.
416 #+name: node-2
417 #+begin_src clojure
418 (defn bind-sense
419 "Bind the sense to the Spatial such that it will maintain its
420 current position relative to the Spatial no matter how the spatial
421 moves. 'sense can be either a Camera or Listener object."
422 [#^Spatial obj sense]
423 (let [sense-offset (.subtract (.getLocation sense)
424 (.getWorldTranslation obj))
425 initial-sense-rotation (Quaternion. (.getRotation sense))
426 base-anti-rotation (.inverse (.getWorldRotation obj))]
427 (.addControl
428 obj
429 (proxy [AbstractControl] []
430 (controlUpdate [tpf]
431 (let [total-rotation
432 (.mult base-anti-rotation (.getWorldRotation obj))]
433 (.setLocation
434 sense
435 (.add
436 (.mult total-rotation sense-offset)
437 (.getWorldTranslation obj)))
438 (.setRotation
439 sense
440 (.mult total-rotation initial-sense-rotation))))
441 (controlRender [_ _])))))
442 #+end_src
444 Here is some example code which shows how a camera bound to a blue box
445 with =bind-sense= moves as the box is buffeted by white cannonballs.
447 #+name: test
448 #+begin_src clojure
449 (defn test-bind-sense
450 "Show a camera that stays in the same relative position to a blue
451 cube."
452 ([] (test-bind-sense false))
453 ([record?]
454 (let [eye-pos (Vector3f. 0 30 0)
455 rock (box 1 1 1 :color ColorRGBA/Blue
456 :position (Vector3f. 0 10 0)
457 :mass 30)
458 table (box 3 1 10 :color ColorRGBA/Gray :mass 0
459 :position (Vector3f. 0 -3 0))]
460 (world
461 (nodify [rock table])
462 standard-debug-controls
463 (fn init [world]
464 (let [cam (doto (.clone (.getCamera world))
465 (.setLocation eye-pos)
466 (.lookAt Vector3f/ZERO
467 Vector3f/UNIT_X))]
468 (bind-sense rock cam)
469 (.setTimer world (RatchetTimer. 60))
470 (if record?
471 (Capture/captureVideo
472 world (File. "/home/r/proj/cortex/render/bind-sense0")))
473 (add-camera!
474 world cam
475 (comp (view-image
476 (if record?
477 (File. "/home/r/proj/cortex/render/bind-sense1")))
478 BufferedImage!))
479 (add-camera! world (.getCamera world) no-op)))
480 no-op))))
481 #+end_src
483 #+begin_html
484 <video controls="controls" width="755">
485 <source src="../video/bind-sense.ogg" type="video/ogg"
486 preload="none" poster="../images/aurellem-1280x480.png" />
487 </video>
488 <br> <a href="http://youtu.be/DvoN2wWQ_6o"> YouTube </a>
489 #+end_html
491 With this, eyes are easy --- you just bind the camera closer to the
492 desired object, and set it to look outward instead of inward as it
493 does in the video.
495 (nb : the video was created with the following commands)
497 *** Combine Frames with ImageMagick
498 #+begin_src clojure :results silent
499 (ns cortex.video.magick
500 (:import java.io.File)
501 (:use clojure.java.shell))
503 (defn combine-images []
504 (let
505 [idx (atom -1)
506 left (rest
507 (sort
508 (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/"))))
509 right (rest
510 (sort
511 (file-seq
512 (File. "/home/r/proj/cortex/render/bind-sense1/"))))
513 sub (rest
514 (sort
515 (file-seq
516 (File. "/home/r/proj/cortex/render/bind-senseB/"))))
517 sub* (concat sub (repeat 1000 (last sub)))]
518 (dorun
519 (map
520 (fn [im-1 im-2 sub]
521 (sh "convert" (.getCanonicalPath im-1)
522 (.getCanonicalPath im-2) "+append"
523 (.getCanonicalPath sub) "-append"
524 (.getCanonicalPath
525 (File. "/home/r/proj/cortex/render/bind-sense/"
526 (format "%07d.png" (swap! idx inc))))))
527 left right sub*))))
528 #+end_src
530 *** Encode Frames with ffmpeg
532 #+begin_src sh :results silent
533 cd /home/r/proj/cortex/render/
534 ffmpeg -r 30 -i bind-sense/%07d.png -b:v 9000k -vcodec libtheora bind-sense.ogg
535 #+end_src
537 * Headers
538 #+name: sense-header
539 #+begin_src clojure
540 (ns cortex.sense
541 "Here are functions useful in the construction of two or more
542 sensors/effectors."
543 {:author "Robert McIntyre"}
544 (:use (cortex world util))
545 (:import ij.process.ImageProcessor)
546 (:import jme3tools.converters.ImageToAwt)
547 (:import java.awt.image.BufferedImage)
548 (:import com.jme3.collision.CollisionResults)
549 (:import com.jme3.bounding.BoundingBox)
550 (:import (com.jme3.scene Node Spatial))
551 (:import com.jme3.scene.control.AbstractControl)
552 (:import (com.jme3.math Quaternion Vector3f))
553 (:import javax.imageio.ImageIO)
554 (:import java.io.File)
555 (:import (javax.swing JPanel JFrame SwingUtilities)))
556 #+end_src
558 #+name: test-header
559 #+begin_src clojure
560 (ns cortex.test.sense
561 (:use (cortex world util sense vision))
562 (:import
563 java.io.File
564 (com.jme3.math Vector3f ColorRGBA)
565 (com.aurellem.capture RatchetTimer Capture)))
566 #+end_src
568 * Source Listing
569 - [[../src/cortex/sense.clj][cortex.sense]]
570 - [[../src/cortex/test/sense.clj][cortex.test.sense]]
571 - [[../assets/Models/subtitles/subtitles.blend][subtitles.blend]]
572 - [[../assets/Models/subtitles/Lake_CraterLake03_sm.hdr][subtitles reflection map]]
573 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
574 - [[http://hg.bortreb.com ][source-repository]]
576 * Next
577 Now that some of the preliminaries are out of the way, in the [[./body.org][next
578 post]] I'll create a simulated body.
581 * COMMENT generate source
582 #+begin_src clojure :tangle ../src/cortex/sense.clj
583 <<sense-header>>
584 <<blender-1>>
585 <<blender-2>>
586 <<topology-1>>
587 <<topology-2>>
588 <<node-1>>
589 <<node-2>>
590 <<view-senses>>
591 #+end_src
593 #+begin_src clojure :tangle ../src/cortex/test/sense.clj
594 <<test-header>>
595 <<test>>
596 #+end_src
598 #+begin_src clojure :tangle ../src/cortex/video/magick.clj
599 <<magick>>
600 #+end_src