view org/sense.org @ 429:b5d0f0adf19f

improvements by Dylan!
author Robert McIntyre <rlm@mit.edu>
date Fri, 21 Mar 2014 20:56:56 -0400
parents 02cc0734a976
children 258078f78b33
line wrap: on
line source
1 #+title: Helper Functions / Motivations
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: sensory utilities
5 #+keywords: simulation, jMonkeyEngine3, clojure, simulated senses
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Blender Utilities
10 In blender, any object can be assigned an arbitrary number of key-value
11 pairs which are called "Custom Properties". These are accessible in
12 jMonkeyEngine when blender files are imported with the
13 =BlenderLoader=. =meta-data= extracts these properties.
15 #+name: blender-1
16 #+begin_src clojure
17 (in-ns 'cortex.sense)
18 (defn meta-data
19 "Get the meta-data for a node created with blender."
20 [blender-node key]
21 (if-let [data (.getUserData blender-node "properties")]
22 ;; this part is to accomodate weird blender properties
23 ;; as well as sensible clojure maps.
24 (.findValue data key)
25 (.getUserData blender-node key)))
27 #+end_src
29 #+results: blender-1
30 : #'cortex.sense/meta-data
32 Blender uses a different coordinate system than jMonkeyEngine so it
33 is useful to be able to convert between the two. These only come into
34 play when the meta-data of a node refers to a vector in the blender
35 coordinate system.
37 #+name: blender-2
38 #+begin_src clojure
39 (defn jme-to-blender
40 "Convert from JME coordinates to Blender coordinates"
41 [#^Vector3f in]
42 (Vector3f. (.getX in) (- (.getZ in)) (.getY in)))
44 (defn blender-to-jme
45 "Convert from Blender coordinates to JME coordinates"
46 [#^Vector3f in]
47 (Vector3f. (.getX in) (.getZ in) (- (.getY in))))
48 #+end_src
50 * Sense Topology
52 Human beings are three-dimensional objects, and the nerves that
53 transmit data from our various sense organs to our brain are
54 essentially one-dimensional. This leaves up to two dimensions in which
55 our sensory information may flow. For example, imagine your skin: it
56 is a two-dimensional surface around a three-dimensional object (your
57 body). It has discrete touch sensors embedded at various points, and
58 the density of these sensors corresponds to the sensitivity of that
59 region of skin. Each touch sensor connects to a nerve, all of which
60 eventually are bundled together as they travel up the spinal cord to
61 the brain. Intersect the spinal nerves with a guillotining plane and
62 you will see all of the sensory data of the skin revealed in a roughly
63 circular two-dimensional image which is the cross section of the
64 spinal cord. Points on this image that are close together in this
65 circle represent touch sensors that are /probably/ close together on
66 the skin, although there is of course some cutting and rearrangement
67 that has to be done to transfer the complicated surface of the skin
68 onto a two dimensional image.
70 Most human senses consist of many discrete sensors of various
71 properties distributed along a surface at various densities. For
72 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
73 disks, and Ruffini's endings, which detect pressure and vibration of
74 various intensities. For ears, it is the stereocilia distributed
75 along the basilar membrane inside the cochlea; each one is sensitive
76 to a slightly different frequency of sound. For eyes, it is rods
77 and cones distributed along the surface of the retina. In each case,
78 we can describe the sense with a surface and a distribution of sensors
79 along that surface.
81 ** UV-maps
83 Blender and jMonkeyEngine already have support for exactly this sort
84 of data structure because it is used to "skin" models for games. It is
85 called [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Textures/Mapping/UV][UV-mapping]]. The three-dimensional surface of a model is cut
86 and smooshed until it fits on a two-dimensional image. You paint
87 whatever you want on that image, and when the three-dimensional shape
88 is rendered in a game the smooshing and cutting us reversed and the
89 image appears on the three-dimensional object.
91 To make a sense, interpret the UV-image as describing the distribution
92 of that senses sensors. To get different types of sensors, you can
93 either use a different color for each type of sensor, or use multiple
94 UV-maps, each labeled with that sensor type. I generally use a white
95 pixel to mean the presence of a sensor and a black pixel to mean the
96 absence of a sensor, and use one UV-map for each sensor-type within a
97 given sense. The paths to the images are not stored as the actual
98 UV-map of the blender object but are instead referenced in the
99 meta-data of the node.
101 #+CAPTION: The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.
102 #+ATTR_HTML: width="300"
103 [[../images/finger-UV.png]]
105 #+CAPTION: Ventral side of the UV-mapped finger. Notice the density of touch sensors at the tip.
106 #+ATTR_HTML: width="300"
107 [[../images/finger-1.png]]
109 #+CAPTION: Side view of the UV-mapped finger.
110 #+ATTR_HTML: width="300"
111 [[../images/finger-2.png]]
113 #+CAPTION: Head on view of the finger. In both the head and side views you can see the divide where the touch-sensors transition from high density to low density.
114 #+ATTR_HTML: width="300"
115 [[../images/finger-3.png]]
117 The following code loads images and gets the locations of the white
118 pixels so that they can be used to create senses. =load-image= finds
119 images using jMonkeyEngine's asset-manager, so the image path is
120 expected to be relative to the =assets= directory. Thanks to Dylan
121 for the beautiful version of =filter-pixels=.
123 #+name: topology-1
124 #+begin_src clojure
125 (defn load-image
126 "Load an image as a BufferedImage using the asset-manager system."
127 [asset-relative-path]
128 (ImageToAwt/convert
129 (.getImage (.loadTexture (asset-manager) asset-relative-path))
130 false false 0))
132 (def white 0xFFFFFF)
134 (defn white? [rgb]
135 (= (bit-and white rgb) white))
137 (defn filter-pixels
138 "List the coordinates of all pixels matching pred, within the bounds
139 provided. If bounds are not specified then the entire image is
140 searched.
141 bounds -> [x0 y0 width height]"
142 {:author "Dylan Holmes"}
143 ([pred #^BufferedImage image]
144 (filter-pixels pred image [0 0 (.getWidth image) (.getHeight image)]))
145 ([pred #^BufferedImage image [x0 y0 width height]]
146 ((fn accumulate [x y matches]
147 (cond
148 (>= y (+ height y0)) matches
149 (>= x (+ width x0)) (recur 0 (inc y) matches)
150 (pred (.getRGB image x y))
151 (recur (inc x) y (conj matches [x y]))
152 :else (recur (inc x) y matches)))
153 x0 y0 [])))
155 (defn white-coordinates
156 "Coordinates of all the white pixels in a subset of the image."
157 ([#^BufferedImage image bounds]
158 (filter-pixels white? image bounds))
159 ([#^BufferedImage image]
160 (filter-pixels white? image)))
161 #+end_src
163 ** Topology
165 Information from the senses is transmitted to the brain via bundles of
166 axons, whether it be the optic nerve or the spinal cord. While these
167 bundles more or less preserve the overall topology of a sense's
168 two-dimensional surface, they do not preserve the precise euclidean
169 distances between every sensor. =collapse= is here to smoosh the
170 sensors described by a UV-map into a contiguous region that still
171 preserves the topology of the original sense.
173 #+name: topology-2
174 #+begin_src clojure
175 (in-ns 'cortex.sense)
177 (defn average [coll]
178 (/ (reduce + coll) (count coll)))
180 (defn- collapse-1d
181 "One dimensional helper for collapse."
182 [center line]
183 (let [length (count line)
184 num-above (count (filter (partial < center) line))
185 num-below (- length num-above)]
186 (range (- center num-below)
187 (+ center num-above))))
189 (defn collapse
190 "Take a sequence of pairs of integers and collapse them into a
191 contiguous bitmap with no \"holes\" or negative entries, as close to
192 the origin [0 0] as the shape permits. The order of the points is
193 preserved.
195 eg.
196 (collapse [[-5 5] [5 5] --> [[0 1] [1 1]
197 [-5 -5] [5 -5]]) --> [0 0] [1 0]]
199 (collapse [[-5 5] [-5 -5] --> [[0 1] [0 0]
200 [ 5 -5] [ 5 5]]) --> [1 0] [1 1]]"
201 [points]
202 (if (empty? points) []
203 (let
204 [num-points (count points)
205 center (vector
206 (int (average (map first points)))
207 (int (average (map first points))))
208 flattened
209 (reduce
210 concat
211 (map
212 (fn [column]
213 (map vector
214 (map first column)
215 (collapse-1d (second center)
216 (map second column))))
217 (partition-by first (sort-by first points))))
218 squeezed
219 (reduce
220 concat
221 (map
222 (fn [row]
223 (map vector
224 (collapse-1d (first center)
225 (map first row))
226 (map second row)))
227 (partition-by second (sort-by second flattened))))
228 relocated
229 (let [min-x (apply min (map first squeezed))
230 min-y (apply min (map second squeezed))]
231 (map (fn [[x y]]
232 [(- x min-x)
233 (- y min-y)])
234 squeezed))
235 point-correspondence
236 (zipmap (sort points) (sort relocated))
238 original-order
239 (vec (map point-correspondence points))]
240 original-order)))
241 #+end_src
242 * Viewing Sense Data
244 It's vital to /see/ the sense data to make sure that everything is
245 behaving as it should. =view-sense= and its helper, =view-image=
246 are here so that each sense can define its own way of turning
247 sense-data into pictures, while the actual rendering of said pictures
248 stays in one central place. =points->image= helps senses generate a
249 base image onto which they can overlay actual sense data.
251 #+name: view-senses
252 #+begin_src clojure
253 (in-ns 'cortex.sense)
255 (defn view-image
256 "Initializes a JPanel on which you may draw a BufferedImage.
257 Returns a function that accepts a BufferedImage and draws it to the
258 JPanel. If given a directory it will save the images as png files
259 starting at 0000000.png and incrementing from there."
260 ([#^File save title]
261 (let [idx (atom -1)
262 image
263 (atom
264 (BufferedImage. 1 1 BufferedImage/TYPE_4BYTE_ABGR))
265 panel
266 (proxy [JPanel] []
267 (paint
268 [graphics]
269 (proxy-super paintComponent graphics)
270 (.drawImage graphics @image 0 0 nil)))
271 frame (JFrame. title)]
272 (SwingUtilities/invokeLater
273 (fn []
274 (doto frame
275 (-> (.getContentPane) (.add panel))
276 (.pack)
277 (.setLocationRelativeTo nil)
278 (.setResizable true)
279 (.setVisible true))))
280 (fn [#^BufferedImage i]
281 (reset! image i)
282 (.setSize frame (+ 8 (.getWidth i)) (+ 28 (.getHeight i)))
283 (.repaint panel 0 0 (.getWidth i) (.getHeight i))
284 (if save
285 (ImageIO/write
286 i "png"
287 (File. save (format "%07d.png" (swap! idx inc))))))))
288 ([#^File save]
289 (view-image save "Display Image"))
290 ([] (view-image nil)))
292 (defn view-sense
293 "Take a kernel that produces a BufferedImage from some sense data
294 and return a function which takes a list of sense data, uses the
295 kernel to convert to images, and displays those images, each in
296 its own JFrame."
297 [sense-display-kernel]
298 (let [windows (atom [])]
299 (fn this
300 ([data]
301 (this data nil))
302 ([data save-to]
303 (if (> (count data) (count @windows))
304 (reset!
305 windows
306 (doall
307 (map
308 (fn [idx]
309 (if save-to
310 (let [dir (File. save-to (str idx))]
311 (.mkdir dir)
312 (view-image dir))
313 (view-image))) (range (count data))))))
314 (dorun
315 (map
316 (fn [display datum]
317 (display (sense-display-kernel datum)))
318 @windows data))))))
321 (defn points->image
322 "Take a collection of points and visualize it as a BufferedImage."
323 [points]
324 (if (empty? points)
325 (BufferedImage. 1 1 BufferedImage/TYPE_BYTE_BINARY)
326 (let [xs (vec (map first points))
327 ys (vec (map second points))
328 x0 (apply min xs)
329 y0 (apply min ys)
330 width (- (apply max xs) x0)
331 height (- (apply max ys) y0)
332 image (BufferedImage. (inc width) (inc height)
333 BufferedImage/TYPE_INT_RGB)]
334 (dorun
335 (for [x (range (.getWidth image))
336 y (range (.getHeight image))]
337 (.setRGB image x y 0xFF0000)))
338 (dorun
339 (for [index (range (count points))]
340 (.setRGB image (- (xs index) x0) (- (ys index) y0) -1)))
341 image)))
343 (defn gray
344 "Create a gray RGB pixel with R, G, and B set to num. num must be
345 between 0 and 255."
346 [num]
347 (+ num
348 (bit-shift-left num 8)
349 (bit-shift-left num 16)))
350 #+end_src
352 * Building a Sense from Nodes
353 My method for defining senses in blender is the following:
355 Senses like vision and hearing are localized to a single point
356 and follow a particular object around. For these:
358 - Create a single top-level empty node whose name is the name of the sense
359 - Add empty nodes which each contain meta-data relevant
360 to the sense, including a UV-map describing the number/distribution
361 of sensors if applicable.
362 - Make each empty-node the child of the top-level
363 node. =sense-nodes= below generates functions to find these children.
365 For touch, store the path to the UV-map which describes touch-sensors in the
366 meta-data of the object to which that map applies.
368 Each sense provides code that analyzes the Node structure of the
369 creature and creates sense-functions. They also modify the Node
370 structure if necessary.
372 Empty nodes created in blender have no appearance or physical presence
373 in jMonkeyEngine, but do appear in the scene graph. Empty nodes that
374 represent a sense which "follows" another geometry (like eyes and
375 ears) follow the closest physical object. =closest-node= finds this
376 closest object given the Creature and a particular empty node.
378 #+name: node-1
379 #+begin_src clojure
380 (defn sense-nodes
381 "For some senses there is a special empty blender node whose
382 children are considered markers for an instance of that sense. This
383 function generates functions to find those children, given the name
384 of the special parent node."
385 [parent-name]
386 (fn [#^Node creature]
387 (if-let [sense-node (.getChild creature parent-name)]
388 (seq (.getChildren sense-node))
389 (do ;;(println-repl "could not find" parent-name "node")
390 []))))
392 (defn closest-node
393 "Return the physical node in creature which is closest to the given
394 node."
395 [#^Node creature #^Node empty]
396 (loop [radius (float 0.01)]
397 (let [results (CollisionResults.)]
398 (.collideWith
399 creature
400 (BoundingBox. (.getWorldTranslation empty)
401 radius radius radius)
402 results)
403 (if-let [target (first results)]
404 (.getGeometry target)
405 (recur (float (* 2 radius)))))))
407 (defn world-to-local
408 "Convert the world coordinates into coordinates relative to the
409 object (i.e. local coordinates), taking into account the rotation
410 of object."
411 [#^Spatial object world-coordinate]
412 (.worldToLocal object world-coordinate nil))
414 (defn local-to-world
415 "Convert the local coordinates into world relative coordinates"
416 [#^Spatial object local-coordinate]
417 (.localToWorld object local-coordinate nil))
418 #+end_src
420 ** Sense Binding
422 =bind-sense= binds either a Camera or a Listener object to any
423 object so that they will follow that object no matter how it
424 moves. It is used to create both eyes and ears.
426 #+name: node-2
427 #+begin_src clojure
428 (defn bind-sense
429 "Bind the sense to the Spatial such that it will maintain its
430 current position relative to the Spatial no matter how the spatial
431 moves. 'sense can be either a Camera or Listener object."
432 [#^Spatial obj sense]
433 (let [sense-offset (.subtract (.getLocation sense)
434 (.getWorldTranslation obj))
435 initial-sense-rotation (Quaternion. (.getRotation sense))
436 base-anti-rotation (.inverse (.getWorldRotation obj))]
437 (.addControl
438 obj
439 (proxy [AbstractControl] []
440 (controlUpdate [tpf]
441 (let [total-rotation
442 (.mult base-anti-rotation (.getWorldRotation obj))]
443 (.setLocation
444 sense
445 (.add
446 (.mult total-rotation sense-offset)
447 (.getWorldTranslation obj)))
448 (.setRotation
449 sense
450 (.mult total-rotation initial-sense-rotation))))
451 (controlRender [_ _])))))
452 #+end_src
454 Here is some example code which shows how a camera bound to a blue box
455 with =bind-sense= moves as the box is buffeted by white cannonballs.
457 #+name: test
458 #+begin_src clojure
459 (in-ns 'cortex.test.sense)
461 (defn test-bind-sense
462 "Show a camera that stays in the same relative position to a blue
463 cube."
464 ([] (test-bind-sense false))
465 ([record?]
466 (let [eye-pos (Vector3f. 0 30 0)
467 rock (box 1 1 1 :color ColorRGBA/Blue
468 :position (Vector3f. 0 10 0)
469 :mass 30)
470 table (box 3 1 10 :color ColorRGBA/Gray :mass 0
471 :position (Vector3f. 0 -3 0))]
472 (world
473 (nodify [rock table])
474 standard-debug-controls
475 (fn init [world]
476 (let [cam (doto (.clone (.getCamera world))
477 (.setLocation eye-pos)
478 (.lookAt Vector3f/ZERO
479 Vector3f/UNIT_X))]
480 (bind-sense rock cam)
481 (.setTimer world (RatchetTimer. 60))
482 (if record?
483 (Capture/captureVideo
484 world
485 (File. "/home/r/proj/cortex/render/bind-sense0")))
486 (add-camera!
487 world cam
488 (comp
489 (view-image
490 (if record?
491 (File. "/home/r/proj/cortex/render/bind-sense1")))
492 BufferedImage!))
493 (add-camera! world (.getCamera world) no-op)))
494 no-op))))
495 #+end_src
497 #+begin_html
498 <video controls="controls" width="755">
499 <source src="../video/bind-sense.ogg" type="video/ogg"
500 preload="none" poster="../images/aurellem-1280x480.png" />
501 </video>
502 <br> <a href="http://youtu.be/DvoN2wWQ_6o"> YouTube </a>
503 #+end_html
505 With this, eyes are easy --- you just bind the camera closer to the
506 desired object, and set it to look outward instead of inward as it
507 does in the video.
509 (nb : the video was created with the following commands)
511 *** Combine Frames with ImageMagick
512 #+begin_src clojure :results silent
513 (ns cortex.video.magick
514 (:import java.io.File)
515 (:use clojure.java.shell))
517 (defn combine-images []
518 (let
519 [idx (atom -1)
520 left (rest
521 (sort
522 (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/"))))
523 right (rest
524 (sort
525 (file-seq
526 (File. "/home/r/proj/cortex/render/bind-sense1/"))))
527 sub (rest
528 (sort
529 (file-seq
530 (File. "/home/r/proj/cortex/render/bind-senseB/"))))
531 sub* (concat sub (repeat 1000 (last sub)))]
532 (dorun
533 (map
534 (fn [im-1 im-2 sub]
535 (sh "convert" (.getCanonicalPath im-1)
536 (.getCanonicalPath im-2) "+append"
537 (.getCanonicalPath sub) "-append"
538 (.getCanonicalPath
539 (File. "/home/r/proj/cortex/render/bind-sense/"
540 (format "%07d.png" (swap! idx inc))))))
541 left right sub*))))
542 #+end_src
544 *** Encode Frames with ffmpeg
546 #+begin_src sh :results silent
547 cd /home/r/proj/cortex/render/
548 ffmpeg -r 30 -i bind-sense/%07d.png -b:v 9000k -vcodec libtheora bind-sense.ogg
549 #+end_src
551 * Headers
552 #+name: sense-header
553 #+begin_src clojure
554 (ns cortex.sense
555 "Here are functions useful in the construction of two or more
556 sensors/effectors."
557 {:author "Robert McIntyre"}
558 (:use (cortex world util))
559 (:import ij.process.ImageProcessor)
560 (:import jme3tools.converters.ImageToAwt)
561 (:import java.awt.image.BufferedImage)
562 (:import com.jme3.collision.CollisionResults)
563 (:import com.jme3.bounding.BoundingBox)
564 (:import (com.jme3.scene Node Spatial))
565 (:import com.jme3.scene.control.AbstractControl)
566 (:import (com.jme3.math Quaternion Vector3f))
567 (:import javax.imageio.ImageIO)
568 (:import java.io.File)
569 (:import (javax.swing JPanel JFrame SwingUtilities)))
570 #+end_src
572 #+name: test-header
573 #+begin_src clojure
574 (ns cortex.test.sense
575 (:use (cortex world util sense vision))
576 (:import
577 java.io.File
578 (com.jme3.math Vector3f ColorRGBA)
579 (com.aurellem.capture RatchetTimer Capture)))
580 #+end_src
582 * Source Listing
583 - [[../src/cortex/sense.clj][cortex.sense]]
584 - [[../src/cortex/test/sense.clj][cortex.test.sense]]
585 - [[../assets/Models/subtitles/subtitles.blend][subtitles.blend]]
586 - [[../assets/Models/subtitles/Lake_CraterLake03_sm.hdr][subtitles reflection map]]
587 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
588 - [[http://hg.bortreb.com ][source-repository]]
590 * Next
591 Now that some of the preliminaries are out of the way, in the [[./body.org][next
592 post]] I'll create a simulated body.
595 * COMMENT generate source
596 #+begin_src clojure :tangle ../src/cortex/sense.clj
597 <<sense-header>>
598 <<blender-1>>
599 <<blender-2>>
600 <<topology-1>>
601 <<topology-2>>
602 <<node-1>>
603 <<node-2>>
604 <<view-senses>>
605 #+end_src
607 #+begin_src clojure :tangle ../src/cortex/test/sense.clj
608 <<test-header>>
609 <<test>>
610 #+end_src
612 #+begin_src clojure :tangle ../src/cortex/video/magick.clj
613 <<magick>>
614 #+end_src