Mercurial > cortex
view org/sense.org @ 258:f4b67005b702
reorganized proprioception
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Tue, 14 Feb 2012 00:39:16 -0700 |
parents | be78d7bd6920 |
children | c39b8b29a79e |
line wrap: on
line source
1 #+title: Helper Functions / Motivations2 #+author: Robert McIntyre3 #+email: rlm@mit.edu4 #+description: sensory utilities5 #+keywords: simulation, jMonkeyEngine3, clojure, simulated senses6 #+SETUPFILE: ../../aurellem/org/setup.org7 #+INCLUDE: ../../aurellem/org/level-0.org10 * Blender Utilities11 In blender, any object can be assigned an arbitray number of key-value12 pairs which are called "Custom Properties". These are accessable in13 jMonkyeEngine when blender files are imported with the14 =BlenderLoader=. =(meta-data)= extracts these properties.16 #+name: blender-117 #+begin_src clojure18 (defn meta-data19 "Get the meta-data for a node created with blender."20 [blender-node key]21 (if-let [data (.getUserData blender-node "properties")]22 (.findValue data key) nil))23 #+end_src25 Blender uses a different coordinate system than jMonkeyEngine so it26 is useful to be able to convert between the two. These only come into27 play when the meta-data of a node refers to a vector in the blender28 coordinate system.30 #+name: blender-231 #+begin_src clojure32 (defn jme-to-blender33 "Convert from JME coordinates to Blender coordinates"34 [#^Vector3f in]35 (Vector3f. (.getX in) (- (.getZ in)) (.getY in)))37 (defn blender-to-jme38 "Convert from Blender coordinates to JME coordinates"39 [#^Vector3f in]40 (Vector3f. (.getX in) (.getZ in) (- (.getY in))))41 #+end_src43 * Sense Topology45 Human beings are three-dimensional objects, and the nerves that46 transmit data from our various sense organs to our brain are47 essentially one-dimensional. This leaves up to two dimensions in which48 our sensory information may flow. For example, imagine your skin: it49 is a two-dimensional surface around a three-dimensional object (your50 body). It has discrete touch sensors embedded at various points, and51 the density of these sensors corresponds to the sensitivity of that52 region of skin. Each touch sensor connects to a nerve, all of which53 eventually are bundled together as they travel up the spinal cord to54 the brain. Intersect the spinal nerves with a guillotining plane and55 you will see all of the sensory data of the skin revealed in a roughly56 circular two-dimensional image which is the cross section of the57 spinal cord. Points on this image that are close together in this58 circle represent touch sensors that are /probably/ close together on59 the skin, although there is of course some cutting and rerangement60 that has to be done to transfer the complicated surface of the skin61 onto a two dimensional image.63 Most human senses consist of many discrete sensors of various64 properties distributed along a surface at various densities. For65 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's66 disks, and Ruffini's endings, which detect pressure and vibration of67 various intensities. For ears, it is the stereocilia distributed68 along the basilar membrane inside the cochlea; each one is sensitive69 to a slightly different frequency of sound. For eyes, it is rods70 and cones distributed along the surface of the retina. In each case,71 we can describe the sense with a surface and a distribution of sensors72 along that surface.74 ** UV-maps76 Blender and jMonkeyEngine already have support for exactly this sort77 of data structure because it is used to "skin" models for games. It is78 called [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Textures/Mapping/UV][UV-mapping]]. The three-dimensional surface of a model is cut79 and smooshed until it fits on a two-dimensional image. You paint80 whatever you want on that image, and when the three-dimensional shape81 is rendered in a game the smooshing and cutting us reversed and the82 image appears on the three-dimensional object.84 To make a sense, interpret the UV-image as describing the distribution85 of that senses sensors. To get different types of sensors, you can86 either use a different color for each type of sensor, or use multiple87 UV-maps, each labeled with that sensor type. I generally use a white88 pixel to mean the presense of a sensor and a black pixel to mean the89 absense of a sensor, and use one UV-map for each sensor-type within a90 given sense. The paths to the images are not stored as the actual91 UV-map of the blender object but are instead referenced in the92 meta-data of the node.94 #+CAPTION: The UV-map for an enlongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.95 #+ATTR_HTML: width="300"96 [[../images/finger-UV.png]]98 #+CAPTION: Ventral side of the UV-mapped finger. Notice the density of touch sensors at the tip.99 #+ATTR_HTML: width="300"100 [[../images/finger-1.png]]102 #+CAPTION: Side view of the UV-mapped finger.103 #+ATTR_HTML: width="300"104 [[../images/finger-2.png]]106 #+CAPTION: Head on view of the finger. In both the head and side views you can see the divide where the touch-sensors transition from high density to low density.107 #+ATTR_HTML: width="300"108 [[../images/finger-3.png]]110 The following code loads images and gets the locations of the white111 pixels so that they can be used to create senses. =(load-image)= finds112 images using jMonkeyEngine's asset-manager, so the image path is113 expected to be relative to the =assets= directory. Thanks to Dylan114 for the beautiful version of =(filter-pixels)=.116 #+name: topology-1117 #+begin_src clojure118 (defn load-image119 "Load an image as a BufferedImage using the asset-manager system."120 [asset-relative-path]121 (ImageToAwt/convert122 (.getImage (.loadTexture (asset-manager) asset-relative-path))123 false false 0))125 (def white 0xFFFFFF)127 (defn white? [rgb]128 (= (bit-and white rgb) white))130 (defn filter-pixels131 "List the coordinates of all pixels matching pred, within the bounds132 provided. If bounds are not specified then the entire image is133 searched.134 bounds -> [x0 y0 width height]"135 {:author "Dylan Holmes"}136 ([pred #^BufferedImage image]137 (filter-pixels pred image [0 0 (.getWidth image) (.getHeight image)]))138 ([pred #^BufferedImage image [x0 y0 width height]]139 ((fn accumulate [x y matches]140 (cond141 (>= y (+ height y0)) matches142 (>= x (+ width x0)) (recur 0 (inc y) matches)143 (pred (.getRGB image x y))144 (recur (inc x) y (conj matches [x y]))145 :else (recur (inc x) y matches)))146 x0 y0 [])))148 (defn white-coordinates149 "Coordinates of all the white pixels in a subset of the image."150 ([#^BufferedImage image bounds]151 (filter-pixels white? image bounds))152 ([#^BufferedImage image]153 (filter-pixels white? image)))154 #+end_src156 ** Topology158 Information from the senses is transmitted to the brain via bundles of159 axons, whether it be the optic nerve or the spinal cord. While these160 bundles more or less perserve the overall topology of a sense's161 two-dimensional surface, they do not perserve the percise euclidean162 distances between every sensor. =(collapse)= is here to smoosh the163 sensors described by a UV-map into a contigous region that still164 perserves the topology of the original sense.166 #+name: topology-2167 #+begin_src clojure168 (in-ns 'cortex.sense)170 (defn average [coll]171 (/ (reduce + coll) (count coll)))173 (defn- collapse-1d174 "One dimensional helper for collapse."175 [center line]176 (let [length (count line)177 num-above (count (filter (partial < center) line))178 num-below (- length num-above)]179 (range (- center num-below)180 (+ center num-above))))182 (defn collapse183 "Take a sequence of pairs of integers and collapse them into a184 contigous bitmap with no \"holes\" or negative entries, as close to185 the origin [0 0] as the shape permits. The order of the points is186 preserved.188 eg.189 (collapse [[-5 5] [5 5] --> [[0 1] [1 1]190 [-5 -5] [5 -5]]) --> [0 0] [1 0]]192 (collapse [[-5 5] [-5 -5] --> [[0 1] [0 0]193 [ 5 -5] [ 5 5]]) --> [1 0] [1 1]]"194 [points]195 (if (empty? points) []196 (let197 [num-points (count points)198 center (vector199 (int (average (map first points)))200 (int (average (map first points))))201 flattened202 (reduce203 concat204 (map205 (fn [column]206 (map vector207 (map first column)208 (collapse-1d (second center)209 (map second column))))210 (partition-by first (sort-by first points))))211 squeezed212 (reduce213 concat214 (map215 (fn [row]216 (map vector217 (collapse-1d (first center)218 (map first row))219 (map second row)))220 (partition-by second (sort-by second flattened))))221 relocated222 (let [min-x (apply min (map first squeezed))223 min-y (apply min (map second squeezed))]224 (map (fn [[x y]]225 [(- x min-x)226 (- y min-y)])227 squeezed))228 point-correspondance229 (zipmap (sort points) (sort relocated))231 original-order232 (vec (map point-correspondance points))]233 original-order)))234 #+end_src235 * Viewing Sense Data237 It's vital to /see/ the sense data to make sure that everything is238 behaving as it should. =(view-sense)= and its helper, =(view-image)=239 are here so that each sense can define its own way of turning240 sense-data into pictures, while the actual rendering of said pictures241 stays in one central place. =(points->image)= helps senses generate a242 base image onto which they can overlay actual sense data.244 #+name: view-senses245 #+begin_src clojure246 (in-ns 'cortex.sense)248 (defn view-image249 "Initailizes a JPanel on which you may draw a BufferedImage.250 Returns a function that accepts a BufferedImage and draws it to the251 JPanel. If given a directory it will save the images as png files252 starting at 0000000.png and incrementing from there."253 ([#^File save]254 (let [idx (atom -1)255 image256 (atom257 (BufferedImage. 1 1 BufferedImage/TYPE_4BYTE_ABGR))258 panel259 (proxy [JPanel] []260 (paint261 [graphics]262 (proxy-super paintComponent graphics)263 (.drawImage graphics @image 0 0 nil)))264 frame (JFrame. "Display Image")]265 (SwingUtilities/invokeLater266 (fn []267 (doto frame268 (-> (.getContentPane) (.add panel))269 (.pack)270 (.setLocationRelativeTo nil)271 (.setResizable true)272 (.setVisible true))))273 (fn [#^BufferedImage i]274 (reset! image i)275 (.setSize frame (+ 8 (.getWidth i)) (+ 28 (.getHeight i)))276 (.repaint panel 0 0 (.getWidth i) (.getHeight i))277 (if save278 (ImageIO/write279 i "png"280 (File. save (format "%07d.png" (swap! idx inc))))))))281 ([] (view-image nil)))283 (defn view-sense284 "Take a kernel that produces a BufferedImage from some sense data285 and return a function which takes a list of sense data, uses the286 kernel to convert to images, and displays those images, each in287 its own JFrame."288 [sense-display-kernel]289 (let [windows (atom [])]290 (fn this291 ([data]292 (this data nil))293 ([data save-to]294 (if (> (count data) (count @windows))295 (reset!296 windows297 (doall298 (map299 (fn [idx]300 (if save-to301 (let [dir (File. save-to (str idx))]302 (.mkdir dir)303 (view-image dir))304 (view-image))) (range (count data))))))305 (dorun306 (map307 (fn [display datum]308 (display (sense-display-kernel datum)))309 @windows data))))))312 (defn points->image313 "Take a collection of points and visuliaze it as a BufferedImage."314 [points]315 (if (empty? points)316 (BufferedImage. 1 1 BufferedImage/TYPE_BYTE_BINARY)317 (let [xs (vec (map first points))318 ys (vec (map second points))319 x0 (apply min xs)320 y0 (apply min ys)321 width (- (apply max xs) x0)322 height (- (apply max ys) y0)323 image (BufferedImage. (inc width) (inc height)324 BufferedImage/TYPE_INT_RGB)]325 (dorun326 (for [x (range (.getWidth image))327 y (range (.getHeight image))]328 (.setRGB image x y 0xFF0000)))329 (dorun330 (for [index (range (count points))]331 (.setRGB image (- (xs index) x0) (- (ys index) y0) -1)))332 image)))334 (defn gray335 "Create a gray RGB pixel with R, G, and B set to num. num must be336 between 0 and 255."337 [num]338 (+ num339 (bit-shift-left num 8)340 (bit-shift-left num 16)))341 #+end_src343 * Building a Sense from Nodes344 My method for defining senses in blender is the following:346 Senses like vision and hearing are localized to a single point347 and follow a particular object around. For these:349 - Create a single top-level empty node whose name is the name of the sense350 - Add empty nodes which each contain meta-data relevant351 to the sense, including a UV-map describing the number/distribution352 of sensors if applicipable.353 - Make each empty-node the child of the top-level354 node. =(sense-nodes)= below generates functions to find these children.356 For touch, store the path to the UV-map which describes touch-sensors in the357 meta-data of the object to which that map applies.359 Each sense provides code that analyzes the Node structure of the360 creature and creates sense-functions. They also modify the Node361 structure if necessary.363 Empty nodes created in blender have no appearance or physical presence364 in jMonkeyEngine, but do appear in the scene graph. Empty nodes that365 represent a sense which "follows" another geometry (like eyes and366 ears) follow the closest physical object. =(closest-node)= finds this367 closest object given the Creature and a particular empty node.369 #+name: node-1370 #+begin_src clojure371 (defn sense-nodes372 "For some senses there is a special empty blender node whose373 children are considered markers for an instance of that sense. This374 function generates functions to find those children, given the name375 of the special parent node."376 [parent-name]377 (fn [#^Node creature]378 (if-let [sense-node (.getChild creature parent-name)]379 (seq (.getChildren sense-node))380 (do (println-repl "could not find" parent-name "node") []))))382 (defn closest-node383 "Return the physical node in creature which is closest to the given384 node."385 [#^Node creature #^Node empty]386 (loop [radius (float 0.01)]387 (let [results (CollisionResults.)]388 (.collideWith389 creature390 (BoundingBox. (.getWorldTranslation empty)391 radius radius radius)392 results)393 (if-let [target (first results)]394 (.getGeometry target)395 (recur (float (* 2 radius)))))))397 (defn world-to-local398 "Convert the world coordinates into coordinates relative to the399 object (i.e. local coordinates), taking into account the rotation400 of object."401 [#^Spatial object world-coordinate]402 (.worldToLocal object world-coordinate nil))404 (defn local-to-world405 "Convert the local coordinates into world relative coordinates"406 [#^Spatial object local-coordinate]407 (.localToWorld object local-coordinate nil))408 #+end_src410 ** Sense Binding412 =(bind-sense)= binds either a Camera or a Listener object to any413 object so that they will follow that object no matter how it414 moves. It is used to create both eyes and ears.416 #+name: node-2417 #+begin_src clojure418 (defn bind-sense419 "Bind the sense to the Spatial such that it will maintain its420 current position relative to the Spatial no matter how the spatial421 moves. 'sense can be either a Camera or Listener object."422 [#^Spatial obj sense]423 (let [sense-offset (.subtract (.getLocation sense)424 (.getWorldTranslation obj))425 initial-sense-rotation (Quaternion. (.getRotation sense))426 base-anti-rotation (.inverse (.getWorldRotation obj))]427 (.addControl428 obj429 (proxy [AbstractControl] []430 (controlUpdate [tpf]431 (let [total-rotation432 (.mult base-anti-rotation (.getWorldRotation obj))]433 (.setLocation434 sense435 (.add436 (.mult total-rotation sense-offset)437 (.getWorldTranslation obj)))438 (.setRotation439 sense440 (.mult total-rotation initial-sense-rotation))))441 (controlRender [_ _])))))442 #+end_src444 Here is some example code which shows how a camera bound to a blue box445 with =(bind-sense)= moves as the box is buffeted by white cannonballs.447 #+name: test448 #+begin_src clojure449 (defn test-bind-sense450 "Show a camera that stays in the same relative position to a blue451 cube."452 []453 (let [eye-pos (Vector3f. 0 30 0)454 rock (box 1 1 1 :color ColorRGBA/Blue455 :position (Vector3f. 0 10 0)456 :mass 30)457 table (box 3 1 10 :color ColorRGBA/Gray :mass 0458 :position (Vector3f. 0 -3 0))]459 (world460 (nodify [rock table])461 standard-debug-controls462 (fn init [world]463 (let [cam (doto (.clone (.getCamera world))464 (.setLocation eye-pos)465 (.lookAt Vector3f/ZERO466 Vector3f/UNIT_X))]467 (bind-sense rock cam)468 (.setTimer world (RatchetTimer. 60))469 (Capture/captureVideo470 world (File. "/home/r/proj/cortex/render/bind-sense0"))471 (add-camera!472 world cam473 (comp (view-image474 (File. "/home/r/proj/cortex/render/bind-sense1"))475 BufferedImage!))476 (add-camera! world (.getCamera world) no-op)))477 no-op)))478 #+end_src480 #+begin_html481 <video controls="controls" width="755">482 <source src="../video/bind-sense.ogg" type="video/ogg"483 preload="none" poster="../images/aurellem-1280x480.png" />484 </video>485 #+end_html487 With this, eyes are easy --- you just bind the camera closer to the488 desired object, and set it to look outward instead of inward as it489 does in the video.491 (nb : the video was created with the following commands)493 *** Combine Frames with ImageMagick494 #+begin_src clojure :results silent495 (ns cortex.video.magick496 (:import java.io.File)497 (:use clojure.contrib.shell-out))499 (defn combine-images []500 (let501 [idx (atom -1)502 left (rest503 (sort504 (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/"))))505 right (rest506 (sort507 (file-seq508 (File. "/home/r/proj/cortex/render/bind-sense1/"))))509 sub (rest510 (sort511 (file-seq512 (File. "/home/r/proj/cortex/render/bind-senseB/"))))513 sub* (concat sub (repeat 1000 (last sub)))]514 (dorun515 (map516 (fn [im-1 im-2 sub]517 (sh "convert" (.getCanonicalPath im-1)518 (.getCanonicalPath im-2) "+append"519 (.getCanonicalPath sub) "-append"520 (.getCanonicalPath521 (File. "/home/r/proj/cortex/render/bind-sense/"522 (format "%07d.png" (swap! idx inc))))))523 left right sub*))))524 #+end_src526 *** Encode Frames with ffmpeg528 #+begin_src sh :results silent529 cd /home/r/proj/cortex/render/530 ffmpeg -r 30 -i bind-sense/%07d.png -b:v 9000k -vcodec libtheora bind-sense.ogg531 #+end_src533 * Headers534 #+name: sense-header535 #+begin_src clojure536 (ns cortex.sense537 "Here are functions useful in the construction of two or more538 sensors/effectors."539 {:author "Robert McInytre"}540 (:use (cortex world util))541 (:import ij.process.ImageProcessor)542 (:import jme3tools.converters.ImageToAwt)543 (:import java.awt.image.BufferedImage)544 (:import com.jme3.collision.CollisionResults)545 (:import com.jme3.bounding.BoundingBox)546 (:import (com.jme3.scene Node Spatial))547 (:import com.jme3.scene.control.AbstractControl)548 (:import (com.jme3.math Quaternion Vector3f))549 (:import javax.imageio.ImageIO)550 (:import java.io.File)551 (:import (javax.swing JPanel JFrame SwingUtilities)))552 #+end_src554 #+name: test-header555 #+begin_src clojure556 (ns cortex.test.sense557 (:use (cortex world util sense vision))558 (:import559 java.io.File560 (com.jme3.math Vector3f ColorRGBA)561 (com.aurellem.capture RatchetTimer Capture)))562 #+end_src564 * Source Listing565 - [[../src/cortex/sense.clj][cortex.sense]]566 - [[../src/cortex/test/sense.clj][cortex.test.sense]]567 - [[../assets/Models/subtitles/subtitles.blend][subtitles.blend]]568 - [[../assets/Models/subtitles/Lake_CraterLake03_sm.hdr][subtitles reflection map]]569 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>570 - [[http://hg.bortreb.com ][source-repository]]572 * Next573 Now that some of the preliminaries are out of the way, in the [[./body.org][next574 post]] I'll create a simulated body.577 * COMMENT generate source578 #+begin_src clojure :tangle ../src/cortex/sense.clj579 <<sense-header>>580 <<blender-1>>581 <<blender-2>>582 <<topology-1>>583 <<topology-2>>584 <<node-1>>585 <<node-2>>586 <<view-senses>>587 #+end_src589 #+begin_src clojure :tangle ../src/cortex/test/sense.clj590 <<test-header>>591 <<test>>592 #+end_src594 #+begin_src clojure :tangle ../src/cortex/video/magick.clj595 <<magick>>596 #+end_src