Mercurial > cortex
view org/sense.org @ 373:6205677359e9
minor formatting.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Tue, 12 Mar 2013 04:22:40 +0000 |
parents | d37ccb6c888f |
children | 02cc0734a976 |
line wrap: on
line source
1 #+title: Helper Functions / Motivations2 #+author: Robert McIntyre3 #+email: rlm@mit.edu4 #+description: sensory utilities5 #+keywords: simulation, jMonkeyEngine3, clojure, simulated senses6 #+SETUPFILE: ../../aurellem/org/setup.org7 #+INCLUDE: ../../aurellem/org/level-0.org9 * Blender Utilities10 In blender, any object can be assigned an arbitrary number of key-value11 pairs which are called "Custom Properties". These are accessible in12 jMonkeyEngine when blender files are imported with the13 =BlenderLoader=. =meta-data= extracts these properties.15 #+name: blender-116 #+begin_src clojure17 (in-ns 'cortex.sense)18 (defn meta-data19 "Get the meta-data for a node created with blender."20 [blender-node key]21 (if-let [data (.getUserData blender-node "properties")]22 ;; this part is to accomodate weird blender properties23 ;; as well as sensible clojure maps.24 (.findValue data key)25 (.getUserData blender-node key)))27 #+end_src29 #+results: blender-130 : #'cortex.sense/meta-data32 Blender uses a different coordinate system than jMonkeyEngine so it33 is useful to be able to convert between the two. These only come into34 play when the meta-data of a node refers to a vector in the blender35 coordinate system.37 #+name: blender-238 #+begin_src clojure39 (defn jme-to-blender40 "Convert from JME coordinates to Blender coordinates"41 [#^Vector3f in]42 (Vector3f. (.getX in) (- (.getZ in)) (.getY in)))44 (defn blender-to-jme45 "Convert from Blender coordinates to JME coordinates"46 [#^Vector3f in]47 (Vector3f. (.getX in) (.getZ in) (- (.getY in))))48 #+end_src50 * Sense Topology52 Human beings are three-dimensional objects, and the nerves that53 transmit data from our various sense organs to our brain are54 essentially one-dimensional. This leaves up to two dimensions in which55 our sensory information may flow. For example, imagine your skin: it56 is a two-dimensional surface around a three-dimensional object (your57 body). It has discrete touch sensors embedded at various points, and58 the density of these sensors corresponds to the sensitivity of that59 region of skin. Each touch sensor connects to a nerve, all of which60 eventually are bundled together as they travel up the spinal cord to61 the brain. Intersect the spinal nerves with a guillotining plane and62 you will see all of the sensory data of the skin revealed in a roughly63 circular two-dimensional image which is the cross section of the64 spinal cord. Points on this image that are close together in this65 circle represent touch sensors that are /probably/ close together on66 the skin, although there is of course some cutting and rearrangement67 that has to be done to transfer the complicated surface of the skin68 onto a two dimensional image.70 Most human senses consist of many discrete sensors of various71 properties distributed along a surface at various densities. For72 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's73 disks, and Ruffini's endings, which detect pressure and vibration of74 various intensities. For ears, it is the stereocilia distributed75 along the basilar membrane inside the cochlea; each one is sensitive76 to a slightly different frequency of sound. For eyes, it is rods77 and cones distributed along the surface of the retina. In each case,78 we can describe the sense with a surface and a distribution of sensors79 along that surface.81 ** UV-maps83 Blender and jMonkeyEngine already have support for exactly this sort84 of data structure because it is used to "skin" models for games. It is85 called [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Textures/Mapping/UV][UV-mapping]]. The three-dimensional surface of a model is cut86 and smooshed until it fits on a two-dimensional image. You paint87 whatever you want on that image, and when the three-dimensional shape88 is rendered in a game the smooshing and cutting us reversed and the89 image appears on the three-dimensional object.91 To make a sense, interpret the UV-image as describing the distribution92 of that senses sensors. To get different types of sensors, you can93 either use a different color for each type of sensor, or use multiple94 UV-maps, each labeled with that sensor type. I generally use a white95 pixel to mean the presence of a sensor and a black pixel to mean the96 absence of a sensor, and use one UV-map for each sensor-type within a97 given sense. The paths to the images are not stored as the actual98 UV-map of the blender object but are instead referenced in the99 meta-data of the node.101 #+CAPTION: The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.102 #+ATTR_HTML: width="300"103 [[../images/finger-UV.png]]105 #+CAPTION: Ventral side of the UV-mapped finger. Notice the density of touch sensors at the tip.106 #+ATTR_HTML: width="300"107 [[../images/finger-1.png]]109 #+CAPTION: Side view of the UV-mapped finger.110 #+ATTR_HTML: width="300"111 [[../images/finger-2.png]]113 #+CAPTION: Head on view of the finger. In both the head and side views you can see the divide where the touch-sensors transition from high density to low density.114 #+ATTR_HTML: width="300"115 [[../images/finger-3.png]]117 The following code loads images and gets the locations of the white118 pixels so that they can be used to create senses. =load-image= finds119 images using jMonkeyEngine's asset-manager, so the image path is120 expected to be relative to the =assets= directory. Thanks to Dylan121 for the beautiful version of =filter-pixels=.123 #+name: topology-1124 #+begin_src clojure125 (defn load-image126 "Load an image as a BufferedImage using the asset-manager system."127 [asset-relative-path]128 (ImageToAwt/convert129 (.getImage (.loadTexture (asset-manager) asset-relative-path))130 false false 0))132 (def white 0xFFFFFF)134 (defn white? [rgb]135 (= (bit-and white rgb) white))137 (defn filter-pixels138 "List the coordinates of all pixels matching pred, within the bounds139 provided. If bounds are not specified then the entire image is140 searched.141 bounds -> [x0 y0 width height]"142 {:author "Dylan Holmes"}143 ([pred #^BufferedImage image]144 (filter-pixels pred image [0 0 (.getWidth image) (.getHeight image)]))145 ([pred #^BufferedImage image [x0 y0 width height]]146 ((fn accumulate [x y matches]147 (cond148 (>= y (+ height y0)) matches149 (>= x (+ width x0)) (recur 0 (inc y) matches)150 (pred (.getRGB image x y))151 (recur (inc x) y (conj matches [x y]))152 :else (recur (inc x) y matches)))153 x0 y0 [])))155 (defn white-coordinates156 "Coordinates of all the white pixels in a subset of the image."157 ([#^BufferedImage image bounds]158 (filter-pixels white? image bounds))159 ([#^BufferedImage image]160 (filter-pixels white? image)))161 #+end_src163 ** Topology165 Information from the senses is transmitted to the brain via bundles of166 axons, whether it be the optic nerve or the spinal cord. While these167 bundles more or less preserve the overall topology of a sense's168 two-dimensional surface, they do not preserve the precise euclidean169 distances between every sensor. =collapse= is here to smoosh the170 sensors described by a UV-map into a contiguous region that still171 preserves the topology of the original sense.173 #+name: topology-2174 #+begin_src clojure175 (in-ns 'cortex.sense)177 (defn average [coll]178 (/ (reduce + coll) (count coll)))180 (defn- collapse-1d181 "One dimensional helper for collapse."182 [center line]183 (let [length (count line)184 num-above (count (filter (partial < center) line))185 num-below (- length num-above)]186 (range (- center num-below)187 (+ center num-above))))189 (defn collapse190 "Take a sequence of pairs of integers and collapse them into a191 contiguous bitmap with no \"holes\" or negative entries, as close to192 the origin [0 0] as the shape permits. The order of the points is193 preserved.195 eg.196 (collapse [[-5 5] [5 5] --> [[0 1] [1 1]197 [-5 -5] [5 -5]]) --> [0 0] [1 0]]199 (collapse [[-5 5] [-5 -5] --> [[0 1] [0 0]200 [ 5 -5] [ 5 5]]) --> [1 0] [1 1]]"201 [points]202 (if (empty? points) []203 (let204 [num-points (count points)205 center (vector206 (int (average (map first points)))207 (int (average (map first points))))208 flattened209 (reduce210 concat211 (map212 (fn [column]213 (map vector214 (map first column)215 (collapse-1d (second center)216 (map second column))))217 (partition-by first (sort-by first points))))218 squeezed219 (reduce220 concat221 (map222 (fn [row]223 (map vector224 (collapse-1d (first center)225 (map first row))226 (map second row)))227 (partition-by second (sort-by second flattened))))228 relocated229 (let [min-x (apply min (map first squeezed))230 min-y (apply min (map second squeezed))]231 (map (fn [[x y]]232 [(- x min-x)233 (- y min-y)])234 squeezed))235 point-correspondence236 (zipmap (sort points) (sort relocated))238 original-order239 (vec (map point-correspondence points))]240 original-order)))241 #+end_src242 * Viewing Sense Data244 It's vital to /see/ the sense data to make sure that everything is245 behaving as it should. =view-sense= and its helper, =view-image=246 are here so that each sense can define its own way of turning247 sense-data into pictures, while the actual rendering of said pictures248 stays in one central place. =points->image= helps senses generate a249 base image onto which they can overlay actual sense data.251 #+name: view-senses252 #+begin_src clojure253 (in-ns 'cortex.sense)255 (defn view-image256 "Initializes a JPanel on which you may draw a BufferedImage.257 Returns a function that accepts a BufferedImage and draws it to the258 JPanel. If given a directory it will save the images as png files259 starting at 0000000.png and incrementing from there."260 ([#^File save]261 (let [idx (atom -1)262 image263 (atom264 (BufferedImage. 1 1 BufferedImage/TYPE_4BYTE_ABGR))265 panel266 (proxy [JPanel] []267 (paint268 [graphics]269 (proxy-super paintComponent graphics)270 (.drawImage graphics @image 0 0 nil)))271 frame (JFrame. "Display Image")]272 (SwingUtilities/invokeLater273 (fn []274 (doto frame275 (-> (.getContentPane) (.add panel))276 (.pack)277 (.setLocationRelativeTo nil)278 (.setResizable true)279 (.setVisible true))))280 (fn [#^BufferedImage i]281 (reset! image i)282 (.setSize frame (+ 8 (.getWidth i)) (+ 28 (.getHeight i)))283 (.repaint panel 0 0 (.getWidth i) (.getHeight i))284 (if save285 (ImageIO/write286 i "png"287 (File. save (format "%07d.png" (swap! idx inc))))))))288 ([] (view-image nil)))290 (defn view-sense291 "Take a kernel that produces a BufferedImage from some sense data292 and return a function which takes a list of sense data, uses the293 kernel to convert to images, and displays those images, each in294 its own JFrame."295 [sense-display-kernel]296 (let [windows (atom [])]297 (fn this298 ([data]299 (this data nil))300 ([data save-to]301 (if (> (count data) (count @windows))302 (reset!303 windows304 (doall305 (map306 (fn [idx]307 (if save-to308 (let [dir (File. save-to (str idx))]309 (.mkdir dir)310 (view-image dir))311 (view-image))) (range (count data))))))312 (dorun313 (map314 (fn [display datum]315 (display (sense-display-kernel datum)))316 @windows data))))))319 (defn points->image320 "Take a collection of points and visualize it as a BufferedImage."321 [points]322 (if (empty? points)323 (BufferedImage. 1 1 BufferedImage/TYPE_BYTE_BINARY)324 (let [xs (vec (map first points))325 ys (vec (map second points))326 x0 (apply min xs)327 y0 (apply min ys)328 width (- (apply max xs) x0)329 height (- (apply max ys) y0)330 image (BufferedImage. (inc width) (inc height)331 BufferedImage/TYPE_INT_RGB)]332 (dorun333 (for [x (range (.getWidth image))334 y (range (.getHeight image))]335 (.setRGB image x y 0xFF0000)))336 (dorun337 (for [index (range (count points))]338 (.setRGB image (- (xs index) x0) (- (ys index) y0) -1)))339 image)))341 (defn gray342 "Create a gray RGB pixel with R, G, and B set to num. num must be343 between 0 and 255."344 [num]345 (+ num346 (bit-shift-left num 8)347 (bit-shift-left num 16)))348 #+end_src350 * Building a Sense from Nodes351 My method for defining senses in blender is the following:353 Senses like vision and hearing are localized to a single point354 and follow a particular object around. For these:356 - Create a single top-level empty node whose name is the name of the sense357 - Add empty nodes which each contain meta-data relevant358 to the sense, including a UV-map describing the number/distribution359 of sensors if applicable.360 - Make each empty-node the child of the top-level361 node. =sense-nodes= below generates functions to find these children.363 For touch, store the path to the UV-map which describes touch-sensors in the364 meta-data of the object to which that map applies.366 Each sense provides code that analyzes the Node structure of the367 creature and creates sense-functions. They also modify the Node368 structure if necessary.370 Empty nodes created in blender have no appearance or physical presence371 in jMonkeyEngine, but do appear in the scene graph. Empty nodes that372 represent a sense which "follows" another geometry (like eyes and373 ears) follow the closest physical object. =closest-node= finds this374 closest object given the Creature and a particular empty node.376 #+name: node-1377 #+begin_src clojure378 (defn sense-nodes379 "For some senses there is a special empty blender node whose380 children are considered markers for an instance of that sense. This381 function generates functions to find those children, given the name382 of the special parent node."383 [parent-name]384 (fn [#^Node creature]385 (if-let [sense-node (.getChild creature parent-name)]386 (seq (.getChildren sense-node))387 (do ;;(println-repl "could not find" parent-name "node")388 []))))390 (defn closest-node391 "Return the physical node in creature which is closest to the given392 node."393 [#^Node creature #^Node empty]394 (loop [radius (float 0.01)]395 (let [results (CollisionResults.)]396 (.collideWith397 creature398 (BoundingBox. (.getWorldTranslation empty)399 radius radius radius)400 results)401 (if-let [target (first results)]402 (.getGeometry target)403 (recur (float (* 2 radius)))))))405 (defn world-to-local406 "Convert the world coordinates into coordinates relative to the407 object (i.e. local coordinates), taking into account the rotation408 of object."409 [#^Spatial object world-coordinate]410 (.worldToLocal object world-coordinate nil))412 (defn local-to-world413 "Convert the local coordinates into world relative coordinates"414 [#^Spatial object local-coordinate]415 (.localToWorld object local-coordinate nil))416 #+end_src418 ** Sense Binding420 =bind-sense= binds either a Camera or a Listener object to any421 object so that they will follow that object no matter how it422 moves. It is used to create both eyes and ears.424 #+name: node-2425 #+begin_src clojure426 (defn bind-sense427 "Bind the sense to the Spatial such that it will maintain its428 current position relative to the Spatial no matter how the spatial429 moves. 'sense can be either a Camera or Listener object."430 [#^Spatial obj sense]431 (let [sense-offset (.subtract (.getLocation sense)432 (.getWorldTranslation obj))433 initial-sense-rotation (Quaternion. (.getRotation sense))434 base-anti-rotation (.inverse (.getWorldRotation obj))]435 (.addControl436 obj437 (proxy [AbstractControl] []438 (controlUpdate [tpf]439 (let [total-rotation440 (.mult base-anti-rotation (.getWorldRotation obj))]441 (.setLocation442 sense443 (.add444 (.mult total-rotation sense-offset)445 (.getWorldTranslation obj)))446 (.setRotation447 sense448 (.mult total-rotation initial-sense-rotation))))449 (controlRender [_ _])))))450 #+end_src452 Here is some example code which shows how a camera bound to a blue box453 with =bind-sense= moves as the box is buffeted by white cannonballs.455 #+name: test456 #+begin_src clojure457 (in-ns 'cortex.test.sense)459 (defn test-bind-sense460 "Show a camera that stays in the same relative position to a blue461 cube."462 ([] (test-bind-sense false))463 ([record?]464 (let [eye-pos (Vector3f. 0 30 0)465 rock (box 1 1 1 :color ColorRGBA/Blue466 :position (Vector3f. 0 10 0)467 :mass 30)468 table (box 3 1 10 :color ColorRGBA/Gray :mass 0469 :position (Vector3f. 0 -3 0))]470 (world471 (nodify [rock table])472 standard-debug-controls473 (fn init [world]474 (let [cam (doto (.clone (.getCamera world))475 (.setLocation eye-pos)476 (.lookAt Vector3f/ZERO477 Vector3f/UNIT_X))]478 (bind-sense rock cam)479 (.setTimer world (RatchetTimer. 60))480 (if record?481 (Capture/captureVideo482 world483 (File. "/home/r/proj/cortex/render/bind-sense0")))484 (add-camera!485 world cam486 (comp487 (view-image488 (if record?489 (File. "/home/r/proj/cortex/render/bind-sense1")))490 BufferedImage!))491 (add-camera! world (.getCamera world) no-op)))492 no-op))))493 #+end_src495 #+begin_html496 <video controls="controls" width="755">497 <source src="../video/bind-sense.ogg" type="video/ogg"498 preload="none" poster="../images/aurellem-1280x480.png" />499 </video>500 <br> <a href="http://youtu.be/DvoN2wWQ_6o"> YouTube </a>501 #+end_html503 With this, eyes are easy --- you just bind the camera closer to the504 desired object, and set it to look outward instead of inward as it505 does in the video.507 (nb : the video was created with the following commands)509 *** Combine Frames with ImageMagick510 #+begin_src clojure :results silent511 (ns cortex.video.magick512 (:import java.io.File)513 (:use clojure.java.shell))515 (defn combine-images []516 (let517 [idx (atom -1)518 left (rest519 (sort520 (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/"))))521 right (rest522 (sort523 (file-seq524 (File. "/home/r/proj/cortex/render/bind-sense1/"))))525 sub (rest526 (sort527 (file-seq528 (File. "/home/r/proj/cortex/render/bind-senseB/"))))529 sub* (concat sub (repeat 1000 (last sub)))]530 (dorun531 (map532 (fn [im-1 im-2 sub]533 (sh "convert" (.getCanonicalPath im-1)534 (.getCanonicalPath im-2) "+append"535 (.getCanonicalPath sub) "-append"536 (.getCanonicalPath537 (File. "/home/r/proj/cortex/render/bind-sense/"538 (format "%07d.png" (swap! idx inc))))))539 left right sub*))))540 #+end_src542 *** Encode Frames with ffmpeg544 #+begin_src sh :results silent545 cd /home/r/proj/cortex/render/546 ffmpeg -r 30 -i bind-sense/%07d.png -b:v 9000k -vcodec libtheora bind-sense.ogg547 #+end_src549 * Headers550 #+name: sense-header551 #+begin_src clojure552 (ns cortex.sense553 "Here are functions useful in the construction of two or more554 sensors/effectors."555 {:author "Robert McIntyre"}556 (:use (cortex world util))557 (:import ij.process.ImageProcessor)558 (:import jme3tools.converters.ImageToAwt)559 (:import java.awt.image.BufferedImage)560 (:import com.jme3.collision.CollisionResults)561 (:import com.jme3.bounding.BoundingBox)562 (:import (com.jme3.scene Node Spatial))563 (:import com.jme3.scene.control.AbstractControl)564 (:import (com.jme3.math Quaternion Vector3f))565 (:import javax.imageio.ImageIO)566 (:import java.io.File)567 (:import (javax.swing JPanel JFrame SwingUtilities)))568 #+end_src570 #+name: test-header571 #+begin_src clojure572 (ns cortex.test.sense573 (:use (cortex world util sense vision))574 (:import575 java.io.File576 (com.jme3.math Vector3f ColorRGBA)577 (com.aurellem.capture RatchetTimer Capture)))578 #+end_src580 * Source Listing581 - [[../src/cortex/sense.clj][cortex.sense]]582 - [[../src/cortex/test/sense.clj][cortex.test.sense]]583 - [[../assets/Models/subtitles/subtitles.blend][subtitles.blend]]584 - [[../assets/Models/subtitles/Lake_CraterLake03_sm.hdr][subtitles reflection map]]585 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>586 - [[http://hg.bortreb.com ][source-repository]]588 * Next589 Now that some of the preliminaries are out of the way, in the [[./body.org][next590 post]] I'll create a simulated body.593 * COMMENT generate source594 #+begin_src clojure :tangle ../src/cortex/sense.clj595 <<sense-header>>596 <<blender-1>>597 <<blender-2>>598 <<topology-1>>599 <<topology-2>>600 <<node-1>>601 <<node-2>>602 <<view-senses>>603 #+end_src605 #+begin_src clojure :tangle ../src/cortex/test/sense.clj606 <<test-header>>607 <<test>>608 #+end_src610 #+begin_src clojure :tangle ../src/cortex/video/magick.clj611 <<magick>>612 #+end_src