Mercurial > cortex

annotate org/sense.org @ 215:f283c62bd212

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixed long standing problem with orientation of eyes in blender, fleshed out text in vision.org
author Robert McIntyre <rlm@mit.edu>
date Fri, 10 Feb 2012 02:19:24 -0700
parents 97b8caf66824
children 7bf3e3d8fb26
rev   line source
rlm@202 1 #+title: Helper Functions / Motivations
rlm@151 2 #+author: Robert McIntyre
rlm@151 3 #+email: rlm@mit.edu
rlm@151 4 #+description: sensory utilities
rlm@151 5 #+keywords: simulation, jMonkeyEngine3, clojure, simulated senses
rlm@151 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@151 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@151 8
rlm@151 9
rlm@197 10 * Blender Utilities
rlm@198 11 In blender, any object can be assigned an arbitray number of key-value
rlm@198 12 pairs which are called "Custom Properties". These are accessable in
rlm@198 13 jMonkyeEngine when blender files are imported with the
rlm@198 14 =BlenderLoader=. =(meta-data)= extracts these properties.
rlm@198 15
rlm@198 16 #+name: blender-1
rlm@197 17 #+begin_src clojure
rlm@181 18 (defn meta-data
rlm@181 19 "Get the meta-data for a node created with blender."
rlm@181 20 [blender-node key]
rlm@151 21 (if-let [data (.getUserData blender-node "properties")]
rlm@198 22 (.findValue data key) nil))
rlm@198 23 #+end_src
rlm@151 24
rlm@198 25 Blender uses a different coordinate system than jMonkeyEngine so it
rlm@198 26 is useful to be able to convert between the two. These only come into
rlm@198 27 play when the meta-data of a node refers to a vector in the blender
rlm@198 28 coordinate system.
rlm@198 29
rlm@198 30 #+name: blender-2
rlm@198 31 #+begin_src clojure
rlm@197 32 (defn jme-to-blender
rlm@197 33 "Convert from JME coordinates to Blender coordinates"
rlm@197 34 [#^Vector3f in]
rlm@198 35 (Vector3f. (.getX in) (- (.getZ in)) (.getY in)))
rlm@151 36
rlm@197 37 (defn blender-to-jme
rlm@197 38 "Convert from Blender coordinates to JME coordinates"
rlm@197 39 [#^Vector3f in]
rlm@198 40 (Vector3f. (.getX in) (.getZ in) (- (.getY in))))
rlm@197 41 #+end_src
rlm@197 42
rlm@198 43 * Sense Topology
rlm@198 44
rlm@198 45 Human beings are three-dimensional objects, and the nerves that
rlm@198 46 transmit data from our various sense organs to our brain are
rlm@198 47 essentially one-dimensional. This leaves up to two dimensions in which
rlm@198 48 our sensory information may flow. For example, imagine your skin: it
rlm@198 49 is a two-dimensional surface around a three-dimensional object (your
rlm@198 50 body). It has discrete touch sensors embedded at various points, and
rlm@198 51 the density of these sensors corresponds to the sensitivity of that
rlm@198 52 region of skin. Each touch sensor connects to a nerve, all of which
rlm@198 53 eventually are bundled together as they travel up the spinal cord to
rlm@198 54 the brain. Intersect the spinal nerves with a guillotining plane and
rlm@198 55 you will see all of the sensory data of the skin revealed in a roughly
rlm@198 56 circular two-dimensional image which is the cross section of the
rlm@198 57 spinal cord. Points on this image that are close together in this
rlm@198 58 circle represent touch sensors that are /probably/ close together on
rlm@198 59 the skin, although there is of course some cutting and rerangement
rlm@198 60 that has to be done to transfer the complicated surface of the skin
rlm@198 61 onto a two dimensional image.
rlm@198 62
rlm@198 63 Most human senses consist of many discrete sensors of various
rlm@198 64 properties distributed along a surface at various densities. For
rlm@198 65 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
rlm@198 66 disks, and Ruffini's endings, which detect pressure and vibration of
rlm@198 67 various intensities. For ears, it is the stereocilia distributed
rlm@198 68 along the basilar membrane inside the cochlea; each one is sensitive
rlm@198 69 to a slightly different frequency of sound. For eyes, it is rods
rlm@198 70 and cones distributed along the surface of the retina. In each case,
rlm@198 71 we can describe the sense with a surface and a distribution of sensors
rlm@198 72 along that surface.
rlm@198 73
rlm@198 74 ** UV-maps
rlm@198 75
rlm@198 76 Blender and jMonkeyEngine already have support for exactly this sort
rlm@198 77 of data structure because it is used to "skin" models for games. It is
rlm@201 78 called [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Textures/Mapping/UV][UV-mapping]]. The three-dimensional surface of a model is cut
rlm@201 79 and smooshed until it fits on a two-dimensional image. You paint
rlm@201 80 whatever you want on that image, and when the three-dimensional shape
rlm@201 81 is rendered in a game the smooshing and cutting us reversed and the
rlm@201 82 image appears on the three-dimensional object.
rlm@198 83
rlm@198 84 To make a sense, interpret the UV-image as describing the distribution
rlm@198 85 of that senses sensors. To get different types of sensors, you can
rlm@198 86 either use a different color for each type of sensor, or use multiple
rlm@198 87 UV-maps, each labeled with that sensor type. I generally use a white
rlm@198 88 pixel to mean the presense of a sensor and a black pixel to mean the
rlm@198 89 absense of a sensor, and use one UV-map for each sensor-type within a
rlm@198 90 given sense. The paths to the images are not stored as the actual
rlm@198 91 UV-map of the blender object but are instead referenced in the
rlm@198 92 meta-data of the node.
rlm@198 93
rlm@198 94 #+CAPTION: The UV-map for an enlongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.
rlm@198 95 #+ATTR_HTML: width="300"
rlm@198 96 [[../images/finger-UV.png]]
rlm@198 97
rlm@198 98 #+CAPTION: Ventral side of the UV-mapped finger. Notice the density of touch sensors at the tip.
rlm@198 99 #+ATTR_HTML: width="300"
rlm@198 100 [[../images/finger-1.png]]
rlm@198 101
rlm@198 102 #+CAPTION: Side view of the UV-mapped finger.
rlm@198 103 #+ATTR_HTML: width="300"
rlm@198 104 [[../images/finger-2.png]]
rlm@198 105
rlm@198 106 #+CAPTION: Head on view of the finger. In both the head and side views you can see the divide where the touch-sensors transition from high density to low density.
rlm@198 107 #+ATTR_HTML: width="300"
rlm@198 108 [[../images/finger-3.png]]
rlm@198 109
rlm@198 110 The following code loads images and gets the locations of the white
rlm@198 111 pixels so that they can be used to create senses. =(load-image)= finds
rlm@198 112 images using jMonkeyEngine's asset-manager, so the image path is
rlm@198 113 expected to be relative to the =assets= directory. Thanks to Dylan
rlm@201 114 for the beautiful version of =(filter-pixels)=.
rlm@198 115
rlm@198 116 #+name: topology-1
rlm@197 117 #+begin_src clojure
rlm@197 118 (defn load-image
rlm@197 119 "Load an image as a BufferedImage using the asset-manager system."
rlm@197 120 [asset-relative-path]
rlm@197 121 (ImageToAwt/convert
rlm@197 122 (.getImage (.loadTexture (asset-manager) asset-relative-path))
rlm@197 123 false false 0))
rlm@151 124
rlm@181 125 (def white 0xFFFFFF)
rlm@181 126
rlm@181 127 (defn white? [rgb]
rlm@181 128 (= (bit-and white rgb) white))
rlm@181 129
rlm@151 130 (defn filter-pixels
rlm@151 131 "List the coordinates of all pixels matching pred, within the bounds
rlm@198 132 provided. If bounds are not specified then the entire image is
rlm@198 133 searched.
rlm@182 134 bounds -> [x0 y0 width height]"
rlm@151 135 {:author "Dylan Holmes"}
rlm@151 136 ([pred #^BufferedImage image]
rlm@151 137 (filter-pixels pred image [0 0 (.getWidth image) (.getHeight image)]))
rlm@151 138 ([pred #^BufferedImage image [x0 y0 width height]]
rlm@151 139 ((fn accumulate [x y matches]
rlm@151 140 (cond
rlm@151 141 (>= y (+ height y0)) matches
rlm@151 142 (>= x (+ width x0)) (recur 0 (inc y) matches)
rlm@151 143 (pred (.getRGB image x y))
rlm@151 144 (recur (inc x) y (conj matches [x y]))
rlm@151 145 :else (recur (inc x) y matches)))
rlm@151 146 x0 y0 [])))
rlm@151 147
rlm@151 148 (defn white-coordinates
rlm@151 149 "Coordinates of all the white pixels in a subset of the image."
rlm@151 150 ([#^BufferedImage image bounds]
rlm@181 151 (filter-pixels white? image bounds))
rlm@151 152 ([#^BufferedImage image]
rlm@181 153 (filter-pixels white? image)))
rlm@198 154 #+end_src
rlm@151 155
rlm@198 156 ** Topology
rlm@151 157
rlm@198 158 Information from the senses is transmitted to the brain via bundles of
rlm@198 159 axons, whether it be the optic nerve or the spinal cord. While these
rlm@198 160 bundles more or less perserve the overall topology of a sense's
rlm@198 161 two-dimensional surface, they do not perserve the percise euclidean
rlm@198 162 distances between every sensor. =(collapse)= is here to smoosh the
rlm@198 163 sensors described by a UV-map into a contigous region that still
rlm@198 164 perserves the topology of the original sense.
rlm@198 165
rlm@198 166 #+name: topology-2
rlm@198 167 #+begin_src clojure
rlm@151 168 (defn average [coll]
rlm@151 169 (/ (reduce + coll) (count coll)))
rlm@151 170
rlm@151 171 (defn collapse-1d
rlm@182 172 "One dimensional analogue of collapse."
rlm@151 173 [center line]
rlm@151 174 (let [length (count line)
rlm@151 175 num-above (count (filter (partial < center) line))
rlm@151 176 num-below (- length num-above)]
rlm@151 177 (range (- center num-below)
rlm@151 178 (+ center num-above))))
rlm@151 179
rlm@151 180 (defn collapse
rlm@151 181 "Take a set of pairs of integers and collapse them into a
rlm@182 182 contigous bitmap with no \"holes\"."
rlm@151 183 [points]
rlm@151 184 (if (empty? points) []
rlm@151 185 (let
rlm@151 186 [num-points (count points)
rlm@151 187 center (vector
rlm@151 188 (int (average (map first points)))
rlm@151 189 (int (average (map first points))))
rlm@151 190 flattened
rlm@151 191 (reduce
rlm@151 192 concat
rlm@151 193 (map
rlm@151 194 (fn [column]
rlm@151 195 (map vector
rlm@151 196 (map first column)
rlm@151 197 (collapse-1d (second center)
rlm@151 198 (map second column))))
rlm@151 199 (partition-by first (sort-by first points))))
rlm@151 200 squeezed
rlm@151 201 (reduce
rlm@151 202 concat
rlm@151 203 (map
rlm@151 204 (fn [row]
rlm@151 205 (map vector
rlm@151 206 (collapse-1d (first center)
rlm@151 207 (map first row))
rlm@151 208 (map second row)))
rlm@151 209 (partition-by second (sort-by second flattened))))
rlm@182 210 relocated
rlm@151 211 (let [min-x (apply min (map first squeezed))
rlm@151 212 min-y (apply min (map second squeezed))]
rlm@151 213 (map (fn [[x y]]
rlm@151 214 [(- x min-x)
rlm@151 215 (- y min-y)])
rlm@151 216 squeezed))]
rlm@182 217 relocated)))
rlm@198 218 #+end_src
rlm@198 219 * Viewing Sense Data
rlm@151 220
rlm@198 221 It's vital to /see/ the sense data to make sure that everything is
rlm@200 222 behaving as it should. =(view-sense)= and its helper, =(view-image)=
rlm@200 223 are here so that each sense can define its own way of turning
rlm@200 224 sense-data into pictures, while the actual rendering of said pictures
rlm@200 225 stays in one central place. =(points->image)= helps senses generate a
rlm@200 226 base image onto which they can overlay actual sense data.
rlm@198 227
rlm@199 228 #+name: view-senses
rlm@198 229 #+begin_src clojure
rlm@199 230 (in-ns 'cortex.sense)
rlm@198 231
rlm@199 232 (defn view-image
rlm@199 233 "Initailizes a JPanel on which you may draw a BufferedImage.
rlm@199 234 Returns a function that accepts a BufferedImage and draws it to the
rlm@199 235 JPanel. If given a directory it will save the images as png files
rlm@199 236 starting at 0000000.png and incrementing from there."
rlm@199 237 ([#^File save]
rlm@199 238 (let [idx (atom -1)
rlm@199 239 image
rlm@199 240 (atom
rlm@199 241 (BufferedImage. 1 1 BufferedImage/TYPE_4BYTE_ABGR))
rlm@199 242 panel
rlm@199 243 (proxy [JPanel] []
rlm@199 244 (paint
rlm@199 245 [graphics]
rlm@199 246 (proxy-super paintComponent graphics)
rlm@199 247 (.drawImage graphics @image 0 0 nil)))
rlm@199 248 frame (JFrame. "Display Image")]
rlm@199 249 (SwingUtilities/invokeLater
rlm@199 250 (fn []
rlm@199 251 (doto frame
rlm@199 252 (-> (.getContentPane) (.add panel))
rlm@199 253 (.pack)
rlm@199 254 (.setLocationRelativeTo nil)
rlm@199 255 (.setResizable true)
rlm@199 256 (.setVisible true))))
rlm@199 257 (fn [#^BufferedImage i]
rlm@199 258 (reset! image i)
rlm@199 259 (.setSize frame (+ 8 (.getWidth i)) (+ 28 (.getHeight i)))
rlm@199 260 (.repaint panel 0 0 (.getWidth i) (.getHeight i))
rlm@199 261 (if save
rlm@199 262 (ImageIO/write
rlm@199 263 i "png"
rlm@199 264 (File. save (format "%07d.png" (swap! idx inc))))))))
rlm@199 265 ([] (view-image nil)))
rlm@199 266
rlm@199 267 (defn view-sense
rlm@199 268 "Take a kernel that produces a BufferedImage from some sense data
rlm@199 269 and return a function which takes a list of sense data, uses the
rlm@199 270 kernel to convert to images, and displays those images, each in
rlm@199 271 its own JFrame."
rlm@199 272 [sense-display-kernel]
rlm@199 273 (let [windows (atom [])]
rlm@215 274 (fn this
rlm@215 275 ([data]
rlm@215 276 (this data nil))
rlm@215 277 ([data save-to]
rlm@215 278 (if (> (count data) (count @windows))
rlm@215 279 (reset!
rlm@215 280 windows
rlm@215 281 (doall
rlm@215 282 (map
rlm@215 283 (fn [idx]
rlm@215 284 (if save-to
rlm@215 285 (let [dir (File. save-to (str idx))]
rlm@215 286 (.mkdir dir)
rlm@215 287 (view-image dir))
rlm@215 288 (view-image))) (range (count data))))))
rlm@215 289 (dorun
rlm@215 290 (map
rlm@215 291 (fn [display datum]
rlm@215 292 (display (sense-display-kernel datum)))
rlm@215 293 @windows data))))))
rlm@215 294
rlm@199 295
rlm@200 296 (defn points->image
rlm@200 297 "Take a collection of points and visuliaze it as a BufferedImage."
rlm@200 298 [points]
rlm@200 299 (if (empty? points)
rlm@200 300 (BufferedImage. 1 1 BufferedImage/TYPE_BYTE_BINARY)
rlm@200 301 (let [xs (vec (map first points))
rlm@200 302 ys (vec (map second points))
rlm@200 303 x0 (apply min xs)
rlm@200 304 y0 (apply min ys)
rlm@200 305 width (- (apply max xs) x0)
rlm@200 306 height (- (apply max ys) y0)
rlm@200 307 image (BufferedImage. (inc width) (inc height)
rlm@200 308 BufferedImage/TYPE_INT_RGB)]
rlm@200 309 (dorun
rlm@200 310 (for [x (range (.getWidth image))
rlm@200 311 y (range (.getHeight image))]
rlm@200 312 (.setRGB image x y 0xFF0000)))
rlm@200 313 (dorun
rlm@200 314 (for [index (range (count points))]
rlm@200 315 (.setRGB image (- (xs index) x0) (- (ys index) y0) -1)))
rlm@200 316 image)))
rlm@200 317
rlm@198 318 (defn gray
rlm@198 319 "Create a gray RGB pixel with R, G, and B set to num. num must be
rlm@198 320 between 0 and 255."
rlm@198 321 [num]
rlm@198 322 (+ num
rlm@198 323 (bit-shift-left num 8)
rlm@198 324 (bit-shift-left num 16)))
rlm@197 325 #+end_src
rlm@197 326
rlm@198 327 * Building a Sense from Nodes
rlm@198 328 My method for defining senses in blender is the following:
rlm@198 329
rlm@198 330 Senses like vision and hearing are localized to a single point
rlm@198 331 and follow a particular object around. For these:
rlm@198 332
rlm@198 333 - Create a single top-level empty node whose name is the name of the sense
rlm@198 334 - Add empty nodes which each contain meta-data relevant
rlm@198 335 to the sense, including a UV-map describing the number/distribution
rlm@198 336 of sensors if applicipable.
rlm@198 337 - Make each empty-node the child of the top-level
rlm@198 338 node. =(sense-nodes)= below generates functions to find these children.
rlm@198 339
rlm@198 340 For touch, store the path to the UV-map which describes touch-sensors in the
rlm@198 341 meta-data of the object to which that map applies.
rlm@198 342
rlm@198 343 Each sense provides code that analyzes the Node structure of the
rlm@198 344 creature and creates sense-functions. They also modify the Node
rlm@198 345 structure if necessary.
rlm@198 346
rlm@198 347 Empty nodes created in blender have no appearance or physical presence
rlm@198 348 in jMonkeyEngine, but do appear in the scene graph. Empty nodes that
rlm@198 349 represent a sense which "follows" another geometry (like eyes and
rlm@198 350 ears) follow the closest physical object. =(closest-node)= finds this
rlm@198 351 closest object given the Creature and a particular empty node.
rlm@198 352
rlm@198 353 #+name: node-1
rlm@197 354 #+begin_src clojure
rlm@198 355 (defn sense-nodes
rlm@198 356 "For some senses there is a special empty blender node whose
rlm@198 357 children are considered markers for an instance of that sense. This
rlm@198 358 function generates functions to find those children, given the name
rlm@198 359 of the special parent node."
rlm@198 360 [parent-name]
rlm@198 361 (fn [#^Node creature]
rlm@198 362 (if-let [sense-node (.getChild creature parent-name)]
rlm@198 363 (seq (.getChildren sense-node))
rlm@198 364 (do (println-repl "could not find" parent-name "node") []))))
rlm@198 365
rlm@197 366 (defn closest-node
rlm@201 367 "Return the physical node in creature which is closest to the given
rlm@201 368 node."
rlm@198 369 [#^Node creature #^Node empty]
rlm@197 370 (loop [radius (float 0.01)]
rlm@197 371 (let [results (CollisionResults.)]
rlm@197 372 (.collideWith
rlm@197 373 creature
rlm@198 374 (BoundingBox. (.getWorldTranslation empty)
rlm@197 375 radius radius radius)
rlm@197 376 results)
rlm@197 377 (if-let [target (first results)]
rlm@197 378 (.getGeometry target)
rlm@197 379 (recur (float (* 2 radius)))))))
rlm@197 380
rlm@198 381 (defn world-to-local
rlm@198 382 "Convert the world coordinates into coordinates relative to the
rlm@198 383 object (i.e. local coordinates), taking into account the rotation
rlm@198 384 of object."
rlm@198 385 [#^Spatial object world-coordinate]
rlm@198 386 (.worldToLocal object world-coordinate nil))
rlm@198 387
rlm@198 388 (defn local-to-world
rlm@198 389 "Convert the local coordinates into world relative coordinates"
rlm@198 390 [#^Spatial object local-coordinate]
rlm@198 391 (.localToWorld object local-coordinate nil))
rlm@198 392 #+end_src
rlm@198 393
rlm@200 394 ** Sense Binding
rlm@200 395
rlm@198 396 =(bind-sense)= binds either a Camera or a Listener object to any
rlm@198 397 object so that they will follow that object no matter how it
rlm@199 398 moves. It is used to create both eyes and ears.
rlm@198 399
rlm@198 400 #+name: node-2
rlm@198 401 #+begin_src clojure
rlm@197 402 (defn bind-sense
rlm@197 403 "Bind the sense to the Spatial such that it will maintain its
rlm@197 404 current position relative to the Spatial no matter how the spatial
rlm@197 405 moves. 'sense can be either a Camera or Listener object."
rlm@197 406 [#^Spatial obj sense]
rlm@197 407 (let [sense-offset (.subtract (.getLocation sense)
rlm@197 408 (.getWorldTranslation obj))
rlm@197 409 initial-sense-rotation (Quaternion. (.getRotation sense))
rlm@197 410 base-anti-rotation (.inverse (.getWorldRotation obj))]
rlm@197 411 (.addControl
rlm@197 412 obj
rlm@197 413 (proxy [AbstractControl] []
rlm@197 414 (controlUpdate [tpf]
rlm@197 415 (let [total-rotation
rlm@197 416 (.mult base-anti-rotation (.getWorldRotation obj))]
rlm@197 417 (.setLocation
rlm@197 418 sense
rlm@197 419 (.add
rlm@197 420 (.mult total-rotation sense-offset)
rlm@197 421 (.getWorldTranslation obj)))
rlm@197 422 (.setRotation
rlm@197 423 sense
rlm@197 424 (.mult total-rotation initial-sense-rotation))))
rlm@197 425 (controlRender [_ _])))))
rlm@197 426 #+end_src
rlm@164 427
rlm@200 428 Here is some example code which shows how a camera bound to a blue box
rlm@200 429 with =(bind-sense)= moves as the box is buffeted by white cannonballs.
rlm@199 430
rlm@199 431 #+name: test
rlm@199 432 #+begin_src clojure
rlm@199 433 (defn test-bind-sense
rlm@201 434 "Show a camera that stays in the same relative position to a blue
rlm@201 435 cube."
rlm@199 436 []
rlm@201 437 (let [eye-pos (Vector3f. 0 30 0)
rlm@199 438 rock (box 1 1 1 :color ColorRGBA/Blue
rlm@199 439 :position (Vector3f. 0 10 0)
rlm@199 440 :mass 30)
rlm@199 441 table (box 3 1 10 :color ColorRGBA/Gray :mass 0
rlm@199 442 :position (Vector3f. 0 -3 0))]
rlm@199 443 (world
rlm@199 444 (nodify [rock table])
rlm@199 445 standard-debug-controls
rlm@201 446 (fn init [world]
rlm@201 447 (let [cam (doto (.clone (.getCamera world))
rlm@201 448 (.setLocation eye-pos)
rlm@201 449 (.lookAt Vector3f/ZERO
rlm@201 450 Vector3f/UNIT_X))]
rlm@199 451 (bind-sense rock cam)
rlm@199 452 (.setTimer world (RatchetTimer. 60))
rlm@199 453 (Capture/captureVideo
rlm@199 454 world (File. "/home/r/proj/cortex/render/bind-sense0"))
rlm@199 455 (add-camera!
rlm@199 456 world cam
rlm@199 457 (comp (view-image
rlm@199 458 (File. "/home/r/proj/cortex/render/bind-sense1"))
rlm@199 459 BufferedImage!))
rlm@199 460 (add-camera! world (.getCamera world) no-op)))
rlm@199 461 no-op)))
rlm@199 462 #+end_src
rlm@199 463
rlm@199 464 #+begin_html
rlm@199 465 <video controls="controls" width="755">
rlm@199 466 <source src="../video/bind-sense.ogg" type="video/ogg"
rlm@199 467 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@199 468 </video>
rlm@199 469 #+end_html
rlm@199 470
rlm@200 471 With this, eyes are easy --- you just bind the camera closer to the
rlm@200 472 desired object, and set it to look outward instead of inward as it
rlm@200 473 does in the video.
rlm@199 474
rlm@200 475 (nb : the video was created with the following commands)
rlm@199 476
rlm@200 477 *** Combine Frames with ImageMagick
rlm@199 478 #+begin_src clojure :results silent
rlm@215 479 (ns cortex.video.magick
rlm@215 480 (:import java.io.File)
rlm@215 481 (:use clojure.contrib.shell-out))
rlm@215 482
rlm@215 483 (defn combine-images []
rlm@215 484 (let
rlm@215 485 [idx (atom -1)
rlm@215 486 left (rest
rlm@215 487 (sort
rlm@215 488 (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/"))))
rlm@215 489 right (rest
rlm@215 490 (sort
rlm@215 491 (file-seq
rlm@215 492 (File. "/home/r/proj/cortex/render/bind-sense1/"))))
rlm@215 493 sub (rest
rlm@199 494 (sort
rlm@200 495 (file-seq
rlm@215 496 (File. "/home/r/proj/cortex/render/bind-senseB/"))))
rlm@215 497 sub* (concat sub (repeat 1000 (last sub)))]
rlm@215 498 (dorun
rlm@215 499 (map
rlm@215 500 (fn [im-1 im-2 sub]
rlm@215 501 (sh "convert" (.getCanonicalPath im-1)
rlm@215 502 (.getCanonicalPath im-2) "+append"
rlm@215 503 (.getCanonicalPath sub) "-append"
rlm@215 504 (.getCanonicalPath
rlm@215 505 (File. "/home/r/proj/cortex/render/bind-sense/"
rlm@215 506 (format "%07d.png" (swap! idx inc))))))
rlm@215 507 left right sub*))))
rlm@199 508 #+end_src
rlm@199 509
rlm@200 510 *** Encode Frames with ffmpeg
rlm@200 511
rlm@199 512 #+begin_src sh :results silent
rlm@199 513 cd /home/r/proj/cortex/render/
rlm@199 514 ffmpeg -r 60 -b 9000k -i bind-sense/%07d.png bind-sense.ogg
rlm@199 515 #+end_src
rlm@199 516
rlm@211 517 * Headers
rlm@211 518 #+name: sense-header
rlm@197 519 #+begin_src clojure
rlm@198 520 (ns cortex.sense
rlm@198 521 "Here are functions useful in the construction of two or more
rlm@198 522 sensors/effectors."
rlm@198 523 {:author "Robert McInytre"}
rlm@198 524 (:use (cortex world util))
rlm@198 525 (:import ij.process.ImageProcessor)
rlm@198 526 (:import jme3tools.converters.ImageToAwt)
rlm@198 527 (:import java.awt.image.BufferedImage)
rlm@198 528 (:import com.jme3.collision.CollisionResults)
rlm@198 529 (:import com.jme3.bounding.BoundingBox)
rlm@198 530 (:import (com.jme3.scene Node Spatial))
rlm@198 531 (:import com.jme3.scene.control.AbstractControl)
rlm@199 532 (:import (com.jme3.math Quaternion Vector3f))
rlm@199 533 (:import javax.imageio.ImageIO)
rlm@199 534 (:import java.io.File)
rlm@199 535 (:import (javax.swing JPanel JFrame SwingUtilities)))
rlm@198 536 #+end_src
rlm@187 537
rlm@211 538 #+name: test-header
rlm@211 539 #+begin_src clojure
rlm@211 540 (ns cortex.test.sense
rlm@211 541 (:use (cortex world util sense vision))
rlm@211 542 (:import
rlm@211 543 java.io.File
rlm@211 544 (com.jme3.math Vector3f ColorRGBA)
rlm@211 545 (com.aurellem.capture RatchetTimer Capture)))
rlm@211 546 #+end_src
rlm@211 547
rlm@198 548 * Source Listing
rlm@211 549 - [[../src/cortex/sense.clj][cortex.sense]]
rlm@211 550 - [[../src/cortex/test/sense.clj][cortex.test.sense]]
rlm@211 551 - [[../assets/Models/subtitles/subtitles.blend][subtitles.blend]]
rlm@211 552 - [[../assets/Models/subtitles/Lake_CraterLake03_sm.hdr][subtitles reflection map]]
rlm@211 553 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
rlm@211 554
rlm@211 555 * Next
rlm@211 556 Now that some of the preliminaries are out of the way, in the [[./body.org][next
rlm@211 557 post]] I'll create a simulated body.
rlm@198 558
rlm@187 559
rlm@151 560 * COMMENT generate source
rlm@151 561 #+begin_src clojure :tangle ../src/cortex/sense.clj
rlm@211 562 <<sense-header>>
rlm@198 563 <<blender-1>>
rlm@198 564 <<blender-2>>
rlm@198 565 <<topology-1>>
rlm@198 566 <<topology-2>>
rlm@198 567 <<node-1>>
rlm@198 568 <<node-2>>
rlm@197 569 <<view-senses>>
rlm@151 570 #+end_src
rlm@199 571
rlm@199 572 #+begin_src clojure :tangle ../src/cortex/test/sense.clj
rlm@211 573 <<test-header>>
rlm@199 574 <<test>>
rlm@199 575 #+end_src
rlm@215 576
rlm@215 577 #+begin_src clojure :tangle ../src/cortex/video/magick.clj
rlm@215 578 <<magick>>
rlm@215 579 #+end_src