view org/vision.org @ 270:aa3641042958

minor formatting changes.
author Robert McIntyre <rlm@mit.edu>
date Tue, 14 Feb 2012 05:30:55 -0700
parents e57d8c52f12f
children 12e6231eae8e c39b8b29a79e
line wrap: on
line source
1 #+title: Simulated Sense of Sight
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both
10 # SUGGEST: Call functions by their name, without
11 # parentheses. e.g. =add-eye!=, not =(add-eye!)=. The reason for this
12 # is that it is potentially easy to confuse the /function/ =f= with its
13 # /value/ at a particular point =(f x)=. Mathematicians have this
14 # problem with their notation; we don't need it in ours.
16 * JMonkeyEngine natively supports multiple views of the same world.
18 Vision is one of the most important senses for humans, so I need to
19 build a simulated sense of vision for my AI. I will do this with
20 simulated eyes. Each eye can be independely moved and should see its
21 own version of the world depending on where it is.
23 Making these simulated eyes a reality is simple bacause jMonkeyEngine
24 already conatains extensive support for multiple views of the same 3D
25 simulated world. The reason jMonkeyEngine has this support is because
26 the support is necessary to create games with split-screen
27 views. Multiple views are also used to create efficient
28 pseudo-reflections by rendering the scene from a certain perspective
29 and then projecting it back onto a surface in the 3D world.
31 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
32 [[../images/goldeneye-4-player.png]]
34 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.
35 # =Viewports= are cameras; =RenderManger= takes snapshots each frame.
36 #* A Brief Description of jMonkeyEngine's Rendering Pipeline
38 jMonkeyEngine allows you to create a =ViewPort=, which represents a
39 view of the simulated world. You can create as many of these as you
40 want. Every frame, the =RenderManager= iterates through each
41 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
42 is a =FrameBuffer= which represents the rendered image in the GPU.
44 #+caption: =ViewPorts= are cameras in the world. During each frame, the =Rendermanager= records a snapshot of what each view is currently seeing.
45 #+ATTR_HTML: width="400"
46 [[../images/diagram_rendermanager.png]]
48 Each =ViewPort= can have any number of attached =SceneProcessor=
49 objects, which are called every time a new frame is rendered. A
50 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do
51 whatever it wants to the data. Often this consists of invoking GPU
52 specific operations on the rendered image. The =SceneProcessor= can
53 also copy the GPU image data to RAM and process it with the CPU.
55 ** From Views to Vision
56 # Appropriating Views for Vision.
58 Each eye in the simulated creature needs its own =ViewPort= so that
59 it can see the world from its own perspective. To this =ViewPort=, I
60 add a =SceneProcessor= that feeds the visual data to any arbitray
61 continuation function for further processing. That continuation
62 function may perform both CPU and GPU operations on the data. To make
63 this easy for the continuation function, the =SceneProcessor=
64 maintains appropriatly sized buffers in RAM to hold the data. It does
65 not do any copying from the GPU to the CPU itself because it is a slow
66 operation.
68 #+name: pipeline-1
69 #+begin_src clojure
70 (defn vision-pipeline
71 "Create a SceneProcessor object which wraps a vision processing
72 continuation function. The continuation is a function that takes
73 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
74 each of which has already been appropiately sized."
75 [continuation]
76 (let [byte-buffer (atom nil)
77 renderer (atom nil)
78 image (atom nil)]
79 (proxy [SceneProcessor] []
80 (initialize
81 [renderManager viewPort]
82 (let [cam (.getCamera viewPort)
83 width (.getWidth cam)
84 height (.getHeight cam)]
85 (reset! renderer (.getRenderer renderManager))
86 (reset! byte-buffer
87 (BufferUtils/createByteBuffer
88 (* width height 4)))
89 (reset! image (BufferedImage.
90 width height
91 BufferedImage/TYPE_4BYTE_ABGR))))
92 (isInitialized [] (not (nil? @byte-buffer)))
93 (reshape [_ _ _])
94 (preFrame [_])
95 (postQueue [_])
96 (postFrame
97 [#^FrameBuffer fb]
98 (.clear @byte-buffer)
99 (continuation @renderer fb @byte-buffer @image))
100 (cleanup []))))
101 #+end_src
103 The continuation function given to =(vision-pipeline)= above will be
104 given a =Renderer= and three containers for image data. The
105 =FrameBuffer= references the GPU image data, but the pixel data can
106 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
107 are initially "empty" but are sized to hold the data in the
108 =FrameBuffer=. I call transfering the GPU image data to the CPU
109 structures "mixing" the image data. I have provided three functions to
110 do this mixing.
112 #+name: pipeline-2
113 #+begin_src clojure
114 (defn frameBuffer->byteBuffer!
115 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
116 the CPU (ByteBuffer)."
117 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
118 (.readFrameBuffer r fb bb) bb)
120 (defn byteBuffer->bufferedImage!
121 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
122 style ABGR image data and place it in BufferedImage bi."
123 [#^ByteBuffer bb #^BufferedImage bi]
124 (Screenshots/convertScreenShot bb bi) bi)
126 (defn BufferedImage!
127 "Continuation which will grab the buffered image from the materials
128 provided by (vision-pipeline)."
129 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
130 (byteBuffer->bufferedImage!
131 (frameBuffer->byteBuffer! r fb bb) bi))
132 #+end_src
134 Note that it is possible to write vision processing algorithms
135 entirely in terms of =BufferedImage= inputs. Just compose that
136 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
137 processing algorithm that is entirely hosted on the GPU does not have
138 to pay for this convienence.
140 * Optical sensor arrays are described with images and referenced with metadata
141 The vision pipeline described above handles the flow of rendered
142 images. Now, we need simulated eyes to serve as the source of these
143 images.
145 An eye is described in blender in the same way as a joint. They are
146 zero dimensional empty objects with no geometry whose local coordinate
147 system determines the orientation of the resulting eye. All eyes are
148 childern of a parent node named "eyes" just as all joints have a
149 parent named "joints". An eye binds to the nearest physical object
150 with =(bind-sense=).
152 #+name: add-eye
153 #+begin_src clojure
154 (in-ns 'cortex.vision)
156 (defn add-eye!
157 "Create a Camera centered on the current position of 'eye which
158 follows the closest physical node in 'creature and sends visual
159 data to 'continuation. The camera will point in the X direction and
160 use the Z vector as up as determined by the rotation of these
161 vectors in blender coordinate space. Use XZY rotation for the node
162 in blender."
163 [#^Node creature #^Spatial eye]
164 (let [target (closest-node creature eye)
165 [cam-width cam-height] (eye-dimensions eye)
166 cam (Camera. cam-width cam-height)
167 rot (.getWorldRotation eye)]
168 (.setLocation cam (.getWorldTranslation eye))
169 (.lookAtDirection
170 cam ; this part is not a mistake and
171 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
172 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
173 (.setFrustumPerspective
174 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
175 (bind-sense target cam) cam))
176 #+end_src
178 Here, the camera is created based on metadata on the eye-node and
179 attached to the nearest physical object with =(bind-sense)=
180 ** The Retina
182 An eye is a surface (the retina) which contains many discrete sensors
183 to detect light. These sensors have can have different light-sensing
184 properties. In humans, each discrete sensor is sensitive to red,
185 blue, green, or gray. These different types of sensors can have
186 different spatial distributions along the retina. In humans, there is
187 a fovea in the center of the retina which has a very high density of
188 color sensors, and a blind spot which has no sensors at all. Sensor
189 density decreases in proportion to distance from the fovea.
191 I want to be able to model any retinal configuration, so my eye-nodes
192 in blender contain metadata pointing to images that describe the
193 percise position of the individual sensors using white pixels. The
194 meta-data also describes the percise sensitivity to light that the
195 sensors described in the image have. An eye can contain any number of
196 these images. For example, the metadata for an eye might look like
197 this:
199 #+begin_src clojure
200 {0xFF0000 "Models/test-creature/retina-small.png"}
201 #+end_src
203 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
204 [[../assets/Models/test-creature/retina-small.png]]
206 Together, the number 0xFF0000 and the image image above describe the
207 placement of red-sensitive sensory elements.
209 Meta-data to very crudely approximate a human eye might be something
210 like this:
212 #+begin_src clojure
213 (let [retinal-profile "Models/test-creature/retina-small.png"]
214 {0xFF0000 retinal-profile
215 0x00FF00 retinal-profile
216 0x0000FF retinal-profile
217 0xFFFFFF retinal-profile})
218 #+end_src
220 The numbers that serve as keys in the map determine a sensor's
221 relative sensitivity to the channels red, green, and blue. These
222 sensitivity values are packed into an integer in the order =|_|R|G|B|=
223 in 8-bit fields. The RGB values of a pixel in the image are added
224 together with these sensitivities as linear weights. Therfore,
225 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
226 all colors equally (gray).
228 For convienence I've defined a few symbols for the more common
229 sensitivity values.
231 #+name: sensitivity
232 #+begin_src clojure
233 (defvar sensitivity-presets
234 {:all 0xFFFFFF
235 :red 0xFF0000
236 :blue 0x0000FF
237 :green 0x00FF00}
238 "Retinal sensitivity presets for sensors that extract one channel
239 (:red :blue :green) or average all channels (:all)")
240 #+end_src
242 ** Metadata Processing
244 =(retina-sensor-profile)= extracts a map from the eye-node in the same
245 format as the example maps above. =(eye-dimensions)= finds the
246 dimensions of the smallest image required to contain all the retinal
247 sensor maps.
249 #+name: retina
250 #+begin_src clojure
251 (defn retina-sensor-profile
252 "Return a map of pixel sensitivity numbers to BufferedImages
253 describing the distribution of light-sensitive components of this
254 eye. :red, :green, :blue, :gray are already defined as extracting
255 the red, green, blue, and average components respectively."
256 [#^Spatial eye]
257 (if-let [eye-map (meta-data eye "eye")]
258 (map-vals
259 load-image
260 (eval (read-string eye-map)))))
262 (defn eye-dimensions
263 "Returns [width, height] determined by the metadata of the eye."
264 [#^Spatial eye]
265 (let [dimensions
266 (map #(vector (.getWidth %) (.getHeight %))
267 (vals (retina-sensor-profile eye)))]
268 [(apply max (map first dimensions))
269 (apply max (map second dimensions))]))
270 #+end_src
272 * Importing and parsing descriptions of eyes.
273 First off, get the children of the "eyes" empty node to find all the
274 eyes the creature has.
275 #+name: eye-node
276 #+begin_src clojure
277 (defvar
278 ^{:arglists '([creature])}
279 eyes
280 (sense-nodes "eyes")
281 "Return the children of the creature's \"eyes\" node.")
282 #+end_src
284 Then, add the camera created by =(add-eye!)= to the simulation by
285 creating a new viewport.
287 #+name: add-camera
288 #+begin_src clojure
289 (defn add-camera!
290 "Add a camera to the world, calling continuation on every frame
291 produced."
292 [#^Application world camera continuation]
293 (let [width (.getWidth camera)
294 height (.getHeight camera)
295 render-manager (.getRenderManager world)
296 viewport (.createMainView render-manager "eye-view" camera)]
297 (doto viewport
298 (.setClearFlags true true true)
299 (.setBackgroundColor ColorRGBA/Black)
300 (.addProcessor (vision-pipeline continuation))
301 (.attachScene (.getRootNode world)))))
302 #+end_src
305 The eye's continuation function should register the viewport with the
306 simulation the first time it is called, use the CPU to extract the
307 appropriate pixels from the rendered image and weight them by each
308 sensor's sensitivity. I have the option to do this processing in
309 native code for a slight gain in speed. I could also do it in the GPU
310 for a massive gain in speed. =(vision-kernel)= generates a list of
311 such continuation functions, one for each channel of the eye.
313 #+name: kernel
314 #+begin_src clojure
315 (in-ns 'cortex.vision)
317 (defrecord attached-viewport [vision-fn viewport-fn]
318 clojure.lang.IFn
319 (invoke [this world] (vision-fn world))
320 (applyTo [this args] (apply vision-fn args)))
322 (defn pixel-sense [sensitivity pixel]
323 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
324 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
325 s-b (bit-and 0x0000FF sensitivity)
327 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
328 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
329 p-b (bit-and 0x0000FF pixel)
331 total-sensitivity (* 255 (+ s-r s-g s-b))]
332 (float (/ (+ (* s-r p-r)
333 (* s-g p-g)
334 (* s-b p-b))
335 total-sensitivity))))
337 (defn vision-kernel
338 "Returns a list of functions, each of which will return a color
339 channel's worth of visual information when called inside a running
340 simulation."
341 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
342 (let [retinal-map (retina-sensor-profile eye)
343 camera (add-eye! creature eye)
344 vision-image
345 (atom
346 (BufferedImage. (.getWidth camera)
347 (.getHeight camera)
348 BufferedImage/TYPE_BYTE_BINARY))
349 register-eye!
350 (runonce
351 (fn [world]
352 (add-camera!
353 world camera
354 (let [counter (atom 0)]
355 (fn [r fb bb bi]
356 (if (zero? (rem (swap! counter inc) (inc skip)))
357 (reset! vision-image
358 (BufferedImage! r fb bb bi))))))))]
359 (vec
360 (map
361 (fn [[key image]]
362 (let [whites (white-coordinates image)
363 topology (vec (collapse whites))
364 sensitivity (sensitivity-presets key key)]
365 (attached-viewport.
366 (fn [world]
367 (register-eye! world)
368 (vector
369 topology
370 (vec
371 (for [[x y] whites]
372 (pixel-sense
373 sensitivity
374 (.getRGB @vision-image x y))))))
375 register-eye!)))
376 retinal-map))))
378 (defn gen-fix-display
379 "Create a function to call to restore a simulation's display when it
380 is disrupted by a Viewport."
381 []
382 (runonce
383 (fn [world]
384 (add-camera! world (.getCamera world) no-op))))
385 #+end_src
387 Note that since each of the functions generated by =(vision-kernel)=
388 shares the same =(register-eye!)= function, the eye will be registered
389 only once the first time any of the functions from the list returned
390 by =(vision-kernel)= is called. Each of the functions returned by
391 =(vision-kernel)= also allows access to the =Viewport= through which
392 it recieves images.
394 The in-game display can be disrupted by all the viewports that the
395 functions greated by =(vision-kernel)= add. This doesn't affect the
396 simulation or the simulated senses, but can be annoying.
397 =(gen-fix-display)= restores the in-simulation display.
399 ** The =vision!= function creates sensory probes.
401 All the hard work has been done; all that remains is to apply
402 =(vision-kernel)= to each eye in the creature and gather the results
403 into one list of functions.
405 #+name: main
406 #+begin_src clojure
407 (defn vision!
408 "Returns a function which returns visual sensory data when called
409 inside a running simulation."
410 [#^Node creature & {skip :skip :or {skip 0}}]
411 (reduce
412 concat
413 (for [eye (eyes creature)]
414 (vision-kernel creature eye))))
415 #+end_src
417 ** Displaying visual data for debugging.
418 # Visualization of Vision. Maybe less alliteration would be better.
419 It's vital to have a visual representation for each sense. Here I use
420 =(view-sense)= to construct a function that will create a display for
421 visual data.
423 #+name: display
424 #+begin_src clojure
425 (in-ns 'cortex.vision)
427 (defn view-vision
428 "Creates a function which accepts a list of visual sensor-data and
429 displays each element of the list to the screen."
430 []
431 (view-sense
432 (fn
433 [[coords sensor-data]]
434 (let [image (points->image coords)]
435 (dorun
436 (for [i (range (count coords))]
437 (.setRGB image ((coords i) 0) ((coords i) 1)
438 (gray (int (* 255 (sensor-data i)))))))
439 image))))
440 #+end_src
442 * Demonstrations
443 ** Demonstrating the vision pipeline.
445 This is a basic test for the vision system. It only tests the
446 vision-pipeline and does not deal with loading eyes from a blender
447 file. The code creates two videos of the same rotating cube from
448 different angles.
450 #+name: test-1
451 #+begin_src clojure
452 (in-ns 'cortex.test.vision)
454 (defn test-pipeline
455 "Testing vision:
456 Tests the vision system by creating two views of the same rotating
457 object from different angles and displaying both of those views in
458 JFrames.
460 You should see a rotating cube, and two windows,
461 each displaying a different view of the cube."
462 []
463 (let [candy
464 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
465 (world
466 (doto (Node.)
467 (.attachChild candy))
468 {}
469 (fn [world]
470 (let [cam (.clone (.getCamera world))
471 width (.getWidth cam)
472 height (.getHeight cam)]
473 (add-camera! world cam
474 (comp
475 (view-image
476 (File. "/home/r/proj/cortex/render/vision/1"))
477 BufferedImage!))
478 (add-camera! world
479 (doto (.clone cam)
480 (.setLocation (Vector3f. -10 0 0))
481 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
482 (comp
483 (view-image
484 (File. "/home/r/proj/cortex/render/vision/2"))
485 BufferedImage!))
486 ;; This is here to restore the main view
487 ;; after the other views have completed processing
488 (add-camera! world (.getCamera world) no-op)))
489 (fn [world tpf]
490 (.rotate candy (* tpf 0.2) 0 0)))))
491 #+end_src
493 #+begin_html
494 <div class="figure">
495 <video controls="controls" width="755">
496 <source src="../video/spinning-cube.ogg" type="video/ogg"
497 preload="none" poster="../images/aurellem-1280x480.png" />
498 </video>
499 <p>A rotating cube viewed from two different perspectives.</p>
500 </div>
501 #+end_html
503 Creating multiple eyes like this can be used for stereoscopic vision
504 simulation in a single creature or for simulating multiple creatures,
505 each with their own sense of vision.
506 ** Demonstrating eye import and parsing.
508 To the worm from the last post, I add a new node that describes its
509 eyes.
511 #+attr_html: width=755
512 #+caption: The worm with newly added empty nodes describing a single eye.
513 [[../images/worm-with-eye.png]]
515 The node highlighted in yellow is the root level "eyes" node. It has
516 a single child, highlighted in orange, which describes a single
517 eye. This is the "eye" node. It is placed so that the worm will have
518 an eye located in the center of the flat portion of its lower
519 hemispherical section.
521 The two nodes which are not highlighted describe the single joint of
522 the worm.
524 The metadata of the eye-node is:
526 #+begin_src clojure :results verbatim :exports both
527 (cortex.sense/meta-data
528 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
529 #+end_src
531 #+results:
532 : "(let [retina \"Models/test-creature/retina-small.png\"]
533 : {:all retina :red retina :green retina :blue retina})"
535 This is the approximation to the human eye described earlier.
537 #+name: test-2
538 #+begin_src clojure
539 (in-ns 'cortex.test.vision)
541 (defn change-color [obj color]
542 (println-repl obj)
543 (if obj
544 (.setColor (.getMaterial obj) "Color" color)))
546 (defn colored-cannon-ball [color]
547 (comp #(change-color % color)
548 (fire-cannon-ball)))
550 (defn test-worm-vision [record]
551 (let [the-worm (doto (worm)(body!))
552 vision (vision! the-worm)
553 vision-display (view-vision)
554 fix-display (gen-fix-display)
555 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
556 x-axis
557 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
558 :position (Vector3f. 0 -5 0))
559 y-axis
560 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
561 :position (Vector3f. 0 -5 0))
562 z-axis
563 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
564 :position (Vector3f. 0 -5 0))
565 timer (RatchetTimer. 60)]
567 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])
568 (assoc standard-debug-controls
569 "key-r" (colored-cannon-ball ColorRGBA/Red)
570 "key-b" (colored-cannon-ball ColorRGBA/Blue)
571 "key-g" (colored-cannon-ball ColorRGBA/Green))
572 (fn [world]
573 (light-up-everything world)
574 (speed-up world)
575 (.setTimer world timer)
576 (display-dialated-time world timer)
577 ;; add a view from the worm's perspective
578 (if record
579 (Capture/captureVideo
580 world
581 (File.
582 "/home/r/proj/cortex/render/worm-vision/main-view")))
584 (add-camera!
585 world
586 (add-eye! the-worm
587 (.getChild
588 (.getChild the-worm "eyes") "eye"))
589 (comp
590 (view-image
591 (if record
592 (File.
593 "/home/r/proj/cortex/render/worm-vision/worm-view")))
594 BufferedImage!))
596 (set-gravity world Vector3f/ZERO))
598 (fn [world _ ]
599 (.setLocalTranslation me (.getLocation (.getCamera world)))
600 (vision-display
601 (map #(% world) vision)
602 (if record (File. "/home/r/proj/cortex/render/worm-vision")))
603 (fix-display world)))))
604 #+end_src
606 The world consists of the worm and a flat gray floor. I can shoot red,
607 green, blue and white cannonballs at the worm. The worm is initially
608 looking down at the floor, and there is no gravity. My perspective
609 (the Main View), the worm's perspective (Worm View) and the 4 sensor
610 channels that comprise the worm's eye are all saved frame-by-frame to
611 disk.
613 * Demonstration of Vision
614 #+begin_html
615 <div class="figure">
616 <video controls="controls" width="755">
617 <source src="../video/worm-vision.ogg" type="video/ogg"
618 preload="none" poster="../images/aurellem-1280x480.png" />
619 </video>
620 <p>Simulated Vision in a Virtual Environment</p>
621 </div>
622 #+end_html
624 ** Generate the Worm Video from Frames
625 #+name: magick2
626 #+begin_src clojure
627 (ns cortex.video.magick2
628 (:import java.io.File)
629 (:use clojure.contrib.shell-out))
631 (defn images [path]
632 (sort (rest (file-seq (File. path)))))
634 (def base "/home/r/proj/cortex/render/worm-vision/")
636 (defn pics [file]
637 (images (str base file)))
639 (defn combine-images []
640 (let [main-view (pics "main-view")
641 worm-view (pics "worm-view")
642 blue (pics "0")
643 green (pics "1")
644 red (pics "2")
645 gray (pics "3")
646 blender (let [b-pics (pics "blender")]
647 (concat b-pics (repeat 9001 (last b-pics))))
648 background (repeat 9001 (File. (str base "background.png")))
649 targets (map
650 #(File. (str base "out/" (format "%07d.png" %)))
651 (range 0 (count main-view)))]
652 (dorun
653 (pmap
654 (comp
655 (fn [[background main-view worm-view red green blue gray blender target]]
656 (println target)
657 (sh "convert"
658 background
659 main-view "-geometry" "+18+17" "-composite"
660 worm-view "-geometry" "+677+17" "-composite"
661 green "-geometry" "+685+430" "-composite"
662 red "-geometry" "+788+430" "-composite"
663 blue "-geometry" "+894+430" "-composite"
664 gray "-geometry" "+1000+430" "-composite"
665 blender "-geometry" "+0+0" "-composite"
666 target))
667 (fn [& args] (map #(.getCanonicalPath %) args)))
668 background main-view worm-view red green blue gray blender targets))))
669 #+end_src
671 #+begin_src sh :results silent
672 cd /home/r/proj/cortex/render/worm-vision
673 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
674 #+end_src
676 * Onward!
677 - As a neat bonus, this idea behind simulated vision also enables one
678 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
679 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
682 #+appendix
684 * Headers
686 #+name: vision-header
687 #+begin_src clojure
688 (ns cortex.vision
689 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
690 eyes from different positions to observe the same world, and pass
691 the observed data to any arbitray function. Automatically reads
692 eye-nodes from specially prepared blender files and instantiates
693 them in the world as actual eyes."
694 {:author "Robert McIntyre"}
695 (:use (cortex world sense util))
696 (:use clojure.contrib.def)
697 (:import com.jme3.post.SceneProcessor)
698 (:import (com.jme3.util BufferUtils Screenshots))
699 (:import java.nio.ByteBuffer)
700 (:import java.awt.image.BufferedImage)
701 (:import (com.jme3.renderer ViewPort Camera))
702 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
703 (:import com.jme3.renderer.Renderer)
704 (:import com.jme3.app.Application)
705 (:import com.jme3.texture.FrameBuffer)
706 (:import (com.jme3.scene Node Spatial)))
707 #+end_src
709 #+name: test-header
710 #+begin_src clojure
711 (ns cortex.test.vision
712 (:use (cortex world sense util body vision))
713 (:use cortex.test.body)
714 (:import java.awt.image.BufferedImage)
715 (:import javax.swing.JPanel)
716 (:import javax.swing.SwingUtilities)
717 (:import java.awt.Dimension)
718 (:import javax.swing.JFrame)
719 (:import com.jme3.math.ColorRGBA)
720 (:import com.jme3.scene.Node)
721 (:import com.jme3.math.Vector3f)
722 (:import java.io.File)
723 (:import (com.aurellem.capture Capture RatchetTimer)))
724 #+end_src
725 * Source Listing
726 - [[../src/cortex/vision.clj][cortex.vision]]
727 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
728 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
729 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
730 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
731 - [[http://hg.bortreb.com ][source-repository]]
735 * COMMENT Generate Source
736 #+begin_src clojure :tangle ../src/cortex/vision.clj
737 <<vision-header>>
738 <<pipeline-1>>
739 <<pipeline-2>>
740 <<retina>>
741 <<add-eye>>
742 <<sensitivity>>
743 <<eye-node>>
744 <<add-camera>>
745 <<kernel>>
746 <<main>>
747 <<display>>
748 #+end_src
750 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
751 <<test-header>>
752 <<test-1>>
753 <<test-2>>
754 #+end_src
756 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
757 <<magick2>>
758 #+end_src