view org/vision.org @ 297:d1206b11ae2d

creating final video
author Robert McIntyre <rlm@mit.edu>
date Thu, 16 Feb 2012 12:48:51 -0700
parents 23aadf376e9d
children 7e7f8d6d9ec5
line wrap: on
line source
1 #+title: Simulated Sense of Sight
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both
10 * JMonkeyEngine natively supports multiple views of the same world.
12 Vision is one of the most important senses for humans, so I need to
13 build a simulated sense of vision for my AI. I will do this with
14 simulated eyes. Each eye can be independely moved and should see its
15 own version of the world depending on where it is.
17 Making these simulated eyes a reality is simple bacause jMonkeyEngine
18 already conatains extensive support for multiple views of the same 3D
19 simulated world. The reason jMonkeyEngine has this support is because
20 the support is necessary to create games with split-screen
21 views. Multiple views are also used to create efficient
22 pseudo-reflections by rendering the scene from a certain perspective
23 and then projecting it back onto a surface in the 3D world.
25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
26 [[../images/goldeneye-4-player.png]]
28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.
29 # =Viewports= are cameras; =RenderManger= takes snapshots each frame.
30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline
32 jMonkeyEngine allows you to create a =ViewPort=, which represents a
33 view of the simulated world. You can create as many of these as you
34 want. Every frame, the =RenderManager= iterates through each
35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
36 is a =FrameBuffer= which represents the rendered image in the GPU.
38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =Rendermanager= records a snapshot of what each view is currently seeing; these snapshots are =Framebuffer= objects.
39 #+ATTR_HTML: width="400"
40 [[../images/diagram_rendermanager2.png]]
42 Each =ViewPort= can have any number of attached =SceneProcessor=
43 objects, which are called every time a new frame is rendered. A
44 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do
45 whatever it wants to the data. Often this consists of invoking GPU
46 specific operations on the rendered image. The =SceneProcessor= can
47 also copy the GPU image data to RAM and process it with the CPU.
49 ** From Views to Vision
50 # Appropriating Views for Vision.
52 Each eye in the simulated creature needs its own =ViewPort= so that
53 it can see the world from its own perspective. To this =ViewPort=, I
54 add a =SceneProcessor= that feeds the visual data to any arbitray
55 continuation function for further processing. That continuation
56 function may perform both CPU and GPU operations on the data. To make
57 this easy for the continuation function, the =SceneProcessor=
58 maintains appropriatly sized buffers in RAM to hold the data. It does
59 not do any copying from the GPU to the CPU itself because it is a slow
60 operation.
62 #+name: pipeline-1
63 #+begin_src clojure
64 (defn vision-pipeline
65 "Create a SceneProcessor object which wraps a vision processing
66 continuation function. The continuation is a function that takes
67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
68 each of which has already been appropiately sized."
69 [continuation]
70 (let [byte-buffer (atom nil)
71 renderer (atom nil)
72 image (atom nil)]
73 (proxy [SceneProcessor] []
74 (initialize
75 [renderManager viewPort]
76 (let [cam (.getCamera viewPort)
77 width (.getWidth cam)
78 height (.getHeight cam)]
79 (reset! renderer (.getRenderer renderManager))
80 (reset! byte-buffer
81 (BufferUtils/createByteBuffer
82 (* width height 4)))
83 (reset! image (BufferedImage.
84 width height
85 BufferedImage/TYPE_4BYTE_ABGR))))
86 (isInitialized [] (not (nil? @byte-buffer)))
87 (reshape [_ _ _])
88 (preFrame [_])
89 (postQueue [_])
90 (postFrame
91 [#^FrameBuffer fb]
92 (.clear @byte-buffer)
93 (continuation @renderer fb @byte-buffer @image))
94 (cleanup []))))
95 #+end_src
97 The continuation function given to =vision-pipeline= above will be
98 given a =Renderer= and three containers for image data. The
99 =FrameBuffer= references the GPU image data, but the pixel data can
100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
101 are initially "empty" but are sized to hold the data in the
102 =FrameBuffer=. I call transfering the GPU image data to the CPU
103 structures "mixing" the image data. I have provided three functions to
104 do this mixing.
106 #+name: pipeline-2
107 #+begin_src clojure
108 (defn frameBuffer->byteBuffer!
109 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
110 the CPU (ByteBuffer)."
111 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
112 (.readFrameBuffer r fb bb) bb)
114 (defn byteBuffer->bufferedImage!
115 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
116 style ABGR image data and place it in BufferedImage bi."
117 [#^ByteBuffer bb #^BufferedImage bi]
118 (Screenshots/convertScreenShot bb bi) bi)
120 (defn BufferedImage!
121 "Continuation which will grab the buffered image from the materials
122 provided by (vision-pipeline)."
123 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
124 (byteBuffer->bufferedImage!
125 (frameBuffer->byteBuffer! r fb bb) bi))
126 #+end_src
128 Note that it is possible to write vision processing algorithms
129 entirely in terms of =BufferedImage= inputs. Just compose that
130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision
131 processing algorithm that is entirely hosted on the GPU does not have
132 to pay for this convienence.
134 * Optical sensor arrays are described with images and referenced with metadata
135 The vision pipeline described above handles the flow of rendered
136 images. Now, we need simulated eyes to serve as the source of these
137 images.
139 An eye is described in blender in the same way as a joint. They are
140 zero dimensional empty objects with no geometry whose local coordinate
141 system determines the orientation of the resulting eye. All eyes are
142 childern of a parent node named "eyes" just as all joints have a
143 parent named "joints". An eye binds to the nearest physical object
144 with =bind-sense=.
146 #+name: add-eye
147 #+begin_src clojure
148 (in-ns 'cortex.vision)
150 (defn add-eye!
151 "Create a Camera centered on the current position of 'eye which
152 follows the closest physical node in 'creature and sends visual
153 data to 'continuation. The camera will point in the X direction and
154 use the Z vector as up as determined by the rotation of these
155 vectors in blender coordinate space. Use XZY rotation for the node
156 in blender."
157 [#^Node creature #^Spatial eye]
158 (let [target (closest-node creature eye)
159 [cam-width cam-height] (eye-dimensions eye)
160 cam (Camera. cam-width cam-height)
161 rot (.getWorldRotation eye)]
162 (.setLocation cam (.getWorldTranslation eye))
163 (.lookAtDirection
164 cam ; this part is not a mistake and
165 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
166 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
167 (.setFrustumPerspective
168 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
169 (bind-sense target cam) cam))
170 #+end_src
172 Here, the camera is created based on metadata on the eye-node and
173 attached to the nearest physical object with =bind-sense=
174 ** The Retina
176 An eye is a surface (the retina) which contains many discrete sensors
177 to detect light. These sensors have can have different light-sensing
178 properties. In humans, each discrete sensor is sensitive to red,
179 blue, green, or gray. These different types of sensors can have
180 different spatial distributions along the retina. In humans, there is
181 a fovea in the center of the retina which has a very high density of
182 color sensors, and a blind spot which has no sensors at all. Sensor
183 density decreases in proportion to distance from the fovea.
185 I want to be able to model any retinal configuration, so my eye-nodes
186 in blender contain metadata pointing to images that describe the
187 percise position of the individual sensors using white pixels. The
188 meta-data also describes the percise sensitivity to light that the
189 sensors described in the image have. An eye can contain any number of
190 these images. For example, the metadata for an eye might look like
191 this:
193 #+begin_src clojure
194 {0xFF0000 "Models/test-creature/retina-small.png"}
195 #+end_src
197 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
198 [[../assets/Models/test-creature/retina-small.png]]
200 Together, the number 0xFF0000 and the image image above describe the
201 placement of red-sensitive sensory elements.
203 Meta-data to very crudely approximate a human eye might be something
204 like this:
206 #+begin_src clojure
207 (let [retinal-profile "Models/test-creature/retina-small.png"]
208 {0xFF0000 retinal-profile
209 0x00FF00 retinal-profile
210 0x0000FF retinal-profile
211 0xFFFFFF retinal-profile})
212 #+end_src
214 The numbers that serve as keys in the map determine a sensor's
215 relative sensitivity to the channels red, green, and blue. These
216 sensitivity values are packed into an integer in the order =|_|R|G|B|=
217 in 8-bit fields. The RGB values of a pixel in the image are added
218 together with these sensitivities as linear weights. Therfore,
219 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
220 all colors equally (gray).
222 For convienence I've defined a few symbols for the more common
223 sensitivity values.
225 #+name: sensitivity
226 #+begin_src clojure
227 (defvar sensitivity-presets
228 {:all 0xFFFFFF
229 :red 0xFF0000
230 :blue 0x0000FF
231 :green 0x00FF00}
232 "Retinal sensitivity presets for sensors that extract one channel
233 (:red :blue :green) or average all channels (:all)")
234 #+end_src
236 ** Metadata Processing
238 =retina-sensor-profile= extracts a map from the eye-node in the same
239 format as the example maps above. =eye-dimensions= finds the
240 dimensions of the smallest image required to contain all the retinal
241 sensor maps.
243 #+name: retina
244 #+begin_src clojure
245 (defn retina-sensor-profile
246 "Return a map of pixel sensitivity numbers to BufferedImages
247 describing the distribution of light-sensitive components of this
248 eye. :red, :green, :blue, :gray are already defined as extracting
249 the red, green, blue, and average components respectively."
250 [#^Spatial eye]
251 (if-let [eye-map (meta-data eye "eye")]
252 (map-vals
253 load-image
254 (eval (read-string eye-map)))))
256 (defn eye-dimensions
257 "Returns [width, height] determined by the metadata of the eye."
258 [#^Spatial eye]
259 (let [dimensions
260 (map #(vector (.getWidth %) (.getHeight %))
261 (vals (retina-sensor-profile eye)))]
262 [(apply max (map first dimensions))
263 (apply max (map second dimensions))]))
264 #+end_src
266 * Importing and parsing descriptions of eyes.
267 First off, get the children of the "eyes" empty node to find all the
268 eyes the creature has.
269 #+name: eye-node
270 #+begin_src clojure
271 (defvar
272 ^{:arglists '([creature])}
273 eyes
274 (sense-nodes "eyes")
275 "Return the children of the creature's \"eyes\" node.")
276 #+end_src
278 Then, add the camera created by =add-eye!= to the simulation by
279 creating a new viewport.
281 #+name: add-camera
282 #+begin_src clojure
283 (defn add-camera!
284 "Add a camera to the world, calling continuation on every frame
285 produced."
286 [#^Application world camera continuation]
287 (let [width (.getWidth camera)
288 height (.getHeight camera)
289 render-manager (.getRenderManager world)
290 viewport (.createMainView render-manager "eye-view" camera)]
291 (doto viewport
292 (.setClearFlags true true true)
293 (.setBackgroundColor ColorRGBA/Black)
294 (.addProcessor (vision-pipeline continuation))
295 (.attachScene (.getRootNode world)))))
296 #+end_src
299 The eye's continuation function should register the viewport with the
300 simulation the first time it is called, use the CPU to extract the
301 appropriate pixels from the rendered image and weight them by each
302 sensor's sensitivity. I have the option to do this processing in
303 native code for a slight gain in speed. I could also do it in the GPU
304 for a massive gain in speed. =vision-kernel= generates a list of
305 such continuation functions, one for each channel of the eye.
307 #+name: kernel
308 #+begin_src clojure
309 (in-ns 'cortex.vision)
311 (defrecord attached-viewport [vision-fn viewport-fn]
312 clojure.lang.IFn
313 (invoke [this world] (vision-fn world))
314 (applyTo [this args] (apply vision-fn args)))
316 (defn pixel-sense [sensitivity pixel]
317 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
318 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
319 s-b (bit-and 0x0000FF sensitivity)
321 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
322 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
323 p-b (bit-and 0x0000FF pixel)
325 total-sensitivity (* 255 (+ s-r s-g s-b))]
326 (float (/ (+ (* s-r p-r)
327 (* s-g p-g)
328 (* s-b p-b))
329 total-sensitivity))))
331 (defn vision-kernel
332 "Returns a list of functions, each of which will return a color
333 channel's worth of visual information when called inside a running
334 simulation."
335 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
336 (let [retinal-map (retina-sensor-profile eye)
337 camera (add-eye! creature eye)
338 vision-image
339 (atom
340 (BufferedImage. (.getWidth camera)
341 (.getHeight camera)
342 BufferedImage/TYPE_BYTE_BINARY))
343 register-eye!
344 (runonce
345 (fn [world]
346 (add-camera!
347 world camera
348 (let [counter (atom 0)]
349 (fn [r fb bb bi]
350 (if (zero? (rem (swap! counter inc) (inc skip)))
351 (reset! vision-image
352 (BufferedImage! r fb bb bi))))))))]
353 (vec
354 (map
355 (fn [[key image]]
356 (let [whites (white-coordinates image)
357 topology (vec (collapse whites))
358 sensitivity (sensitivity-presets key key)]
359 (attached-viewport.
360 (fn [world]
361 (register-eye! world)
362 (vector
363 topology
364 (vec
365 (for [[x y] whites]
366 (pixel-sense
367 sensitivity
368 (.getRGB @vision-image x y))))))
369 register-eye!)))
370 retinal-map))))
372 (defn gen-fix-display
373 "Create a function to call to restore a simulation's display when it
374 is disrupted by a Viewport."
375 []
376 (runonce
377 (fn [world]
378 (add-camera! world (.getCamera world) no-op))))
379 #+end_src
381 Note that since each of the functions generated by =vision-kernel=
382 shares the same =register-eye!= function, the eye will be registered
383 only once the first time any of the functions from the list returned
384 by =vision-kernel= is called. Each of the functions returned by
385 =vision-kernel= also allows access to the =Viewport= through which
386 it recieves images.
388 The in-game display can be disrupted by all the viewports that the
389 functions greated by =vision-kernel= add. This doesn't affect the
390 simulation or the simulated senses, but can be annoying.
391 =gen-fix-display= restores the in-simulation display.
393 ** The =vision!= function creates sensory probes.
395 All the hard work has been done; all that remains is to apply
396 =vision-kernel= to each eye in the creature and gather the results
397 into one list of functions.
399 #+name: main
400 #+begin_src clojure
401 (defn vision!
402 "Returns a function which returns visual sensory data when called
403 inside a running simulation."
404 [#^Node creature & {skip :skip :or {skip 0}}]
405 (reduce
406 concat
407 (for [eye (eyes creature)]
408 (vision-kernel creature eye))))
409 #+end_src
411 ** Displaying visual data for debugging.
412 # Visualization of Vision. Maybe less alliteration would be better.
413 It's vital to have a visual representation for each sense. Here I use
414 =view-sense= to construct a function that will create a display for
415 visual data.
417 #+name: display
418 #+begin_src clojure
419 (in-ns 'cortex.vision)
421 (defn view-vision
422 "Creates a function which accepts a list of visual sensor-data and
423 displays each element of the list to the screen."
424 []
425 (view-sense
426 (fn
427 [[coords sensor-data]]
428 (let [image (points->image coords)]
429 (dorun
430 (for [i (range (count coords))]
431 (.setRGB image ((coords i) 0) ((coords i) 1)
432 (gray (int (* 255 (sensor-data i)))))))
433 image))))
434 #+end_src
436 * Demonstrations
437 ** Demonstrating the vision pipeline.
439 This is a basic test for the vision system. It only tests the
440 vision-pipeline and does not deal with loading eyes from a blender
441 file. The code creates two videos of the same rotating cube from
442 different angles.
444 #+name: test-1
445 #+begin_src clojure
446 (in-ns 'cortex.test.vision)
448 (defn test-pipeline
449 "Testing vision:
450 Tests the vision system by creating two views of the same rotating
451 object from different angles and displaying both of those views in
452 JFrames.
454 You should see a rotating cube, and two windows,
455 each displaying a different view of the cube."
456 ([] (test-pipeline false))
457 ([record?]
458 (let [candy
459 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
460 (world
461 (doto (Node.)
462 (.attachChild candy))
463 {}
464 (fn [world]
465 (let [cam (.clone (.getCamera world))
466 width (.getWidth cam)
467 height (.getHeight cam)]
468 (add-camera! world cam
469 (comp
470 (view-image
471 (if record?
472 (File. "/home/r/proj/cortex/render/vision/1")))
473 BufferedImage!))
474 (add-camera! world
475 (doto (.clone cam)
476 (.setLocation (Vector3f. -10 0 0))
477 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
478 (comp
479 (view-image
480 (if record?
481 (File. "/home/r/proj/cortex/render/vision/2")))
482 BufferedImage!))
483 ;; This is here to restore the main view
484 ;; after the other views have completed processing
485 (add-camera! world (.getCamera world) no-op)))
486 (fn [world tpf]
487 (.rotate candy (* tpf 0.2) 0 0))))))
488 #+end_src
490 #+begin_html
491 <div class="figure">
492 <video controls="controls" width="755">
493 <source src="../video/spinning-cube.ogg" type="video/ogg"
494 preload="none" poster="../images/aurellem-1280x480.png" />
495 </video>
496 <p>A rotating cube viewed from two different perspectives.</p>
497 </div>
498 #+end_html
500 Creating multiple eyes like this can be used for stereoscopic vision
501 simulation in a single creature or for simulating multiple creatures,
502 each with their own sense of vision.
503 ** Demonstrating eye import and parsing.
505 To the worm from the last post, I add a new node that describes its
506 eyes.
508 #+attr_html: width=755
509 #+caption: The worm with newly added empty nodes describing a single eye.
510 [[../images/worm-with-eye.png]]
512 The node highlighted in yellow is the root level "eyes" node. It has
513 a single child, highlighted in orange, which describes a single
514 eye. This is the "eye" node. It is placed so that the worm will have
515 an eye located in the center of the flat portion of its lower
516 hemispherical section.
518 The two nodes which are not highlighted describe the single joint of
519 the worm.
521 The metadata of the eye-node is:
523 #+begin_src clojure :results verbatim :exports both
524 (cortex.sense/meta-data
525 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
526 #+end_src
528 #+results:
529 : "(let [retina \"Models/test-creature/retina-small.png\"]
530 : {:all retina :red retina :green retina :blue retina})"
532 This is the approximation to the human eye described earlier.
534 #+name: test-2
535 #+begin_src clojure
536 (in-ns 'cortex.test.vision)
538 (defn change-color [obj color]
539 (println-repl obj)
540 (if obj
541 (.setColor (.getMaterial obj) "Color" color)))
543 (defn colored-cannon-ball [color]
544 (comp #(change-color % color)
545 (fire-cannon-ball)))
547 (defn test-worm-vision
548 ([] (test-worm-vision false))
549 ([record?]
550 (let [the-worm (doto (worm)(body!))
551 vision (vision! the-worm)
552 vision-display (view-vision)
553 fix-display (gen-fix-display)
554 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
555 x-axis
556 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
557 :position (Vector3f. 0 -5 0))
558 y-axis
559 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
560 :position (Vector3f. 0 -5 0))
561 z-axis
562 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
563 :position (Vector3f. 0 -5 0))
564 timer (RatchetTimer. 60)]
566 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])
567 (assoc standard-debug-controls
568 "key-r" (colored-cannon-ball ColorRGBA/Red)
569 "key-b" (colored-cannon-ball ColorRGBA/Blue)
570 "key-g" (colored-cannon-ball ColorRGBA/Green))
571 (fn [world]
572 (light-up-everything world)
573 (speed-up world)
574 (.setTimer world timer)
575 (display-dialated-time world timer)
576 ;; add a view from the worm's perspective
577 (if record?
578 (Capture/captureVideo
579 world
580 (File.
581 "/home/r/proj/cortex/render/worm-vision/main-view")))
583 (add-camera!
584 world
585 (add-eye! the-worm
586 (.getChild
587 (.getChild the-worm "eyes") "eye"))
588 (comp
589 (view-image
590 (if record?
591 (File.
592 "/home/r/proj/cortex/render/worm-vision/worm-view")))
593 BufferedImage!))
595 (set-gravity world Vector3f/ZERO))
597 (fn [world _ ]
598 (.setLocalTranslation me (.getLocation (.getCamera world)))
599 (vision-display
600 (map #(% world) vision)
601 (if record? (File. "/home/r/proj/cortex/render/worm-vision")))
602 (fix-display world))))))
603 #+end_src
605 The world consists of the worm and a flat gray floor. I can shoot red,
606 green, blue and white cannonballs at the worm. The worm is initially
607 looking down at the floor, and there is no gravity. My perspective
608 (the Main View), the worm's perspective (Worm View) and the 4 sensor
609 channels that comprise the worm's eye are all saved frame-by-frame to
610 disk.
612 * Demonstration of Vision
613 #+begin_html
614 <div class="figure">
615 <video controls="controls" width="755">
616 <source src="../video/worm-vision.ogg" type="video/ogg"
617 preload="none" poster="../images/aurellem-1280x480.png" />
618 </video>
619 <p>Simulated Vision in a Virtual Environment</p>
620 </div>
621 #+end_html
623 ** Generate the Worm Video from Frames
624 #+name: magick2
625 #+begin_src clojure
626 (ns cortex.video.magick2
627 (:import java.io.File)
628 (:use clojure.contrib.shell-out))
630 (defn images [path]
631 (sort (rest (file-seq (File. path)))))
633 (def base "/home/r/proj/cortex/render/worm-vision/")
635 (defn pics [file]
636 (images (str base file)))
638 (defn combine-images []
639 (let [main-view (pics "main-view")
640 worm-view (pics "worm-view")
641 blue (pics "0")
642 green (pics "1")
643 red (pics "2")
644 gray (pics "3")
645 blender (let [b-pics (pics "blender")]
646 (concat b-pics (repeat 9001 (last b-pics))))
647 background (repeat 9001 (File. (str base "background.png")))
648 targets (map
649 #(File. (str base "out/" (format "%07d.png" %)))
650 (range 0 (count main-view)))]
651 (dorun
652 (pmap
653 (comp
654 (fn [[background main-view worm-view red green blue gray blender target]]
655 (println target)
656 (sh "convert"
657 background
658 main-view "-geometry" "+18+17" "-composite"
659 worm-view "-geometry" "+677+17" "-composite"
660 green "-geometry" "+685+430" "-composite"
661 red "-geometry" "+788+430" "-composite"
662 blue "-geometry" "+894+430" "-composite"
663 gray "-geometry" "+1000+430" "-composite"
664 blender "-geometry" "+0+0" "-composite"
665 target))
666 (fn [& args] (map #(.getCanonicalPath %) args)))
667 background main-view worm-view red green blue gray blender targets))))
668 #+end_src
670 #+begin_src sh :results silent
671 cd /home/r/proj/cortex/render/worm-vision
672 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
673 #+end_src
675 * Onward!
676 - As a neat bonus, this idea behind simulated vision also enables one
677 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
678 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
679 #+appendix
681 * Headers
683 #+name: vision-header
684 #+begin_src clojure
685 (ns cortex.vision
686 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
687 eyes from different positions to observe the same world, and pass
688 the observed data to any arbitray function. Automatically reads
689 eye-nodes from specially prepared blender files and instantiates
690 them in the world as actual eyes."
691 {:author "Robert McIntyre"}
692 (:use (cortex world sense util))
693 (:use clojure.contrib.def)
694 (:import com.jme3.post.SceneProcessor)
695 (:import (com.jme3.util BufferUtils Screenshots))
696 (:import java.nio.ByteBuffer)
697 (:import java.awt.image.BufferedImage)
698 (:import (com.jme3.renderer ViewPort Camera))
699 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
700 (:import com.jme3.renderer.Renderer)
701 (:import com.jme3.app.Application)
702 (:import com.jme3.texture.FrameBuffer)
703 (:import (com.jme3.scene Node Spatial)))
704 #+end_src
706 #+name: test-header
707 #+begin_src clojure
708 (ns cortex.test.vision
709 (:use (cortex world sense util body vision))
710 (:use cortex.test.body)
711 (:import java.awt.image.BufferedImage)
712 (:import javax.swing.JPanel)
713 (:import javax.swing.SwingUtilities)
714 (:import java.awt.Dimension)
715 (:import javax.swing.JFrame)
716 (:import com.jme3.math.ColorRGBA)
717 (:import com.jme3.scene.Node)
718 (:import com.jme3.math.Vector3f)
719 (:import java.io.File)
720 (:import (com.aurellem.capture Capture RatchetTimer)))
721 #+end_src
722 * Source Listing
723 - [[../src/cortex/vision.clj][cortex.vision]]
724 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
725 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
726 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
727 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
728 - [[http://hg.bortreb.com ][source-repository]]
731 * Next
732 I find some [[./hearing.org][ears]] for the creature while exploring the guts of
733 jMonkeyEngine's sound system.
735 * COMMENT Generate Source
736 #+begin_src clojure :tangle ../src/cortex/vision.clj
737 <<vision-header>>
738 <<pipeline-1>>
739 <<pipeline-2>>
740 <<retina>>
741 <<add-eye>>
742 <<sensitivity>>
743 <<eye-node>>
744 <<add-camera>>
745 <<kernel>>
746 <<main>>
747 <<display>>
748 #+end_src
750 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
751 <<test-header>>
752 <<test-1>>
753 <<test-2>>
754 #+end_src
756 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
757 <<magick2>>
758 #+end_src