view org/vision.org @ 234:712bd7e5b148

saving progress
author Robert McIntyre <rlm@mit.edu>
date Sun, 12 Feb 2012 09:05:47 -0700
parents 5f14fd7b1288
children 3c9724c8d86b
line wrap: on
line source
1 #+title: Simulated Sense of Sight
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both
10 * Vision
12 Vision is one of the most important senses for humans, so I need to
13 build a simulated sense of vision for my AI. I will do this with
14 simulated eyes. Each eye can be independely moved and should see its
15 own version of the world depending on where it is.
17 Making these simulated eyes a reality is simple bacause jMonkeyEngine
18 already conatains extensive support for multiple views of the same 3D
19 simulated world. The reason jMonkeyEngine has this support is because
20 the support is necessary to create games with split-screen
21 views. Multiple views are also used to create efficient
22 pseudo-reflections by rendering the scene from a certain perspective
23 and then projecting it back onto a surface in the 3D world.
25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
26 [[../images/goldeneye-4-player.png]]
28 * Brief Description of jMonkeyEngine's Rendering Pipeline
30 jMonkeyEngine allows you to create a =ViewPort=, which represents a
31 view of the simulated world. You can create as many of these as you
32 want. Every frame, the =RenderManager= iterates through each
33 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
34 is a =FrameBuffer= which represents the rendered image in the GPU.
36 Each =ViewPort= can have any number of attached =SceneProcessor=
37 objects, which are called every time a new frame is rendered. A
38 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do
39 whatever it wants to the data. Often this consists of invoking GPU
40 specific operations on the rendered image. The =SceneProcessor= can
41 also copy the GPU image data to RAM and process it with the CPU.
43 * The Vision Pipeline
45 Each eye in the simulated creature needs it's own =ViewPort= so that
46 it can see the world from its own perspective. To this =ViewPort=, I
47 add a =SceneProcessor= that feeds the visual data to any arbitray
48 continuation function for further processing. That continuation
49 function may perform both CPU and GPU operations on the data. To make
50 this easy for the continuation function, the =SceneProcessor=
51 maintains appropriatly sized buffers in RAM to hold the data. It does
52 not do any copying from the GPU to the CPU itself because it is a slow
53 operation.
55 #+name: pipeline-1
56 #+begin_src clojure
57 (defn vision-pipeline
58 "Create a SceneProcessor object which wraps a vision processing
59 continuation function. The continuation is a function that takes
60 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
61 each of which has already been appropiately sized."
62 [continuation]
63 (let [byte-buffer (atom nil)
64 renderer (atom nil)
65 image (atom nil)]
66 (proxy [SceneProcessor] []
67 (initialize
68 [renderManager viewPort]
69 (let [cam (.getCamera viewPort)
70 width (.getWidth cam)
71 height (.getHeight cam)]
72 (reset! renderer (.getRenderer renderManager))
73 (reset! byte-buffer
74 (BufferUtils/createByteBuffer
75 (* width height 4)))
76 (reset! image (BufferedImage.
77 width height
78 BufferedImage/TYPE_4BYTE_ABGR))))
79 (isInitialized [] (not (nil? @byte-buffer)))
80 (reshape [_ _ _])
81 (preFrame [_])
82 (postQueue [_])
83 (postFrame
84 [#^FrameBuffer fb]
85 (.clear @byte-buffer)
86 (continuation @renderer fb @byte-buffer @image))
87 (cleanup []))))
88 #+end_src
90 The continuation function given to =(vision-pipeline)= above will be
91 given a =Renderer= and three containers for image data. The
92 =FrameBuffer= references the GPU image data, but the pixel data can
93 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
94 are initially "empty" but are sized to hold the data in the
95 =FrameBuffer=. I call transfering the GPU image data to the CPU
96 structures "mixing" the image data. I have provided three functions to
97 do this mixing.
99 #+name: pipeline-2
100 #+begin_src clojure
101 (defn frameBuffer->byteBuffer!
102 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
103 the CPU (ByteBuffer)."
104 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
105 (.readFrameBuffer r fb bb) bb)
107 (defn byteBuffer->bufferedImage!
108 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
109 style ABGR image data and place it in BufferedImage bi."
110 [#^ByteBuffer bb #^BufferedImage bi]
111 (Screenshots/convertScreenShot bb bi) bi)
113 (defn BufferedImage!
114 "Continuation which will grab the buffered image from the materials
115 provided by (vision-pipeline)."
116 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
117 (byteBuffer->bufferedImage!
118 (frameBuffer->byteBuffer! r fb bb) bi))
119 #+end_src
121 Note that it is possible to write vision processing algorithms
122 entirely in terms of =BufferedImage= inputs. Just compose that
123 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
124 processing algorithm that is entirely hosted on the GPU does not have
125 to pay for this convienence.
127 * COMMENT asdasd
129 (vision creature) will take an optional :skip argument which will
130 inform the continuations in scene processor to skip the given
131 number of cycles 0 means that no cycles will be skipped.
133 (vision creature) will return [init-functions sensor-functions].
134 The init-functions are each single-arg functions that take the
135 world and register the cameras and must each be called before the
136 corresponding sensor-functions. Each init-function returns the
137 viewport for that eye which can be manipulated, saved, etc. Each
138 sensor-function is a thunk and will return data in the same
139 format as the tactile-sensor functions the structure is
140 [topology, sensor-data]. Internally, these sensor-functions
141 maintain a reference to sensor-data which is periodically updated
142 by the continuation function established by its init-function.
143 They can be queried every cycle, but their information may not
144 necessairly be different every cycle.
146 * Physical Eyes
148 The vision pipeline described above handles the flow of rendered
149 images. Now, we need simulated eyes to serve as the source of these
150 images.
152 An eye is described in blender in the same way as a joint. They are
153 zero dimensional empty objects with no geometry whose local coordinate
154 system determines the orientation of the resulting eye. All eyes are
155 childern of a parent node named "eyes" just as all joints have a
156 parent named "joints". An eye binds to the nearest physical object
157 with =(bind-sense=).
159 #+name: add-eye
160 #+begin_src clojure
161 (in-ns 'cortex.vision)
163 (defn add-eye!
164 "Create a Camera centered on the current position of 'eye which
165 follows the closest physical node in 'creature and sends visual
166 data to 'continuation. The camera will point in the X direction and
167 use the Z vector as up as determined by the rotation of these
168 vectors in blender coordinate space. Use XZY rotation for the node
169 in blender."
170 [#^Node creature #^Spatial eye]
171 (let [target (closest-node creature eye)
172 [cam-width cam-height] (eye-dimensions eye)
173 cam (Camera. cam-width cam-height)
174 rot (.getWorldRotation eye)]
175 (.setLocation cam (.getWorldTranslation eye))
176 (.lookAtDirection
177 cam ; this part is not a mistake and
178 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
179 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
180 (.setFrustumPerspective
181 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
182 (bind-sense target cam) cam))
183 #+end_src
185 Here, the camera is created based on metadata on the eye-node and
186 attached to the nearest physical object with =(bind-sense)=
189 ** The Retina
191 An eye is a surface (the retina) which contains many discrete sensors
192 to detect light. These sensors have can have different light-sensing
193 properties. In humans, each discrete sensor is sensitive to red,
194 blue, green, or gray. These different types of sensors can have
195 different spatial distributions along the retina. In humans, there is
196 a fovea in the center of the retina which has a very high density of
197 color sensors, and a blind spot which has no sensors at all. Sensor
198 density decreases in proportion to distance from the fovea.
200 I want to be able to model any retinal configuration, so my eye-nodes
201 in blender contain metadata pointing to images that describe the
202 percise position of the individual sensors using white pixels. The
203 meta-data also describes the percise sensitivity to light that the
204 sensors described in the image have. An eye can contain any number of
205 these images. For example, the metadata for an eye might look like
206 this:
208 #+begin_src clojure
209 {0xFF0000 "Models/test-creature/retina-small.png"}
210 #+end_src
212 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
213 [[../assets/Models/test-creature/retina-small.png]]
215 Together, the number 0xFF0000 and the image image above describe the
216 placement of red-sensitive sensory elements.
218 Meta-data to very crudely approximate a human eye might be something
219 like this:
221 #+begin_src clojure
222 (let [retinal-profile "Models/test-creature/retina-small.png"]
223 {0xFF0000 retinal-profile
224 0x00FF00 retinal-profile
225 0x0000FF retinal-profile
226 0xFFFFFF retinal-profile})
227 #+end_src
229 The numbers that serve as keys in the map determine a sensor's
230 relative sensitivity to the channels red, green, and blue. These
231 sensitivity values are packed into an integer in the order =|_|R|G|B|=
232 in 8-bit fields. The RGB values of a pixel in the image are added
233 together with these sensitivities as linear weights. Therfore,
234 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
235 all colors equally (gray).
237 For convienence I've defined a few symbols for the more common
238 sensitivity values.
240 #+name: sensitivity
241 #+begin_src clojure
242 (defvar sensitivity-presets
243 {:all 0xFFFFFF
244 :red 0xFF0000
245 :blue 0x0000FF
246 :green 0x00FF00}
247 "Retinal sensitivity presets for sensors that extract one channel
248 (:red :blue :green) or average all channels (:all)")
249 #+end_src
251 ** Metadata Processing
253 =(retina-sensor-profile)= extracts a map from the eye-node in the same
254 format as the example maps above. =(eye-dimensions)= finds the
255 dimensions of the smallest image required to contain all the retinal
256 sensor maps.
258 #+name: retina
259 #+begin_src clojure
260 (defn retina-sensor-profile
261 "Return a map of pixel sensitivity numbers to BufferedImages
262 describing the distribution of light-sensitive components of this
263 eye. :red, :green, :blue, :gray are already defined as extracting
264 the red, green, blue, and average components respectively."
265 [#^Spatial eye]
266 (if-let [eye-map (meta-data eye "eye")]
267 (map-vals
268 load-image
269 (eval (read-string eye-map)))))
271 (defn eye-dimensions
272 "Returns [width, height] determined by the metadata of the eye."
273 [#^Spatial eye]
274 (let [dimensions
275 (map #(vector (.getWidth %) (.getHeight %))
276 (vals (retina-sensor-profile eye)))]
277 [(apply max (map first dimensions))
278 (apply max (map second dimensions))]))
279 #+end_src
281 * Eye Creation
282 First off, get the children of the "eyes" empty node to find all the
283 eyes the creature has.
284 #+name: eye-node
285 #+begin_src clojure
286 (defvar
287 ^{:arglists '([creature])}
288 eyes
289 (sense-nodes "eyes")
290 "Return the children of the creature's \"eyes\" node.")
291 #+end_src
293 Then, add the camera created by =(add-eye!)= to the simulation by
294 creating a new viewport.
296 #+name: add-camera
297 #+begin_src clojure
298 (defn add-camera!
299 "Add a camera to the world, calling continuation on every frame
300 produced."
301 [#^Application world camera continuation]
302 (let [width (.getWidth camera)
303 height (.getHeight camera)
304 render-manager (.getRenderManager world)
305 viewport (.createMainView render-manager "eye-view" camera)]
306 (doto viewport
307 (.setClearFlags true true true)
308 (.setBackgroundColor ColorRGBA/Black)
309 (.addProcessor (vision-pipeline continuation))
310 (.attachScene (.getRootNode world)))))
311 #+end_src
314 The eye's continuation function should register the viewport with the
315 simulation the first time it is called, use the CPU to extract the
316 appropriate pixels from the rendered image and weight them by each
317 sensor's sensitivity. I have the option to do this processing in
318 native code for a slight gain in speed. I could also do it in the GPU
319 for a massive gain in speed. =(vision-kernel)= generates a list of
320 such continuation functions, one for each channel of the eye.
322 #+name: kernel
323 #+begin_src clojure
324 (in-ns 'cortex.vision)
326 (defrecord attached-viewport [vision-fn viewport-fn]
327 clojure.lang.IFn
328 (invoke [this world] (vision-fn world))
329 (applyTo [this args] (apply vision-fn args)))
331 (defn pixel-sense [sensitivity pixel]
332 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
333 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
334 s-b (bit-and 0x0000FF sensitivity)
336 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
337 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
338 p-b (bit-and 0x0000FF pixel)
340 total-sensitivity (* 255 (+ s-r s-g s-b))]
341 (float (/ (+ (* s-r p-r)
342 (* s-g p-g)
343 (* s-b p-b))
344 total-sensitivity))))
346 (defn vision-kernel
347 "Returns a list of functions, each of which will return a color
348 channel's worth of visual information when called inside a running
349 simulation."
350 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
351 (let [retinal-map (retina-sensor-profile eye)
352 camera (add-eye! creature eye)
353 vision-image
354 (atom
355 (BufferedImage. (.getWidth camera)
356 (.getHeight camera)
357 BufferedImage/TYPE_BYTE_BINARY))
358 register-eye!
359 (runonce
360 (fn [world]
361 (add-camera!
362 world camera
363 (let [counter (atom 0)]
364 (fn [r fb bb bi]
365 (if (zero? (rem (swap! counter inc) (inc skip)))
366 (reset! vision-image
367 (BufferedImage! r fb bb bi))))))))]
368 (vec
369 (map
370 (fn [[key image]]
371 (let [whites (white-coordinates image)
372 topology (vec (collapse whites))
373 sensitivity (sensitivity-presets key key)]
374 (attached-viewport.
375 (fn [world]
376 (register-eye! world)
377 (vector
378 topology
379 (vec
380 (for [[x y] whites]
381 (pixel-sense
382 sensitivity
383 (.getRGB @vision-image x y))))))
384 register-eye!)))
385 retinal-map))))
387 (defn gen-fix-display
388 "Create a function to call to restore a simulation's display when it
389 is disrupted by a Viewport."
390 []
391 (runonce
392 (fn [world]
393 (add-camera! world (.getCamera world) no-op))))
394 #+end_src
396 Note that since each of the functions generated by =(vision-kernel)=
397 shares the same =(register-eye!)= function, the eye will be registered
398 only once the first time any of the functions from the list returned
399 by =(vision-kernel)= is called. Each of the functions returned by
400 =(vision-kernel)= also allows access to the =Viewport= through which
401 it recieves images.
403 The in-game display can be disrupted by all the viewports that the
404 functions greated by =(vision-kernel)= add. This doesn't affect the
405 simulation or the simulated senses, but can be annoying.
406 =(gen-fix-display)= restores the in-simulation display.
408 ** Vision!
410 All the hard work has been done; all that remains is to apply
411 =(vision-kernel)= to each eye in the creature and gather the results
412 into one list of functions.
414 #+name: main
415 #+begin_src clojure
416 (defn vision!
417 "Returns a function which returns visual sensory data when called
418 inside a running simulation."
419 [#^Node creature & {skip :skip :or {skip 0}}]
420 (reduce
421 concat
422 (for [eye (eyes creature)]
423 (vision-kernel creature eye))))
424 #+end_src
426 ** Visualization of Vision
428 It's vital to have a visual representation for each sense. Here I use
429 =(view-sense)= to construct a function that will create a display for
430 visual data.
432 #+name: display
433 #+begin_src clojure
434 (in-ns 'cortex.vision)
436 (defn view-vision
437 "Creates a function which accepts a list of visual sensor-data and
438 displays each element of the list to the screen."
439 []
440 (view-sense
441 (fn
442 [[coords sensor-data]]
443 (let [image (points->image coords)]
444 (dorun
445 (for [i (range (count coords))]
446 (.setRGB image ((coords i) 0) ((coords i) 1)
447 (gray (int (* 255 (sensor-data i)))))))
448 image))))
449 #+end_src
451 * Tests
453 ** Basic Test
455 This is a basic test for the vision system. It only tests the
456 vision-pipeline and does not deal with loadig eyes from a blender
457 file. The code creates two videos of the same rotating cube from
458 different angles.
460 #+name: test-1
461 #+begin_src clojure
462 (in-ns 'cortex.test.vision)
464 (defn test-pipeline
465 "Testing vision:
466 Tests the vision system by creating two views of the same rotating
467 object from different angles and displaying both of those views in
468 JFrames.
470 You should see a rotating cube, and two windows,
471 each displaying a different view of the cube."
472 []
473 (let [candy
474 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
475 (world
476 (doto (Node.)
477 (.attachChild candy))
478 {}
479 (fn [world]
480 (let [cam (.clone (.getCamera world))
481 width (.getWidth cam)
482 height (.getHeight cam)]
483 (add-camera! world cam
484 (comp
485 (view-image
486 (File. "/home/r/proj/cortex/render/vision/1"))
487 BufferedImage!))
488 (add-camera! world
489 (doto (.clone cam)
490 (.setLocation (Vector3f. -10 0 0))
491 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
492 (comp
493 (view-image
494 (File. "/home/r/proj/cortex/render/vision/2"))
495 BufferedImage!))
496 ;; This is here to restore the main view
497 ;; after the other views have completed processing
498 (add-camera! world (.getCamera world) no-op)))
499 (fn [world tpf]
500 (.rotate candy (* tpf 0.2) 0 0)))))
501 #+end_src
503 #+begin_html
504 <div class="figure">
505 <video controls="controls" width="755">
506 <source src="../video/spinning-cube.ogg" type="video/ogg"
507 preload="none" poster="../images/aurellem-1280x480.png" />
508 </video>
509 <p>A rotating cube viewed from two different perspectives.</p>
510 </div>
511 #+end_html
513 Creating multiple eyes like this can be used for stereoscopic vision
514 simulation in a single creature or for simulating multiple creatures,
515 each with their own sense of vision.
517 ** Adding Vision to the Worm
519 To the worm from the last post, I add a new node that describes its
520 eyes.
522 #+attr_html: width=755
523 #+caption: The worm with newly added empty nodes describing a single eye.
524 [[../images/worm-with-eye.png]]
526 The node highlighted in yellow is the root level "eyes" node. It has
527 a single child, highlighted in orange, which describes a single
528 eye. This is the "eye" node. It is placed so that the worm will have
529 an eye located in the center of the flat portion of its lower
530 hemispherical section.
532 The two nodes which are not highlighted describe the single joint of
533 the worm.
535 The metadata of the eye-node is:
537 #+begin_src clojure :results verbatim :exports both
538 (cortex.sense/meta-data
539 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
540 #+end_src
542 #+results:
543 : "(let [retina \"Models/test-creature/retina-small.png\"]
544 : {:all retina :red retina :green retina :blue retina})"
546 This is the approximation to the human eye described earlier.
548 #+name: test-2
549 #+begin_src clojure
550 (in-ns 'cortex.test.vision)
552 (defn change-color [obj color]
553 (println-repl obj)
554 (if obj
555 (.setColor (.getMaterial obj) "Color" color)))
557 (defn colored-cannon-ball [color]
558 (comp #(change-color % color)
559 (fire-cannon-ball)))
561 (defn test-worm-vision []
562 (let [the-worm (doto (worm)(body!))
563 vision (vision! the-worm)
564 vision-display (view-vision)
565 fix-display (gen-fix-display)
566 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
567 x-axis
568 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
569 :position (Vector3f. 0 -5 0))
570 y-axis
571 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
572 :position (Vector3f. 0 -5 0))
573 z-axis
574 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
575 :position (Vector3f. 0 -5 0))
576 timer (RatchetTimer. 60)]
578 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])
579 (assoc standard-debug-controls
580 "key-r" (colored-cannon-ball ColorRGBA/Red)
581 "key-b" (colored-cannon-ball ColorRGBA/Blue)
582 "key-g" (colored-cannon-ball ColorRGBA/Green))
583 (fn [world]
584 (light-up-everything world)
585 (speed-up world)
586 (.setTimer world timer)
587 (display-dialated-time world timer)
588 ;; add a view from the worm's perspective
589 (add-camera!
590 world
591 (add-eye! the-worm
592 (.getChild
593 (.getChild the-worm "eyes") "eye"))
594 (comp
595 (view-image
596 (File. "/home/r/proj/cortex/render/worm-vision/worm-view"))
597 BufferedImage!))
598 (set-gravity world Vector3f/ZERO)
599 (try
600 (Capture/captureVideo
601 world
602 (File. "/home/r/proj/cortex/render/worm-vision/main-view"))))
604 (fn [world _ ]
605 (.setLocalTranslation me (.getLocation (.getCamera world)))
606 (vision-display
607 (map #(% world) vision)
608 (File. "/home/r/proj/cortex/render/worm-vision"))
609 (fix-display world)))))
610 #+end_src
612 The world consists of the worm and a flat gray floor. I can shoot red,
613 green, blue and white cannonballs at the worm. The worm is initially
614 looking down at the floor, and there is no gravity. My perspective
615 (the Main View), the worm's perspective (Worm View) and the 4 sensor
616 channels that comprise the worm's eye are all saved frame-by-frame to
617 disk.
619 * Demonstration of Vision
620 #+begin_html
621 <div class="figure">
622 <video controls="controls" width="755">
623 <source src="../video/worm-vision.ogg" type="video/ogg"
624 preload="none" poster="../images/aurellem-1280x480.png" />
625 </video>
626 <p>Simulated Vision in a Virtual Environment</p>
627 </div>
628 #+end_html
630 ** Generate the Worm Video from Frames
631 #+name: magick2
632 #+begin_src clojure
633 (ns cortex.video.magick2
634 (:import java.io.File)
635 (:use clojure.contrib.shell-out))
637 (defn images [path]
638 (sort (rest (file-seq (File. path)))))
640 (def base "/home/r/proj/cortex/render/worm-vision/")
642 (defn pics [file]
643 (images (str base file)))
645 (defn combine-images []
646 (let [main-view (pics "main-view")
647 worm-view (pics "worm-view")
648 blue (pics "0")
649 green (pics "1")
650 red (pics "2")
651 gray (pics "3")
652 blender (let [b-pics (pics "blender")]
653 (concat b-pics (repeat 9001 (last b-pics))))
654 background (repeat 9001 (File. (str base "background.png")))
655 targets (map
656 #(File. (str base "out/" (format "%07d.png" %)))
657 (range 0 (count main-view)))]
658 (dorun
659 (pmap
660 (comp
661 (fn [[background main-view worm-view red green blue gray blender target]]
662 (println target)
663 (sh "convert"
664 background
665 main-view "-geometry" "+18+17" "-composite"
666 worm-view "-geometry" "+677+17" "-composite"
667 green "-geometry" "+685+430" "-composite"
668 red "-geometry" "+788+430" "-composite"
669 blue "-geometry" "+894+430" "-composite"
670 gray "-geometry" "+1000+430" "-composite"
671 blender "-geometry" "+0+0" "-composite"
672 target))
673 (fn [& args] (map #(.getCanonicalPath %) args)))
674 background main-view worm-view red green blue gray blender targets))))
675 #+end_src
677 #+begin_src sh :results silent
678 cd /home/r/proj/cortex/render/worm-vision
679 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
680 #+end_src
682 * Headers
684 #+name: vision-header
685 #+begin_src clojure
686 (ns cortex.vision
687 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
688 eyes from different positions to observe the same world, and pass
689 the observed data to any arbitray function. Automatically reads
690 eye-nodes from specially prepared blender files and instantiates
691 them in the world as actual eyes."
692 {:author "Robert McIntyre"}
693 (:use (cortex world sense util))
694 (:use clojure.contrib.def)
695 (:import com.jme3.post.SceneProcessor)
696 (:import (com.jme3.util Buffe rUtils Screenshots))
697 (:import java.nio.ByteBuffer)
698 (:import java.awt.image.BufferedImage)
699 (:import (com.jme3.renderer ViewPort Camera))
700 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
701 (:import com.jme3.renderer.Renderer)
702 (:import com.jme3.app.Application)
703 (:import com.jme3.texture.FrameBuffer)
704 (:import (com.jme3.scene Node Spatial)))
705 #+end_src
707 #+name: test-header
708 #+begin_src clojure
709 (ns cortex.test.vision
710 (:use (cortex world sense util body vision))
711 (:use cortex.test.body)
712 (:import java.awt.image.BufferedImage)
713 (:import javax.swing.JPanel)
714 (:import javax.swing.SwingUtilities)
715 (:import java.awt.Dimension)
716 (:import javax.swing.JFrame)
717 (:import com.jme3.math.ColorRGBA)
718 (:import com.jme3.scene.Node)
719 (:import com.jme3.math.Vector3f)
720 (:import java.io.File)
721 (:import (com.aurellem.capture Capture RatchetTimer)))
722 #+end_src
724 * Onward!
725 - As a neat bonus, this idea behind simulated vision also enables one
726 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
727 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
729 * Source Listing
730 - [[../src/cortex/vision.clj][cortex.vision]]
731 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
732 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
733 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
734 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
735 - [[http://hg.bortreb.com ][source-repository]]
739 * COMMENT Generate Source
740 #+begin_src clojure :tangle ../src/cortex/vision.clj
741 <<vision-header>>
742 <<pipeline-1>>
743 <<pipeline-2>>
744 <<retina>>
745 <<add-eye>>
746 <<sensitivity>>
747 <<eye-node>>
748 <<add-camera>>
749 <<kernel>>
750 <<main>>
751 <<display>>
752 #+end_src
754 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
755 <<test-header>>
756 <<test-1>>
757 <<test-2>>
758 #+end_src
760 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
761 <<magick2>>
762 #+end_src