view org/vision.org @ 338:d37ccb6c888f

determined that the laptop cannot support arbitray dimensions when creating cameras. will need to use desktop from here on out when doing actual simulations.
author Robert McIntyre <rlm@mit.edu>
date Fri, 20 Jul 2012 16:40:25 -0500
parents fdc98824d69b
children 4f5a5d5f1613
line wrap: on
line source
1 #+title: Simulated Sense of Sight
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both
10 * JMonkeyEngine natively supports multiple views of the same world.
12 Vision is one of the most important senses for humans, so I need to
13 build a simulated sense of vision for my AI. I will do this with
14 simulated eyes. Each eye can be independently moved and should see its
15 own version of the world depending on where it is.
17 Making these simulated eyes a reality is simple because jMonkeyEngine
18 already contains extensive support for multiple views of the same 3D
19 simulated world. The reason jMonkeyEngine has this support is because
20 the support is necessary to create games with split-screen
21 views. Multiple views are also used to create efficient
22 pseudo-reflections by rendering the scene from a certain perspective
23 and then projecting it back onto a surface in the 3D world.
25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
26 [[../images/goldeneye-4-player.png]]
28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.
29 # =ViewPorts= are cameras; =RenderManger= takes snapshots each frame.
30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline
32 jMonkeyEngine allows you to create a =ViewPort=, which represents a
33 view of the simulated world. You can create as many of these as you
34 want. Every frame, the =RenderManager= iterates through each
35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
36 is a =FrameBuffer= which represents the rendered image in the GPU.
38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =RenderManager= records a snapshot of what each view is currently seeing; these snapshots are =FrameBuffer= objects.
39 #+ATTR_HTML: width="400"
40 [[../images/diagram_rendermanager2.png]]
42 Each =ViewPort= can have any number of attached =SceneProcessor=
43 objects, which are called every time a new frame is rendered. A
44 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
45 whatever it wants to the data. Often this consists of invoking GPU
46 specific operations on the rendered image. The =SceneProcessor= can
47 also copy the GPU image data to RAM and process it with the CPU.
49 ** From Views to Vision
50 # Appropriating Views for Vision.
52 Each eye in the simulated creature needs its own =ViewPort= so that
53 it can see the world from its own perspective. To this =ViewPort=, I
54 add a =SceneProcessor= that feeds the visual data to any arbitrary
55 continuation function for further processing. That continuation
56 function may perform both CPU and GPU operations on the data. To make
57 this easy for the continuation function, the =SceneProcessor=
58 maintains appropriately sized buffers in RAM to hold the data. It does
59 not do any copying from the GPU to the CPU itself because it is a slow
60 operation.
62 #+name: pipeline-1
63 #+begin_src clojure
64 (defn vision-pipeline
65 "Create a SceneProcessor object which wraps a vision processing
66 continuation function. The continuation is a function that takes
67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
68 each of which has already been appropriately sized."
69 [continuation]
70 (let [byte-buffer (atom nil)
71 renderer (atom nil)
72 image (atom nil)]
73 (proxy [SceneProcessor] []
74 (initialize
75 [renderManager viewPort]
76 (let [cam (.getCamera viewPort)
77 width (.getWidth cam)
78 height (.getHeight cam)]
79 (reset! renderer (.getRenderer renderManager))
80 (reset! byte-buffer
81 (BufferUtils/createByteBuffer
82 (* width height 4)))
83 (reset! image (BufferedImage.
84 width height
85 BufferedImage/TYPE_4BYTE_ABGR))))
86 (isInitialized [] (not (nil? @byte-buffer)))
87 (reshape [_ _ _])
88 (preFrame [_])
89 (postQueue [_])
90 (postFrame
91 [#^FrameBuffer fb]
92 (.clear @byte-buffer)
93 (continuation @renderer fb @byte-buffer @image))
94 (cleanup []))))
95 #+end_src
97 The continuation function given to =vision-pipeline= above will be
98 given a =Renderer= and three containers for image data. The
99 =FrameBuffer= references the GPU image data, but the pixel data can
100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
101 are initially "empty" but are sized to hold the data in the
102 =FrameBuffer=. I call transferring the GPU image data to the CPU
103 structures "mixing" the image data. I have provided three functions to
104 do this mixing.
106 #+name: pipeline-2
107 #+begin_src clojure
108 (defn frameBuffer->byteBuffer!
109 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
110 the CPU (ByteBuffer)."
111 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
112 (.readFrameBuffer r fb bb) bb)
114 (defn byteBuffer->bufferedImage!
115 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
116 style ABGR image data and place it in BufferedImage bi."
117 [#^ByteBuffer bb #^BufferedImage bi]
118 (Screenshots/convertScreenShot bb bi) bi)
120 (defn BufferedImage!
121 "Continuation which will grab the buffered image from the materials
122 provided by (vision-pipeline)."
123 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
124 (byteBuffer->bufferedImage!
125 (frameBuffer->byteBuffer! r fb bb) bi))
126 #+end_src
128 Note that it is possible to write vision processing algorithms
129 entirely in terms of =BufferedImage= inputs. Just compose that
130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision
131 processing algorithm that is entirely hosted on the GPU does not have
132 to pay for this convenience.
134 * Optical sensor arrays are described with images and referenced with metadata
135 The vision pipeline described above handles the flow of rendered
136 images. Now, we need simulated eyes to serve as the source of these
137 images.
139 An eye is described in blender in the same way as a joint. They are
140 zero dimensional empty objects with no geometry whose local coordinate
141 system determines the orientation of the resulting eye. All eyes are
142 children of a parent node named "eyes" just as all joints have a
143 parent named "joints". An eye binds to the nearest physical object
144 with =bind-sense=.
146 #+name: add-eye
147 #+begin_src clojure
148 (in-ns 'cortex.vision)
150 (defn add-eye!
151 "Create a Camera centered on the current position of 'eye which
152 follows the closest physical node in 'creature. The camera will
153 point in the X direction and use the Z vector as up as determined
154 by the rotation of these vectors in blender coordinate space. Use
155 XZY rotation for the node in blender."
156 [#^Node creature #^Spatial eye]
157 (let [target (closest-node creature eye)
158 [cam-width cam-height]
159 ;;[640 480] ;; graphics card on laptop doesn't support
160 ;; arbitray dimensions.
161 (eye-dimensions eye)
162 cam (Camera. cam-width cam-height)
163 rot (.getWorldRotation eye)]
164 (.setLocation cam (.getWorldTranslation eye))
165 (.lookAtDirection
166 cam ; this part is not a mistake and
167 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
168 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
169 (.setFrustumPerspective
170 cam (float 45)
171 (float (/ (.getWidth cam) (.getHeight cam)))
172 (float 1)
173 (float 1000))
174 (bind-sense target cam) cam))
175 #+end_src
177 #+results: add-eye
178 : #'cortex.vision/add-eye!
180 Here, the camera is created based on metadata on the eye-node and
181 attached to the nearest physical object with =bind-sense=
182 ** The Retina
184 An eye is a surface (the retina) which contains many discrete sensors
185 to detect light. These sensors have can have different light-sensing
186 properties. In humans, each discrete sensor is sensitive to red,
187 blue, green, or gray. These different types of sensors can have
188 different spatial distributions along the retina. In humans, there is
189 a fovea in the center of the retina which has a very high density of
190 color sensors, and a blind spot which has no sensors at all. Sensor
191 density decreases in proportion to distance from the fovea.
193 I want to be able to model any retinal configuration, so my eye-nodes
194 in blender contain metadata pointing to images that describe the
195 precise position of the individual sensors using white pixels. The
196 meta-data also describes the precise sensitivity to light that the
197 sensors described in the image have. An eye can contain any number of
198 these images. For example, the metadata for an eye might look like
199 this:
201 #+begin_src clojure
202 {0xFF0000 "Models/test-creature/retina-small.png"}
203 #+end_src
205 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
206 [[../assets/Models/test-creature/retina-small.png]]
208 Together, the number 0xFF0000 and the image image above describe the
209 placement of red-sensitive sensory elements.
211 Meta-data to very crudely approximate a human eye might be something
212 like this:
214 #+begin_src clojure
215 (let [retinal-profile "Models/test-creature/retina-small.png"]
216 {0xFF0000 retinal-profile
217 0x00FF00 retinal-profile
218 0x0000FF retinal-profile
219 0xFFFFFF retinal-profile})
220 #+end_src
222 The numbers that serve as keys in the map determine a sensor's
223 relative sensitivity to the channels red, green, and blue. These
224 sensitivity values are packed into an integer in the order =|_|R|G|B|=
225 in 8-bit fields. The RGB values of a pixel in the image are added
226 together with these sensitivities as linear weights. Therefore,
227 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
228 all colors equally (gray).
230 For convenience I've defined a few symbols for the more common
231 sensitivity values.
233 #+name: sensitivity
234 #+begin_src clojure
235 (def sensitivity-presets
236 "Retinal sensitivity presets for sensors that extract one channel
237 (:red :blue :green) or average all channels (:all)"
238 {:all 0xFFFFFF
239 :red 0xFF0000
240 :blue 0x0000FF
241 :green 0x00FF00})
242 #+end_src
244 ** Metadata Processing
246 =retina-sensor-profile= extracts a map from the eye-node in the same
247 format as the example maps above. =eye-dimensions= finds the
248 dimensions of the smallest image required to contain all the retinal
249 sensor maps.
251 #+name: retina
252 #+begin_src clojure
253 (defn retina-sensor-profile
254 "Return a map of pixel sensitivity numbers to BufferedImages
255 describing the distribution of light-sensitive components of this
256 eye. :red, :green, :blue, :gray are already defined as extracting
257 the red, green, blue, and average components respectively."
258 [#^Spatial eye]
259 (if-let [eye-map (meta-data eye "eye")]
260 (map-vals
261 load-image
262 (eval (read-string eye-map)))))
264 (defn eye-dimensions
265 "Returns [width, height] determined by the metadata of the eye."
266 [#^Spatial eye]
267 (let [dimensions
268 (map #(vector (.getWidth %) (.getHeight %))
269 (vals (retina-sensor-profile eye)))]
270 [(apply max (map first dimensions))
271 (apply max (map second dimensions))]))
272 #+end_src
274 * Importing and parsing descriptions of eyes.
275 First off, get the children of the "eyes" empty node to find all the
276 eyes the creature has.
277 #+name: eye-node
278 #+begin_src clojure
279 (def
280 ^{:doc "Return the children of the creature's \"eyes\" node."
281 :arglists '([creature])}
282 eyes
283 (sense-nodes "eyes"))
284 #+end_src
286 Then, add the camera created by =add-eye!= to the simulation by
287 creating a new viewport.
289 #+name: add-camera
290 #+begin_src clojure
291 (in-ns 'cortex.vision)
292 (defn add-camera!
293 "Add a camera to the world, calling continuation on every frame
294 produced."
295 [#^Application world camera continuation]
296 (let [width (.getWidth camera)
297 height (.getHeight camera)
298 render-manager (.getRenderManager world)
299 viewport (.createMainView render-manager "eye-view" camera)]
300 (doto viewport
301 (.setClearFlags true true true)
302 (.setBackgroundColor ColorRGBA/Black)
303 (.addProcessor (vision-pipeline continuation))
304 (.attachScene (.getRootNode world)))))
305 #+end_src
307 #+results: add-camera
308 : #'cortex.vision/add-camera!
311 The eye's continuation function should register the viewport with the
312 simulation the first time it is called, use the CPU to extract the
313 appropriate pixels from the rendered image and weight them by each
314 sensor's sensitivity. I have the option to do this processing in
315 native code for a slight gain in speed. I could also do it in the GPU
316 for a massive gain in speed. =vision-kernel= generates a list of
317 such continuation functions, one for each channel of the eye.
319 #+name: kernel
320 #+begin_src clojure
321 (in-ns 'cortex.vision)
323 (defrecord attached-viewport [vision-fn viewport-fn]
324 clojure.lang.IFn
325 (invoke [this world] (vision-fn world))
326 (applyTo [this args] (apply vision-fn args)))
328 (defn pixel-sense [sensitivity pixel]
329 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
330 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
331 s-b (bit-and 0x0000FF sensitivity)
333 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
334 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
335 p-b (bit-and 0x0000FF pixel)
337 total-sensitivity (* 255 (+ s-r s-g s-b))]
338 (float (/ (+ (* s-r p-r)
339 (* s-g p-g)
340 (* s-b p-b))
341 total-sensitivity))))
343 (defn vision-kernel
344 "Returns a list of functions, each of which will return a color
345 channel's worth of visual information when called inside a running
346 simulation."
347 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
348 (let [retinal-map (retina-sensor-profile eye)
349 camera (add-eye! creature eye)
350 vision-image
351 (atom
352 (BufferedImage. (.getWidth camera)
353 (.getHeight camera)
354 BufferedImage/TYPE_BYTE_BINARY))
355 register-eye!
356 (runonce
357 (fn [world]
358 (add-camera!
359 world camera
360 (let [counter (atom 0)]
361 (fn [r fb bb bi]
362 (if (zero? (rem (swap! counter inc) (inc skip)))
363 (reset! vision-image
364 (BufferedImage! r fb bb bi))))))))]
365 (vec
366 (map
367 (fn [[key image]]
368 (let [whites (white-coordinates image)
369 topology (vec (collapse whites))
370 sensitivity (sensitivity-presets key key)]
371 (attached-viewport.
372 (fn [world]
373 (register-eye! world)
374 (vector
375 topology
376 (vec
377 (for [[x y] whites]
378 (pixel-sense
379 sensitivity
380 (.getRGB @vision-image x y))))))
381 register-eye!)))
382 retinal-map))))
384 (defn gen-fix-display
385 "Create a function to call to restore a simulation's display when it
386 is disrupted by a Viewport."
387 []
388 (runonce
389 (fn [world]
390 (add-camera! world (.getCamera world) no-op))))
391 #+end_src
393 Note that since each of the functions generated by =vision-kernel=
394 shares the same =register-eye!= function, the eye will be registered
395 only once the first time any of the functions from the list returned
396 by =vision-kernel= is called. Each of the functions returned by
397 =vision-kernel= also allows access to the =Viewport= through which
398 it receives images.
400 The in-game display can be disrupted by all the ViewPorts that the
401 functions generated by =vision-kernel= add. This doesn't affect the
402 simulation or the simulated senses, but can be annoying.
403 =gen-fix-display= restores the in-simulation display.
405 ** The =vision!= function creates sensory probes.
407 All the hard work has been done; all that remains is to apply
408 =vision-kernel= to each eye in the creature and gather the results
409 into one list of functions.
411 #+name: main
412 #+begin_src clojure
413 (defn vision!
414 "Returns a function which returns visual sensory data when called
415 inside a running simulation."
416 [#^Node creature & {skip :skip :or {skip 0}}]
417 (reduce
418 concat
419 (for [eye (eyes creature)]
420 (vision-kernel creature eye))))
421 #+end_src
423 ** Displaying visual data for debugging.
424 # Visualization of Vision. Maybe less alliteration would be better.
425 It's vital to have a visual representation for each sense. Here I use
426 =view-sense= to construct a function that will create a display for
427 visual data.
429 #+name: display
430 #+begin_src clojure
431 (in-ns 'cortex.vision)
433 (defn view-vision
434 "Creates a function which accepts a list of visual sensor-data and
435 displays each element of the list to the screen."
436 []
437 (view-sense
438 (fn
439 [[coords sensor-data]]
440 (let [image (points->image coords)]
441 (dorun
442 (for [i (range (count coords))]
443 (.setRGB image ((coords i) 0) ((coords i) 1)
444 (gray (int (* 255 (sensor-data i)))))))
445 image))))
446 #+end_src
448 * Demonstrations
449 ** Demonstrating the vision pipeline.
451 This is a basic test for the vision system. It only tests the
452 vision-pipeline and does not deal with loading eyes from a blender
453 file. The code creates two videos of the same rotating cube from
454 different angles.
456 #+name: test-1
457 #+begin_src clojure
458 (in-ns 'cortex.test.vision)
460 (defn test-pipeline
461 "Testing vision:
462 Tests the vision system by creating two views of the same rotating
463 object from different angles and displaying both of those views in
464 JFrames.
466 You should see a rotating cube, and two windows,
467 each displaying a different view of the cube."
468 ([] (test-pipeline false))
469 ([record?]
470 (let [candy
471 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
472 (world
473 (doto (Node.)
474 (.attachChild candy))
475 {}
476 (fn [world]
477 (let [cam (.clone (.getCamera world))
478 width (.getWidth cam)
479 height (.getHeight cam)]
480 (add-camera! world cam
481 (comp
482 (view-image
483 (if record?
484 (File. "/home/r/proj/cortex/render/vision/1")))
485 BufferedImage!))
486 (add-camera! world
487 (doto (.clone cam)
488 (.setLocation (Vector3f. -10 0 0))
489 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
490 (comp
491 (view-image
492 (if record?
493 (File. "/home/r/proj/cortex/render/vision/2")))
494 BufferedImage!))
495 ;; This is here to restore the main view
496 ;; after the other views have completed processing
497 (add-camera! world (.getCamera world) no-op)))
498 (fn [world tpf]
499 (.rotate candy (* tpf 0.2) 0 0))))))
500 #+end_src
502 #+begin_html
503 <div class="figure">
504 <video controls="controls" width="755">
505 <source src="../video/spinning-cube.ogg" type="video/ogg"
506 preload="none" poster="../images/aurellem-1280x480.png" />
507 </video>
508 <br> <a href="http://youtu.be/r5Bn2aG7MO0"> YouTube </a>
509 <p>A rotating cube viewed from two different perspectives.</p>
510 </div>
511 #+end_html
513 Creating multiple eyes like this can be used for stereoscopic vision
514 simulation in a single creature or for simulating multiple creatures,
515 each with their own sense of vision.
516 ** Demonstrating eye import and parsing.
518 To the worm from the last post, I add a new node that describes its
519 eyes.
521 #+attr_html: width=755
522 #+caption: The worm with newly added empty nodes describing a single eye.
523 [[../images/worm-with-eye.png]]
525 The node highlighted in yellow is the root level "eyes" node. It has
526 a single child, highlighted in orange, which describes a single
527 eye. This is the "eye" node. It is placed so that the worm will have
528 an eye located in the center of the flat portion of its lower
529 hemispherical section.
531 The two nodes which are not highlighted describe the single joint of
532 the worm.
534 The metadata of the eye-node is:
536 #+begin_src clojure :results verbatim :exports both
537 (cortex.sense/meta-data
538 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
539 #+end_src
541 #+results:
542 : "(let [retina \"Models/test-creature/retina-small.png\"]
543 : {:all retina :red retina :green retina :blue retina})"
545 This is the approximation to the human eye described earlier.
547 #+name: test-2
548 #+begin_src clojure
549 (in-ns 'cortex.test.vision)
551 (defn change-color [obj color]
552 ;;(println-repl obj)
553 (if obj
554 (.setColor (.getMaterial obj) "Color" color)))
556 (defn colored-cannon-ball [color]
557 (comp #(change-color % color)
558 (fire-cannon-ball)))
560 (defn gen-worm
561 "create a creature acceptable for testing as a replacement for the
562 worm."
563 []
564 (nodify
565 "worm"
566 [(nodify
567 "eyes"
568 [(doto
569 (Node. "eye1")
570 (.setLocalTranslation (Vector3f. 0 -1.1 0))
571 (.setUserData
573 "eye"
574 "(let [retina
575 \"Models/test-creature/retina-small.png\"]
576 {:all retina :red retina
577 :green retina :blue retina})"))])
578 (box
579 0.2 0.2 0.2
580 :name "worm-segment"
581 :position (Vector3f. 0 0 0)
582 :color ColorRGBA/Orange)]))
586 (defn test-worm-vision
587 "Testing vision:
588 You should see the worm suspended in mid-air, looking down at a
589 table. There are four small displays, one each for red, green blue,
590 and gray channels. You can fire balls of various colors, and the
591 four channels should react accordingly.
593 Keys:
594 r : fire red-ball
595 b : fire blue-ball
596 g : fire green-ball
597 <space> : fire white ball"
599 ([] (test-worm-vision false))
600 ([record?]
601 (let [the-worm (doto (worm)(body!))
602 ;;the-worm (gen-worm)
603 ;;vision (vision! the-worm)
604 ;;vision-display (view-vision)
605 ;;fix-display (gen-fix-display)
606 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
607 x-axis
608 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
609 :position (Vector3f. 0 -5 0))
610 y-axis
611 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
612 :position (Vector3f. 0 -5 0))
613 z-axis
614 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
615 :position (Vector3f. 0 -5 0))
616 timer (RatchetTimer. 60)
617 ]
619 (world
620 (nodify [(floor) the-worm x-axis y-axis z-axis me])
621 standard-debug-controls
622 ;;"key-r" (colored-cannon-ball ColorRGBA/Red)
623 ;;"key-b" (colored-cannon-ball ColorRGBA/Blue)
624 ;;"key-g" (colored-cannon-ball ColorRGBA/Green))
626 (fn [world]
627 (let
628 [eye-pos (Vector3f. 0 30 0)
629 cam (doto
630 (.clone (.getCamera world))
631 (.setLocation eye-pos)
632 (.lookAt Vector3f/ZERO
633 Vector3f/UNIT_X))
635 bad-cam (doto
636 ;;saved-cam
637 ;;(.clone (.getCamera world))
639 (com.jme3.renderer.Camera. 640 480)
640 (.setFrustumPerspective
641 (float 45)
642 (float (/ 640 480))
643 (float 1)
644 (float 1000))
646 (.setLocation eye-pos)
647 (.lookAt Vector3f/ZERO
648 Vector3f/UNIT_X))
650 bad-cam (add-eye! the-worm (first (eyes the-worm)))
651 ]
654 (light-up-everything world)
655 ;;(speed-up world)
656 (.setTimer world timer)
657 ;;(display-dilated-time world timer)
658 ;; add a view from the worm's perspective
659 (if record?
660 (Capture/captureVideo
661 world
662 (File.
663 "/home/r/proj/cortex/render/worm-vision/main-view")))
665 (bind-sense (last (node-seq the-worm)) cam)
666 (bind-sense (last (node-seq the-worm)) bad-cam)
668 (add-camera!
669 world
670 bad-cam
671 (comp
672 (view-image
673 (if record?
674 (File.
675 "/home/r/proj/cortex/render/worm-vision/worm-view")))
676 BufferedImage!))
680 (add-camera!
681 world cam
682 (comp
683 (view-image
684 (if record?
685 (File.
686 "/home/r/proj/cortex/render/worm-vision/worm-view")))
687 BufferedImage!))
688 (set-gravity world Vector3f/ZERO)
689 (add-camera! world (.getCamera world) no-op)
691 (println-repl cam "\n" bad-cam)
693 ))
695 (fn [world _ ]
696 (.setLocalTranslation me (.getLocation (.getCamera world)))
697 ;; (vision-display
698 ;; (map #(% world) vision)
699 ;; (if record?
700 ;; (File. "/home/r/proj/cortex/render/worm-vision")))
701 ;;(fix-display world)
702 )))))
703 #+end_src
705 #+RESULTS: test-2
706 : #'cortex.test.vision/test-worm-vision
709 The world consists of the worm and a flat gray floor. I can shoot red,
710 green, blue and white cannonballs at the worm. The worm is initially
711 looking down at the floor, and there is no gravity. My perspective
712 (the Main View), the worm's perspective (Worm View) and the 4 sensor
713 channels that comprise the worm's eye are all saved frame-by-frame to
714 disk.
716 * Demonstration of Vision
717 #+begin_html
718 <div class="figure">
719 <video controls="controls" width="755">
720 <source src="../video/worm-vision.ogg" type="video/ogg"
721 preload="none" poster="../images/aurellem-1280x480.png" />
722 </video>
723 <br> <a href="http://youtu.be/J3H3iB_2NPQ"> YouTube </a>
724 <p>Simulated Vision in a Virtual Environment</p>
725 </div>
726 #+end_html
728 ** Generate the Worm Video from Frames
729 #+name: magick2
730 #+begin_src clojure
731 (ns cortex.video.magick2
732 (:import java.io.File)
733 (:use clojure.java.shell))
735 (defn images [path]
736 (sort (rest (file-seq (File. path)))))
738 (def base "/home/r/proj/cortex/render/worm-vision/")
740 (defn pics [file]
741 (images (str base file)))
743 (defn combine-images []
744 (let [main-view (pics "main-view")
745 worm-view (pics "worm-view")
746 blue (pics "0")
747 green (pics "1")
748 red (pics "2")
749 gray (pics "3")
750 blender (let [b-pics (pics "blender")]
751 (concat b-pics (repeat 9001 (last b-pics))))
752 background (repeat 9001 (File. (str base "background.png")))
753 targets (map
754 #(File. (str base "out/" (format "%07d.png" %)))
755 (range 0 (count main-view)))]
756 (dorun
757 (pmap
758 (comp
759 (fn [[background main-view worm-view red green blue gray blender target]]
760 (println target)
761 (sh "convert"
762 background
763 main-view "-geometry" "+18+17" "-composite"
764 worm-view "-geometry" "+677+17" "-composite"
765 green "-geometry" "+685+430" "-composite"
766 red "-geometry" "+788+430" "-composite"
767 blue "-geometry" "+894+430" "-composite"
768 gray "-geometry" "+1000+430" "-composite"
769 blender "-geometry" "+0+0" "-composite"
770 target))
771 (fn [& args] (map #(.getCanonicalPath %) args)))
772 background main-view worm-view red green blue gray blender targets))))
773 #+end_src
775 #+begin_src sh :results silent
776 cd /home/r/proj/cortex/render/worm-vision
777 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
778 #+end_src
780 * Onward!
781 - As a neat bonus, this idea behind simulated vision also enables one
782 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
783 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
784 #+appendix
786 * Headers
788 #+name: vision-header
789 #+begin_src clojure
790 (ns cortex.vision
791 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
792 eyes from different positions to observe the same world, and pass
793 the observed data to any arbitrary function. Automatically reads
794 eye-nodes from specially prepared blender files and instantiates
795 them in the world as actual eyes."
796 {:author "Robert McIntyre"}
797 (:use (cortex world sense util))
798 (:import com.jme3.post.SceneProcessor)
799 (:import (com.jme3.util BufferUtils Screenshots))
800 (:import java.nio.ByteBuffer)
801 (:import java.awt.image.BufferedImage)
802 (:import (com.jme3.renderer ViewPort Camera))
803 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
804 (:import com.jme3.renderer.Renderer)
805 (:import com.jme3.app.Application)
806 (:import com.jme3.texture.FrameBuffer)
807 (:import (com.jme3.scene Node Spatial)))
808 #+end_src
810 #+name: test-header
811 #+begin_src clojure
812 (ns cortex.test.vision
813 (:use (cortex world sense util body vision))
814 (:use cortex.test.body)
815 (:import java.awt.image.BufferedImage)
816 (:import javax.swing.JPanel)
817 (:import javax.swing.SwingUtilities)
818 (:import java.awt.Dimension)
819 (:import javax.swing.JFrame)
820 (:import com.jme3.math.ColorRGBA)
821 (:import com.jme3.scene.Node)
822 (:import com.jme3.math.Vector3f)
823 (:import java.io.File)
824 (:import (com.aurellem.capture Capture RatchetTimer)))
825 #+end_src
826 * Source Listing
827 - [[../src/cortex/vision.clj][cortex.vision]]
828 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
829 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
830 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
831 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
832 - [[http://hg.bortreb.com ][source-repository]]
835 * Next
836 I find some [[./hearing.org][ears]] for the creature while exploring the guts of
837 jMonkeyEngine's sound system.
839 * COMMENT Generate Source
840 #+begin_src clojure :tangle ../src/cortex/vision.clj
841 <<vision-header>>
842 <<pipeline-1>>
843 <<pipeline-2>>
844 <<retina>>
845 <<add-eye>>
846 <<sensitivity>>
847 <<eye-node>>
848 <<add-camera>>
849 <<kernel>>
850 <<main>>
851 <<display>>
852 #+end_src
854 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
855 <<test-header>>
856 <<test-1>>
857 <<test-2>>
858 #+end_src
860 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
861 <<magick2>>
862 #+end_src