Mercurial > cortex
view org/vision.org @ 454:26c13c42481f
work on contributions.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Thu, 27 Mar 2014 17:56:25 -0400 |
parents | 5405f369f4a0 |
children | 3401053124b0 |
line wrap: on
line source
1 #+title: Simulated Sense of Sight2 #+author: Robert McIntyre3 #+email: rlm@mit.edu4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure5 #+keywords: computer vision, jMonkeyEngine3, clojure6 #+SETUPFILE: ../../aurellem/org/setup.org7 #+INCLUDE: ../../aurellem/org/level-0.org8 #+babel: :mkdirp yes :noweb yes :exports both10 * JMonkeyEngine natively supports multiple views of the same world.12 Vision is one of the most important senses for humans, so I need to13 build a simulated sense of vision for my AI. I will do this with14 simulated eyes. Each eye can be independently moved and should see its15 own version of the world depending on where it is.17 Making these simulated eyes a reality is simple because jMonkeyEngine18 already contains extensive support for multiple views of the same 3D19 simulated world. The reason jMonkeyEngine has this support is because20 the support is necessary to create games with split-screen21 views. Multiple views are also used to create efficient22 pseudo-reflections by rendering the scene from a certain perspective23 and then projecting it back onto a surface in the 3D world.25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.26 [[../images/goldeneye-4-player.png]]28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.29 # =ViewPorts= are cameras; =RenderManger= takes snapshots each frame.30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline32 jMonkeyEngine allows you to create a =ViewPort=, which represents a33 view of the simulated world. You can create as many of these as you34 want. Every frame, the =RenderManager= iterates through each35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there36 is a =FrameBuffer= which represents the rendered image in the GPU.38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =RenderManager= records a snapshot of what each view is currently seeing; these snapshots are =FrameBuffer= objects.39 #+ATTR_HTML: width="400"40 [[../images/diagram_rendermanager2.png]]42 Each =ViewPort= can have any number of attached =SceneProcessor=43 objects, which are called every time a new frame is rendered. A44 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do45 whatever it wants to the data. Often this consists of invoking GPU46 specific operations on the rendered image. The =SceneProcessor= can47 also copy the GPU image data to RAM and process it with the CPU.49 ** From Views to Vision50 # Appropriating Views for Vision.52 Each eye in the simulated creature needs its own =ViewPort= so that53 it can see the world from its own perspective. To this =ViewPort=, I54 add a =SceneProcessor= that feeds the visual data to any arbitrary55 continuation function for further processing. That continuation56 function may perform both CPU and GPU operations on the data. To make57 this easy for the continuation function, the =SceneProcessor=58 maintains appropriately sized buffers in RAM to hold the data. It does59 not do any copying from the GPU to the CPU itself because it is a slow60 operation.62 #+name: pipeline-163 #+begin_src clojure64 (defn vision-pipeline65 "Create a SceneProcessor object which wraps a vision processing66 continuation function. The continuation is a function that takes67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],68 each of which has already been appropriately sized."69 [continuation]70 (let [byte-buffer (atom nil)71 renderer (atom nil)72 image (atom nil)]73 (proxy [SceneProcessor] []74 (initialize75 [renderManager viewPort]76 (let [cam (.getCamera viewPort)77 width (.getWidth cam)78 height (.getHeight cam)]79 (reset! renderer (.getRenderer renderManager))80 (reset! byte-buffer81 (BufferUtils/createByteBuffer82 (* width height 4)))83 (reset! image (BufferedImage.84 width height85 BufferedImage/TYPE_4BYTE_ABGR))))86 (isInitialized [] (not (nil? @byte-buffer)))87 (reshape [_ _ _])88 (preFrame [_])89 (postQueue [_])90 (postFrame91 [#^FrameBuffer fb]92 (.clear @byte-buffer)93 (continuation @renderer fb @byte-buffer @image))94 (cleanup []))))95 #+end_src97 The continuation function given to =vision-pipeline= above will be98 given a =Renderer= and three containers for image data. The99 =FrameBuffer= references the GPU image data, but the pixel data can100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=101 are initially "empty" but are sized to hold the data in the102 =FrameBuffer=. I call transferring the GPU image data to the CPU103 structures "mixing" the image data. I have provided three functions to104 do this mixing.106 #+name: pipeline-2107 #+begin_src clojure108 (defn frameBuffer->byteBuffer!109 "Transfer the data in the graphics card (Renderer, FrameBuffer) to110 the CPU (ByteBuffer)."111 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]112 (.readFrameBuffer r fb bb) bb)114 (defn byteBuffer->bufferedImage!115 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT116 style ABGR image data and place it in BufferedImage bi."117 [#^ByteBuffer bb #^BufferedImage bi]118 (Screenshots/convertScreenShot bb bi) bi)120 (defn BufferedImage!121 "Continuation which will grab the buffered image from the materials122 provided by (vision-pipeline)."123 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]124 (byteBuffer->bufferedImage!125 (frameBuffer->byteBuffer! r fb bb) bi))126 #+end_src128 Note that it is possible to write vision processing algorithms129 entirely in terms of =BufferedImage= inputs. Just compose that130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision131 processing algorithm that is entirely hosted on the GPU does not have132 to pay for this convenience.134 * Optical sensor arrays are described with images and referenced with metadata135 The vision pipeline described above handles the flow of rendered136 images. Now, we need simulated eyes to serve as the source of these137 images.139 An eye is described in blender in the same way as a joint. They are140 zero dimensional empty objects with no geometry whose local coordinate141 system determines the orientation of the resulting eye. All eyes are142 children of a parent node named "eyes" just as all joints have a143 parent named "joints". An eye binds to the nearest physical object144 with =bind-sense=.146 #+name: add-eye147 #+begin_src clojure148 (in-ns 'cortex.vision)150 (defn add-eye!151 "Create a Camera centered on the current position of 'eye which152 follows the closest physical node in 'creature. The camera will153 point in the X direction and use the Z vector as up as determined154 by the rotation of these vectors in blender coordinate space. Use155 XZY rotation for the node in blender."156 [#^Node creature #^Spatial eye]157 (let [target (closest-node creature eye)158 [cam-width cam-height]159 ;;[640 480] ;; graphics card on laptop doesn't support160 ;; arbitray dimensions.161 (eye-dimensions eye)162 cam (Camera. cam-width cam-height)163 rot (.getWorldRotation eye)]164 (.setLocation cam (.getWorldTranslation eye))165 (.lookAtDirection166 cam ; this part is not a mistake and167 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in168 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.169 (.setFrustumPerspective170 cam (float 45)171 (float (/ (.getWidth cam) (.getHeight cam)))172 (float 1)173 (float 1000))174 (bind-sense target cam) cam))175 #+end_src177 #+results: add-eye178 : #'cortex.vision/add-eye!180 Here, the camera is created based on metadata on the eye-node and181 attached to the nearest physical object with =bind-sense=182 ** The Retina184 An eye is a surface (the retina) which contains many discrete sensors185 to detect light. These sensors have can have different light-sensing186 properties. In humans, each discrete sensor is sensitive to red,187 blue, green, or gray. These different types of sensors can have188 different spatial distributions along the retina. In humans, there is189 a fovea in the center of the retina which has a very high density of190 color sensors, and a blind spot which has no sensors at all. Sensor191 density decreases in proportion to distance from the fovea.193 I want to be able to model any retinal configuration, so my eye-nodes194 in blender contain metadata pointing to images that describe the195 precise position of the individual sensors using white pixels. The196 meta-data also describes the precise sensitivity to light that the197 sensors described in the image have. An eye can contain any number of198 these images. For example, the metadata for an eye might look like199 this:201 #+begin_src clojure202 {0xFF0000 "Models/test-creature/retina-small.png"}203 #+end_src205 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.206 [[../assets/Models/test-creature/retina-small.png]]208 Together, the number 0xFF0000 and the image image above describe the209 placement of red-sensitive sensory elements.211 Meta-data to very crudely approximate a human eye might be something212 like this:214 #+begin_src clojure215 (let [retinal-profile "Models/test-creature/retina-small.png"]216 {0xFF0000 retinal-profile217 0x00FF00 retinal-profile218 0x0000FF retinal-profile219 0xFFFFFF retinal-profile})220 #+end_src222 The numbers that serve as keys in the map determine a sensor's223 relative sensitivity to the channels red, green, and blue. These224 sensitivity values are packed into an integer in the order =|_|R|G|B|=225 in 8-bit fields. The RGB values of a pixel in the image are added226 together with these sensitivities as linear weights. Therefore,227 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to228 all colors equally (gray).230 For convenience I've defined a few symbols for the more common231 sensitivity values.233 #+name: sensitivity234 #+begin_src clojure235 (def sensitivity-presets236 "Retinal sensitivity presets for sensors that extract one channel237 (:red :blue :green) or average all channels (:all)"238 {:all 0xFFFFFF239 :red 0xFF0000240 :blue 0x0000FF241 :green 0x00FF00})242 #+end_src244 ** Metadata Processing246 =retina-sensor-profile= extracts a map from the eye-node in the same247 format as the example maps above. =eye-dimensions= finds the248 dimensions of the smallest image required to contain all the retinal249 sensor maps.251 #+name: retina252 #+begin_src clojure253 (defn retina-sensor-profile254 "Return a map of pixel sensitivity numbers to BufferedImages255 describing the distribution of light-sensitive components of this256 eye. :red, :green, :blue, :gray are already defined as extracting257 the red, green, blue, and average components respectively."258 [#^Spatial eye]259 (if-let [eye-map (meta-data eye "eye")]260 (map-vals261 load-image262 (eval (read-string eye-map)))))264 (defn eye-dimensions265 "Returns [width, height] determined by the metadata of the eye."266 [#^Spatial eye]267 (let [dimensions268 (map #(vector (.getWidth %) (.getHeight %))269 (vals (retina-sensor-profile eye)))]270 [(apply max (map first dimensions))271 (apply max (map second dimensions))]))272 #+end_src274 * Importing and parsing descriptions of eyes.275 First off, get the children of the "eyes" empty node to find all the276 eyes the creature has.277 #+name: eye-node278 #+begin_src clojure279 (def280 ^{:doc "Return the children of the creature's \"eyes\" node."281 :arglists '([creature])}282 eyes283 (sense-nodes "eyes"))284 #+end_src286 Then, add the camera created by =add-eye!= to the simulation by287 creating a new viewport.289 #+name: add-camera290 #+begin_src clojure291 (in-ns 'cortex.vision)292 (defn add-camera!293 "Add a camera to the world, calling continuation on every frame294 produced."295 [#^Application world camera continuation]296 (let [width (.getWidth camera)297 height (.getHeight camera)298 render-manager (.getRenderManager world)299 viewport (.createMainView render-manager "eye-view" camera)]300 (doto viewport301 (.setClearFlags true true true)302 (.setBackgroundColor ColorRGBA/Black)303 (.addProcessor (vision-pipeline continuation))304 (.attachScene (.getRootNode world)))))305 #+end_src307 #+results: add-camera308 : #'cortex.vision/add-camera!311 The eye's continuation function should register the viewport with the312 simulation the first time it is called, use the CPU to extract the313 appropriate pixels from the rendered image and weight them by each314 sensor's sensitivity. I have the option to do this processing in315 native code for a slight gain in speed. I could also do it in the GPU316 for a massive gain in speed. =vision-kernel= generates a list of317 such continuation functions, one for each channel of the eye.319 #+name: kernel320 #+begin_src clojure321 (in-ns 'cortex.vision)323 (defrecord attached-viewport [vision-fn viewport-fn]324 clojure.lang.IFn325 (invoke [this world] (vision-fn world))326 (applyTo [this args] (apply vision-fn args)))328 (defn pixel-sense [sensitivity pixel]329 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)330 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)331 s-b (bit-and 0x0000FF sensitivity)333 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)334 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)335 p-b (bit-and 0x0000FF pixel)337 total-sensitivity (* 255 (+ s-r s-g s-b))]338 (float (/ (+ (* s-r p-r)339 (* s-g p-g)340 (* s-b p-b))341 total-sensitivity))))343 (defn vision-kernel344 "Returns a list of functions, each of which will return a color345 channel's worth of visual information when called inside a running346 simulation."347 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]348 (let [retinal-map (retina-sensor-profile eye)349 camera (add-eye! creature eye)350 vision-image351 (atom352 (BufferedImage. (.getWidth camera)353 (.getHeight camera)354 BufferedImage/TYPE_BYTE_BINARY))355 register-eye!356 (runonce357 (fn [world]358 (add-camera!359 world camera360 (let [counter (atom 0)]361 (fn [r fb bb bi]362 (if (zero? (rem (swap! counter inc) (inc skip)))363 (reset! vision-image364 (BufferedImage! r fb bb bi))))))))]365 (vec366 (map367 (fn [[key image]]368 (let [whites (white-coordinates image)369 topology (vec (collapse whites))370 sensitivity (sensitivity-presets key key)]371 (attached-viewport.372 (fn [world]373 (register-eye! world)374 (vector375 topology376 (vec377 (for [[x y] whites]378 (pixel-sense379 sensitivity380 (.getRGB @vision-image x y))))))381 register-eye!)))382 retinal-map))))384 (defn gen-fix-display385 "Create a function to call to restore a simulation's display when it386 is disrupted by a Viewport."387 []388 (runonce389 (fn [world]390 (add-camera! world (.getCamera world) no-op))))391 #+end_src393 Note that since each of the functions generated by =vision-kernel=394 shares the same =register-eye!= function, the eye will be registered395 only once the first time any of the functions from the list returned396 by =vision-kernel= is called. Each of the functions returned by397 =vision-kernel= also allows access to the =Viewport= through which398 it receives images.400 The in-game display can be disrupted by all the ViewPorts that the401 functions generated by =vision-kernel= add. This doesn't affect the402 simulation or the simulated senses, but can be annoying.403 =gen-fix-display= restores the in-simulation display.405 ** The =vision!= function creates sensory probes.407 All the hard work has been done; all that remains is to apply408 =vision-kernel= to each eye in the creature and gather the results409 into one list of functions.411 #+name: main412 #+begin_src clojure413 (defn vision!414 "Returns a list of functions, each of which returns visual sensory415 data when called inside a running simulation."416 [#^Node creature & {skip :skip :or {skip 0}}]417 (reduce418 concat419 (for [eye (eyes creature)]420 (vision-kernel creature eye))))421 #+end_src423 ** Displaying visual data for debugging.424 # Visualization of Vision. Maybe less alliteration would be better.425 It's vital to have a visual representation for each sense. Here I use426 =view-sense= to construct a function that will create a display for427 visual data.429 #+name: display430 #+begin_src clojure431 (in-ns 'cortex.vision)433 (defn view-vision434 "Creates a function which accepts a list of visual sensor-data and435 displays each element of the list to the screen."436 []437 (view-sense438 (fn439 [[coords sensor-data]]440 (let [image (points->image coords)]441 (dorun442 (for [i (range (count coords))]443 (.setRGB image ((coords i) 0) ((coords i) 1)444 (gray (int (* 255 (sensor-data i)))))))445 image))))446 #+end_src448 * Demonstrations449 ** Demonstrating the vision pipeline.451 This is a basic test for the vision system. It only tests the452 vision-pipeline and does not deal with loading eyes from a blender453 file. The code creates two videos of the same rotating cube from454 different angles.456 #+name: test-1457 #+begin_src clojure458 (in-ns 'cortex.test.vision)460 (defn test-pipeline461 "Testing vision:462 Tests the vision system by creating two views of the same rotating463 object from different angles and displaying both of those views in464 JFrames.466 You should see a rotating cube, and two windows,467 each displaying a different view of the cube."468 ([] (test-pipeline false))469 ([record?]470 (let [candy471 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]472 (world473 (doto (Node.)474 (.attachChild candy))475 {}476 (fn [world]477 (let [cam (.clone (.getCamera world))478 width (.getWidth cam)479 height (.getHeight cam)]480 (add-camera! world cam481 (comp482 (view-image483 (if record?484 (File. "/home/r/proj/cortex/render/vision/1")))485 BufferedImage!))486 (add-camera! world487 (doto (.clone cam)488 (.setLocation (Vector3f. -10 0 0))489 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))490 (comp491 (view-image492 (if record?493 (File. "/home/r/proj/cortex/render/vision/2")))494 BufferedImage!))495 (let [timer (IsoTimer. 60)]496 (.setTimer world timer)497 (display-dilated-time world timer))498 ;; This is here to restore the main view499 ;; after the other views have completed processing500 (add-camera! world (.getCamera world) no-op)))501 (fn [world tpf]502 (.rotate candy (* tpf 0.2) 0 0))))))503 #+end_src505 #+results: test-1506 : #'cortex.test.vision/test-pipeline508 #+begin_html509 <div class="figure">510 <video controls="controls" width="755">511 <source src="../video/spinning-cube.ogg" type="video/ogg"512 preload="none" poster="../images/aurellem-1280x480.png" />513 </video>514 <br> <a href="http://youtu.be/r5Bn2aG7MO0"> YouTube </a>515 <p>A rotating cube viewed from two different perspectives.</p>516 </div>517 #+end_html519 Creating multiple eyes like this can be used for stereoscopic vision520 simulation in a single creature or for simulating multiple creatures,521 each with their own sense of vision.522 ** Demonstrating eye import and parsing.524 To the worm from the last post, I add a new node that describes its525 eyes.527 #+attr_html: width=755528 #+caption: The worm with newly added empty nodes describing a single eye.529 [[../images/worm-with-eye.png]]531 The node highlighted in yellow is the root level "eyes" node. It has532 a single child, highlighted in orange, which describes a single533 eye. This is the "eye" node. It is placed so that the worm will have534 an eye located in the center of the flat portion of its lower535 hemispherical section.537 The two nodes which are not highlighted describe the single joint of538 the worm.540 The metadata of the eye-node is:542 #+begin_src clojure :results verbatim :exports both543 (cortex.sense/meta-data544 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")545 #+end_src547 #+results:548 : "(let [retina \"Models/test-creature/retina-small.png\"]549 : {:all retina :red retina :green retina :blue retina})"551 This is the approximation to the human eye described earlier.553 #+name: test-2554 #+begin_src clojure555 (in-ns 'cortex.test.vision)557 (defn change-color [obj color]558 ;;(println-repl obj)559 (if obj560 (.setColor (.getMaterial obj) "Color" color)))562 (defn colored-cannon-ball [color]563 (comp #(change-color % color)564 (fire-cannon-ball)))566 (defn gen-worm567 "create a creature acceptable for testing as a replacement for the568 worm."569 []570 (nodify571 "worm"572 [(nodify573 "eyes"574 [(doto575 (Node. "eye1")576 (.setLocalTranslation (Vector3f. 0 -1.1 0))577 (.setUserData579 "eye"580 "(let [retina581 \"Models/test-creature/retina-small.png\"]582 {:all retina :red retina583 :green retina :blue retina})"))])584 (box585 0.2 0.2 0.2586 :name "worm-segment"587 :position (Vector3f. 0 0 0)588 :color ColorRGBA/Orange)]))592 (defn test-worm-vision593 "Testing vision:594 You should see the worm suspended in mid-air, looking down at a595 table. There are four small displays, one each for red, green blue,596 and gray channels. You can fire balls of various colors, and the597 four channels should react accordingly.599 Keys:600 r : fire red-ball601 b : fire blue-ball602 g : fire green-ball603 <space> : fire white ball"605 ([] (test-worm-vision false))606 ([record?]607 (let [the-worm (doto (worm)(body!))608 vision (vision! the-worm)609 vision-display (view-vision)610 fix-display (gen-fix-display)611 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)612 x-axis613 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red614 :position (Vector3f. 0 -5 0))615 y-axis616 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green617 :position (Vector3f. 0 -5 0))618 z-axis619 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue620 :position (Vector3f. 0 -5 0))622 ]624 (world625 (nodify [(floor) the-worm x-axis y-axis z-axis me])626 (merge standard-debug-controls627 {"key-r" (colored-cannon-ball ColorRGBA/Red)628 "key-b" (colored-cannon-ball ColorRGBA/Blue)629 "key-g" (colored-cannon-ball ColorRGBA/Green)})631 (fn [world]632 (light-up-everything world)633 (speed-up world)634 (let [timer (IsoTimer. 60)]635 (.setTimer world timer)636 (display-dilated-time world timer))637 ;; add a view from the worm's perspective638 (if record?639 (Capture/captureVideo640 world641 (File.642 "/home/r/proj/cortex/render/worm-vision/main-view")))644 (add-camera!645 world646 (add-eye! the-worm (first (eyes the-worm)))647 (comp648 (view-image649 (if record?650 (File.651 "/home/r/proj/cortex/render/worm-vision/worm-view")))652 BufferedImage!))654 (set-gravity world Vector3f/ZERO)655 (add-camera! world (.getCamera world) no-op))657 (fn [world _]658 (.setLocalTranslation me (.getLocation (.getCamera world)))659 (vision-display660 (map #(% world) vision)661 (if record?662 (File. "/home/r/proj/cortex/render/worm-vision")))663 (fix-display world)664 )))))665 #+end_src667 #+RESULTS: test-2668 : #'cortex.test.vision/test-worm-vision671 The world consists of the worm and a flat gray floor. I can shoot red,672 green, blue and white cannonballs at the worm. The worm is initially673 looking down at the floor, and there is no gravity. My perspective674 (the Main View), the worm's perspective (Worm View) and the 4 sensor675 channels that comprise the worm's eye are all saved frame-by-frame to676 disk.678 * Demonstration of Vision679 #+begin_html680 <div class="figure">681 <video controls="controls" width="755">682 <source src="../video/worm-vision.ogg" type="video/ogg"683 preload="none" poster="../images/aurellem-1280x480.png" />684 </video>685 <br> <a href="http://youtu.be/J3H3iB_2NPQ"> YouTube </a>686 <p>Simulated Vision in a Virtual Environment</p>687 </div>688 #+end_html690 ** Generate the Worm Video from Frames691 #+name: magick2692 #+begin_src clojure693 (ns cortex.video.magick2694 (:import java.io.File)695 (:use clojure.java.shell))697 (defn images [path]698 (sort (rest (file-seq (File. path)))))700 (def base "/home/r/proj/cortex/render/worm-vision/")702 (defn pics [file]703 (images (str base file)))705 (defn combine-images []706 (let [main-view (pics "main-view")707 worm-view (pics "worm-view")708 blue (pics "0")709 green (pics "1")710 red (pics "2")711 gray (pics "3")712 blender (let [b-pics (pics "blender")]713 (concat b-pics (repeat 9001 (last b-pics))))714 background (repeat 9001 (File. (str base "background.png")))715 targets (map716 #(File. (str base "out/" (format "%07d.png" %)))717 (range 0 (count main-view)))]718 (dorun719 (pmap720 (comp721 (fn [[background main-view worm-view red green blue gray blender target]]722 (println target)723 (sh "convert"724 background725 main-view "-geometry" "+18+17" "-composite"726 worm-view "-geometry" "+677+17" "-composite"727 green "-geometry" "+685+430" "-composite"728 red "-geometry" "+788+430" "-composite"729 blue "-geometry" "+894+430" "-composite"730 gray "-geometry" "+1000+430" "-composite"731 blender "-geometry" "+0+0" "-composite"732 target))733 (fn [& args] (map #(.getCanonicalPath %) args)))734 background main-view worm-view red green blue gray blender targets))))735 #+end_src737 #+begin_src sh :results silent738 cd /home/r/proj/cortex/render/worm-vision739 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg740 #+end_src742 * Onward!743 - As a neat bonus, this idea behind simulated vision also enables one744 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].745 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].746 #+appendix748 * Headers750 #+name: vision-header751 #+begin_src clojure752 (ns cortex.vision753 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple754 eyes from different positions to observe the same world, and pass755 the observed data to any arbitrary function. Automatically reads756 eye-nodes from specially prepared blender files and instantiates757 them in the world as actual eyes."758 {:author "Robert McIntyre"}759 (:use (cortex world sense util))760 (:import com.jme3.post.SceneProcessor)761 (:import (com.jme3.util BufferUtils Screenshots))762 (:import java.nio.ByteBuffer)763 (:import java.awt.image.BufferedImage)764 (:import (com.jme3.renderer ViewPort Camera))765 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))766 (:import com.jme3.renderer.Renderer)767 (:import com.jme3.app.Application)768 (:import com.jme3.texture.FrameBuffer)769 (:import (com.jme3.scene Node Spatial)))770 #+end_src772 #+name: test-header773 #+begin_src clojure774 (ns cortex.test.vision775 (:use (cortex world sense util body vision))776 (:use cortex.test.body)777 (:import java.awt.image.BufferedImage)778 (:import javax.swing.JPanel)779 (:import javax.swing.SwingUtilities)780 (:import java.awt.Dimension)781 (:import javax.swing.JFrame)782 (:import com.jme3.math.ColorRGBA)783 (:import com.jme3.scene.Node)784 (:import com.jme3.math.Vector3f)785 (:import java.io.File)786 (:import (com.aurellem.capture Capture RatchetTimer IsoTimer)))787 #+end_src789 #+results: test-header790 : com.aurellem.capture.IsoTimer792 * Source Listing793 - [[../src/cortex/vision.clj][cortex.vision]]794 - [[../src/cortex/test/vision.clj][cortex.test.vision]]795 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]796 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]797 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>798 - [[http://hg.bortreb.com ][source-repository]]801 * Next802 I find some [[./hearing.org][ears]] for the creature while exploring the guts of803 jMonkeyEngine's sound system.805 * COMMENT Generate Source806 #+begin_src clojure :tangle ../src/cortex/vision.clj807 <<vision-header>>808 <<pipeline-1>>809 <<pipeline-2>>810 <<retina>>811 <<add-eye>>812 <<sensitivity>>813 <<eye-node>>814 <<add-camera>>815 <<kernel>>816 <<main>>817 <<display>>818 #+end_src820 #+begin_src clojure :tangle ../src/cortex/test/vision.clj821 <<test-header>>822 <<test-1>>823 <<test-2>>824 #+end_src826 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj827 <<magick2>>828 #+end_src