Mercurial > cortex
view org/vision.org @ 257:5d7961d7fded
wrote intro text for proprioception
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Tue, 14 Feb 2012 00:37:42 -0700 |
parents | 02b2e6f3fb43 |
children | 0e85237d27a7 |
line wrap: on
line source
1 #+title: Simulated Sense of Sight2 #+author: Robert McIntyre3 #+email: rlm@mit.edu4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure5 #+keywords: computer vision, jMonkeyEngine3, clojure6 #+SETUPFILE: ../../aurellem/org/setup.org7 #+INCLUDE: ../../aurellem/org/level-0.org8 #+babel: :mkdirp yes :noweb yes :exports both10 * Vision12 Vision is one of the most important senses for humans, so I need to13 build a simulated sense of vision for my AI. I will do this with14 simulated eyes. Each eye can be independely moved and should see its15 own version of the world depending on where it is.17 Making these simulated eyes a reality is simple bacause jMonkeyEngine18 already conatains extensive support for multiple views of the same 3D19 simulated world. The reason jMonkeyEngine has this support is because20 the support is necessary to create games with split-screen21 views. Multiple views are also used to create efficient22 pseudo-reflections by rendering the scene from a certain perspective23 and then projecting it back onto a surface in the 3D world.25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.26 [[../images/goldeneye-4-player.png]]28 * Brief Description of jMonkeyEngine's Rendering Pipeline30 jMonkeyEngine allows you to create a =ViewPort=, which represents a31 view of the simulated world. You can create as many of these as you32 want. Every frame, the =RenderManager= iterates through each33 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there34 is a =FrameBuffer= which represents the rendered image in the GPU.36 Each =ViewPort= can have any number of attached =SceneProcessor=37 objects, which are called every time a new frame is rendered. A38 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do39 whatever it wants to the data. Often this consists of invoking GPU40 specific operations on the rendered image. The =SceneProcessor= can41 also copy the GPU image data to RAM and process it with the CPU.43 * The Vision Pipeline45 Each eye in the simulated creature needs it's own =ViewPort= so that46 it can see the world from its own perspective. To this =ViewPort=, I47 add a =SceneProcessor= that feeds the visual data to any arbitray48 continuation function for further processing. That continuation49 function may perform both CPU and GPU operations on the data. To make50 this easy for the continuation function, the =SceneProcessor=51 maintains appropriatly sized buffers in RAM to hold the data. It does52 not do any copying from the GPU to the CPU itself because it is a slow53 operation.55 #+name: pipeline-156 #+begin_src clojure57 (defn vision-pipeline58 "Create a SceneProcessor object which wraps a vision processing59 continuation function. The continuation is a function that takes60 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],61 each of which has already been appropiately sized."62 [continuation]63 (let [byte-buffer (atom nil)64 renderer (atom nil)65 image (atom nil)]66 (proxy [SceneProcessor] []67 (initialize68 [renderManager viewPort]69 (let [cam (.getCamera viewPort)70 width (.getWidth cam)71 height (.getHeight cam)]72 (reset! renderer (.getRenderer renderManager))73 (reset! byte-buffer74 (BufferUtils/createByteBuffer75 (* width height 4)))76 (reset! image (BufferedImage.77 width height78 BufferedImage/TYPE_4BYTE_ABGR))))79 (isInitialized [] (not (nil? @byte-buffer)))80 (reshape [_ _ _])81 (preFrame [_])82 (postQueue [_])83 (postFrame84 [#^FrameBuffer fb]85 (.clear @byte-buffer)86 (continuation @renderer fb @byte-buffer @image))87 (cleanup []))))88 #+end_src90 The continuation function given to =(vision-pipeline)= above will be91 given a =Renderer= and three containers for image data. The92 =FrameBuffer= references the GPU image data, but the pixel data can93 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=94 are initially "empty" but are sized to hold the data in the95 =FrameBuffer=. I call transfering the GPU image data to the CPU96 structures "mixing" the image data. I have provided three functions to97 do this mixing.99 #+name: pipeline-2100 #+begin_src clojure101 (defn frameBuffer->byteBuffer!102 "Transfer the data in the graphics card (Renderer, FrameBuffer) to103 the CPU (ByteBuffer)."104 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]105 (.readFrameBuffer r fb bb) bb)107 (defn byteBuffer->bufferedImage!108 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT109 style ABGR image data and place it in BufferedImage bi."110 [#^ByteBuffer bb #^BufferedImage bi]111 (Screenshots/convertScreenShot bb bi) bi)113 (defn BufferedImage!114 "Continuation which will grab the buffered image from the materials115 provided by (vision-pipeline)."116 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]117 (byteBuffer->bufferedImage!118 (frameBuffer->byteBuffer! r fb bb) bi))119 #+end_src121 Note that it is possible to write vision processing algorithms122 entirely in terms of =BufferedImage= inputs. Just compose that123 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision124 processing algorithm that is entirely hosted on the GPU does not have125 to pay for this convienence.127 * COMMENT asdasd129 (vision creature) will take an optional :skip argument which will130 inform the continuations in scene processor to skip the given131 number of cycles 0 means that no cycles will be skipped.133 (vision creature) will return [init-functions sensor-functions].134 The init-functions are each single-arg functions that take the135 world and register the cameras and must each be called before the136 corresponding sensor-functions. Each init-function returns the137 viewport for that eye which can be manipulated, saved, etc. Each138 sensor-function is a thunk and will return data in the same139 format as the tactile-sensor functions the structure is140 [topology, sensor-data]. Internally, these sensor-functions141 maintain a reference to sensor-data which is periodically updated142 by the continuation function established by its init-function.143 They can be queried every cycle, but their information may not144 necessairly be different every cycle.146 * Physical Eyes148 The vision pipeline described above handles the flow of rendered149 images. Now, we need simulated eyes to serve as the source of these150 images.152 An eye is described in blender in the same way as a joint. They are153 zero dimensional empty objects with no geometry whose local coordinate154 system determines the orientation of the resulting eye. All eyes are155 childern of a parent node named "eyes" just as all joints have a156 parent named "joints". An eye binds to the nearest physical object157 with =(bind-sense=).159 #+name: add-eye160 #+begin_src clojure161 (in-ns 'cortex.vision)163 (defn add-eye!164 "Create a Camera centered on the current position of 'eye which165 follows the closest physical node in 'creature and sends visual166 data to 'continuation. The camera will point in the X direction and167 use the Z vector as up as determined by the rotation of these168 vectors in blender coordinate space. Use XZY rotation for the node169 in blender."170 [#^Node creature #^Spatial eye]171 (let [target (closest-node creature eye)172 [cam-width cam-height] (eye-dimensions eye)173 cam (Camera. cam-width cam-height)174 rot (.getWorldRotation eye)]175 (.setLocation cam (.getWorldTranslation eye))176 (.lookAtDirection177 cam ; this part is not a mistake and178 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in179 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.180 (.setFrustumPerspective181 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)182 (bind-sense target cam) cam))183 #+end_src185 Here, the camera is created based on metadata on the eye-node and186 attached to the nearest physical object with =(bind-sense)=189 ** The Retina191 An eye is a surface (the retina) which contains many discrete sensors192 to detect light. These sensors have can have different light-sensing193 properties. In humans, each discrete sensor is sensitive to red,194 blue, green, or gray. These different types of sensors can have195 different spatial distributions along the retina. In humans, there is196 a fovea in the center of the retina which has a very high density of197 color sensors, and a blind spot which has no sensors at all. Sensor198 density decreases in proportion to distance from the fovea.200 I want to be able to model any retinal configuration, so my eye-nodes201 in blender contain metadata pointing to images that describe the202 percise position of the individual sensors using white pixels. The203 meta-data also describes the percise sensitivity to light that the204 sensors described in the image have. An eye can contain any number of205 these images. For example, the metadata for an eye might look like206 this:208 #+begin_src clojure209 {0xFF0000 "Models/test-creature/retina-small.png"}210 #+end_src212 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.213 [[../assets/Models/test-creature/retina-small.png]]215 Together, the number 0xFF0000 and the image image above describe the216 placement of red-sensitive sensory elements.218 Meta-data to very crudely approximate a human eye might be something219 like this:221 #+begin_src clojure222 (let [retinal-profile "Models/test-creature/retina-small.png"]223 {0xFF0000 retinal-profile224 0x00FF00 retinal-profile225 0x0000FF retinal-profile226 0xFFFFFF retinal-profile})227 #+end_src229 The numbers that serve as keys in the map determine a sensor's230 relative sensitivity to the channels red, green, and blue. These231 sensitivity values are packed into an integer in the order =|_|R|G|B|=232 in 8-bit fields. The RGB values of a pixel in the image are added233 together with these sensitivities as linear weights. Therfore,234 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to235 all colors equally (gray).237 For convienence I've defined a few symbols for the more common238 sensitivity values.240 #+name: sensitivity241 #+begin_src clojure242 (defvar sensitivity-presets243 {:all 0xFFFFFF244 :red 0xFF0000245 :blue 0x0000FF246 :green 0x00FF00}247 "Retinal sensitivity presets for sensors that extract one channel248 (:red :blue :green) or average all channels (:all)")249 #+end_src251 ** Metadata Processing253 =(retina-sensor-profile)= extracts a map from the eye-node in the same254 format as the example maps above. =(eye-dimensions)= finds the255 dimensions of the smallest image required to contain all the retinal256 sensor maps.258 #+name: retina259 #+begin_src clojure260 (defn retina-sensor-profile261 "Return a map of pixel sensitivity numbers to BufferedImages262 describing the distribution of light-sensitive components of this263 eye. :red, :green, :blue, :gray are already defined as extracting264 the red, green, blue, and average components respectively."265 [#^Spatial eye]266 (if-let [eye-map (meta-data eye "eye")]267 (map-vals268 load-image269 (eval (read-string eye-map)))))271 (defn eye-dimensions272 "Returns [width, height] determined by the metadata of the eye."273 [#^Spatial eye]274 (let [dimensions275 (map #(vector (.getWidth %) (.getHeight %))276 (vals (retina-sensor-profile eye)))]277 [(apply max (map first dimensions))278 (apply max (map second dimensions))]))279 #+end_src281 * Eye Creation282 First off, get the children of the "eyes" empty node to find all the283 eyes the creature has.284 #+name: eye-node285 #+begin_src clojure286 (defvar287 ^{:arglists '([creature])}288 eyes289 (sense-nodes "eyes")290 "Return the children of the creature's \"eyes\" node.")291 #+end_src293 Then, add the camera created by =(add-eye!)= to the simulation by294 creating a new viewport.296 #+name: add-camera297 #+begin_src clojure298 (defn add-camera!299 "Add a camera to the world, calling continuation on every frame300 produced."301 [#^Application world camera continuation]302 (let [width (.getWidth camera)303 height (.getHeight camera)304 render-manager (.getRenderManager world)305 viewport (.createMainView render-manager "eye-view" camera)]306 (doto viewport307 (.setClearFlags true true true)308 (.setBackgroundColor ColorRGBA/Black)309 (.addProcessor (vision-pipeline continuation))310 (.attachScene (.getRootNode world)))))311 #+end_src314 The eye's continuation function should register the viewport with the315 simulation the first time it is called, use the CPU to extract the316 appropriate pixels from the rendered image and weight them by each317 sensor's sensitivity. I have the option to do this processing in318 native code for a slight gain in speed. I could also do it in the GPU319 for a massive gain in speed. =(vision-kernel)= generates a list of320 such continuation functions, one for each channel of the eye.322 #+name: kernel323 #+begin_src clojure324 (in-ns 'cortex.vision)326 (defrecord attached-viewport [vision-fn viewport-fn]327 clojure.lang.IFn328 (invoke [this world] (vision-fn world))329 (applyTo [this args] (apply vision-fn args)))331 (defn pixel-sense [sensitivity pixel]332 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)333 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)334 s-b (bit-and 0x0000FF sensitivity)336 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)337 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)338 p-b (bit-and 0x0000FF pixel)340 total-sensitivity (* 255 (+ s-r s-g s-b))]341 (float (/ (+ (* s-r p-r)342 (* s-g p-g)343 (* s-b p-b))344 total-sensitivity))))346 (defn vision-kernel347 "Returns a list of functions, each of which will return a color348 channel's worth of visual information when called inside a running349 simulation."350 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]351 (let [retinal-map (retina-sensor-profile eye)352 camera (add-eye! creature eye)353 vision-image354 (atom355 (BufferedImage. (.getWidth camera)356 (.getHeight camera)357 BufferedImage/TYPE_BYTE_BINARY))358 register-eye!359 (runonce360 (fn [world]361 (add-camera!362 world camera363 (let [counter (atom 0)]364 (fn [r fb bb bi]365 (if (zero? (rem (swap! counter inc) (inc skip)))366 (reset! vision-image367 (BufferedImage! r fb bb bi))))))))]368 (vec369 (map370 (fn [[key image]]371 (let [whites (white-coordinates image)372 topology (vec (collapse whites))373 sensitivity (sensitivity-presets key key)]374 (attached-viewport.375 (fn [world]376 (register-eye! world)377 (vector378 topology379 (vec380 (for [[x y] whites]381 (pixel-sense382 sensitivity383 (.getRGB @vision-image x y))))))384 register-eye!)))385 retinal-map))))387 (defn gen-fix-display388 "Create a function to call to restore a simulation's display when it389 is disrupted by a Viewport."390 []391 (runonce392 (fn [world]393 (add-camera! world (.getCamera world) no-op))))394 #+end_src396 Note that since each of the functions generated by =(vision-kernel)=397 shares the same =(register-eye!)= function, the eye will be registered398 only once the first time any of the functions from the list returned399 by =(vision-kernel)= is called. Each of the functions returned by400 =(vision-kernel)= also allows access to the =Viewport= through which401 it recieves images.403 The in-game display can be disrupted by all the viewports that the404 functions greated by =(vision-kernel)= add. This doesn't affect the405 simulation or the simulated senses, but can be annoying.406 =(gen-fix-display)= restores the in-simulation display.408 ** Vision!410 All the hard work has been done; all that remains is to apply411 =(vision-kernel)= to each eye in the creature and gather the results412 into one list of functions.414 #+name: main415 #+begin_src clojure416 (defn vision!417 "Returns a function which returns visual sensory data when called418 inside a running simulation."419 [#^Node creature & {skip :skip :or {skip 0}}]420 (reduce421 concat422 (for [eye (eyes creature)]423 (vision-kernel creature eye))))424 #+end_src426 ** Visualization of Vision428 It's vital to have a visual representation for each sense. Here I use429 =(view-sense)= to construct a function that will create a display for430 visual data.432 #+name: display433 #+begin_src clojure434 (in-ns 'cortex.vision)436 (defn view-vision437 "Creates a function which accepts a list of visual sensor-data and438 displays each element of the list to the screen."439 []440 (view-sense441 (fn442 [[coords sensor-data]]443 (let [image (points->image coords)]444 (dorun445 (for [i (range (count coords))]446 (.setRGB image ((coords i) 0) ((coords i) 1)447 (gray (int (* 255 (sensor-data i)))))))448 image))))449 #+end_src451 * Tests452 ** Basic Test454 This is a basic test for the vision system. It only tests the455 vision-pipeline and does not deal with loadig eyes from a blender456 file. The code creates two videos of the same rotating cube from457 different angles.459 #+name: test-1460 #+begin_src clojure461 (in-ns 'cortex.test.vision)463 (defn test-pipeline464 "Testing vision:465 Tests the vision system by creating two views of the same rotating466 object from different angles and displaying both of those views in467 JFrames.469 You should see a rotating cube, and two windows,470 each displaying a different view of the cube."471 []472 (let [candy473 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]474 (world475 (doto (Node.)476 (.attachChild candy))477 {}478 (fn [world]479 (let [cam (.clone (.getCamera world))480 width (.getWidth cam)481 height (.getHeight cam)]482 (add-camera! world cam483 (comp484 (view-image485 (File. "/home/r/proj/cortex/render/vision/1"))486 BufferedImage!))487 (add-camera! world488 (doto (.clone cam)489 (.setLocation (Vector3f. -10 0 0))490 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))491 (comp492 (view-image493 (File. "/home/r/proj/cortex/render/vision/2"))494 BufferedImage!))495 ;; This is here to restore the main view496 ;; after the other views have completed processing497 (add-camera! world (.getCamera world) no-op)))498 (fn [world tpf]499 (.rotate candy (* tpf 0.2) 0 0)))))500 #+end_src502 #+begin_html503 <div class="figure">504 <video controls="controls" width="755">505 <source src="../video/spinning-cube.ogg" type="video/ogg"506 preload="none" poster="../images/aurellem-1280x480.png" />507 </video>508 <p>A rotating cube viewed from two different perspectives.</p>509 </div>510 #+end_html512 Creating multiple eyes like this can be used for stereoscopic vision513 simulation in a single creature or for simulating multiple creatures,514 each with their own sense of vision.516 ** Adding Vision to the Worm518 To the worm from the last post, I add a new node that describes its519 eyes.521 #+attr_html: width=755522 #+caption: The worm with newly added empty nodes describing a single eye.523 [[../images/worm-with-eye.png]]525 The node highlighted in yellow is the root level "eyes" node. It has526 a single child, highlighted in orange, which describes a single527 eye. This is the "eye" node. It is placed so that the worm will have528 an eye located in the center of the flat portion of its lower529 hemispherical section.531 The two nodes which are not highlighted describe the single joint of532 the worm.534 The metadata of the eye-node is:536 #+begin_src clojure :results verbatim :exports both537 (cortex.sense/meta-data538 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")539 #+end_src541 #+results:542 : "(let [retina \"Models/test-creature/retina-small.png\"]543 : {:all retina :red retina :green retina :blue retina})"545 This is the approximation to the human eye described earlier.547 #+name: test-2548 #+begin_src clojure549 (in-ns 'cortex.test.vision)551 (defn change-color [obj color]552 (println-repl obj)553 (if obj554 (.setColor (.getMaterial obj) "Color" color)))556 (defn colored-cannon-ball [color]557 (comp #(change-color % color)558 (fire-cannon-ball)))560 (defn test-worm-vision [record]561 (let [the-worm (doto (worm)(body!))562 vision (vision! the-worm)563 vision-display (view-vision)564 fix-display (gen-fix-display)565 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)566 x-axis567 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red568 :position (Vector3f. 0 -5 0))569 y-axis570 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green571 :position (Vector3f. 0 -5 0))572 z-axis573 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue574 :position (Vector3f. 0 -5 0))575 timer (RatchetTimer. 60)]577 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])578 (assoc standard-debug-controls579 "key-r" (colored-cannon-ball ColorRGBA/Red)580 "key-b" (colored-cannon-ball ColorRGBA/Blue)581 "key-g" (colored-cannon-ball ColorRGBA/Green))582 (fn [world]583 (light-up-everything world)584 (speed-up world)585 (.setTimer world timer)586 (display-dialated-time world timer)587 ;; add a view from the worm's perspective588 (if record589 (Capture/captureVideo590 world591 (File.592 "/home/r/proj/cortex/render/worm-vision/main-view")))594 (add-camera!595 world596 (add-eye! the-worm597 (.getChild598 (.getChild the-worm "eyes") "eye"))599 (comp600 (view-image601 (if record602 (File.603 "/home/r/proj/cortex/render/worm-vision/worm-view")))604 BufferedImage!))606 (set-gravity world Vector3f/ZERO))608 (fn [world _ ]609 (.setLocalTranslation me (.getLocation (.getCamera world)))610 (vision-display611 (map #(% world) vision)612 (if record (File. "/home/r/proj/cortex/render/worm-vision")))613 (fix-display world)))))614 #+end_src616 The world consists of the worm and a flat gray floor. I can shoot red,617 green, blue and white cannonballs at the worm. The worm is initially618 looking down at the floor, and there is no gravity. My perspective619 (the Main View), the worm's perspective (Worm View) and the 4 sensor620 channels that comprise the worm's eye are all saved frame-by-frame to621 disk.623 * Demonstration of Vision624 #+begin_html625 <div class="figure">626 <video controls="controls" width="755">627 <source src="../video/worm-vision.ogg" type="video/ogg"628 preload="none" poster="../images/aurellem-1280x480.png" />629 </video>630 <p>Simulated Vision in a Virtual Environment</p>631 </div>632 #+end_html634 ** Generate the Worm Video from Frames635 #+name: magick2636 #+begin_src clojure637 (ns cortex.video.magick2638 (:import java.io.File)639 (:use clojure.contrib.shell-out))641 (defn images [path]642 (sort (rest (file-seq (File. path)))))644 (def base "/home/r/proj/cortex/render/worm-vision/")646 (defn pics [file]647 (images (str base file)))649 (defn combine-images []650 (let [main-view (pics "main-view")651 worm-view (pics "worm-view")652 blue (pics "0")653 green (pics "1")654 red (pics "2")655 gray (pics "3")656 blender (let [b-pics (pics "blender")]657 (concat b-pics (repeat 9001 (last b-pics))))658 background (repeat 9001 (File. (str base "background.png")))659 targets (map660 #(File. (str base "out/" (format "%07d.png" %)))661 (range 0 (count main-view)))]662 (dorun663 (pmap664 (comp665 (fn [[background main-view worm-view red green blue gray blender target]]666 (println target)667 (sh "convert"668 background669 main-view "-geometry" "+18+17" "-composite"670 worm-view "-geometry" "+677+17" "-composite"671 green "-geometry" "+685+430" "-composite"672 red "-geometry" "+788+430" "-composite"673 blue "-geometry" "+894+430" "-composite"674 gray "-geometry" "+1000+430" "-composite"675 blender "-geometry" "+0+0" "-composite"676 target))677 (fn [& args] (map #(.getCanonicalPath %) args)))678 background main-view worm-view red green blue gray blender targets))))679 #+end_src681 #+begin_src sh :results silent682 cd /home/r/proj/cortex/render/worm-vision683 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg684 #+end_src686 * Headers688 #+name: vision-header689 #+begin_src clojure690 (ns cortex.vision691 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple692 eyes from different positions to observe the same world, and pass693 the observed data to any arbitray function. Automatically reads694 eye-nodes from specially prepared blender files and instantiates695 them in the world as actual eyes."696 {:author "Robert McIntyre"}697 (:use (cortex world sense util))698 (:use clojure.contrib.def)699 (:import com.jme3.post.SceneProcessor)700 (:import (com.jme3.util BufferUtils Screenshots))701 (:import java.nio.ByteBuffer)702 (:import java.awt.image.BufferedImage)703 (:import (com.jme3.renderer ViewPort Camera))704 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))705 (:import com.jme3.renderer.Renderer)706 (:import com.jme3.app.Application)707 (:import com.jme3.texture.FrameBuffer)708 (:import (com.jme3.scene Node Spatial)))709 #+end_src711 #+name: test-header712 #+begin_src clojure713 (ns cortex.test.vision714 (:use (cortex world sense util body vision))715 (:use cortex.test.body)716 (:import java.awt.image.BufferedImage)717 (:import javax.swing.JPanel)718 (:import javax.swing.SwingUtilities)719 (:import java.awt.Dimension)720 (:import javax.swing.JFrame)721 (:import com.jme3.math.ColorRGBA)722 (:import com.jme3.scene.Node)723 (:import com.jme3.math.Vector3f)724 (:import java.io.File)725 (:import (com.aurellem.capture Capture RatchetTimer)))726 #+end_src728 * Onward!729 - As a neat bonus, this idea behind simulated vision also enables one730 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].731 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].733 * Source Listing734 - [[../src/cortex/vision.clj][cortex.vision]]735 - [[../src/cortex/test/vision.clj][cortex.test.vision]]736 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]737 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]738 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>739 - [[http://hg.bortreb.com ][source-repository]]743 * COMMENT Generate Source744 #+begin_src clojure :tangle ../src/cortex/vision.clj745 <<vision-header>>746 <<pipeline-1>>747 <<pipeline-2>>748 <<retina>>749 <<add-eye>>750 <<sensitivity>>751 <<eye-node>>752 <<add-camera>>753 <<kernel>>754 <<main>>755 <<display>>756 #+end_src758 #+begin_src clojure :tangle ../src/cortex/test/vision.clj759 <<test-header>>760 <<test-1>>761 <<test-2>>762 #+end_src764 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj765 <<magick2>>766 #+end_src