# HG changeset patch # User Robert McIntyre # Date 1328800270 25200 # Node ID 3199637201798a3e382f1fc605d488b14fcc91de # Parent 8e9825c389415ac74fdf7b6cd9a1f5238d19d433 fleshing out vision diff -r 8e9825c38941 -r 319963720179 org/vision.org --- a/org/vision.org Thu Feb 09 07:39:21 2012 -0700 +++ b/org/vision.org Thu Feb 09 08:11:10 2012 -0700 @@ -26,64 +26,33 @@ #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye [[../images/goldeneye-4-player.png]] +* Brief Description of jMonkeyEngine's Rendering Pipeline +jMonkeyEngine allows you to create a =ViewPort=, which represents a +view of the simulated world. You can create as many of these as you +want. Every frame, the =RenderManager= iterates through each +=ViewPort=, rendering the scene in the GPU. For each =ViewPort= there +is a =FrameBuffer= which represents the rendered image in the GPU. -Make the continuation in scene-processor take FrameBuffer, -byte-buffer, BufferedImage already sized to the correct -dimensions. the continuation will decide wether to "mix" them -into the BufferedImage, lazily ignore them, or mix them halfway -and call c/graphics card routines. +Each =ViewPort= can have any number of attached =SceneProcessor= +objects, which are called every time a new frame is rendered. A +=SceneProcessor= recieves a =FrameBuffer= and can do whatever it wants +to the data. Often this consists of invoking GPU specific operations +on the rendered image. The =SceneProcessor= can also copy the GPU +image data to RAM and process it with the CPU. -(vision creature) will take an optional :skip argument which will -inform the continuations in scene processor to skip the given -number of cycles 0 means that no cycles will be skipped. +* The Vision Pipeline -(vision creature) will return [init-functions sensor-functions]. -The init-functions are each single-arg functions that take the -world and register the cameras and must each be called before the -corresponding sensor-functions. Each init-function returns the -viewport for that eye which can be manipulated, saved, etc. Each -sensor-function is a thunk and will return data in the same -format as the tactile-sensor functions the structure is -[topology, sensor-data]. Internally, these sensor-functions -maintain a reference to sensor-data which is periodically updated -by the continuation function established by its init-function. -They can be queried every cycle, but their information may not -necessairly be different every cycle. - -Each eye in the creature in blender will work the same way as -joints -- a zero dimensional object with no geometry whose local -coordinate system determines the orientation of the resulting -eye. All eyes will have a parent named "eyes" just as all joints -have a parent named "joints". The resulting camera will be a -ChaseCamera or a CameraNode bound to the geo that is closest to -the eye marker. The eye marker will contain the metadata for the -eye, and will be moved by it's bound geometry. The dimensions of -the eye's camera are equal to the dimensions of the eye's "UV" -map. - -#+name: eyes -#+begin_src clojure -(ns cortex.vision - "Simulate the sense of vision in jMonkeyEngine3. Enables multiple - eyes from different positions to observe the same world, and pass - the observed data to any arbitray function. Automatically reads - eye-nodes from specially prepared blender files and instanttiates - them in the world as actual eyes." - {:author "Robert McIntyre"} - (:use (cortex world sense util)) - (:use clojure.contrib.def) - (:import com.jme3.post.SceneProcessor) - (:import (com.jme3.util BufferUtils Screenshots)) - (:import java.nio.ByteBuffer) - (:import java.awt.image.BufferedImage) - (:import (com.jme3.renderer ViewPort Camera)) - (:import com.jme3.math.ColorRGBA) - (:import com.jme3.renderer.Renderer) - (:import com.jme3.app.Application) - (:import com.jme3.texture.FrameBuffer) - (:import (com.jme3.scene Node Spatial))) - +Each eye in the simulated creature needs it's own =ViewPort= so that +it can see the world from its own perspective. To this =ViewPort=, I +add a =SceneProcessor= that feeds the visual data to any arbitra +continuation function for further processing. That continuation +function may perform both CPU and GPU operations on the data. To make +this easy for the continuation function, the =SceneProcessor= +maintains appropriatly sized buffers in RAM to hold the data. It does +not do any copying from the GPU to the CPU itself. +#+name: pipeline-1 +#+begin_src clojure (defn vision-pipeline "Create a SceneProcessor object which wraps a vision processing continuation function. The continuation is a function that takes @@ -115,7 +84,19 @@ (.clear @byte-buffer) (continuation @renderer fb @byte-buffer @image)) (cleanup [])))) - +#+end_src + +The continuation function given to =(vision-pipeline)= above will be +given a =Renderer= and three containers for image data. The +=FrameBuffer= references the GPU image data, but it can not be used +directly on the CPU. The =ByteBuffer= and =BufferedImage= are +initially "empty" but are sized to hold to data in the +=FrameBuffer=. I call transfering the GPU image data to the CPU +structures "mixing" the image data. I have provided three functions to +do this mixing. + +#+name: pipeline-2 +#+begin_src clojure (defn frameBuffer->byteBuffer! "Transfer the data in the graphics card (Renderer, FrameBuffer) to the CPU (ByteBuffer)." @@ -134,7 +115,48 @@ [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi] (byteBuffer->bufferedImage! (frameBuffer->byteBuffer! r fb bb) bi)) +#+end_src +Note that it is possible to write vision processing algorithms +entirely in terms of =BufferedImage= inputs. Just compose that +=BufferedImage= algorithm with =(BufferedImage!)=. However, a vision +processing algorithm that is entirely hosted on the GPU does not have +to pay for this convienence. + + +* Physical Eyes + +The vision pipeline described above only deals with +Each eye in the creature in blender will work the same way as +joints -- a zero dimensional object with no geometry whose local +coordinate system determines the orientation of the resulting +eye. All eyes will have a parent named "eyes" just as all joints +have a parent named "joints". The resulting camera will be a +ChaseCamera or a CameraNode bound to the geo that is closest to +the eye marker. The eye marker will contain the metadata for the +eye, and will be moved by it's bound geometry. The dimensions of +the eye's camera are equal to the dimensions of the eye's "UV" +map. + +(vision creature) will take an optional :skip argument which will +inform the continuations in scene processor to skip the given +number of cycles 0 means that no cycles will be skipped. + +(vision creature) will return [init-functions sensor-functions]. +The init-functions are each single-arg functions that take the +world and register the cameras and must each be called before the +corresponding sensor-functions. Each init-function returns the +viewport for that eye which can be manipulated, saved, etc. Each +sensor-function is a thunk and will return data in the same +format as the tactile-sensor functions the structure is +[topology, sensor-data]. Internally, these sensor-functions +maintain a reference to sensor-data which is periodically updated +by the continuation function established by its init-function. +They can be queried every cycle, but their information may not +necessairly be different every cycle. + + +#+begin_src clojure (defn add-camera! "Add a camera to the world, calling continuation on every frame produced." @@ -326,8 +348,28 @@ (.rotate candy (* tpf 0.2) 0 0))))) #+end_src -#+results: test-vision -: #'cortex.test.vision/test-two-eyes +#+name: vision-header +#+begin_src clojure +(ns cortex.vision + "Simulate the sense of vision in jMonkeyEngine3. Enables multiple + eyes from different positions to observe the same world, and pass + the observed data to any arbitray function. Automatically reads + eye-nodes from specially prepared blender files and instanttiates + them in the world as actual eyes." + {:author "Robert McIntyre"} + (:use (cortex world sense util)) + (:use clojure.contrib.def) + (:import com.jme3.post.SceneProcessor) + (:import (com.jme3.util BufferUtils Screenshots)) + (:import java.nio.ByteBuffer) + (:import java.awt.image.BufferedImage) + (:import (com.jme3.renderer ViewPort Camera)) + (:import com.jme3.math.ColorRGBA) + (:import com.jme3.renderer.Renderer) + (:import com.jme3.app.Application) + (:import com.jme3.texture.FrameBuffer) + (:import (com.jme3.scene Node Spatial))) +#+end_src The example code will create two videos of the same rotating object from different angles. It can be used both for stereoscopic vision