# HG changeset patch # User Robert McIntyre # Date 1328900801 25200 # Node ID ac46ee4e574a309afe78ac5dccd6a86ff6c8b89c # Parent 7bf3e3d8fb2656c642bafa4903bf97035017f5d5 edits to vision.org diff -r 7bf3e3d8fb26 -r ac46ee4e574a org/vision.org --- a/org/vision.org Fri Feb 10 11:37:19 2012 -0700 +++ b/org/vision.org Fri Feb 10 12:06:41 2012 -0700 @@ -14,15 +14,15 @@ simulated eyes. Each eye can be independely moved and should see its own version of the world depending on where it is. -Making these simulated eyes a reality is fairly simple bacause -jMonkeyEngine already conatains extensive support for multiple views -of the same 3D simulated world. The reason jMonkeyEngine has this -support is because the support is necessary to create games with -split-screen views. Multiple views are also used to create efficient +Making these simulated eyes a reality is simple bacause jMonkeyEngine +already conatains extensive support for multiple views of the same 3D +simulated world. The reason jMonkeyEngine has this support is because +the support is necessary to create games with split-screen +views. Multiple views are also used to create efficient pseudo-reflections by rendering the scene from a certain perspective and then projecting it back onto a surface in the 3D world. -#+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye +#+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views. [[../images/goldeneye-4-player.png]] * Brief Description of jMonkeyEngine's Rendering Pipeline @@ -49,7 +49,8 @@ function may perform both CPU and GPU operations on the data. To make this easy for the continuation function, the =SceneProcessor= maintains appropriatly sized buffers in RAM to hold the data. It does -not do any copying from the GPU to the CPU itself. +not do any copying from the GPU to the CPU itself because it is a slow +operation. #+name: pipeline-1 #+begin_src clojure @@ -88,9 +89,9 @@ The continuation function given to =(vision-pipeline)= above will be given a =Renderer= and three containers for image data. The -=FrameBuffer= references the GPU image data, but it can not be used -directly on the CPU. The =ByteBuffer= and =BufferedImage= are -initially "empty" but are sized to hold to data in the +=FrameBuffer= references the GPU image data, but the pixel data can +not be used directly on the CPU. The =ByteBuffer= and =BufferedImage= +are initially "empty" but are sized to hold to data in the =FrameBuffer=. I call transfering the GPU image data to the CPU structures "mixing" the image data. I have provided three functions to do this mixing. @@ -172,10 +173,10 @@ cam (Camera. cam-width cam-height) rot (.getWorldRotation eye)] (.setLocation cam (.getWorldTranslation eye)) - (.lookAtDirection cam (.mult rot Vector3f/UNIT_X) - ;; this part is consistent with using Z in - ;; blender as the UP vector. - (.mult rot Vector3f/UNIT_Y)) + (.lookAtDirection + cam ; this part is not a mistake and + (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in + (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector. (.setFrustumPerspective cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000) (bind-sense target cam) cam)) @@ -188,7 +189,7 @@ ** The Retina An eye is a surface (the retina) which contains many discrete sensors -to detect light. These sensors have can have different-light sensing +to detect light. These sensors have can have different light-sensing properties. In humans, each discrete sensor is sensitive to red, blue, green, or gray. These different types of sensors can have different spatial distributions along the retina. In humans, there is @@ -227,8 +228,8 @@ The numbers that serve as keys in the map determine a sensor's relative sensitivity to the channels red, green, and blue. These -sensitivity values are packed into an integer in the order _RGB in -8-bit fields. The RGB values of a pixel in the image are added +sensitivity values are packed into an integer in the order =|_|R|G|B|= +in 8-bit fields. The RGB values of a pixel in the image are added together with these sensitivities as linear weights. Therfore, 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to all colors equally (gray). @@ -267,8 +268,8 @@ load-image (eval (read-string eye-map))))) -(defn eye-dimensions - "Returns [width, height] specified in the metadata of the eye" +(defn eye-dimensions + "Returns [width, height] determined by the metadata of the eye." [#^Spatial eye] (let [dimensions (map #(vector (.getWidth %) (.getHeight %)) @@ -277,9 +278,7 @@ (apply max (map second dimensions))])) #+end_src - * Eye Creation - First off, get the children of the "eyes" empty node to find all the eyes the creature has. #+name: eye-node @@ -312,13 +311,13 @@ #+end_src -The continuation function registers the viewport with the simulation -the first time it is called, and uses the CPU to extract the +The eye's continuation function should register the viewport with the +simulation the first time it is called, use the CPU to extract the appropriate pixels from the rendered image and weight them by each -sensors sensitivity. I have the option to do this filtering in native -code for a slight gain in speed. I could also do it in the GPU for a -massive gain in speed. =(vision-kernel)= generates a list of such -continuation functions, one for each channel of the eye. +sensor's sensitivity. I have the option to do this processing in +native code for a slight gain in speed. I could also do it in the GPU +for a massive gain in speed. =(vision-kernel)= generates a list of +such continuation functions, one for each channel of the eye. #+name: kernel #+begin_src clojure @@ -408,7 +407,7 @@ ** Vision! -All the hard work has been done, all that remains is to apply +All the hard work has been done; all that remains is to apply =(vision-kernel)= to each eye in the creature and gather the results into one list of functions. @@ -416,7 +415,7 @@ #+begin_src clojure (defn vision! "Returns a function which returns visual sensory data when called - inside a running simulation" + inside a running simulation." [#^Node creature & {skip :skip :or {skip 0}}] (reduce concat @@ -517,7 +516,7 @@ ** Adding Vision to the Worm -To the worm from the last post, we add a new node that describes its +To the worm from the last post, I add a new node that describes its eyes. #+attr_html: width=755 @@ -525,17 +524,19 @@ [[../images/worm-with-eye.png]] The node highlighted in yellow is the root level "eyes" node. It has -a single node, highlighted in orange, which describes a single -eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint -of the worm. +a single child, highlighted in orange, which describes a single +eye. This is the "eye" node. It is placed so that the worm will have +an eye located in the center of the flat portion of its lower +hemispherical section. + +The two nodes which are not highlighted describe the single joint of +the worm. The metadata of the eye-node is: #+begin_src clojure :results verbatim :exports both (cortex.sense/meta-data - (.getChild - (.getChild (cortex.test.body/worm) - "eyes") "eye") "eye") + (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye") #+end_src #+results: @@ -608,7 +609,25 @@ (fix-display world))))) #+end_src -** Methods to Generate the Worm Video +The world consists of the worm and a flat gray floor. I can shoot red, +green, blue and white cannonballs at the worm. The worm is initially +looking down at the floor, and there is no gravity. My perspective +(the Main View), the worm's perspective (Worm View) and the 4 sensor +channels that comprise the worm's eye are all saved frame-by-frame to +disk. + +* Demonstration of Vision +#+begin_html +
+ +

Simulated Vision in a Virtual Environment

+
+#+end_html + +** Generate the Worm Video from Frames #+name: magick2 #+begin_src clojure (ns cortex.video.magick2 @@ -660,17 +679,6 @@ ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg #+end_src -* Demonstration of Vision -#+begin_html -
- -

Simulated Vision in a Virtual Environment

-
-#+end_html - * Headers #+name: vision-header