cortex: org/vision.org comparison

comparison org/vision.org @ 218:ac46ee4e574a

edits to vision.org

author	Robert McIntyre <rlm@mit.edu>
date	Fri, 10 Feb 2012 12:06:41 -0700
parents	f5ea63245b3b
children	5f14fd7b1288

comparison

equal deleted inserted replaced

-:7bf3e3d8fb26
+:ac46ee4e574a
 Vision is one of the most important senses for humans, so I need to
 build a simulated sense of vision for my AI. I will do this with
 simulated eyes. Each eye can be independely moved and should see its
 own version of the world depending on where it is.
-Making these simulated eyes a reality is fairly simple bacause
+Making these simulated eyes a reality is simple bacause jMonkeyEngine
-jMonkeyEngine already conatains extensive support for multiple views
+already conatains extensive support for multiple views of the same 3D
-of the same 3D simulated world. The reason jMonkeyEngine has this
+simulated world. The reason jMonkeyEngine has this support is because
-support is because the support is necessary to create games with
+the support is necessary to create games with split-screen
-split-screen views. Multiple views are also used to create efficient
+views. Multiple views are also used to create efficient
 pseudo-reflections by rendering the scene from a certain perspective
 and then projecting it back onto a surface in the 3D world.
-#+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye
+#+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
 [[../images/goldeneye-4-player.png]]
 * Brief Description of jMonkeyEngine's Rendering Pipeline
 jMonkeyEngine allows you to create a =ViewPort=, which represents a
 add a =SceneProcessor= that feeds the visual data to any arbitray
 continuation function for further processing.  That continuation
 function may perform both CPU and GPU operations on the data. To make
 this easy for the continuation function, the =SceneProcessor=
 maintains appropriatly sized buffers in RAM to hold the data.  It does
-not do any copying from the GPU to the CPU itself.
+not do any copying from the GPU to the CPU itself because it is a slow
+operation.
 #+name: pipeline-1
 #+begin_src clojure
 (defn vision-pipeline
 "Create a SceneProcessor object which wraps a vision processing
 (cleanup []))))
 #+end_src
 The continuation function given to =(vision-pipeline)= above will be
 given a =Renderer= and three containers for image data. The
-=FrameBuffer= references the GPU image data, but it can not be used
+=FrameBuffer= references the GPU image data, but the pixel data can
-directly on the CPU.  The =ByteBuffer= and =BufferedImage= are
+not be used directly on the CPU.  The =ByteBuffer= and =BufferedImage=
-initially "empty" but are sized to hold to data in the
+are initially "empty" but are sized to hold to data in the
 =FrameBuffer=. I call transfering the GPU image data to the CPU
 structures "mixing" the image data. I have provided three functions to
 do this mixing.
 #+name: pipeline-2
 (let [target (closest-node creature eye)
 [cam-width cam-height] (eye-dimensions eye)
 cam (Camera. cam-width cam-height)
 rot (.getWorldRotation eye)]
 (.setLocation cam (.getWorldTranslation eye))
-(.lookAtDirection cam (.mult rot Vector3f/UNIT_X)
+(.lookAtDirection
-;; this part is consistent with using Z in
+cam                                ; this part is not a mistake and
-;; blender as the UP vector.
+(.mult rot Vector3f/UNIT_X)        ; is consistent with using Z in
-(.mult rot Vector3f/UNIT_Y))
+(.mult rot Vector3f/UNIT_Y))       ; blender as the UP vector.
 (.setFrustumPerspective
 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
 (bind-sense target cam) cam))
 #+end_src
 ** The Retina
 An eye is a surface (the retina) which contains many discrete sensors
-to detect light. These sensors have can have different-light sensing
+to detect light. These sensors have can have different light-sensing
 properties.  In humans, each discrete sensor is sensitive to red,
 blue, green, or gray. These different types of sensors can have
 different spatial distributions along the retina. In humans, there is
 a fovea in the center of the retina which has a very high density of
 color sensors, and a blind spot which has no sensors at all. Sensor
 0xFFFFFF retinal-profile})
 #+end_src
 The numbers that serve as keys in the map determine a sensor's
 relative sensitivity to the channels red, green, and blue. These
-sensitivity values are packed into an integer in the order _RGB in
+sensitivity values are packed into an integer in the order =|_|R|G|B|=
-8-bit fields. The RGB values of a pixel in the image are added
+in 8-bit fields. The RGB values of a pixel in the image are added
 together with these sensitivities as linear weights. Therfore,
 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
 all colors equally (gray).
 For convienence I've defined a few symbols for the more common
 (if-let [eye-map (meta-data eye "eye")]
 (map-vals
 load-image
 (eval (read-string eye-map)))))
 (defn eye-dimensions
-"Returns [width, height] specified in the metadata of the eye"
+"Returns [width, height] determined by the metadata of the eye."
 [#^Spatial eye]
 (let [dimensions
 (map #(vector (.getWidth %) (.getHeight %))
 (vals (retina-sensor-profile eye)))]
 [(apply max (map first dimensions))
 (apply max (map second dimensions))]))
 #+end_src
 * Eye Creation
 First off, get the children of the "eyes" empty node to find all the
 eyes the creature has.
 #+name: eye-node
 #+begin_src clojure
 (defvar
 (.addProcessor (vision-pipeline continuation))
 (.attachScene (.getRootNode world)))))
 #+end_src
-The continuation function registers the viewport with the simulation
+The eye's continuation function should register the viewport with the
-the first time it is called, and uses the CPU to extract the
+simulation the first time it is called, use the CPU to extract the
 appropriate pixels from the rendered image and weight them by each
-sensors sensitivity. I have the option to do this filtering in native
+sensor's sensitivity. I have the option to do this processing in
-code for a slight gain in speed. I could also do it in the GPU for a
+native code for a slight gain in speed. I could also do it in the GPU
-massive gain in speed. =(vision-kernel)= generates a list of such
+for a massive gain in speed. =(vision-kernel)= generates a list of
-continuation functions, one for each channel of the eye.
+such continuation functions, one for each channel of the eye.
 #+name: kernel
 #+begin_src clojure
 (in-ns 'cortex.vision)
 simulation or the simulated senses, but can be annoying.
 =(gen-fix-display)= restores the in-simulation display.
 ** Vision!
-All the hard work has been done, all that remains is to apply
+All the hard work has been done; all that remains is to apply
 =(vision-kernel)= to each eye in the creature and gather the results
 into one list of functions.
 #+name: main
 #+begin_src clojure
 (defn vision!
 "Returns a function which returns visual sensory data when called
-inside a running simulation"
+inside a running simulation."
 [#^Node creature & {skip :skip :or {skip 0}}]
 (reduce
 concat
 (for [eye (eyes creature)]
 (vision-kernel creature eye))))
 simulation in a single creature or for simulating multiple creatures,
 each with their own sense of vision.
 ** Adding Vision to the Worm
-To the worm from the last post, we add a new node that describes its
+To the worm from the last post, I add a new node that describes its
 eyes.
 #+attr_html: width=755
 #+caption: The worm with newly added empty nodes describing a single eye.
 [[../images/worm-with-eye.png]]
 The node highlighted in yellow is the root level "eyes" node.  It has
-a single node, highlighted in orange, which describes a single
+a single child, highlighted in orange, which describes a single
-eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint
+eye. This is the "eye" node. It is placed so that the worm will have
-of the worm.
+an eye located in the center of the flat portion of its lower
+hemispherical section.
+The two nodes which are not highlighted describe the single joint of
+the worm.
 The metadata of the eye-node is:
 #+begin_src clojure :results verbatim :exports both
 (cortex.sense/meta-data
-(.getChild
+(.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
-(.getChild (cortex.test.body/worm)
-"eyes") "eye") "eye")
 #+end_src
 #+results:
 : "(let [retina \"Models/test-creature/retina-small.png\"]
 :     {:all retina :red retina :green retina :blue retina})"
 (map #(% world) vision)
 (File. "/home/r/proj/cortex/render/worm-vision"))
 (fix-display world)))))
 #+end_src
-** Methods to Generate the Worm Video
+The world consists of the worm and a flat gray floor. I can shoot red,
+green, blue and white cannonballs at the worm. The worm is initially
+looking down at the floor, and there is no gravity. My perspective
+(the Main View), the worm's perspective (Worm View) and the 4 sensor
+channels that comprise the worm's eye are all saved frame-by-frame to
+disk.
+* Demonstration of Vision
+#+begin_html
+<div class="figure">
+<video controls="controls" width="755">
+<source src="../video/worm-vision.ogg" type="video/ogg"
+	  preload="none" poster="../images/aurellem-1280x480.png" />
+</video>
+<p>Simulated Vision in a Virtual Environment</p>
+</div>
+#+end_html
+** Generate the Worm Video from Frames
 #+name: magick2
 #+begin_src clojure
 (ns cortex.video.magick2
 (:import java.io.File)
 (:use clojure.contrib.shell-out))
 #+begin_src sh :results silent
 cd /home/r/proj/cortex/render/worm-vision
 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
 #+end_src
-* Demonstration of Vision
-#+begin_html
-<div class="figure">
-<video controls="controls" width="755">
-<source src="../video/worm-vision.ogg" type="video/ogg"
-	  preload="none" poster="../images/aurellem-1280x480.png" />
-</video>
-<p>Simulated Vision in a Virtual Environment</p>
-</div>
-#+end_html
 * Headers
 #+name: vision-header
 #+begin_src clojure
 (ns cortex.vision

Mercurial > cortex

comparison org/vision.org @ 218:ac46ee4e574a