cortex: org/vision.org comparison

comparison org/vision.org @ 214:01d3e9855ef9

saving progress, time to sleep.....

author	Robert McIntyre <rlm@mit.edu>
date	Thu, 09 Feb 2012 09:04:17 -0700
parents	319963720179
children	f283c62bd212

comparison

equal deleted inserted replaced

-:319963720179
+:01d3e9855ef9
 * The Vision Pipeline
 Each eye in the simulated creature needs it's own =ViewPort= so that
 it can see the world from its own perspective. To this =ViewPort=, I
-add a =SceneProcessor= that feeds the visual data to any arbitra
+add a =SceneProcessor= that feeds the visual data to any arbitray
 continuation function for further processing.  That continuation
 function may perform both CPU and GPU operations on the data. To make
 this easy for the continuation function, the =SceneProcessor=
 maintains appropriatly sized buffers in RAM to hold the data.  It does
 not do any copying from the GPU to the CPU itself.
 #+name: pipeline-1
 #+begin_src clojure
 (defn vision-pipeline
 "Create a SceneProcessor object which wraps a vision processing
 continuation function. The continuation is a function that takes
 entirely in terms of =BufferedImage= inputs. Just compose that
 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
 processing algorithm that is entirely hosted on the GPU does not have
 to pay for this convienence.
+* COMMENT asdasd
-* Physical Eyes
-The vision pipeline described above only deals with
-Each eye in the creature in blender will work the same way as
-joints -- a zero dimensional object with no geometry whose local
-coordinate system determines the orientation of the resulting
-eye. All eyes will have a parent named "eyes" just as all joints
-have a parent named "joints". The resulting camera will be a
-ChaseCamera or a CameraNode bound to the geo that is closest to
-the eye marker. The eye marker will contain the metadata for the
-eye, and will be moved by it's bound geometry. The dimensions of
-the eye's camera are equal to the dimensions of the eye's "UV"
-map.
 (vision creature) will take an optional :skip argument which will
 inform the continuations in scene processor to skip the given
 number of cycles 0 means that no cycles will be skipped.
 by the continuation function established by its init-function.
 They can be queried every cycle, but their information may not
 necessairly be different every cycle.
-#+begin_src clojure
-(defn add-camera!
+* Physical Eyes
-"Add a camera to the world, calling continuation on every frame
-produced."
+The vision pipeline described above handles the flow of rendered
-[#^Application world camera continuation]
+images. Now, we need simulated eyes to serve as the source of these
-(let [width (.getWidth camera)
+images.
-	height (.getHeight camera)
-	render-manager (.getRenderManager world)
+An eye is described in blender in the same way as a joint. They are
-	viewport (.createMainView render-manager "eye-view" camera)]
+zero dimensional empty objects with no geometry whose local coordinate
-(doto viewport
+system determines the orientation of the resulting eye. All eyes are
-(.setClearFlags true true true)
+childern of a parent node named "eyes" just as all joints have a
-(.setBackgroundColor ColorRGBA/Black)
+parent named "joints". An eye binds to the nearest physical object
-(.addProcessor (vision-pipeline continuation))
+with =(bind-sense=).
-(.attachScene (.getRootNode world)))))
+#+name: add-eye
-(defn retina-sensor-profile
+#+begin_src clojure
-"Return a map of pixel selection functions to BufferedImages
-describing the distribution of light-sensitive components of this
-eye. Each function creates an integer from the rgb values found in
-the pixel. :red, :green, :blue, :gray are already defined as
-extracting the red, green, blue, and average components
-respectively."
-[#^Spatial eye]
-(if-let [eye-map (meta-data eye "eye")]
-(map-vals
-load-image
-(eval (read-string eye-map)))))
-(defn eye-dimensions
-"Returns [width, height] specified in the metadata of the eye"
-[#^Spatial eye]
-(let [dimensions
-(map #(vector (.getWidth %) (.getHeight %))
-(vals (retina-sensor-profile eye)))]
-[(apply max (map first dimensions))
-(apply max (map second dimensions))]))
-(defvar
-^{:arglists '([creature])}
-eyes
-(sense-nodes "eyes")
-"Return the children of the creature's \"eyes\" node.")
 (defn add-eye!
 "Create a Camera centered on the current position of 'eye which
 follows the closest physical node in 'creature and sends visual
 data to 'continuation."
 [#^Node creature #^Spatial eye]
 (.setFrustumPerspective
 cam 45 (/ (.getWidth cam) (.getHeight cam))
 1 1000)
 (bind-sense target cam)
 cam))
+#+end_src
-(defvar color-channel-presets
+Here, the camera is created based on metadata on the eye-node and
+attached to the nearest physical object with =(bind-sense)=
+** The Retina
+An eye is a surface (the retina) which contains many discrete sensors
+to detect light. These sensors have can have different-light sensing
+properties.  In humans, each discrete sensor is sensitive to red,
+blue, green, or gray. These different types of sensors can have
+different spatial distributions along the retina. In humans, there is
+a fovea in the center of the retina which has a very high density of
+color sensors, and a blind spot which has no sensors at all. Sensor
+density decreases in proportion to distance from the retina.
+I want to be able to model any retinal configuration, so my eye-nodes
+in blender contain metadata pointing to images that describe the
+percise position of the individual sensors using white pixels. The
+meta-data also describes the percise sensitivity to light that the
+sensors described in the image have.  An eye can contain any number of
+these images. For example, the metadata for an eye might look like
+this:
+#+begin_src clojure
+{0xFF0000 "Models/test-creature/retina-small.png"}
+#+end_src
+#+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
+[[../assets/Models/test-creature/retina-small.png]]
+Together, the number 0xFF0000 and the image image above describe the
+placement of red-sensitive sensory elements.
+Meta-data to very crudely approximate a human eye might be something
+like this:
+#+begin_src clojure
+(let [retinal-profile "Models/test-creature/retina-small.png"]
+{0xFF0000 retinal-profile
+0x00FF00 retinal-profile
+0x0000FF retinal-profile
+0xFFFFFF retinal-profile})
+#+end_src
+The numbers that serve as keys in the map determine a sensor's
+relative sensitivity to the channels red, green, and blue. These
+sensitivity values are packed into an integer in the order _RGB in
+8-bit fields. The RGB values of a pixel in the image are added
+together with these sensitivities as linear weights. Therfore,
+0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
+all colors equally (gray).
+For convienence I've defined a few symbols for the more common
+sensitivity values.
+#+name: sensitivity
+#+begin_src clojure
+(defvar sensitivity-presets
 {:all    0xFFFFFF
 :red    0xFF0000
 :blue   0x0000FF
 :green  0x00FF00}
-"Bitmasks for common RGB color channels")
+"Retinal sensitivity presets for sensors that extract one channel
+(:red :blue :green) or average all channels (:gray)")
+#+end_src
+** Metadata Processing
+=(retina-sensor-profile)= extracts a map from the eye-node in the same
+format as the example maps above.  =(eye-dimensions)= finds the
+dimansions of the smallest image required to contain all the retinal
+sensor maps.
+#+begin_src clojure
+(defn retina-sensor-profile
+"Return a map of pixel sensitivity numbers to BufferedImages
+describing the distribution of light-sensitive components of this
+eye. :red, :green, :blue, :gray are already defined as extracting
+the red, green, blue, and average components respectively."
+[#^Spatial eye]
+(if-let [eye-map (meta-data eye "eye")]
+(map-vals
+load-image
+(eval (read-string eye-map)))))
+(defn eye-dimensions
+"Returns [width, height] specified in the metadata of the eye"
+[#^Spatial eye]
+(let [dimensions
+(map #(vector (.getWidth %) (.getHeight %))
+(vals (retina-sensor-profile eye)))]
+[(apply max (map first dimensions))
+(apply max (map second dimensions))]))
+#+end_src
+* Eye Creation
+First off, get the children of the "eyes" empty node to find all the
+eyes the creature has.
+#+begin_src clojure
+(defvar
+^{:arglists '([creature])}
+eyes
+(sense-nodes "eyes")
+"Return the children of the creature's \"eyes\" node.")
+#+end_src
+Then,
+#+begin_src clojure
+(defn add-camera!
+"Add a camera to the world, calling continuation on every frame
+produced."
+[#^Application world camera continuation]
+(let [width (.getWidth camera)
+	height (.getHeight camera)
+	render-manager (.getRenderManager world)
+	viewport (.createMainView render-manager "eye-view" camera)]
+(doto viewport
+(.setClearFlags true true true)
+(.setBackgroundColor ColorRGBA/Black)
+(.addProcessor (vision-pipeline continuation))
+(.attachScene (.getRootNode world)))))
 (defn vision-fn
 "Returns a list of functions, each of which will return a color
 channel's worth of visual information when called inside a running
 simulation."
 (vec
 (map
 (fn [[key image]]
 (let [whites (white-coordinates image)
 topology (vec (collapse whites))
-mask (color-channel-presets key)]
+mask (color-channel-presets key key)]
 (fn [world]
 (register-eye! world)
 (vector
 topology
 (vec

Mercurial > cortex

comparison org/vision.org @ 214:01d3e9855ef9