1.1 --- a/org/vision.org	Thu Feb 09 08:11:10 2012 -0700
     1.2 +++ b/org/vision.org	Thu Feb 09 09:04:17 2012 -0700
     1.3 @@ -45,12 +45,13 @@
     1.4  
     1.5  Each eye in the simulated creature needs it's own =ViewPort= so that
     1.6  it can see the world from its own perspective. To this =ViewPort=, I
     1.7 -add a =SceneProcessor= that feeds the visual data to any arbitra
     1.8 +add a =SceneProcessor= that feeds the visual data to any arbitray
     1.9  continuation function for further processing.  That continuation
    1.10  function may perform both CPU and GPU operations on the data. To make
    1.11  this easy for the continuation function, the =SceneProcessor=
    1.12  maintains appropriatly sized buffers in RAM to hold the data.  It does
    1.13  not do any copying from the GPU to the CPU itself.
    1.14 +
    1.15  #+name: pipeline-1
    1.16  #+begin_src clojure
    1.17  (defn vision-pipeline
    1.18 @@ -123,20 +124,7 @@
    1.19  processing algorithm that is entirely hosted on the GPU does not have
    1.20  to pay for this convienence.
    1.21  
    1.22 -
    1.23 -* Physical Eyes
    1.24 -
    1.25 -The vision pipeline described above only deals with 
    1.26 -Each eye in the creature in blender will work the same way as
    1.27 -joints -- a zero dimensional object with no geometry whose local
    1.28 -coordinate system determines the orientation of the resulting
    1.29 -eye. All eyes will have a parent named "eyes" just as all joints
    1.30 -have a parent named "joints". The resulting camera will be a
    1.31 -ChaseCamera or a CameraNode bound to the geo that is closest to
    1.32 -the eye marker. The eye marker will contain the metadata for the
    1.33 -eye, and will be moved by it's bound geometry. The dimensions of
    1.34 -the eye's camera are equal to the dimensions of the eye's "UV"
    1.35 -map.
    1.36 +* COMMENT asdasd 
    1.37  
    1.38  (vision creature) will take an optional :skip argument which will
    1.39  inform the continuations in scene processor to skip the given
    1.40 @@ -156,6 +144,150 @@
    1.41  necessairly be different every cycle.
    1.42  
    1.43  
    1.44 +
    1.45 +* Physical Eyes
    1.46 +
    1.47 +The vision pipeline described above handles the flow of rendered
    1.48 +images. Now, we need simulated eyes to serve as the source of these
    1.49 +images. 
    1.50 +
    1.51 +An eye is described in blender in the same way as a joint. They are
    1.52 +zero dimensional empty objects with no geometry whose local coordinate
    1.53 +system determines the orientation of the resulting eye. All eyes are
    1.54 +childern of a parent node named "eyes" just as all joints have a
    1.55 +parent named "joints". An eye binds to the nearest physical object
    1.56 +with =(bind-sense=).
    1.57 +
    1.58 +#+name: add-eye
    1.59 +#+begin_src clojure
    1.60 +(defn add-eye!
    1.61 +  "Create a Camera centered on the current position of 'eye which
    1.62 +   follows the closest physical node in 'creature and sends visual
    1.63 +   data to 'continuation."
    1.64 +  [#^Node creature #^Spatial eye]
    1.65 +  (let [target (closest-node creature eye)
    1.66 +        [cam-width cam-height] (eye-dimensions eye)
    1.67 +        cam (Camera. cam-width cam-height)]
    1.68 +    (.setLocation cam (.getWorldTranslation eye))
    1.69 +    (.setRotation cam (.getWorldRotation eye))
    1.70 +    (.setFrustumPerspective
    1.71 +     cam 45 (/ (.getWidth cam) (.getHeight cam))
    1.72 +     1 1000)
    1.73 +    (bind-sense target cam)
    1.74 +    cam))
    1.75 +#+end_src
    1.76 +
    1.77 +Here, the camera is created based on metadata on the eye-node and
    1.78 +attached to the nearest physical object with =(bind-sense)=
    1.79 +
    1.80 +
    1.81 +** The Retina
    1.82 +
    1.83 +An eye is a surface (the retina) which contains many discrete sensors
    1.84 +to detect light. These sensors have can have different-light sensing
    1.85 +properties.  In humans, each discrete sensor is sensitive to red,
    1.86 +blue, green, or gray. These different types of sensors can have
    1.87 +different spatial distributions along the retina. In humans, there is
    1.88 +a fovea in the center of the retina which has a very high density of
    1.89 +color sensors, and a blind spot which has no sensors at all. Sensor
    1.90 +density decreases in proportion to distance from the retina.
    1.91 +
    1.92 +I want to be able to model any retinal configuration, so my eye-nodes
    1.93 +in blender contain metadata pointing to images that describe the
    1.94 +percise position of the individual sensors using white pixels. The
    1.95 +meta-data also describes the percise sensitivity to light that the
    1.96 +sensors described in the image have.  An eye can contain any number of
    1.97 +these images. For example, the metadata for an eye might look like
    1.98 +this:
    1.99 +
   1.100 +#+begin_src clojure
   1.101 +{0xFF0000 "Models/test-creature/retina-small.png"}
   1.102 +#+end_src
   1.103 +
   1.104 +#+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
   1.105 +[[../assets/Models/test-creature/retina-small.png]]
   1.106 +
   1.107 +Together, the number 0xFF0000 and the image image above describe the
   1.108 +placement of red-sensitive sensory elements.
   1.109 +
   1.110 +Meta-data to very crudely approximate a human eye might be something
   1.111 +like this:
   1.112 +
   1.113 +#+begin_src clojure
   1.114 +(let [retinal-profile "Models/test-creature/retina-small.png"]
   1.115 +  {0xFF0000 retinal-profile
   1.116 +   0x00FF00 retinal-profile
   1.117 +   0x0000FF retinal-profile
   1.118 +   0xFFFFFF retinal-profile})
   1.119 +#+end_src
   1.120 +
   1.121 +The numbers that serve as keys in the map determine a sensor's
   1.122 +relative sensitivity to the channels red, green, and blue. These
   1.123 +sensitivity values are packed into an integer in the order _RGB in
   1.124 +8-bit fields. The RGB values of a pixel in the image are added
   1.125 +together with these sensitivities as linear weights. Therfore,
   1.126 +0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
   1.127 +all colors equally (gray).
   1.128 +
   1.129 +For convienence I've defined a few symbols for the more common
   1.130 +sensitivity values.
   1.131 +
   1.132 +#+name: sensitivity
   1.133 +#+begin_src clojure
   1.134 +(defvar sensitivity-presets
   1.135 +  {:all    0xFFFFFF
   1.136 +   :red    0xFF0000
   1.137 +   :blue   0x0000FF
   1.138 +   :green  0x00FF00}
   1.139 +  "Retinal sensitivity presets for sensors that extract one channel
   1.140 +   (:red :blue :green) or average all channels (:gray)")
   1.141 +#+end_src
   1.142 +
   1.143 +** Metadata Processing
   1.144 +
   1.145 +=(retina-sensor-profile)= extracts a map from the eye-node in the same
   1.146 +format as the example maps above.  =(eye-dimensions)= finds the
   1.147 +dimansions of the smallest image required to contain all the retinal
   1.148 +sensor maps.
   1.149 +
   1.150 +#+begin_src clojure
   1.151 +(defn retina-sensor-profile
   1.152 +  "Return a map of pixel sensitivity numbers to BufferedImages
   1.153 +   describing the distribution of light-sensitive components of this
   1.154 +   eye. :red, :green, :blue, :gray are already defined as extracting
   1.155 +   the red, green, blue, and average components respectively."
   1.156 +   [#^Spatial eye]
   1.157 +   (if-let [eye-map (meta-data eye "eye")]
   1.158 +     (map-vals
   1.159 +      load-image
   1.160 +      (eval (read-string eye-map)))))
   1.161 +
   1.162 +(defn eye-dimensions
   1.163 +  "Returns [width, height] specified in the metadata of the eye"
   1.164 +  [#^Spatial eye]
   1.165 +  (let [dimensions
   1.166 +          (map #(vector (.getWidth %) (.getHeight %))
   1.167 +               (vals (retina-sensor-profile eye)))]
   1.168 +    [(apply max (map first dimensions))
   1.169 +     (apply max (map second dimensions))]))
   1.170 +#+end_src
   1.171 +
   1.172 +
   1.173 +* Eye Creation 
   1.174 +
   1.175 +First off, get the children of the "eyes" empty node to find all the
   1.176 +eyes the creature has.
   1.177 +
   1.178 +#+begin_src clojure
   1.179 +(defvar 
   1.180 +  ^{:arglists '([creature])}
   1.181 +  eyes
   1.182 +  (sense-nodes "eyes")
   1.183 +  "Return the children of the creature's \"eyes\" node.")
   1.184 +#+end_src
   1.185 +
   1.186 +Then, 
   1.187 +
   1.188  #+begin_src clojure
   1.189  (defn add-camera!
   1.190    "Add a camera to the world, calling continuation on every frame
   1.191 @@ -171,56 +303,9 @@
   1.192        (.addProcessor (vision-pipeline continuation))
   1.193        (.attachScene (.getRootNode world)))))
   1.194  
   1.195 -(defn retina-sensor-profile
   1.196 -  "Return a map of pixel selection functions to BufferedImages
   1.197 -   describing the distribution of light-sensitive components of this
   1.198 -   eye. Each function creates an integer from the rgb values found in
   1.199 -   the pixel. :red, :green, :blue, :gray are already defined as
   1.200 -   extracting the red, green, blue, and average components
   1.201 -   respectively."
   1.202 -   [#^Spatial eye]
   1.203 -   (if-let [eye-map (meta-data eye "eye")]
   1.204 -     (map-vals
   1.205 -      load-image
   1.206 -      (eval (read-string eye-map)))))
   1.207  
   1.208 -(defn eye-dimensions
   1.209 -  "Returns [width, height] specified in the metadata of the eye"
   1.210 -  [#^Spatial eye]
   1.211 -  (let [dimensions
   1.212 -          (map #(vector (.getWidth %) (.getHeight %))
   1.213 -               (vals (retina-sensor-profile eye)))]
   1.214 -    [(apply max (map first dimensions))
   1.215 -     (apply max (map second dimensions))]))
   1.216  
   1.217 -(defvar 
   1.218 -  ^{:arglists '([creature])}
   1.219 -  eyes
   1.220 -  (sense-nodes "eyes")
   1.221 -  "Return the children of the creature's \"eyes\" node.")
   1.222  
   1.223 -(defn add-eye!
   1.224 -  "Create a Camera centered on the current position of 'eye which
   1.225 -   follows the closest physical node in 'creature and sends visual
   1.226 -   data to 'continuation."
   1.227 -  [#^Node creature #^Spatial eye]
   1.228 -  (let [target (closest-node creature eye)
   1.229 -        [cam-width cam-height] (eye-dimensions eye)
   1.230 -        cam (Camera. cam-width cam-height)]
   1.231 -    (.setLocation cam (.getWorldTranslation eye))
   1.232 -    (.setRotation cam (.getWorldRotation eye))
   1.233 -    (.setFrustumPerspective
   1.234 -     cam 45 (/ (.getWidth cam) (.getHeight cam))
   1.235 -     1 1000)
   1.236 -    (bind-sense target cam)
   1.237 -    cam))
   1.238 -
   1.239 -(defvar color-channel-presets
   1.240 -  {:all    0xFFFFFF
   1.241 -   :red    0xFF0000
   1.242 -   :blue   0x0000FF
   1.243 -   :green  0x00FF00}
   1.244 -  "Bitmasks for common RGB color channels")
   1.245  
   1.246  (defn vision-fn
   1.247    "Returns a list of functions, each of which will return a color
   1.248 @@ -249,7 +334,7 @@
   1.249         (fn [[key image]]
   1.250           (let [whites (white-coordinates image)
   1.251                 topology (vec (collapse whites))
   1.252 -               mask (color-channel-presets key)]
   1.253 +               mask (color-channel-presets key key)]
   1.254             (fn [world]
   1.255               (register-eye! world)
   1.256               (vector