Mercurial > cortex

annotate org/vision.org @ 237:02b2e6f3fb43

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
still not happy with collapse, but continuing on touch for now.
author Robert McIntyre <rlm@mit.edu>
date Sun, 12 Feb 2012 10:45:38 -0700
parents 3c9724c8d86b
children 0e85237d27a7
rev   line source
rlm@34 1 #+title: Simulated Sense of Sight
rlm@23 2 #+author: Robert McIntyre
rlm@23 3 #+email: rlm@mit.edu
rlm@38 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
rlm@34 5 #+keywords: computer vision, jMonkeyEngine3, clojure
rlm@23 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@23 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@23 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@23 9
rlm@194 10 * Vision
rlm@216 11
rlm@212 12 Vision is one of the most important senses for humans, so I need to
rlm@212 13 build a simulated sense of vision for my AI. I will do this with
rlm@212 14 simulated eyes. Each eye can be independely moved and should see its
rlm@212 15 own version of the world depending on where it is.
rlm@212 16
rlm@218 17 Making these simulated eyes a reality is simple bacause jMonkeyEngine
rlm@218 18 already conatains extensive support for multiple views of the same 3D
rlm@218 19 simulated world. The reason jMonkeyEngine has this support is because
rlm@218 20 the support is necessary to create games with split-screen
rlm@218 21 views. Multiple views are also used to create efficient
rlm@212 22 pseudo-reflections by rendering the scene from a certain perspective
rlm@212 23 and then projecting it back onto a surface in the 3D world.
rlm@212 24
rlm@218 25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
rlm@212 26 [[../images/goldeneye-4-player.png]]
rlm@212 27
rlm@213 28 * Brief Description of jMonkeyEngine's Rendering Pipeline
rlm@212 29
rlm@213 30 jMonkeyEngine allows you to create a =ViewPort=, which represents a
rlm@213 31 view of the simulated world. You can create as many of these as you
rlm@213 32 want. Every frame, the =RenderManager= iterates through each
rlm@213 33 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
rlm@213 34 is a =FrameBuffer= which represents the rendered image in the GPU.
rlm@151 35
rlm@213 36 Each =ViewPort= can have any number of attached =SceneProcessor=
rlm@213 37 objects, which are called every time a new frame is rendered. A
rlm@219 38 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do
rlm@219 39 whatever it wants to the data. Often this consists of invoking GPU
rlm@219 40 specific operations on the rendered image. The =SceneProcessor= can
rlm@219 41 also copy the GPU image data to RAM and process it with the CPU.
rlm@151 42
rlm@213 43 * The Vision Pipeline
rlm@151 44
rlm@213 45 Each eye in the simulated creature needs it's own =ViewPort= so that
rlm@213 46 it can see the world from its own perspective. To this =ViewPort=, I
rlm@214 47 add a =SceneProcessor= that feeds the visual data to any arbitray
rlm@213 48 continuation function for further processing. That continuation
rlm@213 49 function may perform both CPU and GPU operations on the data. To make
rlm@213 50 this easy for the continuation function, the =SceneProcessor=
rlm@213 51 maintains appropriatly sized buffers in RAM to hold the data. It does
rlm@218 52 not do any copying from the GPU to the CPU itself because it is a slow
rlm@218 53 operation.
rlm@214 54
rlm@213 55 #+name: pipeline-1
rlm@213 56 #+begin_src clojure
rlm@113 57 (defn vision-pipeline
rlm@34 58 "Create a SceneProcessor object which wraps a vision processing
rlm@113 59 continuation function. The continuation is a function that takes
rlm@113 60 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@113 61 each of which has already been appropiately sized."
rlm@23 62 [continuation]
rlm@23 63 (let [byte-buffer (atom nil)
rlm@113 64 renderer (atom nil)
rlm@113 65 image (atom nil)]
rlm@23 66 (proxy [SceneProcessor] []
rlm@23 67 (initialize
rlm@23 68 [renderManager viewPort]
rlm@23 69 (let [cam (.getCamera viewPort)
rlm@23 70 width (.getWidth cam)
rlm@23 71 height (.getHeight cam)]
rlm@23 72 (reset! renderer (.getRenderer renderManager))
rlm@23 73 (reset! byte-buffer
rlm@23 74 (BufferUtils/createByteBuffer
rlm@113 75 (* width height 4)))
rlm@113 76 (reset! image (BufferedImage.
rlm@113 77 width height
rlm@113 78 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@23 79 (isInitialized [] (not (nil? @byte-buffer)))
rlm@23 80 (reshape [_ _ _])
rlm@23 81 (preFrame [_])
rlm@23 82 (postQueue [_])
rlm@23 83 (postFrame
rlm@23 84 [#^FrameBuffer fb]
rlm@23 85 (.clear @byte-buffer)
rlm@113 86 (continuation @renderer fb @byte-buffer @image))
rlm@23 87 (cleanup []))))
rlm@213 88 #+end_src
rlm@213 89
rlm@213 90 The continuation function given to =(vision-pipeline)= above will be
rlm@213 91 given a =Renderer= and three containers for image data. The
rlm@218 92 =FrameBuffer= references the GPU image data, but the pixel data can
rlm@218 93 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
rlm@219 94 are initially "empty" but are sized to hold the data in the
rlm@213 95 =FrameBuffer=. I call transfering the GPU image data to the CPU
rlm@213 96 structures "mixing" the image data. I have provided three functions to
rlm@213 97 do this mixing.
rlm@213 98
rlm@213 99 #+name: pipeline-2
rlm@213 100 #+begin_src clojure
rlm@113 101 (defn frameBuffer->byteBuffer!
rlm@113 102 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
rlm@113 103 the CPU (ByteBuffer)."
rlm@113 104 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
rlm@113 105 (.readFrameBuffer r fb bb) bb)
rlm@113 106
rlm@113 107 (defn byteBuffer->bufferedImage!
rlm@113 108 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
rlm@113 109 style ABGR image data and place it in BufferedImage bi."
rlm@113 110 [#^ByteBuffer bb #^BufferedImage bi]
rlm@113 111 (Screenshots/convertScreenShot bb bi) bi)
rlm@113 112
rlm@113 113 (defn BufferedImage!
rlm@113 114 "Continuation which will grab the buffered image from the materials
rlm@113 115 provided by (vision-pipeline)."
rlm@113 116 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
rlm@113 117 (byteBuffer->bufferedImage!
rlm@113 118 (frameBuffer->byteBuffer! r fb bb) bi))
rlm@213 119 #+end_src
rlm@112 120
rlm@213 121 Note that it is possible to write vision processing algorithms
rlm@213 122 entirely in terms of =BufferedImage= inputs. Just compose that
rlm@213 123 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
rlm@213 124 processing algorithm that is entirely hosted on the GPU does not have
rlm@213 125 to pay for this convienence.
rlm@213 126
rlm@214 127 * COMMENT asdasd
rlm@213 128
rlm@213 129 (vision creature) will take an optional :skip argument which will
rlm@213 130 inform the continuations in scene processor to skip the given
rlm@213 131 number of cycles 0 means that no cycles will be skipped.
rlm@213 132
rlm@213 133 (vision creature) will return [init-functions sensor-functions].
rlm@213 134 The init-functions are each single-arg functions that take the
rlm@213 135 world and register the cameras and must each be called before the
rlm@213 136 corresponding sensor-functions. Each init-function returns the
rlm@213 137 viewport for that eye which can be manipulated, saved, etc. Each
rlm@213 138 sensor-function is a thunk and will return data in the same
rlm@213 139 format as the tactile-sensor functions the structure is
rlm@213 140 [topology, sensor-data]. Internally, these sensor-functions
rlm@213 141 maintain a reference to sensor-data which is periodically updated
rlm@213 142 by the continuation function established by its init-function.
rlm@213 143 They can be queried every cycle, but their information may not
rlm@213 144 necessairly be different every cycle.
rlm@213 145
rlm@214 146 * Physical Eyes
rlm@214 147
rlm@214 148 The vision pipeline described above handles the flow of rendered
rlm@214 149 images. Now, we need simulated eyes to serve as the source of these
rlm@214 150 images.
rlm@214 151
rlm@214 152 An eye is described in blender in the same way as a joint. They are
rlm@214 153 zero dimensional empty objects with no geometry whose local coordinate
rlm@214 154 system determines the orientation of the resulting eye. All eyes are
rlm@214 155 childern of a parent node named "eyes" just as all joints have a
rlm@214 156 parent named "joints". An eye binds to the nearest physical object
rlm@214 157 with =(bind-sense=).
rlm@214 158
rlm@214 159 #+name: add-eye
rlm@214 160 #+begin_src clojure
rlm@215 161 (in-ns 'cortex.vision)
rlm@215 162
rlm@214 163 (defn add-eye!
rlm@214 164 "Create a Camera centered on the current position of 'eye which
rlm@214 165 follows the closest physical node in 'creature and sends visual
rlm@215 166 data to 'continuation. The camera will point in the X direction and
rlm@215 167 use the Z vector as up as determined by the rotation of these
rlm@215 168 vectors in blender coordinate space. Use XZY rotation for the node
rlm@215 169 in blender."
rlm@214 170 [#^Node creature #^Spatial eye]
rlm@214 171 (let [target (closest-node creature eye)
rlm@214 172 [cam-width cam-height] (eye-dimensions eye)
rlm@215 173 cam (Camera. cam-width cam-height)
rlm@215 174 rot (.getWorldRotation eye)]
rlm@214 175 (.setLocation cam (.getWorldTranslation eye))
rlm@218 176 (.lookAtDirection
rlm@218 177 cam ; this part is not a mistake and
rlm@218 178 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
rlm@218 179 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
rlm@214 180 (.setFrustumPerspective
rlm@215 181 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
rlm@215 182 (bind-sense target cam) cam))
rlm@214 183 #+end_src
rlm@214 184
rlm@214 185 Here, the camera is created based on metadata on the eye-node and
rlm@214 186 attached to the nearest physical object with =(bind-sense)=
rlm@214 187
rlm@214 188
rlm@214 189 ** The Retina
rlm@214 190
rlm@214 191 An eye is a surface (the retina) which contains many discrete sensors
rlm@218 192 to detect light. These sensors have can have different light-sensing
rlm@214 193 properties. In humans, each discrete sensor is sensitive to red,
rlm@214 194 blue, green, or gray. These different types of sensors can have
rlm@214 195 different spatial distributions along the retina. In humans, there is
rlm@214 196 a fovea in the center of the retina which has a very high density of
rlm@214 197 color sensors, and a blind spot which has no sensors at all. Sensor
rlm@219 198 density decreases in proportion to distance from the fovea.
rlm@214 199
rlm@214 200 I want to be able to model any retinal configuration, so my eye-nodes
rlm@214 201 in blender contain metadata pointing to images that describe the
rlm@214 202 percise position of the individual sensors using white pixels. The
rlm@214 203 meta-data also describes the percise sensitivity to light that the
rlm@214 204 sensors described in the image have. An eye can contain any number of
rlm@214 205 these images. For example, the metadata for an eye might look like
rlm@214 206 this:
rlm@214 207
rlm@214 208 #+begin_src clojure
rlm@214 209 {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@214 210 #+end_src
rlm@214 211
rlm@214 212 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
rlm@214 213 [[../assets/Models/test-creature/retina-small.png]]
rlm@214 214
rlm@214 215 Together, the number 0xFF0000 and the image image above describe the
rlm@214 216 placement of red-sensitive sensory elements.
rlm@214 217
rlm@214 218 Meta-data to very crudely approximate a human eye might be something
rlm@214 219 like this:
rlm@214 220
rlm@214 221 #+begin_src clojure
rlm@214 222 (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@214 223 {0xFF0000 retinal-profile
rlm@214 224 0x00FF00 retinal-profile
rlm@214 225 0x0000FF retinal-profile
rlm@214 226 0xFFFFFF retinal-profile})
rlm@214 227 #+end_src
rlm@214 228
rlm@214 229 The numbers that serve as keys in the map determine a sensor's
rlm@214 230 relative sensitivity to the channels red, green, and blue. These
rlm@218 231 sensitivity values are packed into an integer in the order =|_|R|G|B|=
rlm@218 232 in 8-bit fields. The RGB values of a pixel in the image are added
rlm@214 233 together with these sensitivities as linear weights. Therfore,
rlm@214 234 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
rlm@214 235 all colors equally (gray).
rlm@214 236
rlm@214 237 For convienence I've defined a few symbols for the more common
rlm@214 238 sensitivity values.
rlm@214 239
rlm@214 240 #+name: sensitivity
rlm@214 241 #+begin_src clojure
rlm@214 242 (defvar sensitivity-presets
rlm@214 243 {:all 0xFFFFFF
rlm@214 244 :red 0xFF0000
rlm@214 245 :blue 0x0000FF
rlm@214 246 :green 0x00FF00}
rlm@214 247 "Retinal sensitivity presets for sensors that extract one channel
rlm@219 248 (:red :blue :green) or average all channels (:all)")
rlm@214 249 #+end_src
rlm@214 250
rlm@214 251 ** Metadata Processing
rlm@214 252
rlm@214 253 =(retina-sensor-profile)= extracts a map from the eye-node in the same
rlm@214 254 format as the example maps above. =(eye-dimensions)= finds the
rlm@219 255 dimensions of the smallest image required to contain all the retinal
rlm@214 256 sensor maps.
rlm@214 257
rlm@216 258 #+name: retina
rlm@214 259 #+begin_src clojure
rlm@214 260 (defn retina-sensor-profile
rlm@214 261 "Return a map of pixel sensitivity numbers to BufferedImages
rlm@214 262 describing the distribution of light-sensitive components of this
rlm@214 263 eye. :red, :green, :blue, :gray are already defined as extracting
rlm@214 264 the red, green, blue, and average components respectively."
rlm@214 265 [#^Spatial eye]
rlm@214 266 (if-let [eye-map (meta-data eye "eye")]
rlm@214 267 (map-vals
rlm@214 268 load-image
rlm@214 269 (eval (read-string eye-map)))))
rlm@214 270
rlm@218 271 (defn eye-dimensions
rlm@218 272 "Returns [width, height] determined by the metadata of the eye."
rlm@214 273 [#^Spatial eye]
rlm@214 274 (let [dimensions
rlm@214 275 (map #(vector (.getWidth %) (.getHeight %))
rlm@214 276 (vals (retina-sensor-profile eye)))]
rlm@214 277 [(apply max (map first dimensions))
rlm@214 278 (apply max (map second dimensions))]))
rlm@214 279 #+end_src
rlm@214 280
rlm@214 281 * Eye Creation
rlm@214 282 First off, get the children of the "eyes" empty node to find all the
rlm@214 283 eyes the creature has.
rlm@216 284 #+name: eye-node
rlm@214 285 #+begin_src clojure
rlm@214 286 (defvar
rlm@214 287 ^{:arglists '([creature])}
rlm@214 288 eyes
rlm@214 289 (sense-nodes "eyes")
rlm@214 290 "Return the children of the creature's \"eyes\" node.")
rlm@214 291 #+end_src
rlm@214 292
rlm@215 293 Then, add the camera created by =(add-eye!)= to the simulation by
rlm@215 294 creating a new viewport.
rlm@214 295
rlm@216 296 #+name: add-camera
rlm@213 297 #+begin_src clojure
rlm@169 298 (defn add-camera!
rlm@169 299 "Add a camera to the world, calling continuation on every frame
rlm@34 300 produced."
rlm@167 301 [#^Application world camera continuation]
rlm@23 302 (let [width (.getWidth camera)
rlm@23 303 height (.getHeight camera)
rlm@23 304 render-manager (.getRenderManager world)
rlm@23 305 viewport (.createMainView render-manager "eye-view" camera)]
rlm@23 306 (doto viewport
rlm@23 307 (.setClearFlags true true true)
rlm@112 308 (.setBackgroundColor ColorRGBA/Black)
rlm@113 309 (.addProcessor (vision-pipeline continuation))
rlm@23 310 (.attachScene (.getRootNode world)))))
rlm@215 311 #+end_src
rlm@151 312
rlm@151 313
rlm@218 314 The eye's continuation function should register the viewport with the
rlm@218 315 simulation the first time it is called, use the CPU to extract the
rlm@215 316 appropriate pixels from the rendered image and weight them by each
rlm@218 317 sensor's sensitivity. I have the option to do this processing in
rlm@218 318 native code for a slight gain in speed. I could also do it in the GPU
rlm@218 319 for a massive gain in speed. =(vision-kernel)= generates a list of
rlm@218 320 such continuation functions, one for each channel of the eye.
rlm@151 321
rlm@216 322 #+name: kernel
rlm@215 323 #+begin_src clojure
rlm@215 324 (in-ns 'cortex.vision)
rlm@151 325
rlm@215 326 (defrecord attached-viewport [vision-fn viewport-fn]
rlm@215 327 clojure.lang.IFn
rlm@215 328 (invoke [this world] (vision-fn world))
rlm@215 329 (applyTo [this args] (apply vision-fn args)))
rlm@151 330
rlm@216 331 (defn pixel-sense [sensitivity pixel]
rlm@216 332 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
rlm@216 333 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
rlm@216 334 s-b (bit-and 0x0000FF sensitivity)
rlm@216 335
rlm@216 336 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
rlm@216 337 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
rlm@216 338 p-b (bit-and 0x0000FF pixel)
rlm@216 339
rlm@216 340 total-sensitivity (* 255 (+ s-r s-g s-b))]
rlm@216 341 (float (/ (+ (* s-r p-r)
rlm@216 342 (* s-g p-g)
rlm@216 343 (* s-b p-b))
rlm@216 344 total-sensitivity))))
rlm@216 345
rlm@215 346 (defn vision-kernel
rlm@171 347 "Returns a list of functions, each of which will return a color
rlm@171 348 channel's worth of visual information when called inside a running
rlm@171 349 simulation."
rlm@151 350 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@169 351 (let [retinal-map (retina-sensor-profile eye)
rlm@169 352 camera (add-eye! creature eye)
rlm@151 353 vision-image
rlm@151 354 (atom
rlm@151 355 (BufferedImage. (.getWidth camera)
rlm@151 356 (.getHeight camera)
rlm@170 357 BufferedImage/TYPE_BYTE_BINARY))
rlm@170 358 register-eye!
rlm@170 359 (runonce
rlm@170 360 (fn [world]
rlm@170 361 (add-camera!
rlm@170 362 world camera
rlm@170 363 (let [counter (atom 0)]
rlm@170 364 (fn [r fb bb bi]
rlm@170 365 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@170 366 (reset! vision-image
rlm@170 367 (BufferedImage! r fb bb bi))))))))]
rlm@151 368 (vec
rlm@151 369 (map
rlm@151 370 (fn [[key image]]
rlm@151 371 (let [whites (white-coordinates image)
rlm@151 372 topology (vec (collapse whites))
rlm@216 373 sensitivity (sensitivity-presets key key)]
rlm@215 374 (attached-viewport.
rlm@215 375 (fn [world]
rlm@215 376 (register-eye! world)
rlm@215 377 (vector
rlm@215 378 topology
rlm@215 379 (vec
rlm@215 380 (for [[x y] whites]
rlm@216 381 (pixel-sense
rlm@216 382 sensitivity
rlm@216 383 (.getRGB @vision-image x y))))))
rlm@215 384 register-eye!)))
rlm@215 385 retinal-map))))
rlm@151 386
rlm@215 387 (defn gen-fix-display
rlm@215 388 "Create a function to call to restore a simulation's display when it
rlm@215 389 is disrupted by a Viewport."
rlm@215 390 []
rlm@215 391 (runonce
rlm@215 392 (fn [world]
rlm@215 393 (add-camera! world (.getCamera world) no-op))))
rlm@215 394 #+end_src
rlm@170 395
rlm@215 396 Note that since each of the functions generated by =(vision-kernel)=
rlm@215 397 shares the same =(register-eye!)= function, the eye will be registered
rlm@215 398 only once the first time any of the functions from the list returned
rlm@215 399 by =(vision-kernel)= is called. Each of the functions returned by
rlm@215 400 =(vision-kernel)= also allows access to the =Viewport= through which
rlm@215 401 it recieves images.
rlm@215 402
rlm@215 403 The in-game display can be disrupted by all the viewports that the
rlm@215 404 functions greated by =(vision-kernel)= add. This doesn't affect the
rlm@215 405 simulation or the simulated senses, but can be annoying.
rlm@215 406 =(gen-fix-display)= restores the in-simulation display.
rlm@215 407
rlm@215 408 ** Vision!
rlm@215 409
rlm@218 410 All the hard work has been done; all that remains is to apply
rlm@215 411 =(vision-kernel)= to each eye in the creature and gather the results
rlm@215 412 into one list of functions.
rlm@215 413
rlm@216 414 #+name: main
rlm@215 415 #+begin_src clojure
rlm@170 416 (defn vision!
rlm@170 417 "Returns a function which returns visual sensory data when called
rlm@218 418 inside a running simulation."
rlm@151 419 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@151 420 (reduce
rlm@170 421 concat
rlm@167 422 (for [eye (eyes creature)]
rlm@215 423 (vision-kernel creature eye))))
rlm@215 424 #+end_src
rlm@151 425
rlm@215 426 ** Visualization of Vision
rlm@215 427
rlm@215 428 It's vital to have a visual representation for each sense. Here I use
rlm@215 429 =(view-sense)= to construct a function that will create a display for
rlm@215 430 visual data.
rlm@215 431
rlm@216 432 #+name: display
rlm@215 433 #+begin_src clojure
rlm@216 434 (in-ns 'cortex.vision)
rlm@216 435
rlm@189 436 (defn view-vision
rlm@189 437 "Creates a function which accepts a list of visual sensor-data and
rlm@189 438 displays each element of the list to the screen."
rlm@189 439 []
rlm@188 440 (view-sense
rlm@188 441 (fn
rlm@188 442 [[coords sensor-data]]
rlm@188 443 (let [image (points->image coords)]
rlm@188 444 (dorun
rlm@188 445 (for [i (range (count coords))]
rlm@188 446 (.setRGB image ((coords i) 0) ((coords i) 1)
rlm@216 447 (gray (int (* 255 (sensor-data i)))))))
rlm@189 448 image))))
rlm@34 449 #+end_src
rlm@23 450
rlm@215 451 * Tests
rlm@215 452 ** Basic Test
rlm@23 453
rlm@215 454 This is a basic test for the vision system. It only tests the
rlm@215 455 vision-pipeline and does not deal with loadig eyes from a blender
rlm@215 456 file. The code creates two videos of the same rotating cube from
rlm@215 457 different angles.
rlm@23 458
rlm@215 459 #+name: test-1
rlm@23 460 #+begin_src clojure
rlm@215 461 (in-ns 'cortex.test.vision)
rlm@23 462
rlm@219 463 (defn test-pipeline
rlm@69 464 "Testing vision:
rlm@69 465 Tests the vision system by creating two views of the same rotating
rlm@69 466 object from different angles and displaying both of those views in
rlm@69 467 JFrames.
rlm@69 468
rlm@69 469 You should see a rotating cube, and two windows,
rlm@69 470 each displaying a different view of the cube."
rlm@36 471 []
rlm@58 472 (let [candy
rlm@58 473 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@112 474 (world
rlm@112 475 (doto (Node.)
rlm@112 476 (.attachChild candy))
rlm@112 477 {}
rlm@112 478 (fn [world]
rlm@112 479 (let [cam (.clone (.getCamera world))
rlm@112 480 width (.getWidth cam)
rlm@112 481 height (.getHeight cam)]
rlm@169 482 (add-camera! world cam
rlm@215 483 (comp
rlm@215 484 (view-image
rlm@215 485 (File. "/home/r/proj/cortex/render/vision/1"))
rlm@215 486 BufferedImage!))
rlm@169 487 (add-camera! world
rlm@112 488 (doto (.clone cam)
rlm@112 489 (.setLocation (Vector3f. -10 0 0))
rlm@112 490 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@215 491 (comp
rlm@215 492 (view-image
rlm@215 493 (File. "/home/r/proj/cortex/render/vision/2"))
rlm@215 494 BufferedImage!))
rlm@112 495 ;; This is here to restore the main view
rlm@112 496 ;; after the other views have completed processing
rlm@169 497 (add-camera! world (.getCamera world) no-op)))
rlm@112 498 (fn [world tpf]
rlm@112 499 (.rotate candy (* tpf 0.2) 0 0)))))
rlm@23 500 #+end_src
rlm@23 501
rlm@215 502 #+begin_html
rlm@215 503 <div class="figure">
rlm@215 504 <video controls="controls" width="755">
rlm@215 505 <source src="../video/spinning-cube.ogg" type="video/ogg"
rlm@215 506 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@215 507 </video>
rlm@215 508 <p>A rotating cube viewed from two different perspectives.</p>
rlm@215 509 </div>
rlm@215 510 #+end_html
rlm@215 511
rlm@215 512 Creating multiple eyes like this can be used for stereoscopic vision
rlm@215 513 simulation in a single creature or for simulating multiple creatures,
rlm@215 514 each with their own sense of vision.
rlm@215 515
rlm@215 516 ** Adding Vision to the Worm
rlm@215 517
rlm@218 518 To the worm from the last post, I add a new node that describes its
rlm@215 519 eyes.
rlm@215 520
rlm@215 521 #+attr_html: width=755
rlm@215 522 #+caption: The worm with newly added empty nodes describing a single eye.
rlm@215 523 [[../images/worm-with-eye.png]]
rlm@215 524
rlm@215 525 The node highlighted in yellow is the root level "eyes" node. It has
rlm@218 526 a single child, highlighted in orange, which describes a single
rlm@218 527 eye. This is the "eye" node. It is placed so that the worm will have
rlm@218 528 an eye located in the center of the flat portion of its lower
rlm@218 529 hemispherical section.
rlm@218 530
rlm@218 531 The two nodes which are not highlighted describe the single joint of
rlm@218 532 the worm.
rlm@215 533
rlm@215 534 The metadata of the eye-node is:
rlm@215 535
rlm@215 536 #+begin_src clojure :results verbatim :exports both
rlm@215 537 (cortex.sense/meta-data
rlm@218 538 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
rlm@215 539 #+end_src
rlm@215 540
rlm@215 541 #+results:
rlm@215 542 : "(let [retina \"Models/test-creature/retina-small.png\"]
rlm@215 543 : {:all retina :red retina :green retina :blue retina})"
rlm@215 544
rlm@215 545 This is the approximation to the human eye described earlier.
rlm@215 546
rlm@216 547 #+name: test-2
rlm@215 548 #+begin_src clojure
rlm@215 549 (in-ns 'cortex.test.vision)
rlm@215 550
rlm@216 551 (defn change-color [obj color]
rlm@216 552 (println-repl obj)
rlm@216 553 (if obj
rlm@216 554 (.setColor (.getMaterial obj) "Color" color)))
rlm@216 555
rlm@216 556 (defn colored-cannon-ball [color]
rlm@216 557 (comp #(change-color % color)
rlm@216 558 (fire-cannon-ball)))
rlm@215 559
rlm@236 560 (defn test-worm-vision [record]
rlm@215 561 (let [the-worm (doto (worm)(body!))
rlm@215 562 vision (vision! the-worm)
rlm@215 563 vision-display (view-vision)
rlm@215 564 fix-display (gen-fix-display)
rlm@215 565 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
rlm@215 566 x-axis
rlm@215 567 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
rlm@215 568 :position (Vector3f. 0 -5 0))
rlm@215 569 y-axis
rlm@215 570 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
rlm@215 571 :position (Vector3f. 0 -5 0))
rlm@215 572 z-axis
rlm@215 573 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
rlm@216 574 :position (Vector3f. 0 -5 0))
rlm@216 575 timer (RatchetTimer. 60)]
rlm@215 576
rlm@215 577 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])
rlm@216 578 (assoc standard-debug-controls
rlm@216 579 "key-r" (colored-cannon-ball ColorRGBA/Red)
rlm@216 580 "key-b" (colored-cannon-ball ColorRGBA/Blue)
rlm@216 581 "key-g" (colored-cannon-ball ColorRGBA/Green))
rlm@215 582 (fn [world]
rlm@215 583 (light-up-everything world)
rlm@216 584 (speed-up world)
rlm@216 585 (.setTimer world timer)
rlm@216 586 (display-dialated-time world timer)
rlm@215 587 ;; add a view from the worm's perspective
rlm@236 588 (if record
rlm@236 589 (Capture/captureVideo
rlm@236 590 world
rlm@236 591 (File.
rlm@236 592 "/home/r/proj/cortex/render/worm-vision/main-view")))
rlm@236 593
rlm@215 594 (add-camera!
rlm@215 595 world
rlm@215 596 (add-eye! the-worm
rlm@215 597 (.getChild
rlm@215 598 (.getChild the-worm "eyes") "eye"))
rlm@215 599 (comp
rlm@215 600 (view-image
rlm@236 601 (if record
rlm@236 602 (File.
rlm@236 603 "/home/r/proj/cortex/render/worm-vision/worm-view")))
rlm@215 604 BufferedImage!))
rlm@236 605
rlm@236 606 (set-gravity world Vector3f/ZERO))
rlm@216 607
rlm@215 608 (fn [world _ ]
rlm@215 609 (.setLocalTranslation me (.getLocation (.getCamera world)))
rlm@215 610 (vision-display
rlm@215 611 (map #(% world) vision)
rlm@236 612 (if record (File. "/home/r/proj/cortex/render/worm-vision")))
rlm@215 613 (fix-display world)))))
rlm@215 614 #+end_src
rlm@215 615
rlm@218 616 The world consists of the worm and a flat gray floor. I can shoot red,
rlm@218 617 green, blue and white cannonballs at the worm. The worm is initially
rlm@218 618 looking down at the floor, and there is no gravity. My perspective
rlm@218 619 (the Main View), the worm's perspective (Worm View) and the 4 sensor
rlm@218 620 channels that comprise the worm's eye are all saved frame-by-frame to
rlm@218 621 disk.
rlm@218 622
rlm@218 623 * Demonstration of Vision
rlm@218 624 #+begin_html
rlm@218 625 <div class="figure">
rlm@218 626 <video controls="controls" width="755">
rlm@218 627 <source src="../video/worm-vision.ogg" type="video/ogg"
rlm@218 628 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@218 629 </video>
rlm@218 630 <p>Simulated Vision in a Virtual Environment</p>
rlm@218 631 </div>
rlm@218 632 #+end_html
rlm@218 633
rlm@218 634 ** Generate the Worm Video from Frames
rlm@216 635 #+name: magick2
rlm@216 636 #+begin_src clojure
rlm@216 637 (ns cortex.video.magick2
rlm@216 638 (:import java.io.File)
rlm@216 639 (:use clojure.contrib.shell-out))
rlm@216 640
rlm@216 641 (defn images [path]
rlm@216 642 (sort (rest (file-seq (File. path)))))
rlm@216 643
rlm@216 644 (def base "/home/r/proj/cortex/render/worm-vision/")
rlm@216 645
rlm@216 646 (defn pics [file]
rlm@216 647 (images (str base file)))
rlm@216 648
rlm@216 649 (defn combine-images []
rlm@216 650 (let [main-view (pics "main-view")
rlm@216 651 worm-view (pics "worm-view")
rlm@216 652 blue (pics "0")
rlm@216 653 green (pics "1")
rlm@216 654 red (pics "2")
rlm@216 655 gray (pics "3")
rlm@216 656 blender (let [b-pics (pics "blender")]
rlm@216 657 (concat b-pics (repeat 9001 (last b-pics))))
rlm@216 658 background (repeat 9001 (File. (str base "background.png")))
rlm@216 659 targets (map
rlm@216 660 #(File. (str base "out/" (format "%07d.png" %)))
rlm@216 661 (range 0 (count main-view)))]
rlm@216 662 (dorun
rlm@216 663 (pmap
rlm@216 664 (comp
rlm@216 665 (fn [[background main-view worm-view red green blue gray blender target]]
rlm@216 666 (println target)
rlm@216 667 (sh "convert"
rlm@216 668 background
rlm@216 669 main-view "-geometry" "+18+17" "-composite"
rlm@216 670 worm-view "-geometry" "+677+17" "-composite"
rlm@216 671 green "-geometry" "+685+430" "-composite"
rlm@216 672 red "-geometry" "+788+430" "-composite"
rlm@216 673 blue "-geometry" "+894+430" "-composite"
rlm@216 674 gray "-geometry" "+1000+430" "-composite"
rlm@216 675 blender "-geometry" "+0+0" "-composite"
rlm@216 676 target))
rlm@216 677 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@216 678 background main-view worm-view red green blue gray blender targets))))
rlm@216 679 #+end_src
rlm@216 680
rlm@216 681 #+begin_src sh :results silent
rlm@216 682 cd /home/r/proj/cortex/render/worm-vision
rlm@216 683 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
rlm@216 684 #+end_src
rlm@236 685
rlm@215 686 * Headers
rlm@215 687
rlm@213 688 #+name: vision-header
rlm@213 689 #+begin_src clojure
rlm@213 690 (ns cortex.vision
rlm@213 691 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
rlm@213 692 eyes from different positions to observe the same world, and pass
rlm@213 693 the observed data to any arbitray function. Automatically reads
rlm@216 694 eye-nodes from specially prepared blender files and instantiates
rlm@213 695 them in the world as actual eyes."
rlm@213 696 {:author "Robert McIntyre"}
rlm@213 697 (:use (cortex world sense util))
rlm@213 698 (:use clojure.contrib.def)
rlm@213 699 (:import com.jme3.post.SceneProcessor)
rlm@237 700 (:import (com.jme3.util BufferUtils Screenshots))
rlm@213 701 (:import java.nio.ByteBuffer)
rlm@213 702 (:import java.awt.image.BufferedImage)
rlm@213 703 (:import (com.jme3.renderer ViewPort Camera))
rlm@216 704 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
rlm@213 705 (:import com.jme3.renderer.Renderer)
rlm@213 706 (:import com.jme3.app.Application)
rlm@213 707 (:import com.jme3.texture.FrameBuffer)
rlm@213 708 (:import (com.jme3.scene Node Spatial)))
rlm@213 709 #+end_src
rlm@112 710
rlm@215 711 #+name: test-header
rlm@215 712 #+begin_src clojure
rlm@215 713 (ns cortex.test.vision
rlm@215 714 (:use (cortex world sense util body vision))
rlm@215 715 (:use cortex.test.body)
rlm@215 716 (:import java.awt.image.BufferedImage)
rlm@215 717 (:import javax.swing.JPanel)
rlm@215 718 (:import javax.swing.SwingUtilities)
rlm@215 719 (:import java.awt.Dimension)
rlm@215 720 (:import javax.swing.JFrame)
rlm@215 721 (:import com.jme3.math.ColorRGBA)
rlm@215 722 (:import com.jme3.scene.Node)
rlm@215 723 (:import com.jme3.math.Vector3f)
rlm@216 724 (:import java.io.File)
rlm@216 725 (:import (com.aurellem.capture Capture RatchetTimer)))
rlm@215 726 #+end_src
rlm@215 727
rlm@216 728 * Onward!
rlm@216 729 - As a neat bonus, this idea behind simulated vision also enables one
rlm@216 730 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
rlm@216 731 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
rlm@215 732
rlm@216 733 * Source Listing
rlm@216 734 - [[../src/cortex/vision.clj][cortex.vision]]
rlm@216 735 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
rlm@216 736 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
rlm@216 737 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
rlm@216 738 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
rlm@216 739 - [[http://hg.bortreb.com ][source-repository]]
rlm@216 740
rlm@35 741
rlm@24 742
rlm@212 743 * COMMENT Generate Source
rlm@34 744 #+begin_src clojure :tangle ../src/cortex/vision.clj
rlm@216 745 <<vision-header>>
rlm@216 746 <<pipeline-1>>
rlm@216 747 <<pipeline-2>>
rlm@216 748 <<retina>>
rlm@216 749 <<add-eye>>
rlm@216 750 <<sensitivity>>
rlm@216 751 <<eye-node>>
rlm@216 752 <<add-camera>>
rlm@216 753 <<kernel>>
rlm@216 754 <<main>>
rlm@216 755 <<display>>
rlm@24 756 #+end_src
rlm@24 757
rlm@68 758 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
rlm@215 759 <<test-header>>
rlm@215 760 <<test-1>>
rlm@216 761 <<test-2>>
rlm@24 762 #+end_src
rlm@216 763
rlm@216 764 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
rlm@216 765 <<magick2>>
rlm@216 766 #+end_src