Mercurial > cortex

annotate org/vision.org @ 338:d37ccb6c888f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
determined that the laptop cannot support arbitray dimensions when creating cameras. will need to use desktop from here on out when doing actual simulations.
author Robert McIntyre <rlm@mit.edu>
date Fri, 20 Jul 2012 16:40:25 -0500
parents fdc98824d69b
children 4f5a5d5f1613
rev   line source
rlm@34 1 #+title: Simulated Sense of Sight
rlm@23 2 #+author: Robert McIntyre
rlm@23 3 #+email: rlm@mit.edu
rlm@38 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
rlm@34 5 #+keywords: computer vision, jMonkeyEngine3, clojure
rlm@23 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@23 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@23 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@23 9
ocsenave@264 10 * JMonkeyEngine natively supports multiple views of the same world.
ocsenave@264 11
rlm@212 12 Vision is one of the most important senses for humans, so I need to
rlm@212 13 build a simulated sense of vision for my AI. I will do this with
rlm@306 14 simulated eyes. Each eye can be independently moved and should see its
rlm@212 15 own version of the world depending on where it is.
rlm@212 16
rlm@306 17 Making these simulated eyes a reality is simple because jMonkeyEngine
rlm@306 18 already contains extensive support for multiple views of the same 3D
rlm@218 19 simulated world. The reason jMonkeyEngine has this support is because
rlm@218 20 the support is necessary to create games with split-screen
rlm@218 21 views. Multiple views are also used to create efficient
rlm@212 22 pseudo-reflections by rendering the scene from a certain perspective
rlm@212 23 and then projecting it back onto a surface in the 3D world.
rlm@212 24
rlm@218 25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
rlm@212 26 [[../images/goldeneye-4-player.png]]
rlm@212 27
ocsenave@264 28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.
rlm@306 29 # =ViewPorts= are cameras; =RenderManger= takes snapshots each frame.
ocsenave@264 30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline
rlm@212 31
rlm@213 32 jMonkeyEngine allows you to create a =ViewPort=, which represents a
rlm@213 33 view of the simulated world. You can create as many of these as you
rlm@213 34 want. Every frame, the =RenderManager= iterates through each
rlm@213 35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
rlm@213 36 is a =FrameBuffer= which represents the rendered image in the GPU.
rlm@151 37
rlm@306 38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =RenderManager= records a snapshot of what each view is currently seeing; these snapshots are =FrameBuffer= objects.
ocsenave@265 39 #+ATTR_HTML: width="400"
ocsenave@272 40 [[../images/diagram_rendermanager2.png]]
ocsenave@262 41
rlm@213 42 Each =ViewPort= can have any number of attached =SceneProcessor=
rlm@213 43 objects, which are called every time a new frame is rendered. A
rlm@306 44 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
rlm@219 45 whatever it wants to the data. Often this consists of invoking GPU
rlm@219 46 specific operations on the rendered image. The =SceneProcessor= can
rlm@219 47 also copy the GPU image data to RAM and process it with the CPU.
rlm@151 48
ocsenave@264 49 ** From Views to Vision
ocsenave@264 50 # Appropriating Views for Vision.
rlm@151 51
ocsenave@264 52 Each eye in the simulated creature needs its own =ViewPort= so that
rlm@213 53 it can see the world from its own perspective. To this =ViewPort=, I
rlm@306 54 add a =SceneProcessor= that feeds the visual data to any arbitrary
rlm@213 55 continuation function for further processing. That continuation
rlm@213 56 function may perform both CPU and GPU operations on the data. To make
rlm@213 57 this easy for the continuation function, the =SceneProcessor=
rlm@306 58 maintains appropriately sized buffers in RAM to hold the data. It does
rlm@218 59 not do any copying from the GPU to the CPU itself because it is a slow
rlm@218 60 operation.
rlm@214 61
rlm@213 62 #+name: pipeline-1
rlm@213 63 #+begin_src clojure
rlm@113 64 (defn vision-pipeline
rlm@34 65 "Create a SceneProcessor object which wraps a vision processing
rlm@113 66 continuation function. The continuation is a function that takes
rlm@113 67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@306 68 each of which has already been appropriately sized."
rlm@23 69 [continuation]
rlm@23 70 (let [byte-buffer (atom nil)
rlm@113 71 renderer (atom nil)
rlm@113 72 image (atom nil)]
rlm@23 73 (proxy [SceneProcessor] []
rlm@23 74 (initialize
rlm@23 75 [renderManager viewPort]
rlm@23 76 (let [cam (.getCamera viewPort)
rlm@23 77 width (.getWidth cam)
rlm@23 78 height (.getHeight cam)]
rlm@23 79 (reset! renderer (.getRenderer renderManager))
rlm@23 80 (reset! byte-buffer
rlm@23 81 (BufferUtils/createByteBuffer
rlm@113 82 (* width height 4)))
rlm@113 83 (reset! image (BufferedImage.
rlm@113 84 width height
rlm@113 85 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@23 86 (isInitialized [] (not (nil? @byte-buffer)))
rlm@23 87 (reshape [_ _ _])
rlm@23 88 (preFrame [_])
rlm@23 89 (postQueue [_])
rlm@23 90 (postFrame
rlm@23 91 [#^FrameBuffer fb]
rlm@23 92 (.clear @byte-buffer)
rlm@113 93 (continuation @renderer fb @byte-buffer @image))
rlm@23 94 (cleanup []))))
rlm@213 95 #+end_src
rlm@213 96
rlm@273 97 The continuation function given to =vision-pipeline= above will be
rlm@213 98 given a =Renderer= and three containers for image data. The
rlm@218 99 =FrameBuffer= references the GPU image data, but the pixel data can
rlm@218 100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
rlm@219 101 are initially "empty" but are sized to hold the data in the
rlm@306 102 =FrameBuffer=. I call transferring the GPU image data to the CPU
rlm@213 103 structures "mixing" the image data. I have provided three functions to
rlm@213 104 do this mixing.
rlm@213 105
rlm@213 106 #+name: pipeline-2
rlm@213 107 #+begin_src clojure
rlm@113 108 (defn frameBuffer->byteBuffer!
rlm@113 109 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
rlm@113 110 the CPU (ByteBuffer)."
rlm@113 111 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
rlm@113 112 (.readFrameBuffer r fb bb) bb)
rlm@113 113
rlm@113 114 (defn byteBuffer->bufferedImage!
rlm@113 115 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
rlm@113 116 style ABGR image data and place it in BufferedImage bi."
rlm@113 117 [#^ByteBuffer bb #^BufferedImage bi]
rlm@113 118 (Screenshots/convertScreenShot bb bi) bi)
rlm@113 119
rlm@113 120 (defn BufferedImage!
rlm@113 121 "Continuation which will grab the buffered image from the materials
rlm@113 122 provided by (vision-pipeline)."
rlm@113 123 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
rlm@113 124 (byteBuffer->bufferedImage!
rlm@113 125 (frameBuffer->byteBuffer! r fb bb) bi))
rlm@213 126 #+end_src
rlm@112 127
rlm@213 128 Note that it is possible to write vision processing algorithms
rlm@213 129 entirely in terms of =BufferedImage= inputs. Just compose that
rlm@273 130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision
rlm@213 131 processing algorithm that is entirely hosted on the GPU does not have
rlm@306 132 to pay for this convenience.
rlm@213 133
ocsenave@265 134 * Optical sensor arrays are described with images and referenced with metadata
rlm@214 135 The vision pipeline described above handles the flow of rendered
rlm@214 136 images. Now, we need simulated eyes to serve as the source of these
rlm@214 137 images.
rlm@214 138
rlm@214 139 An eye is described in blender in the same way as a joint. They are
rlm@214 140 zero dimensional empty objects with no geometry whose local coordinate
rlm@214 141 system determines the orientation of the resulting eye. All eyes are
rlm@306 142 children of a parent node named "eyes" just as all joints have a
rlm@214 143 parent named "joints". An eye binds to the nearest physical object
rlm@273 144 with =bind-sense=.
rlm@214 145
rlm@214 146 #+name: add-eye
rlm@214 147 #+begin_src clojure
rlm@215 148 (in-ns 'cortex.vision)
rlm@215 149
rlm@214 150 (defn add-eye!
rlm@214 151 "Create a Camera centered on the current position of 'eye which
rlm@338 152 follows the closest physical node in 'creature. The camera will
rlm@338 153 point in the X direction and use the Z vector as up as determined
rlm@338 154 by the rotation of these vectors in blender coordinate space. Use
rlm@338 155 XZY rotation for the node in blender."
rlm@214 156 [#^Node creature #^Spatial eye]
rlm@214 157 (let [target (closest-node creature eye)
rlm@338 158 [cam-width cam-height]
rlm@338 159 ;;[640 480] ;; graphics card on laptop doesn't support
rlm@338 160 ;; arbitray dimensions.
rlm@338 161 (eye-dimensions eye)
rlm@215 162 cam (Camera. cam-width cam-height)
rlm@215 163 rot (.getWorldRotation eye)]
rlm@214 164 (.setLocation cam (.getWorldTranslation eye))
rlm@218 165 (.lookAtDirection
rlm@338 166 cam ; this part is not a mistake and
rlm@338 167 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
rlm@338 168 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
rlm@214 169 (.setFrustumPerspective
rlm@338 170 cam (float 45)
rlm@338 171 (float (/ (.getWidth cam) (.getHeight cam)))
rlm@338 172 (float 1)
rlm@338 173 (float 1000))
rlm@215 174 (bind-sense target cam) cam))
rlm@214 175 #+end_src
rlm@214 176
rlm@338 177 #+results: add-eye
rlm@338 178 : #'cortex.vision/add-eye!
rlm@338 179
rlm@214 180 Here, the camera is created based on metadata on the eye-node and
rlm@273 181 attached to the nearest physical object with =bind-sense=
rlm@214 182 ** The Retina
rlm@214 183
rlm@214 184 An eye is a surface (the retina) which contains many discrete sensors
rlm@218 185 to detect light. These sensors have can have different light-sensing
rlm@214 186 properties. In humans, each discrete sensor is sensitive to red,
rlm@214 187 blue, green, or gray. These different types of sensors can have
rlm@214 188 different spatial distributions along the retina. In humans, there is
rlm@214 189 a fovea in the center of the retina which has a very high density of
rlm@214 190 color sensors, and a blind spot which has no sensors at all. Sensor
rlm@219 191 density decreases in proportion to distance from the fovea.
rlm@214 192
rlm@214 193 I want to be able to model any retinal configuration, so my eye-nodes
rlm@214 194 in blender contain metadata pointing to images that describe the
rlm@306 195 precise position of the individual sensors using white pixels. The
rlm@306 196 meta-data also describes the precise sensitivity to light that the
rlm@214 197 sensors described in the image have. An eye can contain any number of
rlm@214 198 these images. For example, the metadata for an eye might look like
rlm@214 199 this:
rlm@214 200
rlm@214 201 #+begin_src clojure
rlm@214 202 {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@214 203 #+end_src
rlm@214 204
rlm@214 205 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
rlm@214 206 [[../assets/Models/test-creature/retina-small.png]]
rlm@214 207
rlm@214 208 Together, the number 0xFF0000 and the image image above describe the
rlm@214 209 placement of red-sensitive sensory elements.
rlm@214 210
rlm@214 211 Meta-data to very crudely approximate a human eye might be something
rlm@214 212 like this:
rlm@214 213
rlm@214 214 #+begin_src clojure
rlm@214 215 (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@214 216 {0xFF0000 retinal-profile
rlm@214 217 0x00FF00 retinal-profile
rlm@214 218 0x0000FF retinal-profile
rlm@214 219 0xFFFFFF retinal-profile})
rlm@214 220 #+end_src
rlm@214 221
rlm@214 222 The numbers that serve as keys in the map determine a sensor's
rlm@214 223 relative sensitivity to the channels red, green, and blue. These
rlm@218 224 sensitivity values are packed into an integer in the order =|_|R|G|B|=
rlm@218 225 in 8-bit fields. The RGB values of a pixel in the image are added
rlm@306 226 together with these sensitivities as linear weights. Therefore,
rlm@214 227 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
rlm@214 228 all colors equally (gray).
rlm@214 229
rlm@306 230 For convenience I've defined a few symbols for the more common
rlm@214 231 sensitivity values.
rlm@214 232
rlm@214 233 #+name: sensitivity
rlm@214 234 #+begin_src clojure
rlm@317 235 (def sensitivity-presets
rlm@317 236 "Retinal sensitivity presets for sensors that extract one channel
rlm@317 237 (:red :blue :green) or average all channels (:all)"
rlm@214 238 {:all 0xFFFFFF
rlm@214 239 :red 0xFF0000
rlm@214 240 :blue 0x0000FF
rlm@317 241 :green 0x00FF00})
rlm@214 242 #+end_src
rlm@214 243
rlm@214 244 ** Metadata Processing
rlm@214 245
rlm@273 246 =retina-sensor-profile= extracts a map from the eye-node in the same
rlm@273 247 format as the example maps above. =eye-dimensions= finds the
rlm@219 248 dimensions of the smallest image required to contain all the retinal
rlm@214 249 sensor maps.
rlm@214 250
rlm@216 251 #+name: retina
rlm@214 252 #+begin_src clojure
rlm@214 253 (defn retina-sensor-profile
rlm@214 254 "Return a map of pixel sensitivity numbers to BufferedImages
rlm@214 255 describing the distribution of light-sensitive components of this
rlm@214 256 eye. :red, :green, :blue, :gray are already defined as extracting
rlm@214 257 the red, green, blue, and average components respectively."
rlm@214 258 [#^Spatial eye]
rlm@214 259 (if-let [eye-map (meta-data eye "eye")]
rlm@214 260 (map-vals
rlm@214 261 load-image
rlm@214 262 (eval (read-string eye-map)))))
rlm@214 263
rlm@218 264 (defn eye-dimensions
rlm@218 265 "Returns [width, height] determined by the metadata of the eye."
rlm@214 266 [#^Spatial eye]
rlm@214 267 (let [dimensions
rlm@214 268 (map #(vector (.getWidth %) (.getHeight %))
rlm@214 269 (vals (retina-sensor-profile eye)))]
rlm@214 270 [(apply max (map first dimensions))
rlm@214 271 (apply max (map second dimensions))]))
rlm@214 272 #+end_src
rlm@214 273
ocsenave@265 274 * Importing and parsing descriptions of eyes.
rlm@214 275 First off, get the children of the "eyes" empty node to find all the
rlm@214 276 eyes the creature has.
rlm@216 277 #+name: eye-node
rlm@214 278 #+begin_src clojure
rlm@317 279 (def
rlm@317 280 ^{:doc "Return the children of the creature's \"eyes\" node."
rlm@317 281 :arglists '([creature])}
rlm@214 282 eyes
rlm@317 283 (sense-nodes "eyes"))
rlm@214 284 #+end_src
rlm@214 285
rlm@273 286 Then, add the camera created by =add-eye!= to the simulation by
rlm@215 287 creating a new viewport.
rlm@214 288
rlm@216 289 #+name: add-camera
rlm@213 290 #+begin_src clojure
rlm@338 291 (in-ns 'cortex.vision)
rlm@169 292 (defn add-camera!
rlm@169 293 "Add a camera to the world, calling continuation on every frame
rlm@34 294 produced."
rlm@167 295 [#^Application world camera continuation]
rlm@23 296 (let [width (.getWidth camera)
rlm@23 297 height (.getHeight camera)
rlm@23 298 render-manager (.getRenderManager world)
rlm@23 299 viewport (.createMainView render-manager "eye-view" camera)]
rlm@23 300 (doto viewport
rlm@23 301 (.setClearFlags true true true)
rlm@112 302 (.setBackgroundColor ColorRGBA/Black)
rlm@113 303 (.addProcessor (vision-pipeline continuation))
rlm@23 304 (.attachScene (.getRootNode world)))))
rlm@215 305 #+end_src
rlm@151 306
rlm@338 307 #+results: add-camera
rlm@338 308 : #'cortex.vision/add-camera!
rlm@338 309
rlm@151 310
rlm@218 311 The eye's continuation function should register the viewport with the
rlm@218 312 simulation the first time it is called, use the CPU to extract the
rlm@215 313 appropriate pixels from the rendered image and weight them by each
rlm@218 314 sensor's sensitivity. I have the option to do this processing in
rlm@218 315 native code for a slight gain in speed. I could also do it in the GPU
rlm@273 316 for a massive gain in speed. =vision-kernel= generates a list of
rlm@218 317 such continuation functions, one for each channel of the eye.
rlm@151 318
rlm@216 319 #+name: kernel
rlm@215 320 #+begin_src clojure
rlm@215 321 (in-ns 'cortex.vision)
rlm@151 322
rlm@215 323 (defrecord attached-viewport [vision-fn viewport-fn]
rlm@215 324 clojure.lang.IFn
rlm@215 325 (invoke [this world] (vision-fn world))
rlm@215 326 (applyTo [this args] (apply vision-fn args)))
rlm@151 327
rlm@216 328 (defn pixel-sense [sensitivity pixel]
rlm@216 329 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
rlm@216 330 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
rlm@216 331 s-b (bit-and 0x0000FF sensitivity)
rlm@216 332
rlm@216 333 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
rlm@216 334 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
rlm@216 335 p-b (bit-and 0x0000FF pixel)
rlm@216 336
rlm@216 337 total-sensitivity (* 255 (+ s-r s-g s-b))]
rlm@216 338 (float (/ (+ (* s-r p-r)
rlm@216 339 (* s-g p-g)
rlm@216 340 (* s-b p-b))
rlm@216 341 total-sensitivity))))
rlm@216 342
rlm@215 343 (defn vision-kernel
rlm@171 344 "Returns a list of functions, each of which will return a color
rlm@171 345 channel's worth of visual information when called inside a running
rlm@171 346 simulation."
rlm@151 347 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@169 348 (let [retinal-map (retina-sensor-profile eye)
rlm@169 349 camera (add-eye! creature eye)
rlm@151 350 vision-image
rlm@151 351 (atom
rlm@151 352 (BufferedImage. (.getWidth camera)
rlm@151 353 (.getHeight camera)
rlm@170 354 BufferedImage/TYPE_BYTE_BINARY))
rlm@170 355 register-eye!
rlm@170 356 (runonce
rlm@170 357 (fn [world]
rlm@170 358 (add-camera!
rlm@170 359 world camera
rlm@170 360 (let [counter (atom 0)]
rlm@170 361 (fn [r fb bb bi]
rlm@170 362 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@170 363 (reset! vision-image
rlm@170 364 (BufferedImage! r fb bb bi))))))))]
rlm@151 365 (vec
rlm@151 366 (map
rlm@151 367 (fn [[key image]]
rlm@151 368 (let [whites (white-coordinates image)
rlm@151 369 topology (vec (collapse whites))
rlm@216 370 sensitivity (sensitivity-presets key key)]
rlm@215 371 (attached-viewport.
rlm@215 372 (fn [world]
rlm@215 373 (register-eye! world)
rlm@215 374 (vector
rlm@215 375 topology
rlm@215 376 (vec
rlm@215 377 (for [[x y] whites]
rlm@216 378 (pixel-sense
rlm@216 379 sensitivity
rlm@216 380 (.getRGB @vision-image x y))))))
rlm@215 381 register-eye!)))
rlm@215 382 retinal-map))))
rlm@151 383
rlm@215 384 (defn gen-fix-display
rlm@215 385 "Create a function to call to restore a simulation's display when it
rlm@215 386 is disrupted by a Viewport."
rlm@215 387 []
rlm@215 388 (runonce
rlm@215 389 (fn [world]
rlm@215 390 (add-camera! world (.getCamera world) no-op))))
rlm@215 391 #+end_src
rlm@170 392
rlm@273 393 Note that since each of the functions generated by =vision-kernel=
rlm@273 394 shares the same =register-eye!= function, the eye will be registered
rlm@215 395 only once the first time any of the functions from the list returned
rlm@273 396 by =vision-kernel= is called. Each of the functions returned by
rlm@273 397 =vision-kernel= also allows access to the =Viewport= through which
rlm@306 398 it receives images.
rlm@215 399
rlm@306 400 The in-game display can be disrupted by all the ViewPorts that the
rlm@306 401 functions generated by =vision-kernel= add. This doesn't affect the
rlm@215 402 simulation or the simulated senses, but can be annoying.
rlm@273 403 =gen-fix-display= restores the in-simulation display.
rlm@215 404
ocsenave@265 405 ** The =vision!= function creates sensory probes.
rlm@215 406
rlm@218 407 All the hard work has been done; all that remains is to apply
rlm@273 408 =vision-kernel= to each eye in the creature and gather the results
rlm@215 409 into one list of functions.
rlm@215 410
rlm@216 411 #+name: main
rlm@215 412 #+begin_src clojure
rlm@170 413 (defn vision!
rlm@170 414 "Returns a function which returns visual sensory data when called
rlm@218 415 inside a running simulation."
rlm@151 416 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@151 417 (reduce
rlm@170 418 concat
rlm@167 419 (for [eye (eyes creature)]
rlm@215 420 (vision-kernel creature eye))))
rlm@215 421 #+end_src
rlm@151 422
ocsenave@265 423 ** Displaying visual data for debugging.
ocsenave@265 424 # Visualization of Vision. Maybe less alliteration would be better.
rlm@215 425 It's vital to have a visual representation for each sense. Here I use
rlm@273 426 =view-sense= to construct a function that will create a display for
rlm@215 427 visual data.
rlm@215 428
rlm@216 429 #+name: display
rlm@215 430 #+begin_src clojure
rlm@216 431 (in-ns 'cortex.vision)
rlm@216 432
rlm@189 433 (defn view-vision
rlm@189 434 "Creates a function which accepts a list of visual sensor-data and
rlm@189 435 displays each element of the list to the screen."
rlm@189 436 []
rlm@188 437 (view-sense
rlm@188 438 (fn
rlm@188 439 [[coords sensor-data]]
rlm@188 440 (let [image (points->image coords)]
rlm@188 441 (dorun
rlm@188 442 (for [i (range (count coords))]
rlm@188 443 (.setRGB image ((coords i) 0) ((coords i) 1)
rlm@216 444 (gray (int (* 255 (sensor-data i)))))))
rlm@189 445 image))))
rlm@34 446 #+end_src
rlm@23 447
ocsenave@264 448 * Demonstrations
ocsenave@264 449 ** Demonstrating the vision pipeline.
rlm@23 450
rlm@215 451 This is a basic test for the vision system. It only tests the
ocsenave@264 452 vision-pipeline and does not deal with loading eyes from a blender
rlm@215 453 file. The code creates two videos of the same rotating cube from
rlm@215 454 different angles.
rlm@23 455
rlm@215 456 #+name: test-1
rlm@23 457 #+begin_src clojure
rlm@215 458 (in-ns 'cortex.test.vision)
rlm@23 459
rlm@219 460 (defn test-pipeline
rlm@69 461 "Testing vision:
rlm@69 462 Tests the vision system by creating two views of the same rotating
rlm@69 463 object from different angles and displaying both of those views in
rlm@69 464 JFrames.
rlm@69 465
rlm@69 466 You should see a rotating cube, and two windows,
rlm@69 467 each displaying a different view of the cube."
rlm@283 468 ([] (test-pipeline false))
rlm@283 469 ([record?]
rlm@283 470 (let [candy
rlm@283 471 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@283 472 (world
rlm@283 473 (doto (Node.)
rlm@283 474 (.attachChild candy))
rlm@283 475 {}
rlm@283 476 (fn [world]
rlm@283 477 (let [cam (.clone (.getCamera world))
rlm@283 478 width (.getWidth cam)
rlm@283 479 height (.getHeight cam)]
rlm@283 480 (add-camera! world cam
rlm@283 481 (comp
rlm@283 482 (view-image
rlm@283 483 (if record?
rlm@283 484 (File. "/home/r/proj/cortex/render/vision/1")))
rlm@283 485 BufferedImage!))
rlm@283 486 (add-camera! world
rlm@283 487 (doto (.clone cam)
rlm@283 488 (.setLocation (Vector3f. -10 0 0))
rlm@283 489 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@283 490 (comp
rlm@283 491 (view-image
rlm@283 492 (if record?
rlm@283 493 (File. "/home/r/proj/cortex/render/vision/2")))
rlm@283 494 BufferedImage!))
rlm@283 495 ;; This is here to restore the main view
rlm@112 496 ;; after the other views have completed processing
rlm@283 497 (add-camera! world (.getCamera world) no-op)))
rlm@283 498 (fn [world tpf]
rlm@283 499 (.rotate candy (* tpf 0.2) 0 0))))))
rlm@23 500 #+end_src
rlm@23 501
rlm@215 502 #+begin_html
rlm@215 503 <div class="figure">
rlm@215 504 <video controls="controls" width="755">
rlm@215 505 <source src="../video/spinning-cube.ogg" type="video/ogg"
rlm@215 506 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@215 507 </video>
rlm@309 508 <br> <a href="http://youtu.be/r5Bn2aG7MO0"> YouTube </a>
rlm@215 509 <p>A rotating cube viewed from two different perspectives.</p>
rlm@215 510 </div>
rlm@215 511 #+end_html
rlm@215 512
rlm@215 513 Creating multiple eyes like this can be used for stereoscopic vision
rlm@215 514 simulation in a single creature or for simulating multiple creatures,
rlm@215 515 each with their own sense of vision.
ocsenave@264 516 ** Demonstrating eye import and parsing.
rlm@215 517
rlm@218 518 To the worm from the last post, I add a new node that describes its
rlm@215 519 eyes.
rlm@215 520
rlm@215 521 #+attr_html: width=755
rlm@215 522 #+caption: The worm with newly added empty nodes describing a single eye.
rlm@215 523 [[../images/worm-with-eye.png]]
rlm@215 524
rlm@215 525 The node highlighted in yellow is the root level "eyes" node. It has
rlm@218 526 a single child, highlighted in orange, which describes a single
rlm@218 527 eye. This is the "eye" node. It is placed so that the worm will have
rlm@218 528 an eye located in the center of the flat portion of its lower
rlm@218 529 hemispherical section.
rlm@218 530
rlm@218 531 The two nodes which are not highlighted describe the single joint of
rlm@218 532 the worm.
rlm@215 533
rlm@215 534 The metadata of the eye-node is:
rlm@215 535
rlm@215 536 #+begin_src clojure :results verbatim :exports both
rlm@215 537 (cortex.sense/meta-data
rlm@218 538 (.getChild (.getChild (cortex.test.body/worm) "eyes") "eye") "eye")
rlm@215 539 #+end_src
rlm@215 540
rlm@215 541 #+results:
rlm@215 542 : "(let [retina \"Models/test-creature/retina-small.png\"]
rlm@215 543 : {:all retina :red retina :green retina :blue retina})"
rlm@215 544
rlm@215 545 This is the approximation to the human eye described earlier.
rlm@215 546
rlm@216 547 #+name: test-2
rlm@215 548 #+begin_src clojure
rlm@215 549 (in-ns 'cortex.test.vision)
rlm@215 550
rlm@216 551 (defn change-color [obj color]
rlm@321 552 ;;(println-repl obj)
rlm@216 553 (if obj
rlm@216 554 (.setColor (.getMaterial obj) "Color" color)))
rlm@216 555
rlm@216 556 (defn colored-cannon-ball [color]
rlm@216 557 (comp #(change-color % color)
rlm@216 558 (fire-cannon-ball)))
rlm@215 559
rlm@338 560 (defn gen-worm
rlm@338 561 "create a creature acceptable for testing as a replacement for the
rlm@338 562 worm."
rlm@338 563 []
rlm@338 564 (nodify
rlm@338 565 "worm"
rlm@338 566 [(nodify
rlm@338 567 "eyes"
rlm@338 568 [(doto
rlm@338 569 (Node. "eye1")
rlm@338 570 (.setLocalTranslation (Vector3f. 0 -1.1 0))
rlm@338 571 (.setUserData
rlm@338 572
rlm@338 573 "eye"
rlm@338 574 "(let [retina
rlm@338 575 \"Models/test-creature/retina-small.png\"]
rlm@338 576 {:all retina :red retina
rlm@338 577 :green retina :blue retina})"))])
rlm@338 578 (box
rlm@338 579 0.2 0.2 0.2
rlm@338 580 :name "worm-segment"
rlm@338 581 :position (Vector3f. 0 0 0)
rlm@338 582 :color ColorRGBA/Orange)]))
rlm@338 583
rlm@338 584
rlm@338 585
rlm@283 586 (defn test-worm-vision
rlm@321 587 "Testing vision:
rlm@321 588 You should see the worm suspended in mid-air, looking down at a
rlm@321 589 table. There are four small displays, one each for red, green blue,
rlm@321 590 and gray channels. You can fire balls of various colors, and the
rlm@321 591 four channels should react accordingly.
rlm@321 592
rlm@321 593 Keys:
rlm@321 594 r : fire red-ball
rlm@321 595 b : fire blue-ball
rlm@321 596 g : fire green-ball
rlm@321 597 <space> : fire white ball"
rlm@338 598
rlm@283 599 ([] (test-worm-vision false))
rlm@283 600 ([record?]
rlm@283 601 (let [the-worm (doto (worm)(body!))
rlm@338 602 ;;the-worm (gen-worm)
rlm@338 603 ;;vision (vision! the-worm)
rlm@337 604 ;;vision-display (view-vision)
rlm@338 605 ;;fix-display (gen-fix-display)
rlm@283 606 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
rlm@283 607 x-axis
rlm@283 608 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
rlm@283 609 :position (Vector3f. 0 -5 0))
rlm@283 610 y-axis
rlm@283 611 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
rlm@283 612 :position (Vector3f. 0 -5 0))
rlm@283 613 z-axis
rlm@283 614 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
rlm@283 615 :position (Vector3f. 0 -5 0))
rlm@338 616 timer (RatchetTimer. 60)
rlm@338 617 ]
rlm@215 618
rlm@335 619 (world
rlm@335 620 (nodify [(floor) the-worm x-axis y-axis z-axis me])
rlm@338 621 standard-debug-controls
rlm@338 622 ;;"key-r" (colored-cannon-ball ColorRGBA/Red)
rlm@338 623 ;;"key-b" (colored-cannon-ball ColorRGBA/Blue)
rlm@338 624 ;;"key-g" (colored-cannon-ball ColorRGBA/Green))
rlm@338 625
rlm@335 626 (fn [world]
rlm@338 627 (let
rlm@338 628 [eye-pos (Vector3f. 0 30 0)
rlm@338 629 cam (doto
rlm@338 630 (.clone (.getCamera world))
rlm@338 631 (.setLocation eye-pos)
rlm@338 632 (.lookAt Vector3f/ZERO
rlm@338 633 Vector3f/UNIT_X))
rlm@338 634
rlm@338 635 bad-cam (doto
rlm@338 636 ;;saved-cam
rlm@338 637 ;;(.clone (.getCamera world))
rlm@338 638
rlm@338 639 (com.jme3.renderer.Camera. 640 480)
rlm@338 640 (.setFrustumPerspective
rlm@338 641 (float 45)
rlm@338 642 (float (/ 640 480))
rlm@338 643 (float 1)
rlm@338 644 (float 1000))
rlm@338 645
rlm@338 646 (.setLocation eye-pos)
rlm@338 647 (.lookAt Vector3f/ZERO
rlm@338 648 Vector3f/UNIT_X))
rlm@338 649
rlm@338 650 bad-cam (add-eye! the-worm (first (eyes the-worm)))
rlm@338 651 ]
rlm@338 652
rlm@338 653
rlm@338 654 (light-up-everything world)
rlm@338 655 ;;(speed-up world)
rlm@338 656 (.setTimer world timer)
rlm@338 657 ;;(display-dilated-time world timer)
rlm@338 658 ;; add a view from the worm's perspective
rlm@338 659 (if record?
rlm@338 660 (Capture/captureVideo
rlm@338 661 world
rlm@338 662 (File.
rlm@338 663 "/home/r/proj/cortex/render/worm-vision/main-view")))
rlm@338 664
rlm@338 665 (bind-sense (last (node-seq the-worm)) cam)
rlm@338 666 (bind-sense (last (node-seq the-worm)) bad-cam)
rlm@338 667
rlm@338 668 (add-camera!
rlm@335 669 world
rlm@338 670 bad-cam
rlm@338 671 (comp
rlm@338 672 (view-image
rlm@338 673 (if record?
rlm@338 674 (File.
rlm@338 675 "/home/r/proj/cortex/render/worm-vision/worm-view")))
rlm@338 676 BufferedImage!))
rlm@338 677
rlm@338 678
rlm@338 679
rlm@338 680 (add-camera!
rlm@338 681 world cam
rlm@338 682 (comp
rlm@338 683 (view-image
rlm@338 684 (if record?
rlm@338 685 (File.
rlm@338 686 "/home/r/proj/cortex/render/worm-vision/worm-view")))
rlm@338 687 BufferedImage!))
rlm@338 688 (set-gravity world Vector3f/ZERO)
rlm@338 689 (add-camera! world (.getCamera world) no-op)
rlm@338 690
rlm@338 691 (println-repl cam "\n" bad-cam)
rlm@338 692
rlm@338 693 ))
rlm@335 694
rlm@335 695 (fn [world _ ]
rlm@335 696 (.setLocalTranslation me (.getLocation (.getCamera world)))
rlm@337 697 ;; (vision-display
rlm@337 698 ;; (map #(% world) vision)
rlm@337 699 ;; (if record?
rlm@337 700 ;; (File. "/home/r/proj/cortex/render/worm-vision")))
rlm@338 701 ;;(fix-display world)
rlm@335 702 )))))
rlm@215 703 #+end_src
rlm@215 704
rlm@335 705 #+RESULTS: test-2
rlm@337 706 : #'cortex.test.vision/test-worm-vision
rlm@335 707
rlm@335 708
rlm@218 709 The world consists of the worm and a flat gray floor. I can shoot red,
rlm@218 710 green, blue and white cannonballs at the worm. The worm is initially
rlm@218 711 looking down at the floor, and there is no gravity. My perspective
rlm@218 712 (the Main View), the worm's perspective (Worm View) and the 4 sensor
rlm@218 713 channels that comprise the worm's eye are all saved frame-by-frame to
rlm@218 714 disk.
rlm@218 715
rlm@218 716 * Demonstration of Vision
rlm@218 717 #+begin_html
rlm@218 718 <div class="figure">
rlm@218 719 <video controls="controls" width="755">
rlm@218 720 <source src="../video/worm-vision.ogg" type="video/ogg"
rlm@218 721 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@218 722 </video>
rlm@309 723 <br> <a href="http://youtu.be/J3H3iB_2NPQ"> YouTube </a>
rlm@218 724 <p>Simulated Vision in a Virtual Environment</p>
rlm@218 725 </div>
rlm@218 726 #+end_html
rlm@218 727
rlm@218 728 ** Generate the Worm Video from Frames
rlm@216 729 #+name: magick2
rlm@216 730 #+begin_src clojure
rlm@216 731 (ns cortex.video.magick2
rlm@216 732 (:import java.io.File)
rlm@316 733 (:use clojure.java.shell))
rlm@216 734
rlm@216 735 (defn images [path]
rlm@216 736 (sort (rest (file-seq (File. path)))))
rlm@216 737
rlm@216 738 (def base "/home/r/proj/cortex/render/worm-vision/")
rlm@216 739
rlm@216 740 (defn pics [file]
rlm@216 741 (images (str base file)))
rlm@216 742
rlm@216 743 (defn combine-images []
rlm@216 744 (let [main-view (pics "main-view")
rlm@216 745 worm-view (pics "worm-view")
rlm@216 746 blue (pics "0")
rlm@216 747 green (pics "1")
rlm@216 748 red (pics "2")
rlm@216 749 gray (pics "3")
rlm@216 750 blender (let [b-pics (pics "blender")]
rlm@216 751 (concat b-pics (repeat 9001 (last b-pics))))
rlm@216 752 background (repeat 9001 (File. (str base "background.png")))
rlm@216 753 targets (map
rlm@216 754 #(File. (str base "out/" (format "%07d.png" %)))
rlm@216 755 (range 0 (count main-view)))]
rlm@216 756 (dorun
rlm@216 757 (pmap
rlm@216 758 (comp
rlm@216 759 (fn [[background main-view worm-view red green blue gray blender target]]
rlm@216 760 (println target)
rlm@216 761 (sh "convert"
rlm@216 762 background
rlm@216 763 main-view "-geometry" "+18+17" "-composite"
rlm@216 764 worm-view "-geometry" "+677+17" "-composite"
rlm@216 765 green "-geometry" "+685+430" "-composite"
rlm@216 766 red "-geometry" "+788+430" "-composite"
rlm@216 767 blue "-geometry" "+894+430" "-composite"
rlm@216 768 gray "-geometry" "+1000+430" "-composite"
rlm@216 769 blender "-geometry" "+0+0" "-composite"
rlm@216 770 target))
rlm@216 771 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@216 772 background main-view worm-view red green blue gray blender targets))))
rlm@216 773 #+end_src
rlm@216 774
rlm@216 775 #+begin_src sh :results silent
rlm@216 776 cd /home/r/proj/cortex/render/worm-vision
rlm@216 777 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
rlm@216 778 #+end_src
rlm@236 779
ocsenave@265 780 * Onward!
ocsenave@265 781 - As a neat bonus, this idea behind simulated vision also enables one
ocsenave@265 782 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
ocsenave@265 783 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
ocsenave@265 784 #+appendix
ocsenave@265 785
rlm@215 786 * Headers
rlm@215 787
rlm@213 788 #+name: vision-header
rlm@213 789 #+begin_src clojure
rlm@213 790 (ns cortex.vision
rlm@213 791 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
rlm@213 792 eyes from different positions to observe the same world, and pass
rlm@306 793 the observed data to any arbitrary function. Automatically reads
rlm@216 794 eye-nodes from specially prepared blender files and instantiates
rlm@213 795 them in the world as actual eyes."
rlm@213 796 {:author "Robert McIntyre"}
rlm@213 797 (:use (cortex world sense util))
rlm@213 798 (:import com.jme3.post.SceneProcessor)
rlm@237 799 (:import (com.jme3.util BufferUtils Screenshots))
rlm@213 800 (:import java.nio.ByteBuffer)
rlm@213 801 (:import java.awt.image.BufferedImage)
rlm@213 802 (:import (com.jme3.renderer ViewPort Camera))
rlm@216 803 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
rlm@213 804 (:import com.jme3.renderer.Renderer)
rlm@213 805 (:import com.jme3.app.Application)
rlm@213 806 (:import com.jme3.texture.FrameBuffer)
rlm@213 807 (:import (com.jme3.scene Node Spatial)))
rlm@213 808 #+end_src
rlm@112 809
rlm@215 810 #+name: test-header
rlm@215 811 #+begin_src clojure
rlm@215 812 (ns cortex.test.vision
rlm@215 813 (:use (cortex world sense util body vision))
rlm@215 814 (:use cortex.test.body)
rlm@215 815 (:import java.awt.image.BufferedImage)
rlm@215 816 (:import javax.swing.JPanel)
rlm@215 817 (:import javax.swing.SwingUtilities)
rlm@215 818 (:import java.awt.Dimension)
rlm@215 819 (:import javax.swing.JFrame)
rlm@215 820 (:import com.jme3.math.ColorRGBA)
rlm@215 821 (:import com.jme3.scene.Node)
rlm@215 822 (:import com.jme3.math.Vector3f)
rlm@216 823 (:import java.io.File)
rlm@216 824 (:import (com.aurellem.capture Capture RatchetTimer)))
rlm@215 825 #+end_src
rlm@216 826 * Source Listing
rlm@216 827 - [[../src/cortex/vision.clj][cortex.vision]]
rlm@216 828 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
rlm@216 829 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
rlm@216 830 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
rlm@216 831 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
rlm@216 832 - [[http://hg.bortreb.com ][source-repository]]
rlm@216 833
rlm@35 834
rlm@273 835 * Next
rlm@273 836 I find some [[./hearing.org][ears]] for the creature while exploring the guts of
rlm@273 837 jMonkeyEngine's sound system.
rlm@24 838
rlm@212 839 * COMMENT Generate Source
rlm@34 840 #+begin_src clojure :tangle ../src/cortex/vision.clj
rlm@216 841 <<vision-header>>
rlm@216 842 <<pipeline-1>>
rlm@216 843 <<pipeline-2>>
rlm@216 844 <<retina>>
rlm@216 845 <<add-eye>>
rlm@216 846 <<sensitivity>>
rlm@216 847 <<eye-node>>
rlm@216 848 <<add-camera>>
rlm@216 849 <<kernel>>
rlm@216 850 <<main>>
rlm@216 851 <<display>>
rlm@24 852 #+end_src
rlm@24 853
rlm@68 854 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
rlm@215 855 <<test-header>>
rlm@215 856 <<test-1>>
rlm@216 857 <<test-2>>
rlm@24 858 #+end_src
rlm@216 859
rlm@216 860 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
rlm@216 861 <<magick2>>
rlm@216 862 #+end_src