Mercurial > cortex

annotate org/vision.org @ 216:f5ea63245b3b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
completed vision demonstration video and first draft of vision.org
author Robert McIntyre <rlm@mit.edu>
date Fri, 10 Feb 2012 11:34:07 -0700
parents f283c62bd212
children ac46ee4e574a
rev   line source
rlm@34 1 #+title: Simulated Sense of Sight
rlm@23 2 #+author: Robert McIntyre
rlm@23 3 #+email: rlm@mit.edu
rlm@38 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
rlm@34 5 #+keywords: computer vision, jMonkeyEngine3, clojure
rlm@23 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@23 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@23 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@23 9
rlm@194 10 * Vision
rlm@216 11
rlm@212 12 Vision is one of the most important senses for humans, so I need to
rlm@212 13 build a simulated sense of vision for my AI. I will do this with
rlm@212 14 simulated eyes. Each eye can be independely moved and should see its
rlm@212 15 own version of the world depending on where it is.
rlm@212 16
rlm@212 17 Making these simulated eyes a reality is fairly simple bacause
rlm@212 18 jMonkeyEngine already conatains extensive support for multiple views
rlm@212 19 of the same 3D simulated world. The reason jMonkeyEngine has this
rlm@212 20 support is because the support is necessary to create games with
rlm@212 21 split-screen views. Multiple views are also used to create efficient
rlm@212 22 pseudo-reflections by rendering the scene from a certain perspective
rlm@212 23 and then projecting it back onto a surface in the 3D world.
rlm@212 24
rlm@212 25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye
rlm@212 26 [[../images/goldeneye-4-player.png]]
rlm@212 27
rlm@213 28 * Brief Description of jMonkeyEngine's Rendering Pipeline
rlm@212 29
rlm@213 30 jMonkeyEngine allows you to create a =ViewPort=, which represents a
rlm@213 31 view of the simulated world. You can create as many of these as you
rlm@213 32 want. Every frame, the =RenderManager= iterates through each
rlm@213 33 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
rlm@213 34 is a =FrameBuffer= which represents the rendered image in the GPU.
rlm@151 35
rlm@213 36 Each =ViewPort= can have any number of attached =SceneProcessor=
rlm@213 37 objects, which are called every time a new frame is rendered. A
rlm@213 38 =SceneProcessor= recieves a =FrameBuffer= and can do whatever it wants
rlm@213 39 to the data. Often this consists of invoking GPU specific operations
rlm@213 40 on the rendered image. The =SceneProcessor= can also copy the GPU
rlm@213 41 image data to RAM and process it with the CPU.
rlm@151 42
rlm@213 43 * The Vision Pipeline
rlm@151 44
rlm@213 45 Each eye in the simulated creature needs it's own =ViewPort= so that
rlm@213 46 it can see the world from its own perspective. To this =ViewPort=, I
rlm@214 47 add a =SceneProcessor= that feeds the visual data to any arbitray
rlm@213 48 continuation function for further processing. That continuation
rlm@213 49 function may perform both CPU and GPU operations on the data. To make
rlm@213 50 this easy for the continuation function, the =SceneProcessor=
rlm@213 51 maintains appropriatly sized buffers in RAM to hold the data. It does
rlm@213 52 not do any copying from the GPU to the CPU itself.
rlm@214 53
rlm@213 54 #+name: pipeline-1
rlm@213 55 #+begin_src clojure
rlm@113 56 (defn vision-pipeline
rlm@34 57 "Create a SceneProcessor object which wraps a vision processing
rlm@113 58 continuation function. The continuation is a function that takes
rlm@113 59 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@113 60 each of which has already been appropiately sized."
rlm@23 61 [continuation]
rlm@23 62 (let [byte-buffer (atom nil)
rlm@113 63 renderer (atom nil)
rlm@113 64 image (atom nil)]
rlm@23 65 (proxy [SceneProcessor] []
rlm@23 66 (initialize
rlm@23 67 [renderManager viewPort]
rlm@23 68 (let [cam (.getCamera viewPort)
rlm@23 69 width (.getWidth cam)
rlm@23 70 height (.getHeight cam)]
rlm@23 71 (reset! renderer (.getRenderer renderManager))
rlm@23 72 (reset! byte-buffer
rlm@23 73 (BufferUtils/createByteBuffer
rlm@113 74 (* width height 4)))
rlm@113 75 (reset! image (BufferedImage.
rlm@113 76 width height
rlm@113 77 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@23 78 (isInitialized [] (not (nil? @byte-buffer)))
rlm@23 79 (reshape [_ _ _])
rlm@23 80 (preFrame [_])
rlm@23 81 (postQueue [_])
rlm@23 82 (postFrame
rlm@23 83 [#^FrameBuffer fb]
rlm@23 84 (.clear @byte-buffer)
rlm@113 85 (continuation @renderer fb @byte-buffer @image))
rlm@23 86 (cleanup []))))
rlm@213 87 #+end_src
rlm@213 88
rlm@213 89 The continuation function given to =(vision-pipeline)= above will be
rlm@213 90 given a =Renderer= and three containers for image data. The
rlm@213 91 =FrameBuffer= references the GPU image data, but it can not be used
rlm@213 92 directly on the CPU. The =ByteBuffer= and =BufferedImage= are
rlm@213 93 initially "empty" but are sized to hold to data in the
rlm@213 94 =FrameBuffer=. I call transfering the GPU image data to the CPU
rlm@213 95 structures "mixing" the image data. I have provided three functions to
rlm@213 96 do this mixing.
rlm@213 97
rlm@213 98 #+name: pipeline-2
rlm@213 99 #+begin_src clojure
rlm@113 100 (defn frameBuffer->byteBuffer!
rlm@113 101 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
rlm@113 102 the CPU (ByteBuffer)."
rlm@113 103 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
rlm@113 104 (.readFrameBuffer r fb bb) bb)
rlm@113 105
rlm@113 106 (defn byteBuffer->bufferedImage!
rlm@113 107 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
rlm@113 108 style ABGR image data and place it in BufferedImage bi."
rlm@113 109 [#^ByteBuffer bb #^BufferedImage bi]
rlm@113 110 (Screenshots/convertScreenShot bb bi) bi)
rlm@113 111
rlm@113 112 (defn BufferedImage!
rlm@113 113 "Continuation which will grab the buffered image from the materials
rlm@113 114 provided by (vision-pipeline)."
rlm@113 115 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
rlm@113 116 (byteBuffer->bufferedImage!
rlm@113 117 (frameBuffer->byteBuffer! r fb bb) bi))
rlm@213 118 #+end_src
rlm@112 119
rlm@213 120 Note that it is possible to write vision processing algorithms
rlm@213 121 entirely in terms of =BufferedImage= inputs. Just compose that
rlm@213 122 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
rlm@213 123 processing algorithm that is entirely hosted on the GPU does not have
rlm@213 124 to pay for this convienence.
rlm@213 125
rlm@214 126 * COMMENT asdasd
rlm@213 127
rlm@213 128 (vision creature) will take an optional :skip argument which will
rlm@213 129 inform the continuations in scene processor to skip the given
rlm@213 130 number of cycles 0 means that no cycles will be skipped.
rlm@213 131
rlm@213 132 (vision creature) will return [init-functions sensor-functions].
rlm@213 133 The init-functions are each single-arg functions that take the
rlm@213 134 world and register the cameras and must each be called before the
rlm@213 135 corresponding sensor-functions. Each init-function returns the
rlm@213 136 viewport for that eye which can be manipulated, saved, etc. Each
rlm@213 137 sensor-function is a thunk and will return data in the same
rlm@213 138 format as the tactile-sensor functions the structure is
rlm@213 139 [topology, sensor-data]. Internally, these sensor-functions
rlm@213 140 maintain a reference to sensor-data which is periodically updated
rlm@213 141 by the continuation function established by its init-function.
rlm@213 142 They can be queried every cycle, but their information may not
rlm@213 143 necessairly be different every cycle.
rlm@213 144
rlm@214 145 * Physical Eyes
rlm@214 146
rlm@214 147 The vision pipeline described above handles the flow of rendered
rlm@214 148 images. Now, we need simulated eyes to serve as the source of these
rlm@214 149 images.
rlm@214 150
rlm@214 151 An eye is described in blender in the same way as a joint. They are
rlm@214 152 zero dimensional empty objects with no geometry whose local coordinate
rlm@214 153 system determines the orientation of the resulting eye. All eyes are
rlm@214 154 childern of a parent node named "eyes" just as all joints have a
rlm@214 155 parent named "joints". An eye binds to the nearest physical object
rlm@214 156 with =(bind-sense=).
rlm@214 157
rlm@214 158 #+name: add-eye
rlm@214 159 #+begin_src clojure
rlm@215 160 (in-ns 'cortex.vision)
rlm@215 161
rlm@214 162 (defn add-eye!
rlm@214 163 "Create a Camera centered on the current position of 'eye which
rlm@214 164 follows the closest physical node in 'creature and sends visual
rlm@215 165 data to 'continuation. The camera will point in the X direction and
rlm@215 166 use the Z vector as up as determined by the rotation of these
rlm@215 167 vectors in blender coordinate space. Use XZY rotation for the node
rlm@215 168 in blender."
rlm@214 169 [#^Node creature #^Spatial eye]
rlm@214 170 (let [target (closest-node creature eye)
rlm@214 171 [cam-width cam-height] (eye-dimensions eye)
rlm@215 172 cam (Camera. cam-width cam-height)
rlm@215 173 rot (.getWorldRotation eye)]
rlm@214 174 (.setLocation cam (.getWorldTranslation eye))
rlm@215 175 (.lookAtDirection cam (.mult rot Vector3f/UNIT_X)
rlm@215 176 ;; this part is consistent with using Z in
rlm@215 177 ;; blender as the UP vector.
rlm@215 178 (.mult rot Vector3f/UNIT_Y))
rlm@214 179 (.setFrustumPerspective
rlm@215 180 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
rlm@215 181 (bind-sense target cam) cam))
rlm@214 182 #+end_src
rlm@214 183
rlm@214 184 Here, the camera is created based on metadata on the eye-node and
rlm@214 185 attached to the nearest physical object with =(bind-sense)=
rlm@214 186
rlm@214 187
rlm@214 188 ** The Retina
rlm@214 189
rlm@214 190 An eye is a surface (the retina) which contains many discrete sensors
rlm@214 191 to detect light. These sensors have can have different-light sensing
rlm@214 192 properties. In humans, each discrete sensor is sensitive to red,
rlm@214 193 blue, green, or gray. These different types of sensors can have
rlm@214 194 different spatial distributions along the retina. In humans, there is
rlm@214 195 a fovea in the center of the retina which has a very high density of
rlm@214 196 color sensors, and a blind spot which has no sensors at all. Sensor
rlm@214 197 density decreases in proportion to distance from the retina.
rlm@214 198
rlm@214 199 I want to be able to model any retinal configuration, so my eye-nodes
rlm@214 200 in blender contain metadata pointing to images that describe the
rlm@214 201 percise position of the individual sensors using white pixels. The
rlm@214 202 meta-data also describes the percise sensitivity to light that the
rlm@214 203 sensors described in the image have. An eye can contain any number of
rlm@214 204 these images. For example, the metadata for an eye might look like
rlm@214 205 this:
rlm@214 206
rlm@214 207 #+begin_src clojure
rlm@214 208 {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@214 209 #+end_src
rlm@214 210
rlm@214 211 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
rlm@214 212 [[../assets/Models/test-creature/retina-small.png]]
rlm@214 213
rlm@214 214 Together, the number 0xFF0000 and the image image above describe the
rlm@214 215 placement of red-sensitive sensory elements.
rlm@214 216
rlm@214 217 Meta-data to very crudely approximate a human eye might be something
rlm@214 218 like this:
rlm@214 219
rlm@214 220 #+begin_src clojure
rlm@214 221 (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@214 222 {0xFF0000 retinal-profile
rlm@214 223 0x00FF00 retinal-profile
rlm@214 224 0x0000FF retinal-profile
rlm@214 225 0xFFFFFF retinal-profile})
rlm@214 226 #+end_src
rlm@214 227
rlm@214 228 The numbers that serve as keys in the map determine a sensor's
rlm@214 229 relative sensitivity to the channels red, green, and blue. These
rlm@214 230 sensitivity values are packed into an integer in the order _RGB in
rlm@214 231 8-bit fields. The RGB values of a pixel in the image are added
rlm@214 232 together with these sensitivities as linear weights. Therfore,
rlm@214 233 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
rlm@214 234 all colors equally (gray).
rlm@214 235
rlm@214 236 For convienence I've defined a few symbols for the more common
rlm@214 237 sensitivity values.
rlm@214 238
rlm@214 239 #+name: sensitivity
rlm@214 240 #+begin_src clojure
rlm@214 241 (defvar sensitivity-presets
rlm@214 242 {:all 0xFFFFFF
rlm@214 243 :red 0xFF0000
rlm@214 244 :blue 0x0000FF
rlm@214 245 :green 0x00FF00}
rlm@214 246 "Retinal sensitivity presets for sensors that extract one channel
rlm@214 247 (:red :blue :green) or average all channels (:gray)")
rlm@214 248 #+end_src
rlm@214 249
rlm@214 250 ** Metadata Processing
rlm@214 251
rlm@214 252 =(retina-sensor-profile)= extracts a map from the eye-node in the same
rlm@214 253 format as the example maps above. =(eye-dimensions)= finds the
rlm@214 254 dimansions of the smallest image required to contain all the retinal
rlm@214 255 sensor maps.
rlm@214 256
rlm@216 257 #+name: retina
rlm@214 258 #+begin_src clojure
rlm@214 259 (defn retina-sensor-profile
rlm@214 260 "Return a map of pixel sensitivity numbers to BufferedImages
rlm@214 261 describing the distribution of light-sensitive components of this
rlm@214 262 eye. :red, :green, :blue, :gray are already defined as extracting
rlm@214 263 the red, green, blue, and average components respectively."
rlm@214 264 [#^Spatial eye]
rlm@214 265 (if-let [eye-map (meta-data eye "eye")]
rlm@214 266 (map-vals
rlm@214 267 load-image
rlm@214 268 (eval (read-string eye-map)))))
rlm@214 269
rlm@214 270 (defn eye-dimensions
rlm@214 271 "Returns [width, height] specified in the metadata of the eye"
rlm@214 272 [#^Spatial eye]
rlm@214 273 (let [dimensions
rlm@214 274 (map #(vector (.getWidth %) (.getHeight %))
rlm@214 275 (vals (retina-sensor-profile eye)))]
rlm@214 276 [(apply max (map first dimensions))
rlm@214 277 (apply max (map second dimensions))]))
rlm@214 278 #+end_src
rlm@214 279
rlm@214 280
rlm@214 281 * Eye Creation
rlm@214 282
rlm@214 283 First off, get the children of the "eyes" empty node to find all the
rlm@214 284 eyes the creature has.
rlm@216 285 #+name: eye-node
rlm@214 286 #+begin_src clojure
rlm@214 287 (defvar
rlm@214 288 ^{:arglists '([creature])}
rlm@214 289 eyes
rlm@214 290 (sense-nodes "eyes")
rlm@214 291 "Return the children of the creature's \"eyes\" node.")
rlm@214 292 #+end_src
rlm@214 293
rlm@215 294 Then, add the camera created by =(add-eye!)= to the simulation by
rlm@215 295 creating a new viewport.
rlm@214 296
rlm@216 297 #+name: add-camera
rlm@213 298 #+begin_src clojure
rlm@169 299 (defn add-camera!
rlm@169 300 "Add a camera to the world, calling continuation on every frame
rlm@34 301 produced."
rlm@167 302 [#^Application world camera continuation]
rlm@23 303 (let [width (.getWidth camera)
rlm@23 304 height (.getHeight camera)
rlm@23 305 render-manager (.getRenderManager world)
rlm@23 306 viewport (.createMainView render-manager "eye-view" camera)]
rlm@23 307 (doto viewport
rlm@23 308 (.setClearFlags true true true)
rlm@112 309 (.setBackgroundColor ColorRGBA/Black)
rlm@113 310 (.addProcessor (vision-pipeline continuation))
rlm@23 311 (.attachScene (.getRootNode world)))))
rlm@215 312 #+end_src
rlm@151 313
rlm@151 314
rlm@215 315 The continuation function registers the viewport with the simulation
rlm@215 316 the first time it is called, and uses the CPU to extract the
rlm@215 317 appropriate pixels from the rendered image and weight them by each
rlm@215 318 sensors sensitivity. I have the option to do this filtering in native
rlm@215 319 code for a slight gain in speed. I could also do it in the GPU for a
rlm@215 320 massive gain in speed. =(vision-kernel)= generates a list of such
rlm@215 321 continuation functions, one for each channel of the eye.
rlm@151 322
rlm@216 323 #+name: kernel
rlm@215 324 #+begin_src clojure
rlm@215 325 (in-ns 'cortex.vision)
rlm@151 326
rlm@215 327 (defrecord attached-viewport [vision-fn viewport-fn]
rlm@215 328 clojure.lang.IFn
rlm@215 329 (invoke [this world] (vision-fn world))
rlm@215 330 (applyTo [this args] (apply vision-fn args)))
rlm@151 331
rlm@216 332 (defn pixel-sense [sensitivity pixel]
rlm@216 333 (let [s-r (bit-shift-right (bit-and 0xFF0000 sensitivity) 16)
rlm@216 334 s-g (bit-shift-right (bit-and 0x00FF00 sensitivity) 8)
rlm@216 335 s-b (bit-and 0x0000FF sensitivity)
rlm@216 336
rlm@216 337 p-r (bit-shift-right (bit-and 0xFF0000 pixel) 16)
rlm@216 338 p-g (bit-shift-right (bit-and 0x00FF00 pixel) 8)
rlm@216 339 p-b (bit-and 0x0000FF pixel)
rlm@216 340
rlm@216 341 total-sensitivity (* 255 (+ s-r s-g s-b))]
rlm@216 342 (float (/ (+ (* s-r p-r)
rlm@216 343 (* s-g p-g)
rlm@216 344 (* s-b p-b))
rlm@216 345 total-sensitivity))))
rlm@216 346
rlm@215 347 (defn vision-kernel
rlm@171 348 "Returns a list of functions, each of which will return a color
rlm@171 349 channel's worth of visual information when called inside a running
rlm@171 350 simulation."
rlm@151 351 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@169 352 (let [retinal-map (retina-sensor-profile eye)
rlm@169 353 camera (add-eye! creature eye)
rlm@151 354 vision-image
rlm@151 355 (atom
rlm@151 356 (BufferedImage. (.getWidth camera)
rlm@151 357 (.getHeight camera)
rlm@170 358 BufferedImage/TYPE_BYTE_BINARY))
rlm@170 359 register-eye!
rlm@170 360 (runonce
rlm@170 361 (fn [world]
rlm@170 362 (add-camera!
rlm@170 363 world camera
rlm@170 364 (let [counter (atom 0)]
rlm@170 365 (fn [r fb bb bi]
rlm@170 366 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@170 367 (reset! vision-image
rlm@170 368 (BufferedImage! r fb bb bi))))))))]
rlm@151 369 (vec
rlm@151 370 (map
rlm@151 371 (fn [[key image]]
rlm@151 372 (let [whites (white-coordinates image)
rlm@151 373 topology (vec (collapse whites))
rlm@216 374 sensitivity (sensitivity-presets key key)]
rlm@215 375 (attached-viewport.
rlm@215 376 (fn [world]
rlm@215 377 (register-eye! world)
rlm@215 378 (vector
rlm@215 379 topology
rlm@215 380 (vec
rlm@215 381 (for [[x y] whites]
rlm@216 382 (pixel-sense
rlm@216 383 sensitivity
rlm@216 384 (.getRGB @vision-image x y))))))
rlm@215 385 register-eye!)))
rlm@215 386 retinal-map))))
rlm@151 387
rlm@215 388 (defn gen-fix-display
rlm@215 389 "Create a function to call to restore a simulation's display when it
rlm@215 390 is disrupted by a Viewport."
rlm@215 391 []
rlm@215 392 (runonce
rlm@215 393 (fn [world]
rlm@215 394 (add-camera! world (.getCamera world) no-op))))
rlm@215 395 #+end_src
rlm@170 396
rlm@215 397 Note that since each of the functions generated by =(vision-kernel)=
rlm@215 398 shares the same =(register-eye!)= function, the eye will be registered
rlm@215 399 only once the first time any of the functions from the list returned
rlm@215 400 by =(vision-kernel)= is called. Each of the functions returned by
rlm@215 401 =(vision-kernel)= also allows access to the =Viewport= through which
rlm@215 402 it recieves images.
rlm@215 403
rlm@215 404 The in-game display can be disrupted by all the viewports that the
rlm@215 405 functions greated by =(vision-kernel)= add. This doesn't affect the
rlm@215 406 simulation or the simulated senses, but can be annoying.
rlm@215 407 =(gen-fix-display)= restores the in-simulation display.
rlm@215 408
rlm@215 409 ** Vision!
rlm@215 410
rlm@215 411 All the hard work has been done, all that remains is to apply
rlm@215 412 =(vision-kernel)= to each eye in the creature and gather the results
rlm@215 413 into one list of functions.
rlm@215 414
rlm@216 415 #+name: main
rlm@215 416 #+begin_src clojure
rlm@170 417 (defn vision!
rlm@170 418 "Returns a function which returns visual sensory data when called
rlm@170 419 inside a running simulation"
rlm@151 420 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@151 421 (reduce
rlm@170 422 concat
rlm@167 423 (for [eye (eyes creature)]
rlm@215 424 (vision-kernel creature eye))))
rlm@215 425 #+end_src
rlm@151 426
rlm@215 427 ** Visualization of Vision
rlm@215 428
rlm@215 429 It's vital to have a visual representation for each sense. Here I use
rlm@215 430 =(view-sense)= to construct a function that will create a display for
rlm@215 431 visual data.
rlm@215 432
rlm@216 433 #+name: display
rlm@215 434 #+begin_src clojure
rlm@216 435 (in-ns 'cortex.vision)
rlm@216 436
rlm@189 437 (defn view-vision
rlm@189 438 "Creates a function which accepts a list of visual sensor-data and
rlm@189 439 displays each element of the list to the screen."
rlm@189 440 []
rlm@188 441 (view-sense
rlm@188 442 (fn
rlm@188 443 [[coords sensor-data]]
rlm@188 444 (let [image (points->image coords)]
rlm@188 445 (dorun
rlm@188 446 (for [i (range (count coords))]
rlm@188 447 (.setRGB image ((coords i) 0) ((coords i) 1)
rlm@216 448 (gray (int (* 255 (sensor-data i)))))))
rlm@189 449 image))))
rlm@34 450 #+end_src
rlm@23 451
rlm@215 452 * Tests
rlm@112 453
rlm@215 454 ** Basic Test
rlm@23 455
rlm@215 456 This is a basic test for the vision system. It only tests the
rlm@215 457 vision-pipeline and does not deal with loadig eyes from a blender
rlm@215 458 file. The code creates two videos of the same rotating cube from
rlm@215 459 different angles.
rlm@23 460
rlm@215 461 #+name: test-1
rlm@23 462 #+begin_src clojure
rlm@215 463 (in-ns 'cortex.test.vision)
rlm@23 464
rlm@36 465 (defn test-two-eyes
rlm@69 466 "Testing vision:
rlm@69 467 Tests the vision system by creating two views of the same rotating
rlm@69 468 object from different angles and displaying both of those views in
rlm@69 469 JFrames.
rlm@69 470
rlm@69 471 You should see a rotating cube, and two windows,
rlm@69 472 each displaying a different view of the cube."
rlm@36 473 []
rlm@58 474 (let [candy
rlm@58 475 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@112 476 (world
rlm@112 477 (doto (Node.)
rlm@112 478 (.attachChild candy))
rlm@112 479 {}
rlm@112 480 (fn [world]
rlm@112 481 (let [cam (.clone (.getCamera world))
rlm@112 482 width (.getWidth cam)
rlm@112 483 height (.getHeight cam)]
rlm@169 484 (add-camera! world cam
rlm@215 485 (comp
rlm@215 486 (view-image
rlm@215 487 (File. "/home/r/proj/cortex/render/vision/1"))
rlm@215 488 BufferedImage!))
rlm@169 489 (add-camera! world
rlm@112 490 (doto (.clone cam)
rlm@112 491 (.setLocation (Vector3f. -10 0 0))
rlm@112 492 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@215 493 (comp
rlm@215 494 (view-image
rlm@215 495 (File. "/home/r/proj/cortex/render/vision/2"))
rlm@215 496 BufferedImage!))
rlm@112 497 ;; This is here to restore the main view
rlm@112 498 ;; after the other views have completed processing
rlm@169 499 (add-camera! world (.getCamera world) no-op)))
rlm@112 500 (fn [world tpf]
rlm@112 501 (.rotate candy (* tpf 0.2) 0 0)))))
rlm@23 502 #+end_src
rlm@23 503
rlm@215 504 #+begin_html
rlm@215 505 <div class="figure">
rlm@215 506 <video controls="controls" width="755">
rlm@215 507 <source src="../video/spinning-cube.ogg" type="video/ogg"
rlm@215 508 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@215 509 </video>
rlm@215 510 <p>A rotating cube viewed from two different perspectives.</p>
rlm@215 511 </div>
rlm@215 512 #+end_html
rlm@215 513
rlm@215 514 Creating multiple eyes like this can be used for stereoscopic vision
rlm@215 515 simulation in a single creature or for simulating multiple creatures,
rlm@215 516 each with their own sense of vision.
rlm@215 517
rlm@215 518 ** Adding Vision to the Worm
rlm@215 519
rlm@215 520 To the worm from the last post, we add a new node that describes its
rlm@215 521 eyes.
rlm@215 522
rlm@215 523 #+attr_html: width=755
rlm@215 524 #+caption: The worm with newly added empty nodes describing a single eye.
rlm@215 525 [[../images/worm-with-eye.png]]
rlm@215 526
rlm@215 527 The node highlighted in yellow is the root level "eyes" node. It has
rlm@215 528 a single node, highlighted in orange, which describes a single
rlm@215 529 eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint
rlm@215 530 of the worm.
rlm@215 531
rlm@215 532 The metadata of the eye-node is:
rlm@215 533
rlm@215 534 #+begin_src clojure :results verbatim :exports both
rlm@215 535 (cortex.sense/meta-data
rlm@215 536 (.getChild
rlm@215 537 (.getChild (cortex.test.body/worm)
rlm@215 538 "eyes") "eye") "eye")
rlm@215 539 #+end_src
rlm@215 540
rlm@215 541 #+results:
rlm@215 542 : "(let [retina \"Models/test-creature/retina-small.png\"]
rlm@215 543 : {:all retina :red retina :green retina :blue retina})"
rlm@215 544
rlm@215 545 This is the approximation to the human eye described earlier.
rlm@215 546
rlm@216 547 #+name: test-2
rlm@215 548 #+begin_src clojure
rlm@215 549 (in-ns 'cortex.test.vision)
rlm@215 550
rlm@216 551 (defn change-color [obj color]
rlm@216 552 (println-repl obj)
rlm@216 553 (if obj
rlm@216 554 (.setColor (.getMaterial obj) "Color" color)))
rlm@216 555
rlm@216 556 (defn colored-cannon-ball [color]
rlm@216 557 (comp #(change-color % color)
rlm@216 558 (fire-cannon-ball)))
rlm@215 559
rlm@215 560 (defn test-worm-vision []
rlm@215 561 (let [the-worm (doto (worm)(body!))
rlm@215 562 vision (vision! the-worm)
rlm@215 563 vision-display (view-vision)
rlm@215 564 fix-display (gen-fix-display)
rlm@215 565 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
rlm@215 566 x-axis
rlm@215 567 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
rlm@215 568 :position (Vector3f. 0 -5 0))
rlm@215 569 y-axis
rlm@215 570 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
rlm@215 571 :position (Vector3f. 0 -5 0))
rlm@215 572 z-axis
rlm@215 573 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
rlm@216 574 :position (Vector3f. 0 -5 0))
rlm@216 575 timer (RatchetTimer. 60)]
rlm@215 576
rlm@215 577 (world (nodify [(floor) the-worm x-axis y-axis z-axis me])
rlm@216 578 (assoc standard-debug-controls
rlm@216 579 "key-r" (colored-cannon-ball ColorRGBA/Red)
rlm@216 580 "key-b" (colored-cannon-ball ColorRGBA/Blue)
rlm@216 581 "key-g" (colored-cannon-ball ColorRGBA/Green))
rlm@215 582 (fn [world]
rlm@215 583 (light-up-everything world)
rlm@216 584 (speed-up world)
rlm@216 585 (.setTimer world timer)
rlm@216 586 (display-dialated-time world timer)
rlm@215 587 ;; add a view from the worm's perspective
rlm@215 588 (add-camera!
rlm@215 589 world
rlm@215 590 (add-eye! the-worm
rlm@215 591 (.getChild
rlm@215 592 (.getChild the-worm "eyes") "eye"))
rlm@215 593 (comp
rlm@215 594 (view-image
rlm@215 595 (File. "/home/r/proj/cortex/render/worm-vision/worm-view"))
rlm@215 596 BufferedImage!))
rlm@215 597 (set-gravity world Vector3f/ZERO)
rlm@216 598 (try
rlm@216 599 (Capture/captureVideo
rlm@216 600 world
rlm@216 601 (File. "/home/r/proj/cortex/render/worm-vision/main-view"))))
rlm@216 602
rlm@215 603 (fn [world _ ]
rlm@215 604 (.setLocalTranslation me (.getLocation (.getCamera world)))
rlm@215 605 (vision-display
rlm@215 606 (map #(% world) vision)
rlm@215 607 (File. "/home/r/proj/cortex/render/worm-vision"))
rlm@215 608 (fix-display world)))))
rlm@215 609 #+end_src
rlm@215 610
rlm@216 611 ** Methods to Generate the Worm Video
rlm@216 612 #+name: magick2
rlm@216 613 #+begin_src clojure
rlm@216 614 (ns cortex.video.magick2
rlm@216 615 (:import java.io.File)
rlm@216 616 (:use clojure.contrib.shell-out))
rlm@216 617
rlm@216 618 (defn images [path]
rlm@216 619 (sort (rest (file-seq (File. path)))))
rlm@216 620
rlm@216 621 (def base "/home/r/proj/cortex/render/worm-vision/")
rlm@216 622
rlm@216 623 (defn pics [file]
rlm@216 624 (images (str base file)))
rlm@216 625
rlm@216 626 (defn combine-images []
rlm@216 627 (let [main-view (pics "main-view")
rlm@216 628 worm-view (pics "worm-view")
rlm@216 629 blue (pics "0")
rlm@216 630 green (pics "1")
rlm@216 631 red (pics "2")
rlm@216 632 gray (pics "3")
rlm@216 633 blender (let [b-pics (pics "blender")]
rlm@216 634 (concat b-pics (repeat 9001 (last b-pics))))
rlm@216 635 background (repeat 9001 (File. (str base "background.png")))
rlm@216 636 targets (map
rlm@216 637 #(File. (str base "out/" (format "%07d.png" %)))
rlm@216 638 (range 0 (count main-view)))]
rlm@216 639 (dorun
rlm@216 640 (pmap
rlm@216 641 (comp
rlm@216 642 (fn [[background main-view worm-view red green blue gray blender target]]
rlm@216 643 (println target)
rlm@216 644 (sh "convert"
rlm@216 645 background
rlm@216 646 main-view "-geometry" "+18+17" "-composite"
rlm@216 647 worm-view "-geometry" "+677+17" "-composite"
rlm@216 648 green "-geometry" "+685+430" "-composite"
rlm@216 649 red "-geometry" "+788+430" "-composite"
rlm@216 650 blue "-geometry" "+894+430" "-composite"
rlm@216 651 gray "-geometry" "+1000+430" "-composite"
rlm@216 652 blender "-geometry" "+0+0" "-composite"
rlm@216 653 target))
rlm@216 654 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@216 655 background main-view worm-view red green blue gray blender targets))))
rlm@216 656 #+end_src
rlm@216 657
rlm@216 658 #+begin_src sh :results silent
rlm@216 659 cd /home/r/proj/cortex/render/worm-vision
rlm@216 660 ffmpeg -r 25 -b 9001k -i out/%07d.png -vcodec libtheora worm-vision.ogg
rlm@216 661 #+end_src
rlm@216 662
rlm@216 663 * Demonstration of Vision
rlm@216 664 #+begin_html
rlm@216 665 <div class="figure">
rlm@216 666 <video controls="controls" width="755">
rlm@216 667 <source src="../video/worm-vision.ogg" type="video/ogg"
rlm@216 668 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@216 669 </video>
rlm@216 670 <p>Simulated Vision in a Virtual Environment</p>
rlm@216 671 </div>
rlm@216 672 #+end_html
rlm@216 673
rlm@215 674 * Headers
rlm@215 675
rlm@213 676 #+name: vision-header
rlm@213 677 #+begin_src clojure
rlm@213 678 (ns cortex.vision
rlm@213 679 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
rlm@213 680 eyes from different positions to observe the same world, and pass
rlm@213 681 the observed data to any arbitray function. Automatically reads
rlm@216 682 eye-nodes from specially prepared blender files and instantiates
rlm@213 683 them in the world as actual eyes."
rlm@213 684 {:author "Robert McIntyre"}
rlm@213 685 (:use (cortex world sense util))
rlm@213 686 (:use clojure.contrib.def)
rlm@213 687 (:import com.jme3.post.SceneProcessor)
rlm@213 688 (:import (com.jme3.util BufferUtils Screenshots))
rlm@213 689 (:import java.nio.ByteBuffer)
rlm@213 690 (:import java.awt.image.BufferedImage)
rlm@213 691 (:import (com.jme3.renderer ViewPort Camera))
rlm@216 692 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f))
rlm@213 693 (:import com.jme3.renderer.Renderer)
rlm@213 694 (:import com.jme3.app.Application)
rlm@213 695 (:import com.jme3.texture.FrameBuffer)
rlm@213 696 (:import (com.jme3.scene Node Spatial)))
rlm@213 697 #+end_src
rlm@112 698
rlm@215 699 #+name: test-header
rlm@215 700 #+begin_src clojure
rlm@215 701 (ns cortex.test.vision
rlm@215 702 (:use (cortex world sense util body vision))
rlm@215 703 (:use cortex.test.body)
rlm@215 704 (:import java.awt.image.BufferedImage)
rlm@215 705 (:import javax.swing.JPanel)
rlm@215 706 (:import javax.swing.SwingUtilities)
rlm@215 707 (:import java.awt.Dimension)
rlm@215 708 (:import javax.swing.JFrame)
rlm@215 709 (:import com.jme3.math.ColorRGBA)
rlm@215 710 (:import com.jme3.scene.Node)
rlm@215 711 (:import com.jme3.math.Vector3f)
rlm@216 712 (:import java.io.File)
rlm@216 713 (:import (com.aurellem.capture Capture RatchetTimer)))
rlm@215 714 #+end_src
rlm@215 715
rlm@216 716 * Onward!
rlm@216 717 - As a neat bonus, this idea behind simulated vision also enables one
rlm@216 718 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
rlm@216 719 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
rlm@215 720
rlm@216 721 * Source Listing
rlm@216 722 - [[../src/cortex/vision.clj][cortex.vision]]
rlm@216 723 - [[../src/cortex/test/vision.clj][cortex.test.vision]]
rlm@216 724 - [[../src/cortex/video/magick2.clj][cortex.video.magick2]]
rlm@216 725 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
rlm@216 726 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
rlm@216 727 - [[http://hg.bortreb.com ][source-repository]]
rlm@216 728
rlm@35 729
rlm@24 730
rlm@212 731 * COMMENT Generate Source
rlm@34 732 #+begin_src clojure :tangle ../src/cortex/vision.clj
rlm@216 733 <<vision-header>>
rlm@216 734 <<pipeline-1>>
rlm@216 735 <<pipeline-2>>
rlm@216 736 <<retina>>
rlm@216 737 <<add-eye>>
rlm@216 738 <<sensitivity>>
rlm@216 739 <<eye-node>>
rlm@216 740 <<add-camera>>
rlm@216 741 <<kernel>>
rlm@216 742 <<main>>
rlm@216 743 <<display>>
rlm@24 744 #+end_src
rlm@24 745
rlm@68 746 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
rlm@215 747 <<test-header>>
rlm@215 748 <<test-1>>
rlm@216 749 <<test-2>>
rlm@24 750 #+end_src
rlm@216 751
rlm@216 752 #+begin_src clojure :tangle ../src/cortex/video/magick2.clj
rlm@216 753 <<magick2>>
rlm@216 754 #+end_src