Mercurial > cortex

annotate org/vision.org @ 168:1c8e9d389ea4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
renamed eyes.org to vision.org
author Robert McIntyre <rlm@mit.edu>
date Sat, 04 Feb 2012 04:08:08 -0700
parents org/eyes.org@9e6a30b8c99a
children 94b79c191fc7
rev   line source
rlm@34 1 #+title: Simulated Sense of Sight
rlm@23 2 #+author: Robert McIntyre
rlm@23 3 #+email: rlm@mit.edu
rlm@38 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
rlm@34 5 #+keywords: computer vision, jMonkeyEngine3, clojure
rlm@23 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@23 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@23 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@23 9
rlm@34 10 * Vision
rlm@23 11
rlm@34 12 I want to make creatures with eyes. Each eye can be independely moved
rlm@34 13 and should see its own version of the world depending on where it is.
rlm@151 14
rlm@151 15 Here's how vision will work.
rlm@151 16
rlm@151 17 Make the continuation in scene-processor take FrameBuffer,
rlm@151 18 byte-buffer, BufferedImage already sized to the correct
rlm@151 19 dimensions. the continuation will decide wether to "mix" them
rlm@151 20 into the BufferedImage, lazily ignore them, or mix them halfway
rlm@151 21 and call c/graphics card routines.
rlm@151 22
rlm@151 23 (vision creature) will take an optional :skip argument which will
rlm@151 24 inform the continuations in scene processor to skip the given
rlm@151 25 number of cycles 0 means that no cycles will be skipped.
rlm@151 26
rlm@151 27 (vision creature) will return [init-functions sensor-functions].
rlm@151 28 The init-functions are each single-arg functions that take the
rlm@151 29 world and register the cameras and must each be called before the
rlm@151 30 corresponding sensor-functions. Each init-function returns the
rlm@151 31 viewport for that eye which can be manipulated, saved, etc. Each
rlm@151 32 sensor-function is a thunk and will return data in the same
rlm@151 33 format as the tactile-sensor functions the structure is
rlm@151 34 [topology, sensor-data]. Internally, these sensor-functions
rlm@151 35 maintain a reference to sensor-data which is periodically updated
rlm@151 36 by the continuation function established by its init-function.
rlm@151 37 They can be queried every cycle, but their information may not
rlm@151 38 necessairly be different every cycle.
rlm@151 39
rlm@151 40 Each eye in the creature in blender will work the same way as
rlm@151 41 joints -- a zero dimensional object with no geometry whose local
rlm@151 42 coordinate system determines the orientation of the resulting
rlm@151 43 eye. All eyes will have a parent named "eyes" just as all joints
rlm@151 44 have a parent named "joints". The resulting camera will be a
rlm@151 45 ChaseCamera or a CameraNode bound to the geo that is closest to
rlm@151 46 the eye marker. The eye marker will contain the metadata for the
rlm@151 47 eye, and will be moved by it's bound geometry. The dimensions of
rlm@151 48 the eye's camera are equal to the dimensions of the eye's "UV"
rlm@151 49 map.
rlm@151 50
rlm@66 51 #+name: eyes
rlm@23 52 #+begin_src clojure
rlm@34 53 (ns cortex.vision
rlm@34 54 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
rlm@34 55 eyes from different positions to observe the same world, and pass
rlm@34 56 the observed data to any arbitray function."
rlm@34 57 {:author "Robert McIntyre"}
rlm@151 58 (:use (cortex world sense util))
rlm@167 59 (:use clojure.contrib.def)
rlm@34 60 (:import com.jme3.post.SceneProcessor)
rlm@113 61 (:import (com.jme3.util BufferUtils Screenshots))
rlm@34 62 (:import java.nio.ByteBuffer)
rlm@34 63 (:import java.awt.image.BufferedImage)
rlm@34 64 (:import com.jme3.renderer.ViewPort)
rlm@113 65 (:import com.jme3.math.ColorRGBA)
rlm@151 66 (:import com.jme3.renderer.Renderer)
rlm@151 67 (:import com.jme3.scene.Node))
rlm@151 68
rlm@151 69 (cortex.import/mega-import-jme3)
rlm@23 70
rlm@113 71
rlm@113 72 (defn vision-pipeline
rlm@34 73 "Create a SceneProcessor object which wraps a vision processing
rlm@113 74 continuation function. The continuation is a function that takes
rlm@113 75 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@113 76 each of which has already been appropiately sized."
rlm@23 77 [continuation]
rlm@23 78 (let [byte-buffer (atom nil)
rlm@113 79 renderer (atom nil)
rlm@113 80 image (atom nil)]
rlm@23 81 (proxy [SceneProcessor] []
rlm@23 82 (initialize
rlm@23 83 [renderManager viewPort]
rlm@23 84 (let [cam (.getCamera viewPort)
rlm@23 85 width (.getWidth cam)
rlm@23 86 height (.getHeight cam)]
rlm@23 87 (reset! renderer (.getRenderer renderManager))
rlm@23 88 (reset! byte-buffer
rlm@23 89 (BufferUtils/createByteBuffer
rlm@113 90 (* width height 4)))
rlm@113 91 (reset! image (BufferedImage.
rlm@113 92 width height
rlm@113 93 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@23 94 (isInitialized [] (not (nil? @byte-buffer)))
rlm@23 95 (reshape [_ _ _])
rlm@23 96 (preFrame [_])
rlm@23 97 (postQueue [_])
rlm@23 98 (postFrame
rlm@23 99 [#^FrameBuffer fb]
rlm@23 100 (.clear @byte-buffer)
rlm@113 101 (continuation @renderer fb @byte-buffer @image))
rlm@23 102 (cleanup []))))
rlm@23 103
rlm@113 104 (defn frameBuffer->byteBuffer!
rlm@113 105 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
rlm@113 106 the CPU (ByteBuffer)."
rlm@113 107 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
rlm@113 108 (.readFrameBuffer r fb bb) bb)
rlm@113 109
rlm@113 110 (defn byteBuffer->bufferedImage!
rlm@113 111 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
rlm@113 112 style ABGR image data and place it in BufferedImage bi."
rlm@113 113 [#^ByteBuffer bb #^BufferedImage bi]
rlm@113 114 (Screenshots/convertScreenShot bb bi) bi)
rlm@113 115
rlm@113 116 (defn BufferedImage!
rlm@113 117 "Continuation which will grab the buffered image from the materials
rlm@113 118 provided by (vision-pipeline)."
rlm@113 119 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
rlm@113 120 (byteBuffer->bufferedImage!
rlm@113 121 (frameBuffer->byteBuffer! r fb bb) bi))
rlm@112 122
rlm@167 123 (defn add-eye!
rlm@34 124 "Add an eye to the world, calling continuation on every frame
rlm@34 125 produced."
rlm@167 126 [#^Application world camera continuation]
rlm@23 127 (let [width (.getWidth camera)
rlm@23 128 height (.getHeight camera)
rlm@23 129 render-manager (.getRenderManager world)
rlm@23 130 viewport (.createMainView render-manager "eye-view" camera)]
rlm@23 131 (doto viewport
rlm@23 132 (.setClearFlags true true true)
rlm@112 133 (.setBackgroundColor ColorRGBA/Black)
rlm@113 134 (.addProcessor (vision-pipeline continuation))
rlm@23 135 (.attachScene (.getRootNode world)))))
rlm@151 136
rlm@151 137 (defn retina-sensor-image
rlm@151 138 "Return a map of pixel selection functions to BufferedImages
rlm@151 139 describing the distribution of light-sensitive components on this
rlm@151 140 geometry's surface. Each function creates an integer from the rgb
rlm@151 141 values found in the pixel. :red, :green, :blue, :gray are already
rlm@151 142 defined as extracting the red green blue and average components
rlm@151 143 respectively."
rlm@151 144 [#^Spatial eye]
rlm@151 145 (if-let [eye-map (meta-data eye "eye")]
rlm@151 146 (map-vals
rlm@167 147 load-image
rlm@151 148 (eval (read-string eye-map)))))
rlm@151 149
rlm@151 150 (defn eye-dimensions
rlm@151 151 "returns the width and height specified in the metadata of the eye"
rlm@151 152 [#^Spatial eye]
rlm@151 153 (let [dimensions
rlm@151 154 (map #(vector (.getWidth %) (.getHeight %))
rlm@151 155 (vals (retina-sensor-image eye)))]
rlm@151 156 [(apply max (map first dimensions))
rlm@151 157 (apply max (map second dimensions))]))
rlm@151 158
rlm@167 159 (defvar
rlm@167 160 ^{:arglists '([creature])}
rlm@167 161 eyes
rlm@167 162 (sense-nodes "eyes")
rlm@167 163 "Return the children of the creature's \"eyes\" node.")
rlm@151 164
rlm@151 165 (defn attach-eye
rlm@151 166 "Attach a Camera to the appropiate area and return the Camera."
rlm@151 167 [#^Node creature #^Spatial eye]
rlm@151 168 (let [target (closest-node creature eye)
rlm@151 169 [cam-width cam-height] (eye-dimensions eye)
rlm@151 170 cam (Camera. cam-width cam-height)]
rlm@151 171 (.setLocation cam (.getWorldTranslation eye))
rlm@151 172 (.setRotation cam (.getWorldRotation eye))
rlm@151 173 (.setFrustumPerspective
rlm@151 174 cam 45 (/ (.getWidth cam) (.getHeight cam))
rlm@151 175 1 1000)
rlm@151 176 (bind-sense target cam)
rlm@151 177 cam))
rlm@151 178
rlm@151 179 (def presets
rlm@151 180 {:all 0xFFFFFF
rlm@151 181 :red 0xFF0000
rlm@151 182 :blue 0x0000FF
rlm@151 183 :green 0x00FF00})
rlm@151 184
rlm@151 185 (defn enable-vision
rlm@151 186 "return [init-function sensor-functions] for a particular eye"
rlm@151 187 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@151 188 (let [retinal-map (retina-sensor-image eye)
rlm@151 189 camera (attach-eye creature eye)
rlm@151 190 vision-image
rlm@151 191 (atom
rlm@151 192 (BufferedImage. (.getWidth camera)
rlm@151 193 (.getHeight camera)
rlm@151 194 BufferedImage/TYPE_BYTE_BINARY))]
rlm@151 195 [(fn [world]
rlm@167 196 (add-eye!
rlm@151 197 world camera
rlm@151 198 (let [counter (atom 0)]
rlm@151 199 (fn [r fb bb bi]
rlm@151 200 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@151 201 (reset! vision-image (BufferedImage! r fb bb bi)))))))
rlm@151 202 (vec
rlm@151 203 (map
rlm@151 204 (fn [[key image]]
rlm@151 205 (let [whites (white-coordinates image)
rlm@151 206 topology (vec (collapse whites))
rlm@151 207 mask (presets key)]
rlm@151 208 (fn []
rlm@151 209 (vector
rlm@151 210 topology
rlm@151 211 (vec
rlm@151 212 (for [[x y] whites]
rlm@151 213 (bit-and
rlm@151 214 mask (.getRGB @vision-image x y))))))))
rlm@151 215 retinal-map))]))
rlm@151 216
rlm@151 217 (defn vision
rlm@151 218 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@151 219 (reduce
rlm@151 220 (fn [[init-a senses-a]
rlm@151 221 [init-b senses-b]]
rlm@151 222 [(conj init-a init-b)
rlm@151 223 (into senses-a senses-b)])
rlm@151 224 [[][]]
rlm@167 225 (for [eye (eyes creature)]
rlm@151 226 (enable-vision creature eye))))
rlm@151 227
rlm@151 228
rlm@34 229 #+end_src
rlm@23 230
rlm@112 231
rlm@34 232 Note the use of continuation passing style for connecting the eye to a
rlm@34 233 function to process the output. You can create any number of eyes, and
rlm@34 234 each of them will see the world from their own =Camera=. Once every
rlm@34 235 frame, the rendered image is copied to a =BufferedImage=, and that
rlm@34 236 data is sent off to the continuation function. Moving the =Camera=
rlm@34 237 which was used to create the eye will change what the eye sees.
rlm@23 238
rlm@34 239 * Example
rlm@23 240
rlm@66 241 #+name: test-vision
rlm@23 242 #+begin_src clojure
rlm@68 243 (ns cortex.test.vision
rlm@34 244 (:use (cortex world util vision))
rlm@34 245 (:import java.awt.image.BufferedImage)
rlm@34 246 (:import javax.swing.JPanel)
rlm@34 247 (:import javax.swing.SwingUtilities)
rlm@34 248 (:import java.awt.Dimension)
rlm@34 249 (:import javax.swing.JFrame)
rlm@34 250 (:import com.jme3.math.ColorRGBA)
rlm@45 251 (:import com.jme3.scene.Node)
rlm@113 252 (:import com.jme3.math.Vector3f))
rlm@23 253
rlm@36 254 (defn test-two-eyes
rlm@69 255 "Testing vision:
rlm@69 256 Tests the vision system by creating two views of the same rotating
rlm@69 257 object from different angles and displaying both of those views in
rlm@69 258 JFrames.
rlm@69 259
rlm@69 260 You should see a rotating cube, and two windows,
rlm@69 261 each displaying a different view of the cube."
rlm@36 262 []
rlm@58 263 (let [candy
rlm@58 264 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@112 265 (world
rlm@112 266 (doto (Node.)
rlm@112 267 (.attachChild candy))
rlm@112 268 {}
rlm@112 269 (fn [world]
rlm@112 270 (let [cam (.clone (.getCamera world))
rlm@112 271 width (.getWidth cam)
rlm@112 272 height (.getHeight cam)]
rlm@167 273 (add-eye! world cam
rlm@113 274 ;;no-op
rlm@113 275 (comp (view-image) BufferedImage!)
rlm@112 276 )
rlm@167 277 (add-eye! world
rlm@112 278 (doto (.clone cam)
rlm@112 279 (.setLocation (Vector3f. -10 0 0))
rlm@112 280 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@113 281 ;;no-op
rlm@113 282 (comp (view-image) BufferedImage!))
rlm@112 283 ;; This is here to restore the main view
rlm@112 284 ;; after the other views have completed processing
rlm@167 285 (add-eye! world (.getCamera world) no-op)))
rlm@112 286 (fn [world tpf]
rlm@112 287 (.rotate candy (* tpf 0.2) 0 0)))))
rlm@23 288 #+end_src
rlm@23 289
rlm@112 290 #+results: test-vision
rlm@112 291 : #'cortex.test.vision/test-two-eyes
rlm@112 292
rlm@34 293 The example code will create two videos of the same rotating object
rlm@34 294 from different angles. It can be used both for stereoscopic vision
rlm@34 295 simulation or for simulating multiple creatures, each with their own
rlm@34 296 sense of vision.
rlm@24 297
rlm@35 298 - As a neat bonus, this idea behind simulated vision also enables one
rlm@35 299 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
rlm@35 300
rlm@24 301
rlm@24 302 * COMMENT code generation
rlm@34 303 #+begin_src clojure :tangle ../src/cortex/vision.clj
rlm@24 304 <<eyes>>
rlm@24 305 #+end_src
rlm@24 306
rlm@68 307 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
rlm@24 308 <<test-vision>>
rlm@24 309 #+end_src