Mercurial > cortex

annotate org/eyes.org @ 153:c95179907951

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
saving progress
author Robert McIntyre <rlm@mit.edu>
date Fri, 03 Feb 2012 05:59:32 -0700
parents aaacf087504c
children 9e6a30b8c99a
rev   line source
rlm@34 1 #+title: Simulated Sense of Sight
rlm@23 2 #+author: Robert McIntyre
rlm@23 3 #+email: rlm@mit.edu
rlm@38 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
rlm@34 5 #+keywords: computer vision, jMonkeyEngine3, clojure
rlm@23 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@23 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@23 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@23 9
rlm@34 10 * Vision
rlm@23 11
rlm@34 12 I want to make creatures with eyes. Each eye can be independely moved
rlm@34 13 and should see its own version of the world depending on where it is.
rlm@151 14
rlm@151 15 Here's how vision will work.
rlm@151 16
rlm@151 17 Make the continuation in scene-processor take FrameBuffer,
rlm@151 18 byte-buffer, BufferedImage already sized to the correct
rlm@151 19 dimensions. the continuation will decide wether to "mix" them
rlm@151 20 into the BufferedImage, lazily ignore them, or mix them halfway
rlm@151 21 and call c/graphics card routines.
rlm@151 22
rlm@151 23 (vision creature) will take an optional :skip argument which will
rlm@151 24 inform the continuations in scene processor to skip the given
rlm@151 25 number of cycles 0 means that no cycles will be skipped.
rlm@151 26
rlm@151 27 (vision creature) will return [init-functions sensor-functions].
rlm@151 28 The init-functions are each single-arg functions that take the
rlm@151 29 world and register the cameras and must each be called before the
rlm@151 30 corresponding sensor-functions. Each init-function returns the
rlm@151 31 viewport for that eye which can be manipulated, saved, etc. Each
rlm@151 32 sensor-function is a thunk and will return data in the same
rlm@151 33 format as the tactile-sensor functions the structure is
rlm@151 34 [topology, sensor-data]. Internally, these sensor-functions
rlm@151 35 maintain a reference to sensor-data which is periodically updated
rlm@151 36 by the continuation function established by its init-function.
rlm@151 37 They can be queried every cycle, but their information may not
rlm@151 38 necessairly be different every cycle.
rlm@151 39
rlm@151 40 Each eye in the creature in blender will work the same way as
rlm@151 41 joints -- a zero dimensional object with no geometry whose local
rlm@151 42 coordinate system determines the orientation of the resulting
rlm@151 43 eye. All eyes will have a parent named "eyes" just as all joints
rlm@151 44 have a parent named "joints". The resulting camera will be a
rlm@151 45 ChaseCamera or a CameraNode bound to the geo that is closest to
rlm@151 46 the eye marker. The eye marker will contain the metadata for the
rlm@151 47 eye, and will be moved by it's bound geometry. The dimensions of
rlm@151 48 the eye's camera are equal to the dimensions of the eye's "UV"
rlm@151 49 map.
rlm@151 50
rlm@151 51
rlm@151 52
rlm@151 53
rlm@151 54
rlm@66 55 #+name: eyes
rlm@23 56 #+begin_src clojure
rlm@34 57 (ns cortex.vision
rlm@34 58 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
rlm@34 59 eyes from different positions to observe the same world, and pass
rlm@34 60 the observed data to any arbitray function."
rlm@34 61 {:author "Robert McIntyre"}
rlm@151 62 (:use (cortex world sense util))
rlm@34 63 (:import com.jme3.post.SceneProcessor)
rlm@113 64 (:import (com.jme3.util BufferUtils Screenshots))
rlm@34 65 (:import java.nio.ByteBuffer)
rlm@34 66 (:import java.awt.image.BufferedImage)
rlm@34 67 (:import com.jme3.renderer.ViewPort)
rlm@113 68 (:import com.jme3.math.ColorRGBA)
rlm@151 69 (:import com.jme3.renderer.Renderer)
rlm@151 70 (:import jme3tools.converters.ImageToAwt)
rlm@151 71 (:import com.jme3.scene.Node))
rlm@151 72
rlm@151 73 (cortex.import/mega-import-jme3)
rlm@23 74
rlm@113 75
rlm@113 76 (defn vision-pipeline
rlm@34 77 "Create a SceneProcessor object which wraps a vision processing
rlm@113 78 continuation function. The continuation is a function that takes
rlm@113 79 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@113 80 each of which has already been appropiately sized."
rlm@23 81 [continuation]
rlm@23 82 (let [byte-buffer (atom nil)
rlm@113 83 renderer (atom nil)
rlm@113 84 image (atom nil)]
rlm@23 85 (proxy [SceneProcessor] []
rlm@23 86 (initialize
rlm@23 87 [renderManager viewPort]
rlm@23 88 (let [cam (.getCamera viewPort)
rlm@23 89 width (.getWidth cam)
rlm@23 90 height (.getHeight cam)]
rlm@23 91 (reset! renderer (.getRenderer renderManager))
rlm@23 92 (reset! byte-buffer
rlm@23 93 (BufferUtils/createByteBuffer
rlm@113 94 (* width height 4)))
rlm@113 95 (reset! image (BufferedImage.
rlm@113 96 width height
rlm@113 97 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@23 98 (isInitialized [] (not (nil? @byte-buffer)))
rlm@23 99 (reshape [_ _ _])
rlm@23 100 (preFrame [_])
rlm@23 101 (postQueue [_])
rlm@23 102 (postFrame
rlm@23 103 [#^FrameBuffer fb]
rlm@23 104 (.clear @byte-buffer)
rlm@113 105 (continuation @renderer fb @byte-buffer @image))
rlm@23 106 (cleanup []))))
rlm@23 107
rlm@113 108 (defn frameBuffer->byteBuffer!
rlm@113 109 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
rlm@113 110 the CPU (ByteBuffer)."
rlm@113 111 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
rlm@113 112 (.readFrameBuffer r fb bb) bb)
rlm@113 113
rlm@113 114 (defn byteBuffer->bufferedImage!
rlm@113 115 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
rlm@113 116 style ABGR image data and place it in BufferedImage bi."
rlm@113 117 [#^ByteBuffer bb #^BufferedImage bi]
rlm@113 118 (Screenshots/convertScreenShot bb bi) bi)
rlm@113 119
rlm@113 120 (defn BufferedImage!
rlm@113 121 "Continuation which will grab the buffered image from the materials
rlm@113 122 provided by (vision-pipeline)."
rlm@113 123 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
rlm@113 124 (byteBuffer->bufferedImage!
rlm@113 125 (frameBuffer->byteBuffer! r fb bb) bi))
rlm@112 126
rlm@23 127 (defn add-eye
rlm@34 128 "Add an eye to the world, calling continuation on every frame
rlm@34 129 produced."
rlm@23 130 [world camera continuation]
rlm@23 131 (let [width (.getWidth camera)
rlm@23 132 height (.getHeight camera)
rlm@23 133 render-manager (.getRenderManager world)
rlm@23 134 viewport (.createMainView render-manager "eye-view" camera)]
rlm@23 135 (doto viewport
rlm@23 136 (.setClearFlags true true true)
rlm@112 137 (.setBackgroundColor ColorRGBA/Black)
rlm@113 138 (.addProcessor (vision-pipeline continuation))
rlm@23 139 (.attachScene (.getRootNode world)))))
rlm@151 140
rlm@151 141 (defn retina-sensor-image
rlm@151 142 "Return a map of pixel selection functions to BufferedImages
rlm@151 143 describing the distribution of light-sensitive components on this
rlm@151 144 geometry's surface. Each function creates an integer from the rgb
rlm@151 145 values found in the pixel. :red, :green, :blue, :gray are already
rlm@151 146 defined as extracting the red green blue and average components
rlm@151 147 respectively."
rlm@151 148 [#^Spatial eye]
rlm@151 149 (if-let [eye-map (meta-data eye "eye")]
rlm@151 150 (map-vals
rlm@151 151 #(ImageToAwt/convert
rlm@151 152 (.getImage (.loadTexture (asset-manager) %))
rlm@151 153 false false 0)
rlm@151 154 (eval (read-string eye-map)))))
rlm@151 155
rlm@151 156 (defn eye-dimensions
rlm@151 157 "returns the width and height specified in the metadata of the eye"
rlm@151 158 [#^Spatial eye]
rlm@151 159 (let [dimensions
rlm@151 160 (map #(vector (.getWidth %) (.getHeight %))
rlm@151 161 (vals (retina-sensor-image eye)))]
rlm@151 162 [(apply max (map first dimensions))
rlm@151 163 (apply max (map second dimensions))]))
rlm@151 164
rlm@151 165 (defn creature-eyes
rlm@151 166 ;;dylan
rlm@151 167 "Return the children of the creature's \"eyes\" node."
rlm@151 168 ;;"The eye nodes which are children of the \"eyes\" node in the
rlm@151 169 ;;creature."
rlm@151 170 [#^Node creature]
rlm@151 171 (if-let [eye-node (.getChild creature "eyes")]
rlm@151 172 (seq (.getChildren eye-node))
rlm@151 173 (do (println-repl "could not find eyes node") [])))
rlm@151 174
rlm@151 175
rlm@151 176 (defn attach-eye
rlm@151 177 "Attach a Camera to the appropiate area and return the Camera."
rlm@151 178 [#^Node creature #^Spatial eye]
rlm@151 179 (let [target (closest-node creature eye)
rlm@151 180 [cam-width cam-height] (eye-dimensions eye)
rlm@151 181 cam (Camera. cam-width cam-height)]
rlm@151 182 (.setLocation cam (.getWorldTranslation eye))
rlm@151 183 (.setRotation cam (.getWorldRotation eye))
rlm@151 184 (.setFrustumPerspective
rlm@151 185 cam 45 (/ (.getWidth cam) (.getHeight cam))
rlm@151 186 1 1000)
rlm@151 187 (bind-sense target cam)
rlm@151 188 cam))
rlm@151 189
rlm@151 190 (def presets
rlm@151 191 {:all 0xFFFFFF
rlm@151 192 :red 0xFF0000
rlm@151 193 :blue 0x0000FF
rlm@151 194 :green 0x00FF00})
rlm@151 195
rlm@151 196 (defn enable-vision
rlm@151 197 "return [init-function sensor-functions] for a particular eye"
rlm@151 198 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@151 199 (let [retinal-map (retina-sensor-image eye)
rlm@151 200 camera (attach-eye creature eye)
rlm@151 201 vision-image
rlm@151 202 (atom
rlm@151 203 (BufferedImage. (.getWidth camera)
rlm@151 204 (.getHeight camera)
rlm@151 205 BufferedImage/TYPE_BYTE_BINARY))]
rlm@151 206 [(fn [world]
rlm@151 207 (add-eye
rlm@151 208 world camera
rlm@151 209 (let [counter (atom 0)]
rlm@151 210 (fn [r fb bb bi]
rlm@151 211 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@151 212 (reset! vision-image (BufferedImage! r fb bb bi)))))))
rlm@151 213 (vec
rlm@151 214 (map
rlm@151 215 (fn [[key image]]
rlm@151 216 (let [whites (white-coordinates image)
rlm@151 217 topology (vec (collapse whites))
rlm@151 218 mask (presets key)]
rlm@151 219 (fn []
rlm@151 220 (vector
rlm@151 221 topology
rlm@151 222 (vec
rlm@151 223 (for [[x y] whites]
rlm@151 224 (bit-and
rlm@151 225 mask (.getRGB @vision-image x y))))))))
rlm@151 226 retinal-map))]))
rlm@151 227
rlm@151 228 (defn vision
rlm@151 229 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@151 230 (reduce
rlm@151 231 (fn [[init-a senses-a]
rlm@151 232 [init-b senses-b]]
rlm@151 233 [(conj init-a init-b)
rlm@151 234 (into senses-a senses-b)])
rlm@151 235 [[][]]
rlm@151 236 (for [eye (creature-eyes creature)]
rlm@151 237 (enable-vision creature eye))))
rlm@151 238
rlm@151 239
rlm@34 240 #+end_src
rlm@23 241
rlm@112 242
rlm@34 243 Note the use of continuation passing style for connecting the eye to a
rlm@34 244 function to process the output. You can create any number of eyes, and
rlm@34 245 each of them will see the world from their own =Camera=. Once every
rlm@34 246 frame, the rendered image is copied to a =BufferedImage=, and that
rlm@34 247 data is sent off to the continuation function. Moving the =Camera=
rlm@34 248 which was used to create the eye will change what the eye sees.
rlm@23 249
rlm@34 250 * Example
rlm@23 251
rlm@66 252 #+name: test-vision
rlm@23 253 #+begin_src clojure
rlm@68 254 (ns cortex.test.vision
rlm@34 255 (:use (cortex world util vision))
rlm@34 256 (:import java.awt.image.BufferedImage)
rlm@34 257 (:import javax.swing.JPanel)
rlm@34 258 (:import javax.swing.SwingUtilities)
rlm@34 259 (:import java.awt.Dimension)
rlm@34 260 (:import javax.swing.JFrame)
rlm@34 261 (:import com.jme3.math.ColorRGBA)
rlm@45 262 (:import com.jme3.scene.Node)
rlm@113 263 (:import com.jme3.math.Vector3f))
rlm@23 264
rlm@36 265 (defn test-two-eyes
rlm@69 266 "Testing vision:
rlm@69 267 Tests the vision system by creating two views of the same rotating
rlm@69 268 object from different angles and displaying both of those views in
rlm@69 269 JFrames.
rlm@69 270
rlm@69 271 You should see a rotating cube, and two windows,
rlm@69 272 each displaying a different view of the cube."
rlm@36 273 []
rlm@58 274 (let [candy
rlm@58 275 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@112 276 (world
rlm@112 277 (doto (Node.)
rlm@112 278 (.attachChild candy))
rlm@112 279 {}
rlm@112 280 (fn [world]
rlm@112 281 (let [cam (.clone (.getCamera world))
rlm@112 282 width (.getWidth cam)
rlm@112 283 height (.getHeight cam)]
rlm@112 284 (add-eye world cam
rlm@113 285 ;;no-op
rlm@113 286 (comp (view-image) BufferedImage!)
rlm@112 287 )
rlm@112 288 (add-eye world
rlm@112 289 (doto (.clone cam)
rlm@112 290 (.setLocation (Vector3f. -10 0 0))
rlm@112 291 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@113 292 ;;no-op
rlm@113 293 (comp (view-image) BufferedImage!))
rlm@112 294 ;; This is here to restore the main view
rlm@112 295 ;; after the other views have completed processing
rlm@112 296 (add-eye world (.getCamera world) no-op)))
rlm@112 297 (fn [world tpf]
rlm@112 298 (.rotate candy (* tpf 0.2) 0 0)))))
rlm@23 299 #+end_src
rlm@23 300
rlm@112 301 #+results: test-vision
rlm@112 302 : #'cortex.test.vision/test-two-eyes
rlm@112 303
rlm@34 304 The example code will create two videos of the same rotating object
rlm@34 305 from different angles. It can be used both for stereoscopic vision
rlm@34 306 simulation or for simulating multiple creatures, each with their own
rlm@34 307 sense of vision.
rlm@24 308
rlm@35 309 - As a neat bonus, this idea behind simulated vision also enables one
rlm@35 310 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
rlm@35 311
rlm@24 312
rlm@24 313 * COMMENT code generation
rlm@34 314 #+begin_src clojure :tangle ../src/cortex/vision.clj
rlm@24 315 <<eyes>>
rlm@24 316 #+end_src
rlm@24 317
rlm@68 318 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
rlm@24 319 <<test-vision>>
rlm@24 320 #+end_src