view org/vision.org @ 201:1c915cc1118b

minor edits for sense.org
author Robert McIntyre <rlm@mit.edu>
date Tue, 07 Feb 2012 07:13:45 -0700
parents ac158a976443
children 8e9825c38941
line wrap: on
line source
1 #+title: Simulated Sense of Sight
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both
10 * Vision
12 I want to make creatures with eyes. Each eye can be independely moved
13 and should see its own version of the world depending on where it is.
15 Here's how vision will work.
17 Make the continuation in scene-processor take FrameBuffer,
18 byte-buffer, BufferedImage already sized to the correct
19 dimensions. the continuation will decide wether to "mix" them
20 into the BufferedImage, lazily ignore them, or mix them halfway
21 and call c/graphics card routines.
23 (vision creature) will take an optional :skip argument which will
24 inform the continuations in scene processor to skip the given
25 number of cycles 0 means that no cycles will be skipped.
27 (vision creature) will return [init-functions sensor-functions].
28 The init-functions are each single-arg functions that take the
29 world and register the cameras and must each be called before the
30 corresponding sensor-functions. Each init-function returns the
31 viewport for that eye which can be manipulated, saved, etc. Each
32 sensor-function is a thunk and will return data in the same
33 format as the tactile-sensor functions the structure is
34 [topology, sensor-data]. Internally, these sensor-functions
35 maintain a reference to sensor-data which is periodically updated
36 by the continuation function established by its init-function.
37 They can be queried every cycle, but their information may not
38 necessairly be different every cycle.
40 Each eye in the creature in blender will work the same way as
41 joints -- a zero dimensional object with no geometry whose local
42 coordinate system determines the orientation of the resulting
43 eye. All eyes will have a parent named "eyes" just as all joints
44 have a parent named "joints". The resulting camera will be a
45 ChaseCamera or a CameraNode bound to the geo that is closest to
46 the eye marker. The eye marker will contain the metadata for the
47 eye, and will be moved by it's bound geometry. The dimensions of
48 the eye's camera are equal to the dimensions of the eye's "UV"
49 map.
51 #+name: eyes
52 #+begin_src clojure
53 (ns cortex.vision
54 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
55 eyes from different positions to observe the same world, and pass
56 the observed data to any arbitray function. Automatically reads
57 eye-nodes from specially prepared blender files and instanttiates
58 them in the world as actual eyes."
59 {:author "Robert McIntyre"}
60 (:use (cortex world sense util))
61 (:use clojure.contrib.def)
62 (:import com.jme3.post.SceneProcessor)
63 (:import (com.jme3.util BufferUtils Screenshots))
64 (:import java.nio.ByteBuffer)
65 (:import java.awt.image.BufferedImage)
66 (:import (com.jme3.renderer ViewPort Camera))
67 (:import com.jme3.math.ColorRGBA)
68 (:import com.jme3.renderer.Renderer)
69 (:import com.jme3.app.Application)
70 (:import com.jme3.texture.FrameBuffer)
71 (:import (com.jme3.scene Node Spatial)))
73 (defn vision-pipeline
74 "Create a SceneProcessor object which wraps a vision processing
75 continuation function. The continuation is a function that takes
76 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
77 each of which has already been appropiately sized."
78 [continuation]
79 (let [byte-buffer (atom nil)
80 renderer (atom nil)
81 image (atom nil)]
82 (proxy [SceneProcessor] []
83 (initialize
84 [renderManager viewPort]
85 (let [cam (.getCamera viewPort)
86 width (.getWidth cam)
87 height (.getHeight cam)]
88 (reset! renderer (.getRenderer renderManager))
89 (reset! byte-buffer
90 (BufferUtils/createByteBuffer
91 (* width height 4)))
92 (reset! image (BufferedImage.
93 width height
94 BufferedImage/TYPE_4BYTE_ABGR))))
95 (isInitialized [] (not (nil? @byte-buffer)))
96 (reshape [_ _ _])
97 (preFrame [_])
98 (postQueue [_])
99 (postFrame
100 [#^FrameBuffer fb]
101 (.clear @byte-buffer)
102 (continuation @renderer fb @byte-buffer @image))
103 (cleanup []))))
105 (defn frameBuffer->byteBuffer!
106 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
107 the CPU (ByteBuffer)."
108 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
109 (.readFrameBuffer r fb bb) bb)
111 (defn byteBuffer->bufferedImage!
112 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
113 style ABGR image data and place it in BufferedImage bi."
114 [#^ByteBuffer bb #^BufferedImage bi]
115 (Screenshots/convertScreenShot bb bi) bi)
117 (defn BufferedImage!
118 "Continuation which will grab the buffered image from the materials
119 provided by (vision-pipeline)."
120 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
121 (byteBuffer->bufferedImage!
122 (frameBuffer->byteBuffer! r fb bb) bi))
124 (defn add-camera!
125 "Add a camera to the world, calling continuation on every frame
126 produced."
127 [#^Application world camera continuation]
128 (let [width (.getWidth camera)
129 height (.getHeight camera)
130 render-manager (.getRenderManager world)
131 viewport (.createMainView render-manager "eye-view" camera)]
132 (doto viewport
133 (.setClearFlags true true true)
134 (.setBackgroundColor ColorRGBA/Black)
135 (.addProcessor (vision-pipeline continuation))
136 (.attachScene (.getRootNode world)))))
138 (defn retina-sensor-profile
139 "Return a map of pixel selection functions to BufferedImages
140 describing the distribution of light-sensitive components of this
141 eye. Each function creates an integer from the rgb values found in
142 the pixel. :red, :green, :blue, :gray are already defined as
143 extracting the red, green, blue, and average components
144 respectively."
145 [#^Spatial eye]
146 (if-let [eye-map (meta-data eye "eye")]
147 (map-vals
148 load-image
149 (eval (read-string eye-map)))))
151 (defn eye-dimensions
152 "Returns [width, height] specified in the metadata of the eye"
153 [#^Spatial eye]
154 (let [dimensions
155 (map #(vector (.getWidth %) (.getHeight %))
156 (vals (retina-sensor-profile eye)))]
157 [(apply max (map first dimensions))
158 (apply max (map second dimensions))]))
160 (defvar
161 ^{:arglists '([creature])}
162 eyes
163 (sense-nodes "eyes")
164 "Return the children of the creature's \"eyes\" node.")
166 (defn add-eye!
167 "Create a Camera centered on the current position of 'eye which
168 follows the closest physical node in 'creature and sends visual
169 data to 'continuation."
170 [#^Node creature #^Spatial eye]
171 (let [target (closest-node creature eye)
172 [cam-width cam-height] (eye-dimensions eye)
173 cam (Camera. cam-width cam-height)]
174 (.setLocation cam (.getWorldTranslation eye))
175 (.setRotation cam (.getWorldRotation eye))
176 (.setFrustumPerspective
177 cam 45 (/ (.getWidth cam) (.getHeight cam))
178 1 1000)
179 (bind-sense target cam)
180 cam))
182 (defvar color-channel-presets
183 {:all 0xFFFFFF
184 :red 0xFF0000
185 :blue 0x0000FF
186 :green 0x00FF00}
187 "Bitmasks for common RGB color channels")
189 (defn vision-fn
190 "Returns a list of functions, each of which will return a color
191 channel's worth of visual information when called inside a running
192 simulation."
193 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
194 (let [retinal-map (retina-sensor-profile eye)
195 camera (add-eye! creature eye)
196 vision-image
197 (atom
198 (BufferedImage. (.getWidth camera)
199 (.getHeight camera)
200 BufferedImage/TYPE_BYTE_BINARY))
201 register-eye!
202 (runonce
203 (fn [world]
204 (add-camera!
205 world camera
206 (let [counter (atom 0)]
207 (fn [r fb bb bi]
208 (if (zero? (rem (swap! counter inc) (inc skip)))
209 (reset! vision-image
210 (BufferedImage! r fb bb bi))))))))]
211 (vec
212 (map
213 (fn [[key image]]
214 (let [whites (white-coordinates image)
215 topology (vec (collapse whites))
216 mask (color-channel-presets key)]
217 (fn [world]
218 (register-eye! world)
219 (vector
220 topology
221 (vec
222 (for [[x y] whites]
223 (bit-and
224 mask (.getRGB @vision-image x y))))))))
225 retinal-map))))
228 ;; TODO maybe should add a viewport-manipulation function to
229 ;; automatically change viewport settings, attach shadow filters, etc.
231 (defn vision!
232 "Returns a function which returns visual sensory data when called
233 inside a running simulation"
234 [#^Node creature & {skip :skip :or {skip 0}}]
235 (reduce
236 concat
237 (for [eye (eyes creature)]
238 (vision-fn creature eye))))
240 (defn view-vision
241 "Creates a function which accepts a list of visual sensor-data and
242 displays each element of the list to the screen."
243 []
244 (view-sense
245 (fn
246 [[coords sensor-data]]
247 (let [image (points->image coords)]
248 (dorun
249 (for [i (range (count coords))]
250 (.setRGB image ((coords i) 0) ((coords i) 1)
251 (sensor-data i))))
252 image))))
254 #+end_src
257 Note the use of continuation passing style for connecting the eye to a
258 function to process the output. You can create any number of eyes, and
259 each of them will see the world from their own =Camera=. Once every
260 frame, the rendered image is copied to a =BufferedImage=, and that
261 data is sent off to the continuation function. Moving the =Camera=
262 which was used to create the eye will change what the eye sees.
264 * Example
266 #+name: test-vision
267 #+begin_src clojure
268 (ns cortex.test.vision
269 (:use (cortex world util vision))
270 (:import java.awt.image.BufferedImage)
271 (:import javax.swing.JPanel)
272 (:import javax.swing.SwingUtilities)
273 (:import java.awt.Dimension)
274 (:import javax.swing.JFrame)
275 (:import com.jme3.math.ColorRGBA)
276 (:import com.jme3.scene.Node)
277 (:import com.jme3.math.Vector3f))
279 (defn test-two-eyes
280 "Testing vision:
281 Tests the vision system by creating two views of the same rotating
282 object from different angles and displaying both of those views in
283 JFrames.
285 You should see a rotating cube, and two windows,
286 each displaying a different view of the cube."
287 []
288 (let [candy
289 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
290 (world
291 (doto (Node.)
292 (.attachChild candy))
293 {}
294 (fn [world]
295 (let [cam (.clone (.getCamera world))
296 width (.getWidth cam)
297 height (.getHeight cam)]
298 (add-camera! world cam
299 ;;no-op
300 (comp (view-image) BufferedImage!)
301 )
302 (add-camera! world
303 (doto (.clone cam)
304 (.setLocation (Vector3f. -10 0 0))
305 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
306 ;;no-op
307 (comp (view-image) BufferedImage!))
308 ;; This is here to restore the main view
309 ;; after the other views have completed processing
310 (add-camera! world (.getCamera world) no-op)))
311 (fn [world tpf]
312 (.rotate candy (* tpf 0.2) 0 0)))))
313 #+end_src
315 #+results: test-vision
316 : #'cortex.test.vision/test-two-eyes
318 The example code will create two videos of the same rotating object
319 from different angles. It can be used both for stereoscopic vision
320 simulation or for simulating multiple creatures, each with their own
321 sense of vision.
323 - As a neat bonus, this idea behind simulated vision also enables one
324 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
327 * COMMENT code generation
328 #+begin_src clojure :tangle ../src/cortex/vision.clj
329 <<eyes>>
330 #+end_src
332 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
333 <<test-vision>>
334 #+end_src