|
rlm@34
|
1 #+title: Simulated Sense of Sight
|
|
rlm@23
|
2 #+author: Robert McIntyre
|
|
rlm@23
|
3 #+email: rlm@mit.edu
|
|
rlm@38
|
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
|
|
rlm@34
|
5 #+keywords: computer vision, jMonkeyEngine3, clojure
|
|
rlm@23
|
6 #+SETUPFILE: ../../aurellem/org/setup.org
|
|
rlm@23
|
7 #+INCLUDE: ../../aurellem/org/level-0.org
|
|
rlm@23
|
8 #+babel: :mkdirp yes :noweb yes :exports both
|
|
rlm@23
|
9
|
|
rlm@194
|
10 * Vision
|
|
rlm@23
|
11
|
|
rlm@151
|
12
|
|
rlm@212
|
13 Vision is one of the most important senses for humans, so I need to
|
|
rlm@212
|
14 build a simulated sense of vision for my AI. I will do this with
|
|
rlm@212
|
15 simulated eyes. Each eye can be independely moved and should see its
|
|
rlm@212
|
16 own version of the world depending on where it is.
|
|
rlm@212
|
17
|
|
rlm@212
|
18 Making these simulated eyes a reality is fairly simple bacause
|
|
rlm@212
|
19 jMonkeyEngine already conatains extensive support for multiple views
|
|
rlm@212
|
20 of the same 3D simulated world. The reason jMonkeyEngine has this
|
|
rlm@212
|
21 support is because the support is necessary to create games with
|
|
rlm@212
|
22 split-screen views. Multiple views are also used to create efficient
|
|
rlm@212
|
23 pseudo-reflections by rendering the scene from a certain perspective
|
|
rlm@212
|
24 and then projecting it back onto a surface in the 3D world.
|
|
rlm@212
|
25
|
|
rlm@212
|
26 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye
|
|
rlm@212
|
27 [[../images/goldeneye-4-player.png]]
|
|
rlm@212
|
28
|
|
rlm@213
|
29 * Brief Description of jMonkeyEngine's Rendering Pipeline
|
|
rlm@212
|
30
|
|
rlm@213
|
31 jMonkeyEngine allows you to create a =ViewPort=, which represents a
|
|
rlm@213
|
32 view of the simulated world. You can create as many of these as you
|
|
rlm@213
|
33 want. Every frame, the =RenderManager= iterates through each
|
|
rlm@213
|
34 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
|
|
rlm@213
|
35 is a =FrameBuffer= which represents the rendered image in the GPU.
|
|
rlm@151
|
36
|
|
rlm@213
|
37 Each =ViewPort= can have any number of attached =SceneProcessor=
|
|
rlm@213
|
38 objects, which are called every time a new frame is rendered. A
|
|
rlm@213
|
39 =SceneProcessor= recieves a =FrameBuffer= and can do whatever it wants
|
|
rlm@213
|
40 to the data. Often this consists of invoking GPU specific operations
|
|
rlm@213
|
41 on the rendered image. The =SceneProcessor= can also copy the GPU
|
|
rlm@213
|
42 image data to RAM and process it with the CPU.
|
|
rlm@151
|
43
|
|
rlm@213
|
44 * The Vision Pipeline
|
|
rlm@151
|
45
|
|
rlm@213
|
46 Each eye in the simulated creature needs it's own =ViewPort= so that
|
|
rlm@213
|
47 it can see the world from its own perspective. To this =ViewPort=, I
|
|
rlm@214
|
48 add a =SceneProcessor= that feeds the visual data to any arbitray
|
|
rlm@213
|
49 continuation function for further processing. That continuation
|
|
rlm@213
|
50 function may perform both CPU and GPU operations on the data. To make
|
|
rlm@213
|
51 this easy for the continuation function, the =SceneProcessor=
|
|
rlm@213
|
52 maintains appropriatly sized buffers in RAM to hold the data. It does
|
|
rlm@213
|
53 not do any copying from the GPU to the CPU itself.
|
|
rlm@214
|
54
|
|
rlm@213
|
55 #+name: pipeline-1
|
|
rlm@213
|
56 #+begin_src clojure
|
|
rlm@113
|
57 (defn vision-pipeline
|
|
rlm@34
|
58 "Create a SceneProcessor object which wraps a vision processing
|
|
rlm@113
|
59 continuation function. The continuation is a function that takes
|
|
rlm@113
|
60 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
|
|
rlm@113
|
61 each of which has already been appropiately sized."
|
|
rlm@23
|
62 [continuation]
|
|
rlm@23
|
63 (let [byte-buffer (atom nil)
|
|
rlm@113
|
64 renderer (atom nil)
|
|
rlm@113
|
65 image (atom nil)]
|
|
rlm@23
|
66 (proxy [SceneProcessor] []
|
|
rlm@23
|
67 (initialize
|
|
rlm@23
|
68 [renderManager viewPort]
|
|
rlm@23
|
69 (let [cam (.getCamera viewPort)
|
|
rlm@23
|
70 width (.getWidth cam)
|
|
rlm@23
|
71 height (.getHeight cam)]
|
|
rlm@23
|
72 (reset! renderer (.getRenderer renderManager))
|
|
rlm@23
|
73 (reset! byte-buffer
|
|
rlm@23
|
74 (BufferUtils/createByteBuffer
|
|
rlm@113
|
75 (* width height 4)))
|
|
rlm@113
|
76 (reset! image (BufferedImage.
|
|
rlm@113
|
77 width height
|
|
rlm@113
|
78 BufferedImage/TYPE_4BYTE_ABGR))))
|
|
rlm@23
|
79 (isInitialized [] (not (nil? @byte-buffer)))
|
|
rlm@23
|
80 (reshape [_ _ _])
|
|
rlm@23
|
81 (preFrame [_])
|
|
rlm@23
|
82 (postQueue [_])
|
|
rlm@23
|
83 (postFrame
|
|
rlm@23
|
84 [#^FrameBuffer fb]
|
|
rlm@23
|
85 (.clear @byte-buffer)
|
|
rlm@113
|
86 (continuation @renderer fb @byte-buffer @image))
|
|
rlm@23
|
87 (cleanup []))))
|
|
rlm@213
|
88 #+end_src
|
|
rlm@213
|
89
|
|
rlm@213
|
90 The continuation function given to =(vision-pipeline)= above will be
|
|
rlm@213
|
91 given a =Renderer= and three containers for image data. The
|
|
rlm@213
|
92 =FrameBuffer= references the GPU image data, but it can not be used
|
|
rlm@213
|
93 directly on the CPU. The =ByteBuffer= and =BufferedImage= are
|
|
rlm@213
|
94 initially "empty" but are sized to hold to data in the
|
|
rlm@213
|
95 =FrameBuffer=. I call transfering the GPU image data to the CPU
|
|
rlm@213
|
96 structures "mixing" the image data. I have provided three functions to
|
|
rlm@213
|
97 do this mixing.
|
|
rlm@213
|
98
|
|
rlm@213
|
99 #+name: pipeline-2
|
|
rlm@213
|
100 #+begin_src clojure
|
|
rlm@113
|
101 (defn frameBuffer->byteBuffer!
|
|
rlm@113
|
102 "Transfer the data in the graphics card (Renderer, FrameBuffer) to
|
|
rlm@113
|
103 the CPU (ByteBuffer)."
|
|
rlm@113
|
104 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]
|
|
rlm@113
|
105 (.readFrameBuffer r fb bb) bb)
|
|
rlm@113
|
106
|
|
rlm@113
|
107 (defn byteBuffer->bufferedImage!
|
|
rlm@113
|
108 "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT
|
|
rlm@113
|
109 style ABGR image data and place it in BufferedImage bi."
|
|
rlm@113
|
110 [#^ByteBuffer bb #^BufferedImage bi]
|
|
rlm@113
|
111 (Screenshots/convertScreenShot bb bi) bi)
|
|
rlm@113
|
112
|
|
rlm@113
|
113 (defn BufferedImage!
|
|
rlm@113
|
114 "Continuation which will grab the buffered image from the materials
|
|
rlm@113
|
115 provided by (vision-pipeline)."
|
|
rlm@113
|
116 [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]
|
|
rlm@113
|
117 (byteBuffer->bufferedImage!
|
|
rlm@113
|
118 (frameBuffer->byteBuffer! r fb bb) bi))
|
|
rlm@213
|
119 #+end_src
|
|
rlm@112
|
120
|
|
rlm@213
|
121 Note that it is possible to write vision processing algorithms
|
|
rlm@213
|
122 entirely in terms of =BufferedImage= inputs. Just compose that
|
|
rlm@213
|
123 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision
|
|
rlm@213
|
124 processing algorithm that is entirely hosted on the GPU does not have
|
|
rlm@213
|
125 to pay for this convienence.
|
|
rlm@213
|
126
|
|
rlm@214
|
127 * COMMENT asdasd
|
|
rlm@213
|
128
|
|
rlm@213
|
129 (vision creature) will take an optional :skip argument which will
|
|
rlm@213
|
130 inform the continuations in scene processor to skip the given
|
|
rlm@213
|
131 number of cycles 0 means that no cycles will be skipped.
|
|
rlm@213
|
132
|
|
rlm@213
|
133 (vision creature) will return [init-functions sensor-functions].
|
|
rlm@213
|
134 The init-functions are each single-arg functions that take the
|
|
rlm@213
|
135 world and register the cameras and must each be called before the
|
|
rlm@213
|
136 corresponding sensor-functions. Each init-function returns the
|
|
rlm@213
|
137 viewport for that eye which can be manipulated, saved, etc. Each
|
|
rlm@213
|
138 sensor-function is a thunk and will return data in the same
|
|
rlm@213
|
139 format as the tactile-sensor functions the structure is
|
|
rlm@213
|
140 [topology, sensor-data]. Internally, these sensor-functions
|
|
rlm@213
|
141 maintain a reference to sensor-data which is periodically updated
|
|
rlm@213
|
142 by the continuation function established by its init-function.
|
|
rlm@213
|
143 They can be queried every cycle, but their information may not
|
|
rlm@213
|
144 necessairly be different every cycle.
|
|
rlm@213
|
145
|
|
rlm@213
|
146
|
|
rlm@214
|
147
|
|
rlm@214
|
148 * Physical Eyes
|
|
rlm@214
|
149
|
|
rlm@214
|
150 The vision pipeline described above handles the flow of rendered
|
|
rlm@214
|
151 images. Now, we need simulated eyes to serve as the source of these
|
|
rlm@214
|
152 images.
|
|
rlm@214
|
153
|
|
rlm@214
|
154 An eye is described in blender in the same way as a joint. They are
|
|
rlm@214
|
155 zero dimensional empty objects with no geometry whose local coordinate
|
|
rlm@214
|
156 system determines the orientation of the resulting eye. All eyes are
|
|
rlm@214
|
157 childern of a parent node named "eyes" just as all joints have a
|
|
rlm@214
|
158 parent named "joints". An eye binds to the nearest physical object
|
|
rlm@214
|
159 with =(bind-sense=).
|
|
rlm@214
|
160
|
|
rlm@214
|
161 #+name: add-eye
|
|
rlm@214
|
162 #+begin_src clojure
|
|
rlm@214
|
163 (defn add-eye!
|
|
rlm@214
|
164 "Create a Camera centered on the current position of 'eye which
|
|
rlm@214
|
165 follows the closest physical node in 'creature and sends visual
|
|
rlm@214
|
166 data to 'continuation."
|
|
rlm@214
|
167 [#^Node creature #^Spatial eye]
|
|
rlm@214
|
168 (let [target (closest-node creature eye)
|
|
rlm@214
|
169 [cam-width cam-height] (eye-dimensions eye)
|
|
rlm@214
|
170 cam (Camera. cam-width cam-height)]
|
|
rlm@214
|
171 (.setLocation cam (.getWorldTranslation eye))
|
|
rlm@214
|
172 (.setRotation cam (.getWorldRotation eye))
|
|
rlm@214
|
173 (.setFrustumPerspective
|
|
rlm@214
|
174 cam 45 (/ (.getWidth cam) (.getHeight cam))
|
|
rlm@214
|
175 1 1000)
|
|
rlm@214
|
176 (bind-sense target cam)
|
|
rlm@214
|
177 cam))
|
|
rlm@214
|
178 #+end_src
|
|
rlm@214
|
179
|
|
rlm@214
|
180 Here, the camera is created based on metadata on the eye-node and
|
|
rlm@214
|
181 attached to the nearest physical object with =(bind-sense)=
|
|
rlm@214
|
182
|
|
rlm@214
|
183
|
|
rlm@214
|
184 ** The Retina
|
|
rlm@214
|
185
|
|
rlm@214
|
186 An eye is a surface (the retina) which contains many discrete sensors
|
|
rlm@214
|
187 to detect light. These sensors have can have different-light sensing
|
|
rlm@214
|
188 properties. In humans, each discrete sensor is sensitive to red,
|
|
rlm@214
|
189 blue, green, or gray. These different types of sensors can have
|
|
rlm@214
|
190 different spatial distributions along the retina. In humans, there is
|
|
rlm@214
|
191 a fovea in the center of the retina which has a very high density of
|
|
rlm@214
|
192 color sensors, and a blind spot which has no sensors at all. Sensor
|
|
rlm@214
|
193 density decreases in proportion to distance from the retina.
|
|
rlm@214
|
194
|
|
rlm@214
|
195 I want to be able to model any retinal configuration, so my eye-nodes
|
|
rlm@214
|
196 in blender contain metadata pointing to images that describe the
|
|
rlm@214
|
197 percise position of the individual sensors using white pixels. The
|
|
rlm@214
|
198 meta-data also describes the percise sensitivity to light that the
|
|
rlm@214
|
199 sensors described in the image have. An eye can contain any number of
|
|
rlm@214
|
200 these images. For example, the metadata for an eye might look like
|
|
rlm@214
|
201 this:
|
|
rlm@214
|
202
|
|
rlm@214
|
203 #+begin_src clojure
|
|
rlm@214
|
204 {0xFF0000 "Models/test-creature/retina-small.png"}
|
|
rlm@214
|
205 #+end_src
|
|
rlm@214
|
206
|
|
rlm@214
|
207 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.
|
|
rlm@214
|
208 [[../assets/Models/test-creature/retina-small.png]]
|
|
rlm@214
|
209
|
|
rlm@214
|
210 Together, the number 0xFF0000 and the image image above describe the
|
|
rlm@214
|
211 placement of red-sensitive sensory elements.
|
|
rlm@214
|
212
|
|
rlm@214
|
213 Meta-data to very crudely approximate a human eye might be something
|
|
rlm@214
|
214 like this:
|
|
rlm@214
|
215
|
|
rlm@214
|
216 #+begin_src clojure
|
|
rlm@214
|
217 (let [retinal-profile "Models/test-creature/retina-small.png"]
|
|
rlm@214
|
218 {0xFF0000 retinal-profile
|
|
rlm@214
|
219 0x00FF00 retinal-profile
|
|
rlm@214
|
220 0x0000FF retinal-profile
|
|
rlm@214
|
221 0xFFFFFF retinal-profile})
|
|
rlm@214
|
222 #+end_src
|
|
rlm@214
|
223
|
|
rlm@214
|
224 The numbers that serve as keys in the map determine a sensor's
|
|
rlm@214
|
225 relative sensitivity to the channels red, green, and blue. These
|
|
rlm@214
|
226 sensitivity values are packed into an integer in the order _RGB in
|
|
rlm@214
|
227 8-bit fields. The RGB values of a pixel in the image are added
|
|
rlm@214
|
228 together with these sensitivities as linear weights. Therfore,
|
|
rlm@214
|
229 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
|
|
rlm@214
|
230 all colors equally (gray).
|
|
rlm@214
|
231
|
|
rlm@214
|
232 For convienence I've defined a few symbols for the more common
|
|
rlm@214
|
233 sensitivity values.
|
|
rlm@214
|
234
|
|
rlm@214
|
235 #+name: sensitivity
|
|
rlm@214
|
236 #+begin_src clojure
|
|
rlm@214
|
237 (defvar sensitivity-presets
|
|
rlm@214
|
238 {:all 0xFFFFFF
|
|
rlm@214
|
239 :red 0xFF0000
|
|
rlm@214
|
240 :blue 0x0000FF
|
|
rlm@214
|
241 :green 0x00FF00}
|
|
rlm@214
|
242 "Retinal sensitivity presets for sensors that extract one channel
|
|
rlm@214
|
243 (:red :blue :green) or average all channels (:gray)")
|
|
rlm@214
|
244 #+end_src
|
|
rlm@214
|
245
|
|
rlm@214
|
246 ** Metadata Processing
|
|
rlm@214
|
247
|
|
rlm@214
|
248 =(retina-sensor-profile)= extracts a map from the eye-node in the same
|
|
rlm@214
|
249 format as the example maps above. =(eye-dimensions)= finds the
|
|
rlm@214
|
250 dimansions of the smallest image required to contain all the retinal
|
|
rlm@214
|
251 sensor maps.
|
|
rlm@214
|
252
|
|
rlm@214
|
253 #+begin_src clojure
|
|
rlm@214
|
254 (defn retina-sensor-profile
|
|
rlm@214
|
255 "Return a map of pixel sensitivity numbers to BufferedImages
|
|
rlm@214
|
256 describing the distribution of light-sensitive components of this
|
|
rlm@214
|
257 eye. :red, :green, :blue, :gray are already defined as extracting
|
|
rlm@214
|
258 the red, green, blue, and average components respectively."
|
|
rlm@214
|
259 [#^Spatial eye]
|
|
rlm@214
|
260 (if-let [eye-map (meta-data eye "eye")]
|
|
rlm@214
|
261 (map-vals
|
|
rlm@214
|
262 load-image
|
|
rlm@214
|
263 (eval (read-string eye-map)))))
|
|
rlm@214
|
264
|
|
rlm@214
|
265 (defn eye-dimensions
|
|
rlm@214
|
266 "Returns [width, height] specified in the metadata of the eye"
|
|
rlm@214
|
267 [#^Spatial eye]
|
|
rlm@214
|
268 (let [dimensions
|
|
rlm@214
|
269 (map #(vector (.getWidth %) (.getHeight %))
|
|
rlm@214
|
270 (vals (retina-sensor-profile eye)))]
|
|
rlm@214
|
271 [(apply max (map first dimensions))
|
|
rlm@214
|
272 (apply max (map second dimensions))]))
|
|
rlm@214
|
273 #+end_src
|
|
rlm@214
|
274
|
|
rlm@214
|
275
|
|
rlm@214
|
276 * Eye Creation
|
|
rlm@214
|
277
|
|
rlm@214
|
278 First off, get the children of the "eyes" empty node to find all the
|
|
rlm@214
|
279 eyes the creature has.
|
|
rlm@214
|
280
|
|
rlm@214
|
281 #+begin_src clojure
|
|
rlm@214
|
282 (defvar
|
|
rlm@214
|
283 ^{:arglists '([creature])}
|
|
rlm@214
|
284 eyes
|
|
rlm@214
|
285 (sense-nodes "eyes")
|
|
rlm@214
|
286 "Return the children of the creature's \"eyes\" node.")
|
|
rlm@214
|
287 #+end_src
|
|
rlm@214
|
288
|
|
rlm@214
|
289 Then,
|
|
rlm@214
|
290
|
|
rlm@213
|
291 #+begin_src clojure
|
|
rlm@169
|
292 (defn add-camera!
|
|
rlm@169
|
293 "Add a camera to the world, calling continuation on every frame
|
|
rlm@34
|
294 produced."
|
|
rlm@167
|
295 [#^Application world camera continuation]
|
|
rlm@23
|
296 (let [width (.getWidth camera)
|
|
rlm@23
|
297 height (.getHeight camera)
|
|
rlm@23
|
298 render-manager (.getRenderManager world)
|
|
rlm@23
|
299 viewport (.createMainView render-manager "eye-view" camera)]
|
|
rlm@23
|
300 (doto viewport
|
|
rlm@23
|
301 (.setClearFlags true true true)
|
|
rlm@112
|
302 (.setBackgroundColor ColorRGBA/Black)
|
|
rlm@113
|
303 (.addProcessor (vision-pipeline continuation))
|
|
rlm@23
|
304 (.attachScene (.getRootNode world)))))
|
|
rlm@151
|
305
|
|
rlm@151
|
306
|
|
rlm@151
|
307
|
|
rlm@151
|
308
|
|
rlm@151
|
309
|
|
rlm@169
|
310 (defn vision-fn
|
|
rlm@171
|
311 "Returns a list of functions, each of which will return a color
|
|
rlm@171
|
312 channel's worth of visual information when called inside a running
|
|
rlm@171
|
313 simulation."
|
|
rlm@151
|
314 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
|
|
rlm@169
|
315 (let [retinal-map (retina-sensor-profile eye)
|
|
rlm@169
|
316 camera (add-eye! creature eye)
|
|
rlm@151
|
317 vision-image
|
|
rlm@151
|
318 (atom
|
|
rlm@151
|
319 (BufferedImage. (.getWidth camera)
|
|
rlm@151
|
320 (.getHeight camera)
|
|
rlm@170
|
321 BufferedImage/TYPE_BYTE_BINARY))
|
|
rlm@170
|
322 register-eye!
|
|
rlm@170
|
323 (runonce
|
|
rlm@170
|
324 (fn [world]
|
|
rlm@170
|
325 (add-camera!
|
|
rlm@170
|
326 world camera
|
|
rlm@170
|
327 (let [counter (atom 0)]
|
|
rlm@170
|
328 (fn [r fb bb bi]
|
|
rlm@170
|
329 (if (zero? (rem (swap! counter inc) (inc skip)))
|
|
rlm@170
|
330 (reset! vision-image
|
|
rlm@170
|
331 (BufferedImage! r fb bb bi))))))))]
|
|
rlm@151
|
332 (vec
|
|
rlm@151
|
333 (map
|
|
rlm@151
|
334 (fn [[key image]]
|
|
rlm@151
|
335 (let [whites (white-coordinates image)
|
|
rlm@151
|
336 topology (vec (collapse whites))
|
|
rlm@214
|
337 mask (color-channel-presets key key)]
|
|
rlm@170
|
338 (fn [world]
|
|
rlm@170
|
339 (register-eye! world)
|
|
rlm@151
|
340 (vector
|
|
rlm@151
|
341 topology
|
|
rlm@151
|
342 (vec
|
|
rlm@151
|
343 (for [[x y] whites]
|
|
rlm@151
|
344 (bit-and
|
|
rlm@151
|
345 mask (.getRGB @vision-image x y))))))))
|
|
rlm@170
|
346 retinal-map))))
|
|
rlm@151
|
347
|
|
rlm@170
|
348
|
|
rlm@170
|
349 ;; TODO maybe should add a viewport-manipulation function to
|
|
rlm@170
|
350 ;; automatically change viewport settings, attach shadow filters, etc.
|
|
rlm@170
|
351
|
|
rlm@170
|
352 (defn vision!
|
|
rlm@170
|
353 "Returns a function which returns visual sensory data when called
|
|
rlm@170
|
354 inside a running simulation"
|
|
rlm@151
|
355 [#^Node creature & {skip :skip :or {skip 0}}]
|
|
rlm@151
|
356 (reduce
|
|
rlm@170
|
357 concat
|
|
rlm@167
|
358 (for [eye (eyes creature)]
|
|
rlm@169
|
359 (vision-fn creature eye))))
|
|
rlm@151
|
360
|
|
rlm@189
|
361 (defn view-vision
|
|
rlm@189
|
362 "Creates a function which accepts a list of visual sensor-data and
|
|
rlm@189
|
363 displays each element of the list to the screen."
|
|
rlm@189
|
364 []
|
|
rlm@188
|
365 (view-sense
|
|
rlm@188
|
366 (fn
|
|
rlm@188
|
367 [[coords sensor-data]]
|
|
rlm@188
|
368 (let [image (points->image coords)]
|
|
rlm@188
|
369 (dorun
|
|
rlm@188
|
370 (for [i (range (count coords))]
|
|
rlm@188
|
371 (.setRGB image ((coords i) 0) ((coords i) 1)
|
|
rlm@188
|
372 (sensor-data i))))
|
|
rlm@189
|
373 image))))
|
|
rlm@188
|
374
|
|
rlm@34
|
375 #+end_src
|
|
rlm@23
|
376
|
|
rlm@112
|
377
|
|
rlm@34
|
378 Note the use of continuation passing style for connecting the eye to a
|
|
rlm@34
|
379 function to process the output. You can create any number of eyes, and
|
|
rlm@34
|
380 each of them will see the world from their own =Camera=. Once every
|
|
rlm@34
|
381 frame, the rendered image is copied to a =BufferedImage=, and that
|
|
rlm@34
|
382 data is sent off to the continuation function. Moving the =Camera=
|
|
rlm@34
|
383 which was used to create the eye will change what the eye sees.
|
|
rlm@23
|
384
|
|
rlm@34
|
385 * Example
|
|
rlm@23
|
386
|
|
rlm@66
|
387 #+name: test-vision
|
|
rlm@23
|
388 #+begin_src clojure
|
|
rlm@68
|
389 (ns cortex.test.vision
|
|
rlm@34
|
390 (:use (cortex world util vision))
|
|
rlm@34
|
391 (:import java.awt.image.BufferedImage)
|
|
rlm@34
|
392 (:import javax.swing.JPanel)
|
|
rlm@34
|
393 (:import javax.swing.SwingUtilities)
|
|
rlm@34
|
394 (:import java.awt.Dimension)
|
|
rlm@34
|
395 (:import javax.swing.JFrame)
|
|
rlm@34
|
396 (:import com.jme3.math.ColorRGBA)
|
|
rlm@45
|
397 (:import com.jme3.scene.Node)
|
|
rlm@113
|
398 (:import com.jme3.math.Vector3f))
|
|
rlm@23
|
399
|
|
rlm@36
|
400 (defn test-two-eyes
|
|
rlm@69
|
401 "Testing vision:
|
|
rlm@69
|
402 Tests the vision system by creating two views of the same rotating
|
|
rlm@69
|
403 object from different angles and displaying both of those views in
|
|
rlm@69
|
404 JFrames.
|
|
rlm@69
|
405
|
|
rlm@69
|
406 You should see a rotating cube, and two windows,
|
|
rlm@69
|
407 each displaying a different view of the cube."
|
|
rlm@36
|
408 []
|
|
rlm@58
|
409 (let [candy
|
|
rlm@58
|
410 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
|
|
rlm@112
|
411 (world
|
|
rlm@112
|
412 (doto (Node.)
|
|
rlm@112
|
413 (.attachChild candy))
|
|
rlm@112
|
414 {}
|
|
rlm@112
|
415 (fn [world]
|
|
rlm@112
|
416 (let [cam (.clone (.getCamera world))
|
|
rlm@112
|
417 width (.getWidth cam)
|
|
rlm@112
|
418 height (.getHeight cam)]
|
|
rlm@169
|
419 (add-camera! world cam
|
|
rlm@113
|
420 ;;no-op
|
|
rlm@113
|
421 (comp (view-image) BufferedImage!)
|
|
rlm@112
|
422 )
|
|
rlm@169
|
423 (add-camera! world
|
|
rlm@112
|
424 (doto (.clone cam)
|
|
rlm@112
|
425 (.setLocation (Vector3f. -10 0 0))
|
|
rlm@112
|
426 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
|
|
rlm@113
|
427 ;;no-op
|
|
rlm@113
|
428 (comp (view-image) BufferedImage!))
|
|
rlm@112
|
429 ;; This is here to restore the main view
|
|
rlm@112
|
430 ;; after the other views have completed processing
|
|
rlm@169
|
431 (add-camera! world (.getCamera world) no-op)))
|
|
rlm@112
|
432 (fn [world tpf]
|
|
rlm@112
|
433 (.rotate candy (* tpf 0.2) 0 0)))))
|
|
rlm@23
|
434 #+end_src
|
|
rlm@23
|
435
|
|
rlm@213
|
436 #+name: vision-header
|
|
rlm@213
|
437 #+begin_src clojure
|
|
rlm@213
|
438 (ns cortex.vision
|
|
rlm@213
|
439 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
|
|
rlm@213
|
440 eyes from different positions to observe the same world, and pass
|
|
rlm@213
|
441 the observed data to any arbitray function. Automatically reads
|
|
rlm@213
|
442 eye-nodes from specially prepared blender files and instanttiates
|
|
rlm@213
|
443 them in the world as actual eyes."
|
|
rlm@213
|
444 {:author "Robert McIntyre"}
|
|
rlm@213
|
445 (:use (cortex world sense util))
|
|
rlm@213
|
446 (:use clojure.contrib.def)
|
|
rlm@213
|
447 (:import com.jme3.post.SceneProcessor)
|
|
rlm@213
|
448 (:import (com.jme3.util BufferUtils Screenshots))
|
|
rlm@213
|
449 (:import java.nio.ByteBuffer)
|
|
rlm@213
|
450 (:import java.awt.image.BufferedImage)
|
|
rlm@213
|
451 (:import (com.jme3.renderer ViewPort Camera))
|
|
rlm@213
|
452 (:import com.jme3.math.ColorRGBA)
|
|
rlm@213
|
453 (:import com.jme3.renderer.Renderer)
|
|
rlm@213
|
454 (:import com.jme3.app.Application)
|
|
rlm@213
|
455 (:import com.jme3.texture.FrameBuffer)
|
|
rlm@213
|
456 (:import (com.jme3.scene Node Spatial)))
|
|
rlm@213
|
457 #+end_src
|
|
rlm@112
|
458
|
|
rlm@34
|
459 The example code will create two videos of the same rotating object
|
|
rlm@34
|
460 from different angles. It can be used both for stereoscopic vision
|
|
rlm@34
|
461 simulation or for simulating multiple creatures, each with their own
|
|
rlm@34
|
462 sense of vision.
|
|
rlm@24
|
463
|
|
rlm@35
|
464 - As a neat bonus, this idea behind simulated vision also enables one
|
|
rlm@35
|
465 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
|
|
rlm@35
|
466
|
|
rlm@24
|
467
|
|
rlm@212
|
468 * COMMENT Generate Source
|
|
rlm@34
|
469 #+begin_src clojure :tangle ../src/cortex/vision.clj
|
|
rlm@24
|
470 <<eyes>>
|
|
rlm@24
|
471 #+end_src
|
|
rlm@24
|
472
|
|
rlm@68
|
473 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
|
|
rlm@24
|
474 <<test-vision>>
|
|
rlm@24
|
475 #+end_src
|
