Mercurial > cortex

comparison org/vision.org @ 306:7e7f8d6d9ec5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
massive spellchecking
author Robert McIntyre <rlm@mit.edu>
date Sat, 18 Feb 2012 10:59:41 -0700
parents 23aadf376e9d
children 5d448182c807
comparison
equal deleted inserted replaced
305:19c43ec6958d 306:7e7f8d6d9ec5
9 9
10 * JMonkeyEngine natively supports multiple views of the same world. 10 * JMonkeyEngine natively supports multiple views of the same world.
11 11
12 Vision is one of the most important senses for humans, so I need to 12 Vision is one of the most important senses for humans, so I need to
13 build a simulated sense of vision for my AI. I will do this with 13 build a simulated sense of vision for my AI. I will do this with
14 simulated eyes. Each eye can be independely moved and should see its 14 simulated eyes. Each eye can be independently moved and should see its
15 own version of the world depending on where it is. 15 own version of the world depending on where it is.
16 16
17 Making these simulated eyes a reality is simple bacause jMonkeyEngine 17 Making these simulated eyes a reality is simple because jMonkeyEngine
18 already conatains extensive support for multiple views of the same 3D 18 already contains extensive support for multiple views of the same 3D
19 simulated world. The reason jMonkeyEngine has this support is because 19 simulated world. The reason jMonkeyEngine has this support is because
20 the support is necessary to create games with split-screen 20 the support is necessary to create games with split-screen
21 views. Multiple views are also used to create efficient 21 views. Multiple views are also used to create efficient
22 pseudo-reflections by rendering the scene from a certain perspective 22 pseudo-reflections by rendering the scene from a certain perspective
23 and then projecting it back onto a surface in the 3D world. 23 and then projecting it back onto a surface in the 3D world.
24 24
25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views. 25 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.
26 [[../images/goldeneye-4-player.png]] 26 [[../images/goldeneye-4-player.png]]
27 27
28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=. 28 ** =ViewPorts=, =SceneProcessors=, and the =RenderManager=.
29 # =Viewports= are cameras; =RenderManger= takes snapshots each frame. 29 # =ViewPorts= are cameras; =RenderManger= takes snapshots each frame.
30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline 30 #* A Brief Description of jMonkeyEngine's Rendering Pipeline
31 31
32 jMonkeyEngine allows you to create a =ViewPort=, which represents a 32 jMonkeyEngine allows you to create a =ViewPort=, which represents a
33 view of the simulated world. You can create as many of these as you 33 view of the simulated world. You can create as many of these as you
34 want. Every frame, the =RenderManager= iterates through each 34 want. Every frame, the =RenderManager= iterates through each
35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there 35 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
36 is a =FrameBuffer= which represents the rendered image in the GPU. 36 is a =FrameBuffer= which represents the rendered image in the GPU.
37 37
38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =Rendermanager= records a snapshot of what each view is currently seeing; these snapshots are =Framebuffer= objects. 38 #+caption: =ViewPorts= are cameras in the world. During each frame, the =RenderManager= records a snapshot of what each view is currently seeing; these snapshots are =FrameBuffer= objects.
39 #+ATTR_HTML: width="400" 39 #+ATTR_HTML: width="400"
40 [[../images/diagram_rendermanager2.png]] 40 [[../images/diagram_rendermanager2.png]]
41 41
42 Each =ViewPort= can have any number of attached =SceneProcessor= 42 Each =ViewPort= can have any number of attached =SceneProcessor=
43 objects, which are called every time a new frame is rendered. A 43 objects, which are called every time a new frame is rendered. A
44 =SceneProcessor= recieves its =ViewPort's= =FrameBuffer= and can do 44 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
45 whatever it wants to the data. Often this consists of invoking GPU 45 whatever it wants to the data. Often this consists of invoking GPU
46 specific operations on the rendered image. The =SceneProcessor= can 46 specific operations on the rendered image. The =SceneProcessor= can
47 also copy the GPU image data to RAM and process it with the CPU. 47 also copy the GPU image data to RAM and process it with the CPU.
48 48
49 ** From Views to Vision 49 ** From Views to Vision
50 # Appropriating Views for Vision. 50 # Appropriating Views for Vision.
51 51
52 Each eye in the simulated creature needs its own =ViewPort= so that 52 Each eye in the simulated creature needs its own =ViewPort= so that
53 it can see the world from its own perspective. To this =ViewPort=, I 53 it can see the world from its own perspective. To this =ViewPort=, I
54 add a =SceneProcessor= that feeds the visual data to any arbitray 54 add a =SceneProcessor= that feeds the visual data to any arbitrary
55 continuation function for further processing. That continuation 55 continuation function for further processing. That continuation
56 function may perform both CPU and GPU operations on the data. To make 56 function may perform both CPU and GPU operations on the data. To make
57 this easy for the continuation function, the =SceneProcessor= 57 this easy for the continuation function, the =SceneProcessor=
58 maintains appropriatly sized buffers in RAM to hold the data. It does 58 maintains appropriately sized buffers in RAM to hold the data. It does
59 not do any copying from the GPU to the CPU itself because it is a slow 59 not do any copying from the GPU to the CPU itself because it is a slow
60 operation. 60 operation.
61 61
62 #+name: pipeline-1 62 #+name: pipeline-1
63 #+begin_src clojure 63 #+begin_src clojure
64 (defn vision-pipeline 64 (defn vision-pipeline
65 "Create a SceneProcessor object which wraps a vision processing 65 "Create a SceneProcessor object which wraps a vision processing
66 continuation function. The continuation is a function that takes 66 continuation function. The continuation is a function that takes
67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi], 67 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
68 each of which has already been appropiately sized." 68 each of which has already been appropriately sized."
69 [continuation] 69 [continuation]
70 (let [byte-buffer (atom nil) 70 (let [byte-buffer (atom nil)
71 renderer (atom nil) 71 renderer (atom nil)
72 image (atom nil)] 72 image (atom nil)]
73 (proxy [SceneProcessor] [] 73 (proxy [SceneProcessor] []
97 The continuation function given to =vision-pipeline= above will be 97 The continuation function given to =vision-pipeline= above will be
98 given a =Renderer= and three containers for image data. The 98 given a =Renderer= and three containers for image data. The
99 =FrameBuffer= references the GPU image data, but the pixel data can 99 =FrameBuffer= references the GPU image data, but the pixel data can
100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage= 100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
101 are initially "empty" but are sized to hold the data in the 101 are initially "empty" but are sized to hold the data in the
102 =FrameBuffer=. I call transfering the GPU image data to the CPU 102 =FrameBuffer=. I call transferring the GPU image data to the CPU
103 structures "mixing" the image data. I have provided three functions to 103 structures "mixing" the image data. I have provided three functions to
104 do this mixing. 104 do this mixing.
105 105
106 #+name: pipeline-2 106 #+name: pipeline-2
107 #+begin_src clojure 107 #+begin_src clojure
127 127
128 Note that it is possible to write vision processing algorithms 128 Note that it is possible to write vision processing algorithms
129 entirely in terms of =BufferedImage= inputs. Just compose that 129 entirely in terms of =BufferedImage= inputs. Just compose that
130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision 130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision
131 processing algorithm that is entirely hosted on the GPU does not have 131 processing algorithm that is entirely hosted on the GPU does not have
132 to pay for this convienence. 132 to pay for this convenience.
133 133
134 * Optical sensor arrays are described with images and referenced with metadata 134 * Optical sensor arrays are described with images and referenced with metadata
135 The vision pipeline described above handles the flow of rendered 135 The vision pipeline described above handles the flow of rendered
136 images. Now, we need simulated eyes to serve as the source of these 136 images. Now, we need simulated eyes to serve as the source of these
137 images. 137 images.
138 138
139 An eye is described in blender in the same way as a joint. They are 139 An eye is described in blender in the same way as a joint. They are
140 zero dimensional empty objects with no geometry whose local coordinate 140 zero dimensional empty objects with no geometry whose local coordinate
141 system determines the orientation of the resulting eye. All eyes are 141 system determines the orientation of the resulting eye. All eyes are
142 childern of a parent node named "eyes" just as all joints have a 142 children of a parent node named "eyes" just as all joints have a
143 parent named "joints". An eye binds to the nearest physical object 143 parent named "joints". An eye binds to the nearest physical object
144 with =bind-sense=. 144 with =bind-sense=.
145 145
146 #+name: add-eye 146 #+name: add-eye
147 #+begin_src clojure 147 #+begin_src clojure
182 color sensors, and a blind spot which has no sensors at all. Sensor 182 color sensors, and a blind spot which has no sensors at all. Sensor
183 density decreases in proportion to distance from the fovea. 183 density decreases in proportion to distance from the fovea.
184 184
185 I want to be able to model any retinal configuration, so my eye-nodes 185 I want to be able to model any retinal configuration, so my eye-nodes
186 in blender contain metadata pointing to images that describe the 186 in blender contain metadata pointing to images that describe the
187 percise position of the individual sensors using white pixels. The 187 precise position of the individual sensors using white pixels. The
188 meta-data also describes the percise sensitivity to light that the 188 meta-data also describes the precise sensitivity to light that the
189 sensors described in the image have. An eye can contain any number of 189 sensors described in the image have. An eye can contain any number of
190 these images. For example, the metadata for an eye might look like 190 these images. For example, the metadata for an eye might look like
191 this: 191 this:
192 192
193 #+begin_src clojure 193 #+begin_src clojure
213 213
214 The numbers that serve as keys in the map determine a sensor's 214 The numbers that serve as keys in the map determine a sensor's
215 relative sensitivity to the channels red, green, and blue. These 215 relative sensitivity to the channels red, green, and blue. These
216 sensitivity values are packed into an integer in the order =|_|R|G|B|= 216 sensitivity values are packed into an integer in the order =|_|R|G|B|=
217 in 8-bit fields. The RGB values of a pixel in the image are added 217 in 8-bit fields. The RGB values of a pixel in the image are added
218 together with these sensitivities as linear weights. Therfore, 218 together with these sensitivities as linear weights. Therefore,
219 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to 219 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to
220 all colors equally (gray). 220 all colors equally (gray).
221 221
222 For convienence I've defined a few symbols for the more common 222 For convenience I've defined a few symbols for the more common
223 sensitivity values. 223 sensitivity values.
224 224
225 #+name: sensitivity 225 #+name: sensitivity
226 #+begin_src clojure 226 #+begin_src clojure
227 (defvar sensitivity-presets 227 (defvar sensitivity-presets
381 Note that since each of the functions generated by =vision-kernel= 381 Note that since each of the functions generated by =vision-kernel=
382 shares the same =register-eye!= function, the eye will be registered 382 shares the same =register-eye!= function, the eye will be registered
383 only once the first time any of the functions from the list returned 383 only once the first time any of the functions from the list returned
384 by =vision-kernel= is called. Each of the functions returned by 384 by =vision-kernel= is called. Each of the functions returned by
385 =vision-kernel= also allows access to the =Viewport= through which 385 =vision-kernel= also allows access to the =Viewport= through which
386 it recieves images. 386 it receives images.
387 387
388 The in-game display can be disrupted by all the viewports that the 388 The in-game display can be disrupted by all the ViewPorts that the
389 functions greated by =vision-kernel= add. This doesn't affect the 389 functions generated by =vision-kernel= add. This doesn't affect the
390 simulation or the simulated senses, but can be annoying. 390 simulation or the simulated senses, but can be annoying.
391 =gen-fix-display= restores the in-simulation display. 391 =gen-fix-display= restores the in-simulation display.
392 392
393 ** The =vision!= function creates sensory probes. 393 ** The =vision!= function creates sensory probes.
394 394
570 "key-g" (colored-cannon-ball ColorRGBA/Green)) 570 "key-g" (colored-cannon-ball ColorRGBA/Green))
571 (fn [world] 571 (fn [world]
572 (light-up-everything world) 572 (light-up-everything world)
573 (speed-up world) 573 (speed-up world)
574 (.setTimer world timer) 574 (.setTimer world timer)
575 (display-dialated-time world timer) 575 (display-dilated-time world timer)
576 ;; add a view from the worm's perspective 576 ;; add a view from the worm's perspective
577 (if record? 577 (if record?
578 (Capture/captureVideo 578 (Capture/captureVideo
579 world 579 world
580 (File. 580 (File.
683 #+name: vision-header 683 #+name: vision-header
684 #+begin_src clojure 684 #+begin_src clojure
685 (ns cortex.vision 685 (ns cortex.vision
686 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple 686 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
687 eyes from different positions to observe the same world, and pass 687 eyes from different positions to observe the same world, and pass
688 the observed data to any arbitray function. Automatically reads 688 the observed data to any arbitrary function. Automatically reads
689 eye-nodes from specially prepared blender files and instantiates 689 eye-nodes from specially prepared blender files and instantiates
690 them in the world as actual eyes." 690 them in the world as actual eyes."
691 {:author "Robert McIntyre"} 691 {:author "Robert McIntyre"}
692 (:use (cortex world sense util)) 692 (:use (cortex world sense util))
693 (:use clojure.contrib.def) 693 (:use clojure.contrib.def)