Mercurial > cortex

comparison org/vision.org @ 273:c39b8b29a79e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixed ambigous in-text function references
author Robert McIntyre <rlm@mit.edu>
date Wed, 15 Feb 2012 06:56:47 -0700
parents aa3641042958
children dbecd276b51a
comparison
equal deleted inserted replaced
271:5833b4ce877a 273:c39b8b29a79e
4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure 4 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure
5 #+keywords: computer vision, jMonkeyEngine3, clojure 5 #+keywords: computer vision, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org 6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org 7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+babel: :mkdirp yes :noweb yes :exports both 8 #+babel: :mkdirp yes :noweb yes :exports both
9
10 # SUGGEST: Call functions by their name, without
11 # parentheses. e.g. =add-eye!=, not =(add-eye!)=. The reason for this
12 # is that it is potentially easy to confuse the /function/ =f= with its
13 # /value/ at a particular point =(f x)=. Mathematicians have this
14 # problem with their notation; we don't need it in ours.
15 9
16 * JMonkeyEngine natively supports multiple views of the same world. 10 * JMonkeyEngine natively supports multiple views of the same world.
17 11
18 Vision is one of the most important senses for humans, so I need to 12 Vision is one of the most important senses for humans, so I need to
19 build a simulated sense of vision for my AI. I will do this with 13 build a simulated sense of vision for my AI. I will do this with
98 (.clear @byte-buffer) 92 (.clear @byte-buffer)
99 (continuation @renderer fb @byte-buffer @image)) 93 (continuation @renderer fb @byte-buffer @image))
100 (cleanup [])))) 94 (cleanup []))))
101 #+end_src 95 #+end_src
102 96
103 The continuation function given to =(vision-pipeline)= above will be 97 The continuation function given to =vision-pipeline= above will be
104 given a =Renderer= and three containers for image data. The 98 given a =Renderer= and three containers for image data. The
105 =FrameBuffer= references the GPU image data, but the pixel data can 99 =FrameBuffer= references the GPU image data, but the pixel data can
106 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage= 100 not be used directly on the CPU. The =ByteBuffer= and =BufferedImage=
107 are initially "empty" but are sized to hold the data in the 101 are initially "empty" but are sized to hold the data in the
108 =FrameBuffer=. I call transfering the GPU image data to the CPU 102 =FrameBuffer=. I call transfering the GPU image data to the CPU
131 (frameBuffer->byteBuffer! r fb bb) bi)) 125 (frameBuffer->byteBuffer! r fb bb) bi))
132 #+end_src 126 #+end_src
133 127
134 Note that it is possible to write vision processing algorithms 128 Note that it is possible to write vision processing algorithms
135 entirely in terms of =BufferedImage= inputs. Just compose that 129 entirely in terms of =BufferedImage= inputs. Just compose that
136 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision 130 =BufferedImage= algorithm with =BufferedImage!=. However, a vision
137 processing algorithm that is entirely hosted on the GPU does not have 131 processing algorithm that is entirely hosted on the GPU does not have
138 to pay for this convienence. 132 to pay for this convienence.
139 133
140 * Optical sensor arrays are described with images and referenced with metadata 134 * Optical sensor arrays are described with images and referenced with metadata
141 The vision pipeline described above handles the flow of rendered 135 The vision pipeline described above handles the flow of rendered
145 An eye is described in blender in the same way as a joint. They are 139 An eye is described in blender in the same way as a joint. They are
146 zero dimensional empty objects with no geometry whose local coordinate 140 zero dimensional empty objects with no geometry whose local coordinate
147 system determines the orientation of the resulting eye. All eyes are 141 system determines the orientation of the resulting eye. All eyes are
148 childern of a parent node named "eyes" just as all joints have a 142 childern of a parent node named "eyes" just as all joints have a
149 parent named "joints". An eye binds to the nearest physical object 143 parent named "joints". An eye binds to the nearest physical object
150 with =(bind-sense=). 144 with =bind-sense=.
151 145
152 #+name: add-eye 146 #+name: add-eye
153 #+begin_src clojure 147 #+begin_src clojure
154 (in-ns 'cortex.vision) 148 (in-ns 'cortex.vision)
155 149
174 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000) 168 cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
175 (bind-sense target cam) cam)) 169 (bind-sense target cam) cam))
176 #+end_src 170 #+end_src
177 171
178 Here, the camera is created based on metadata on the eye-node and 172 Here, the camera is created based on metadata on the eye-node and
179 attached to the nearest physical object with =(bind-sense)= 173 attached to the nearest physical object with =bind-sense=
180 ** The Retina 174 ** The Retina
181 175
182 An eye is a surface (the retina) which contains many discrete sensors 176 An eye is a surface (the retina) which contains many discrete sensors
183 to detect light. These sensors have can have different light-sensing 177 to detect light. These sensors have can have different light-sensing
184 properties. In humans, each discrete sensor is sensitive to red, 178 properties. In humans, each discrete sensor is sensitive to red,
239 (:red :blue :green) or average all channels (:all)") 233 (:red :blue :green) or average all channels (:all)")
240 #+end_src 234 #+end_src
241 235
242 ** Metadata Processing 236 ** Metadata Processing
243 237
244 =(retina-sensor-profile)= extracts a map from the eye-node in the same 238 =retina-sensor-profile= extracts a map from the eye-node in the same
245 format as the example maps above. =(eye-dimensions)= finds the 239 format as the example maps above. =eye-dimensions= finds the
246 dimensions of the smallest image required to contain all the retinal 240 dimensions of the smallest image required to contain all the retinal
247 sensor maps. 241 sensor maps.
248 242
249 #+name: retina 243 #+name: retina
250 #+begin_src clojure 244 #+begin_src clojure
279 eyes 273 eyes
280 (sense-nodes "eyes") 274 (sense-nodes "eyes")
281 "Return the children of the creature's \"eyes\" node.") 275 "Return the children of the creature's \"eyes\" node.")
282 #+end_src 276 #+end_src
283 277
284 Then, add the camera created by =(add-eye!)= to the simulation by 278 Then, add the camera created by =add-eye!= to the simulation by
285 creating a new viewport. 279 creating a new viewport.
286 280
287 #+name: add-camera 281 #+name: add-camera
288 #+begin_src clojure 282 #+begin_src clojure
289 (defn add-camera! 283 (defn add-camera!
305 The eye's continuation function should register the viewport with the 299 The eye's continuation function should register the viewport with the
306 simulation the first time it is called, use the CPU to extract the 300 simulation the first time it is called, use the CPU to extract the
307 appropriate pixels from the rendered image and weight them by each 301 appropriate pixels from the rendered image and weight them by each
308 sensor's sensitivity. I have the option to do this processing in 302 sensor's sensitivity. I have the option to do this processing in
309 native code for a slight gain in speed. I could also do it in the GPU 303 native code for a slight gain in speed. I could also do it in the GPU
310 for a massive gain in speed. =(vision-kernel)= generates a list of 304 for a massive gain in speed. =vision-kernel= generates a list of
311 such continuation functions, one for each channel of the eye. 305 such continuation functions, one for each channel of the eye.
312 306
313 #+name: kernel 307 #+name: kernel
314 #+begin_src clojure 308 #+begin_src clojure
315 (in-ns 'cortex.vision) 309 (in-ns 'cortex.vision)
382 (runonce 376 (runonce
383 (fn [world] 377 (fn [world]
384 (add-camera! world (.getCamera world) no-op)))) 378 (add-camera! world (.getCamera world) no-op))))
385 #+end_src 379 #+end_src
386 380
387 Note that since each of the functions generated by =(vision-kernel)= 381 Note that since each of the functions generated by =vision-kernel=
388 shares the same =(register-eye!)= function, the eye will be registered 382 shares the same =register-eye!= function, the eye will be registered
389 only once the first time any of the functions from the list returned 383 only once the first time any of the functions from the list returned
390 by =(vision-kernel)= is called. Each of the functions returned by 384 by =vision-kernel= is called. Each of the functions returned by
391 =(vision-kernel)= also allows access to the =Viewport= through which 385 =vision-kernel= also allows access to the =Viewport= through which
392 it recieves images. 386 it recieves images.
393 387
394 The in-game display can be disrupted by all the viewports that the 388 The in-game display can be disrupted by all the viewports that the
395 functions greated by =(vision-kernel)= add. This doesn't affect the 389 functions greated by =vision-kernel= add. This doesn't affect the
396 simulation or the simulated senses, but can be annoying. 390 simulation or the simulated senses, but can be annoying.
397 =(gen-fix-display)= restores the in-simulation display. 391 =gen-fix-display= restores the in-simulation display.
398 392
399 ** The =vision!= function creates sensory probes. 393 ** The =vision!= function creates sensory probes.
400 394
401 All the hard work has been done; all that remains is to apply 395 All the hard work has been done; all that remains is to apply
402 =(vision-kernel)= to each eye in the creature and gather the results 396 =vision-kernel= to each eye in the creature and gather the results
403 into one list of functions. 397 into one list of functions.
404 398
405 #+name: main 399 #+name: main
406 #+begin_src clojure 400 #+begin_src clojure
407 (defn vision! 401 (defn vision!
415 #+end_src 409 #+end_src
416 410
417 ** Displaying visual data for debugging. 411 ** Displaying visual data for debugging.
418 # Visualization of Vision. Maybe less alliteration would be better. 412 # Visualization of Vision. Maybe less alliteration would be better.
419 It's vital to have a visual representation for each sense. Here I use 413 It's vital to have a visual representation for each sense. Here I use
420 =(view-sense)= to construct a function that will create a display for 414 =view-sense= to construct a function that will create a display for
421 visual data. 415 visual data.
422 416
423 #+name: display 417 #+name: display
424 #+begin_src clojure 418 #+begin_src clojure
425 (in-ns 'cortex.vision) 419 (in-ns 'cortex.vision)
675 669
676 * Onward! 670 * Onward!
677 - As a neat bonus, this idea behind simulated vision also enables one 671 - As a neat bonus, this idea behind simulated vision also enables one
678 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]]. 672 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
679 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]]. 673 - Now that we have vision, it's time to tackle [[./hearing.org][hearing]].
680
681
682 #+appendix 674 #+appendix
683 675
684 * Headers 676 * Headers
685 677
686 #+name: vision-header 678 #+name: vision-header
729 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]] 721 - [[../assets/Models/subtitles/worm-vision-subtitles.blend][worm-vision-subtitles.blend]]
730 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul> 722 #+html: <ul> <li> <a href="../org/sense.org">This org file</a> </li> </ul>
731 - [[http://hg.bortreb.com ][source-repository]] 723 - [[http://hg.bortreb.com ][source-repository]]
732 724
733 725
726 * Next
727 I find some [[./hearing.org][ears]] for the creature while exploring the guts of
728 jMonkeyEngine's sound system.
734 729
735 * COMMENT Generate Source 730 * COMMENT Generate Source
736 #+begin_src clojure :tangle ../src/cortex/vision.clj 731 #+begin_src clojure :tangle ../src/cortex/vision.clj
737 <<vision-header>> 732 <<vision-header>>
738 <<pipeline-1>> 733 <<pipeline-1>>