#+title: Simulated Sense of Sight

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Simulated sight for AI research using JMonkeyEngine3 and clojure

 #+keywords: computer vision, jMonkeyEngine3, clojure

 #+SETUPFILE: ../../aurellem/org/setup.org

 #+INCLUDE: ../../aurellem/org/level-0.org

 #+babel: :mkdirp yes :noweb yes :exports both

 

 * Vision

 

 

 Vision is one of the most important senses for humans, so I need to

 build a simulated sense of vision for my AI. I will do this with

 simulated eyes. Each eye can be independely moved and should see its

 own version of the world depending on where it is.

 

 Making these simulated eyes a reality is fairly simple bacause

 jMonkeyEngine already conatains extensive support for multiple views

 of the same 3D simulated world. The reason jMonkeyEngine has this

 support is because the support is necessary to create games with

 split-screen views. Multiple views are also used to create efficient

 pseudo-reflections by rendering the scene from a certain perspective

 and then projecting it back onto a surface in the 3D world.

 

 #+caption: jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye

 [[../images/goldeneye-4-player.png]]

 

 * Brief Description of jMonkeyEngine's Rendering Pipeline

 

 jMonkeyEngine allows you to create a =ViewPort=, which represents a

 view of the simulated world. You can create as many of these as you

 want. Every frame, the =RenderManager= iterates through each

 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there

 is a =FrameBuffer= which represents the rendered image in the GPU.

 

 Each =ViewPort= can have any number of attached =SceneProcessor=

 objects, which are called every time a new frame is rendered. A

 =SceneProcessor= recieves a =FrameBuffer= and can do whatever it wants

 to the data.  Often this consists of invoking GPU specific operations

 on the rendered image.  The =SceneProcessor= can also copy the GPU

 image data to RAM and process it with the CPU.

 

 * The Vision Pipeline

 

 Each eye in the simulated creature needs it's own =ViewPort= so that

 it can see the world from its own perspective. To this =ViewPort=, I

 add a =SceneProcessor= that feeds the visual data to any arbitray

 continuation function for further processing.  That continuation

 function may perform both CPU and GPU operations on the data. To make

 this easy for the continuation function, the =SceneProcessor=

 maintains appropriatly sized buffers in RAM to hold the data.  It does

 not do any copying from the GPU to the CPU itself.

 

 #+name: pipeline-1

 #+begin_src clojure

 (defn vision-pipeline

   "Create a SceneProcessor object which wraps a vision processing

   continuation function. The continuation is a function that takes 

   [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],

   each of which has already been appropiately sized."

   [continuation]

   (let [byte-buffer (atom nil)

 	renderer (atom nil)

         image (atom nil)]

   (proxy [SceneProcessor] []

     (initialize

      [renderManager viewPort]

      (let [cam (.getCamera viewPort)

 	   width (.getWidth cam)

 	   height (.getHeight cam)]

        (reset! renderer (.getRenderer renderManager))

        (reset! byte-buffer

 	     (BufferUtils/createByteBuffer

 	      (* width height 4)))

         (reset! image (BufferedImage.

                       width height

                       BufferedImage/TYPE_4BYTE_ABGR))))

     (isInitialized [] (not (nil? @byte-buffer)))

     (reshape [_ _ _])

     (preFrame [_])

     (postQueue [_])

     (postFrame

      [#^FrameBuffer fb]

      (.clear @byte-buffer)

      (continuation @renderer fb @byte-buffer @image))

     (cleanup []))))

 #+end_src

 

 The continuation function given to =(vision-pipeline)= above will be

 given a =Renderer= and three containers for image data. The

 =FrameBuffer= references the GPU image data, but it can not be used

 directly on the CPU.  The =ByteBuffer= and =BufferedImage= are

 initially "empty" but are sized to hold to data in the

 =FrameBuffer=. I call transfering the GPU image data to the CPU

 structures "mixing" the image data. I have provided three functions to

 do this mixing.

 

 #+name: pipeline-2

 #+begin_src clojure

 (defn frameBuffer->byteBuffer!

   "Transfer the data in the graphics card (Renderer, FrameBuffer) to

    the CPU (ByteBuffer)."  

   [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb]

   (.readFrameBuffer r fb bb) bb)

 

 (defn byteBuffer->bufferedImage!

   "Convert the C-style BGRA image data in the ByteBuffer bb to the AWT

    style ABGR image data and place it in BufferedImage bi."

   [#^ByteBuffer bb #^BufferedImage bi]

   (Screenshots/convertScreenShot bb bi) bi)

 

 (defn BufferedImage!

   "Continuation which will grab the buffered image from the materials

    provided by (vision-pipeline)."

   [#^Renderer r #^FrameBuffer fb #^ByteBuffer bb #^BufferedImage bi]

   (byteBuffer->bufferedImage!

    (frameBuffer->byteBuffer! r fb bb) bi))

 #+end_src

 

 Note that it is possible to write vision processing algorithms

 entirely in terms of =BufferedImage= inputs. Just compose that

 =BufferedImage= algorithm with =(BufferedImage!)=. However, a vision

 processing algorithm that is entirely hosted on the GPU does not have

 to pay for this convienence.

 

 * COMMENT asdasd 

 

 (vision creature) will take an optional :skip argument which will

 inform the continuations in scene processor to skip the given

 number of cycles 0 means that no cycles will be skipped.

 

 (vision creature) will return [init-functions sensor-functions].

 The init-functions are each single-arg functions that take the

 world and register the cameras and must each be called before the

 corresponding sensor-functions.  Each init-function returns the

 viewport for that eye which can be manipulated, saved, etc. Each

 sensor-function is a thunk and will return data in the same

 format as the tactile-sensor functions the structure is

 [topology, sensor-data]. Internally, these sensor-functions

 maintain a reference to sensor-data which is periodically updated

 by the continuation function established by its init-function.

 They can be queried every cycle, but their information may not

 necessairly be different every cycle.

 

 

 

 * Physical Eyes

 

 The vision pipeline described above handles the flow of rendered

 images. Now, we need simulated eyes to serve as the source of these

 images. 

 

 An eye is described in blender in the same way as a joint. They are

 zero dimensional empty objects with no geometry whose local coordinate

 system determines the orientation of the resulting eye. All eyes are

 childern of a parent node named "eyes" just as all joints have a

 parent named "joints". An eye binds to the nearest physical object

 with =(bind-sense=).

 

 #+name: add-eye

 #+begin_src clojure

 (in-ns 'cortex.vision)

 

 (import com.jme3.math.Vector3f)

 

 (def blender-rotation-correction

   (doto (Quaternion.)

     (.fromRotationMatrix

      (doto (Matrix3f.)

        (.setColumn 0

                    (Vector3f. 1 0 0))

        (.setColumn 1

                    (Vector3f. 0 -1 0))

        (.setColumn 2

                    (Vector3f. 0 0 -1)))

 

      (doto (Matrix3f.)

        (.setColumn 0

                    (Vector3f. 

 

 

 (defn add-eye!

   "Create a Camera centered on the current position of 'eye which

    follows the closest physical node in 'creature and sends visual

    data to 'continuation. The camera will point in the X direction and

    use the Z vector as up as determined by the rotation of these

    vectors in blender coordinate space. Use XZY rotation for the node

    in blender."

   [#^Node creature #^Spatial eye]

   (let [target (closest-node creature eye)

         [cam-width cam-height] (eye-dimensions eye)

         cam (Camera. cam-width cam-height)

         rot (.getWorldRotation eye)]

     (.setLocation cam (.getWorldTranslation eye))

     (.lookAtDirection cam (.mult rot Vector3f/UNIT_X)

                       ;; this part is consistent with using Z in

                       ;; blender as the UP vector.

                       (.mult rot Vector3f/UNIT_Y))

 

     (println-repl "eye unit-z ->"  (.mult rot Vector3f/UNIT_Z))

     (println-repl "eye unit-y ->"  (.mult rot Vector3f/UNIT_Y))

     (println-repl "eye unit-x ->"  (.mult rot Vector3f/UNIT_X))

     (.setFrustumPerspective

      cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)

     (bind-sense target cam) cam))

 #+end_src

 

 Here, the camera is created based on metadata on the eye-node and

 attached to the nearest physical object with =(bind-sense)=

 

 

 ** The Retina

 

 An eye is a surface (the retina) which contains many discrete sensors

 to detect light. These sensors have can have different-light sensing

 properties.  In humans, each discrete sensor is sensitive to red,

 blue, green, or gray. These different types of sensors can have

 different spatial distributions along the retina. In humans, there is

 a fovea in the center of the retina which has a very high density of

 color sensors, and a blind spot which has no sensors at all. Sensor

 density decreases in proportion to distance from the retina.

 

 I want to be able to model any retinal configuration, so my eye-nodes

 in blender contain metadata pointing to images that describe the

 percise position of the individual sensors using white pixels. The

 meta-data also describes the percise sensitivity to light that the

 sensors described in the image have.  An eye can contain any number of

 these images. For example, the metadata for an eye might look like

 this:

 

 #+begin_src clojure

 {0xFF0000 "Models/test-creature/retina-small.png"}

 #+end_src

 

 #+caption: The retinal profile image "Models/test-creature/retina-small.png". White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.

 [[../assets/Models/test-creature/retina-small.png]]

 

 Together, the number 0xFF0000 and the image image above describe the

 placement of red-sensitive sensory elements.

 

 Meta-data to very crudely approximate a human eye might be something

 like this:

 

 #+begin_src clojure

 (let [retinal-profile "Models/test-creature/retina-small.png"]

   {0xFF0000 retinal-profile

    0x00FF00 retinal-profile

    0x0000FF retinal-profile

    0xFFFFFF retinal-profile})

 #+end_src

 

 The numbers that serve as keys in the map determine a sensor's

 relative sensitivity to the channels red, green, and blue. These

 sensitivity values are packed into an integer in the order _RGB in

 8-bit fields. The RGB values of a pixel in the image are added

 together with these sensitivities as linear weights. Therfore,

 0xFF0000 means sensitive to red only while 0xFFFFFF means sensitive to

 all colors equally (gray).

 

 For convienence I've defined a few symbols for the more common

 sensitivity values.

 

 #+name: sensitivity

 #+begin_src clojure

 (defvar sensitivity-presets

   {:all    0xFFFFFF

    :red    0xFF0000

    :blue   0x0000FF

    :green  0x00FF00}

   "Retinal sensitivity presets for sensors that extract one channel

    (:red :blue :green) or average all channels (:gray)")

 #+end_src

 

 ** Metadata Processing

 

 =(retina-sensor-profile)= extracts a map from the eye-node in the same

 format as the example maps above.  =(eye-dimensions)= finds the

 dimansions of the smallest image required to contain all the retinal

 sensor maps.

 

 #+begin_src clojure

 (defn retina-sensor-profile

   "Return a map of pixel sensitivity numbers to BufferedImages

    describing the distribution of light-sensitive components of this

    eye. :red, :green, :blue, :gray are already defined as extracting

    the red, green, blue, and average components respectively."

    [#^Spatial eye]

    (if-let [eye-map (meta-data eye "eye")]

      (map-vals

       load-image

       (eval (read-string eye-map)))))

 

 (defn eye-dimensions

   "Returns [width, height] specified in the metadata of the eye"

   [#^Spatial eye]

   (let [dimensions

           (map #(vector (.getWidth %) (.getHeight %))

                (vals (retina-sensor-profile eye)))]

     [(apply max (map first dimensions))

      (apply max (map second dimensions))]))

 #+end_src

 

 

 * Eye Creation 

 

 First off, get the children of the "eyes" empty node to find all the

 eyes the creature has.

 

 #+begin_src clojure

 (defvar 

   ^{:arglists '([creature])}

   eyes

   (sense-nodes "eyes")

   "Return the children of the creature's \"eyes\" node.")

 #+end_src

 

 Then, add the camera created by =(add-eye!)= to the simulation by

 creating a new viewport.

 

 #+begin_src clojure

 (defn add-camera!

   "Add a camera to the world, calling continuation on every frame

   produced." 

   [#^Application world camera continuation]

   (let [width (.getWidth camera)

 	height (.getHeight camera)

 	render-manager (.getRenderManager world)

 	viewport (.createMainView render-manager "eye-view" camera)]

     (doto viewport

       (.setClearFlags true true true)

       (.setBackgroundColor ColorRGBA/Black)

       (.addProcessor (vision-pipeline continuation))

       (.attachScene (.getRootNode world)))))

 #+end_src

 

 

 The continuation function registers the viewport with the simulation

 the first time it is called, and uses the CPU to extract the

 appropriate pixels from the rendered image and weight them by each

 sensors sensitivity. I have the option to do this filtering in native

 code for a slight gain in speed. I could also do it in the GPU for a

 massive gain in speed. =(vision-kernel)= generates a list of such

 continuation functions, one for each channel of the eye.

 

 #+begin_src clojure 

 (in-ns 'cortex.vision)

 

 (defrecord attached-viewport [vision-fn viewport-fn]

   clojure.lang.IFn

   (invoke [this world] (vision-fn world))

   (applyTo [this args] (apply vision-fn args)))

 

 (defn vision-kernel

   "Returns a list of functions, each of which will return a color

    channel's worth of visual information when called inside a running

    simulation."

   [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]

   (let [retinal-map (retina-sensor-profile eye)

         camera (add-eye! creature eye)

         vision-image

         (atom

          (BufferedImage. (.getWidth camera)

                          (.getHeight camera)

                          BufferedImage/TYPE_BYTE_BINARY))

         register-eye!

         (runonce

          (fn [world]

            (add-camera!

             world camera

             (let [counter  (atom 0)]

               (fn [r fb bb bi]

                 (if (zero? (rem (swap! counter inc) (inc skip)))

                   (reset! vision-image

                           (BufferedImage! r fb bb bi))))))))]

      (vec

       (map

        (fn [[key image]]

          (let [whites (white-coordinates image)

                topology (vec (collapse whites))

                mask (color-channel-presets key key)]

            (attached-viewport.

             (fn [world]

               (register-eye! world)

               (vector

                topology

                (vec 

                 (for [[x y] whites]

                   (bit-and

                    mask (.getRGB @vision-image x y))))))

             register-eye!)))

          retinal-map))))

 

 (defn gen-fix-display

   "Create a function to call to restore a simulation's display when it

    is disrupted by a Viewport."

   []

   (runonce

    (fn [world]

      (add-camera! world (.getCamera world) no-op))))

 

 #+end_src

 

 Note that since each of the functions generated by =(vision-kernel)=

 shares the same =(register-eye!)= function, the eye will be registered

 only once the first time any of the functions from the list returned

 by =(vision-kernel)= is called.  Each of the functions returned by

 =(vision-kernel)= also allows access to the =Viewport= through which

 it recieves images.

 

 The in-game display can be disrupted by all the viewports that the

 functions greated by =(vision-kernel)= add. This doesn't affect the

 simulation or the simulated senses, but can be annoying.

 =(gen-fix-display)= restores the in-simulation display.

 

 ** Vision!

 

 All the hard work has been done, all that remains is to apply

 =(vision-kernel)= to each eye in the creature and gather the results

 into one list of functions.

 

 #+begin_src clojure

 (defn vision!

   "Returns a function which returns visual sensory data when called

    inside a running simulation"

   [#^Node creature & {skip :skip :or {skip 0}}]

   (reduce

    concat 

    (for [eye (eyes creature)]

      (vision-kernel creature eye))))

 #+end_src

 

 ** Visualization of Vision

 

 It's vital to have a visual representation for each sense. Here I use

 =(view-sense)= to construct a function that will create a display for

 visual data.

 

 #+begin_src clojure 

 (defn view-vision

   "Creates a function which accepts a list of visual sensor-data and

   displays each element of the list to the screen." 

   []

   (view-sense

    (fn 

      [[coords sensor-data]]

      (let [image (points->image coords)]

        (dorun

         (for [i (range (count coords))]

           (.setRGB image ((coords i) 0) ((coords i) 1)

                    (sensor-data i))))

        image))))

 #+end_src

 

 * Tests

 

 ** Basic Test

 

 This is a basic test for the vision system.  It only tests the

 vision-pipeline and does not deal with loadig eyes from a blender

 file. The code creates two videos of the same rotating cube from

 different angles. 

 

 #+name: test-1

 #+begin_src clojure

 (in-ns 'cortex.test.vision)

 

 (defn test-two-eyes

   "Testing vision:

    Tests the vision system by creating two views of the same rotating

    object from different angles and displaying both of those views in

    JFrames.

 

    You should see a rotating cube, and two windows,

    each displaying a different view of the cube."

   []

   (let [candy

         (box 1 1 1 :physical? false :color ColorRGBA/Blue)]

     (world

      (doto (Node.)

        (.attachChild candy))

      {}

      (fn [world]

        (let [cam (.clone (.getCamera world))

              width (.getWidth cam)

              height (.getHeight cam)]

          (add-camera! world cam 

                       (comp

                        (view-image

                         (File. "/home/r/proj/cortex/render/vision/1"))

                         BufferedImage!))

          (add-camera! world

                   (doto (.clone cam)

                     (.setLocation (Vector3f. -10 0 0))

                     (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))

                   (comp

                    (view-image

                     (File. "/home/r/proj/cortex/render/vision/2"))

                         BufferedImage!))

          ;; This is here to restore the main view

          ;; after the other views have completed processing

          (add-camera! world (.getCamera world) no-op)))

      (fn [world tpf]

        (.rotate candy (* tpf 0.2) 0 0)))))

 #+end_src

 

 #+begin_html

 <div class="figure">

 <video controls="controls" width="755">

   <source src="../video/spinning-cube.ogg" type="video/ogg"

 	  preload="none" poster="../images/aurellem-1280x480.png" />

 </video>

 <p>A rotating cube viewed from two different perspectives.</p>

 </div>

 #+end_html

 

 Creating multiple eyes like this can be used for stereoscopic vision

 simulation in a single creature or for simulating multiple creatures,

 each with their own sense of vision.

 

 ** Adding Vision to the Worm

 

 To the worm from the last post, we add a new node that describes its

 eyes.

 

 #+attr_html: width=755

 #+caption: The worm with newly added empty nodes describing a single eye.

 [[../images/worm-with-eye.png]]

 

 The node highlighted in yellow is the root level "eyes" node.  It has

 a single node, highlighted in orange, which describes a single

 eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint

 of the worm.

 

 The metadata of the eye-node is:

 

 #+begin_src clojure :results verbatim :exports both

 (cortex.sense/meta-data

  (.getChild

   (.getChild (cortex.test.body/worm)

             "eyes") "eye") "eye")

 #+end_src

 

 #+results:

 : "(let [retina \"Models/test-creature/retina-small.png\"]

 :     {:all retina :red retina :green retina :blue retina})"

 

 This is the approximation to the human eye described earlier.

 

 #+begin_src clojure

 (in-ns 'cortex.test.vision)

 

 (import com.aurellem.capture.Capture)

 

 (defn test-worm-vision [] 

   (let [the-worm (doto (worm)(body!))

         vision (vision! the-worm)

         vision-display (view-vision)

         fix-display (gen-fix-display)

         me (sphere 0.5 :color ColorRGBA/Blue :physical? false)

         x-axis

         (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red

              :position (Vector3f. 0 -5 0))

         y-axis

         (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green

              :position (Vector3f. 0 -5 0))

         z-axis

         (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue

              :position (Vector3f. 0 -5 0))]

 

     (world (nodify [(floor) the-worm x-axis y-axis z-axis me])

            standard-debug-controls

            (fn [world]

              (light-up-everything world)

              ;; add a view from the worm's perspective

              (add-camera!

               world

               (add-eye! the-worm

                         (.getChild 

                          (.getChild the-worm "eyes") "eye"))

               (comp

                (view-image

                 (File. "/home/r/proj/cortex/render/worm-vision/worm-view"))

                BufferedImage!))

              (set-gravity world Vector3f/ZERO)

              (Capture/captureVideo

               world

               (File. "/home/r/proj/cortex/render/worm-vision/main-view")))

            (fn [world _ ]

              (.setLocalTranslation me (.getLocation (.getCamera world)))

              (vision-display

               (map #(% world) vision)

               (File. "/home/r/proj/cortex/render/worm-vision"))

              (fix-display world)))))

 #+end_src

 

 * Headers

 

 #+name: vision-header

 #+begin_src clojure 

 (ns cortex.vision

   "Simulate the sense of vision in jMonkeyEngine3. Enables multiple

   eyes from different positions to observe the same world, and pass

   the observed data to any arbitray function. Automatically reads

   eye-nodes from specially prepared blender files and instanttiates

   them in the world as actual eyes."

   {:author "Robert McIntyre"}

   (:use (cortex world sense util))

   (:use clojure.contrib.def)

   (:import com.jme3.post.SceneProcessor)

   (:import (com.jme3.util BufferUtils Screenshots))

   (:import java.nio.ByteBuffer)

   (:import java.awt.image.BufferedImage)

   (:import (com.jme3.renderer ViewPort Camera))

   (:import com.jme3.math.ColorRGBA)

   (:import com.jme3.renderer.Renderer)

   (:import com.jme3.app.Application)

   (:import com.jme3.texture.FrameBuffer)

   (:import (com.jme3.scene Node Spatial)))

 #+end_src

 

 #+name: test-header

 #+begin_src clojure

 (ns cortex.test.vision

   (:use (cortex world sense util body vision))

   (:use cortex.test.body)

   (:import java.awt.image.BufferedImage)

   (:import javax.swing.JPanel)

   (:import javax.swing.SwingUtilities)

   (:import java.awt.Dimension)

   (:import javax.swing.JFrame)

   (:import com.jme3.math.ColorRGBA)

   (:import com.jme3.scene.Node)

   (:import com.jme3.math.Vector3f)

   (:import java.io.File))

 #+end_src

 

 

 

 - As a neat bonus, this idea behind simulated vision also enables one

   to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].

 

 

 * COMMENT Generate Source

 #+begin_src clojure :tangle ../src/cortex/vision.clj

 <<eyes>>

 #+end_src

 

 #+begin_src clojure :tangle ../src/cortex/test/vision.clj

 <<test-header>>

 <<test-1>>

 #+end_src