cortex: org/vision.org comparison

comparison org/vision.org @ 215:f283c62bd212

fixed long standing problem with orientation of eyes in blender, fleshed out text in vision.org

author	Robert McIntyre <rlm@mit.edu>
date	Fri, 10 Feb 2012 02:19:24 -0700
parents	01d3e9855ef9
children	f5ea63245b3b

comparison

equal deleted inserted replaced

-:01d3e9855ef9
+:f283c62bd212
 parent named "joints". An eye binds to the nearest physical object
 with =(bind-sense=).
 #+name: add-eye
 #+begin_src clojure
+(in-ns 'cortex.vision)
+(import com.jme3.math.Vector3f)
+(def blender-rotation-correction
+(doto (Quaternion.)
+(.fromRotationMatrix
+(doto (Matrix3f.)
+(.setColumn 0
+(Vector3f. 1 0 0))
+(.setColumn 1
+(Vector3f. 0 -1 0))
+(.setColumn 2
+(Vector3f. 0 0 -1)))
+(doto (Matrix3f.)
+(.setColumn 0
+(Vector3f.
 (defn add-eye!
 "Create a Camera centered on the current position of 'eye which
 follows the closest physical node in 'creature and sends visual
-data to 'continuation."
+data to 'continuation. The camera will point in the X direction and
+use the Z vector as up as determined by the rotation of these
+vectors in blender coordinate space. Use XZY rotation for the node
+in blender."
 [#^Node creature #^Spatial eye]
 (let [target (closest-node creature eye)
 [cam-width cam-height] (eye-dimensions eye)
-cam (Camera. cam-width cam-height)]
+cam (Camera. cam-width cam-height)
+rot (.getWorldRotation eye)]
 (.setLocation cam (.getWorldTranslation eye))
-(.setRotation cam (.getWorldRotation eye))
+(.lookAtDirection cam (.mult rot Vector3f/UNIT_X)
+;; this part is consistent with using Z in
+;; blender as the UP vector.
+(.mult rot Vector3f/UNIT_Y))
+(println-repl "eye unit-z ->"  (.mult rot Vector3f/UNIT_Z))
+(println-repl "eye unit-y ->"  (.mult rot Vector3f/UNIT_Y))
+(println-repl "eye unit-x ->"  (.mult rot Vector3f/UNIT_X))
 (.setFrustumPerspective
-cam 45 (/ (.getWidth cam) (.getHeight cam))
+cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000)
-1 1000)
+(bind-sense target cam) cam))
-(bind-sense target cam)
-cam))
 #+end_src
 Here, the camera is created based on metadata on the eye-node and
 attached to the nearest physical object with =(bind-sense)=
 eyes
 (sense-nodes "eyes")
 "Return the children of the creature's \"eyes\" node.")
 #+end_src
-Then,
+Then, add the camera created by =(add-eye!)= to the simulation by
+creating a new viewport.
 #+begin_src clojure
 (defn add-camera!
 "Add a camera to the world, calling continuation on every frame
 produced."
 (doto viewport
 (.setClearFlags true true true)
 (.setBackgroundColor ColorRGBA/Black)
 (.addProcessor (vision-pipeline continuation))
 (.attachScene (.getRootNode world)))))
+#+end_src
+The continuation function registers the viewport with the simulation
+the first time it is called, and uses the CPU to extract the
-(defn vision-fn
+appropriate pixels from the rendered image and weight them by each
+sensors sensitivity. I have the option to do this filtering in native
+code for a slight gain in speed. I could also do it in the GPU for a
+massive gain in speed. =(vision-kernel)= generates a list of such
+continuation functions, one for each channel of the eye.
+#+begin_src clojure
+(in-ns 'cortex.vision)
+(defrecord attached-viewport [vision-fn viewport-fn]
+clojure.lang.IFn
+(invoke [this world] (vision-fn world))
+(applyTo [this args] (apply vision-fn args)))
+(defn vision-kernel
 "Returns a list of functions, each of which will return a color
 channel's worth of visual information when called inside a running
 simulation."
 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
 (let [retinal-map (retina-sensor-profile eye)
 (map
 (fn [[key image]]
 (let [whites (white-coordinates image)
 topology (vec (collapse whites))
 mask (color-channel-presets key key)]
-(fn [world]
+(attached-viewport.
-(register-eye! world)
+(fn [world]
-(vector
+(register-eye! world)
-topology
+(vector
-(vec
+topology
-(for [[x y] whites]
+(vec
-(bit-and
+(for [[x y] whites]
-mask (.getRGB @vision-image x y))))))))
+(bit-and
-retinal-map))))
+mask (.getRGB @vision-image x y))))))
+register-eye!)))
+retinal-map))))
-;; TODO maybe should add a viewport-manipulation function to
-;; automatically change viewport settings, attach shadow filters, etc.
+(defn gen-fix-display
+"Create a function to call to restore a simulation's display when it
+is disrupted by a Viewport."
+[]
+(runonce
+(fn [world]
+(add-camera! world (.getCamera world) no-op))))
+#+end_src
+Note that since each of the functions generated by =(vision-kernel)=
+shares the same =(register-eye!)= function, the eye will be registered
+only once the first time any of the functions from the list returned
+by =(vision-kernel)= is called.  Each of the functions returned by
+=(vision-kernel)= also allows access to the =Viewport= through which
+it recieves images.
+The in-game display can be disrupted by all the viewports that the
+functions greated by =(vision-kernel)= add. This doesn't affect the
+simulation or the simulated senses, but can be annoying.
+=(gen-fix-display)= restores the in-simulation display.
+** Vision!
+All the hard work has been done, all that remains is to apply
+=(vision-kernel)= to each eye in the creature and gather the results
+into one list of functions.
+#+begin_src clojure
 (defn vision!
 "Returns a function which returns visual sensory data when called
 inside a running simulation"
 [#^Node creature & {skip :skip :or {skip 0}}]
 (reduce
 concat
 (for [eye (eyes creature)]
-(vision-fn creature eye))))
+(vision-kernel creature eye))))
+#+end_src
+** Visualization of Vision
+It's vital to have a visual representation for each sense. Here I use
+=(view-sense)= to construct a function that will create a display for
+visual data.
+#+begin_src clojure
 (defn view-vision
 "Creates a function which accepts a list of visual sensor-data and
 displays each element of the list to the screen."
 []
 (view-sense
 (dorun
 (for [i (range (count coords))]
 (.setRGB image ((coords i) 0) ((coords i) 1)
 (sensor-data i))))
 image))))
+#+end_src
-#+end_src
+* Tests
-Note the use of continuation passing style for connecting the eye to a
+** Basic Test
-function to process the output. You can create any number of eyes, and
-each of them will see the world from their own =Camera=. Once every
+This is a basic test for the vision system.  It only tests the
-frame, the rendered image is copied to a =BufferedImage=, and that
+vision-pipeline and does not deal with loadig eyes from a blender
-data is sent off to the continuation function. Moving the =Camera=
+file. The code creates two videos of the same rotating cube from
-which was used to create the eye will change what the eye sees.
+different angles.
-* Example
+#+name: test-1
+#+begin_src clojure
-#+name: test-vision
+(in-ns 'cortex.test.vision)
-#+begin_src clojure
-(ns cortex.test.vision
-(:use (cortex world util vision))
-(:import java.awt.image.BufferedImage)
-(:import javax.swing.JPanel)
-(:import javax.swing.SwingUtilities)
-(:import java.awt.Dimension)
-(:import javax.swing.JFrame)
-(:import com.jme3.math.ColorRGBA)
-(:import com.jme3.scene.Node)
-(:import com.jme3.math.Vector3f))
 (defn test-two-eyes
 "Testing vision:
 Tests the vision system by creating two views of the same rotating
 object from different angles and displaying both of those views in
 (fn [world]
 (let [cam (.clone (.getCamera world))
 width (.getWidth cam)
 height (.getHeight cam)]
 (add-camera! world cam
-;;no-op
+(comp
-(comp (view-image) BufferedImage!)
+(view-image
-)
+(File. "/home/r/proj/cortex/render/vision/1"))
+BufferedImage!))
 (add-camera! world
 (doto (.clone cam)
 (.setLocation (Vector3f. -10 0 0))
 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
-;;no-op
+(comp
-(comp (view-image) BufferedImage!))
+(view-image
+(File. "/home/r/proj/cortex/render/vision/2"))
+BufferedImage!))
 ;; This is here to restore the main view
 ;; after the other views have completed processing
 (add-camera! world (.getCamera world) no-op)))
 (fn [world tpf]
 (.rotate candy (* tpf 0.2) 0 0)))))
 #+end_src
+#+begin_html
+<div class="figure">
+<video controls="controls" width="755">
+<source src="../video/spinning-cube.ogg" type="video/ogg"
+	  preload="none" poster="../images/aurellem-1280x480.png" />
+</video>
+<p>A rotating cube viewed from two different perspectives.</p>
+</div>
+#+end_html
+Creating multiple eyes like this can be used for stereoscopic vision
+simulation in a single creature or for simulating multiple creatures,
+each with their own sense of vision.
+** Adding Vision to the Worm
+To the worm from the last post, we add a new node that describes its
+eyes.
+#+attr_html: width=755
+#+caption: The worm with newly added empty nodes describing a single eye.
+[[../images/worm-with-eye.png]]
+The node highlighted in yellow is the root level "eyes" node.  It has
+a single node, highlighted in orange, which describes a single
+eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint
+of the worm.
+The metadata of the eye-node is:
+#+begin_src clojure :results verbatim :exports both
+(cortex.sense/meta-data
+(.getChild
+(.getChild (cortex.test.body/worm)
+"eyes") "eye") "eye")
+#+end_src
+#+results:
+: "(let [retina \"Models/test-creature/retina-small.png\"]
+:     {:all retina :red retina :green retina :blue retina})"
+This is the approximation to the human eye described earlier.
+#+begin_src clojure
+(in-ns 'cortex.test.vision)
+(import com.aurellem.capture.Capture)
+(defn test-worm-vision []
+(let [the-worm (doto (worm)(body!))
+vision (vision! the-worm)
+vision-display (view-vision)
+fix-display (gen-fix-display)
+me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
+x-axis
+(box 1 0.01 0.01 :physical? false :color ColorRGBA/Red
+:position (Vector3f. 0 -5 0))
+y-axis
+(box 0.01 1 0.01 :physical? false :color ColorRGBA/Green
+:position (Vector3f. 0 -5 0))
+z-axis
+(box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue
+:position (Vector3f. 0 -5 0))]
+(world (nodify [(floor) the-worm x-axis y-axis z-axis me])
+standard-debug-controls
+(fn [world]
+(light-up-everything world)
+;; add a view from the worm's perspective
+(add-camera!
+world
+(add-eye! the-worm
+(.getChild
+(.getChild the-worm "eyes") "eye"))
+(comp
+(view-image
+(File. "/home/r/proj/cortex/render/worm-vision/worm-view"))
+BufferedImage!))
+(set-gravity world Vector3f/ZERO)
+(Capture/captureVideo
+world
+(File. "/home/r/proj/cortex/render/worm-vision/main-view")))
+(fn [world _ ]
+(.setLocalTranslation me (.getLocation (.getCamera world)))
+(vision-display
+(map #(% world) vision)
+(File. "/home/r/proj/cortex/render/worm-vision"))
+(fix-display world)))))
+#+end_src
+* Headers
 #+name: vision-header
 #+begin_src clojure
 (ns cortex.vision
 "Simulate the sense of vision in jMonkeyEngine3. Enables multiple
 (:import com.jme3.app.Application)
 (:import com.jme3.texture.FrameBuffer)
 (:import (com.jme3.scene Node Spatial)))
 #+end_src
-The example code will create two videos of the same rotating object
+#+name: test-header
-from different angles. It can be used both for stereoscopic vision
+#+begin_src clojure
-simulation or for simulating multiple creatures, each with their own
+(ns cortex.test.vision
-sense of vision.
+(:use (cortex world sense util body vision))
+(:use cortex.test.body)
+(:import java.awt.image.BufferedImage)
+(:import javax.swing.JPanel)
+(:import javax.swing.SwingUtilities)
+(:import java.awt.Dimension)
+(:import javax.swing.JFrame)
+(:import com.jme3.math.ColorRGBA)
+(:import com.jme3.scene.Node)
+(:import com.jme3.math.Vector3f)
+(:import java.io.File))
+#+end_src
 - As a neat bonus, this idea behind simulated vision also enables one
 to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]].
 #+begin_src clojure :tangle ../src/cortex/vision.clj
 <<eyes>>
 #+end_src
 #+begin_src clojure :tangle ../src/cortex/test/vision.clj
-<<test-vision>>
+<<test-header>>
-#+end_src
+<<test-1>>
+#+end_src

Mercurial > cortex

comparison org/vision.org @ 215:f283c62bd212