# HG changeset patch # User Robert McIntyre # Date 1328865564 25200 # Node ID f283c62bd2128f27881f5f7d78f9b697e9b65175 # Parent 01d3e9855ef9e385627d4488628752ef4a2a342c fixed long standing problem with orientation of eyes in blender, fleshed out text in vision.org diff -r 01d3e9855ef9 -r f283c62bd212 .hgignore --- a/.hgignore Thu Feb 09 09:04:17 2012 -0700 +++ b/.hgignore Fri Feb 10 02:19:24 2012 -0700 @@ -7,6 +7,10 @@ render* sources* *.hdr +libbulletjme64.so +liblwjgl64.so +libopenal64.so + syntax: regexp ^.*blend\d$ \ No newline at end of file diff -r 01d3e9855ef9 -r f283c62bd212 assets/Models/test-creature/worm.blend Binary file assets/Models/test-creature/worm.blend has changed diff -r 01d3e9855ef9 -r f283c62bd212 images/worm-with-eye.png Binary file images/worm-with-eye.png has changed diff -r 01d3e9855ef9 -r f283c62bd212 org/body.org --- a/org/body.org Thu Feb 09 09:04:17 2012 -0700 +++ b/org/body.org Fri Feb 10 02:19:24 2012 -0700 @@ -491,13 +491,15 @@ #+begin_src clojure (in-ns 'cortex.test.body) +(defn worm [] + (load-blender-model + "Models/test-creature/worm.blend")) + (defn worm-1 [] (let [timer (RatchetTimer. 60)] (world (nodify - [(doto - (load-blender-model - "Models/test-creature/worm.blend") + [(doto (worm) (body!)) (floor)]) (merge standard-debug-controls debug-control) diff -r 01d3e9855ef9 -r f283c62bd212 org/sense.org --- a/org/sense.org Thu Feb 09 09:04:17 2012 -0700 +++ b/org/sense.org Fri Feb 10 02:19:24 2012 -0700 @@ -271,15 +271,27 @@ its own JFrame." [sense-display-kernel] (let [windows (atom [])] - (fn [data] - (if (> (count data) (count @windows)) - (reset! - windows (map (fn [_] (view-image)) (range (count data))))) - (dorun - (map - (fn [display datum] - (display (sense-display-kernel datum))) - @windows data))))) + (fn this + ([data] + (this data nil)) + ([data save-to] + (if (> (count data) (count @windows)) + (reset! + windows + (doall + (map + (fn [idx] + (if save-to + (let [dir (File. save-to (str idx))] + (.mkdir dir) + (view-image dir)) + (view-image))) (range (count data)))))) + (dorun + (map + (fn [display datum] + (display (sense-display-kernel datum))) + @windows data)))))) + (defn points->image "Take a collection of points and visuliaze it as a BufferedImage." @@ -464,33 +476,35 @@ *** Combine Frames with ImageMagick #+begin_src clojure :results silent -(in-ns 'user) -(import java.io.File) -(use 'clojure.contrib.shell-out) -(let - [idx (atom -1) - left (rest - (sort - (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/")))) - right (rest +(ns cortex.video.magick + (:import java.io.File) + (:use clojure.contrib.shell-out)) + +(defn combine-images [] + (let + [idx (atom -1) + left (rest + (sort + (file-seq (File. "/home/r/proj/cortex/render/bind-sense0/")))) + right (rest + (sort + (file-seq + (File. "/home/r/proj/cortex/render/bind-sense1/")))) + sub (rest (sort (file-seq - (File. "/home/r/proj/cortex/render/bind-sense1/")))) - sub (rest - (sort - (file-seq - (File. "/home/r/proj/cortex/render/bind-senseB/")))) - sub* (concat sub (repeat 1000 (last sub)))] - (dorun - (map - (fn [im-1 im-2 sub] - (sh "convert" (.getCanonicalPath im-1) - (.getCanonicalPath im-2) "+append" - (.getCanonicalPath sub) "-append" - (.getCanonicalPath - (File. "/home/r/proj/cortex/render/bind-sense/" - (format "%07d.png" (swap! idx inc)))))) - left right sub*))) + (File. "/home/r/proj/cortex/render/bind-senseB/")))) + sub* (concat sub (repeat 1000 (last sub)))] + (dorun + (map + (fn [im-1 im-2 sub] + (sh "convert" (.getCanonicalPath im-1) + (.getCanonicalPath im-2) "+append" + (.getCanonicalPath sub) "-append" + (.getCanonicalPath + (File. "/home/r/proj/cortex/render/bind-sense/" + (format "%07d.png" (swap! idx inc)))))) + left right sub*)))) #+end_src *** Encode Frames with ffmpeg @@ -559,3 +573,7 @@ <> <> #+end_src + +#+begin_src clojure :tangle ../src/cortex/video/magick.clj +<> +#+end_src diff -r 01d3e9855ef9 -r f283c62bd212 org/vision.org --- a/org/vision.org Thu Feb 09 09:04:17 2012 -0700 +++ b/org/vision.org Fri Feb 10 02:19:24 2012 -0700 @@ -160,21 +160,50 @@ #+name: add-eye #+begin_src clojure +(in-ns 'cortex.vision) + +(import com.jme3.math.Vector3f) + +(def blender-rotation-correction + (doto (Quaternion.) + (.fromRotationMatrix + (doto (Matrix3f.) + (.setColumn 0 + (Vector3f. 1 0 0)) + (.setColumn 1 + (Vector3f. 0 -1 0)) + (.setColumn 2 + (Vector3f. 0 0 -1))) + + (doto (Matrix3f.) + (.setColumn 0 + (Vector3f. + + (defn add-eye! "Create a Camera centered on the current position of 'eye which follows the closest physical node in 'creature and sends visual - data to 'continuation." + data to 'continuation. The camera will point in the X direction and + use the Z vector as up as determined by the rotation of these + vectors in blender coordinate space. Use XZY rotation for the node + in blender." [#^Node creature #^Spatial eye] (let [target (closest-node creature eye) [cam-width cam-height] (eye-dimensions eye) - cam (Camera. cam-width cam-height)] + cam (Camera. cam-width cam-height) + rot (.getWorldRotation eye)] (.setLocation cam (.getWorldTranslation eye)) - (.setRotation cam (.getWorldRotation eye)) + (.lookAtDirection cam (.mult rot Vector3f/UNIT_X) + ;; this part is consistent with using Z in + ;; blender as the UP vector. + (.mult rot Vector3f/UNIT_Y)) + + (println-repl "eye unit-z ->" (.mult rot Vector3f/UNIT_Z)) + (println-repl "eye unit-y ->" (.mult rot Vector3f/UNIT_Y)) + (println-repl "eye unit-x ->" (.mult rot Vector3f/UNIT_X)) (.setFrustumPerspective - cam 45 (/ (.getWidth cam) (.getHeight cam)) - 1 1000) - (bind-sense target cam) - cam)) + cam 45 (/ (.getWidth cam) (.getHeight cam)) 1 1000) + (bind-sense target cam) cam)) #+end_src Here, the camera is created based on metadata on the eye-node and @@ -286,7 +315,8 @@ "Return the children of the creature's \"eyes\" node.") #+end_src -Then, +Then, add the camera created by =(add-eye!)= to the simulation by +creating a new viewport. #+begin_src clojure (defn add-camera! @@ -302,12 +332,26 @@ (.setBackgroundColor ColorRGBA/Black) (.addProcessor (vision-pipeline continuation)) (.attachScene (.getRootNode world))))) +#+end_src +The continuation function registers the viewport with the simulation +the first time it is called, and uses the CPU to extract the +appropriate pixels from the rendered image and weight them by each +sensors sensitivity. I have the option to do this filtering in native +code for a slight gain in speed. I could also do it in the GPU for a +massive gain in speed. =(vision-kernel)= generates a list of such +continuation functions, one for each channel of the eye. +#+begin_src clojure +(in-ns 'cortex.vision) +(defrecord attached-viewport [vision-fn viewport-fn] + clojure.lang.IFn + (invoke [this world] (vision-fn world)) + (applyTo [this args] (apply vision-fn args))) -(defn vision-fn +(defn vision-kernel "Returns a list of functions, each of which will return a color channel's worth of visual information when called inside a running simulation." @@ -335,20 +379,47 @@ (let [whites (white-coordinates image) topology (vec (collapse whites)) mask (color-channel-presets key key)] - (fn [world] - (register-eye! world) - (vector - topology - (vec - (for [[x y] whites] - (bit-and - mask (.getRGB @vision-image x y)))))))) - retinal-map)))) + (attached-viewport. + (fn [world] + (register-eye! world) + (vector + topology + (vec + (for [[x y] whites] + (bit-and + mask (.getRGB @vision-image x y)))))) + register-eye!))) + retinal-map)))) +(defn gen-fix-display + "Create a function to call to restore a simulation's display when it + is disrupted by a Viewport." + [] + (runonce + (fn [world] + (add-camera! world (.getCamera world) no-op)))) -;; TODO maybe should add a viewport-manipulation function to -;; automatically change viewport settings, attach shadow filters, etc. +#+end_src +Note that since each of the functions generated by =(vision-kernel)= +shares the same =(register-eye!)= function, the eye will be registered +only once the first time any of the functions from the list returned +by =(vision-kernel)= is called. Each of the functions returned by +=(vision-kernel)= also allows access to the =Viewport= through which +it recieves images. + +The in-game display can be disrupted by all the viewports that the +functions greated by =(vision-kernel)= add. This doesn't affect the +simulation or the simulated senses, but can be annoying. +=(gen-fix-display)= restores the in-simulation display. + +** Vision! + +All the hard work has been done, all that remains is to apply +=(vision-kernel)= to each eye in the creature and gather the results +into one list of functions. + +#+begin_src clojure (defn vision! "Returns a function which returns visual sensory data when called inside a running simulation" @@ -356,8 +427,16 @@ (reduce concat (for [eye (eyes creature)] - (vision-fn creature eye)))) + (vision-kernel creature eye)))) +#+end_src +** Visualization of Vision + +It's vital to have a visual representation for each sense. Here I use +=(view-sense)= to construct a function that will create a display for +visual data. + +#+begin_src clojure (defn view-vision "Creates a function which accepts a list of visual sensor-data and displays each element of the list to the screen." @@ -371,31 +450,20 @@ (.setRGB image ((coords i) 0) ((coords i) 1) (sensor-data i)))) image)))) - #+end_src +* Tests -Note the use of continuation passing style for connecting the eye to a -function to process the output. You can create any number of eyes, and -each of them will see the world from their own =Camera=. Once every -frame, the rendered image is copied to a =BufferedImage=, and that -data is sent off to the continuation function. Moving the =Camera= -which was used to create the eye will change what the eye sees. +** Basic Test -* Example +This is a basic test for the vision system. It only tests the +vision-pipeline and does not deal with loadig eyes from a blender +file. The code creates two videos of the same rotating cube from +different angles. -#+name: test-vision +#+name: test-1 #+begin_src clojure -(ns cortex.test.vision - (:use (cortex world util vision)) - (:import java.awt.image.BufferedImage) - (:import javax.swing.JPanel) - (:import javax.swing.SwingUtilities) - (:import java.awt.Dimension) - (:import javax.swing.JFrame) - (:import com.jme3.math.ColorRGBA) - (:import com.jme3.scene.Node) - (:import com.jme3.math.Vector3f)) +(in-ns 'cortex.test.vision) (defn test-two-eyes "Testing vision: @@ -417,15 +485,18 @@ width (.getWidth cam) height (.getHeight cam)] (add-camera! world cam - ;;no-op - (comp (view-image) BufferedImage!) - ) + (comp + (view-image + (File. "/home/r/proj/cortex/render/vision/1")) + BufferedImage!)) (add-camera! world (doto (.clone cam) (.setLocation (Vector3f. -10 0 0)) (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) - ;;no-op - (comp (view-image) BufferedImage!)) + (comp + (view-image + (File. "/home/r/proj/cortex/render/vision/2")) + BufferedImage!)) ;; This is here to restore the main view ;; after the other views have completed processing (add-camera! world (.getCamera world) no-op))) @@ -433,6 +504,98 @@ (.rotate candy (* tpf 0.2) 0 0))))) #+end_src +#+begin_html +
+ +

A rotating cube viewed from two different perspectives.

+
+#+end_html + +Creating multiple eyes like this can be used for stereoscopic vision +simulation in a single creature or for simulating multiple creatures, +each with their own sense of vision. + +** Adding Vision to the Worm + +To the worm from the last post, we add a new node that describes its +eyes. + +#+attr_html: width=755 +#+caption: The worm with newly added empty nodes describing a single eye. +[[../images/worm-with-eye.png]] + +The node highlighted in yellow is the root level "eyes" node. It has +a single node, highlighted in orange, which describes a single +eye. This is the "eye" node. The two nodes which are not highlighted describe the single joint +of the worm. + +The metadata of the eye-node is: + +#+begin_src clojure :results verbatim :exports both +(cortex.sense/meta-data + (.getChild + (.getChild (cortex.test.body/worm) + "eyes") "eye") "eye") +#+end_src + +#+results: +: "(let [retina \"Models/test-creature/retina-small.png\"] +: {:all retina :red retina :green retina :blue retina})" + +This is the approximation to the human eye described earlier. + +#+begin_src clojure +(in-ns 'cortex.test.vision) + +(import com.aurellem.capture.Capture) + +(defn test-worm-vision [] + (let [the-worm (doto (worm)(body!)) + vision (vision! the-worm) + vision-display (view-vision) + fix-display (gen-fix-display) + me (sphere 0.5 :color ColorRGBA/Blue :physical? false) + x-axis + (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red + :position (Vector3f. 0 -5 0)) + y-axis + (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green + :position (Vector3f. 0 -5 0)) + z-axis + (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue + :position (Vector3f. 0 -5 0))] + + (world (nodify [(floor) the-worm x-axis y-axis z-axis me]) + standard-debug-controls + (fn [world] + (light-up-everything world) + ;; add a view from the worm's perspective + (add-camera! + world + (add-eye! the-worm + (.getChild + (.getChild the-worm "eyes") "eye")) + (comp + (view-image + (File. "/home/r/proj/cortex/render/worm-vision/worm-view")) + BufferedImage!)) + (set-gravity world Vector3f/ZERO) + (Capture/captureVideo + world + (File. "/home/r/proj/cortex/render/worm-vision/main-view"))) + (fn [world _ ] + (.setLocalTranslation me (.getLocation (.getCamera world))) + (vision-display + (map #(% world) vision) + (File. "/home/r/proj/cortex/render/worm-vision")) + (fix-display world))))) +#+end_src + +* Headers + #+name: vision-header #+begin_src clojure (ns cortex.vision @@ -456,10 +619,23 @@ (:import (com.jme3.scene Node Spatial))) #+end_src -The example code will create two videos of the same rotating object -from different angles. It can be used both for stereoscopic vision -simulation or for simulating multiple creatures, each with their own -sense of vision. +#+name: test-header +#+begin_src clojure +(ns cortex.test.vision + (:use (cortex world sense util body vision)) + (:use cortex.test.body) + (:import java.awt.image.BufferedImage) + (:import javax.swing.JPanel) + (:import javax.swing.SwingUtilities) + (:import java.awt.Dimension) + (:import javax.swing.JFrame) + (:import com.jme3.math.ColorRGBA) + (:import com.jme3.scene.Node) + (:import com.jme3.math.Vector3f) + (:import java.io.File)) +#+end_src + + - As a neat bonus, this idea behind simulated vision also enables one to [[../../cortex/html/capture-video.html][capture live video feeds from jMonkeyEngine]]. @@ -471,5 +647,6 @@ #+end_src #+begin_src clojure :tangle ../src/cortex/test/vision.clj -<> +<> +<> #+end_src