# HG changeset patch # User Robert McIntyre # Date 1328987707 25200 # Node ID 7c374c6cfe170e79faa3e273a447c97333d1d783 # Parent c5f6d880558b015ca512f5697406395abd6b6da2 hearing.org rough draft is ready diff -r c5f6d880558b -r 7c374c6cfe17 assets/Models/test-creature/worm.blend Binary file assets/Models/test-creature/worm.blend has changed diff -r c5f6d880558b -r 7c374c6cfe17 images/worm-with-ear.png Binary file images/worm-with-ear.png has changed diff -r c5f6d880558b -r 7c374c6cfe17 org/hearing.org --- a/org/hearing.org Sat Feb 11 07:08:38 2012 -0700 +++ b/org/hearing.org Sat Feb 11 12:15:07 2012 -0700 @@ -788,7 +788,7 @@ rarefaction of the air while 1.0 represents maximum compression of the air at a given instant. -#+name: ears +#+name: hearing-pipeline #+begin_src clojure (in-ns 'cortex.hearing) @@ -832,6 +832,7 @@ listeners, =(update-listener-velocity!)= ensures that this velocity information is always up-to-date. +#+name: hearing-ears #+begin_src clojure (defvar ^{:arglists '([creature])} @@ -874,6 +875,7 @@ ** Ear Creation +#+name: hearing-kernel #+begin_src clojure (defn hearing-kernel "Returns a functon which returns auditory sensory data when called @@ -920,7 +922,10 @@ [-1.0, 1.0] to the range [0 255], converts to integer, and displays the number as a greyscale pixel. +#+name: hearing-display #+begin_src clojure +(in-ns 'cortex.hearing) + (defn view-hearing "Creates a function which accepts a list of auditory data and display each element of the list to the screen as an image." @@ -933,7 +938,7 @@ #(rem (int (* 255 (/ (+ 1 %) 2))) 256) sensor-data)) height 50 - image (BufferedImage. (count coords) height + image (BufferedImage. (max 1 (count coords)) height BufferedImage/TYPE_INT_RGB)] (dorun (for [x (range (count coords))] @@ -968,7 +973,6 @@ detects sound which is louder than a certain threshold. As the blue sphere travels along the path, it excites each of the cubes in turn.

- #+end_html #+include "../../jmeCapture/src/com/aurellem/capture/examples/Advanced.java" src java @@ -976,7 +980,7 @@ Here is a small clojure program to drive the java program and make it available as part of my test suite. -#+name: test-hearing +#+name: test-hearing-1 #+begin_src clojure (in-ns 'cortex.test.hearing) @@ -994,9 +998,139 @@ (.setPauseOnLostFocus false))) #+end_src + + + ** Adding Hearing to the Worm +To the worm, I add a new node called "ears" with one child which +represents the worm's single ear. +#+attr_html: width=755 +#+caption: The Worm with a newly added nodes describing an ear. +[[../images/worm-with-ear.png]] + +The node highlighted in yellow it the top-level "ears" node. It's +child, highlighted in orange, represents a the single ear the creature +has. The ear will be localized right above the curved part of the +worm's lower hemispherical region opposite the eye. + +The other empty nodes represent the worm's single joint and eye and are +described in [[./body.org][body]] and [[./vision.org][vision]]. + +#+name: test-hearing-2 +#+begin_src clojure +(in-ns 'cortex.test.hearing) + +(cortex.import/mega-import-jme3) +(import java.io.File) + +(use 'cortex.body) + +(defn test-worm-hearing [] + (let [the-worm (doto (worm) (body!)) + hearing (hearing! the-worm) + hearing-display (view-hearing) + + tone (AudioNode. (asset-manager) + "Sounds/pure.wav" false) + + hymn (AudioNode. (asset-manager) + "Sounds/ear-and-eye.wav" false)] + (world + (nodify [the-worm (floor)]) + (merge standard-debug-controls + {"key-return" + (fn [_ value] + (if value (.play tone))) + "key-l" + (fn [_ value] + (if value (.play hymn)))}) + (fn [world] + (light-up-everything world) + (com.aurellem.capture.Capture/captureVideo + world + (File."/home/r/proj/cortex/render/worm-audio/frames")) + (com.aurellem.capture.Capture/captureAudio + world + (File."/home/r/proj/cortex/render/worm-audio/audio.wav"))) + + (fn [world tpf] + (hearing-display + (map #(% world) hearing) + (File. "/home/r/proj/cortex/render/worm-audio/hearing-data")))))) +#+end_src + +In this test, I load the worm with its newly formed ear and let it +hear sounds. The sound the worm is hearing is localized to the origin +of the world, and you can see that as the worm moves farther away from +the origin when it is hit by balls, it hears the sound less intensely. + +The sound you hear in the video is from the worm's perspective. Notice +how the pure tone becomes fainter and the visual display of the +auditory data becomes less pronounced as the worm falls farther away +from the source of the sound. + +#+begin_html +
+
+ +
+

The worm can now hear the sound pulses produced from the + hymn. Notice the strikingly different pattern that human speech + makes compared to the insturments. Once the worm is pushed off the + floor, the sound it hears is attenuated, and the display of the + sound it hears beomes fainter. This shows the 3D localization of + sound in this world.

+
+ +#+end_html + +*** Creating the Ear Video +#+name: magick-3 +#+begin_src clojure +(ns cortex.video.magick3 + (:import java.io.File) + (:use clojure.contrib.shell-out)) + +(defn images [path] + (sort (rest (file-seq (File. path))))) + +(def base "/home/r/proj/cortex/render/worm-audio/") + +(defn pics [file] + (images (str base file))) + +(defn combine-images [] + (let [main-view (pics "frames") + hearing (pics "hearing-data") + background (repeat 9001 (File. (str base "background.png"))) + targets (map + #(File. (str base "out/" (format "%07d.png" %))) + (range 0 (count main-view)))] + (dorun + (pmap + (comp + (fn [[background main-view hearing target]] + (println target) + (sh "convert" + background + main-view "-geometry" "+66+21" "-composite" + hearing "-geometry" "+21+526" "-composite" + target)) + (fn [& args] (map #(.getCanonicalPath %) args))) + background main-view hearing targets)))) +#+end_src + +#+begin_src sh +cd /home/r/proj/cortex/render/worm-audio +ffmpeg -r 60 -i out/%07d.png -i audio.wav \ + -b:a 128k -b:v 9001k \ + -acodec libvorbis -vcodec libtheora worm-hearing.ogg +#+end_src * Headers @@ -1006,7 +1140,7 @@ "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple listeners at different positions in the same world. Automatically reads ear-nodes from specially prepared blender files and - instantiates them in the world as actual ears." + instantiates them in the world as simulated ears." {:author "Robert McIntyre"} (:use (cortex world util sense)) (:use clojure.contrib.def) @@ -1022,9 +1156,11 @@ (:import com.jme3.scene.control.AbstractControl)) #+end_src +#+name: test-header #+begin_src clojure (ns cortex.test.hearing (:use (cortex world util hearing)) + (:use cortex.test.body) (:import (com.jme3.audio AudioNode Listener)) (:import com.jme3.scene.Node com.jme3.system.AppSettings)) @@ -1042,11 +1178,20 @@ #+begin_src clojure :tangle ../src/cortex/hearing.clj <> -<> +<> +<> +<> +<> #+end_src #+begin_src clojure :tangle ../src/cortex/test/hearing.clj -<> +<> +<> +<> +#+end_src + +#+begin_src clojure :tangle ../src/cortex/video/magick3.clj +<> #+end_src #+begin_src C :tangle ../../audio-send/Alc/backends/send.c diff -r c5f6d880558b -r 7c374c6cfe17 org/sense.org --- a/org/sense.org Sat Feb 11 07:08:38 2012 -0700 +++ b/org/sense.org Sat Feb 11 12:15:07 2012 -0700 @@ -511,7 +511,7 @@ #+begin_src sh :results silent cd /home/r/proj/cortex/render/ -ffmpeg -r 60 -b 9000k -i bind-sense/%07d.png bind-sense.ogg +ffmpeg -r 30 -i bind-sense/%07d.png -b:v 9000k -vcodec libtheora bind-sense.ogg #+end_src * Headers