cortex: org/hearing.org comparison

comparison org/hearing.org @ 221:7c374c6cfe17

hearing.org rough draft is ready

author	Robert McIntyre <rlm@mit.edu>
date	Sat, 11 Feb 2012 12:15:07 -0700
parents	c5f6d880558b
children	49d0a545a872

comparison

equal deleted inserted replaced

-:c5f6d880558b
+:7c374c6cfe17
 floats which represent the linear PCM encoded waveform of the
 sound. With linear PCM (pulse code modulation) -1.0 represents maximum
 rarefaction of the air while 1.0 represents maximum compression of the
 air at a given instant.
-#+name: ears
+#+name: hearing-pipeline
 #+begin_src clojure
 (in-ns 'cortex.hearing)
 (defn hearing-pipeline
 "Creates a SoundProcessor which wraps a sound processing
 object with which it is bound, just as the camera in the [[http://aurellem.localhost/cortex/html/sense.html#sec-4-1][sense binding
 demonstration]].  =OpenAL= simulates the doppler effect for moving
 listeners, =(update-listener-velocity!)= ensures that this velocity
 information is always up-to-date.
+#+name: hearing-ears
 #+begin_src clojure
 (defvar
 ^{:arglists '([creature])}
 ears
 (sense-nodes "ears")
 (.registerSoundProcessor audio-renderer lis sp)))
 #+end_src
 ** Ear Creation
+#+name: hearing-kernel
 #+begin_src clojure
 (defn hearing-kernel
 "Returns a functon which returns auditory sensory data when called
 inside a running simulation."
 [#^Node creature #^Spatial ear]
 reported by the simulated sense of hearing. It converts the values
 reported in the vector returned by the hearing function from the range
 [-1.0, 1.0] to the range [0 255], converts to integer, and displays
 the number as a greyscale pixel.
+#+name: hearing-display
 #+begin_src clojure
+(in-ns 'cortex.hearing)
 (defn view-hearing
 "Creates a function which accepts a list of auditory data and
 display each element of the list to the screen as an image."
 []
 (view-sense
 (vec
 (map
 #(rem (int (* 255 (/ (+ 1 %) 2))) 256)
 sensor-data))
 height 50
-image (BufferedImage. (count coords) height
+image (BufferedImage. (max 1 (count coords)) height
 BufferedImage/TYPE_INT_RGB)]
 (dorun
 (for [x (range (count coords))]
 (dorun
 (for [y (range height)]
 <p>The blue ball is emitting a constant sound. Each blue box is
 listening for sound, and will change color from blue to green if it
 detects sound which is louder than a certain threshold. As the blue
 sphere travels along the path, it excites each of the cubes in turn.</p>
 </div>
 #+end_html
 #+include "../../jmeCapture/src/com/aurellem/capture/examples/Advanced.java" src java
 Here is a small clojure program to drive the java program and make it
 available as part of my test suite.
-#+name: test-hearing
+#+name: test-hearing-1
 #+begin_src clojure
 (in-ns 'cortex.test.hearing)
 (defn test-java-hearing
 "Testing hearing:
 (.setAudioRenderer "Send")))
 (.setShowSettings false)
 (.setPauseOnLostFocus false)))
 #+end_src
 ** Adding Hearing to the Worm
+To the worm, I add a new node called "ears" with one child which
+represents the worm's single ear.
+#+attr_html: width=755
+#+caption: The Worm with a newly added nodes describing an ear.
+[[../images/worm-with-ear.png]]
+The node highlighted in yellow it the top-level "ears" node. It's
+child, highlighted in orange, represents a the single ear the creature
+has. The ear will be localized right above the curved part of the
+worm's lower hemispherical region opposite the eye.
+The other empty nodes represent the worm's single joint and eye and are
+described in [[./body.org][body]] and [[./vision.org][vision]].
+#+name: test-hearing-2
+#+begin_src clojure
+(in-ns 'cortex.test.hearing)
+(cortex.import/mega-import-jme3)
+(import java.io.File)
+(use 'cortex.body)
+(defn test-worm-hearing []
+(let [the-worm (doto (worm) (body!))
+hearing (hearing! the-worm)
+hearing-display (view-hearing)
+tone (AudioNode. (asset-manager)
+"Sounds/pure.wav" false)
+hymn (AudioNode. (asset-manager)
+"Sounds/ear-and-eye.wav" false)]
+(world
+(nodify [the-worm (floor)])
+(merge standard-debug-controls
+{"key-return"
+(fn [_ value]
+(if value (.play tone)))
+"key-l"
+(fn [_ value]
+(if value (.play hymn)))})
+(fn [world]
+(light-up-everything world)
+(com.aurellem.capture.Capture/captureVideo
+world
+(File."/home/r/proj/cortex/render/worm-audio/frames"))
+(com.aurellem.capture.Capture/captureAudio
+world
+(File."/home/r/proj/cortex/render/worm-audio/audio.wav")))
+(fn [world tpf]
+(hearing-display
+(map #(% world) hearing)
+(File. "/home/r/proj/cortex/render/worm-audio/hearing-data"))))))
+#+end_src
+In this test, I load the worm with its newly formed ear and let it
+hear sounds. The sound the worm is hearing is localized to the origin
+of the world, and you can see that as the worm moves farther away from
+the origin when it is hit by balls, it hears the sound less intensely.
+The sound you hear in the video is from the worm's perspective. Notice
+how the pure tone becomes fainter and the visual display of the
+auditory data becomes less pronounced as the worm falls farther away
+from the source of the sound.
+#+begin_html
+<div class="figure">
+<center>
+<video controls="controls" width="600">
+<source src="../video/worm-hearing.ogg" type="video/ogg"
+	  preload="none" poster="../images/aurellem-1280x480.png" />
+</video>
+</center>
+<p>The worm can now hear the sound pulses produced from the
+hymn. Notice the strikingly different pattern that human speech
+makes compared to the insturments. Once the worm is pushed off the
+floor, the sound it hears is attenuated, and the display of the
+sound it hears beomes fainter. This shows the 3D localization of
+sound in this world.</p>
+</div>
+#+end_html
+*** Creating the Ear Video
+#+name: magick-3
+#+begin_src clojure
+(ns cortex.video.magick3
+(:import java.io.File)
+(:use clojure.contrib.shell-out))
+(defn images [path]
+(sort (rest (file-seq (File. path)))))
+(def base "/home/r/proj/cortex/render/worm-audio/")
+(defn pics [file]
+(images (str base file)))
+(defn combine-images []
+(let [main-view (pics "frames")
+hearing (pics "hearing-data")
+background (repeat 9001 (File. (str base "background.png")))
+targets (map
+#(File. (str base "out/" (format "%07d.png" %)))
+(range 0 (count main-view)))]
+(dorun
+(pmap
+(comp
+(fn [[background main-view hearing target]]
+(println target)
+(sh "convert"
+background
+main-view "-geometry" "+66+21"   "-composite"
+hearing   "-geometry" "+21+526"  "-composite"
+target))
+(fn [& args] (map #(.getCanonicalPath %) args)))
+background main-view hearing targets))))
+#+end_src
+#+begin_src sh
+cd /home/r/proj/cortex/render/worm-audio
+ffmpeg -r 60 -i out/%07d.png -i audio.wav \
+-b:a 128k  -b:v 9001k \
+-acodec libvorbis -vcodec libtheora worm-hearing.ogg
+#+end_src
 * Headers
 #+name: hearing-header
 #+begin_src clojure
 (ns cortex.hearing
 "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple
 listeners at different positions in the same world. Automatically
 reads ear-nodes from specially prepared blender files and
-instantiates them in the world as actual ears."
+instantiates them in the world as simulated ears."
 {:author "Robert McIntyre"}
 (:use (cortex world util sense))
 (:use clojure.contrib.def)
 (:import java.nio.ByteBuffer)
 (:import java.awt.image.BufferedImage)
 (:import com.jme3.audio.Listener)
 (:import com.jme3.app.Application)
 (:import com.jme3.scene.control.AbstractControl))
 #+end_src
+#+name: test-header
 #+begin_src clojure
 (ns cortex.test.hearing
 (:use (cortex world util hearing))
+(:use cortex.test.body)
 (:import (com.jme3.audio AudioNode Listener))
 (:import com.jme3.scene.Node
 	   com.jme3.system.AppSettings))
 #+end_src
 * COMMENT Code Generation
 #+begin_src clojure :tangle ../src/cortex/hearing.clj
 <<hearing-header>>
-<<ears>>
+<<hearing-pipeline>>
+<<hearing-ears>>
+<<hearing-kernel>>
+<<hearing-display>>
 #+end_src
 #+begin_src clojure :tangle ../src/cortex/test/hearing.clj
-<<test-hearing>>
+<<test-header>>
+<<test-hearing-1>>
+<<test-hearing-2>>
+#+end_src
+#+begin_src clojure :tangle ../src/cortex/video/magick3.clj
+<<magick-3>>
 #+end_src
 #+begin_src C :tangle ../../audio-send/Alc/backends/send.c
 <<send-header>>
 <<send-state>>

Mercurial > cortex

comparison org/hearing.org @ 221:7c374c6cfe17