Mercurial > cortex

comparison org/hearing.org @ 306:7e7f8d6d9ec5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
massive spellchecking
author Robert McIntyre <rlm@mit.edu>
date Sat, 18 Feb 2012 10:59:41 -0700
parents 23aadf376e9d
children 5d448182c807
comparison
equal deleted inserted replaced
305:19c43ec6958d 306:7e7f8d6d9ec5
89 89
90 - Support Multiple Listeners from jMonkeyEngine3 90 - Support Multiple Listeners from jMonkeyEngine3
91 - Get access to the rendered sound data for further processing from 91 - Get access to the rendered sound data for further processing from
92 clojure. 92 clojure.
93 93
94 I named it the "Multiple Audio Send" Deives, or =Send= Device for 94 I named it the "Multiple Audio Send" Device, or =Send= Device for
95 short, since it sends audio data back to the callig application like 95 short, since it sends audio data back to the calling application like
96 an Aux-Send cable on a mixing board. 96 an Aux-Send cable on a mixing board.
97 97
98 Onward to the actual Device! 98 Onward to the actual Device!
99 99
100 ** =send.c= 100 ** =send.c=
760 760
761 #+include: "../../jmeCapture/src/com/aurellem/capture/audio/MultiListener.java" src java 761 #+include: "../../jmeCapture/src/com/aurellem/capture/audio/MultiListener.java" src java
762 762
763 ** SoundProcessors are like SceneProcessors 763 ** SoundProcessors are like SceneProcessors
764 764
765 A =SoundProcessor= is analgous to a =SceneProcessor=. Every frame, the 765 A =SoundProcessor= is analogous to a =SceneProcessor=. Every frame, the
766 =SoundProcessor= registered with a given =Listener= recieves the 766 =SoundProcessor= registered with a given =Listener= receives the
767 rendered sound data and can do whatever processing it wants with it. 767 rendered sound data and can do whatever processing it wants with it.
768 768
769 #+include "../../jmeCapture/src/com/aurellem/capture/audio/SoundProcessor.java" src java 769 #+include "../../jmeCapture/src/com/aurellem/capture/audio/SoundProcessor.java" src java
770 770
771 * Finally, Ears in clojure! 771 * Finally, Ears in clojure!
775 abstraction. 775 abstraction.
776 776
777 ** Hearing Pipeline 777 ** Hearing Pipeline
778 778
779 All sound rendering is done in the CPU, so =hearing-pipeline= is 779 All sound rendering is done in the CPU, so =hearing-pipeline= is
780 much less complicated than =vision-pipelie= The bytes available in 780 much less complicated than =vision-pipeline= The bytes available in
781 the ByteBuffer obtained from the =send= Device have different meanings 781 the ByteBuffer obtained from the =send= Device have different meanings
782 dependant upon the particular hardware or your system. That is why 782 dependent upon the particular hardware or your system. That is why
783 the =AudioFormat= object is necessary to provide the meaning that the 783 the =AudioFormat= object is necessary to provide the meaning that the
784 raw bytes lack. =byteBuffer->pulse-vector= uses the excellent 784 raw bytes lack. =byteBuffer->pulse-vector= uses the excellent
785 conversion facilities from [[http://www.tritonus.org/ ][tritonus]] ([[http://tritonus.sourceforge.net/apidoc/org/tritonus/share/sampled/FloatSampleTools.html#byte2floatInterleaved%2528byte%5B%5D,%2520int,%2520float%5B%5D,%2520int,%2520int,%2520javax.sound.sampled.AudioFormat%2529][javadoc]]) to generate a clojure vector of 785 conversion facilities from [[http://www.tritonus.org/ ][tritonus]] ([[http://tritonus.sourceforge.net/apidoc/org/tritonus/share/sampled/FloatSampleTools.html#byte2floatInterleaved%2528byte%5B%5D,%2520int,%2520float%5B%5D,%2520int,%2520int,%2520javax.sound.sampled.AudioFormat%2529][javadoc]]) to generate a clojure vector of
786 floats which represent the linear PCM encoded waveform of the 786 floats which represent the linear PCM encoded waveform of the
787 sound. With linear PCM (pulse code modulation) -1.0 represents maximum 787 sound. With linear PCM (pulse code modulation) -1.0 represents maximum
794 794
795 (defn hearing-pipeline 795 (defn hearing-pipeline
796 "Creates a SoundProcessor which wraps a sound processing 796 "Creates a SoundProcessor which wraps a sound processing
797 continuation function. The continuation is a function that takes 797 continuation function. The continuation is a function that takes
798 [#^ByteBuffer b #^Integer int numSamples #^AudioFormat af ], each of which 798 [#^ByteBuffer b #^Integer int numSamples #^AudioFormat af ], each of which
799 has already been apprpiately sized." 799 has already been appropriately sized."
800 [continuation] 800 [continuation]
801 (proxy [SoundProcessor] [] 801 (proxy [SoundProcessor] []
802 (cleanup []) 802 (cleanup [])
803 (process 803 (process
804 [#^ByteBuffer audioSamples numSamples #^AudioFormat audioFormat] 804 [#^ByteBuffer audioSamples numSamples #^AudioFormat audioFormat]
826 named "ears". =add-ear!= and =update-listener-velocity!= use 826 named "ears". =add-ear!= and =update-listener-velocity!= use
827 =bind-sense= to bind a =Listener= object located at the initial 827 =bind-sense= to bind a =Listener= object located at the initial
828 position of an "ear" node to the closest physical object in the 828 position of an "ear" node to the closest physical object in the
829 creature. That =Listener= will stay in the same orientation to the 829 creature. That =Listener= will stay in the same orientation to the
830 object with which it is bound, just as the camera in the [[http://aurellem.localhost/cortex/html/sense.html#sec-4-1][sense binding 830 object with which it is bound, just as the camera in the [[http://aurellem.localhost/cortex/html/sense.html#sec-4-1][sense binding
831 demonstration]]. =OpenAL= simulates the doppler effect for moving 831 demonstration]]. =OpenAL= simulates the Doppler effect for moving
832 listeners, =update-listener-velocity!= ensures that this velocity 832 listeners, =update-listener-velocity!= ensures that this velocity
833 information is always up-to-date. 833 information is always up-to-date.
834 834
835 #+name: hearing-ears 835 #+name: hearing-ears
836 #+begin_src clojure 836 #+begin_src clojure
876 ** Ear Creation 876 ** Ear Creation
877 877
878 #+name: hearing-kernel 878 #+name: hearing-kernel
879 #+begin_src clojure 879 #+begin_src clojure
880 (defn hearing-kernel 880 (defn hearing-kernel
881 "Returns a functon which returns auditory sensory data when called 881 "Returns a function which returns auditory sensory data when called
882 inside a running simulation." 882 inside a running simulation."
883 [#^Node creature #^Spatial ear] 883 [#^Node creature #^Spatial ear]
884 (let [hearing-data (atom []) 884 (let [hearing-data (atom [])
885 register-listener! 885 register-listener!
886 (runonce 886 (runonce
914 frequency of 44,100 samples per second, the vector will have exactly 914 frequency of 44,100 samples per second, the vector will have exactly
915 735 elements. 915 735 elements.
916 916
917 ** Visualizing Hearing 917 ** Visualizing Hearing
918 918
919 This is a simple visualization function which displaye the waveform 919 This is a simple visualization function which displays the waveform
920 reported by the simulated sense of hearing. It converts the values 920 reported by the simulated sense of hearing. It converts the values
921 reported in the vector returned by the hearing function from the range 921 reported in the vector returned by the hearing function from the range
922 [-1.0, 1.0] to the range [0 255], converts to integer, and displays 922 [-1.0, 1.0] to the range [0 255], converts to integer, and displays
923 the number as a greyscale pixel. 923 the number as a greyscale pixel.
924 924
1075 preload="none" poster="../images/aurellem-1280x480.png" /> 1075 preload="none" poster="../images/aurellem-1280x480.png" />
1076 </video> 1076 </video>
1077 </center> 1077 </center>
1078 <p>The worm can now hear the sound pulses produced from the 1078 <p>The worm can now hear the sound pulses produced from the
1079 hymn. Notice the strikingly different pattern that human speech 1079 hymn. Notice the strikingly different pattern that human speech
1080 makes compared to the insturments. Once the worm is pushed off the 1080 makes compared to the instruments. Once the worm is pushed off the
1081 floor, the sound it hears is attenuated, and the display of the 1081 floor, the sound it hears is attenuated, and the display of the
1082 sound it hears beomes fainter. This shows the 3D localization of 1082 sound it hears becomes fainter. This shows the 3D localization of
1083 sound in this world.</p> 1083 sound in this world.</p>
1084 </div> 1084 </div>
1085 #+end_html 1085 #+end_html
1086 1086
1087 *** Creating the Ear Video 1087 *** Creating the Ear Video