comparison org/ear.org @ 20:e8ae40c9848c

fixed 1,000,000 spelling errors
author Robert McIntyre <rlm@mit.edu>
date Thu, 03 Nov 2011 15:02:18 -0700
parents 22ac5a0367cd
children 0ee04505a37f
comparison
equal deleted inserted replaced
19:22ac5a0367cd 20:e8ae40c9848c
5 #+keywords: simulated hearing, openal, clojure, jMonkeyEngine3, LWJGL, AI 5 #+keywords: simulated hearing, openal, clojure, jMonkeyEngine3, LWJGL, AI
6 #+SETUPFILE: ../../aurellem/org/setup.org 6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org 7 #+INCLUDE: ../../aurellem/org/level-0.org
8 #+BABEL: :exports both :noweb yes :cache no :mkdirp yes 8 #+BABEL: :exports both :noweb yes :cache no :mkdirp yes
9 9
10
11
12
13 * Hearing 10 * Hearing
14 11
15 I want to be able to place ears in a similiar manner to how I place 12 I want to be able to place ears in a similar manner to how I place
16 the eyes. I want to be able to place ears in a unique spatial 13 the eyes. I want to be able to place ears in a unique spatial
17 position, and recieve as output at every tick the FFT of whatever 14 position, and receive as output at every tick the F.F.T. of whatever
18 signals are happening at that point. 15 signals are happening at that point.
19 16
20 Hearing is one of the more difficult senses to simulate, because there 17 Hearing is one of the more difficult senses to simulate, because there
21 is less support for obtaining the actual sound data that is processed 18 is less support for obtaining the actual sound data that is processed
22 by jMonkeyEngine3. 19 by jMonkeyEngine3.
23 20
24 jMonkeyEngine's sound system works as follows: 21 jMonkeyEngine's sound system works as follows:
25 22
26 - jMonkeyEngine uese the =AppSettings= for the particular application 23 - jMonkeyEngine uses the =AppSettings= for the particular application
27 to determine what sort of =AudioRenderer= should be used. 24 to determine what sort of =AudioRenderer= should be used.
28 - although some support is provided for multiple AudioRendering 25 - although some support is provided for multiple AudioRendering
29 backends, jMonkeyEngine at the time of this writing will either 26 backends, jMonkeyEngine at the time of this writing will either
30 pick no AudioRender at all, or the =LwjglAudioRenderer= 27 pick no AudioRenderer at all, or the =LwjglAudioRenderer=
31 - jMonkeyEngine tries to figure out what sort of system you're 28 - jMonkeyEngine tries to figure out what sort of system you're
32 running and extracts the appropiate native libraries. 29 running and extracts the appropriate native libraries.
33 - the =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game 30 - the =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
34 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]] 31 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
35 - =OpenAL= calculates the 3D sound localization and feeds a stream of 32 - =OpenAL= calculates the 3D sound localization and feeds a stream of
36 sound to any of various sound output devices with which it knows 33 sound to any of various sound output devices with which it knows
37 how to communicate. 34 how to communicate.
38 35
39 A consequence of this is that there's no way to access the actual 36 A consequence of this is that there's no way to access the actual
40 sound data produced by =OpenAL=. Even worse, =OpanAL= only supports 37 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
41 one /listener/, which normally isn't a problem for games, but becomes 38 one /listener/, which normally isn't a problem for games, but becomes
42 a problem when trying to make multiple AI creatures that can each hear 39 a problem when trying to make multiple AI creatures that can each hear
43 the world from a different perspective. 40 the world from a different perspective.
44 41
45 To make many AI creatures in jMonkeyEngine that can each hear the 42 To make many AI creatures in jMonkeyEngine that can each hear the
48 45
49 * Extending =OpenAL= 46 * Extending =OpenAL=
50 ** =OpenAL= Devices 47 ** =OpenAL= Devices
51 48
52 =OpenAL= goes to great lengths to support many different systems, all 49 =OpenAL= goes to great lengths to support many different systems, all
53 with different sound capabilities and interfaces. It acomplishes this 50 with different sound capabilities and interfaces. It accomplishes this
54 difficult task by providing code for many different sound backends in 51 difficult task by providing code for many different sound backends in
55 pseudo-objects called /Devices/. There's a device for the Linux Open 52 pseudo-objects called /Devices/. There's a device for the Linux Open
56 Sound System and the Advanced Linxu Sound Architechture, there's one 53 Sound System and the Advanced Linux Sound Architecture, there's one
57 for Direct Sound on Windows, there's even one for Solaris. =OpenAL= 54 for Direct Sound on Windows, there's even one for Solaris. =OpenAL=
58 solves the problem of platform independence by providing all these 55 solves the problem of platform independence by providing all these
59 Devices. 56 Devices.
60 57
61 Wrapper libraries such as LWJGL are free to examine the system on 58 Wrapper libraries such as LWJGL are free to examine the system on
62 which they are running and then select an appropiate device for that 59 which they are running and then select an appropriate device for that
63 system. 60 system.
64 61
65 There are also a few "special" devices that don't interface with any 62 There are also a few "special" devices that don't interface with any
66 particular system. These include the Null Device, which doesn't do 63 particular system. These include the Null Device, which doesn't do
67 anything, and the Wave Device, which writes whatever sound it recieves 64 anything, and the Wave Device, which writes whatever sound it receives
68 to a file, if everything has been set up correctly when configuring 65 to a file, if everything has been set up correctly when configuring
69 =OpenAL=. 66 =OpenAL=.
70 67
71 Actual mixing of the sound data happens in the Devices, and they are 68 Actual mixing of the sound data happens in the Devices, and they are
72 the only point in the sound rendering process where this data is 69 the only point in the sound rendering process where this data is
129 static void renderData(ALCdevice *, int samples); 126 static void renderData(ALCdevice *, int samples);
130 127
131 #define UNUSED(x) (void)(x) 128 #define UNUSED(x) (void)(x)
132 #+end_src 129 #+end_src
133 130
134 The main idea behing the Send device is to take advantage of the fact 131 The main idea behind the Send device is to take advantage of the fact
135 that LWJGL only manages one /context/ when using OpenAL. A /context/ 132 that LWJGL only manages one /context/ when using OpenAL. A /context/
136 is like a container that holds samples and keeps track of where the 133 is like a container that holds samples and keeps track of where the
137 listener is. In order to support multiple listeners, the Send device 134 listener is. In order to support multiple listeners, the Send device
138 identifies the LWJGL context as the master context, and creates any 135 identifies the LWJGL context as the master context, and creates any
139 number of slave contexts to represent additional listeners. Every 136 number of slave contexts to represent additional listeners. Every
144 141
145 To recap, the process is: 142 To recap, the process is:
146 - Set the LWJGL context as "master" in the =init()= method. 143 - Set the LWJGL context as "master" in the =init()= method.
147 - Create any number of additional contexts via =addContext()= 144 - Create any number of additional contexts via =addContext()=
148 - At every call to =renderData()= sync the master context with the 145 - At every call to =renderData()= sync the master context with the
149 slave contexts vit =syncContexts()= 146 slave contexts with =syncContexts()=
150 - =syncContexts()= calls =syncSources()= to sync all the sources 147 - =syncContexts()= calls =syncSources()= to sync all the sources
151 which are in the master context. 148 which are in the master context.
152 - =limitContext()= and =unLimitContext()= make it possible to render 149 - =limitContext()= and =unLimitContext()= make it possible to render
153 only one context at a time. 150 only one context at a time.
154 151
214 MAKE_SYNC3(syncSource3i, ALint, alGetSource3i, alSource3i); 211 MAKE_SYNC3(syncSource3i, ALint, alGetSource3i, alSource3i);
215 MAKE_SYNC3(syncSource3f, ALfloat, alGetSource3f, alSource3f); 212 MAKE_SYNC3(syncSource3f, ALfloat, alGetSource3f, alSource3f);
216 213
217 #+end_src 214 #+end_src
218 215
219 Setting the state of an =OpenAl= source is done with the =alSourcei=, 216 Setting the state of an =OpenAL= source is done with the =alSourcei=,
220 =alSourcef=, =alSource3i=, and =alSource3f= functions. In order to 217 =alSourcef=, =alSource3i=, and =alSource3f= functions. In order to
221 complely synchronize two sources, it is necessary to use all of 218 completely synchronize two sources, it is necessary to use all of
222 them. These macros help to condense the otherwise repetitive 219 them. These macros help to condense the otherwise repetitive
223 synchronization code involving these simillar low-level =OpenAL= functions. 220 synchronization code involving these similar low-level =OpenAL= functions.
224 221
225 ** Source Synchronization 222 ** Source Synchronization
226 #+begin_src C 223 #+begin_src C
227 void syncSources(ALsource *masterSource, ALsource *slaveSource, 224 void syncSources(ALsource *masterSource, ALsource *slaveSource,
228 ALCcontext *masterCtx, ALCcontext *slaveCtx){ 225 ALCcontext *masterCtx, ALCcontext *slaveCtx){
286 } 283 }
287 // Restore whatever context was previously active. 284 // Restore whatever context was previously active.
288 alcMakeContextCurrent(current); 285 alcMakeContextCurrent(current);
289 } 286 }
290 #+end_src 287 #+end_src
291 This function is long because it has to exaustively go through all the 288 This function is long because it has to exhaustively go through all the
292 possible state that a source can have and make sure that it is the 289 possible state that a source can have and make sure that it is the
293 same between the master and slave sources. I'd like to take this 290 same between the master and slave sources. I'd like to take this
294 moment to salute the [[http://connect.creativelabs.com/openal/Documentation/Forms/AllItems.aspx][=OpenAL= Reference Manual]], which provides a very 291 moment to salute the [[http://connect.creativelabs.com/openal/Documentation/Forms/AllItems.aspx][=OpenAL= Reference Manual]], which provides a very
295 good description of =OpenAL='s internals. 292 good description of =OpenAL='s internals.
296 293
311 ALuint numMissingSources = numMasterSources - numSlaveSources; 308 ALuint numMissingSources = numMasterSources - numSlaveSources;
312 ALuint newSources[numMissingSources]; 309 ALuint newSources[numMissingSources];
313 alGenSources(numMissingSources, newSources); 310 alGenSources(numMissingSources, newSources);
314 } 311 }
315 312
316 /* Now, slave is gauranteed to have at least as many sources 313 /* Now, slave is guaranteed to have at least as many sources
317 as master. Sync each source from master to the corresponding 314 as master. Sync each source from master to the corresponding
318 source in slave. */ 315 source in slave. */
319 int i; 316 int i;
320 for(i = 0; i < masterSourceMap->size; i++){ 317 for(i = 0; i < masterSourceMap->size; i++){
321 syncSources((ALsource*)masterSourceMap->array[i].value, 318 syncSources((ALsource*)masterSourceMap->array[i].value,
326 } 323 }
327 #+end_src 324 #+end_src
328 325
329 Most of the hard work in Context Synchronization is done in 326 Most of the hard work in Context Synchronization is done in
330 =syncSources()=. The only thing that =syncContexts()= has to worry 327 =syncSources()=. The only thing that =syncContexts()= has to worry
331 about is automoatically creating new sources whenever a slave context 328 about is automatically creating new sources whenever a slave context
332 does not have the same number of sources as the master context. 329 does not have the same number of sources as the master context.
333 330
334 ** Context Creation 331 ** Context Creation
335 #+begin_src C 332 #+begin_src C
336 static void addContext(ALCdevice *Device, ALCcontext *context){ 333 static void addContext(ALCdevice *Device, ALCcontext *context){
378 375
379 static ALCcontext **currentContext; 376 static ALCcontext **currentContext;
380 static ALuint currentNumContext; 377 static ALuint currentNumContext;
381 378
382 /* By default, all contexts are rendered at once for each call to aluMixData. 379 /* By default, all contexts are rendered at once for each call to aluMixData.
383 * This function uses the internals of the ALCdecice struct to temporarly 380 * This function uses the internals of the ALCdevice struct to temporally
384 * cause aluMixData to only render the chosen context. 381 * cause aluMixData to only render the chosen context.
385 */ 382 */
386 static void limitContext(ALCdevice *Device, ALCcontext *ctx){ 383 static void limitContext(ALCdevice *Device, ALCcontext *ctx){
387 currentContext = Device->Contexts; 384 currentContext = Device->Contexts;
388 currentNumContext = Device->NumContexts; 385 currentNumContext = Device->NumContexts;
394 Device->Contexts = currentContext; 391 Device->Contexts = currentContext;
395 Device->NumContexts = currentNumContext; 392 Device->NumContexts = currentNumContext;
396 } 393 }
397 #+end_src 394 #+end_src
398 395
399 =OpenAL= normally reneders all Contexts in parallel, outputting the 396 =OpenAL= normally renders all Contexts in parallel, outputting the
400 whole result to the buffer. It does this by iterating over the 397 whole result to the buffer. It does this by iterating over the
401 Device->Contexts array and rendering each context to the buffer in 398 Device->Contexts array and rendering each context to the buffer in
402 turn. By temporarly setting Device->NumContexts to 1 and adjusting 399 turn. By temporally setting Device->NumContexts to 1 and adjusting
403 the Device's context list to put the desired context-to-be-rendered 400 the Device's context list to put the desired context-to-be-rendered
404 into position 0, we can get trick =OpenAL= into rendering each slave 401 into position 0, we can get trick =OpenAL= into rendering each slave
405 context separate from all the others. 402 context separate from all the others.
406 403
407 ** Main Device Loop 404 ** Main Device Loop
574 alListenerf(param, v1); 571 alListenerf(param, v1);
575 alcMakeContextCurrent(current); 572 alcMakeContextCurrent(current);
576 } 573 }
577 #+end_src 574 #+end_src
578 575
579 *** Initilazation 576 *** Initialization
580 =initDevice= is called from the Java side after LWJGL has created its 577 =initDevice= is called from the Java side after LWJGL has created its
581 context, and before any calls to =addListener=. It establishes the 578 context, and before any calls to =addListener=. It establishes the
582 LWJGL context as the master context. 579 LWJGL context as the master context.
583 580
584 =getAudioFormat= is a convienence function that uses JNI to build up a 581 =getAudioFormat= is a convenience function that uses JNI to build up a
585 =javax.sound.sampled.AudioFormat= object from data in the Device. This 582 =javax.sound.sampled.AudioFormat= object from data in the Device. This
586 way, there is no ambiguity about what the bits created by =step= and 583 way, there is no ambiguity about what the bits created by =step= and
587 returned by =getSamples= mean. 584 returned by =getSamples= mean.
588 585
589 #+begin_src C 586 #+begin_src C
638 *** Boring Device management stuff 635 *** Boring Device management stuff
639 This code is more-or-less copied verbatim from the other =OpenAL= 636 This code is more-or-less copied verbatim from the other =OpenAL=
640 backends. It's the basis for =OpenAL='s primitive object system. 637 backends. It's the basis for =OpenAL='s primitive object system.
641 638
642 #+begin_src C 639 #+begin_src C
643 //////////////////// Device Initilization / Management 640 //////////////////// Device Initialization / Management
644 641
645 static const ALCchar sendDevice[] = "Multiple Audio Send"; 642 static const ALCchar sendDevice[] = "Multiple Audio Send";
646 643
647 static ALCboolean send_open_playback(ALCdevice *device, 644 static ALCboolean send_open_playback(ALCdevice *device,
648 const ALCchar *deviceName) 645 const ALCchar *deviceName)
649 { 646 {
650 send_data *data; 647 send_data *data;
651 // stop any buffering for stdout, so that I can 648 // stop any buffering for stdout, so that I can
652 // see the printf statements in my terminal immediatley 649 // see the printf statements in my terminal immediately
653 setbuf(stdout, NULL); 650 setbuf(stdout, NULL);
654 651
655 if(!deviceName) 652 if(!deviceName)
656 deviceName = sendDevice; 653 deviceName = sendDevice;
657 else if(strcmp(deviceName, sendDevice) != 0) 654 else if(strcmp(deviceName, sendDevice) != 0)
728 * The Java interface, =AudioSend= 725 * The Java interface, =AudioSend=
729 726
730 The Java interface to the Send Device follows naturally from the JNI 727 The Java interface to the Send Device follows naturally from the JNI
731 definitions. It is included here for completeness. The only thing here 728 definitions. It is included here for completeness. The only thing here
732 of note is the =deviceID=. This is available from LWJGL, but to only 729 of note is the =deviceID=. This is available from LWJGL, but to only
733 way to get it is reflection. Unfornatuently, there is no other way to 730 way to get it is reflection. Unfortunately, there is no other way to
734 control the Send device than to obtain a pointer to it. 731 control the Send device than to obtain a pointer to it.
735 732
736 #+include: "../java/src/com/aurellem/send/AudioSend.java" src java :exports code 733 #+include: "../java/src/com/aurellem/send/AudioSend.java" src java :exports code
737 734
738 * Finally, Ears in clojure! 735 * Finally, Ears in clojure!
739 736
740 Now that the infastructure is complete (modulo a few patches to 737 Now that the infrastructure is complete (modulo a few patches to
741 jMonkeyEngine3 to support accessing this modified version of =OpenAL= 738 jMonkeyEngine3 to support accessing this modified version of =OpenAL=
742 that are not worth discussing), the clojure ear abstraction is rather 739 that are not worth discussing), the clojure ear abstraction is rather
743 simple. Just as there were =SceneProcessors= for vision, there are 740 simple. Just as there were =SceneProcessors= for vision, there are
744 now =SoundProcessors= for hearing. 741 now =SoundProcessors= for hearing.
745 742
748 #+srcname: ears 745 #+srcname: ears
749 #+begin_src clojure 746 #+begin_src clojure
750 (ns cortex.hearing 747 (ns cortex.hearing
751 "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple 748 "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple
752 listeners at different positions in the same world. Passes vectors 749 listeners at different positions in the same world. Passes vectors
753 of floats in the range [-1.0 -- 1.0] in PCM format to any arbitray 750 of floats in the range [-1.0 -- 1.0] in PCM format to any arbitrary
754 function." 751 function."
755 {:author "Robert McIntyre"} 752 {:author "Robert McIntyre"}
756 (:use (cortex world util)) 753 (:use (cortex world util))
757 (:import java.nio.ByteBuffer) 754 (:import java.nio.ByteBuffer)
758 (:import org.tritonus.share.sampled.FloatSampleTools) 755 (:import org.tritonus.share.sampled.FloatSampleTools)