Mercurial > audio-send
comparison org/ear.org @ 20:e8ae40c9848c
fixed 1,000,000 spelling errors
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Thu, 03 Nov 2011 15:02:18 -0700 |
parents | 22ac5a0367cd |
children | 0ee04505a37f |
comparison
equal
deleted
inserted
replaced
19:22ac5a0367cd | 20:e8ae40c9848c |
---|---|
5 #+keywords: simulated hearing, openal, clojure, jMonkeyEngine3, LWJGL, AI | 5 #+keywords: simulated hearing, openal, clojure, jMonkeyEngine3, LWJGL, AI |
6 #+SETUPFILE: ../../aurellem/org/setup.org | 6 #+SETUPFILE: ../../aurellem/org/setup.org |
7 #+INCLUDE: ../../aurellem/org/level-0.org | 7 #+INCLUDE: ../../aurellem/org/level-0.org |
8 #+BABEL: :exports both :noweb yes :cache no :mkdirp yes | 8 #+BABEL: :exports both :noweb yes :cache no :mkdirp yes |
9 | 9 |
10 | |
11 | |
12 | |
13 * Hearing | 10 * Hearing |
14 | 11 |
15 I want to be able to place ears in a similiar manner to how I place | 12 I want to be able to place ears in a similar manner to how I place |
16 the eyes. I want to be able to place ears in a unique spatial | 13 the eyes. I want to be able to place ears in a unique spatial |
17 position, and recieve as output at every tick the FFT of whatever | 14 position, and receive as output at every tick the F.F.T. of whatever |
18 signals are happening at that point. | 15 signals are happening at that point. |
19 | 16 |
20 Hearing is one of the more difficult senses to simulate, because there | 17 Hearing is one of the more difficult senses to simulate, because there |
21 is less support for obtaining the actual sound data that is processed | 18 is less support for obtaining the actual sound data that is processed |
22 by jMonkeyEngine3. | 19 by jMonkeyEngine3. |
23 | 20 |
24 jMonkeyEngine's sound system works as follows: | 21 jMonkeyEngine's sound system works as follows: |
25 | 22 |
26 - jMonkeyEngine uese the =AppSettings= for the particular application | 23 - jMonkeyEngine uses the =AppSettings= for the particular application |
27 to determine what sort of =AudioRenderer= should be used. | 24 to determine what sort of =AudioRenderer= should be used. |
28 - although some support is provided for multiple AudioRendering | 25 - although some support is provided for multiple AudioRendering |
29 backends, jMonkeyEngine at the time of this writing will either | 26 backends, jMonkeyEngine at the time of this writing will either |
30 pick no AudioRender at all, or the =LwjglAudioRenderer= | 27 pick no AudioRenderer at all, or the =LwjglAudioRenderer= |
31 - jMonkeyEngine tries to figure out what sort of system you're | 28 - jMonkeyEngine tries to figure out what sort of system you're |
32 running and extracts the appropiate native libraries. | 29 running and extracts the appropriate native libraries. |
33 - the =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game | 30 - the =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game |
34 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]] | 31 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]] |
35 - =OpenAL= calculates the 3D sound localization and feeds a stream of | 32 - =OpenAL= calculates the 3D sound localization and feeds a stream of |
36 sound to any of various sound output devices with which it knows | 33 sound to any of various sound output devices with which it knows |
37 how to communicate. | 34 how to communicate. |
38 | 35 |
39 A consequence of this is that there's no way to access the actual | 36 A consequence of this is that there's no way to access the actual |
40 sound data produced by =OpenAL=. Even worse, =OpanAL= only supports | 37 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports |
41 one /listener/, which normally isn't a problem for games, but becomes | 38 one /listener/, which normally isn't a problem for games, but becomes |
42 a problem when trying to make multiple AI creatures that can each hear | 39 a problem when trying to make multiple AI creatures that can each hear |
43 the world from a different perspective. | 40 the world from a different perspective. |
44 | 41 |
45 To make many AI creatures in jMonkeyEngine that can each hear the | 42 To make many AI creatures in jMonkeyEngine that can each hear the |
48 | 45 |
49 * Extending =OpenAL= | 46 * Extending =OpenAL= |
50 ** =OpenAL= Devices | 47 ** =OpenAL= Devices |
51 | 48 |
52 =OpenAL= goes to great lengths to support many different systems, all | 49 =OpenAL= goes to great lengths to support many different systems, all |
53 with different sound capabilities and interfaces. It acomplishes this | 50 with different sound capabilities and interfaces. It accomplishes this |
54 difficult task by providing code for many different sound backends in | 51 difficult task by providing code for many different sound backends in |
55 pseudo-objects called /Devices/. There's a device for the Linux Open | 52 pseudo-objects called /Devices/. There's a device for the Linux Open |
56 Sound System and the Advanced Linxu Sound Architechture, there's one | 53 Sound System and the Advanced Linux Sound Architecture, there's one |
57 for Direct Sound on Windows, there's even one for Solaris. =OpenAL= | 54 for Direct Sound on Windows, there's even one for Solaris. =OpenAL= |
58 solves the problem of platform independence by providing all these | 55 solves the problem of platform independence by providing all these |
59 Devices. | 56 Devices. |
60 | 57 |
61 Wrapper libraries such as LWJGL are free to examine the system on | 58 Wrapper libraries such as LWJGL are free to examine the system on |
62 which they are running and then select an appropiate device for that | 59 which they are running and then select an appropriate device for that |
63 system. | 60 system. |
64 | 61 |
65 There are also a few "special" devices that don't interface with any | 62 There are also a few "special" devices that don't interface with any |
66 particular system. These include the Null Device, which doesn't do | 63 particular system. These include the Null Device, which doesn't do |
67 anything, and the Wave Device, which writes whatever sound it recieves | 64 anything, and the Wave Device, which writes whatever sound it receives |
68 to a file, if everything has been set up correctly when configuring | 65 to a file, if everything has been set up correctly when configuring |
69 =OpenAL=. | 66 =OpenAL=. |
70 | 67 |
71 Actual mixing of the sound data happens in the Devices, and they are | 68 Actual mixing of the sound data happens in the Devices, and they are |
72 the only point in the sound rendering process where this data is | 69 the only point in the sound rendering process where this data is |
129 static void renderData(ALCdevice *, int samples); | 126 static void renderData(ALCdevice *, int samples); |
130 | 127 |
131 #define UNUSED(x) (void)(x) | 128 #define UNUSED(x) (void)(x) |
132 #+end_src | 129 #+end_src |
133 | 130 |
134 The main idea behing the Send device is to take advantage of the fact | 131 The main idea behind the Send device is to take advantage of the fact |
135 that LWJGL only manages one /context/ when using OpenAL. A /context/ | 132 that LWJGL only manages one /context/ when using OpenAL. A /context/ |
136 is like a container that holds samples and keeps track of where the | 133 is like a container that holds samples and keeps track of where the |
137 listener is. In order to support multiple listeners, the Send device | 134 listener is. In order to support multiple listeners, the Send device |
138 identifies the LWJGL context as the master context, and creates any | 135 identifies the LWJGL context as the master context, and creates any |
139 number of slave contexts to represent additional listeners. Every | 136 number of slave contexts to represent additional listeners. Every |
144 | 141 |
145 To recap, the process is: | 142 To recap, the process is: |
146 - Set the LWJGL context as "master" in the =init()= method. | 143 - Set the LWJGL context as "master" in the =init()= method. |
147 - Create any number of additional contexts via =addContext()= | 144 - Create any number of additional contexts via =addContext()= |
148 - At every call to =renderData()= sync the master context with the | 145 - At every call to =renderData()= sync the master context with the |
149 slave contexts vit =syncContexts()= | 146 slave contexts with =syncContexts()= |
150 - =syncContexts()= calls =syncSources()= to sync all the sources | 147 - =syncContexts()= calls =syncSources()= to sync all the sources |
151 which are in the master context. | 148 which are in the master context. |
152 - =limitContext()= and =unLimitContext()= make it possible to render | 149 - =limitContext()= and =unLimitContext()= make it possible to render |
153 only one context at a time. | 150 only one context at a time. |
154 | 151 |
214 MAKE_SYNC3(syncSource3i, ALint, alGetSource3i, alSource3i); | 211 MAKE_SYNC3(syncSource3i, ALint, alGetSource3i, alSource3i); |
215 MAKE_SYNC3(syncSource3f, ALfloat, alGetSource3f, alSource3f); | 212 MAKE_SYNC3(syncSource3f, ALfloat, alGetSource3f, alSource3f); |
216 | 213 |
217 #+end_src | 214 #+end_src |
218 | 215 |
219 Setting the state of an =OpenAl= source is done with the =alSourcei=, | 216 Setting the state of an =OpenAL= source is done with the =alSourcei=, |
220 =alSourcef=, =alSource3i=, and =alSource3f= functions. In order to | 217 =alSourcef=, =alSource3i=, and =alSource3f= functions. In order to |
221 complely synchronize two sources, it is necessary to use all of | 218 completely synchronize two sources, it is necessary to use all of |
222 them. These macros help to condense the otherwise repetitive | 219 them. These macros help to condense the otherwise repetitive |
223 synchronization code involving these simillar low-level =OpenAL= functions. | 220 synchronization code involving these similar low-level =OpenAL= functions. |
224 | 221 |
225 ** Source Synchronization | 222 ** Source Synchronization |
226 #+begin_src C | 223 #+begin_src C |
227 void syncSources(ALsource *masterSource, ALsource *slaveSource, | 224 void syncSources(ALsource *masterSource, ALsource *slaveSource, |
228 ALCcontext *masterCtx, ALCcontext *slaveCtx){ | 225 ALCcontext *masterCtx, ALCcontext *slaveCtx){ |
286 } | 283 } |
287 // Restore whatever context was previously active. | 284 // Restore whatever context was previously active. |
288 alcMakeContextCurrent(current); | 285 alcMakeContextCurrent(current); |
289 } | 286 } |
290 #+end_src | 287 #+end_src |
291 This function is long because it has to exaustively go through all the | 288 This function is long because it has to exhaustively go through all the |
292 possible state that a source can have and make sure that it is the | 289 possible state that a source can have and make sure that it is the |
293 same between the master and slave sources. I'd like to take this | 290 same between the master and slave sources. I'd like to take this |
294 moment to salute the [[http://connect.creativelabs.com/openal/Documentation/Forms/AllItems.aspx][=OpenAL= Reference Manual]], which provides a very | 291 moment to salute the [[http://connect.creativelabs.com/openal/Documentation/Forms/AllItems.aspx][=OpenAL= Reference Manual]], which provides a very |
295 good description of =OpenAL='s internals. | 292 good description of =OpenAL='s internals. |
296 | 293 |
311 ALuint numMissingSources = numMasterSources - numSlaveSources; | 308 ALuint numMissingSources = numMasterSources - numSlaveSources; |
312 ALuint newSources[numMissingSources]; | 309 ALuint newSources[numMissingSources]; |
313 alGenSources(numMissingSources, newSources); | 310 alGenSources(numMissingSources, newSources); |
314 } | 311 } |
315 | 312 |
316 /* Now, slave is gauranteed to have at least as many sources | 313 /* Now, slave is guaranteed to have at least as many sources |
317 as master. Sync each source from master to the corresponding | 314 as master. Sync each source from master to the corresponding |
318 source in slave. */ | 315 source in slave. */ |
319 int i; | 316 int i; |
320 for(i = 0; i < masterSourceMap->size; i++){ | 317 for(i = 0; i < masterSourceMap->size; i++){ |
321 syncSources((ALsource*)masterSourceMap->array[i].value, | 318 syncSources((ALsource*)masterSourceMap->array[i].value, |
326 } | 323 } |
327 #+end_src | 324 #+end_src |
328 | 325 |
329 Most of the hard work in Context Synchronization is done in | 326 Most of the hard work in Context Synchronization is done in |
330 =syncSources()=. The only thing that =syncContexts()= has to worry | 327 =syncSources()=. The only thing that =syncContexts()= has to worry |
331 about is automoatically creating new sources whenever a slave context | 328 about is automatically creating new sources whenever a slave context |
332 does not have the same number of sources as the master context. | 329 does not have the same number of sources as the master context. |
333 | 330 |
334 ** Context Creation | 331 ** Context Creation |
335 #+begin_src C | 332 #+begin_src C |
336 static void addContext(ALCdevice *Device, ALCcontext *context){ | 333 static void addContext(ALCdevice *Device, ALCcontext *context){ |
378 | 375 |
379 static ALCcontext **currentContext; | 376 static ALCcontext **currentContext; |
380 static ALuint currentNumContext; | 377 static ALuint currentNumContext; |
381 | 378 |
382 /* By default, all contexts are rendered at once for each call to aluMixData. | 379 /* By default, all contexts are rendered at once for each call to aluMixData. |
383 * This function uses the internals of the ALCdecice struct to temporarly | 380 * This function uses the internals of the ALCdevice struct to temporally |
384 * cause aluMixData to only render the chosen context. | 381 * cause aluMixData to only render the chosen context. |
385 */ | 382 */ |
386 static void limitContext(ALCdevice *Device, ALCcontext *ctx){ | 383 static void limitContext(ALCdevice *Device, ALCcontext *ctx){ |
387 currentContext = Device->Contexts; | 384 currentContext = Device->Contexts; |
388 currentNumContext = Device->NumContexts; | 385 currentNumContext = Device->NumContexts; |
394 Device->Contexts = currentContext; | 391 Device->Contexts = currentContext; |
395 Device->NumContexts = currentNumContext; | 392 Device->NumContexts = currentNumContext; |
396 } | 393 } |
397 #+end_src | 394 #+end_src |
398 | 395 |
399 =OpenAL= normally reneders all Contexts in parallel, outputting the | 396 =OpenAL= normally renders all Contexts in parallel, outputting the |
400 whole result to the buffer. It does this by iterating over the | 397 whole result to the buffer. It does this by iterating over the |
401 Device->Contexts array and rendering each context to the buffer in | 398 Device->Contexts array and rendering each context to the buffer in |
402 turn. By temporarly setting Device->NumContexts to 1 and adjusting | 399 turn. By temporally setting Device->NumContexts to 1 and adjusting |
403 the Device's context list to put the desired context-to-be-rendered | 400 the Device's context list to put the desired context-to-be-rendered |
404 into position 0, we can get trick =OpenAL= into rendering each slave | 401 into position 0, we can get trick =OpenAL= into rendering each slave |
405 context separate from all the others. | 402 context separate from all the others. |
406 | 403 |
407 ** Main Device Loop | 404 ** Main Device Loop |
574 alListenerf(param, v1); | 571 alListenerf(param, v1); |
575 alcMakeContextCurrent(current); | 572 alcMakeContextCurrent(current); |
576 } | 573 } |
577 #+end_src | 574 #+end_src |
578 | 575 |
579 *** Initilazation | 576 *** Initialization |
580 =initDevice= is called from the Java side after LWJGL has created its | 577 =initDevice= is called from the Java side after LWJGL has created its |
581 context, and before any calls to =addListener=. It establishes the | 578 context, and before any calls to =addListener=. It establishes the |
582 LWJGL context as the master context. | 579 LWJGL context as the master context. |
583 | 580 |
584 =getAudioFormat= is a convienence function that uses JNI to build up a | 581 =getAudioFormat= is a convenience function that uses JNI to build up a |
585 =javax.sound.sampled.AudioFormat= object from data in the Device. This | 582 =javax.sound.sampled.AudioFormat= object from data in the Device. This |
586 way, there is no ambiguity about what the bits created by =step= and | 583 way, there is no ambiguity about what the bits created by =step= and |
587 returned by =getSamples= mean. | 584 returned by =getSamples= mean. |
588 | 585 |
589 #+begin_src C | 586 #+begin_src C |
638 *** Boring Device management stuff | 635 *** Boring Device management stuff |
639 This code is more-or-less copied verbatim from the other =OpenAL= | 636 This code is more-or-less copied verbatim from the other =OpenAL= |
640 backends. It's the basis for =OpenAL='s primitive object system. | 637 backends. It's the basis for =OpenAL='s primitive object system. |
641 | 638 |
642 #+begin_src C | 639 #+begin_src C |
643 //////////////////// Device Initilization / Management | 640 //////////////////// Device Initialization / Management |
644 | 641 |
645 static const ALCchar sendDevice[] = "Multiple Audio Send"; | 642 static const ALCchar sendDevice[] = "Multiple Audio Send"; |
646 | 643 |
647 static ALCboolean send_open_playback(ALCdevice *device, | 644 static ALCboolean send_open_playback(ALCdevice *device, |
648 const ALCchar *deviceName) | 645 const ALCchar *deviceName) |
649 { | 646 { |
650 send_data *data; | 647 send_data *data; |
651 // stop any buffering for stdout, so that I can | 648 // stop any buffering for stdout, so that I can |
652 // see the printf statements in my terminal immediatley | 649 // see the printf statements in my terminal immediately |
653 setbuf(stdout, NULL); | 650 setbuf(stdout, NULL); |
654 | 651 |
655 if(!deviceName) | 652 if(!deviceName) |
656 deviceName = sendDevice; | 653 deviceName = sendDevice; |
657 else if(strcmp(deviceName, sendDevice) != 0) | 654 else if(strcmp(deviceName, sendDevice) != 0) |
728 * The Java interface, =AudioSend= | 725 * The Java interface, =AudioSend= |
729 | 726 |
730 The Java interface to the Send Device follows naturally from the JNI | 727 The Java interface to the Send Device follows naturally from the JNI |
731 definitions. It is included here for completeness. The only thing here | 728 definitions. It is included here for completeness. The only thing here |
732 of note is the =deviceID=. This is available from LWJGL, but to only | 729 of note is the =deviceID=. This is available from LWJGL, but to only |
733 way to get it is reflection. Unfornatuently, there is no other way to | 730 way to get it is reflection. Unfortunately, there is no other way to |
734 control the Send device than to obtain a pointer to it. | 731 control the Send device than to obtain a pointer to it. |
735 | 732 |
736 #+include: "../java/src/com/aurellem/send/AudioSend.java" src java :exports code | 733 #+include: "../java/src/com/aurellem/send/AudioSend.java" src java :exports code |
737 | 734 |
738 * Finally, Ears in clojure! | 735 * Finally, Ears in clojure! |
739 | 736 |
740 Now that the infastructure is complete (modulo a few patches to | 737 Now that the infrastructure is complete (modulo a few patches to |
741 jMonkeyEngine3 to support accessing this modified version of =OpenAL= | 738 jMonkeyEngine3 to support accessing this modified version of =OpenAL= |
742 that are not worth discussing), the clojure ear abstraction is rather | 739 that are not worth discussing), the clojure ear abstraction is rather |
743 simple. Just as there were =SceneProcessors= for vision, there are | 740 simple. Just as there were =SceneProcessors= for vision, there are |
744 now =SoundProcessors= for hearing. | 741 now =SoundProcessors= for hearing. |
745 | 742 |
748 #+srcname: ears | 745 #+srcname: ears |
749 #+begin_src clojure | 746 #+begin_src clojure |
750 (ns cortex.hearing | 747 (ns cortex.hearing |
751 "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple | 748 "Simulate the sense of hearing in jMonkeyEngine3. Enables multiple |
752 listeners at different positions in the same world. Passes vectors | 749 listeners at different positions in the same world. Passes vectors |
753 of floats in the range [-1.0 -- 1.0] in PCM format to any arbitray | 750 of floats in the range [-1.0 -- 1.0] in PCM format to any arbitrary |
754 function." | 751 function." |
755 {:author "Robert McIntyre"} | 752 {:author "Robert McIntyre"} |
756 (:use (cortex world util)) | 753 (:use (cortex world util)) |
757 (:import java.nio.ByteBuffer) | 754 (:import java.nio.ByteBuffer) |
758 (:import org.tritonus.share.sampled.FloatSampleTools) | 755 (:import org.tritonus.share.sampled.FloatSampleTools) |