======Capture Audio/Video to a File======

 

 <note>A/V recording is still in development.  It works for all of jMonkeyEngine's test cases. If you experience any problems or

 of something isn't clear, please let me know. -- bortreb</note>

 

 So you've made your cool new JMonkeyEngine3 game and you want to

 create a demo video to show off your hard work. Or maybe you want to

 make a cutscene for your game using the physics and characters in the

 game itself.  Screen capturing is the most straightforward way to do

 this, but it can slow down your game and produce low-quality video and

 audio as a result. A better way is to record video and audio directly

 from the game while it is running.

 

 <note tip>Combine this method with jMonkeyEngine's [[:jme3:advanced:Cinematics]] feature to record high-quality game trailers!</note>

 

 ===== Simple Way =====

 

 First off, if all you just want to record video at 30fps with no sound, then look

 no further then jMonkeyEngine3's build in ''VideoRecorderAppState''

 class.

 

 Add the following code to your ''simpleInitApp()'' method. 

 

 <code java>

 stateManager.attach(new VideoRecorderAppState()); //start recording

 </code>

 

 The game will run slow, but the recording will be in high-quality and

 normal speed. The video files will be stored in your user home

 directory, if you want to save to another file, specify it in the

 VideoRecorderAppState constructor. Recording starts when the state is

 attached and ends when the application quits or the state is detached.

 

 That's all!

 

 ===== Advanced Way =====

 

 If you want to record audio as well, record at different framerates,

 or record from multiple viewpoints at once, then there's a full

 solution for doing this already made for you here:

 

 http://www.aurellem.com/releases/jmeCapture-latest.zip

 

 http://www.aurellem.com/releases/jmeCapture-latest.tar.bz2

 

 Download the archive in your preferred format, extract,

 add the jars to your project, and you are ready to go.

 

 The javadoc is here:

 http://www.aurellem.com/jmeCapture/docs/

 

 To capture video and audio you use the

 ''com.aurellem.capture.Capture'' class, which has two methods,

 ''captureAudio()'' and ''captureVideo()'', and the

 ''com.aurellem.capture.IsoTimer'' class, which sets the audio and

 video framerate.

 

 The steps are as simple as:

 

 <code java>

 yourApp.setTimer(new IsoTimer(desiredFramesPerSecond));

 </code>

 

 This causes jMonkeyEngine to take as much time as it needs to fully

 calculate every frame of the video and audio.  You will see your game

 speed up and slow down depending on how computationally demanding your

 game is, but the final recorded audio and video will be perfectly

 sychronized and will run at exactly the fps which you specified.

 

 <code java>

 captureVideo(yourApp, targetVideoFile);

 captureAudio(yourApp, targetAudioFile);

 </code>

 

 These will cause the app to record audio and video when it is run.

 Audio and video will stop being recorded when the app stops. Your

 audio will be recorded as a 44,100 Hz linear PCM wav file, while the

 video will be recorded according to the following rules:

 

 1.) (Preferred) If you supply an empty directory as the file, then

 the video will be saved as a sequence of .png files, one file per

 frame.  The files start at 0000000.png and increment from there.

 You can then combine the frames into your preferred

 container/codec. If the directory is not empty, then writing

 video frames to it will fail, and nothing will be written.

     

 2.) If the filename ends in ".avi" then the frames will be encoded as

 a RAW stream inside an AVI 1.0 container.  The resulting file

 will be quite large and you will probably want to re-encode it to

 your preferred container/codec format.  Be advised that some

 video payers cannot process AVI with a RAW stream, and that AVI

 1.0 files generated by this method that exceed 2.0GB are invalid

 according to the AVI 1.0 spec (but many programs can still deal

 with them.)  Thanks to Werner Randelshofer for his excellent work

 which made the AVI file writer option possible.

  

 3.) Any non-directory file ending in anything other than ".avi" will

 be processed through Xuggle.  Xuggle provides the option to use

 many codecs/containers, but you will have to install it on your

 system yourself in order to use this option. Please visit

 http://www.xuggle.com/ to learn how to do this.

 

 Note that you will not hear any sound if you choose to record sound to

 a file.

 

 ==== Basic Example ====

 

 Here is a complete example showing how to capture both audio and video

 from one of jMonkeyEngine3's advanced demo applications.

 

 <code java>

 import java.io.File;

 import java.io.IOException;

 

 import jme3test.water.TestPostWater;

 

 import com.aurellem.capture.Capture;

 import com.aurellem.capture.IsoTimer;

 import com.jme3.app.SimpleApplication;

 

 

 /**

  * Demonstrates how to use basic Audio/Video capture with a

  * jMonkeyEngine application. You can use these techniques to make

  * high quality cutscenes or demo videos, even on very slow laptops.

  * 

  * @author Robert McIntyre

  */

 

 public class Basic {

 	

     public static void main(String[] ignore) throws IOException{

 	File video = File.createTempFile("JME-water-video", ".avi");

 	File audio = File.createTempFile("JME-water-audio", ".wav");

 		

 	SimpleApplication app = new TestPostWater();

 	app.setTimer(new IsoTimer(60));

 	app.setShowSettings(false);

 		

 	Capture.captureVideo(app, video);

 	Capture.captureAudio(app, audio);

 		

 	app.start();

 		

 	System.out.println(video.getCanonicalPath());

 	System.out.println(audio.getCanonicalPath());

     }

 }

 </code>

 

 ==== How it works ====

 

 A standard JME3 application that extends ''SimpleApplication'' or

 ''Application'' tries as hard as it can to keep in sync with

 //user-time//.  If a ball is rolling at 1 game-mile per game-hour in the

 game, and you wait for one user-hour as measured by the clock on your

 wall, then the ball should have traveled exactly one game-mile. In

 order to keep sync with the real world, the game throttles its physics

 engine and graphics display.  If the computations involved in running

 the game are too intense, then the game will first skip frames, then

 sacrifice physics accuracy.  If there are particuraly demanding

 computations, then you may only get 1 fps, and the ball may tunnel

 through the floor or obstacles due to inaccurate physics simulation,

 but after the end of one user-hour, that ball will have traveled one

 game-mile.

 

 When we're recording video, we don't care if the game-time syncs with

 user-time, but instead whether the time in the recorded video

 (video-time) syncs with user-time. To continue the analogy, if we

 recorded the ball rolling at 1 game-mile per game-hour and watched the

 video later, we would want to see 30 fps video of the ball rolling at

 1 video-mile per //user-hour//. It doesn't matter how much user-time it

 took to simulate that hour of game-time to make the high-quality

 recording.

 

 The IsoTimer ignores real-time and always reports that the same amount

 of time has passed every time it is called. That way, one can put code

 to write each video/audio frame to a file without worrying about that

 code itself slowing down the game to the point where the recording

 would be useless.

 

 

 ==== Advanced Example ==== 

 

 The package from aurellem.com was made for AI research and can do more

 than just record a single stream of audio and video. You can use it

 to:

 

 1.) Create multiple independent listeners that each hear the world

 from their own perspective.

 

 2.) Process the sound data in any way you wish.

 

 3.) Do the same for visual data.

 

 Here is a more advanced example, which can also be found along with

 other examples in the jmeCapture.jar file included in the

 distribution.

 

 <code java>

 package com.aurellem.capture.examples;

 

 import java.io.File;

 import java.io.FileNotFoundException;

 import java.io.IOException;

 import java.lang.reflect.Field;

 import java.nio.ByteBuffer;

 

 import javax.sound.sampled.AudioFormat;

 

 import org.tritonus.share.sampled.FloatSampleTools;

 

 import com.aurellem.capture.AurellemSystemDelegate;

 import com.aurellem.capture.Capture;

 import com.aurellem.capture.IsoTimer;

 import com.aurellem.capture.audio.CompositeSoundProcessor;

 import com.aurellem.capture.audio.MultiListener;

 import com.aurellem.capture.audio.SoundProcessor;

 import com.aurellem.capture.audio.WaveFileWriter;

 import com.jme3.app.SimpleApplication;

 import com.jme3.audio.AudioNode;

 import com.jme3.audio.Listener;

 import com.jme3.cinematic.MotionPath;

 import com.jme3.cinematic.events.AbstractCinematicEvent;

 import com.jme3.cinematic.events.MotionTrack;

 import com.jme3.material.Material;

 import com.jme3.math.ColorRGBA;

 import com.jme3.math.FastMath;

 import com.jme3.math.Quaternion;

 import com.jme3.math.Vector3f;

 import com.jme3.scene.Geometry;

 import com.jme3.scene.Node;

 import com.jme3.scene.shape.Box;

 import com.jme3.scene.shape.Sphere;

 import com.jme3.system.AppSettings;

 import com.jme3.system.JmeSystem;

 

 /**

  * 

  * Demonstrates advanced use of the audio capture and recording

  * features.  Multiple perspectives of the same scene are

  * simultaneously rendered to different sound files.

  * 

  * A key limitation of the way multiple listeners are implemented is

  * that only 3D positioning effects are realized for listeners other

  * than the main LWJGL listener.  This means that audio effects such

  * as environment settings will *not* be heard on any auxiliary

  * listeners, though sound attenuation will work correctly.

  * 

  * Multiple listeners as realized here might be used to make AI

  * entities that can each hear the world from their own perspective.

  * 

  * @author Robert McIntyre

  */

 

 public class Advanced extends SimpleApplication {

 

     /**

      * You will see three grey cubes, a blue sphere, and a path which

      * circles each cube.  The blue sphere is generating a constant

      * monotone sound as it moves along the track.  Each cube is

      * listening for sound; when a cube hears sound whose intensity is

      * greater than a certain threshold, it changes its color from

      * grey to green.

      * 

      *  Each cube is also saving whatever it hears to a file.  The

      *  scene from the perspective of the viewer is also saved to a

      *  video file.  When you listen to each of the sound files

      *  alongside the video, the sound will get louder when the sphere

      *  approaches the cube that generated that sound file.  This

      *  shows that each listener is hearing the world from its own

      *  perspective.

      * 

      */

     public static void main(String[] args) {

 	Advanced app = new Advanced();

 	AppSettings settings = new AppSettings(true);

 	settings.setAudioRenderer(AurellemSystemDelegate.SEND);

 	JmeSystem.setSystemDelegate(new AurellemSystemDelegate());

 	app.setSettings(settings);

 	app.setShowSettings(false);

 	app.setPauseOnLostFocus(false);

 		

 	try {

 	    Capture.captureVideo(app, File.createTempFile("advanced",".avi"));

 	    Capture.captureAudio(app, File.createTempFile("advanced", ".wav"));

 	}

 	catch (IOException e) {e.printStackTrace();}

 		

 	app.start();

     }

 

 	

     private Geometry bell;

     private Geometry ear1;

     private Geometry ear2;

     private Geometry ear3;

     private AudioNode music;

     private MotionTrack motionControl;

 		

     private Geometry makeEar(Node root, Vector3f position){

 	Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");

 	Geometry ear = new Geometry("ear", new Box(1.0f, 1.0f, 1.0f));

 	ear.setLocalTranslation(position);

 	mat.setColor("Color", ColorRGBA.Green);

 	ear.setMaterial(mat);

 	root.attachChild(ear);

 	return ear;

     } 

 

     private Vector3f[] path = new Vector3f[]{

 	// loop 1

 	new Vector3f(0, 0, 0),

 	new Vector3f(0, 0, -10),

 	new Vector3f(-2, 0, -14),

 	new Vector3f(-6, 0, -20),

 	new Vector3f(0, 0, -26),

 	new Vector3f(6, 0, -20),

 	new Vector3f(0, 0, -14),

 	new Vector3f(-6, 0, -20),

 	new Vector3f(0, 0, -26),

 	new Vector3f(6, 0, -20),

 	// loop 2

 	new Vector3f(5, 0, -5),

 	new Vector3f(7, 0, 1.5f),

 	new Vector3f(14, 0, 2),

 	new Vector3f(20, 0, 6),

 	new Vector3f(26, 0, 0),

 	new Vector3f(20, 0, -6),

 	new Vector3f(14, 0, 0),

 	new Vector3f(20, 0, 6),

 	new Vector3f(26, 0, 0),

 	new Vector3f(20, 0, -6),

 	new Vector3f(14, 0, 0),

 	// loop 3

 	new Vector3f(8, 0, 7.5f),

 	new Vector3f(7, 0, 10.5f),

 	new Vector3f(6, 0, 20),

 	new Vector3f(0, 0, 26),

 	new Vector3f(-6, 0, 20),

 	new Vector3f(0, 0, 14),

 	new Vector3f(6, 0, 20),

 	new Vector3f(0, 0, 26),

 	new Vector3f(-6, 0, 20),

 	new Vector3f(0, 0, 14),

 	// begin ellipse

 	new Vector3f(16, 5, 20),

 	new Vector3f(0, 0, 26),

 	new Vector3f(-16, -10, 20),

 	new Vector3f(0, 0, 14),

 	new Vector3f(16, 20, 20),

 	new Vector3f(0, 0, 26),

 	new Vector3f(-10, -25, 10),

 	new Vector3f(-10, 0, 0),

 	// come at me!

 	new Vector3f(-28.00242f, 48.005623f, -34.648228f),

 	new Vector3f(0, 0 , -20),

     };

 

     private void createScene() {

 	Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");

 	bell = new Geometry( "sound-emitter" , new Sphere(15,15,1));

 	mat.setColor("Color", ColorRGBA.Blue);

 	bell.setMaterial(mat);

 	rootNode.attachChild(bell);

 

 	ear1 = makeEar(rootNode, new Vector3f(0, 0 ,-20));

 	ear2 = makeEar(rootNode, new Vector3f(0, 0 ,20));

 	ear3 = makeEar(rootNode, new Vector3f(20, 0 ,0));

 

 	MotionPath track = new MotionPath();

 

 	for (Vector3f v : path){

 	    track.addWayPoint(v);

 	}

 	track.setCurveTension(0.80f);

 

 	motionControl = new MotionTrack(bell,track);

 		

 	// for now, use reflection to change the timer... 

 	// motionControl.setTimer(new IsoTimer(60));

 	try {

 	    Field timerField;

 	    timerField = AbstractCinematicEvent.class.getDeclaredField("timer");

 	    timerField.setAccessible(true);

 	    try {timerField.set(motionControl, new IsoTimer(60));} 

 	    catch (IllegalArgumentException e) {e.printStackTrace();} 

 	    catch (IllegalAccessException e) {e.printStackTrace();}

 	} 

 	catch (SecurityException e) {e.printStackTrace();} 

 	catch (NoSuchFieldException e) {e.printStackTrace();}

 		

 	motionControl.setDirectionType(MotionTrack.Direction.PathAndRotation);

 	motionControl.setRotation(new Quaternion().fromAngleNormalAxis(-FastMath.HALF_PI, Vector3f.UNIT_Y));

 	motionControl.setInitialDuration(20f);

 	motionControl.setSpeed(1f);

 

 	track.enableDebugShape(assetManager, rootNode);

 	positionCamera();

     }

 

 

     private void positionCamera(){

 	this.cam.setLocation(new Vector3f(-28.00242f, 48.005623f, -34.648228f));

 	this.cam.setRotation(new Quaternion(0.3359635f, 0.34280345f, -0.13281013f, 0.8671653f));

     }

 

     private void initAudio() {

 	org.lwjgl.input.Mouse.setGrabbed(false);	

 	music = new AudioNode(assetManager, "Sound/Effects/Beep.ogg", false);

 

 	rootNode.attachChild(music);

 	audioRenderer.playSource(music);

 	music.setPositional(true);

 	music.setVolume(1f);

 	music.setReverbEnabled(false);

 	music.setDirectional(false);

 	music.setMaxDistance(200.0f);

 	music.setRefDistance(1f);

 	//music.setRolloffFactor(1f);

 	music.setLooping(false);

 	audioRenderer.pauseSource(music); 

     }

 

     public class Dancer implements SoundProcessor {

 	Geometry entity;

 	float scale = 2;

 	public Dancer(Geometry entity){

 	    this.entity = entity;

 	}

 

 	/**

 	 * this method is irrelevant since there is no state to cleanup.

 	 */

 	public void cleanup() {}

 

 

 	/**

 	 * Respond to sound!  This is the brain of an AI entity that 

 	 * hears it's surroundings and reacts to them.

 	 */

 	public void process(ByteBuffer audioSamples, int numSamples, AudioFormat format) {

 	    audioSamples.clear();

 	    byte[] data = new byte[numSamples];

 	    float[] out = new float[numSamples];

 	    audioSamples.get(data);

 	    FloatSampleTools.byte2floatInterleaved(data, 0, out, 0, 

 						   numSamples/format.getFrameSize(), format);

 

 	    float max = Float.NEGATIVE_INFINITY;

 	    for (float f : out){if (f > max) max = f;}

 	    audioSamples.clear();

 

 	    if (max > 0.1){entity.getMaterial().setColor("Color", ColorRGBA.Green);}

 	    else {entity.getMaterial().setColor("Color", ColorRGBA.Gray);}

 	}

     }

 

     private void prepareEar(Geometry ear, int n){

 	if (this.audioRenderer instanceof MultiListener){

 	    MultiListener rf = (MultiListener)this.audioRenderer;

 

 	    Listener auxListener = new Listener();

 	    auxListener.setLocation(ear.getLocalTranslation());

 

 	    rf.addListener(auxListener);

 	    WaveFileWriter aux = null;

 

 	    try {aux = new WaveFileWriter(new File("/home/r/tmp/ear"+n+".wav"));} 

 	    catch (FileNotFoundException e) {e.printStackTrace();}

 

 	    rf.registerSoundProcessor(auxListener, 

 				      new CompositeSoundProcessor(new Dancer(ear), aux));

 	}   

     }

 

 

     public void simpleInitApp() {

 	this.setTimer(new IsoTimer(60));

 	initAudio();

 		

 	createScene();

 

 	prepareEar(ear1, 1);

 	prepareEar(ear2, 1);

 	prepareEar(ear3, 1);

 

 	motionControl.play();

     }

 

     public void simpleUpdate(float tpf) {

 	if (music.getStatus() != AudioNode.Status.Playing){

 	    music.play();

 	}

 	Vector3f loc = cam.getLocation();

 	Quaternion rot = cam.getRotation();

 	listener.setLocation(loc);

 	listener.setRotation(rot);

 	music.setLocalTranslation(bell.getLocalTranslation());

     }

 

 }

 </code>

 

 {{http://www.youtube.com/v/oCEfK0yhDrY?.swf?400×333}}

 

 ===== More Information =====

 

 This is the old page showing the first version of this idea

 http://aurellem.org/cortex/html/capture-video.html

 

 All source code can be found here:

 

 http://hg.bortreb.com/audio-send

 

 http://hg.bortreb.com/jmeCapture

  

 More information on the modifications to OpenAL to support multiple

 listeners can be found here.

 

 http://aurellem.org/audio-send/html/ear.html