# HG changeset patch # User Robert McIntyre # Date 1342801264 18000 # Node ID f4ef73370da1e9f169814192eb06f0e1a5842ca2 # Parent 698d48b91cd50449da6ebd0abec06742a4649742 add file. diff -r 698d48b91cd5 -r f4ef73370da1 MIT-media-projects.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/MIT-media-projects.org Fri Jul 20 11:21:04 2012 -0500 @@ -0,0 +1,21 @@ +*Machine Learning and Pattern Recognition with Multiple Modalities +Hyungil Ahn and Rosalind W. Picard + +This project develops new theory and algorithms to enable computers to +make rapid and accurate inferences from multiple modes of data, such +as determining a person's affective state from multiple sensors—video, +mouse behavior, chair pressure patterns, typed selections, or +physiology. Recent efforts focus on understanding the level of a +person's attention, useful for things such as determining when to +interrupt. Our approach is Bayesian: formulating probabilistic models +on the basis of domain knowledge and training data, and then +performing inference according to the rules of probability +theory. This type of sensor fusion work is especially challenging due +to problems of sensor channel drop-out, different kinds of noise in +different channels, dependence between channels, scarce and sometimes +inaccurate labels, and patterns to detect that are inherently +time-varying. We have constructed a variety of new algorithms for +solving these problems and demonstrated their performance gains over +other state-of-the-art methods. + +http://affect.media.mit.edu/projectpages/multimodal/ \ No newline at end of file diff -r 698d48b91cd5 -r f4ef73370da1 org/intro.org --- a/org/intro.org Thu Jul 19 19:38:45 2012 -0500 +++ b/org/intro.org Fri Jul 20 11:21:04 2012 -0500 @@ -8,170 +8,193 @@ #+babel: :mkdirp yes :noweb yes * Background -Artificial Intelligence has tried and failed for more than half a -century to produce programs as flexible, creative, and "intelligent" -as the human mind itself. Clearly, we are still missing some important -ideas concerning intelligent programs or we would have strong AI -already. What idea could be missing? + +Artificial Intelligence has tried and failed for more than +half a century to produce programs as flexible, creative, +and "intelligent" as the human mind itself. Clearly, we are +still missing some important ideas concerning intelligent +programs or we would have strong AI already. What idea could +be missing? When Turing first proposed his famous "Turing Test" in the -groundbreaking paper [[../sources/turing.pdf][/Computing Machines and Intelligence/]], he gave -little importance to how a computer program might interact with the -world: +groundbreaking paper [[../sources/turing.pdf][/Computing Machines and Intelligence/]], +he gave little importance to how a computer program might +interact with the world: #+BEGIN_QUOTE -\ldquo{}We need not be too concerned about the legs, eyes, etc. The example of -Miss Helen Keller shows that education can take place provided that -communication in both directions between teacher and pupil can take -place by some means or other.\rdquo{} +\ldquo{}We need not be too concerned about the legs, eyes, +etc. The example of Miss Helen Keller shows that education +can take place provided that communication in both +directions between teacher and pupil can take place by some +means or other.\rdquo{} #+END_QUOTE -And from the example of Hellen Keller he went on to assume that the -only thing a fledgling AI program could need by way of communication -is a teletypewriter. But Hellen Keller did possess vision and hearing -for the first few months of her life, and her tactile sense was far -more rich than any text-stream could hope to achieve. She possessed a -body she could move freely, and had continual access to the real world -to learn from her actions. +And from the example of Hellen Keller he went on to assume +that the only thing a fledgling AI program could need by way +of communication is a teletypewriter. But Hellen Keller did +possess vision and hearing for the first few months of her +life, and her tactile sense was far more rich than any +text-stream could hope to achieve. She possessed a body she +could move freely, and had continual access to the real +world to learn from her actions. -I believe that our programs are suffering from too little sensory -input to become really intelligent. Imagine for a moment that you -lived in a world completely cut off form all sensory stimulation. You -have no eyes to see, no ears to hear, no mouth to speak. No body, no -taste, no feeling whatsoever. The only sense you get at all is a -single point of light, flickering on and off in the void. If this was -your life from birth, you would never learn anything, and could never -become intelligent. Actual humans placed in sensory deprivation -chambers experience hallucinations and can begin to loose their sense -of reality. Most of the time, the programs we write are in exactly -this situation. They do not interface with cameras and microphones, -and they do not control a real or simulated body or interact with any -sort of world. +I believe that our programs are suffering from too little +sensory input to become really intelligent. Imagine for a +moment that you lived in a world completely cut off form all +sensory stimulation. You have no eyes to see, no ears to +hear, no mouth to speak. No body, no taste, no feeling +whatsoever. The only sense you get at all is a single point +of light, flickering on and off in the void. If this was +your life from birth, you would never learn anything, and +could never become intelligent. Actual humans placed in +sensory deprivation chambers experience hallucinations and +can begin to loose their sense of reality. Most of the time, +the programs we write are in exactly this situation. They do +not interface with cameras and microphones, and they do not +control a real or simulated body or interact with any sort +of world. * Simulation vs. Reality + I want demonstrate that multiple senses are what enable -intelligence. There are two ways of playing around with senses and -computer programs: - +intelligence. There are two ways of playing around with +senses and computer programs: ** Simulation -The first is to go entirely with simulation: virtual world, virtual -character, virtual senses. The advantages are that when everything is -a simulation, experiments in that simulation are absolutely -reproducible. It's also easier to change the character and world to -explore new situations and different sensory combinations. -If the world is to be simulated on a computer, then not only do you -have to worry about whether the character's senses are rich enough to -learn from the world, but whether the world itself is rendered with -enough detail and realism to give enough working material to the -character's senses. To name just a few difficulties facing modern -physics simulators: destructibility of the environment, simulation of -water/other fluids, large areas, nonrigid bodies, lots of objects, -smoke. I don't know of any computer simulation that would allow a -character to take a rock and grind it into fine dust, then use that -dust to make a clay sculpture, at least not without spending years -calculating the interactions of every single small grain of -dust. Maybe a simulated world with today's limitations doesn't provide +The first is to go entirely with simulation: virtual world, +virtual character, virtual senses. The advantages are that +when everything is a simulation, experiments in that +simulation are absolutely reproducible. It's also easier to +change the character and world to explore new situations and +different sensory combinations. + +If the world is to be simulated on a computer, then not only +do you have to worry about whether the character's senses +are rich enough to learn from the world, but whether the +world itself is rendered with enough detail and realism to +give enough working material to the character's senses. To +name just a few difficulties facing modern physics +simulators: destructibility of the environment, simulation +of water/other fluids, large areas, nonrigid bodies, lots of +objects, smoke. I don't know of any computer simulation that +would allow a character to take a rock and grind it into +fine dust, then use that dust to make a clay sculpture, at +least not without spending years calculating the +interactions of every single small grain of dust. Maybe a +simulated world with today's limitations doesn't provide enough richness for real intelligence to evolve. ** Reality -The other approach for playing with senses is to hook your software up -to real cameras, microphones, robots, etc., and let it loose in the -real world. This has the advantage of eliminating concerns about -simulating the world at the expense of increasing the complexity of -implementing the senses. Instead of just grabbing the current rendered -frame for processing, you have to use an actual camera with real -lenses and interact with photons to get an image. It is much harder to -change the character, which is now partly a physical robot of some -sort, since doing so involves changing things around in the real world -instead of modifying lines of code. While the real world is very rich -and definitely provides enough stimulation for intelligence to develop -as evidenced by our own existence, it is also uncontrollable in the -sense that a particular situation cannot be recreated perfectly or -saved for later use. It is harder to conduct science because it is -harder to repeat an experiment. The worst thing about using the real -world instead of a simulation is the matter of time. Instead of -simulated time you get the constant and unstoppable flow of real -time. This severely limits the sorts of software you can use to -program the AI because all sense inputs must be handled in real -time. Complicated ideas may have to be implemented in hardware or may -simply be impossible given the current speed of our -processors. Contrast this with a simulation, in which the flow of time -in the simulated world can be slowed down to accommodate the -limitations of the character's programming. In terms of cost, doing -everything in software is far cheaper than building custom real-time +The other approach for playing with senses is to hook your +software up to real cameras, microphones, robots, etc., and +let it loose in the real world. This has the advantage of +eliminating concerns about simulating the world at the +expense of increasing the complexity of implementing the +senses. Instead of just grabbing the current rendered frame +for processing, you have to use an actual camera with real +lenses and interact with photons to get an image. It is much +harder to change the character, which is now partly a +physical robot of some sort, since doing so involves +changing things around in the real world instead of +modifying lines of code. While the real world is very rich +and definitely provides enough stimulation for intelligence +to develop as evidenced by our own existence, it is also +uncontrollable in the sense that a particular situation +cannot be recreated perfectly or saved for later use. It is +harder to conduct science because it is harder to repeat an +experiment. The worst thing about using the real world +instead of a simulation is the matter of time. Instead of +simulated time you get the constant and unstoppable flow of +real time. This severely limits the sorts of software you +can use to program the AI because all sense inputs must be +handled in real time. Complicated ideas may have to be +implemented in hardware or may simply be impossible given +the current speed of our processors. Contrast this with a +simulation, in which the flow of time in the simulated world +can be slowed down to accommodate the limitations of the +character's programming. In terms of cost, doing everything +in software is far cheaper than building custom real-time hardware. All you need is a laptop and some patience. * Choose a Simulation Engine -Mainly because of issues with controlling the flow of time, I chose to -simulate both the world and the character. I set out to make a world -in which I could embed a character with multiple senses. My main goal -is to make an environment where I can perform further experiments in -simulated senses. +Mainly because of issues with controlling the flow of time, +I chose to simulate both the world and the character. I set +out to make a world in which I could embed a character with +multiple senses. My main goal is to make an environment +where I can perform further experiments in simulated senses. -I examined many different 3D environments to try and find something I -would use as the base for my simulation; eventually the choice came -down to three engines: the Quake II engine, the Source Engine, and -jMonkeyEngine. +I examined many different 3D environments to try and find +something I would use as the base for my simulation; +eventually the choice came down to three engines: the Quake +II engine, the Source Engine, and jMonkeyEngine. ** [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] -I spent a bit more than a month working with the Quake II Engine from -ID software to see if I could use it for my purposes. All the source -code was released by ID software into the Public Domain several years -ago, and as a result it has been ported and modified for many -different reasons. This engine was famous for its advanced use of +I spent a bit more than a month working with the Quake II +Engine from ID software to see if I could use it for my +purposes. All the source code was released by ID software +into the Public Domain several years ago, and as a result it +has been ported and modified for many different +reasons. This engine was famous for its advanced use of realistic shading and had decent and fast physics -simulation. Researchers at Princeton [[http://papers.cnl.salk.edu/PDFs/Intracelllular%20Dynamics%20of%20Virtual%20Place%20Cells%202011-4178.pdf][used this code]] ([[http://brainwindows.wordpress.com/2009/10/14/playing-quake-with-a-real-mouse/][video]]) to study -spatial information encoding in the hippocampal cells of rats. Those -researchers created a special Quake II level that simulated a maze, -and added an interface where a mouse could run on top of a ball in -various directions to move the character in the simulated maze. They -measured hippocampal activity during this exercise to try and tease -out the method in which spatial data was stored in that area of the -brain. I find this promising because if a real living rat can interact -with a computer simulation of a maze in the same way as it interacts -with a real-world maze, then maybe that simulation is close enough to -reality that a simulated sense of vision and motor control interacting -with that simulation could reveal useful information about the real -thing. There is a Java port of the original C source code called -Jake2. The port demonstrates Java's OpenGL bindings and runs anywhere -from 90% to 105% as fast as the C version. After reviewing much of the -source of Jake2, I eventually rejected it because the engine is too -tied to the concept of a first-person shooter game. One of the -problems I had was that there do not seem to be any easy way to attach -multiple cameras to a single character. There are also several physics -clipping issues that are corrected in a way that only applies to the -main character and does not apply to arbitrary objects. While there is -a large community of level modders, I couldn't find a community to -support using the engine to make new things. +simulation. Researchers at Princeton [[http://papers.cnl.salk.edu/PDFs/Intracelllular%20Dynamics%20of%20Virtual%20Place%20Cells%202011-4178.pdf][used this code]] ([[http://brainwindows.wordpress.com/2009/10/14/playing-quake-with-a-real-mouse/][video]]) +to study spatial information encoding in the hippocampal +cells of rats. Those researchers created a special Quake II +level that simulated a maze, and added an interface where a +mouse could run on top of a ball in various directions to +move the character in the simulated maze. They measured +hippocampal activity during this exercise to try and tease +out the method in which spatial data was stored in that area +of the brain. I find this promising because if a real living +rat can interact with a computer simulation of a maze in the +same way as it interacts with a real-world maze, then maybe +that simulation is close enough to reality that a simulated +sense of vision and motor control interacting with that +simulation could reveal useful information about the real +thing. There is a Java port of the original C source code +called Jake2. The port demonstrates Java's OpenGL bindings +and runs anywhere from 90% to 105% as fast as the C +version. After reviewing much of the source of Jake2, I +rejected it because the engine is too tied to the concept of +a first-person shooter game. One of the problems I had was +that there does not seem to be any easy way to attach +multiple cameras to a single character. There are also +several physics clipping issues that are corrected in a way +that only applies to the main character and do not apply to +arbitrary objects. While there is a large community of level +modders, I couldn't find a community to support using the +engine to make new things. ** [[http://source.valvesoftware.com/][Source Engine]] -The Source Engine evolved from the Quake II and Quake I engines and is -used by Valve in the Half-Life series of games. The physics simulation -in the Source Engine is quite accurate and probably the best out of -all the engines I investigated. There is also an extensive community -actively working with the engine. However, applications that use the -Source Engine must be written in C++, the code is not open, it only -runs on Windows, and the tools that come with the SDK to handle models -and textures are complicated and awkward to use. +The Source Engine evolved from the Quake II and Quake I +engines and is used by Valve in the Half-Life series of +games. The physics simulation in the Source Engine is quite +accurate and probably the best out of all the engines I +investigated. There is also an extensive community actively +working with the engine. However, applications that use the +Source Engine must be written in C++, the code is not open, +it only runs on Windows, and the tools that come with the +SDK to handle models and textures are complicated and +awkward to use. ** [[http://jmonkeyengine.com/][jMonkeyEngine3]] -jMonkeyEngine is a new library for creating games in Java. It uses -OpenGL to render to the screen and uses screengraphs to avoid drawing -things that do not appear on the screen. It has an active community -and several games in the pipeline. The engine was not built to serve -any particular game but is instead meant to be used for any 3D -game. After experimenting with each of these three engines and a few -others for about 2 months I settled on jMonkeyEngine. I chose it -because it had the most features out of all the open projects I looked -at, and because I could then write my code in Clojure, an -implementation of LISP that runs on the JVM. +jMonkeyEngine is a new library for creating games in +Java. It uses OpenGL to render to the screen and uses +screengraphs to avoid drawing things that do not appear on +the screen. It has an active community and several games in +the pipeline. The engine was not built to serve any +particular game but is instead meant to be used for any 3D +game. After experimenting with each of these three engines +and a few others for about 2 months I settled on +jMonkeyEngine. I chose it because it had the most features +out of all the open projects I looked at, and because I +could then write my code in Clojure, an implementation of +LISP that runs on the JVM. diff -r 698d48b91cd5 -r f4ef73370da1 org/thesis.org --- a/org/thesis.org Thu Jul 19 19:38:45 2012 -0500 +++ b/org/thesis.org Fri Jul 20 11:21:04 2012 -0500 @@ -9,7 +9,7 @@ #+include: "./intro.org" * Virtual reality is vastly easier than actual reality. - + * There is no framework for AI experimenmts with multiple senses. * Cortex is my attempt at such a framework. @@ -51,6 +51,6 @@ * Rat in a Maze * Swarm Creatures - + * Simulated Imagination