1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/MIT-media-projects.org	Fri Jul 20 11:21:04 2012 -0500
     1.3 @@ -0,0 +1,21 @@
     1.4 +*Machine Learning and Pattern Recognition with Multiple Modalities
     1.5 +Hyungil Ahn and Rosalind W. Picard
     1.6 +
     1.7 +This project develops new theory and algorithms to enable computers to
     1.8 +make rapid and accurate inferences from multiple modes of data, such
     1.9 +as determining a person's affective state from multiple sensors—video,
    1.10 +mouse behavior, chair pressure patterns, typed selections, or
    1.11 +physiology. Recent efforts focus on understanding the level of a
    1.12 +person's attention, useful for things such as determining when to
    1.13 +interrupt. Our approach is Bayesian: formulating probabilistic models
    1.14 +on the basis of domain knowledge and training data, and then
    1.15 +performing inference according to the rules of probability
    1.16 +theory. This type of sensor fusion work is especially challenging due
    1.17 +to problems of sensor channel drop-out, different kinds of noise in
    1.18 +different channels, dependence between channels, scarce and sometimes
    1.19 +inaccurate labels, and patterns to detect that are inherently
    1.20 +time-varying. We have constructed a variety of new algorithms for
    1.21 +solving these problems and demonstrated their performance gains over
    1.22 +other state-of-the-art methods.
    1.23 +
    1.24 +http://affect.media.mit.edu/projectpages/multimodal/
    1.25 \ No newline at end of file

     2.1 --- a/org/intro.org	Thu Jul 19 19:38:45 2012 -0500
     2.2 +++ b/org/intro.org	Fri Jul 20 11:21:04 2012 -0500
     2.3 @@ -8,170 +8,193 @@
     2.4  #+babel: :mkdirp yes :noweb yes
     2.5  
     2.6  * Background
     2.7 -Artificial Intelligence has tried and failed for more than half a
     2.8 -century to produce programs as flexible, creative, and "intelligent"
     2.9 -as the human mind itself. Clearly, we are still missing some important
    2.10 -ideas concerning intelligent programs or we would have strong AI
    2.11 -already. What idea could be missing?
    2.12 +
    2.13 +Artificial Intelligence has tried and failed for more than
    2.14 +half a century to produce programs as flexible, creative,
    2.15 +and "intelligent" as the human mind itself. Clearly, we are
    2.16 +still missing some important ideas concerning intelligent
    2.17 +programs or we would have strong AI already. What idea could
    2.18 +be missing?
    2.19  
    2.20  When Turing first proposed his famous "Turing Test" in the
    2.21 -groundbreaking paper [[../sources/turing.pdf][/Computing Machines and Intelligence/]], he gave
    2.22 -little importance to how a computer program might interact with the
    2.23 -world:
    2.24 +groundbreaking paper [[../sources/turing.pdf][/Computing Machines and Intelligence/]],
    2.25 +he gave little importance to how a computer program might
    2.26 +interact with the world:
    2.27  
    2.28  #+BEGIN_QUOTE
    2.29 -\ldquo{}We need not be too concerned about the legs, eyes, etc. The example of
    2.30 -Miss Helen Keller shows that education can take place provided that
    2.31 -communication in both directions between teacher and pupil can take
    2.32 -place by some means or other.\rdquo{}
    2.33 +\ldquo{}We need not be too concerned about the legs, eyes,
    2.34 +etc. The example of Miss Helen Keller shows that education
    2.35 +can take place provided that communication in both
    2.36 +directions between teacher and pupil can take place by some
    2.37 +means or other.\rdquo{}
    2.38  #+END_QUOTE
    2.39  
    2.40 -And from the example of Hellen Keller he went on to assume that the
    2.41 -only thing a fledgling AI program could need by way of communication
    2.42 -is a teletypewriter. But Hellen Keller did possess vision and hearing
    2.43 -for the first few months of her life, and her tactile sense was far
    2.44 -more rich than any text-stream could hope to achieve. She possessed a
    2.45 -body she could move freely, and had continual access to the real world
    2.46 -to learn from her actions.
    2.47 +And from the example of Hellen Keller he went on to assume
    2.48 +that the only thing a fledgling AI program could need by way
    2.49 +of communication is a teletypewriter. But Hellen Keller did
    2.50 +possess vision and hearing for the first few months of her
    2.51 +life, and her tactile sense was far more rich than any
    2.52 +text-stream could hope to achieve. She possessed a body she
    2.53 +could move freely, and had continual access to the real
    2.54 +world to learn from her actions.
    2.55  
    2.56 -I believe that our programs are suffering from too little sensory
    2.57 -input to become really intelligent. Imagine for a moment that you
    2.58 -lived in a world completely cut off form all sensory stimulation. You
    2.59 -have no eyes to see, no ears to hear, no mouth to speak. No body, no
    2.60 -taste, no feeling whatsoever. The only sense you get at all is a
    2.61 -single point of light, flickering on and off in the void. If this was
    2.62 -your life from birth, you would never learn anything, and could never
    2.63 -become intelligent. Actual humans placed in sensory deprivation
    2.64 -chambers experience hallucinations and can begin to loose their sense
    2.65 -of reality. Most of the time, the programs we write are in exactly
    2.66 -this situation. They do not interface with cameras and microphones,
    2.67 -and they do not control a real or simulated body or interact with any
    2.68 -sort of world.
    2.69 +I believe that our programs are suffering from too little
    2.70 +sensory input to become really intelligent. Imagine for a
    2.71 +moment that you lived in a world completely cut off form all
    2.72 +sensory stimulation. You have no eyes to see, no ears to
    2.73 +hear, no mouth to speak. No body, no taste, no feeling
    2.74 +whatsoever. The only sense you get at all is a single point
    2.75 +of light, flickering on and off in the void. If this was
    2.76 +your life from birth, you would never learn anything, and
    2.77 +could never become intelligent. Actual humans placed in
    2.78 +sensory deprivation chambers experience hallucinations and
    2.79 +can begin to loose their sense of reality. Most of the time,
    2.80 +the programs we write are in exactly this situation. They do
    2.81 +not interface with cameras and microphones, and they do not
    2.82 +control a real or simulated body or interact with any sort
    2.83 +of world.
    2.84  
    2.85  * Simulation vs. Reality
    2.86 +
    2.87  I want demonstrate that multiple senses are what enable
    2.88 -intelligence. There are two ways of playing around with senses and
    2.89 -computer programs:
    2.90 -
    2.91 +intelligence. There are two ways of playing around with
    2.92 +senses and computer programs:
    2.93  
    2.94  ** Simulation
    2.95 -The first is to go entirely with simulation: virtual world, virtual
    2.96 -character, virtual senses. The advantages are that when everything is
    2.97 -a simulation, experiments in that simulation are absolutely
    2.98 -reproducible. It's also easier to change the character and world to
    2.99 -explore new situations and different sensory combinations.
   2.100  
   2.101 -If the world is to be simulated on a computer, then not only do you
   2.102 -have to worry about whether the character's senses are rich enough to
   2.103 -learn from the world, but whether the world itself is rendered with
   2.104 -enough detail and realism to give enough working material to the
   2.105 -character's senses. To name just a few difficulties facing modern
   2.106 -physics simulators: destructibility of the environment, simulation of
   2.107 -water/other fluids, large areas, nonrigid bodies, lots of objects,
   2.108 -smoke. I don't know of any computer simulation that would allow a
   2.109 -character to take a rock and grind it into fine dust, then use that
   2.110 -dust to make a clay sculpture, at least not without spending years
   2.111 -calculating the interactions of every single small grain of
   2.112 -dust. Maybe a simulated world with today's limitations doesn't provide
   2.113 +The first is to go entirely with simulation: virtual world,
   2.114 +virtual character, virtual senses. The advantages are that
   2.115 +when everything is a simulation, experiments in that
   2.116 +simulation are absolutely reproducible. It's also easier to
   2.117 +change the character and world to explore new situations and
   2.118 +different sensory combinations.
   2.119 +
   2.120 +If the world is to be simulated on a computer, then not only
   2.121 +do you have to worry about whether the character's senses
   2.122 +are rich enough to learn from the world, but whether the
   2.123 +world itself is rendered with enough detail and realism to
   2.124 +give enough working material to the character's senses. To
   2.125 +name just a few difficulties facing modern physics
   2.126 +simulators: destructibility of the environment, simulation
   2.127 +of water/other fluids, large areas, nonrigid bodies, lots of
   2.128 +objects, smoke. I don't know of any computer simulation that
   2.129 +would allow a character to take a rock and grind it into
   2.130 +fine dust, then use that dust to make a clay sculpture, at
   2.131 +least not without spending years calculating the
   2.132 +interactions of every single small grain of dust. Maybe a
   2.133 +simulated world with today's limitations doesn't provide
   2.134  enough richness for real intelligence to evolve.
   2.135  
   2.136  ** Reality
   2.137  
   2.138 -The other approach for playing with senses is to hook your software up
   2.139 -to real cameras, microphones, robots, etc., and let it loose in the
   2.140 -real world. This has the advantage of eliminating concerns about
   2.141 -simulating the world at the expense of increasing the complexity of
   2.142 -implementing the senses. Instead of just grabbing the current rendered
   2.143 -frame for processing, you have to use an actual camera with real
   2.144 -lenses and interact with photons to get an image. It is much harder to
   2.145 -change the character, which is now partly a physical robot of some
   2.146 -sort, since doing so involves changing things around in the real world
   2.147 -instead of modifying lines of code. While the real world is very rich
   2.148 -and definitely provides enough stimulation for intelligence to develop
   2.149 -as evidenced by our own existence, it is also uncontrollable in the
   2.150 -sense that a particular situation cannot be recreated perfectly or
   2.151 -saved for later use. It is harder to conduct science because it is
   2.152 -harder to repeat an experiment. The worst thing about using the real
   2.153 -world instead of a simulation is the matter of time. Instead of
   2.154 -simulated time you get the constant and unstoppable flow of real
   2.155 -time. This severely limits the sorts of software you can use to
   2.156 -program the AI because all sense inputs must be handled in real
   2.157 -time. Complicated ideas may have to be implemented in hardware or may
   2.158 -simply be impossible given the current speed of our
   2.159 -processors. Contrast this with a simulation, in which the flow of time
   2.160 -in the simulated world can be slowed down to accommodate the
   2.161 -limitations of the character's programming. In terms of cost, doing
   2.162 -everything in software is far cheaper than building custom real-time
   2.163 +The other approach for playing with senses is to hook your
   2.164 +software up to real cameras, microphones, robots, etc., and
   2.165 +let it loose in the real world. This has the advantage of
   2.166 +eliminating concerns about simulating the world at the
   2.167 +expense of increasing the complexity of implementing the
   2.168 +senses. Instead of just grabbing the current rendered frame
   2.169 +for processing, you have to use an actual camera with real
   2.170 +lenses and interact with photons to get an image. It is much
   2.171 +harder to change the character, which is now partly a
   2.172 +physical robot of some sort, since doing so involves
   2.173 +changing things around in the real world instead of
   2.174 +modifying lines of code. While the real world is very rich
   2.175 +and definitely provides enough stimulation for intelligence
   2.176 +to develop as evidenced by our own existence, it is also
   2.177 +uncontrollable in the sense that a particular situation
   2.178 +cannot be recreated perfectly or saved for later use. It is
   2.179 +harder to conduct science because it is harder to repeat an
   2.180 +experiment. The worst thing about using the real world
   2.181 +instead of a simulation is the matter of time. Instead of
   2.182 +simulated time you get the constant and unstoppable flow of
   2.183 +real time. This severely limits the sorts of software you
   2.184 +can use to program the AI because all sense inputs must be
   2.185 +handled in real time. Complicated ideas may have to be
   2.186 +implemented in hardware or may simply be impossible given
   2.187 +the current speed of our processors. Contrast this with a
   2.188 +simulation, in which the flow of time in the simulated world
   2.189 +can be slowed down to accommodate the limitations of the
   2.190 +character's programming. In terms of cost, doing everything
   2.191 +in software is far cheaper than building custom real-time
   2.192  hardware. All you need is a laptop and some patience.
   2.193  
   2.194  * Choose a Simulation Engine
   2.195  
   2.196 -Mainly because of issues with controlling the flow of time, I chose to
   2.197 -simulate both the world and the character. I set out to make a world
   2.198 -in which I could embed a character with multiple senses. My main goal
   2.199 -is to make an environment where I can perform further experiments in
   2.200 -simulated senses.
   2.201 +Mainly because of issues with controlling the flow of time,
   2.202 +I chose to simulate both the world and the character. I set
   2.203 +out to make a world in which I could embed a character with
   2.204 +multiple senses. My main goal is to make an environment
   2.205 +where I can perform further experiments in simulated senses.
   2.206  
   2.207 -I examined many different 3D environments to try and find something I
   2.208 -would use as the base for my simulation; eventually the choice came
   2.209 -down to three engines: the Quake II engine, the Source Engine, and
   2.210 -jMonkeyEngine.
   2.211 +I examined many different 3D environments to try and find
   2.212 +something I would use as the base for my simulation;
   2.213 +eventually the choice came down to three engines: the Quake
   2.214 +II engine, the Source Engine, and jMonkeyEngine.
   2.215  
   2.216  ** [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]]
   2.217  
   2.218 -I spent a bit more than a month working with the Quake II Engine from
   2.219 -ID software to see if I could use it for my purposes. All the source
   2.220 -code was released by ID software into the Public Domain several years
   2.221 -ago, and as a result it has been ported and modified for many
   2.222 -different reasons. This engine was famous for its advanced use of
   2.223 +I spent a bit more than a month working with the Quake II
   2.224 +Engine from ID software to see if I could use it for my
   2.225 +purposes. All the source code was released by ID software
   2.226 +into the Public Domain several years ago, and as a result it
   2.227 +has been ported and modified for many different
   2.228 +reasons. This engine was famous for its advanced use of
   2.229  realistic shading and had decent and fast physics
   2.230 -simulation. Researchers at Princeton [[http://papers.cnl.salk.edu/PDFs/Intracelllular%20Dynamics%20of%20Virtual%20Place%20Cells%202011-4178.pdf][used this code]] ([[http://brainwindows.wordpress.com/2009/10/14/playing-quake-with-a-real-mouse/][video]]) to study
   2.231 -spatial information encoding in the hippocampal cells of rats. Those
   2.232 -researchers created a special Quake II level that simulated a maze,
   2.233 -and added an interface where a mouse could run on top of a ball in
   2.234 -various directions to move the character in the simulated maze. They
   2.235 -measured hippocampal activity during this exercise to try and tease
   2.236 -out the method in which spatial data was stored in that area of the
   2.237 -brain. I find this promising because if a real living rat can interact
   2.238 -with a computer simulation of a maze in the same way as it interacts
   2.239 -with a real-world maze, then maybe that simulation is close enough to
   2.240 -reality that a simulated sense of vision and motor control interacting
   2.241 -with that simulation could reveal useful information about the real
   2.242 -thing. There is a Java port of the original C source code called
   2.243 -Jake2. The port demonstrates Java's OpenGL bindings and runs anywhere
   2.244 -from 90% to 105% as fast as the C version. After reviewing much of the
   2.245 -source of Jake2, I eventually rejected it because the engine is too
   2.246 -tied to the concept of a first-person shooter game. One of the
   2.247 -problems I had was that there do not seem to be any easy way to attach
   2.248 -multiple cameras to a single character. There are also several physics
   2.249 -clipping issues that are corrected in a way that only applies to the
   2.250 -main character and does not apply to arbitrary objects. While there is
   2.251 -a large community of level modders, I couldn't find a community to
   2.252 -support using the engine to make new things.
   2.253 +simulation. Researchers at Princeton [[http://papers.cnl.salk.edu/PDFs/Intracelllular%20Dynamics%20of%20Virtual%20Place%20Cells%202011-4178.pdf][used this code]] ([[http://brainwindows.wordpress.com/2009/10/14/playing-quake-with-a-real-mouse/][video]])
   2.254 +to study spatial information encoding in the hippocampal
   2.255 +cells of rats. Those researchers created a special Quake II
   2.256 +level that simulated a maze, and added an interface where a
   2.257 +mouse could run on top of a ball in various directions to
   2.258 +move the character in the simulated maze. They measured
   2.259 +hippocampal activity during this exercise to try and tease
   2.260 +out the method in which spatial data was stored in that area
   2.261 +of the brain. I find this promising because if a real living
   2.262 +rat can interact with a computer simulation of a maze in the
   2.263 +same way as it interacts with a real-world maze, then maybe
   2.264 +that simulation is close enough to reality that a simulated
   2.265 +sense of vision and motor control interacting with that
   2.266 +simulation could reveal useful information about the real
   2.267 +thing. There is a Java port of the original C source code
   2.268 +called Jake2. The port demonstrates Java's OpenGL bindings
   2.269 +and runs anywhere from 90% to 105% as fast as the C
   2.270 +version. After reviewing much of the source of Jake2, I
   2.271 +rejected it because the engine is too tied to the concept of
   2.272 +a first-person shooter game. One of the problems I had was
   2.273 +that there does not seem to be any easy way to attach
   2.274 +multiple cameras to a single character. There are also
   2.275 +several physics clipping issues that are corrected in a way
   2.276 +that only applies to the main character and do not apply to
   2.277 +arbitrary objects. While there is a large community of level
   2.278 +modders, I couldn't find a community to support using the
   2.279 +engine to make new things.
   2.280  
   2.281  ** [[http://source.valvesoftware.com/][Source Engine]]
   2.282  
   2.283 -The Source Engine evolved from the Quake II and Quake I engines and is
   2.284 -used by Valve in the Half-Life series of games. The physics simulation
   2.285 -in the Source Engine is quite accurate and probably the best out of
   2.286 -all the engines I investigated. There is also an extensive community
   2.287 -actively working with the engine. However, applications that use the
   2.288 -Source Engine must be written in C++, the code is not open, it only
   2.289 -runs on Windows, and the tools that come with the SDK to handle models
   2.290 -and textures are complicated and awkward to use.
   2.291 +The Source Engine evolved from the Quake II and Quake I
   2.292 +engines and is used by Valve in the Half-Life series of
   2.293 +games. The physics simulation in the Source Engine is quite
   2.294 +accurate and probably the best out of all the engines I
   2.295 +investigated. There is also an extensive community actively
   2.296 +working with the engine. However, applications that use the
   2.297 +Source Engine must be written in C++, the code is not open,
   2.298 +it only runs on Windows, and the tools that come with the
   2.299 +SDK to handle models and textures are complicated and
   2.300 +awkward to use.
   2.301  
   2.302  ** [[http://jmonkeyengine.com/][jMonkeyEngine3]]
   2.303  
   2.304 -jMonkeyEngine is a new library for creating games in Java. It uses
   2.305 -OpenGL to render to the screen and uses screengraphs to avoid drawing
   2.306 -things that do not appear on the screen. It has an active community
   2.307 -and several games in the pipeline. The engine was not built to serve
   2.308 -any particular game but is instead meant to be used for any 3D
   2.309 -game. After experimenting with each of these three engines and a few
   2.310 -others for about 2 months I settled on jMonkeyEngine. I chose it
   2.311 -because it had the most features out of all the open projects I looked
   2.312 -at, and because I could then write my code in Clojure, an
   2.313 -implementation of LISP that runs on the JVM.
   2.314 +jMonkeyEngine is a new library for creating games in
   2.315 +Java. It uses OpenGL to render to the screen and uses
   2.316 +screengraphs to avoid drawing things that do not appear on
   2.317 +the screen. It has an active community and several games in
   2.318 +the pipeline. The engine was not built to serve any
   2.319 +particular game but is instead meant to be used for any 3D
   2.320 +game. After experimenting with each of these three engines
   2.321 +and a few others for about 2 months I settled on
   2.322 +jMonkeyEngine. I chose it because it had the most features
   2.323 +out of all the open projects I looked at, and because I
   2.324 +could then write my code in Clojure, an implementation of
   2.325 +LISP that runs on the JVM.
   2.326  
   2.327  
   2.328  

     3.1 --- a/org/thesis.org	Thu Jul 19 19:38:45 2012 -0500
     3.2 +++ b/org/thesis.org	Fri Jul 20 11:21:04 2012 -0500
     3.3 @@ -9,7 +9,7 @@
     3.4  #+include: "./intro.org"
     3.5  
     3.6  * Virtual reality is vastly easier than actual reality.
     3.7 -
     3.8 +  
     3.9  * There is no framework for AI experimenmts with multiple senses.
    3.10  
    3.11  * Cortex is my attempt at such a framework.
    3.12 @@ -51,6 +51,6 @@
    3.13  * Rat in a Maze
    3.14  
    3.15  * Swarm Creatures
    3.16 -
    3.17 +  
    3.18  * Simulated Imagination
    3.19