1.1 --- a/thesis/abstract.org	Sun Mar 23 17:59:16 2014 -0400
     1.2 +++ b/thesis/abstract.org	Sun Mar 23 19:09:14 2014 -0400
     1.3 @@ -1,34 +1,23 @@
     1.4 -Here I explore the design and capabilities of my system (called
     1.5 -=CORTEX=) which enables experiments in /embodied artificial
     1.6 -intelligence/ -- that is, AI which uses a physical simulation of
     1.7 -reality accompanied by a simulated body to solve problems.
     1.8 +Here I demonstrate the power of using embodied artificial intelligence
     1.9 +to attack the /action recognition/ problem, which is the challenge of
    1.10 +recognizing actions performed by a creature given limited data about
    1.11 +the creature's actions, such as a video recording. I solve this problem
    1.12 +in the case of a worm-like creature performing actions such as curling
    1.13 +and wiggling.
    1.14  
    1.15 -In the first half of the thesis I describe the construction of
    1.16 -=CORTEX= and the rationale behind my architecture choices. =CORTEX= is
    1.17 -a complete platform for embodied AI research. It provides multiple
    1.18 -senses for simulated creatures, including vision, touch,
    1.19 -proprioception, muscle tension, and hearing. Each of these senses
    1.20 -provides a wealth of parameters that are biologically
    1.21 -inspired. =CORTEX= is able to simulate any number of creatures and
    1.22 -senses, and provides facilities for easily modeling and creating new
    1.23 -creatures. As a research platform it is more complete than any other
    1.24 -system currently available.
    1.25 +To attack the action recognition problem, I developed a computational
    1.26 +model of empathy which allows me to use simple, embodied
    1.27 +representations of actions (which require rich sensory data), even
    1.28 +when that sensory data is not actually available. The missing sense
    1.29 +data is ``imagined'' by the system by combining previous experiences
    1.30 +gained from unsupervised free play.
    1.31  
    1.32 -In the second half of the thesis I develop a computational model of
    1.33 -empathy, using =CORTEX= as a base. Empathy in this context is the
    1.34 -ability to observe another creature and infer what sorts of sensations
    1.35 -that creature is feeling. My empathy algorithm involves multiple
    1.36 -phases. First is free-play, where the creature moves around and gains
    1.37 -sensory experience. From this experience I construct a representation
    1.38 -of the creature's sensory state space, which I call \phi-space. Using
    1.39 -\phi-space, I construct an efficient function for enriching the
    1.40 -limited data that comes from observing another creature with a full
    1.41 -compliment of imagined sensory data based on previous experience. I
    1.42 -can then use the imagined sensory data to recognize what the observed
    1.43 -creature is doing and feeling, using straightforward embodied action
    1.44 -predicates. This is all demonstrated with using a simple worm-like
    1.45 -creature, and recognizing worm-actions in video.
    1.46 -
    1.47 -
    1.48 -empathy is important 
    1.49 -cortex tests that idea
    1.50 \ No newline at end of file
    1.51 +In order to build this empathic, action-recognizing system, I created
    1.52 +a program called =CORTEX=, which is a complete platform for embodied
    1.53 +AI research. It provides multiple senses for simulated creatures,
    1.54 +including vision, touch, proprioception, muscle tension, and
    1.55 +hearing. Each of these senses provides a wealth of parameters that are
    1.56 +biologically inspired. =CORTEX= is able to simulate any number of
    1.57 +creatures and senses, and provides facilities for easily modeling and
    1.58 +creating new creatures. As a research platform it is more complete
    1.59 +than any other system currently available.

     2.1 --- a/thesis/cortex.org	Sun Mar 23 17:59:16 2014 -0400
     2.2 +++ b/thesis/cortex.org	Sun Mar 23 19:09:14 2014 -0400
     2.3 @@ -4,86 +4,64 @@
     2.4  #+description: Using embodied AI to facilitate Artificial Imagination.
     2.5  #+keywords: AI, clojure, embodiment
     2.6  
     2.7 --- show hand
     2.8 +* Embodiment is a critical component of Intelligence
     2.9  
    2.10 -* Embodiment is a critical component to Intelligence
    2.11 +** Recognizing actions in video is extremely difficult
    2.12 +   cat drinking, mimes, leaning, common sense
    2.13  
    2.14 -* To explore embodiment, we need a world, body, and senses
    2.15 +** Embodiment is the the right language for the job
    2.16  
    2.17 -* Because of Time, simulation is perferable to reality
    2.18 +   a new possibility for the question ``what is a chair?'' -- it's the
    2.19 +   feeling of your butt on something and your knees bent, with your
    2.20 +   back muscles and legs relaxed.
    2.21  
    2.22 -* Video game engines are a great starting point
    2.23 +** =CORTEX= is a system for exploring embodiment
    2.24  
    2.25 -* Bodies are composed of segments connected by joints
    2.26 +   Hand integration demo
    2.27  
    2.28 -* Eyes reuse standard video game components
    2.29 +** =CORTEX= solves recognition problems using empathy
    2.30 +   
    2.31 +   worm empathy demo
    2.32  
    2.33 -* Hearing is hard; =CORTEX= does it right
    2.34 +** Overview
    2.35  
    2.36 -* Touch uses hundreds of hair-like elements
    2.37 +* Building =CORTEX=
    2.38  
    2.39 -* Proprioception is the force that makes everything ``real''
    2.40 +** To explore embodiment, we need a world, body, and senses
    2.41  
    2.42 -* Muscles are both effectors and sensors
    2.43 +** Because of Time, simulation is perferable to reality
    2.44  
    2.45 -* =CORTEX= brings complex creatures to life!
    2.46 +** Video game engines are a great starting point
    2.47  
    2.48 -* =CORTEX= enables many possiblities for further research
    2.49 +** Bodies are composed of segments connected by joints
    2.50  
    2.51 -* =CORTEX= User Guide
    2.52 +** Eyes reuse standard video game components
    2.53 +
    2.54 +** Hearing is hard; =CORTEX= does it right
    2.55 +
    2.56 +** Touch uses hundreds of hair-like elements
    2.57 +
    2.58 +** Proprioception is the force that makes everything ``real''
    2.59 +
    2.60 +** Muscles are both effectors and sensors
    2.61 +
    2.62 +** =CORTEX= brings complex creatures to life!
    2.63 +
    2.64 +** =CORTEX= enables many possiblities for further research
    2.65  
    2.66  * Empathy in a simulated worm
    2.67  
    2.68 -* Embodiment factors action recognition into managable parts
    2.69 +** Embodiment factors action recognition into managable parts
    2.70  
    2.71 -* Action recognition is easy with a full gamut of senses
    2.72 +** Action recognition is easy with a full gamut of senses
    2.73  
    2.74 -* Digression: bootstrapping with multiple senses
    2.75 +** Digression: bootstrapping with multiple senses
    2.76  
    2.77 -* \Phi-space describes the worm's experiences
    2.78 +** \Phi-space describes the worm's experiences
    2.79  
    2.80 -* Empathy is the process of tracing though \Phi-space 
    2.81 +** Empathy is the process of tracing though \Phi-space 
    2.82    
    2.83 -* Efficient action recognition via empathy
    2.84 -
    2.85 -* Contributions
    2.86 -
    2.87 -
    2.88 -* Vision 
    2.89 -
    2.90 -  System for understanding what the actors in a video are doing --
    2.91 -  Action Recognition.
    2.92 -  
    2.93 -  Separate action recognition into three components:
    2.94 -
    2.95 -  - free play
    2.96 -  - embodied action predicates
    2.97 -  - model alignment 
    2.98 -  - sensory imagination
    2.99 -
   2.100 -* Steps 
   2.101 -  
   2.102 - - Build cortex, a simulated environment for sensate AI
   2.103 -   - solid bodies w/ joints
   2.104 -   - vision
   2.105 -   - touch
   2.106 -   - vision
   2.107 -   - hearing
   2.108 -   - proprioception
   2.109 -   - muscle contraction
   2.110 -
   2.111 - - Build experimental framework for worm-actions
   2.112 -  - embodied stream predicates
   2.113 -  - \phi-space 
   2.114 -  - \phi-scan
   2.115 -
   2.116 -* News
   2.117 -  
   2.118 -  Experimental results:
   2.119 -
   2.120 -  - \phi-space actually works very well for the worm!
   2.121 -  - self organizing touch map
   2.122 -
   2.123 +** Efficient action recognition via empathy
   2.124  
   2.125  * Contributions
   2.126    - Built =CORTEX=, a comprehensive platform for embodied AI
   2.127 @@ -92,9 +70,30 @@
   2.128    - created a novel concept for action recognition by using artificial
   2.129      imagination. 
   2.130  
   2.131 +* =CORTEX= User Guide
   2.132  
   2.133  
   2.134  
   2.135 +In the second half of the thesis I develop a computational model of
   2.136 +empathy, using =CORTEX= as a base. Empathy in this context is the
   2.137 +ability to observe another creature and infer what sorts of sensations
   2.138 +that creature is feeling. My empathy algorithm involves multiple
   2.139 +phases. First is free-play, where the creature moves around and gains
   2.140 +sensory experience. From this experience I construct a representation
   2.141 +of the creature's sensory state space, which I call \phi-space. Using
   2.142 +\phi-space, I construct an efficient function for enriching the
   2.143 +limited data that comes from observing another creature with a full
   2.144 +compliment of imagined sensory data based on previous experience. I
   2.145 +can then use the imagined sensory data to recognize what the observed
   2.146 +creature is doing and feeling, using straightforward embodied action
   2.147 +predicates. This is all demonstrated with using a simple worm-like
   2.148 +creature, and recognizing worm-actions based on limited data.
   2.149  
   2.150 +Embodied representation using multiple senses such as touch,
   2.151 +proprioception, and muscle tension turns out be be exceedingly
   2.152 +efficient at describing body-centered actions. It is the ``right
   2.153 +language for the job''. For example, it takes only around 5 lines of
   2.154 +LISP code to describe the action of ``curling'' using embodied
   2.155 +primitives. It takes about 8 lines to describe the seemingly
   2.156 +complicated action of wiggling.
   2.157  
   2.158 -