# HG changeset patch # User Robert McIntyre # Date 1395616154 14400 # Node ID 853377051f1e61fc282d55232d31c0fb56fbe5b6 # Parent ae3bfc82ac7c6a2953691d9c68af8f4937171164 abstract v. 2 diff -r ae3bfc82ac7c -r 853377051f1e thesis/abstract.org --- a/thesis/abstract.org Sun Mar 23 17:59:16 2014 -0400 +++ b/thesis/abstract.org Sun Mar 23 19:09:14 2014 -0400 @@ -1,34 +1,23 @@ -Here I explore the design and capabilities of my system (called -=CORTEX=) which enables experiments in /embodied artificial -intelligence/ -- that is, AI which uses a physical simulation of -reality accompanied by a simulated body to solve problems. +Here I demonstrate the power of using embodied artificial intelligence +to attack the /action recognition/ problem, which is the challenge of +recognizing actions performed by a creature given limited data about +the creature's actions, such as a video recording. I solve this problem +in the case of a worm-like creature performing actions such as curling +and wiggling. -In the first half of the thesis I describe the construction of -=CORTEX= and the rationale behind my architecture choices. =CORTEX= is -a complete platform for embodied AI research. It provides multiple -senses for simulated creatures, including vision, touch, -proprioception, muscle tension, and hearing. Each of these senses -provides a wealth of parameters that are biologically -inspired. =CORTEX= is able to simulate any number of creatures and -senses, and provides facilities for easily modeling and creating new -creatures. As a research platform it is more complete than any other -system currently available. +To attack the action recognition problem, I developed a computational +model of empathy which allows me to use simple, embodied +representations of actions (which require rich sensory data), even +when that sensory data is not actually available. The missing sense +data is ``imagined'' by the system by combining previous experiences +gained from unsupervised free play. -In the second half of the thesis I develop a computational model of -empathy, using =CORTEX= as a base. Empathy in this context is the -ability to observe another creature and infer what sorts of sensations -that creature is feeling. My empathy algorithm involves multiple -phases. First is free-play, where the creature moves around and gains -sensory experience. From this experience I construct a representation -of the creature's sensory state space, which I call \phi-space. Using -\phi-space, I construct an efficient function for enriching the -limited data that comes from observing another creature with a full -compliment of imagined sensory data based on previous experience. I -can then use the imagined sensory data to recognize what the observed -creature is doing and feeling, using straightforward embodied action -predicates. This is all demonstrated with using a simple worm-like -creature, and recognizing worm-actions in video. - - -empathy is important -cortex tests that idea \ No newline at end of file +In order to build this empathic, action-recognizing system, I created +a program called =CORTEX=, which is a complete platform for embodied +AI research. It provides multiple senses for simulated creatures, +including vision, touch, proprioception, muscle tension, and +hearing. Each of these senses provides a wealth of parameters that are +biologically inspired. =CORTEX= is able to simulate any number of +creatures and senses, and provides facilities for easily modeling and +creating new creatures. As a research platform it is more complete +than any other system currently available. diff -r ae3bfc82ac7c -r 853377051f1e thesis/cortex.org --- a/thesis/cortex.org Sun Mar 23 17:59:16 2014 -0400 +++ b/thesis/cortex.org Sun Mar 23 19:09:14 2014 -0400 @@ -4,86 +4,64 @@ #+description: Using embodied AI to facilitate Artificial Imagination. #+keywords: AI, clojure, embodiment --- show hand +* Embodiment is a critical component of Intelligence -* Embodiment is a critical component to Intelligence +** Recognizing actions in video is extremely difficult + cat drinking, mimes, leaning, common sense -* To explore embodiment, we need a world, body, and senses +** Embodiment is the the right language for the job -* Because of Time, simulation is perferable to reality + a new possibility for the question ``what is a chair?'' -- it's the + feeling of your butt on something and your knees bent, with your + back muscles and legs relaxed. -* Video game engines are a great starting point +** =CORTEX= is a system for exploring embodiment -* Bodies are composed of segments connected by joints + Hand integration demo -* Eyes reuse standard video game components +** =CORTEX= solves recognition problems using empathy + + worm empathy demo -* Hearing is hard; =CORTEX= does it right +** Overview -* Touch uses hundreds of hair-like elements +* Building =CORTEX= -* Proprioception is the force that makes everything ``real'' +** To explore embodiment, we need a world, body, and senses -* Muscles are both effectors and sensors +** Because of Time, simulation is perferable to reality -* =CORTEX= brings complex creatures to life! +** Video game engines are a great starting point -* =CORTEX= enables many possiblities for further research +** Bodies are composed of segments connected by joints -* =CORTEX= User Guide +** Eyes reuse standard video game components + +** Hearing is hard; =CORTEX= does it right + +** Touch uses hundreds of hair-like elements + +** Proprioception is the force that makes everything ``real'' + +** Muscles are both effectors and sensors + +** =CORTEX= brings complex creatures to life! + +** =CORTEX= enables many possiblities for further research * Empathy in a simulated worm -* Embodiment factors action recognition into managable parts +** Embodiment factors action recognition into managable parts -* Action recognition is easy with a full gamut of senses +** Action recognition is easy with a full gamut of senses -* Digression: bootstrapping with multiple senses +** Digression: bootstrapping with multiple senses -* \Phi-space describes the worm's experiences +** \Phi-space describes the worm's experiences -* Empathy is the process of tracing though \Phi-space +** Empathy is the process of tracing though \Phi-space -* Efficient action recognition via empathy - -* Contributions - - -* Vision - - System for understanding what the actors in a video are doing -- - Action Recognition. - - Separate action recognition into three components: - - - free play - - embodied action predicates - - model alignment - - sensory imagination - -* Steps - - - Build cortex, a simulated environment for sensate AI - - solid bodies w/ joints - - vision - - touch - - vision - - hearing - - proprioception - - muscle contraction - - - Build experimental framework for worm-actions - - embodied stream predicates - - \phi-space - - \phi-scan - -* News - - Experimental results: - - - \phi-space actually works very well for the worm! - - self organizing touch map - +** Efficient action recognition via empathy * Contributions - Built =CORTEX=, a comprehensive platform for embodied AI @@ -92,9 +70,30 @@ - created a novel concept for action recognition by using artificial imagination. +* =CORTEX= User Guide +In the second half of the thesis I develop a computational model of +empathy, using =CORTEX= as a base. Empathy in this context is the +ability to observe another creature and infer what sorts of sensations +that creature is feeling. My empathy algorithm involves multiple +phases. First is free-play, where the creature moves around and gains +sensory experience. From this experience I construct a representation +of the creature's sensory state space, which I call \phi-space. Using +\phi-space, I construct an efficient function for enriching the +limited data that comes from observing another creature with a full +compliment of imagined sensory data based on previous experience. I +can then use the imagined sensory data to recognize what the observed +creature is doing and feeling, using straightforward embodied action +predicates. This is all demonstrated with using a simple worm-like +creature, and recognizing worm-actions based on limited data. +Embodied representation using multiple senses such as touch, +proprioception, and muscle tension turns out be be exceedingly +efficient at describing body-centered actions. It is the ``right +language for the job''. For example, it takes only around 5 lines of +LISP code to describe the action of ``curling'' using embodied +primitives. It takes about 8 lines to describe the seemingly +complicated action of wiggling. -