diff -r f639e2139ce2 thesis/cortex.org

 --- a/thesis/cortex.org	Sun Mar 30 01:34:43 2014 -0400

 +++ b/thesis/cortex.org	Sun Mar 30 10:07:17 2014 -0400

 @@ -41,49 +41,46 @@

      [[./images/aurellem-gray.png]]

  

  

 -* Empathy and Embodiment as problem solving strategies

 +* Empathy \& Embodiment: problem solving strategies

    

 -  By the end of this thesis, you will have seen a novel approach to

 -  interpreting video using embodiment and empathy. You will have also

 -  seen one way to efficiently implement empathy for embodied

 -  creatures. Finally, you will become familiar with =CORTEX=, a system

 -  for designing and simulating creatures with rich senses, which you

 -  may choose to use in your own research.

 -  

 -  This is the core vision of my thesis: That one of the important ways

 -  in which we understand others is by imagining ourselves in their

 -  position and emphatically feeling experiences relative to our own

 -  bodies. By understanding events in terms of our own previous

 -  corporeal experience, we greatly constrain the possibilities of what

 -  would otherwise be an unwieldy exponential search. This extra

 -  constraint can be the difference between easily understanding what

 -  is happening in a video and being completely lost in a sea of

 -  incomprehensible color and movement.

 -  

 -** Recognizing actions in video is extremely difficult

 -

 -   Consider for example the problem of determining what is happening

 -   in a video of which this is one frame:

 -

 +** The problem: recognizing actions in video is extremely difficult

 +# developing / requires useful representations

 +   

 +   Examine the following collection of images. As you, and indeed very

 +   young children, can easily determine, each one is a picture of

 +   someone drinking. 

 +

 +   # dxh: cat, cup, drinking fountain, rain, straw, coconut

     #+caption: A cat drinking some water. Identifying this action is 

 -   #+caption: beyond the state of the art for computers.

 +   #+caption: beyond the capabilities of existing computer vision systems.

     #+ATTR_LaTeX: :width 7cm

     [[./images/cat-drinking.jpg]]

 +     

 +   Nevertheless, it is beyond the state of the art for a computer

 +   vision program to describe what's happening in each of these

 +   images, or what's common to them. Part of the problem is that many

 +   computer vision systems focus on pixel-level details or probability

 +   distributions of pixels, with little focus on [...]

 +

 +

 +   In fact, the contents of scene may have much less to do with pixel

 +   probabilities than with recognizing various affordances: things you

 +   can move, objects you can grasp, spaces that can be filled

 +   (Gibson). For example, what processes might enable you to see the

 +   chair in figure \ref{hidden-chair}? 

 +   # Or suppose that you are building a program that recognizes chairs.

 +   # How could you ``see'' the chair ?

     

 -   It is currently impossible for any computer program to reliably

 -   label such a video as ``drinking''. And rightly so -- it is a very

 -   hard problem! What features can you describe in terms of low level

 -   functions of pixels that can even begin to describe at a high level

 -   what is happening here?

 -  

 -   Or suppose that you are building a program that recognizes chairs.

 -   How could you ``see'' the chair in figure \ref{hidden-chair}?

 -   

 +   # dxh: blur chair

     #+caption: The chair in this image is quite obvious to humans, but I 

     #+caption: doubt that any modern computer vision program can find it.

     #+name: hidden-chair

     #+ATTR_LaTeX: :width 10cm

     [[./images/fat-person-sitting-at-desk.jpg]]

 +

 +

 +   

 +

     

     Finally, how is it that you can easily tell the difference between

     how the girls /muscles/ are working in figure \ref{girl}?

 @@ -95,10 +92,13 @@

     #+ATTR_LaTeX: :width 7cm

     [[./images/wall-push.png]]

    

 +

 +

 +

     Each of these examples tells us something about what might be going

     on in our minds as we easily solve these recognition problems.

     

 -   The hidden chairs show us that we are strongly triggered by cues

 +   The hidden chair shows us that we are strongly triggered by cues

     relating to the position of human bodies, and that we can determine

     the overall physical configuration of a human body even if much of

     that body is occluded.

 @@ -109,10 +109,107 @@

     most positions, and we can easily project this self-knowledge to

     imagined positions triggered by images of the human body.

  

 -** =EMPATH= neatly solves recognition problems  

 +** A step forward: the sensorimotor-centered approach

 +# ** =EMPATH= recognizes what creatures are doing

 +# neatly solves recognition problems  

 +   In this thesis, I explore the idea that our knowledge of our own

 +   bodies enables us to recognize the actions of others. 

 +

 +   First, I built a system for constructing virtual creatures with

 +   physiologically plausible sensorimotor systems and detailed

 +   environments. The result is =CORTEX=, which is described in section

 +   \ref{sec-2}. (=CORTEX= was built to be flexible and useful to other

 +   AI researchers; it is provided in full with detailed instructions

 +   on the web [here].)

 +

 +   Next, I wrote routines which enabled a simple worm-like creature to

 +   infer the actions of a second worm-like creature, using only its

 +   own prior sensorimotor experiences and knowledge of the second

 +   worm's joint positions. This program, =EMPATH=, is described in

 +   section \ref{sec-3}, and the key results of this experiment are

 +   summarized below.

 +

 +  #+caption: From only \emph{proprioceptive} data, =EMPATH= was able to infer 

 +  #+caption: the complete sensory experience and classify these four poses.

 +  #+caption: The last image is a composite, depicting the intermediate stages of \emph{wriggling}.

 +  #+name: worm-recognition-intro-2

 +  #+ATTR_LaTeX: :width 15cm

 +   [[./images/empathy-1.png]]

 +

 +   # =CORTEX= provides a language for describing the sensorimotor

 +   # experiences of various creatures. 

 +

 +   # Next, I developed an experiment to test the power of =CORTEX='s

 +   # sensorimotor-centered language for solving recognition problems. As

 +   # a proof of concept, I wrote routines which enabled a simple

 +   # worm-like creature to infer the actions of a second worm-like

 +   # creature, using only its own previous sensorimotor experiences and

 +   # knowledge of the second worm's joints (figure

 +   # \ref{worm-recognition-intro-2}). The result of this proof of

 +   # concept was the program =EMPATH=, described in section

 +   # \ref{sec-3}. The key results of this

 +

 +   # Using only first-person sensorimotor experiences and third-person

 +   # proprioceptive data, 

 +

 +*** Key results

 +   - After one-shot supervised training, =EMPATH= was able recognize a

 +     wide variety of static poses and dynamic actions---ranging from

 +     curling in a circle to wriggling with a particular frequency ---

 +     with 95\% accuracy.

 +   - These results were completely independent of viewing angle

 +     because the underlying body-centered language fundamentally is;

 +     once an action is learned, it can be recognized equally well from

 +     any viewing angle.

 +   - =EMPATH= is surprisingly short; the sensorimotor-centered

 +     language provided by =CORTEX= resulted in extremely economical

 +     recognition routines --- about 0000 lines in all --- suggesting

 +     that such representations are very powerful, and often

 +     indispensible for the types of recognition tasks considered here.

 +   - Although for expediency's sake, I relied on direct knowledge of

 +     joint positions in this proof of concept, it would be

 +     straightforward to extend =EMPATH= so that it (more

 +     realistically) infers joint positions from its visual data.

 +

 +# because the underlying language is fundamentally orientation-independent

 +

 +# recognize the actions of a worm with 95\% accuracy. The

 +#      recognition tasks 

     

 -   I propose a system that can express the types of recognition

 -   problems above in a form amenable to computation. It is split into

 +

 +

 +

 +   [Talk about these results and what you find promising about them]

 +

 +** Roadmap

 +   [I'm going to explain how =CORTEX= works, then break down how

 +   =EMPATH= does its thing. Because the details reveal such-and-such

 +   about the approach.]

 +

 +   # The success of this simple proof-of-concept offers a tantalizing

 +

 +

 +   # explore the idea 

 +   # The key contribution of this thesis is the idea that body-centered

 +   # representations (which express 

 +

 +

 +   # the

 +   # body-centered approach --- in which I try to determine what's

 +   # happening in a scene by bringing it into registration with my own

 +   # bodily experiences --- are indispensible for recognizing what

 +   # creatures are doing in a scene.

 +

 +* COMMENT

 +# body-centered language

 +   

 +   In this thesis, I'll describe =EMPATH=, which solves a certain

 +   class of recognition problems 

 +

 +   The key idea is to use self-centered (or first-person) language.

 +

 +   I have built a system that can express the types of recognition

 +   problems in a form amenable to computation. It is split into

     four parts:

  

     - Free/Guided Play :: The creature moves around and experiences the

 @@ -286,14 +383,14 @@

       code to create a creature, and can use a wide library of

       pre-existing blender models as a base for your own creatures.

  

 -   - =CORTEX= implements a wide variety of senses, including touch,

 +   - =CORTEX= implements a wide variety of senses: touch,

       proprioception, vision, hearing, and muscle tension. Complicated

       senses like touch, and vision involve multiple sensory elements

       embedded in a 2D surface. You have complete control over the

       distribution of these sensor elements through the use of simple

       png image files. In particular, =CORTEX= implements more

       comprehensive hearing than any other creature simulation system

 -     available. 

 +     available.

  

     - =CORTEX= supports any number of creatures and any number of

       senses. Time in =CORTEX= dialates so that the simulated creatures

 @@ -353,7 +450,24 @@

     \end{sidewaysfigure}

  #+END_LaTeX

  

 -** Contributions

 +** Road map

 +

 +   By the end of this thesis, you will have seen a novel approach to

 +  interpreting video using embodiment and empathy. You will have also

 +  seen one way to efficiently implement empathy for embodied

 +  creatures. Finally, you will become familiar with =CORTEX=, a system

 +  for designing and simulating creatures with rich senses, which you

 +  may choose to use in your own research.

 +  

 +  This is the core vision of my thesis: That one of the important ways

 +  in which we understand others is by imagining ourselves in their

 +  position and emphatically feeling experiences relative to our own

 +  bodies. By understanding events in terms of our own previous

 +  corporeal experience, we greatly constrain the possibilities of what

 +  would otherwise be an unwieldy exponential search. This extra

 +  constraint can be the difference between easily understanding what

 +  is happening in a video and being completely lost in a sea of

 +  incomprehensible color and movement.

  

     - I built =CORTEX=, a comprehensive platform for embodied AI

       experiments. =CORTEX= supports many features lacking in other

 @@ -363,18 +477,22 @@

     - I built =EMPATH=, which uses =CORTEX= to identify the actions of

       a worm-like creature using a computational model of empathy.

     

 -* Building =CORTEX=

 -

 -  I intend for =CORTEX= to be used as a general-purpose library for

 -  building creatures and outfitting them with senses, so that it will

 -  be useful for other researchers who want to test out ideas of their

 -  own. To this end, wherver I have had to make archetictural choices

 -  about =CORTEX=, I have chosen to give as much freedom to the user as

 -  possible, so that =CORTEX= may be used for things I have not

 -  forseen.

 -

 -** Simulation or Reality?

 -   

 +

 +* Designing =CORTEX=

 +  In this section, I outline the design decisions that went into

 +  making =CORTEX=, along with some details about its

 +  implementation. (A practical guide to getting started with =CORTEX=,

 +  which skips over the history and implementation details presented

 +  here, is provided in an appendix \ref{} at the end of this paper.)

 +

 +  Throughout this project, I intended for =CORTEX= to be flexible and

 +  extensible enough to be useful for other researchers who want to

 +  test out ideas of their own. To this end, wherver I have had to make

 +  archetictural choices about =CORTEX=, I have chosen to give as much

 +  freedom to the user as possible, so that =CORTEX= may be used for

 +  things I have not forseen.

 +

 +** Building in simulation versus reality

     The most important archetictural decision of all is the choice to

     use a computer-simulated environemnt in the first place! The world

     is a vast and rich place, and for now simulations are a very poor

 @@ -436,7 +554,7 @@

      doing everything in software is far cheaper than building custom

      real-time hardware. All you need is a laptop and some patience.

  

 -** Because of Time, simulation is perferable to reality

 +** Simulated time enables rapid prototyping and complex scenes 

  

     I envision =CORTEX= being used to support rapid prototyping and

     iteration of ideas. Even if I could put together a well constructed