#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 

 

 * Empathy and Embodiment as problem solving strategies

   

   By the end of this thesis, you will have seen a novel approach to

   interpreting video using embodiment and empathy. You will have also

   seen one way to efficiently implement empathy for embodied

   creatures. Finally, you will become familiar with =CORTEX=, a system

   for designing and simulating creatures with rich senses, which you

   may choose to use in your own research.

   

   This is the core vision of my thesis: That one of the important ways

   in which we understand others is by imagining ourselves in their

   position and emphatically feeling experiences relative to our own

   bodies. By understanding events in terms of our own previous

   corporeal experience, we greatly constrain the possibilities of what

   would otherwise be an unwieldy exponential search. This extra

   constraint can be the difference between easily understanding what

   is happening in a video and being completely lost in a sea of

   incomprehensible color and movement.

 

 ** Recognizing actions in video is extremely difficult

 

    Consider for example the problem of determining what is happening

    in a video of which this is one frame:

 

    #+caption: A cat drinking some water. Identifying this action is 

    #+caption: beyond the state of the art for computers.

    #+ATTR_LaTeX: :width 7cm

    [[./images/cat-drinking.jpg]]

    

    It is currently impossible for any computer program to reliably

    label such a video as ``drinking''. And rightly so -- it is a very

    hard problem! What features can you describe in terms of low level

    functions of pixels that can even begin to describe at a high level

    what is happening here?

   

    Or suppose that you are building a program that recognizes chairs.

    How could you ``see'' the chair in figure \ref{invisible-chair} and

    figure \ref{hidden-chair}?

    

    #+caption: When you look at this, do you think ``chair''? I certainly do.

    #+name: invisible-chair

    #+ATTR_LaTeX: :width 10cm

    [[./images/invisible-chair.png]]

    

    #+caption: The chair in this image is quite obvious to humans, but I 

    #+caption: doubt that any computer program can find it.

    #+name: hidden-chair

    #+ATTR_LaTeX: :width 10cm

    [[./images/fat-person-sitting-at-desk.jpg]]

    

    Finally, how is it that you can easily tell the difference between

    how the girls /muscles/ are working in figure \ref{girl}?

    

    #+caption: The mysterious ``common sense'' appears here as you are able 

    #+caption: to discern the difference in how the girl's arm muscles

    #+caption: are activated between the two images.

    #+name: girl

    #+ATTR_LaTeX: :width 10cm

    [[./images/wall-push.png]]

   

    Each of these examples tells us something about what might be going

    on in our minds as we easily solve these recognition problems.

    

    The hidden chairs show us that we are strongly triggered by cues

    relating to the position of human bodies, and that we can determine

    the overall physical configuration of a human body even if much of

    that body is occluded.

 

    The picture of the girl pushing against the wall tells us that we

    have common sense knowledge about the kinetics of our own bodies.

    We know well how our muscles would have to work to maintain us in

    most positions, and we can easily project this self-knowledge to

    imagined positions triggered by images of the human body.

 

 ** =EMPATH= neatly solves recognition problems  

    

    I propose a system that can express the types of recognition

    problems above in a form amenable to computation. It is split into

    four parts:

 

    - Free/Guided Play (Training) :: The creature moves around and

         experiences the world through its unique perspective. Many

         otherwise complicated actions are easily described in the

         language of a full suite of body-centered, rich senses. For

         example, drinking is the feeling of water sliding down your

         throat, and cooling your insides. It's often accompanied by

         bringing your hand close to your face, or bringing your face

         close to water. Sitting down is the feeling of bending your

         knees, activating your quadriceps, then feeling a surface with

         your bottom and relaxing your legs. These body-centered action

         descriptions can be either learned or hard coded.

    - Alignment (Posture imitation) :: When trying to interpret a video

         or image, the creature takes a model of itself and aligns it

         with whatever it sees. This alignment can even cross species,

         as when humans try to align themselves with things like

         ponies, dogs, or other humans with a different body type.

    - Empathy (Sensory extrapolation) :: The alignment triggers

         associations with sensory data from prior experiences. For

         example, the alignment itself easily maps to proprioceptive

         data. Any sounds or obvious skin contact in the video can to a

         lesser extent trigger previous experience. Segments of

         previous experiences are stitched together to form a coherent

         and complete sensory portrait of the scene.

    - Recognition (Classification) :: With the scene described in terms

         of first person sensory events, the creature can now run its

         action-identification programs on this synthesized sensory

         data, just as it would if it were actually experiencing the

         scene first-hand. If previous experience has been accurately

         retrieved, and if it is analogous enough to the scene, then

         the creature will correctly identify the action in the scene.

    

    For example, I think humans are able to label the cat video as

    ``drinking'' because they imagine /themselves/ as the cat, and

    imagine putting their face up against a stream of water and

    sticking out their tongue. In that imagined world, they can feel

    the cool water hitting their tongue, and feel the water entering

    their body, and are able to recognize that /feeling/ as drinking.

    So, the label of the action is not really in the pixels of the

    image, but is found clearly in a simulation inspired by those

    pixels. An imaginative system, having been trained on drinking and

    non-drinking examples and learning that the most important

    component of drinking is the feeling of water sliding down one's

    throat, would analyze a video of a cat drinking in the following

    manner:

    

    1. Create a physical model of the video by putting a ``fuzzy''

       model of its own body in place of the cat. Possibly also create

       a simulation of the stream of water.

 

    2. Play out this simulated scene and generate imagined sensory

       experience. This will include relevant muscle contractions, a

       close up view of the stream from the cat's perspective, and most

       importantly, the imagined feeling of water entering the

       mouth. The imagined sensory experience can come from a

       simulation of the event, but can also be pattern-matched from

       previous, similar embodied experience.

 

    3. The action is now easily identified as drinking by the sense of

       taste alone. The other senses (such as the tongue moving in and

       out) help to give plausibility to the simulated action. Note that

       the sense of vision, while critical in creating the simulation,

       is not critical for identifying the action from the simulation.

 

    For the chair examples, the process is even easier:

 

     1. Align a model of your body to the person in the image.

 

     2. Generate proprioceptive sensory data from this alignment.

   

     3. Use the imagined proprioceptive data as a key to lookup related

        sensory experience associated with that particular proproceptive

        feeling.

 

     4. Retrieve the feeling of your bottom resting on a surface, your

        knees bent, and your leg muscles relaxed.

 

     5. This sensory information is consistent with the =sitting?=

        sensory predicate, so you (and the entity in the image) must be

        sitting.

 

     6. There must be a chair-like object since you are sitting.

 

    Empathy offers yet another alternative to the age-old AI

    representation question: ``What is a chair?'' --- A chair is the

    feeling of sitting.

 

    My program, =EMPATH= uses this empathic problem solving technique

    to interpret the actions of a simple, worm-like creature. 

    

    #+caption: The worm performs many actions during free play such as 

    #+caption: curling, wiggling, and resting.

    #+name: worm-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-intro-white.png]]

 

    #+caption: =EMPATH= recognized and classified each of these poses by

    #+caption: inferring the complete sensory experience from 

    #+caption: proprioceptive data.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-poses.png]]

    

    One powerful advantage of empathic problem solving is that it

    factors the action recognition problem into two easier problems. To

    use empathy, you need an /aligner/, which takes the video and a

    model of your body, and aligns the model with the video. Then, you

    need a /recognizer/, which uses the aligned model to interpret the

    action. The power in this method lies in the fact that you describe

    all actions form a body-centered, viewpoint You are less tied to

    the particulars of any visual representation of the actions. If you

    teach the system what ``running'' is, and you have a good enough

    aligner, the system will from then on be able to recognize running

    from any point of view, even strange points of view like above or

    underneath the runner. This is in contrast to action recognition

    schemes that try to identify actions using a non-embodied approach

    such as TODO:REFERENCE. If these systems learn about running as

    viewed from the side, they will not automatically be able to

    recognize running from any other viewpoint.

 

    Another powerful advantage is that using the language of multiple

    body-centered rich senses to describe body-centerd actions offers a

    massive boost in descriptive capability. Consider how difficult it

    would be to compose a set of HOG filters to describe the action of

    a simple worm-creature ``curling'' so that its head touches its

    tail, and then behold the simplicity of describing thus action in a

    language designed for the task (listing \ref{grand-circle-intro}):

 

    #+caption: Body-centerd actions are best expressed in a body-centered 

    #+caption: language. This code detects when the worm has curled into a 

    #+caption: full circle. Imagine how you would replicate this functionality

    #+caption: using low-level pixel features such as HOG filters!

    #+name: grand-circle-intro

    #+begin_listing clojure

    #+begin_src clojure

 (defn grand-circle?

   "Does the worm form a majestic circle (one end touching the other)?"

   [experiences]

   (and (curled? experiences)

        (let [worm-touch (:touch (peek experiences))

              tail-touch (worm-touch 0)

              head-touch (worm-touch 4)]

          (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))

               (< 0.55 (contact worm-segment-top-tip    head-touch))))))

    #+end_src

    #+end_listing

 

 

 ** =CORTEX= is a toolkit for building sensate creatures

 

    Hand integration demo

 

 ** Contributions

 

 * Building =CORTEX=

 

 ** To explore embodiment, we need a world, body, and senses

 

 ** Because of Time, simulation is perferable to reality

 

 ** Video game engines are a great starting point

 

 ** Bodies are composed of segments connected by joints

 

 ** Eyes reuse standard video game components

 

 ** Hearing is hard; =CORTEX= does it right

 

 ** Touch uses hundreds of hair-like elements

 

 ** Proprioception is the sense that makes everything ``real''

 

 ** Muscles are both effectors and sensors

 

 ** =CORTEX= brings complex creatures to life!

 

 ** =CORTEX= enables many possiblities for further research

 

 * Empathy in a simulated worm

 

 ** Embodiment factors action recognition into managable parts

 

 ** Action recognition is easy with a full gamut of senses

 

 ** Digression: bootstrapping touch using free exploration

 

 ** \Phi-space describes the worm's experiences

 

 ** Empathy is the process of tracing though \Phi-space 

   

 ** Efficient action recognition with =EMPATH=

 

 * Contributions

   - Built =CORTEX=, a comprehensive platform for embodied AI

     experiments. Has many new features lacking in other systems, such

     as sound. Easy to model/create new creatures.

   - created a novel concept for action recognition by using artificial

     imagination. 

 

 In the second half of the thesis I develop a computational model of

 empathy, using =CORTEX= as a base. Empathy in this context is the

 ability to observe another creature and infer what sorts of sensations

 that creature is feeling. My empathy algorithm involves multiple

 phases. First is free-play, where the creature moves around and gains

 sensory experience. From this experience I construct a representation

 of the creature's sensory state space, which I call \Phi-space. Using

 \Phi-space, I construct an efficient function for enriching the

 limited data that comes from observing another creature with a full

 compliment of imagined sensory data based on previous experience. I

 can then use the imagined sensory data to recognize what the observed

 creature is doing and feeling, using straightforward embodied action

 predicates. This is all demonstrated with using a simple worm-like

 creature, and recognizing worm-actions based on limited data.

 

 Embodied representation using multiple senses such as touch,

 proprioception, and muscle tension turns out be be exceedingly

 efficient at describing body-centered actions. It is the ``right

 language for the job''. For example, it takes only around 5 lines of

 LISP code to describe the action of ``curling'' using embodied

 primitives. It takes about 8 lines to describe the seemingly

 complicated action of wiggling.

 

 

 

 * COMMENT names for cortex

  - bioland

 

 

 

 

 # An anatomical joke:

 # - Training

 # - Skeletal imitation

 # - Sensory fleshing-out

 # - Classification