#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 

 

 * Empathy and Embodiment as problem solving strategies

   

   By the end of this thesis, you will have seen a novel approach to

   interpreting video using embodiment and empathy. You will have also

   seen one way to efficiently implement empathy for embodied

   creatures.

   

   The core vision of this thesis is that one of the important ways in

   which we understand others is by imagining ourselves in their

   posistion and empathicaly feeling experiences based on our own past

   experiences and imagination.

 

   By understanding events in terms of our own previous corperal

   experience, we greatly constrain the possibilities of what would

   otherwise be an unweidly exponential search. This extra constraint

   can be the difference between easily understanding what is happening

   in a video and being completely lost in a sea of incomprehensible

   color and movement.

 

 ** Recognizing actions in video is extremely difficult

 

   Consider for example the problem of determining what is happening in

   a video of which this is one frame:

 

   #+caption: A cat drinking some water. Identifying this action is 

   #+caption: beyond the state of the art for computers.

   #+ATTR_LaTeX: :width 7cm

   [[./images/cat-drinking.jpg]]

   

   It is currently impossible for any computer program to reliably

   label such an video as "drinking".  And rightly so -- it is a very

   hard problem! What features can you describe in terms of low level

   functions of pixels that can even begin to describe what is

   happening here? 

   

   Or suppose that you are building a program that recognizes

   chairs. How could you ``see'' the chair in the following pictures?

 

   #+caption: When you look at this, do you think ``chair''? I certainly do.

   #+ATTR_LaTeX: :width 10cm

   [[./images/invisible-chair.png]]

   

   #+caption: The chair in this image is quite obvious to humans, but I 

   #+caption: doubt that any computer program can find it.

   #+ATTR_LaTeX: :width 10cm

   [[./images/fat-person-sitting-at-desk.jpg]]

 

   Finally, how is it that you can easily tell the difference between

   how the girls /muscles/ are working in \ref{girl}?

 

   #+caption: The mysterious ``common sense'' appears here as you are able 

   #+caption: to ``see'' the difference in how the girl's arm muscles

   #+caption: are activated differently in the two images.

   #+name: girl

   #+ATTR_LaTeX: :width 10cm

   [[./images/wall-push.png]]

   

 

   These problems are difficult because the language of pixels is far

   removed from what we would consider to be an acceptable description

   of the events in these images. In order to process them, we must

   raise the images into some higher level of abstraction where their

   descriptions become more similar to how we would describe them in

   English. The question is, how can we raise 

   

 

   I think humans are able to label such video as "drinking" because

   they imagine /themselves/ as the cat, and imagine putting their face

   up against a stream of water and sticking out their tongue. In that

   imagined world, they can feel the cool water hitting their tongue,

   and feel the water entering their body, and are able to recognize

   that /feeling/ as drinking. So, the label of the action is not

   really in the pixels of the image, but is found clearly in a

   simulation inspired by those pixels. An imaginative system, having

   been trained on drinking and non-drinking examples and learning that

   the most important component of drinking is the feeling of water

   sliding down one's throat, would analyze a video of a cat drinking

   in the following manner:

    

    - Create a physical model of the video by putting a "fuzzy" model

      of its own body in place of the cat. Also, create a simulation of

      the stream of water.

 

    - Play out this simulated scene and generate imagined sensory

      experience. This will include relevant muscle contractions, a

      close up view of the stream from the cat's perspective, and most

      importantly, the imagined feeling of water entering the mouth.

 

    - The action is now easily identified as drinking by the sense of

      taste alone. The other senses (such as the tongue moving in and

      out) help to give plausibility to the simulated action. Note that

      the sense of vision, while critical in creating the simulation,

      is not critical for identifying the action from the simulation.

 

    cat drinking, mimes, leaning, common sense

 

 ** =EMPATH= neatly solves recognition problems

 

    factorization , right language, etc

 

    a new possibility for the question ``what is a chair?'' -- it's the

    feeling of your butt on something and your knees bent, with your

    back muscles and legs relaxed.

 

 ** =CORTEX= is a toolkit for building sensate creatures

 

    Hand integration demo

 

 ** Contributions

 

 * Building =CORTEX=

 

 ** To explore embodiment, we need a world, body, and senses

 

 ** Because of Time, simulation is perferable to reality

 

 ** Video game engines are a great starting point

 

 ** Bodies are composed of segments connected by joints

 

 ** Eyes reuse standard video game components

 

 ** Hearing is hard; =CORTEX= does it right

 

 ** Touch uses hundreds of hair-like elements

 

 ** Proprioception is the sense that makes everything ``real''

 

 ** Muscles are both effectors and sensors

 

 ** =CORTEX= brings complex creatures to life!

 

 ** =CORTEX= enables many possiblities for further research

 

 * Empathy in a simulated worm

 

 ** Embodiment factors action recognition into managable parts

 

 ** Action recognition is easy with a full gamut of senses

 

 ** Digression: bootstrapping touch using free exploration

 

 ** \Phi-space describes the worm's experiences

 

 ** Empathy is the process of tracing though \Phi-space 

   

 ** Efficient action recognition =EMPATH=

 

 * Contributions

   - Built =CORTEX=, a comprehensive platform for embodied AI

     experiments. Has many new features lacking in other systems, such

     as sound. Easy to model/create new creatures.

   - created a novel concept for action recognition by using artificial

     imagination. 

 

 In the second half of the thesis I develop a computational model of

 empathy, using =CORTEX= as a base. Empathy in this context is the

 ability to observe another creature and infer what sorts of sensations

 that creature is feeling. My empathy algorithm involves multiple

 phases. First is free-play, where the creature moves around and gains

 sensory experience. From this experience I construct a representation

 of the creature's sensory state space, which I call \phi-space. Using

 \phi-space, I construct an efficient function for enriching the

 limited data that comes from observing another creature with a full

 compliment of imagined sensory data based on previous experience. I

 can then use the imagined sensory data to recognize what the observed

 creature is doing and feeling, using straightforward embodied action

 predicates. This is all demonstrated with using a simple worm-like

 creature, and recognizing worm-actions based on limited data.

 

 Embodied representation using multiple senses such as touch,

 proprioception, and muscle tension turns out be be exceedingly

 efficient at describing body-centered actions. It is the ``right

 language for the job''. For example, it takes only around 5 lines of

 LISP code to describe the action of ``curling'' using embodied

 primitives. It takes about 8 lines to describe the seemingly

 complicated action of wiggling.

 

 

 

 * COMMENT names for cortex

  - bioland