#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 

 

 * Empathy and Embodiment: problem solving strategies

   

   By the end of this thesis, you will have seen a novel approach to

   interpreting video using embodiment and empathy. You will have also

   seen one way to efficiently implement empathy for embodied

   creatures.

   

   The core vision of this thesis is that one of the important ways in

   which we understand others is by imagining ourselves in their

   posistion and empathicaly feeling experiences based on our own past

   experiences and imagination.

 

   By understanding events in terms of our own previous corperal

   experience, we greatly constrain the possibilities of what would

   otherwise be an unweidly exponential search. This extra constraint

   can be the difference between easily understanding what is happening

   in a video and being completely lost in a sea of incomprehensible

   color and movement.

 

 ** Recognizing actions in video is extremely difficult

 

   Consider for example the problem of determining what is happening in

   a video of which this is one frame:

 

   #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.

   #+ATTR_LaTeX: :width 7cm

   [[./images/cat-drinking.jpg]]

   

   It is currently impossible for any computer program to reliably

   label such an video as "drinking".  And rightly so -- it is a very

   hard problem! What features can you describe in terms of low level

   functions of pixels that can even begin to describe what is

   happening here? 

   

   Or suppose that you are building a program that recognizes

   chairs. How could you ``see'' the chair in the following picture?

 

   #+caption: When you look at this, do you think ``chair''? I certainly do.

   #+ATTR_LaTeX: :width 10cm

   [[./images/invisible-chair.png]]

   

   #+caption: The chair in this image is quite obvious to humans, but I doubt any computer program can find it.

   #+ATTR_LaTeX: :width 10cm

   [[./images/fat-person-sitting-at-desk.jpg]]

 

 

   I think humans are able to label

   such video as "drinking" because they imagine /themselves/ as the

   cat, and imagine putting their face up against a stream of water and

   sticking out their tongue. In that imagined world, they can feel the

   cool water hitting their tongue, and feel the water entering their

   body, and are able to recognize that /feeling/ as drinking. So, the

   label of the action is not really in the pixels of the image, but is

   found clearly in a simulation inspired by those pixels. An

   imaginative system, having been trained on drinking and non-drinking

   examples and learning that the most important component of drinking

   is the feeling of water sliding down one's throat, would analyze a

   video of a cat drinking in the following manner:

    

    - Create a physical model of the video by putting a "fuzzy" model

      of its own body in place of the cat. Also, create a simulation of

      the stream of water.

 

    - Play out this simulated scene and generate imagined sensory

      experience. This will include relevant muscle contractions, a

      close up view of the stream from the cat's perspective, and most

      importantly, the imagined feeling of water entering the mouth.

 

    - The action is now easily identified as drinking by the sense of

      taste alone. The other senses (such as the tongue moving in and

      out) help to give plausibility to the simulated action. Note that

      the sense of vision, while critical in creating the simulation,

      is not critical for identifying the action from the simulation.

 

 

 

 

 

 

 

    cat drinking, mimes, leaning, common sense

 

 ** =EMPATH= neatly solves recognition problems

 

    factorization , right language, etc

 

    a new possibility for the question ``what is a chair?'' -- it's the

    feeling of your butt on something and your knees bent, with your

    back muscles and legs relaxed.

 

 ** =CORTEX= is a toolkit for building sensate creatures

 

    Hand integration demo

 

 ** Contributions

 

 * Building =CORTEX=

 

 ** To explore embodiment, we need a world, body, and senses

 

 ** Because of Time, simulation is perferable to reality

 

 ** Video game engines are a great starting point

 

 ** Bodies are composed of segments connected by joints

 

 ** Eyes reuse standard video game components

 

 ** Hearing is hard; =CORTEX= does it right

 

 ** Touch uses hundreds of hair-like elements

 

 ** Proprioception is the force that makes everything ``real''

 

 ** Muscles are both effectors and sensors

 

 ** =CORTEX= brings complex creatures to life!

 

 ** =CORTEX= enables many possiblities for further research

 

 * Empathy in a simulated worm

 

 ** Embodiment factors action recognition into managable parts

 

 ** Action recognition is easy with a full gamut of senses

 

 ** Digression: bootstrapping touch using free exploration

 

 ** \Phi-space describes the worm's experiences

 

 ** Empathy is the process of tracing though \Phi-space 

   

 ** Efficient action recognition via empathy

 

 * Contributions

   - Built =CORTEX=, a comprehensive platform for embodied AI

     experiments. Has many new features lacking in other systems, such

     as sound. Easy to model/create new creatures.

   - created a novel concept for action recognition by using artificial

     imagination. 

 

 In the second half of the thesis I develop a computational model of

 empathy, using =CORTEX= as a base. Empathy in this context is the

 ability to observe another creature and infer what sorts of sensations

 that creature is feeling. My empathy algorithm involves multiple

 phases. First is free-play, where the creature moves around and gains

 sensory experience. From this experience I construct a representation

 of the creature's sensory state space, which I call \phi-space. Using

 \phi-space, I construct an efficient function for enriching the

 limited data that comes from observing another creature with a full

 compliment of imagined sensory data based on previous experience. I

 can then use the imagined sensory data to recognize what the observed

 creature is doing and feeling, using straightforward embodied action

 predicates. This is all demonstrated with using a simple worm-like

 creature, and recognizing worm-actions based on limited data.

 

 Embodied representation using multiple senses such as touch,

 proprioception, and muscle tension turns out be be exceedingly

 efficient at describing body-centered actions. It is the ``right

 language for the job''. For example, it takes only around 5 lines of

 LISP code to describe the action of ``curling'' using embodied

 primitives. It takes about 8 lines to describe the seemingly

 complicated action of wiggling.

 

 

 

 * COMMENT names for cortex

  - bioland