#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 

 

 * Empathy and Embodiment as problem solving strategies

   

   By the end of this thesis, you will have seen a novel approach to

   interpreting video using embodiment and empathy. You will have also

   seen one way to efficiently implement empathy for embodied

   creatures. Finally, you will become familiar with =CORTEX=, a system

   for designing and simulating creatures with rich senses, which you

   may choose to use in your own research.

   

   This is the core vision of my thesis: That one of the important ways

   in which we understand others is by imagining ourselves in their

   position and emphatically feeling experiences relative to our own

   bodies. By understanding events in terms of our own previous

   corporeal experience, we greatly constrain the possibilities of what

   would otherwise be an unwieldy exponential search. This extra

   constraint can be the difference between easily understanding what

   is happening in a video and being completely lost in a sea of

   incomprehensible color and movement.

 

 ** Recognizing actions in video is extremely difficult

 

    Consider for example the problem of determining what is happening

    in a video of which this is one frame:

 

    #+caption: A cat drinking some water. Identifying this action is 

    #+caption: beyond the state of the art for computers.

    #+ATTR_LaTeX: :width 7cm

    [[./images/cat-drinking.jpg]]

    

    It is currently impossible for any computer program to reliably

    label such a video as ``drinking''. And rightly so -- it is a very

    hard problem! What features can you describe in terms of low level

    functions of pixels that can even begin to describe at a high level

    what is happening here?

   

    Or suppose that you are building a program that recognizes chairs.

    How could you ``see'' the chair in figure \ref{hidden-chair}?

    

    #+caption: The chair in this image is quite obvious to humans, but I 

    #+caption: doubt that any modern computer vision program can find it.

    #+name: hidden-chair

    #+ATTR_LaTeX: :width 10cm

    [[./images/fat-person-sitting-at-desk.jpg]]

    

    Finally, how is it that you can easily tell the difference between

    how the girls /muscles/ are working in figure \ref{girl}?

    

    #+caption: The mysterious ``common sense'' appears here as you are able 

    #+caption: to discern the difference in how the girl's arm muscles

    #+caption: are activated between the two images.

    #+name: girl

    #+ATTR_LaTeX: :width 7cm

    [[./images/wall-push.png]]

   

    Each of these examples tells us something about what might be going

    on in our minds as we easily solve these recognition problems.

    

    The hidden chairs show us that we are strongly triggered by cues

    relating to the position of human bodies, and that we can determine

    the overall physical configuration of a human body even if much of

    that body is occluded.

 

    The picture of the girl pushing against the wall tells us that we

    have common sense knowledge about the kinetics of our own bodies.

    We know well how our muscles would have to work to maintain us in

    most positions, and we can easily project this self-knowledge to

    imagined positions triggered by images of the human body.

 

 ** =EMPATH= neatly solves recognition problems  

    

    I propose a system that can express the types of recognition

    problems above in a form amenable to computation. It is split into

    four parts:

 

    - Free/Guided Play :: The creature moves around and experiences the

         world through its unique perspective. Many otherwise

         complicated actions are easily described in the language of a

         full suite of body-centered, rich senses. For example,

         drinking is the feeling of water sliding down your throat, and

         cooling your insides. It's often accompanied by bringing your

         hand close to your face, or bringing your face close to water.

         Sitting down is the feeling of bending your knees, activating

         your quadriceps, then feeling a surface with your bottom and

         relaxing your legs. These body-centered action descriptions

         can be either learned or hard coded.

    - Posture Imitation :: When trying to interpret a video or image,

         the creature takes a model of itself and aligns it with

         whatever it sees. This alignment can even cross species, as

         when humans try to align themselves with things like ponies,

         dogs, or other humans with a different body type.

    - Empathy         :: The alignment triggers associations with

         sensory data from prior experiences. For example, the

         alignment itself easily maps to proprioceptive data. Any

         sounds or obvious skin contact in the video can to a lesser

         extent trigger previous experience. Segments of previous

         experiences are stitched together to form a coherent and

         complete sensory portrait of the scene.

    - Recognition      :: With the scene described in terms of first

         person sensory events, the creature can now run its

         action-identification programs on this synthesized sensory

         data, just as it would if it were actually experiencing the

         scene first-hand. If previous experience has been accurately

         retrieved, and if it is analogous enough to the scene, then

         the creature will correctly identify the action in the scene.

    

    For example, I think humans are able to label the cat video as

    ``drinking'' because they imagine /themselves/ as the cat, and

    imagine putting their face up against a stream of water and

    sticking out their tongue. In that imagined world, they can feel

    the cool water hitting their tongue, and feel the water entering

    their body, and are able to recognize that /feeling/ as drinking.

    So, the label of the action is not really in the pixels of the

    image, but is found clearly in a simulation inspired by those

    pixels. An imaginative system, having been trained on drinking and

    non-drinking examples and learning that the most important

    component of drinking is the feeling of water sliding down one's

    throat, would analyze a video of a cat drinking in the following

    manner:

    

    1. Create a physical model of the video by putting a ``fuzzy''

       model of its own body in place of the cat. Possibly also create

       a simulation of the stream of water.

 

    2. Play out this simulated scene and generate imagined sensory

       experience. This will include relevant muscle contractions, a

       close up view of the stream from the cat's perspective, and most

       importantly, the imagined feeling of water entering the

       mouth. The imagined sensory experience can come from a

       simulation of the event, but can also be pattern-matched from

       previous, similar embodied experience.

 

    3. The action is now easily identified as drinking by the sense of

       taste alone. The other senses (such as the tongue moving in and

       out) help to give plausibility to the simulated action. Note that

       the sense of vision, while critical in creating the simulation,

       is not critical for identifying the action from the simulation.

 

    For the chair examples, the process is even easier:

 

     1. Align a model of your body to the person in the image.

 

     2. Generate proprioceptive sensory data from this alignment.

   

     3. Use the imagined proprioceptive data as a key to lookup related

        sensory experience associated with that particular proproceptive

        feeling.

 

     4. Retrieve the feeling of your bottom resting on a surface, your

        knees bent, and your leg muscles relaxed.

 

     5. This sensory information is consistent with the =sitting?=

        sensory predicate, so you (and the entity in the image) must be

        sitting.

 

     6. There must be a chair-like object since you are sitting.

 

    Empathy offers yet another alternative to the age-old AI

    representation question: ``What is a chair?'' --- A chair is the

    feeling of sitting.

 

    My program, =EMPATH= uses this empathic problem solving technique

    to interpret the actions of a simple, worm-like creature. 

    

    #+caption: The worm performs many actions during free play such as 

    #+caption: curling, wiggling, and resting.

    #+name: worm-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-intro-white.png]]

 

    #+caption: =EMPATH= recognized and classified each of these poses by

    #+caption: inferring the complete sensory experience from 

    #+caption: proprioceptive data.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-poses.png]]

    

    One powerful advantage of empathic problem solving is that it

    factors the action recognition problem into two easier problems. To

    use empathy, you need an /aligner/, which takes the video and a

    model of your body, and aligns the model with the video. Then, you

    need a /recognizer/, which uses the aligned model to interpret the

    action. The power in this method lies in the fact that you describe

    all actions form a body-centered viewpoint. You are less tied to

    the particulars of any visual representation of the actions. If you

    teach the system what ``running'' is, and you have a good enough

    aligner, the system will from then on be able to recognize running

    from any point of view, even strange points of view like above or

    underneath the runner. This is in contrast to action recognition

    schemes that try to identify actions using a non-embodied approach.

    If these systems learn about running as viewed from the side, they

    will not automatically be able to recognize running from any other

    viewpoint.

 

    Another powerful advantage is that using the language of multiple

    body-centered rich senses to describe body-centerd actions offers a

    massive boost in descriptive capability. Consider how difficult it

    would be to compose a set of HOG filters to describe the action of

    a simple worm-creature ``curling'' so that its head touches its

    tail, and then behold the simplicity of describing thus action in a

    language designed for the task (listing \ref{grand-circle-intro}):

 

    #+caption: Body-centerd actions are best expressed in a body-centered 

    #+caption: language. This code detects when the worm has curled into a 

    #+caption: full circle. Imagine how you would replicate this functionality

    #+caption: using low-level pixel features such as HOG filters!

    #+name: grand-circle-intro

    #+begin_listing clojure

    #+begin_src clojure

 (defn grand-circle?

   "Does the worm form a majestic circle (one end touching the other)?"

   [experiences]

   (and (curled? experiences)

        (let [worm-touch (:touch (peek experiences))

              tail-touch (worm-touch 0)

              head-touch (worm-touch 4)]

          (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))

               (< 0.55 (contact worm-segment-top-tip    head-touch))))))

    #+end_src

    #+end_listing

 

 

 ** =CORTEX= is a toolkit for building sensate creatures

 

    I built =CORTEX= to be a general AI research platform for doing

    experiments involving multiple rich senses and a wide variety and

    number of creatures. I intend it to be useful as a library for many

    more projects than just this one. =CORTEX= was necessary to meet a

    need among AI researchers at CSAIL and beyond, which is that people

    often will invent neat ideas that are best expressed in the

    language of creatures and senses, but in order to explore those

    ideas they must first build a platform in which they can create

    simulated creatures with rich senses! There are many ideas that

    would be simple to execute (such as =EMPATH=), but attached to them

    is the multi-month effort to make a good creature simulator. Often,

    that initial investment of time proves to be too much, and the

    project must make do with a lesser environment.

 

    =CORTEX= is well suited as an environment for embodied AI research

    for three reasons:

 

    - You can create new creatures using Blender, a popular 3D modeling

      program. Each sense can be specified using special blender nodes

      with biologically inspired paramaters. You need not write any

      code to create a creature, and can use a wide library of

      pre-existing blender models as a base for your own creatures.

 

    - =CORTEX= implements a wide variety of senses, including touch,

      proprioception, vision, hearing, and muscle tension. Complicated

      senses like touch, and vision involve multiple sensory elements

      embedded in a 2D surface. You have complete control over the

      distribution of these sensor elements through the use of simple

      png image files. In particular, =CORTEX= implements more

      comprehensive hearing than any other creature simulation system

      available. 

 

    - =CORTEX= supports any number of creatures and any number of

      senses. Time in =CORTEX= dialates so that the simulated creatures

      always precieve a perfectly smooth flow of time, regardless of

      the actual computational load.

 

    =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game

    engine designed to create cross-platform 3D desktop games. =CORTEX=

    is mainly written in clojure, a dialect of =LISP= that runs on the

    java virtual machine (JVM). The API for creating and simulating

    creatures is entirely expressed in clojure. Hearing is implemented

    as a layer of clojure code on top of a layer of java code on top of

    a layer of =C++= code which implements a modified version of

    =OpenAL= to support multiple listeners. =CORTEX= is the only

    simulation environment that I know of that can support multiple

    entities that can each hear the world from their own perspective.

    Other senses also require a small layer of Java code. =CORTEX= also

    uses =bullet=, a physics simulator written in =C=.

 

    #+caption: Here is the worm from above modeled in Blender, a free 

    #+caption: 3D-modeling program. Senses and joints are described

    #+caption: using special nodes in Blender.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 12cm

    [[./images/blender-worm.png]]

 

    During one test with =CORTEX=, I created 3,000 entities each with

    their own independent senses and ran them all at only 1/80 real

    time. In another test, I created a detailed model of my own hand,

    equipped with a realistic distribution of touch (more sensitive at

    the fingertips), as well as eyes and ears, and it ran at around 1/4

    real time.

 

    #+caption: Here is the worm from above modeled in Blender, a free 

    #+caption: 3D-modeling program. Senses and joints are described

    #+caption: using special nodes in Blender.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/full-hand.png]]

    

    

    

 

    

 ** Contributions

 

 * Building =CORTEX=

 

 ** To explore embodiment, we need a world, body, and senses

 

 ** Because of Time, simulation is perferable to reality

 

 ** Video game engines are a great starting point

 

 ** Bodies are composed of segments connected by joints

 

 ** Eyes reuse standard video game components

 

 ** Hearing is hard; =CORTEX= does it right

 

 ** Touch uses hundreds of hair-like elements

 

 ** Proprioception is the sense that makes everything ``real''

 

 ** Muscles are both effectors and sensors

 

 ** =CORTEX= brings complex creatures to life!

 

 ** =CORTEX= enables many possiblities for further research

 

 * Empathy in a simulated worm

 

 ** Embodiment factors action recognition into managable parts

 

 ** Action recognition is easy with a full gamut of senses

 

 ** Digression: bootstrapping touch using free exploration

 

 ** \Phi-space describes the worm's experiences

 

 ** Empathy is the process of tracing though \Phi-space 

   

 ** Efficient action recognition with =EMPATH=

 

 * Contributions

   - Built =CORTEX=, a comprehensive platform for embodied AI

     experiments. Has many new features lacking in other systems, such

     as sound. Easy to model/create new creatures.

   - created a novel concept for action recognition by using artificial

     imagination. 

 

 In the second half of the thesis I develop a computational model of

 empathy, using =CORTEX= as a base. Empathy in this context is the

 ability to observe another creature and infer what sorts of sensations

 that creature is feeling. My empathy algorithm involves multiple

 phases. First is free-play, where the creature moves around and gains

 sensory experience. From this experience I construct a representation

 of the creature's sensory state space, which I call \Phi-space. Using

 \Phi-space, I construct an efficient function for enriching the

 limited data that comes from observing another creature with a full

 compliment of imagined sensory data based on previous experience. I

 can then use the imagined sensory data to recognize what the observed

 creature is doing and feeling, using straightforward embodied action

 predicates. This is all demonstrated with using a simple worm-like

 creature, and recognizing worm-actions based on limited data.

 

 Embodied representation using multiple senses such as touch,

 proprioception, and muscle tension turns out be be exceedingly

 efficient at describing body-centered actions. It is the ``right

 language for the job''. For example, it takes only around 5 lines of

 LISP code to describe the action of ``curling'' using embodied

 primitives. It takes about 8 lines to describe the seemingly

 complicated action of wiggling.

 

 

 

 * COMMENT names for cortex

  - bioland

 

 

 

 

 # An anatomical joke:

 # - Training

 # - Skeletal imitation

 # - Sensory fleshing-out

 # - Classification