#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 #+LaTeX_CLASS_OPTIONS: [nofloat]

 

 * COMMENT templates

   #+caption: 

   #+caption: 

   #+caption: 

   #+caption: 

   #+name: name

   #+begin_listing clojure

   #+begin_src clojure

   #+end_src

   #+end_listing

 

   #+caption: 

   #+caption: 

   #+caption: 

   #+name: name

   #+ATTR_LaTeX: :width 10cm

   [[./images/aurellem-gray.png]]

 

 * COMMENT Empathy and Embodiment as problem solving strategies

   

   By the end of this thesis, you will have seen a novel approach to

   interpreting video using embodiment and empathy. You will have also

   seen one way to efficiently implement empathy for embodied

   creatures. Finally, you will become familiar with =CORTEX=, a system

   for designing and simulating creatures with rich senses, which you

   may choose to use in your own research.

   

   This is the core vision of my thesis: That one of the important ways

   in which we understand others is by imagining ourselves in their

   position and emphatically feeling experiences relative to our own

   bodies. By understanding events in terms of our own previous

   corporeal experience, we greatly constrain the possibilities of what

   would otherwise be an unwieldy exponential search. This extra

   constraint can be the difference between easily understanding what

   is happening in a video and being completely lost in a sea of

   incomprehensible color and movement.

 

 ** Recognizing actions in video is extremely difficult

 

    Consider for example the problem of determining what is happening

    in a video of which this is one frame:

 

    #+caption: A cat drinking some water. Identifying this action is 

    #+caption: beyond the state of the art for computers.

    #+ATTR_LaTeX: :width 7cm

    [[./images/cat-drinking.jpg]]

    

    It is currently impossible for any computer program to reliably

    label such a video as ``drinking''. And rightly so -- it is a very

    hard problem! What features can you describe in terms of low level

    functions of pixels that can even begin to describe at a high level

    what is happening here?

   

    Or suppose that you are building a program that recognizes chairs.

    How could you ``see'' the chair in figure \ref{hidden-chair}?

    

    #+caption: The chair in this image is quite obvious to humans, but I 

    #+caption: doubt that any modern computer vision program can find it.

    #+name: hidden-chair

    #+ATTR_LaTeX: :width 10cm

    [[./images/fat-person-sitting-at-desk.jpg]]

    

    Finally, how is it that you can easily tell the difference between

    how the girls /muscles/ are working in figure \ref{girl}?

    

    #+caption: The mysterious ``common sense'' appears here as you are able 

    #+caption: to discern the difference in how the girl's arm muscles

    #+caption: are activated between the two images.

    #+name: girl

    #+ATTR_LaTeX: :width 7cm

    [[./images/wall-push.png]]

   

    Each of these examples tells us something about what might be going

    on in our minds as we easily solve these recognition problems.

    

    The hidden chairs show us that we are strongly triggered by cues

    relating to the position of human bodies, and that we can determine

    the overall physical configuration of a human body even if much of

    that body is occluded.

 

    The picture of the girl pushing against the wall tells us that we

    have common sense knowledge about the kinetics of our own bodies.

    We know well how our muscles would have to work to maintain us in

    most positions, and we can easily project this self-knowledge to

    imagined positions triggered by images of the human body.

 

 ** =EMPATH= neatly solves recognition problems  

    

    I propose a system that can express the types of recognition

    problems above in a form amenable to computation. It is split into

    four parts:

 

    - Free/Guided Play :: The creature moves around and experiences the

         world through its unique perspective. Many otherwise

         complicated actions are easily described in the language of a

         full suite of body-centered, rich senses. For example,

         drinking is the feeling of water sliding down your throat, and

         cooling your insides. It's often accompanied by bringing your

         hand close to your face, or bringing your face close to water.

         Sitting down is the feeling of bending your knees, activating

         your quadriceps, then feeling a surface with your bottom and

         relaxing your legs. These body-centered action descriptions

         can be either learned or hard coded.

    - Posture Imitation :: When trying to interpret a video or image,

         the creature takes a model of itself and aligns it with

         whatever it sees. This alignment can even cross species, as

         when humans try to align themselves with things like ponies,

         dogs, or other humans with a different body type.

    - Empathy         :: The alignment triggers associations with

         sensory data from prior experiences. For example, the

         alignment itself easily maps to proprioceptive data. Any

         sounds or obvious skin contact in the video can to a lesser

         extent trigger previous experience. Segments of previous

         experiences are stitched together to form a coherent and

         complete sensory portrait of the scene.

    - Recognition      :: With the scene described in terms of first

         person sensory events, the creature can now run its

         action-identification programs on this synthesized sensory

         data, just as it would if it were actually experiencing the

         scene first-hand. If previous experience has been accurately

         retrieved, and if it is analogous enough to the scene, then

         the creature will correctly identify the action in the scene.

    

    For example, I think humans are able to label the cat video as

    ``drinking'' because they imagine /themselves/ as the cat, and

    imagine putting their face up against a stream of water and

    sticking out their tongue. In that imagined world, they can feel

    the cool water hitting their tongue, and feel the water entering

    their body, and are able to recognize that /feeling/ as drinking.

    So, the label of the action is not really in the pixels of the

    image, but is found clearly in a simulation inspired by those

    pixels. An imaginative system, having been trained on drinking and

    non-drinking examples and learning that the most important

    component of drinking is the feeling of water sliding down one's

    throat, would analyze a video of a cat drinking in the following

    manner:

    

    1. Create a physical model of the video by putting a ``fuzzy''

       model of its own body in place of the cat. Possibly also create

       a simulation of the stream of water.

 

    2. Play out this simulated scene and generate imagined sensory

       experience. This will include relevant muscle contractions, a

       close up view of the stream from the cat's perspective, and most

       importantly, the imagined feeling of water entering the

       mouth. The imagined sensory experience can come from a

       simulation of the event, but can also be pattern-matched from

       previous, similar embodied experience.

 

    3. The action is now easily identified as drinking by the sense of

       taste alone. The other senses (such as the tongue moving in and

       out) help to give plausibility to the simulated action. Note that

       the sense of vision, while critical in creating the simulation,

       is not critical for identifying the action from the simulation.

 

    For the chair examples, the process is even easier:

 

     1. Align a model of your body to the person in the image.

 

     2. Generate proprioceptive sensory data from this alignment.

   

     3. Use the imagined proprioceptive data as a key to lookup related

        sensory experience associated with that particular proproceptive

        feeling.

 

     4. Retrieve the feeling of your bottom resting on a surface, your

        knees bent, and your leg muscles relaxed.

 

     5. This sensory information is consistent with the =sitting?=

        sensory predicate, so you (and the entity in the image) must be

        sitting.

 

     6. There must be a chair-like object since you are sitting.

 

    Empathy offers yet another alternative to the age-old AI

    representation question: ``What is a chair?'' --- A chair is the

    feeling of sitting.

 

    My program, =EMPATH= uses this empathic problem solving technique

    to interpret the actions of a simple, worm-like creature. 

    

    #+caption: The worm performs many actions during free play such as 

    #+caption: curling, wiggling, and resting.

    #+name: worm-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-intro-white.png]]

 

    #+caption: =EMPATH= recognized and classified each of these 

    #+caption: poses by inferring the complete sensory experience 

    #+caption: from proprioceptive data.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 15cm

    [[./images/worm-poses.png]]

    

    One powerful advantage of empathic problem solving is that it

    factors the action recognition problem into two easier problems. To

    use empathy, you need an /aligner/, which takes the video and a

    model of your body, and aligns the model with the video. Then, you

    need a /recognizer/, which uses the aligned model to interpret the

    action. The power in this method lies in the fact that you describe

    all actions form a body-centered viewpoint. You are less tied to

    the particulars of any visual representation of the actions. If you

    teach the system what ``running'' is, and you have a good enough

    aligner, the system will from then on be able to recognize running

    from any point of view, even strange points of view like above or

    underneath the runner. This is in contrast to action recognition

    schemes that try to identify actions using a non-embodied approach.

    If these systems learn about running as viewed from the side, they

    will not automatically be able to recognize running from any other

    viewpoint.

 

    Another powerful advantage is that using the language of multiple

    body-centered rich senses to describe body-centerd actions offers a

    massive boost in descriptive capability. Consider how difficult it

    would be to compose a set of HOG filters to describe the action of

    a simple worm-creature ``curling'' so that its head touches its

    tail, and then behold the simplicity of describing thus action in a

    language designed for the task (listing \ref{grand-circle-intro}):

 

    #+caption: Body-centerd actions are best expressed in a body-centered 

    #+caption: language. This code detects when the worm has curled into a 

    #+caption: full circle. Imagine how you would replicate this functionality

    #+caption: using low-level pixel features such as HOG filters!

    #+name: grand-circle-intro

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn grand-circle?

   "Does the worm form a majestic circle (one end touching the other)?"

   [experiences]

   (and (curled? experiences)

        (let [worm-touch (:touch (peek experiences))

              tail-touch (worm-touch 0)

              head-touch (worm-touch 4)]

          (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))

               (< 0.2 (contact worm-segment-top-tip    head-touch))))))

    #+end_src

    #+end_listing

 

 

 **  =CORTEX= is a toolkit for building sensate creatures

 

    I built =CORTEX= to be a general AI research platform for doing

    experiments involving multiple rich senses and a wide variety and

    number of creatures. I intend it to be useful as a library for many

    more projects than just this thesis. =CORTEX= was necessary to meet

    a need among AI researchers at CSAIL and beyond, which is that

    people often will invent neat ideas that are best expressed in the

    language of creatures and senses, but in order to explore those

    ideas they must first build a platform in which they can create

    simulated creatures with rich senses! There are many ideas that

    would be simple to execute (such as =EMPATH=), but attached to them

    is the multi-month effort to make a good creature simulator. Often,

    that initial investment of time proves to be too much, and the

    project must make do with a lesser environment.

 

    =CORTEX= is well suited as an environment for embodied AI research

    for three reasons:

 

    - You can create new creatures using Blender, a popular 3D modeling

      program. Each sense can be specified using special blender nodes

      with biologically inspired paramaters. You need not write any

      code to create a creature, and can use a wide library of

      pre-existing blender models as a base for your own creatures.

 

    - =CORTEX= implements a wide variety of senses, including touch,

      proprioception, vision, hearing, and muscle tension. Complicated

      senses like touch, and vision involve multiple sensory elements

      embedded in a 2D surface. You have complete control over the

      distribution of these sensor elements through the use of simple

      png image files. In particular, =CORTEX= implements more

      comprehensive hearing than any other creature simulation system

      available. 

 

    - =CORTEX= supports any number of creatures and any number of

      senses. Time in =CORTEX= dialates so that the simulated creatures

      always precieve a perfectly smooth flow of time, regardless of

      the actual computational load.

 

    =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game

    engine designed to create cross-platform 3D desktop games. =CORTEX=

    is mainly written in clojure, a dialect of =LISP= that runs on the

    java virtual machine (JVM). The API for creating and simulating

    creatures and senses is entirely expressed in clojure, though many

    senses are implemented at the layer of jMonkeyEngine or below. For

    example, for the sense of hearing I use a layer of clojure code on

    top of a layer of java JNI bindings that drive a layer of =C++=

    code which implements a modified version of =OpenAL= to support

    multiple listeners. =CORTEX= is the only simulation environment

    that I know of that can support multiple entities that can each

    hear the world from their own perspective. Other senses also

    require a small layer of Java code. =CORTEX= also uses =bullet=, a

    physics simulator written in =C=.

 

    #+caption: Here is the worm from above modeled in Blender, a free 

    #+caption: 3D-modeling program. Senses and joints are described

    #+caption: using special nodes in Blender.

    #+name: worm-recognition-intro

    #+ATTR_LaTeX: :width 12cm

    [[./images/blender-worm.png]]

 

    Here are some thing I anticipate that =CORTEX= might be used for:

 

    - exploring new ideas about sensory integration

    - distributed communication among swarm creatures

    - self-learning using free exploration, 

    - evolutionary algorithms involving creature construction

    - exploration of exoitic senses and effectors that are not possible

      in the real world (such as telekenisis or a semantic sense)

    - imagination using subworlds

 

    During one test with =CORTEX=, I created 3,000 creatures each with

    their own independent senses and ran them all at only 1/80 real

    time. In another test, I created a detailed model of my own hand,

    equipped with a realistic distribution of touch (more sensitive at

    the fingertips), as well as eyes and ears, and it ran at around 1/4

    real time.

 

 #+BEGIN_LaTeX

    \begin{sidewaysfigure}

    \includegraphics[width=9.5in]{images/full-hand.png}

    \caption{

    I modeled my own right hand in Blender and rigged it with all the

    senses that {\tt CORTEX} supports. My simulated hand has a

    biologically inspired distribution of touch sensors. The senses are

    displayed on the right, and the simulation is displayed on the

    left. Notice that my hand is curling its fingers, that it can see

    its own finger from the eye in its palm, and that it can feel its

    own thumb touching its palm.}

    \end{sidewaysfigure}

 #+END_LaTeX

 

 ** Contributions

 

    - I built =CORTEX=, a comprehensive platform for embodied AI

      experiments. =CORTEX= supports many features lacking in other

      systems, such proper simulation of hearing. It is easy to create

      new =CORTEX= creatures using Blender, a free 3D modeling program.

 

    - I built =EMPATH=, which uses =CORTEX= to identify the actions of

      a worm-like creature using a computational model of empathy.

    

 * Building =CORTEX=

 

   I intend for =CORTEX= to be used as a general purpose library for

   building creatures and outfitting them with senses, so that it will

   be useful for other researchers who want to test out ideas of their

   own. To this end, wherver I have had to make archetictural choices

   about =CORTEX=, I have chosen to give as much freedom to the user as

   possible, so that =CORTEX= may be used for things I have not

   forseen.

 

 ** COMMENT Simulation or Reality?

    

    The most important archetictural decision of all is the choice to

    use a computer-simulated environemnt in the first place! The world

    is a vast and rich place, and for now simulations are a very poor

    reflection of its complexity. It may be that there is a significant

    qualatative difference between dealing with senses in the real

    world and dealing with pale facilimilies of them in a simulation.

    What are the advantages and disadvantages of a simulation vs.

    reality?

 

 *** Simulation

 

     The advantages of virtual reality are that when everything is a

     simulation, experiments in that simulation are absolutely

     reproducible. It's also easier to change the character and world

     to explore new situations and different sensory combinations.

 

     If the world is to be simulated on a computer, then not only do

     you have to worry about whether the character's senses are rich

     enough to learn from the world, but whether the world itself is

     rendered with enough detail and realism to give enough working

     material to the character's senses. To name just a few

     difficulties facing modern physics simulators: destructibility of

     the environment, simulation of water/other fluids, large areas,

     nonrigid bodies, lots of objects, smoke. I don't know of any

     computer simulation that would allow a character to take a rock

     and grind it into fine dust, then use that dust to make a clay

     sculpture, at least not without spending years calculating the

     interactions of every single small grain of dust. Maybe a

     simulated world with today's limitations doesn't provide enough

     richness for real intelligence to evolve.

 

 *** Reality

 

     The other approach for playing with senses is to hook your

     software up to real cameras, microphones, robots, etc., and let it

     loose in the real world. This has the advantage of eliminating

     concerns about simulating the world at the expense of increasing

     the complexity of implementing the senses. Instead of just

     grabbing the current rendered frame for processing, you have to

     use an actual camera with real lenses and interact with photons to

     get an image. It is much harder to change the character, which is

     now partly a physical robot of some sort, since doing so involves

     changing things around in the real world instead of modifying

     lines of code. While the real world is very rich and definitely

     provides enough stimulation for intelligence to develop as

     evidenced by our own existence, it is also uncontrollable in the

     sense that a particular situation cannot be recreated perfectly or

     saved for later use. It is harder to conduct science because it is

     harder to repeat an experiment. The worst thing about using the

     real world instead of a simulation is the matter of time. Instead

     of simulated time you get the constant and unstoppable flow of

     real time. This severely limits the sorts of software you can use

     to program the AI because all sense inputs must be handled in real

     time. Complicated ideas may have to be implemented in hardware or

     may simply be impossible given the current speed of our

     processors. Contrast this with a simulation, in which the flow of

     time in the simulated world can be slowed down to accommodate the

     limitations of the character's programming. In terms of cost,

     doing everything in software is far cheaper than building custom

     real-time hardware. All you need is a laptop and some patience.

 

 ** COMMENT Because of Time, simulation is perferable to reality

 

    I envision =CORTEX= being used to support rapid prototyping and

    iteration of ideas. Even if I could put together a well constructed

    kit for creating robots, it would still not be enough because of

    the scourge of real-time processing. Anyone who wants to test their

    ideas in the real world must always worry about getting their

    algorithms to run fast enough to process information in real time.

    The need for real time processing only increases if multiple senses

    are involved. In the extreme case, even simple algorithms will have

    to be accelerated by ASIC chips or FPGAs, turning what would

    otherwise be a few lines of code and a 10x speed penality into a

    multi-month ordeal. For this reason, =CORTEX= supports

    /time-dialiation/, which scales back the framerate of the

    simulation in proportion to the amount of processing each frame.

    From the perspective of the creatures inside the simulation, time

    always appears to flow at a constant rate, regardless of how

    complicated the envorimnent becomes or how many creatures are in

    the simulation. The cost is that =CORTEX= can sometimes run slower

    than real time. This can also be an advantage, however ---

    simulations of very simple creatures in =CORTEX= generally run at

    40x on my machine!

 

 ** COMMENT What is a sense?

    

    If =CORTEX= is to support a wide variety of senses, it would help

    to have a better understanding of what a ``sense'' actually is!

    While vision, touch, and hearing all seem like they are quite

    different things, I was supprised to learn during the course of

    this thesis that they (and all physical senses) can be expressed as

    exactly the same mathematical object due to a dimensional argument!

 

    Human beings are three-dimensional objects, and the nerves that

    transmit data from our various sense organs to our brain are

    essentially one-dimensional. This leaves up to two dimensions in

    which our sensory information may flow. For example, imagine your

    skin: it is a two-dimensional surface around a three-dimensional

    object (your body). It has discrete touch sensors embedded at

    various points, and the density of these sensors corresponds to the

    sensitivity of that region of skin. Each touch sensor connects to a

    nerve, all of which eventually are bundled together as they travel

    up the spinal cord to the brain. Intersect the spinal nerves with a

    guillotining plane and you will see all of the sensory data of the

    skin revealed in a roughly circular two-dimensional image which is

    the cross section of the spinal cord. Points on this image that are

    close together in this circle represent touch sensors that are

    /probably/ close together on the skin, although there is of course

    some cutting and rearrangement that has to be done to transfer the

    complicated surface of the skin onto a two dimensional image.

 

    Most human senses consist of many discrete sensors of various

    properties distributed along a surface at various densities. For

    skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's

    disks, and Ruffini's endings, which detect pressure and vibration

    of various intensities. For ears, it is the stereocilia distributed

    along the basilar membrane inside the cochlea; each one is

    sensitive to a slightly different frequency of sound. For eyes, it

    is rods and cones distributed along the surface of the retina. In

    each case, we can describe the sense with a surface and a

    distribution of sensors along that surface.

 

    The neat idea is that every human sense can be effectively

    described in terms of a surface containing embedded sensors. If the

    sense had any more dimensions, then there wouldn't be enough room

    in the spinal chord to transmit the information!

 

    Therefore, =CORTEX= must support the ability to create objects and

    then be able to ``paint'' points along their surfaces to describe

    each sense. 

 

    Fortunately this idea is already a well known computer graphics

    technique called called /UV-mapping/. The three-dimensional surface

    of a model is cut and smooshed until it fits on a two-dimensional

    image. You paint whatever you want on that image, and when the

    three-dimensional shape is rendered in a game the smooshing and

    cutting is reversed and the image appears on the three-dimensional

    object.

 

    To make a sense, interpret the UV-image as describing the

    distribution of that senses sensors. To get different types of

    sensors, you can either use a different color for each type of

    sensor, or use multiple UV-maps, each labeled with that sensor

    type. I generally use a white pixel to mean the presence of a

    sensor and a black pixel to mean the absence of a sensor, and use

    one UV-map for each sensor-type within a given sense. 

 

    #+CAPTION: The UV-map for an elongated icososphere. The white

    #+caption: dots each represent a touch sensor. They are dense 

    #+caption: in the regions that describe the tip of the finger, 

    #+caption: and less dense along the dorsal side of the finger 

    #+caption: opposite the tip.

    #+name: finger-UV

    #+ATTR_latex: :width 10cm

    [[./images/finger-UV.png]]

 

    #+caption: Ventral side of the UV-mapped finger. Notice the 

    #+caption: density of touch sensors at the tip.

    #+name: finger-side-view

    #+ATTR_LaTeX: :width 10cm

    [[./images/finger-1.png]]

 

 ** COMMENT Video game engines are a great starting point

    

    I did not need to write my own physics simulation code or shader to

    build =CORTEX=. Doing so would lead to a system that is impossible

    for anyone but myself to use anyway. Instead, I use a video game

    engine as a base and modify it to accomodate the additional needs

    of =CORTEX=. Video game engines are an ideal starting point to

    build =CORTEX=, because they are not far from being creature

    building systems themselves.

    

    First off, general purpose video game engines come with a physics

    engine and lighting / sound system. The physics system provides

    tools that can be co-opted to serve as touch, proprioception, and

    muscles. Since some games support split screen views, a good video

    game engine will allow you to efficiently create multiple cameras

    in the simulated world that can be used as eyes. Video game systems

    offer integrated asset management for things like textures and

    creatures models, providing an avenue for defining creatures. They

    also understand UV-mapping, since this technique is used to apply a

    texture to a model. Finally, because video game engines support a

    large number of users, as long as =CORTEX= doesn't stray too far

    from the base system, other researchers can turn to this community

    for help when doing their research.

    

 ** COMMENT =CORTEX= is based on jMonkeyEngine3

 

    While preparing to build =CORTEX= I studied several video game

    engines to see which would best serve as a base. The top contenders

    were:

 

    - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]]    :: The Quake II engine was designed by ID

         software in 1997.  All the source code was released by ID

         software into the Public Domain several years ago, and as a

         result it has been ported to many different languages. This

         engine was famous for its advanced use of realistic shading

         and had decent and fast physics simulation. The main advantage

         of the Quake II engine is its simplicity, but I ultimately

         rejected it because the engine is too tied to the concept of a

         first-person shooter game. One of the problems I had was that

         there does not seem to be any easy way to attach multiple

         cameras to a single character. There are also several physics

         clipping issues that are corrected in a way that only applies

         to the main character and do not apply to arbitrary objects.

 

    - [[http://source.valvesoftware.com/][Source Engine]]     :: The Source Engine evolved from the Quake II

         and Quake I engines and is used by Valve in the Half-Life

         series of games. The physics simulation in the Source Engine

         is quite accurate and probably the best out of all the engines

         I investigated. There is also an extensive community actively

         working with the engine. However, applications that use the

         Source Engine must be written in C++, the code is not open, it

         only runs on Windows, and the tools that come with the SDK to

         handle models and textures are complicated and awkward to use.

 

    -  [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating

         games in Java. It uses OpenGL to render to the screen and uses

         screengraphs to avoid drawing things that do not appear on the

         screen. It has an active community and several games in the

         pipeline. The engine was not built to serve any particular

         game but is instead meant to be used for any 3D game. 

 

    I chose jMonkeyEngine3 because it because it had the most features

    out of all the free projects I looked at, and because I could then

    write my code in clojure, an implementation of =LISP= that runs on

    the JVM.

 

 ** COMMENT =CORTEX= uses Blender to create creature models

 

    For the simple worm-like creatures I will use later on in this

    thesis, I could define a simple API in =CORTEX= that would allow

    one to create boxes, spheres, etc., and leave that API as the sole

    way to create creatures. However, for =CORTEX= to truly be useful

    for other projects, it needs a way to construct complicated

    creatures. If possible, it would be nice to leverage work that has

    already been done by the community of 3D modelers, or at least

    enable people who are talented at moedling but not programming to

    design =CORTEX= creatures.

 

    Therefore, I use Blender, a free 3D modeling program, as the main

    way to create creatures in =CORTEX=. However, the creatures modeled

    in Blender must also be simple to simulate in jMonkeyEngine3's game

    engine, and must also be easy to rig with =CORTEX='s senses. I

    accomplish this with extensive use of Blender's ``empty nodes.'' 

 

    Empty nodes have no mass, physical presence, or appearance, but

    they can hold metadata and have names. I use a tree structure of

    empty nodes to specify senses in the following manner:

 

    - Create a single top-level empty node whose name is the name of

      the sense.

    - Add empty nodes which each contain meta-data relevant to the

      sense, including a UV-map describing the number/distribution of

      sensors if applicable.

    - Make each empty-node the child of the top-level node.

      

    #+caption: An example of annoting a creature model with empty

    #+caption: nodes to describe the layout of senses. There are 

    #+caption: multiple empty nodes which each describe the position

    #+caption: of muscles, ears, eyes, or joints.

    #+name: sense-nodes

    #+ATTR_LaTeX: :width 10cm

    [[./images/empty-sense-nodes.png]]

 

 ** COMMENT Bodies are composed of segments connected by joints

 

    Blender is a general purpose animation tool, which has been used in

    the past to create high quality movies such as Sintel

    \cite{sintel}. Though Blender can model and render even complicated

    things like water, it is crucual to keep models that are meant to

    be simulated as creatures simple. =Bullet=, which =CORTEX= uses

    though jMonkeyEngine3, is a rigid-body physics system. This offers

    a compromise between the expressiveness of a game level and the

    speed at which it can be simulated, and it means that creatures

    should be naturally expressed as rigid components held together by

    joint constraints.

 

    But humans are more like a squishy bag with wrapped around some

    hard bones which define the overall shape. When we move, our skin

    bends and stretches to accomodate the new positions of our bones. 

 

    One way to make bodies composed of rigid pieces connected by joints

    /seem/ more human-like is to use an /armature/, (or /rigging/)

    system, which defines a overall ``body mesh'' and defines how the

    mesh deforms as a function of the position of each ``bone'' which

    is a standard rigid body. This technique is used extensively to

    model humans and create realistic animations. It is not a good

    technique for physical simulation, however because it creates a lie

    -- the skin is not a physical part of the simulation and does not

    interact with any objects in the world or itself. Objects will pass

    right though the skin until they come in contact with the

    underlying bone, which is a physical object. Whithout simulating

    the skin, the sense of touch has little meaning, and the creature's

    own vision will lie to it about the true extent of its body.

    Simulating the skin as a physical object requires some way to

    continuously update the physical model of the skin along with the

    movement of the bones, which is unacceptably slow compared to rigid

    body simulation. 

 

    Therefore, instead of using the human-like ``deformable bag of

    bones'' approach, I decided to base my body plans on multiple solid

    objects that are connected by joints, inspired by the robot =EVE=

    from the movie WALL-E.

    

    #+caption: =EVE= from the movie WALL-E.  This body plan turns 

    #+caption: out to be much better suited to my purposes than a more 

    #+caption: human-like one.

    #+ATTR_LaTeX: :width 10cm

    [[./images/Eve.jpg]]

 

    =EVE='s body is composed of several rigid components that are held

    together by invisible joint constraints. This is what I mean by

    ``eve-like''. The main reason that I use eve-style bodies is for

    efficiency, and so that there will be correspondence between the

    AI's semses and the physical presence of its body. Each individual

    section is simulated by a separate rigid body that corresponds

    exactly with its visual representation and does not change.

    Sections are connected by invisible joints that are well supported

    in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,

    can efficiently simulate hundreds of rigid bodies connected by

    joints. Just because sections are rigid does not mean they have to

    stay as one piece forever; they can be dynamically replaced with

    multiple sections to simulate splitting in two. This could be used

    to simulate retractable claws or =EVE='s hands, which are able to

    coalesce into one object in the movie.

 

 *** Solidifying/Connecting a body

 

     =CORTEX= creates a creature in two steps: first, it traverses the

     nodes in the blender file and creates physical representations for

     any of them that have mass defined in their blender meta-data.

 

    #+caption: Program for iterating through the nodes in a blender file

    #+caption: and generating physical jMonkeyEngine3 objects with mass

    #+caption: and a matching physics shape.

    #+name: name

    #+begin_listing clojure

    #+begin_src clojure

 (defn physical!

   "Iterate through the nodes in creature and make them real physical

    objects in the simulation."

   [#^Node creature]

   (dorun

    (map

     (fn [geom]

       (let [physics-control

             (RigidBodyControl.

              (HullCollisionShape.

               (.getMesh geom))

              (if-let [mass (meta-data geom "mass")]

                (float mass) (float 1)))]

         (.addControl geom physics-control)))

     (filter #(isa? (class %) Geometry )

             (node-seq creature)))))

    #+end_src

    #+end_listing

    

     The next step to making a proper body is to connect those pieces

     together with joints. jMonkeyEngine has a large array of joints

     available via =bullet=, such as Point2Point, Cone, Hinge, and a

     generic Six Degree of Freedom joint, with or without spring

     restitution. 

 

     Joints are treated a lot like proper senses, in that there is a

     top-level empty node named ``joints'' whose children each

     represent a joint.

 

     #+caption: View of the hand model in Blender showing the main ``joints''

     #+caption: node (highlighted in yellow) and its children which each

     #+caption: represent a joint in the hand. Each joint node has metadata

     #+caption: specifying what sort of joint it is.

     #+name: blender-hand

     #+ATTR_LaTeX: :width 10cm

     [[./images/hand-screenshot1.png]]

 

 

     =CORTEX='s procedure for binding the creature together with joints

     is as follows:

     

     - Find the children of the ``joints'' node.

     - Determine the two spatials the joint is meant to connect.

     - Create the joint based on the meta-data of the empty node.

 

     The higher order function =sense-nodes= from =cortex.sense=

     simplifies finding the joints based on their parent ``joints''

     node.

 

    #+caption: Retrieving the children empty nodes from a single 

    #+caption: named empty node is a common pattern in =CORTEX=

    #+caption: further instances of this technique for the senses 

    #+caption: will be omitted

    #+name: get-empty-nodes

    #+begin_listing clojure

    #+begin_src clojure

 (defn sense-nodes

   "For some senses there is a special empty blender node whose

    children are considered markers for an instance of that sense. This

    function generates functions to find those children, given the name

    of the special parent node."

   [parent-name]

   (fn [#^Node creature]

     (if-let [sense-node (.getChild creature parent-name)]

       (seq (.getChildren sense-node)) [])))

 

 (def

   ^{:doc "Return the children of the creature's \"joints\" node."

     :arglists '([creature])}

   joints

   (sense-nodes "joints"))

    #+end_src

    #+end_listing

 

     To find a joint's targets, =CORTEX= creates a small cube, centered

     around the empty-node, and grows the cube exponentially until it

     intersects two physical objects. The objects are ordered according

     to the joint's rotation, with the first one being the object that

     has more negative coordinates in the joint's reference frame.

     Since the objects must be physical, the empty-node itself escapes

     detection. Because the objects must be physical, =joint-targets=

     must be called /after/ =physical!= is called.

    

     #+caption: Program to find the targets of a joint node by 

     #+caption: exponentiallly growth of a search cube.

     #+name: joint-targets

     #+begin_listing clojure

     #+begin_src clojure

 (defn joint-targets

   "Return the two closest two objects to the joint object, ordered

   from bottom to top according to the joint's rotation."

   [#^Node parts #^Node joint]

   (loop [radius (float 0.01)]

     (let [results (CollisionResults.)]

       (.collideWith

        parts

        (BoundingBox. (.getWorldTranslation joint)

                      radius radius radius) results)

       (let [targets

             (distinct

              (map  #(.getGeometry %) results))]

         (if (>= (count targets) 2)

           (sort-by

            #(let [joint-ref-frame-position

                   (jme-to-blender

                    (.mult

                     (.inverse (.getWorldRotation joint))

                     (.subtract (.getWorldTranslation %)

                                (.getWorldTranslation joint))))]

               (.dot (Vector3f. 1 1 1) joint-ref-frame-position))                  

            (take 2 targets))

           (recur (float (* radius 2))))))))

     #+end_src

     #+end_listing

    

     Once =CORTEX= finds all joints and targets, it creates them using

     a dispatch on the metadata of each joint node.

 

     #+caption: Program to dispatch on blender metadata and create joints

     #+caption: sutiable for physical simulation.

     #+name: joint-dispatch

     #+begin_listing clojure

     #+begin_src clojure

 (defmulti joint-dispatch

   "Translate blender pseudo-joints into real JME joints."

   (fn [constraints & _] 

     (:type constraints)))

 

 (defmethod joint-dispatch :point

   [constraints control-a control-b pivot-a pivot-b rotation]

   (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)

     (.setLinearLowerLimit Vector3f/ZERO)

     (.setLinearUpperLimit Vector3f/ZERO)))

 

 (defmethod joint-dispatch :hinge

   [constraints control-a control-b pivot-a pivot-b rotation]

   (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)

         [limit-1 limit-2] (:limit constraints)

         hinge-axis (.mult rotation (blender-to-jme axis))]

     (doto (HingeJoint. control-a control-b pivot-a pivot-b 

                        hinge-axis hinge-axis)

       (.setLimit limit-1 limit-2))))

 

 (defmethod joint-dispatch :cone

   [constraints control-a control-b pivot-a pivot-b rotation]

   (let [limit-xz (:limit-xz constraints)

         limit-xy (:limit-xy constraints)

         twist    (:twist constraints)]

     (doto (ConeJoint. control-a control-b pivot-a pivot-b

                       rotation rotation)

       (.setLimit (float limit-xz) (float limit-xy)

                  (float twist)))))

     #+end_src

     #+end_listing

 

     All that is left for joints it to combine the above pieces into a

     something that can operate on the collection of nodes that a

     blender file represents.

 

     #+caption: Program to completely create a joint given information 

     #+caption: from a blender file.

     #+name: connect

     #+begin_listing clojure

    #+begin_src clojure

 (defn connect

   "Create a joint between 'obj-a and 'obj-b at the location of

   'joint. The type of joint is determined by the metadata on 'joint.

 

    Here are some examples:

    {:type :point}

    {:type :hinge  :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}

    (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)

 

    {:type :cone :limit-xz 0]

                 :limit-xy 0]

                 :twist 0]}   (use XZY rotation mode in blender!)"

   [#^Node obj-a #^Node obj-b #^Node joint]

   (let [control-a (.getControl obj-a RigidBodyControl)

         control-b (.getControl obj-b RigidBodyControl)

         joint-center (.getWorldTranslation joint)

         joint-rotation (.toRotationMatrix (.getWorldRotation joint))

         pivot-a (world-to-local obj-a joint-center)

         pivot-b (world-to-local obj-b joint-center)]

     (if-let

         [constraints (map-vals eval (read-string (meta-data joint "joint")))]

       ;; A side-effect of creating a joint registers

       ;; it with both physics objects which in turn

       ;; will register the joint with the physics system

       ;; when the simulation is started.

         (joint-dispatch constraints

                         control-a control-b

                         pivot-a pivot-b

                         joint-rotation))))

     #+end_src

     #+end_listing

 

     In general, whenever =CORTEX= exposes a sense (or in this case

     physicality), it provides a function of the type =sense!=, which

     takes in a collection of nodes and augments it to support that

     sense. The function returns any controlls necessary to use that

     sense. In this case =body!= cerates a physical body and returns no

     control functions.

 

     #+caption: Program to give joints to a creature.

     #+name: name

     #+begin_listing clojure

     #+begin_src clojure

 (defn joints!

   "Connect the solid parts of the creature with physical joints. The

    joints are taken from the \"joints\" node in the creature."

   [#^Node creature]

   (dorun

    (map

     (fn [joint]

       (let [[obj-a obj-b] (joint-targets creature joint)]

         (connect obj-a obj-b joint)))

     (joints creature))))

 (defn body!

   "Endow the creature with a physical body connected with joints.  The

    particulars of the joints and the masses of each body part are

    determined in blender."

   [#^Node creature]

   (physical! creature)

   (joints! creature))

     #+end_src

     #+end_listing

 

     All of the code you have just seen amounts to only 130 lines, yet

     because it builds on top of Blender and jMonkeyEngine3, those few

     lines pack quite a punch!

 

     The hand from figure \ref{blender-hand}, which was modeled after

     my own right hand, can now be given joints and simulated as a

     creature.

    

     #+caption: With the ability to create physical creatures from blender,

     #+caption: =CORTEX= gets one step closer to becomming a full creature

     #+caption: simulation environment.

     #+name: name

     #+ATTR_LaTeX: :width 15cm

     [[./images/physical-hand.png]]

 

 ** Eyes reuse standard video game components

 

 ** Hearing is hard; =CORTEX= does it right

 

 ** Touch uses hundreds of hair-like elements

 

 ** Proprioception is the sense that makes everything ``real''

 

 ** Muscles are both effectors and sensors

 

 ** =CORTEX= brings complex creatures to life!

 

 ** =CORTEX= enables many possiblities for further research

 

 * COMMENT Empathy in a simulated worm

 

   Here I develop a computational model of empathy, using =CORTEX= as a

   base. Empathy in this context is the ability to observe another

   creature and infer what sorts of sensations that creature is

   feeling. My empathy algorithm involves multiple phases. First is

   free-play, where the creature moves around and gains sensory

   experience. From this experience I construct a representation of the

   creature's sensory state space, which I call \Phi-space. Using

   \Phi-space, I construct an efficient function which takes the

   limited data that comes from observing another creature and enriches

   it full compliment of imagined sensory data. I can then use the

   imagined sensory data to recognize what the observed creature is

   doing and feeling, using straightforward embodied action predicates.

   This is all demonstrated with using a simple worm-like creature, and

   recognizing worm-actions based on limited data.

 

   #+caption: Here is the worm with which we will be working. 

   #+caption: It is composed of 5 segments. Each segment has a 

   #+caption: pair of extensor and flexor muscles. Each of the 

   #+caption: worm's four joints is a hinge joint which allows 

   #+caption: about 30 degrees of rotation to either side. Each segment

   #+caption: of the worm is touch-capable and has a uniform 

   #+caption: distribution of touch sensors on each of its faces.

   #+caption: Each joint has a proprioceptive sense to detect 

   #+caption: relative positions. The worm segments are all the 

   #+caption: same except for the first one, which has a much

   #+caption: higher weight than the others to allow for easy 

   #+caption: manual motor control.

   #+name: basic-worm-view

   #+ATTR_LaTeX: :width 10cm

   [[./images/basic-worm-view.png]]

 

   #+caption: Program for reading a worm from a blender file and 

   #+caption: outfitting it with the senses of proprioception, 

   #+caption: touch, and the ability to move, as specified in the 

   #+caption: blender file.

   #+name: get-worm

   #+begin_listing clojure

   #+begin_src clojure

 (defn worm []

   (let [model (load-blender-model "Models/worm/worm.blend")]

     {:body (doto model (body!))

      :touch (touch! model)

      :proprioception (proprioception! model)

      :muscles (movement! model)}))

   #+end_src

   #+end_listing

 

 ** Embodiment factors action recognition into managable parts

 

    Using empathy, I divide the problem of action recognition into a

    recognition process expressed in the language of a full compliment

    of senses, and an imaganitive process that generates full sensory

    data from partial sensory data. Splitting the action recognition

    problem in this manner greatly reduces the total amount of work to

    recognize actions: The imaganitive process is mostly just matching

    previous experience, and the recognition process gets to use all

    the senses to directly describe any action.

 

 ** Action recognition is easy with a full gamut of senses

 

    Embodied representations using multiple senses such as touch,

    proprioception, and muscle tension turns out be be exceedingly

    efficient at describing body-centered actions. It is the ``right

    language for the job''. For example, it takes only around 5 lines

    of LISP code to describe the action of ``curling'' using embodied

    primitives. It takes about 10 lines to describe the seemingly

    complicated action of wiggling.

 

    The following action predicates each take a stream of sensory

    experience, observe however much of it they desire, and decide

    whether the worm is doing the action they describe. =curled?=

    relies on proprioception, =resting?= relies on touch, =wiggling?=

    relies on a fourier analysis of muscle contraction, and

    =grand-circle?= relies on touch and reuses =curled?= as a gaurd.

    

    #+caption: Program for detecting whether the worm is curled. This is the 

    #+caption: simplest action predicate, because it only uses the last frame 

    #+caption: of sensory experience, and only uses proprioceptive data. Even 

    #+caption: this simple predicate, however, is automatically frame 

    #+caption: independent and ignores vermopomorphic differences such as 

    #+caption: worm textures and colors.

    #+name: curled

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn curled?

   "Is the worm curled up?"

   [experiences]

   (every?

    (fn [[_ _ bend]]

      (> (Math/sin bend) 0.64))

    (:proprioception (peek experiences))))

    #+end_src

    #+end_listing

 

    #+caption: Program for summarizing the touch information in a patch 

    #+caption: of skin.

    #+name: touch-summary

    #+attr_latex: [htpb]

 

 #+begin_listing clojure

    #+begin_src clojure

 (defn contact

   "Determine how much contact a particular worm segment has with

    other objects. Returns a value between 0 and 1, where 1 is full

    contact and 0 is no contact."

   [touch-region [coords contact :as touch]]

   (-> (zipmap coords contact)

       (select-keys touch-region)

       (vals)

       (#(map first %))

       (average)

       (* 10)

       (- 1)

       (Math/abs)))

    #+end_src

    #+end_listing

 

 

    #+caption: Program for detecting whether the worm is at rest. This program

    #+caption: uses a summary of the tactile information from the underbelly 

    #+caption: of the worm, and is only true if every segment is touching the 

    #+caption: floor. Note that this function contains no references to 

    #+caption: proprioction at all.

    #+name: resting

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (def worm-segment-bottom (rect-region [8 15] [14 22]))

 

 (defn resting?

   "Is the worm resting on the ground?"

   [experiences]

   (every?

    (fn [touch-data]

      (< 0.9 (contact worm-segment-bottom touch-data)))

    (:touch (peek experiences))))

    #+end_src

    #+end_listing

 

    #+caption: Program for detecting whether the worm is curled up into a 

    #+caption: full circle. Here the embodied approach begins to shine, as

    #+caption: I am able to both use a previous action predicate (=curled?=)

    #+caption: as well as the direct tactile experience of the head and tail.

    #+name: grand-circle

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))

 

 (def worm-segment-top-tip (rect-region [0 15] [7 22]))

 

 (defn grand-circle?

   "Does the worm form a majestic circle (one end touching the other)?"

   [experiences]

   (and (curled? experiences)

        (let [worm-touch (:touch (peek experiences))

              tail-touch (worm-touch 0)

              head-touch (worm-touch 4)]

          (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))

               (< 0.55 (contact worm-segment-top-tip    head-touch))))))

    #+end_src

    #+end_listing

 

 

    #+caption: Program for detecting whether the worm has been wiggling for 

    #+caption: the last few frames. It uses a fourier analysis of the muscle 

    #+caption: contractions of the worm's tail to determine wiggling. This is 

    #+caption: signigicant because there is no particular frame that clearly 

    #+caption: indicates that the worm is wiggling --- only when multiple frames 

    #+caption: are analyzed together is the wiggling revealed. Defining 

    #+caption: wiggling this way also gives the worm an opportunity to learn 

    #+caption: and recognize ``frustrated wiggling'', where the worm tries to 

    #+caption: wiggle but can't. Frustrated wiggling is very visually different 

    #+caption: from actual wiggling, but this definition gives it to us for free.

    #+name: wiggling

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn fft [nums]

   (map

    #(.getReal %)

    (.transform

     (FastFourierTransformer. DftNormalization/STANDARD)

     (double-array nums) TransformType/FORWARD)))

 

 (def indexed (partial map-indexed vector))

 

 (defn max-indexed [s]

   (first (sort-by (comp - second) (indexed s))))

 

 (defn wiggling?

   "Is the worm wiggling?"

   [experiences]

   (let [analysis-interval 0x40]

     (when (> (count experiences) analysis-interval)

       (let [a-flex 3

             a-ex   2

             muscle-activity

             (map :muscle (vector:last-n experiences analysis-interval))

             base-activity

             (map #(- (% a-flex) (% a-ex)) muscle-activity)]

         (= 2

            (first

             (max-indexed

              (map #(Math/abs %)

                   (take 20 (fft base-activity))))))))))

    #+end_src

    #+end_listing

 

    With these action predicates, I can now recognize the actions of

    the worm while it is moving under my control and I have access to

    all the worm's senses.

 

    #+caption: Use the action predicates defined earlier to report on 

    #+caption: what the worm is doing while in simulation.

    #+name: report-worm-activity

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn debug-experience

   [experiences text]

   (cond

    (grand-circle? experiences) (.setText text "Grand Circle")

    (curled? experiences)       (.setText text "Curled")

    (wiggling? experiences)     (.setText text "Wiggling")

    (resting? experiences)      (.setText text "Resting")))

    #+end_src

    #+end_listing

 

    #+caption: Using =debug-experience=, the body-centered predicates

    #+caption: work together to classify the behaviour of the worm. 

    #+caption: the predicates are operating with access to the worm's

    #+caption: full sensory data.

    #+name: basic-worm-view

    #+ATTR_LaTeX: :width 10cm

    [[./images/worm-identify-init.png]]

 

    These action predicates satisfy the recognition requirement of an

    empathic recognition system. There is power in the simplicity of

    the action predicates. They describe their actions without getting

    confused in visual details of the worm. Each one is frame

    independent, but more than that, they are each indepent of

    irrelevant visual details of the worm and the environment. They

    will work regardless of whether the worm is a different color or

    hevaily textured, or if the environment has strange lighting.

 

    The trick now is to make the action predicates work even when the

    sensory data on which they depend is absent. If I can do that, then

    I will have gained much,

 

 ** \Phi-space describes the worm's experiences

    

    As a first step towards building empathy, I need to gather all of

    the worm's experiences during free play. I use a simple vector to

    store all the experiences. 

 

    Each element of the experience vector exists in the vast space of

    all possible worm-experiences. Most of this vast space is actually

    unreachable due to physical constraints of the worm's body. For

    example, the worm's segments are connected by hinge joints that put

    a practical limit on the worm's range of motions without limiting

    its degrees of freedom. Some groupings of senses are impossible;

    the worm can not be bent into a circle so that its ends are

    touching and at the same time not also experience the sensation of

    touching itself.

 

    As the worm moves around during free play and its experience vector

    grows larger, the vector begins to define a subspace which is all

    the sensations the worm can practicaly experience during normal

    operation. I call this subspace \Phi-space, short for

    physical-space. The experience vector defines a path through

    \Phi-space. This path has interesting properties that all derive

    from physical embodiment. The proprioceptive components are

    completely smooth, because in order for the worm to move from one

    position to another, it must pass through the intermediate

    positions. The path invariably forms loops as actions are repeated.

    Finally and most importantly, proprioception actually gives very

    strong inference about the other senses. For example, when the worm

    is flat, you can infer that it is touching the ground and that its

    muscles are not active, because if the muscles were active, the

    worm would be moving and would not be perfectly flat. In order to

    stay flat, the worm has to be touching the ground, or it would

    again be moving out of the flat position due to gravity. If the

    worm is positioned in such a way that it interacts with itself,

    then it is very likely to be feeling the same tactile feelings as

    the last time it was in that position, because it has the same body

    as then. If you observe multiple frames of proprioceptive data,

    then you can become increasingly confident about the exact

    activations of the worm's muscles, because it generally takes a

    unique combination of muscle contractions to transform the worm's

    body along a specific path through \Phi-space.

 

    There is a simple way of taking \Phi-space and the total ordering

    provided by an experience vector and reliably infering the rest of

    the senses.

 

 ** Empathy is the process of tracing though \Phi-space 

 

    Here is the core of a basic empathy algorithm, starting with an

    experience vector:

 

    First, group the experiences into tiered proprioceptive bins. I use

    powers of 10 and 3 bins, and the smallest bin has an approximate

    size of 0.001 radians in all proprioceptive dimensions.

    

    Then, given a sequence of proprioceptive input, generate a set of

    matching experience records for each input, using the tiered

    proprioceptive bins. 

 

    Finally, to infer sensory data, select the longest consective chain

    of experiences. Conecutive experience means that the experiences

    appear next to each other in the experience vector.

 

    This algorithm has three advantages: 

 

    1. It's simple

 

    3. It's very fast -- retrieving possible interpretations takes

       constant time. Tracing through chains of interpretations takes

       time proportional to the average number of experiences in a

       proprioceptive bin. Redundant experiences in \Phi-space can be

       merged to save computation.

 

    2. It protects from wrong interpretations of transient ambiguous

       proprioceptive data. For example, if the worm is flat for just

       an instant, this flattness will not be interpreted as implying

       that the worm has its muscles relaxed, since the flattness is

       part of a longer chain which includes a distinct pattern of

       muscle activation. Markov chains or other memoryless statistical

       models that operate on individual frames may very well make this

       mistake.

 

    #+caption: Program to convert an experience vector into a 

    #+caption: proprioceptively binned lookup function.

    #+name: bin

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn bin [digits]

   (fn [angles]

     (->> angles

          (flatten)

          (map (juxt #(Math/sin %) #(Math/cos %)))

          (flatten)

          (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))

 

 (defn gen-phi-scan 

   "Nearest-neighbors with binning. Only returns a result if

    the propriceptive data is within 10% of a previously recorded

    result in all dimensions."

   [phi-space]

   (let [bin-keys (map bin [3 2 1])

         bin-maps

         (map (fn [bin-key]

                (group-by

                 (comp bin-key :proprioception phi-space)

                 (range (count phi-space)))) bin-keys)

         lookups (map (fn [bin-key bin-map]

                        (fn [proprio] (bin-map (bin-key proprio))))

                      bin-keys bin-maps)]

     (fn lookup [proprio-data]

       (set (some #(% proprio-data) lookups)))))

    #+end_src

    #+end_listing

 

    #+caption: =longest-thread= finds the longest path of consecutive 

    #+caption: experiences to explain proprioceptive worm data.

    #+name: phi-space-history-scan

    #+ATTR_LaTeX: :width 10cm

    [[./images/aurellem-gray.png]]

 

    =longest-thread= infers sensory data by stitching together pieces

    from previous experience. It prefers longer chains of previous

    experience to shorter ones. For example, during training the worm

    might rest on the ground for one second before it performs its

    excercises. If during recognition the worm rests on the ground for

    five seconds, =longest-thread= will accomodate this five second

    rest period by looping the one second rest chain five times.

 

    =longest-thread= takes time proportinal to the average number of

    entries in a proprioceptive bin, because for each element in the

    starting bin it performes a series of set lookups in the preceeding

    bins. If the total history is limited, then this is only a constant

    multiple times the number of entries in the starting bin. This

    analysis also applies even if the action requires multiple longest

    chains -- it's still the average number of entries in a

    proprioceptive bin times the desired chain length. Because

    =longest-thread= is so efficient and simple, I can interpret

    worm-actions in real time.

 

    #+caption: Program to calculate empathy by tracing though \Phi-space

    #+caption: and finding the longest (ie. most coherent) interpretation

    #+caption: of the data.

    #+name: longest-thread

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn longest-thread

   "Find the longest thread from phi-index-sets. The index sets should

    be ordered from most recent to least recent."

   [phi-index-sets]

   (loop [result '()

          [thread-bases & remaining :as phi-index-sets] phi-index-sets]

     (if (empty? phi-index-sets)

       (vec result)

       (let [threads

             (for [thread-base thread-bases]

               (loop [thread (list thread-base)

                      remaining remaining]

                 (let [next-index (dec (first thread))]

                   (cond (empty? remaining) thread

                         (contains? (first remaining) next-index)

                         (recur

                          (cons next-index thread) (rest remaining))

                         :else thread))))

             longest-thread

             (reduce (fn [thread-a thread-b]

                       (if (> (count thread-a) (count thread-b))

                         thread-a thread-b))

                     '(nil)

                     threads)]

         (recur (concat longest-thread result)

                (drop (count longest-thread) phi-index-sets))))))

    #+end_src

    #+end_listing

 

    There is one final piece, which is to replace missing sensory data

    with a best-guess estimate. While I could fill in missing data by

    using a gradient over the closest known sensory data points,

    averages can be misleading. It is certainly possible to create an

    impossible sensory state by averaging two possible sensory states.

    Therefore, I simply replicate the most recent sensory experience to

    fill in the gaps.

 

    #+caption: Fill in blanks in sensory experience by replicating the most 

    #+caption: recent experience.

    #+name: infer-nils

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn infer-nils

   "Replace nils with the next available non-nil element in the

    sequence, or barring that, 0."

   [s]

   (loop [i (dec (count s))

          v (transient s)]

     (if (zero? i) (persistent! v)

         (if-let [cur (v i)]

           (if (get v (dec i) 0)

             (recur (dec i) v)

             (recur (dec i) (assoc! v (dec i) cur)))

           (recur i (assoc! v i 0))))))

    #+end_src

    #+end_listing

   

 ** Efficient action recognition with =EMPATH=

    

    To use =EMPATH= with the worm, I first need to gather a set of

    experiences from the worm that includes the actions I want to

    recognize. The =generate-phi-space= program (listing

    \ref{generate-phi-space} runs the worm through a series of

    exercices and gatheres those experiences into a vector. The

    =do-all-the-things= program is a routine expressed in a simple

    muscle contraction script language for automated worm control. It

    causes the worm to rest, curl, and wiggle over about 700 frames

    (approx. 11 seconds).

 

    #+caption: Program to gather the worm's experiences into a vector for 

    #+caption: further processing. The =motor-control-program= line uses

    #+caption: a motor control script that causes the worm to execute a series

    #+caption: of ``exercices'' that include all the action predicates.

    #+name: generate-phi-space

    #+attr_latex: [htpb]

 #+begin_listing clojure 

    #+begin_src clojure

 (def do-all-the-things 

   (concat

    curl-script

    [[300 :d-ex 40]

     [320 :d-ex 0]]

    (shift-script 280 (take 16 wiggle-script))))

 

 (defn generate-phi-space []

   (let [experiences (atom [])]

     (run-world

      (apply-map 

       worm-world

       (merge

        (worm-world-defaults)

        {:end-frame 700

         :motor-control

         (motor-control-program worm-muscle-labels do-all-the-things)

         :experiences experiences})))

     @experiences))

    #+end_src

    #+end_listing

 

    #+caption: Use longest thread and a phi-space generated from a short

    #+caption: exercise routine to interpret actions during free play.

    #+name: empathy-debug

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (defn init []

   (def phi-space (generate-phi-space))

   (def phi-scan (gen-phi-scan phi-space)))

 

 (defn empathy-demonstration []

   (let [proprio (atom ())]

     (fn

       [experiences text]

       (let [phi-indices (phi-scan (:proprioception (peek experiences)))]

         (swap! proprio (partial cons phi-indices))

         (let [exp-thread (longest-thread (take 300 @proprio))

               empathy (mapv phi-space (infer-nils exp-thread))]

           (println-repl (vector:last-n exp-thread 22))

           (cond

            (grand-circle? empathy) (.setText text "Grand Circle")

            (curled? empathy)       (.setText text "Curled")

            (wiggling? empathy)     (.setText text "Wiggling")

            (resting? empathy)      (.setText text "Resting")

            :else                       (.setText text "Unknown")))))))

 

 (defn empathy-experiment [record]

   (.start (worm-world :experience-watch (debug-experience-phi)

                       :record record :worm worm*)))

    #+end_src

    #+end_listing

    

    The result of running =empathy-experiment= is that the system is

    generally able to interpret worm actions using the action-predicates

    on simulated sensory data just as well as with actual data. Figure

    \ref{empathy-debug-image} was generated using =empathy-experiment=:

 

   #+caption: From only proprioceptive data, =EMPATH= was able to infer 

   #+caption: the complete sensory experience and classify four poses

   #+caption: (The last panel shows a composite image of \emph{wriggling}, 

   #+caption: a dynamic pose.)

   #+name: empathy-debug-image

   #+ATTR_LaTeX: :width 10cm :placement [H]

   [[./images/empathy-1.png]]

 

   One way to measure the performance of =EMPATH= is to compare the

   sutiability of the imagined sense experience to trigger the same

   action predicates as the real sensory experience. 

   

    #+caption: Determine how closely empathy approximates actual 

    #+caption: sensory data.

    #+name: test-empathy-accuracy

    #+attr_latex: [htpb]

 #+begin_listing clojure

    #+begin_src clojure

 (def worm-action-label

   (juxt grand-circle? curled? wiggling?))

 

 (defn compare-empathy-with-baseline [matches]

   (let [proprio (atom ())]

     (fn

       [experiences text]

       (let [phi-indices (phi-scan (:proprioception (peek experiences)))]

         (swap! proprio (partial cons phi-indices))

         (let [exp-thread (longest-thread (take 300 @proprio))

               empathy (mapv phi-space (infer-nils exp-thread))

               experience-matches-empathy

               (= (worm-action-label experiences)

                  (worm-action-label empathy))]

           (println-repl experience-matches-empathy)

           (swap! matches #(conj % experience-matches-empathy)))))))

               

 (defn accuracy [v]

   (float (/ (count (filter true? v)) (count v))))

 

 (defn test-empathy-accuracy []

   (let [res (atom [])]

     (run-world

      (worm-world :experience-watch

                  (compare-empathy-with-baseline res)

                  :worm worm*))

     (accuracy @res)))

    #+end_src

    #+end_listing

 

   Running =test-empathy-accuracy= using the very short exercise

   program defined in listing \ref{generate-phi-space}, and then doing

   a similar pattern of activity manually yeilds an accuracy of around

   73%. This is based on very limited worm experience. By training the

   worm for longer, the accuracy dramatically improves.

 

    #+caption: Program to generate \Phi-space using manual training.

    #+name: manual-phi-space

    #+attr_latex: [htpb]

    #+begin_listing clojure

    #+begin_src clojure

 (defn init-interactive []

   (def phi-space

     (let [experiences (atom [])]

       (run-world

        (apply-map 

         worm-world

         (merge

          (worm-world-defaults)

          {:experiences experiences})))

       @experiences))

   (def phi-scan (gen-phi-scan phi-space)))

    #+end_src

    #+end_listing

 

   After about 1 minute of manual training, I was able to achieve 95%

   accuracy on manual testing of the worm using =init-interactive= and

   =test-empathy-accuracy=. The majority of errors are near the

   boundaries of transitioning from one type of action to another.

   During these transitions the exact label for the action is more open

   to interpretation, and dissaggrement between empathy and experience

   is more excusable.

 

 ** Digression: bootstrapping touch using free exploration

 

    In the previous section I showed how to compute actions in terms of

    body-centered predicates which relied averate touch activation of

    pre-defined regions of the worm's skin. What if, instead of recieving

    touch pre-grouped into the six faces of each worm segment, the true

    topology of the worm's skin was unknown? This is more similiar to how

    a nerve fiber bundle might be arranged. While two fibers that are

    close in a nerve bundle /might/ correspond to two touch sensors that

    are close together on the skin, the process of taking a complicated

    surface and forcing it into essentially a circle requires some cuts

    and rerragenments.

    

    In this section I show how to automatically learn the skin-topology of

    a worm segment by free exploration. As the worm rolls around on the

    floor, large sections of its surface get activated. If the worm has

    stopped moving, then whatever region of skin that is touching the

    floor is probably an important region, and should be recorded.

    

    #+caption: Program to detect whether the worm is in a resting state 

    #+caption: with one face touching the floor.

    #+name: pure-touch

    #+begin_listing clojure

    #+begin_src clojure

 (def full-contact [(float 0.0) (float 0.1)])

 

 (defn pure-touch?

   "This is worm specific code to determine if a large region of touch

    sensors is either all on or all off."

   [[coords touch :as touch-data]]

   (= (set (map first touch)) (set full-contact)))

    #+end_src

    #+end_listing

 

    After collecting these important regions, there will many nearly

    similiar touch regions. While for some purposes the subtle

    differences between these regions will be important, for my

    purposes I colapse them into mostly non-overlapping sets using

    =remove-similiar= in listing \ref{remove-similiar}

 

    #+caption: Program to take a lits of set of points and ``collapse them''

    #+caption: so that the remaining sets in the list are siginificantly 

    #+caption: different from each other. Prefer smaller sets to larger ones.

    #+name: remove-similiar

    #+begin_listing clojure

    #+begin_src clojure

 (defn remove-similar

   [coll]

   (loop [result () coll (sort-by (comp - count) coll)]

     (if (empty? coll) result

         (let  [[x & xs] coll

                c (count x)]

           (if (some

                (fn [other-set]

                  (let [oc (count other-set)]

                    (< (- (count (union other-set x)) c) (* oc 0.1))))

                xs)

             (recur result xs)

             (recur (cons x result) xs))))))

    #+end_src

    #+end_listing

 

    Actually running this simulation is easy given =CORTEX='s facilities.

 

    #+caption: Collect experiences while the worm moves around. Filter the touch 

    #+caption: sensations by stable ones, collapse similiar ones together, 

    #+caption: and report the regions learned.

    #+name: learn-touch

    #+begin_listing clojure

    #+begin_src clojure

 (defn learn-touch-regions []

   (let [experiences (atom [])

         world (apply-map

                worm-world

                (assoc (worm-segment-defaults)

                  :experiences experiences))]

     (run-world world)

     (->>

      @experiences

      (drop 175)

      ;; access the single segment's touch data

      (map (comp first :touch))

      ;; only deal with "pure" touch data to determine surfaces

      (filter pure-touch?)

      ;; associate coordinates with touch values

      (map (partial apply zipmap))

      ;; select those regions where contact is being made

      (map (partial group-by second))

      (map #(get % full-contact))

      (map (partial map first))

      ;; remove redundant/subset regions

      (map set)

      remove-similar)))

 

 (defn learn-and-view-touch-regions []

   (map view-touch-region

        (learn-touch-regions)))

    #+end_src

    #+end_listing

 

    The only thing remining to define is the particular motion the worm

    must take. I accomplish this with a simple motor control program.

 

    #+caption: Motor control program for making the worm roll on the ground.

    #+caption: This could also be replaced with random motion.

    #+name: worm-roll

    #+begin_listing clojure

    #+begin_src clojure

 (defn touch-kinesthetics []

   [[170 :lift-1 40]

    [190 :lift-1 19]

    [206 :lift-1  0]

 

    [400 :lift-2 40]

    [410 :lift-2  0]

 

    [570 :lift-2 40]

    [590 :lift-2 21]

    [606 :lift-2  0]

 

    [800 :lift-1 30]

    [809 :lift-1 0]

 

    [900 :roll-2 40]

    [905 :roll-2 20]

    [910 :roll-2  0]

 

    [1000 :roll-2 40]

    [1005 :roll-2 20]

    [1010 :roll-2  0]

    

    [1100 :roll-2 40]

    [1105 :roll-2 20]

    [1110 :roll-2  0]

    ])

    #+end_src

    #+end_listing

 

 

    #+caption: The small worm rolls around on the floor, driven

    #+caption: by the motor control program in listing \ref{worm-roll}.

    #+name: worm-roll

    #+ATTR_LaTeX: :width 12cm

    [[./images/worm-roll.png]]

 

 

    #+caption: After completing its adventures, the worm now knows 

    #+caption: how its touch sensors are arranged along its skin. These 

    #+caption: are the regions that were deemed important by 

    #+caption: =learn-touch-regions=. Note that the worm has discovered

    #+caption: that it has six sides.

    #+name: worm-touch-map

    #+ATTR_LaTeX: :width 12cm

    [[./images/touch-learn.png]]

 

    While simple, =learn-touch-regions= exploits regularities in both

    the worm's physiology and the worm's environment to correctly

    deduce that the worm has six sides. Note that =learn-touch-regions=

    would work just as well even if the worm's touch sense data were

    completely scrambled. The cross shape is just for convienence. This

    example justifies the use of pre-defined touch regions in =EMPATH=.

 

 * COMMENT Contributions

   

   In this thesis you have seen the =CORTEX= system, a complete

   environment for creating simulated creatures. You have seen how to

   implement five senses including touch, proprioception, hearing,

   vision, and muscle tension. You have seen how to create new creatues

   using blender, a 3D modeling tool. I hope that =CORTEX= will be

   useful in further research projects. To this end I have included the

   full source to =CORTEX= along with a large suite of tests and

   examples. I have also created a user guide for =CORTEX= which is

   inculded in an appendix to this thesis.

 

   You have also seen how I used =CORTEX= as a platform to attach the

   /action recognition/ problem, which is the problem of recognizing

   actions in video. You saw a simple system called =EMPATH= which

   ientifies actions by first describing actions in a body-centerd,

   rich sense language, then infering a full range of sensory

   experience from limited data using previous experience gained from

   free play.

 

   As a minor digression, you also saw how I used =CORTEX= to enable a

   tiny worm to discover the topology of its skin simply by rolling on

   the ground. 

 

   In conclusion, the main contributions of this thesis are:

 

   - =CORTEX=, a system for creating simulated creatures with rich

     senses.

   - =EMPATH=, a program for recognizing actions by imagining sensory

     experience. 

 

 # An anatomical joke:

 # - Training

 # - Skeletal imitation

 # - Sensory fleshing-out

 # - Classification