1.1 --- a/thesis/abstract.org	Fri May 02 03:39:19 2014 -0400
     1.2 +++ b/thesis/abstract.org	Fri May 02 13:40:47 2014 -0400
     1.3 @@ -9,10 +9,10 @@
     1.4  model of empathy (=EMPATH=) which allows me to recognize actions using
     1.5  simple, embodied representations of actions (which require rich
     1.6  sensory data), even when that sensory data is not actually available.
     1.7 -The missing sense data is ``imagined'' by the system by combining
     1.8 -previous experiences gained from unsupervised free play. The worm is a
     1.9 -five-segment creature equipped with touch, proprioception, and muscle
    1.10 -tension senses. It recognizes actions using only proprioception data.
    1.11 +The missing sense data is imagined by combining previous experiences
    1.12 +gained from unsupervised free play. The worm is a five-segment
    1.13 +creature equipped with touch, proprioception, and muscle tension
    1.14 +senses. It recognizes actions using only proprioception data.
    1.15  
    1.16  In order to build this empathic, action-recognizing system, I created
    1.17  a program called =CORTEX=, which is a complete platform for embodied

     2.1 --- a/thesis/cortex.org	Fri May 02 03:39:19 2014 -0400
     2.2 +++ b/thesis/cortex.org	Fri May 02 13:40:47 2014 -0400
     2.3 @@ -43,15 +43,15 @@
     2.4  
     2.5  * Empathy \& Embodiment: problem solving strategies
     2.6  
     2.7 -  By the end of this thesis, you will have a novel approach to
     2.8 -  representing an recognizing physical actions using embodiment and
     2.9 -  empathy. You will also see one way to efficiently implement physical
    2.10 -  empathy for embodied creatures. Finally, you will become familiar
    2.11 -  with =CORTEX=, a system for designing and simulating creatures with
    2.12 -  rich senses, which I have designed as a library that you can use in
    2.13 -  your own research. Note that I /do not/ process video directly --- I
    2.14 -  start with knowledge of the positions of a creature's body parts and
    2.15 -  works from there.
    2.16 +  By the time you have read this thesis, you will understand a novel
    2.17 +  approach to representing and recognizing physical actions using
    2.18 +  embodiment and empathy. You will also see one way to efficiently
    2.19 +  implement physical empathy for embodied creatures. Finally, you will
    2.20 +  become familiar with =CORTEX=, a system for designing and simulating
    2.21 +  creatures with rich senses, which I have designed as a library that
    2.22 +  you can use in your own research. Note that I /do not/ process video
    2.23 +  directly --- I start with knowledge of the positions of a creature's
    2.24 +  body parts and work from there.
    2.25    
    2.26    This is the core vision of my thesis: That one of the important ways
    2.27    in which we understand others is by imagining ourselves in their
    2.28 @@ -65,12 +65,13 @@
    2.29    
    2.30  ** The problem: recognizing actions is hard!
    2.31     
    2.32 -   Examine the following image. What is happening? As you, and indeed
    2.33 -   very young children, can easily determine, this is an image of
    2.34 -   drinking. 
    2.35 +   Examine figure \ref{cat-drink}. What is happening? As you, and
    2.36 +   indeed very young children, can easily determine, this is an image
    2.37 +   of drinking.
    2.38  
    2.39     #+caption: A cat drinking some water. Identifying this action is 
    2.40     #+caption: beyond the capabilities of existing computer vision systems.
    2.41 +   #+name: cat-drink
    2.42     #+ATTR_LaTeX: :width 7cm
    2.43     [[./images/cat-drinking.jpg]]
    2.44       
    2.45 @@ -94,11 +95,13 @@
    2.46     [[./images/fat-person-sitting-at-desk.jpg]]
    2.47  
    2.48     Finally, how is it that you can easily tell the difference between
    2.49 -   how the girls /muscles/ are working in figure \ref{girl}?
    2.50 +   how the girl's /muscles/ are working in figure \ref{girl}?
    2.51     
    2.52     #+caption: The mysterious ``common sense'' appears here as you are able 
    2.53     #+caption: to discern the difference in how the girl's arm muscles
    2.54 -   #+caption: are activated between the two images.
    2.55 +   #+caption: are activated between the two images. When you compare
    2.56 +   #+caption: these two images, do you feel something in your own arm
    2.57 +   #+caption: muscles?
    2.58     #+name: girl
    2.59     #+ATTR_LaTeX: :width 7cm
    2.60     [[./images/wall-push.png]]
    2.61 @@ -138,7 +141,7 @@
    2.62     image, but is found clearly in a simulation / recollection inspired
    2.63     by those pixels. An imaginative system, having been trained on
    2.64     drinking and non-drinking examples and learning that the most
    2.65 -   important component of drinking is the feeling of water sliding
    2.66 +   important component of drinking is the feeling of water flowing
    2.67     down one's throat, would analyze a video of a cat drinking in the
    2.68     following manner:
    2.69     
    2.70 @@ -146,7 +149,7 @@
    2.71        model of its own body in place of the cat. Possibly also create
    2.72        a simulation of the stream of water.
    2.73  
    2.74 -   2. ``Play out'' this simulated scene and generate imagined sensory
    2.75 +   2. Play out this simulated scene and generate imagined sensory
    2.76        experience. This will include relevant muscle contractions, a
    2.77        close up view of the stream from the cat's perspective, and most
    2.78        importantly, the imagined feeling of water entering the mouth.
    2.79 @@ -233,19 +236,19 @@
    2.80     Exploring these ideas further demands a concrete implementation, so
    2.81     first, I built a system for constructing virtual creatures with
    2.82     physiologically plausible sensorimotor systems and detailed
    2.83 -   environments. The result is =CORTEX=, which is described in section
    2.84 +   environments. The result is =CORTEX=, which I describe in chapter
    2.85     \ref{sec-2}.
    2.86  
    2.87     Next, I wrote routines which enabled a simple worm-like creature to
    2.88     infer the actions of a second worm-like creature, using only its
    2.89     own prior sensorimotor experiences and knowledge of the second
    2.90     worm's joint positions. This program, =EMPATH=, is described in
    2.91 -   section \ref{sec-3}. It's main components are:
    2.92 +   chapter \ref{sec-3}. It's main components are:
    2.93  
    2.94     - Embodied Action Definitions :: Many otherwise complicated actions
    2.95          are easily described in the language of a full suite of
    2.96          body-centered, rich senses and experiences. For example,
    2.97 -        drinking is the feeling of water sliding down your throat, and
    2.98 +        drinking is the feeling of water flowing down your throat, and
    2.99          cooling your insides. It's often accompanied by bringing your
   2.100          hand close to your face, or bringing your face close to water.
   2.101          Sitting down is the feeling of bending your knees, activating
   2.102 @@ -316,10 +319,10 @@
   2.103       that such representations are very powerful, and often
   2.104       indispensable for the types of recognition tasks considered here.
   2.105  
   2.106 -   - Although for expediency's sake, I relied on direct knowledge of
   2.107 -     joint positions in this proof of concept, it would be
   2.108 -     straightforward to extend =EMPATH= so that it (more
   2.109 -     realistically) infers joint positions from its visual data.
   2.110 +   - For expediency's sake, I relied on direct knowledge of joint
   2.111 +     positions in this proof of concept. However, I believe that the
   2.112 +     structure of =EMPATH= and =CORTEX= will make future work to
   2.113 +     enable video analysis much easier than it would otherwise be.
   2.114  
   2.115  ** =EMPATH= is built on =CORTEX=, a creature builder.
   2.116  
   2.117 @@ -343,19 +346,19 @@
   2.118     for three reasons:
   2.119  
   2.120     - You can design new creatures using Blender (\cite{blender}), a
   2.121 -     popular 3D modeling program. Each sense can be specified using
   2.122 -     special blender nodes with biologically inspired parameters. You
   2.123 -     need not write any code to create a creature, and can use a wide
   2.124 -     library of pre-existing blender models as a base for your own
   2.125 -     creatures.
   2.126 +     popular, free 3D modeling program. Each sense can be specified
   2.127 +     using special blender nodes with biologically inspired
   2.128 +     parameters. You need not write any code to create a creature, and
   2.129 +     can use a wide library of pre-existing blender models as a base
   2.130 +     for your own creatures.
   2.131  
   2.132     - =CORTEX= implements a wide variety of senses: touch,
   2.133       proprioception, vision, hearing, and muscle tension. Complicated
   2.134       senses like touch and vision involve multiple sensory elements
   2.135       embedded in a 2D surface. You have complete control over the
   2.136       distribution of these sensor elements through the use of simple
   2.137 -     png image files. =CORTEX= implements more comprehensive hearing
   2.138 -     than any other creature simulation system available.
   2.139 +     image files. =CORTEX= implements more comprehensive hearing than
   2.140 +     any other creature simulation system available.
   2.141  
   2.142     - =CORTEX= supports any number of creatures and any number of
   2.143       senses. Time in =CORTEX= dilates so that the simulated creatures
   2.144 @@ -365,8 +368,8 @@
   2.145     =CORTEX= is built on top of =jMonkeyEngine3=
   2.146     (\cite{jmonkeyengine}), which is a video game engine designed to
   2.147     create cross-platform 3D desktop games. =CORTEX= is mainly written
   2.148 -   in clojure, a dialect of =LISP= that runs on the java virtual
   2.149 -   machine (JVM). The API for creating and simulating creatures and
   2.150 +   in clojure, a dialect of =LISP= that runs on the Java Virtual
   2.151 +   Machine (JVM). The API for creating and simulating creatures and
   2.152     senses is entirely expressed in clojure, though many senses are
   2.153     implemented at the layer of jMonkeyEngine or below. For example,
   2.154     for the sense of hearing I use a layer of clojure code on top of a
   2.155 @@ -396,8 +399,8 @@
   2.156     - imagination using subworlds
   2.157  
   2.158     During one test with =CORTEX=, I created 3,000 creatures each with
   2.159 -   their own independent senses and ran them all at only 1/80 real
   2.160 -   time. In another test, I created a detailed model of my own hand,
   2.161 +   its own independent senses and ran them all at only 1/80 real time.
   2.162 +   In another test, I created a detailed model of my own hand,
   2.163     equipped with a realistic distribution of touch (more sensitive at
   2.164     the fingertips), as well as eyes and ears, and it ran at around 1/4
   2.165     real time.
   2.166 @@ -416,9 +419,9 @@
   2.167     \end{sidewaysfigure}
   2.168  #+END_LaTeX
   2.169  
   2.170 -* Designing =CORTEX=
   2.171 -
   2.172 -  In this section, I outline the design decisions that went into
   2.173 +* COMMENT Designing =CORTEX=
   2.174 +
   2.175 +  In this chapter, I outline the design decisions that went into
   2.176    making =CORTEX=, along with some details about its implementation.
   2.177    (A practical guide to getting started with =CORTEX=, which skips
   2.178    over the history and implementation details presented here, is
   2.179 @@ -1317,8 +1320,8 @@
   2.180  
   2.181  ** ...but hearing must be built from scratch
   2.182  
   2.183 -   At the end of this section I will have simulated ears that work the
   2.184 -   same way as the simulated eyes in the last section. I will be able to
   2.185 +   At the end of this chapter I will have simulated ears that work the
   2.186 +   same way as the simulated eyes in the last chapter. I will be able to
   2.187     place any number of ear-nodes in a blender file, and they will bind to
   2.188     the closest physical object and follow it as it moves around. Each ear
   2.189     will provide access to the sound data it picks up between every frame.
   2.190 @@ -1332,7 +1335,7 @@
   2.191     =CORTEX='s hearing is unique because it does not have any
   2.192     limitations compared to other simulation environments. As far as I
   2.193     know, there is no other system that supports multiple listeners,
   2.194 -   and the sound demo at the end of this section is the first time
   2.195 +   and the sound demo at the end of this chapter is the first time
   2.196     it's been done in a video game environment.
   2.197  
   2.198  *** Brief Description of jMonkeyEngine's Sound System
   2.199 @@ -2146,7 +2149,7 @@
   2.200     joint from the rest position defined in the blender file. This
   2.201     simulates the muscle-spindles and joint capsules. I will deal with
   2.202     Golgi tendon organs, which calculate muscle strain, in the next
   2.203 -   section.
   2.204 +   chapter.
   2.205  
   2.206  *** Helper functions
   2.207  
   2.208 @@ -2392,7 +2395,7 @@
   2.209      also a sense function: it returns the percent of the total muscle
   2.210      strength that is currently being employed. This is analogous to
   2.211      muscle tension in humans and completes the sense of proprioception
   2.212 -    begun in the last section.
   2.213 +    begun in the last chapter.
   2.214      
   2.215  ** =CORTEX= brings complex creatures to life!
   2.216     
   2.217 @@ -2499,7 +2502,7 @@
   2.218  
   2.219  \newpage
   2.220  
   2.221 -* =EMPATH=: action recognition in a simulated worm
   2.222 +* COMMENT =EMPATH=: action recognition in a simulated worm
   2.223  
   2.224    Here I develop a computational model of empathy, using =CORTEX= as a
   2.225    base. Empathy in this context is the ability to observe another
   2.226 @@ -3220,7 +3223,7 @@
   2.227  
   2.228  ** Digression: Learning touch sensor layout through free play
   2.229  
   2.230 -   In the previous section I showed how to compute actions in terms of
   2.231 +   In the previous chapter I showed how to compute actions in terms of
   2.232     body-centered predicates, but some of those predicates relied on
   2.233     the average touch activation of pre-defined regions of the worm's
   2.234     skin. What if, instead of receiving touch pre-grouped into the six
   2.235 @@ -3233,7 +3236,7 @@
   2.236     of skin that are close together in the animal end up far apart in
   2.237     the nerve bundle.
   2.238     
   2.239 -   In this section I show how to automatically learn the skin-topology of
   2.240 +   In this chapter I show how to automatically learn the skin-topology of
   2.241     a worm segment by free exploration. As the worm rolls around on the
   2.242     floor, large sections of its surface get activated. If the worm has
   2.243     stopped moving, then whatever region of skin that is touching the
   2.244 @@ -3484,7 +3487,7 @@
   2.245  \clearpage
   2.246  #+END_LaTeX
   2.247  
   2.248 -* Contributions
   2.249 +* COMMENT Contributions
   2.250  
   2.251    The big idea behind this thesis is a new way to represent and
   2.252    recognize physical actions, which I call /empathic representation/.
   2.253 @@ -3544,7 +3547,7 @@
   2.254  \appendix
   2.255  #+END_LaTeX
   2.256  
   2.257 -* Appendix: =CORTEX= User Guide
   2.258 +* COMMENT Appendix: =CORTEX= User Guide
   2.259  
   2.260    Those who write a thesis should endeavor to make their code not only
   2.261    accessible, but actually usable, as a way to pay back the community