# HG changeset patch # User Robert McIntyre # Date 1398060771 14400 # Node ID fd74479db5cb90bc5dda1d36cedfeed557662752 # Parent ac747fa0a67844fe9c93d5c66b2ad530ef2a49d4# Parent 25f23cfd56ceae2e15108f8621aeb252a24221ca merge some winston updates. diff -r ac747fa0a678 -r fd74479db5cb images/finger-1.png Binary file images/finger-1.png has changed diff -r ac747fa0a678 -r fd74479db5cb thesis/abstract.org --- a/thesis/abstract.org Mon Apr 21 02:11:59 2014 -0400 +++ b/thesis/abstract.org Mon Apr 21 02:12:51 2014 -0400 @@ -3,20 +3,22 @@ recognizing actions performed by a creature given limited data about the creature's actions, such as a video recording. I solve this problem in the case of a worm-like creature performing actions such as -curling and wiggling. +curling and wiggling. To attack the action recognition problem, I developed a computational model of empathy (=EMPATH=) which allows me to recognize actions using simple, embodied representations of actions (which require rich -sensory data), even when that sensory data is not actually -available. The missing sense data is ``imagined'' by the system by -combining previous experiences gained from unsupervised free play. +sensory data), even when that sensory data is not actually available. +The missing sense data is ``imagined'' by the system by combining +previous experiences gained from unsupervised free play. The worm is a +five-segment creature equipped with touch, proprioception, and muscle +tension senses. It recognizes actions using only proprioception data. In order to build this empathic, action-recognizing system, I created a program called =CORTEX=, which is a complete platform for embodied AI research. It provides multiple senses for simulated creatures, -including vision, touch, proprioception, muscle tension, and -hearing. Each of these senses provides a wealth of parameters that are +including vision, touch, proprioception, muscle tension, and hearing. +Each of these senses provides a wealth of parameters that are biologically inspired. =CORTEX= is able to simulate any number of creatures and senses, and provides facilities for easily modeling and creating new creatures. As a research platform it is more complete diff -r ac747fa0a678 -r fd74479db5cb thesis/cortex.org --- a/thesis/cortex.org Mon Apr 21 02:11:59 2014 -0400 +++ b/thesis/cortex.org Mon Apr 21 02:12:51 2014 -0400 @@ -44,11 +44,14 @@ * Empathy \& Embodiment: problem solving strategies By the end of this thesis, you will have seen a novel approach to - interpreting video using embodiment and empathy. You will have also - seen one way to efficiently implement empathy for embodied + interpreting video using embodiment and empathy. You will also see + one way to efficiently implement physical empathy for embodied creatures. Finally, you will become familiar with =CORTEX=, a system - for designing and simulating creatures with rich senses, which you - may choose to use in your own research. + for designing and simulating creatures with rich senses, which I + have designed as a library that you can use in your own research. + Note that I /do not/ process video directly --- I start with + knowledge of the positions of a creature's body parts and works from + there. This is the core vision of my thesis: That one of the important ways in which we understand others is by imagining ourselves in their @@ -60,7 +63,7 @@ is happening in a video and being completely lost in a sea of incomprehensible color and movement. -** The problem: recognizing actions in video is hard! +** The problem: recognizing actions is hard! Examine the following image. What is happening? As you, and indeed very young children, can easily determine, this is an image of @@ -84,8 +87,8 @@ example, what processes might enable you to see the chair in figure \ref{hidden-chair}? - #+caption: The chair in this image is quite obvious to humans, but I - #+caption: doubt that any modern computer vision program can find it. + #+caption: The chair in this image is quite obvious to humans, but + #+caption: it can't be found by any modern computer vision program. #+name: hidden-chair #+ATTR_LaTeX: :width 10cm [[./images/fat-person-sitting-at-desk.jpg]] @@ -480,7 +483,7 @@ real world instead of a simulation is the matter of time. Instead of simulated time you get the constant and unstoppable flow of real time. This severely limits the sorts of software you can use - to program the AI because all sense inputs must be handled in real + to program an AI, because all sense inputs must be handled in real time. Complicated ideas may have to be implemented in hardware or may simply be impossible given the current speed of our processors. Contrast this with a simulation, in which the flow of @@ -550,10 +553,10 @@ of the retina. In each case, we can describe the sense with a surface and a distribution of sensors along that surface. - The neat idea is that every human sense can be effectively - described in terms of a surface containing embedded sensors. If the - sense had any more dimensions, then there wouldn't be enough room - in the spinal chord to transmit the information! + In fact, almost every human sense can be effectively described in + terms of a surface containing embedded sensors. If the sense had + any more dimensions, then there wouldn't be enough room in the + spinal chord to transmit the information! Therefore, =CORTEX= must support the ability to create objects and then be able to ``paint'' points along their surfaces to describe @@ -2378,14 +2381,14 @@ #+end_listing - =movement-kernel= creates a function that will move the nearest - physical object to the muscle node. The muscle exerts a rotational - force dependent on it's orientation to the object in the blender - file. The function returned by =movement-kernel= is also a sense - function: it returns the percent of the total muscle strength that - is currently being employed. This is analogous to muscle tension - in humans and completes the sense of proprioception begun in the - last section. + =movement-kernel= creates a function that controlls the movement + of the nearest physical node to the muscle node. The muscle exerts + a rotational force dependent on it's orientation to the object in + the blender file. The function returned by =movement-kernel= is + also a sense function: it returns the percent of the total muscle + strength that is currently being employed. This is analogous to + muscle tension in humans and completes the sense of proprioception + begun in the last section. ** =CORTEX= brings complex creatures to life! @@ -2491,6 +2494,8 @@ hard control problems without worrying about physics or senses. +\newpage + * =EMPATH=: action recognition in a simulated worm Here I develop a computational model of empathy, using =CORTEX= as a @@ -2502,8 +2507,8 @@ creature's sensory state space, which I call \Phi-space. Using \Phi-space, I construct an efficient function which takes the limited data that comes from observing another creature and enriches - it full compliment of imagined sensory data. I can then use the - imagined sensory data to recognize what the observed creature is + it with a full compliment of imagined sensory data. I can then use + the imagined sensory data to recognize what the observed creature is doing and feeling, using straightforward embodied action predicates. This is all demonstrated with using a simple worm-like creature, and recognizing worm-actions based on limited data. @@ -2555,9 +2560,9 @@ Embodied representations using multiple senses such as touch, proprioception, and muscle tension turns out be be exceedingly - efficient at describing body-centered actions. It is the ``right - language for the job''. For example, it takes only around 5 lines - of LISP code to describe the action of ``curling'' using embodied + efficient at describing body-centered actions. It is the right + language for the job. For example, it takes only around 5 lines of + LISP code to describe the action of curling using embodied primitives. It takes about 10 lines to describe the seemingly complicated action of wiggling. @@ -2566,14 +2571,16 @@ whether the worm is doing the action they describe. =curled?= relies on proprioception, =resting?= relies on touch, =wiggling?= relies on a Fourier analysis of muscle contraction, and - =grand-circle?= relies on touch and reuses =curled?= as a guard. + =grand-circle?= relies on touch and reuses =curled?= in its + definition, showing how embodied predicates can be composed. #+caption: Program for detecting whether the worm is curled. This is the #+caption: simplest action predicate, because it only uses the last frame #+caption: of sensory experience, and only uses proprioceptive data. Even #+caption: this simple predicate, however, is automatically frame - #+caption: independent and ignores vermopomorphic differences such as - #+caption: worm textures and colors. + #+caption: independent and ignores vermopomorphic \footnote{Like + #+caption: \emph{anthropomorphic}, except for worms instead of humans.} + #+caption: differences such as worm textures and colors. #+name: curled #+begin_listing clojure #+begin_src clojure @@ -2735,7 +2742,7 @@ The trick now is to make the action predicates work even when the sensory data on which they depend is absent. If I can do that, then - I will have gained much, + I will have gained much. ** \Phi-space describes the worm's experiences diff -r ac747fa0a678 -r fd74479db5cb thesis/cover.tex --- a/thesis/cover.tex Mon Apr 21 02:11:59 2014 -0400 +++ b/thesis/cover.tex Mon Apr 21 02:12:51 2014 -0400 @@ -45,7 +45,7 @@ % however the specifications can change. We recommend that you verify the % layout of your title page with your thesis advisor and/or the MIT % Libraries before printing your final copy. -\title{Solving Problems using Embodiment \& Empathy} +\title{Recognizing Actions using Embodiment \& Empathy} \author{Robert Louis M\raisebox{\depth}{\small \underline{\underline{c}}}Intyre} %\author{Robert McIntyre} diff -r ac747fa0a678 -r fd74479db5cb thesis/images/empty-sense-nodes.png Binary file thesis/images/empty-sense-nodes.png has changed diff -r ac747fa0a678 -r fd74479db5cb thesis/images/finger-UV.png Binary file thesis/images/finger-UV.png has changed diff -r ac747fa0a678 -r fd74479db5cb thesis/images/hand-screenshot1.png Binary file thesis/images/hand-screenshot1.png has changed diff -r ac747fa0a678 -r fd74479db5cb thesis/rlm-cortex-meng.tex --- a/thesis/rlm-cortex-meng.tex Mon Apr 21 02:11:59 2014 -0400 +++ b/thesis/rlm-cortex-meng.tex Mon Apr 21 02:12:51 2014 -0400 @@ -62,6 +62,9 @@ % Configure minted source code listings. \usemintedstyle{default} \newminted{clojure}{fontsize=\footnotesize} +\newminted{java}{fontsize=\footnotesize} +\newminted{c}{fontsize=\footnotesize} + % Allow colored source code listing to break across pages. \newenvironment{anchoredListing}{\captionsetup{type=listing}}{}