# HG changeset patch # User Robert McIntyre # Date 1399052447 14400 # Node ID d304b2ea7c585580d27dadf70c5d2857bdc31a56 # Parent b1d8d9b4b569a2b7ddfc4e13ae6e8c7a3bee08da some changes from winston. diff -r b1d8d9b4b569 -r d304b2ea7c58 thesis/abstract.org --- a/thesis/abstract.org Fri May 02 03:39:19 2014 -0400 +++ b/thesis/abstract.org Fri May 02 13:40:47 2014 -0400 @@ -9,10 +9,10 @@ model of empathy (=EMPATH=) which allows me to recognize actions using simple, embodied representations of actions (which require rich sensory data), even when that sensory data is not actually available. -The missing sense data is ``imagined'' by the system by combining -previous experiences gained from unsupervised free play. The worm is a -five-segment creature equipped with touch, proprioception, and muscle -tension senses. It recognizes actions using only proprioception data. +The missing sense data is imagined by combining previous experiences +gained from unsupervised free play. The worm is a five-segment +creature equipped with touch, proprioception, and muscle tension +senses. It recognizes actions using only proprioception data. In order to build this empathic, action-recognizing system, I created a program called =CORTEX=, which is a complete platform for embodied diff -r b1d8d9b4b569 -r d304b2ea7c58 thesis/cortex.org --- a/thesis/cortex.org Fri May 02 03:39:19 2014 -0400 +++ b/thesis/cortex.org Fri May 02 13:40:47 2014 -0400 @@ -43,15 +43,15 @@ * Empathy \& Embodiment: problem solving strategies - By the end of this thesis, you will have a novel approach to - representing an recognizing physical actions using embodiment and - empathy. You will also see one way to efficiently implement physical - empathy for embodied creatures. Finally, you will become familiar - with =CORTEX=, a system for designing and simulating creatures with - rich senses, which I have designed as a library that you can use in - your own research. Note that I /do not/ process video directly --- I - start with knowledge of the positions of a creature's body parts and - works from there. + By the time you have read this thesis, you will understand a novel + approach to representing and recognizing physical actions using + embodiment and empathy. You will also see one way to efficiently + implement physical empathy for embodied creatures. Finally, you will + become familiar with =CORTEX=, a system for designing and simulating + creatures with rich senses, which I have designed as a library that + you can use in your own research. Note that I /do not/ process video + directly --- I start with knowledge of the positions of a creature's + body parts and work from there. This is the core vision of my thesis: That one of the important ways in which we understand others is by imagining ourselves in their @@ -65,12 +65,13 @@ ** The problem: recognizing actions is hard! - Examine the following image. What is happening? As you, and indeed - very young children, can easily determine, this is an image of - drinking. + Examine figure \ref{cat-drink}. What is happening? As you, and + indeed very young children, can easily determine, this is an image + of drinking. #+caption: A cat drinking some water. Identifying this action is #+caption: beyond the capabilities of existing computer vision systems. + #+name: cat-drink #+ATTR_LaTeX: :width 7cm [[./images/cat-drinking.jpg]] @@ -94,11 +95,13 @@ [[./images/fat-person-sitting-at-desk.jpg]] Finally, how is it that you can easily tell the difference between - how the girls /muscles/ are working in figure \ref{girl}? + how the girl's /muscles/ are working in figure \ref{girl}? #+caption: The mysterious ``common sense'' appears here as you are able #+caption: to discern the difference in how the girl's arm muscles - #+caption: are activated between the two images. + #+caption: are activated between the two images. When you compare + #+caption: these two images, do you feel something in your own arm + #+caption: muscles? #+name: girl #+ATTR_LaTeX: :width 7cm [[./images/wall-push.png]] @@ -138,7 +141,7 @@ image, but is found clearly in a simulation / recollection inspired by those pixels. An imaginative system, having been trained on drinking and non-drinking examples and learning that the most - important component of drinking is the feeling of water sliding + important component of drinking is the feeling of water flowing down one's throat, would analyze a video of a cat drinking in the following manner: @@ -146,7 +149,7 @@ model of its own body in place of the cat. Possibly also create a simulation of the stream of water. - 2. ``Play out'' this simulated scene and generate imagined sensory + 2. Play out this simulated scene and generate imagined sensory experience. This will include relevant muscle contractions, a close up view of the stream from the cat's perspective, and most importantly, the imagined feeling of water entering the mouth. @@ -233,19 +236,19 @@ Exploring these ideas further demands a concrete implementation, so first, I built a system for constructing virtual creatures with physiologically plausible sensorimotor systems and detailed - environments. The result is =CORTEX=, which is described in section + environments. The result is =CORTEX=, which I describe in chapter \ref{sec-2}. Next, I wrote routines which enabled a simple worm-like creature to infer the actions of a second worm-like creature, using only its own prior sensorimotor experiences and knowledge of the second worm's joint positions. This program, =EMPATH=, is described in - section \ref{sec-3}. It's main components are: + chapter \ref{sec-3}. It's main components are: - Embodied Action Definitions :: Many otherwise complicated actions are easily described in the language of a full suite of body-centered, rich senses and experiences. For example, - drinking is the feeling of water sliding down your throat, and + drinking is the feeling of water flowing down your throat, and cooling your insides. It's often accompanied by bringing your hand close to your face, or bringing your face close to water. Sitting down is the feeling of bending your knees, activating @@ -316,10 +319,10 @@ that such representations are very powerful, and often indispensable for the types of recognition tasks considered here. - - Although for expediency's sake, I relied on direct knowledge of - joint positions in this proof of concept, it would be - straightforward to extend =EMPATH= so that it (more - realistically) infers joint positions from its visual data. + - For expediency's sake, I relied on direct knowledge of joint + positions in this proof of concept. However, I believe that the + structure of =EMPATH= and =CORTEX= will make future work to + enable video analysis much easier than it would otherwise be. ** =EMPATH= is built on =CORTEX=, a creature builder. @@ -343,19 +346,19 @@ for three reasons: - You can design new creatures using Blender (\cite{blender}), a - popular 3D modeling program. Each sense can be specified using - special blender nodes with biologically inspired parameters. You - need not write any code to create a creature, and can use a wide - library of pre-existing blender models as a base for your own - creatures. + popular, free 3D modeling program. Each sense can be specified + using special blender nodes with biologically inspired + parameters. You need not write any code to create a creature, and + can use a wide library of pre-existing blender models as a base + for your own creatures. - =CORTEX= implements a wide variety of senses: touch, proprioception, vision, hearing, and muscle tension. Complicated senses like touch and vision involve multiple sensory elements embedded in a 2D surface. You have complete control over the distribution of these sensor elements through the use of simple - png image files. =CORTEX= implements more comprehensive hearing - than any other creature simulation system available. + image files. =CORTEX= implements more comprehensive hearing than + any other creature simulation system available. - =CORTEX= supports any number of creatures and any number of senses. Time in =CORTEX= dilates so that the simulated creatures @@ -365,8 +368,8 @@ =CORTEX= is built on top of =jMonkeyEngine3= (\cite{jmonkeyengine}), which is a video game engine designed to create cross-platform 3D desktop games. =CORTEX= is mainly written - in clojure, a dialect of =LISP= that runs on the java virtual - machine (JVM). The API for creating and simulating creatures and + in clojure, a dialect of =LISP= that runs on the Java Virtual + Machine (JVM). The API for creating and simulating creatures and senses is entirely expressed in clojure, though many senses are implemented at the layer of jMonkeyEngine or below. For example, for the sense of hearing I use a layer of clojure code on top of a @@ -396,8 +399,8 @@ - imagination using subworlds During one test with =CORTEX=, I created 3,000 creatures each with - their own independent senses and ran them all at only 1/80 real - time. In another test, I created a detailed model of my own hand, + its own independent senses and ran them all at only 1/80 real time. + In another test, I created a detailed model of my own hand, equipped with a realistic distribution of touch (more sensitive at the fingertips), as well as eyes and ears, and it ran at around 1/4 real time. @@ -416,9 +419,9 @@ \end{sidewaysfigure} #+END_LaTeX -* Designing =CORTEX= - - In this section, I outline the design decisions that went into +* COMMENT Designing =CORTEX= + + In this chapter, I outline the design decisions that went into making =CORTEX=, along with some details about its implementation. (A practical guide to getting started with =CORTEX=, which skips over the history and implementation details presented here, is @@ -1317,8 +1320,8 @@ ** ...but hearing must be built from scratch - At the end of this section I will have simulated ears that work the - same way as the simulated eyes in the last section. I will be able to + At the end of this chapter I will have simulated ears that work the + same way as the simulated eyes in the last chapter. I will be able to place any number of ear-nodes in a blender file, and they will bind to the closest physical object and follow it as it moves around. Each ear will provide access to the sound data it picks up between every frame. @@ -1332,7 +1335,7 @@ =CORTEX='s hearing is unique because it does not have any limitations compared to other simulation environments. As far as I know, there is no other system that supports multiple listeners, - and the sound demo at the end of this section is the first time + and the sound demo at the end of this chapter is the first time it's been done in a video game environment. *** Brief Description of jMonkeyEngine's Sound System @@ -2146,7 +2149,7 @@ joint from the rest position defined in the blender file. This simulates the muscle-spindles and joint capsules. I will deal with Golgi tendon organs, which calculate muscle strain, in the next - section. + chapter. *** Helper functions @@ -2392,7 +2395,7 @@ also a sense function: it returns the percent of the total muscle strength that is currently being employed. This is analogous to muscle tension in humans and completes the sense of proprioception - begun in the last section. + begun in the last chapter. ** =CORTEX= brings complex creatures to life! @@ -2499,7 +2502,7 @@ \newpage -* =EMPATH=: action recognition in a simulated worm +* COMMENT =EMPATH=: action recognition in a simulated worm Here I develop a computational model of empathy, using =CORTEX= as a base. Empathy in this context is the ability to observe another @@ -3220,7 +3223,7 @@ ** Digression: Learning touch sensor layout through free play - In the previous section I showed how to compute actions in terms of + In the previous chapter I showed how to compute actions in terms of body-centered predicates, but some of those predicates relied on the average touch activation of pre-defined regions of the worm's skin. What if, instead of receiving touch pre-grouped into the six @@ -3233,7 +3236,7 @@ of skin that are close together in the animal end up far apart in the nerve bundle. - In this section I show how to automatically learn the skin-topology of + In this chapter I show how to automatically learn the skin-topology of a worm segment by free exploration. As the worm rolls around on the floor, large sections of its surface get activated. If the worm has stopped moving, then whatever region of skin that is touching the @@ -3484,7 +3487,7 @@ \clearpage #+END_LaTeX -* Contributions +* COMMENT Contributions The big idea behind this thesis is a new way to represent and recognize physical actions, which I call /empathic representation/. @@ -3544,7 +3547,7 @@ \appendix #+END_LaTeX -* Appendix: =CORTEX= User Guide +* COMMENT Appendix: =CORTEX= User Guide Those who write a thesis should endeavor to make their code not only accessible, but actually usable, as a way to pay back the community