cortex: thesis/cortex.org comparison

comparison thesis/cortex.org @ 516:ced955c3c84f

resurrect old cortex to fix flow issues.

author	Robert McIntyre <rlm@mit.edu>
date	Sun, 30 Mar 2014 22:48:19 -0400
parents	58fa1ffd481e
children	68665d2c32a7

comparison

equal deleted inserted replaced

-:58fa1ffd481e
+:ced955c3c84f
 #+ATTR_LaTeX: :width 10cm
 [[./images/aurellem-gray.png]]
 * Empathy \& Embodiment: problem solving strategies
+By the end of this thesis, you will have seen a novel approach to
+interpreting video using embodiment and empathy. You will have also
+seen one way to efficiently implement empathy for embodied
+creatures. Finally, you will become familiar with =CORTEX=, a system
+for designing and simulating creatures with rich senses, which you
+may choose to use in your own research.
-** The problem: recognizing actions in video is extremely difficult
+This is the core vision of my thesis: That one of the important ways
-# developing / requires useful representations
+in which we understand others is by imagining ourselves in their
+position and emphatically feeling experiences relative to our own
-Examine the following collection of images. As you, and indeed very
+bodies. By understanding events in terms of our own previous
-young children, can easily determine, each one is a picture of
+corporeal experience, we greatly constrain the possibilities of what
-someone drinking.
+would otherwise be an unwieldy exponential search. This extra
+constraint can be the difference between easily understanding what
-# dxh: cat, cup, drinking fountain, rain, straw, coconut
+is happening in a video and being completely lost in a sea of
+incomprehensible color and movement.
+** The problem: recognizing actions in video is hard!
+Examine the following image. What is happening? As you, and indeed
+very young children, can easily determine, this is an image of
+drinking.
 #+caption: A cat drinking some water. Identifying this action is
 #+caption: beyond the capabilities of existing computer vision systems.
 #+ATTR_LaTeX: :width 7cm
 [[./images/cat-drinking.jpg]]
 Nevertheless, it is beyond the state of the art for a computer
-vision program to describe what's happening in each of these
+vision program to describe what's happening in this image. Part of
-images, or what's common to them. Part of the problem is that many
+the problem is that many computer vision systems focus on
-computer vision systems focus on pixel-level details or probability
+pixel-level details or comparisons to example images (such as
-distributions of pixels, with little focus on [...]
+\cite{volume-action-recognition}), but the 3D world is so variable
+that it is hard to descrive the world in terms of possible images.
 In fact, the contents of scene may have much less to do with pixel
 probabilities than with recognizing various affordances: things you
-can move, objects you can grasp, spaces that can be filled
+can move, objects you can grasp, spaces that can be filled . For
-(Gibson). For example, what processes might enable you to see the
+example, what processes might enable you to see the chair in figure
-chair in figure \ref{hidden-chair}?
+\ref{hidden-chair}?
-# Or suppose that you are building a program that recognizes chairs.
-# How could you ``see'' the chair ?
-# dxh: blur chair
 #+caption: The chair in this image is quite obvious to humans, but I
 #+caption: doubt that any modern computer vision program can find it.
 #+name: hidden-chair
 #+ATTR_LaTeX: :width 10cm
 [[./images/fat-person-sitting-at-desk.jpg]]
 Finally, how is it that you can easily tell the difference between
 how the girls /muscles/ are working in figure \ref{girl}?
 #+caption: The mysterious ``common sense'' appears here as you are able
 #+caption: to discern the difference in how the girl's arm muscles
 #+caption: are activated between the two images.
 #+name: girl
 #+ATTR_LaTeX: :width 7cm
 [[./images/wall-push.png]]
 Each of these examples tells us something about what might be going
 on in our minds as we easily solve these recognition problems.
 The hidden chair shows us that we are strongly triggered by cues
 relating to the position of human bodies, and that we can determine
 We know well how our muscles would have to work to maintain us in
 most positions, and we can easily project this self-knowledge to
 imagined positions triggered by images of the human body.
 ** A step forward: the sensorimotor-centered approach
-# ** =EMPATH= recognizes what creatures are doing
-# neatly solves recognition problems
 In this thesis, I explore the idea that our knowledge of our own
-bodies enables us to recognize the actions of others.
+bodies, combined with our own rich senses, enables us to recognize
+the actions of others.
+For example, I think humans are able to label the cat video as
+``drinking'' because they imagine /themselves/ as the cat, and
+imagine putting their face up against a stream of water and
+sticking out their tongue. In that imagined world, they can feel
+the cool water hitting their tongue, and feel the water entering
+their body, and are able to recognize that /feeling/ as drinking.
+So, the label of the action is not really in the pixels of the
+image, but is found clearly in a simulation inspired by those
+pixels. An imaginative system, having been trained on drinking and
+non-drinking examples and learning that the most important
+component of drinking is the feeling of water sliding down one's
+throat, would analyze a video of a cat drinking in the following
+manner:
+1. Create a physical model of the video by putting a ``fuzzy''
+model of its own body in place of the cat. Possibly also create
+a simulation of the stream of water.
+2. Play out this simulated scene and generate imagined sensory
+experience. This will include relevant muscle contractions, a
+close up view of the stream from the cat's perspective, and most
+importantly, the imagined feeling of water entering the
+mouth. The imagined sensory experience can come from a
+simulation of the event, but can also be pattern-matched from
+previous, similar embodied experience.
+3. The action is now easily identified as drinking by the sense of
+taste alone. The other senses (such as the tongue moving in and
+out) help to give plausibility to the simulated action. Note that
+the sense of vision, while critical in creating the simulation,
+is not critical for identifying the action from the simulation.
+For the chair examples, the process is even easier:
+1. Align a model of your body to the person in the image.
+2. Generate proprioceptive sensory data from this alignment.
+3. Use the imagined proprioceptive data as a key to lookup related
+sensory experience associated with that particular proproceptive
+feeling.
+4. Retrieve the feeling of your bottom resting on a surface, your
+knees bent, and your leg muscles relaxed.
+5. This sensory information is consistent with your =sitting?=
+sensory predicate, so you (and the entity in the image) must be
+sitting.
+6. There must be a chair-like object since you are sitting.
+Empathy offers yet another alternative to the age-old AI
+representation question: ``What is a chair?'' --- A chair is the
+feeling of sitting!
+One powerful advantage of empathic problem solving is that it
+factors the action recognition problem into two easier problems. To
+use empathy, you need an /aligner/, which takes the video and a
+model of your body, and aligns the model with the video. Then, you
+need a /recognizer/, which uses the aligned model to interpret the
+action. The power in this method lies in the fact that you describe
+all actions form a body-centered viewpoint. You are less tied to
+the particulars of any visual representation of the actions. If you
+teach the system what ``running'' is, and you have a good enough
+aligner, the system will from then on be able to recognize running
+from any point of view, even strange points of view like above or
+underneath the runner. This is in contrast to action recognition
+schemes that try to identify actions using a non-embodied approach.
+If these systems learn about running as viewed from the side, they
+will not automatically be able to recognize running from any other
+viewpoint.
+Another powerful advantage is that using the language of multiple
+body-centered rich senses to describe body-centerd actions offers a
+massive boost in descriptive capability. Consider how difficult it
+would be to compose a set of HOG filters to describe the action of
+a simple worm-creature ``curling'' so that its head touches its
+tail, and then behold the simplicity of describing thus action in a
+language designed for the task (listing \ref{grand-circle-intro}):
+#+caption: Body-centerd actions are best expressed in a body-centered
+#+caption: language. This code detects when the worm has curled into a
+#+caption: full circle. Imagine how you would replicate this functionality
+#+caption: using low-level pixel features such as HOG filters!
+#+name: grand-circle-intro
+#+begin_listing clojure
+#+begin_src clojure
+(defn grand-circle?
+"Does the worm form a majestic circle (one end touching the other)?"
+[experiences]
+(and (curled? experiences)
+(let [worm-touch (:touch (peek experiences))
+tail-touch (worm-touch 0)
+head-touch (worm-touch 4)]
+(and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
+(< 0.2 (contact worm-segment-top-tip    head-touch))))))
+#+end_src
+#+end_listing
+** =EMPATH= regognizes actions using empathy
 First, I built a system for constructing virtual creatures with
 physiologically plausible sensorimotor systems and detailed
 environments. The result is =CORTEX=, which is described in section
 \ref{sec-2}. (=CORTEX= was built to be flexible and useful to other
 infer the actions of a second worm-like creature, using only its
 own prior sensorimotor experiences and knowledge of the second
 worm's joint positions. This program, =EMPATH=, is described in
 section \ref{sec-3}, and the key results of this experiment are
 summarized below.
-#+caption: From only \emph{proprioceptive} data, =EMPATH= was able to infer
-#+caption: the complete sensory experience and classify these four poses.
-#+caption: The last image is a composite, depicting the intermediate stages of \emph{wriggling}.
-#+name: worm-recognition-intro-2
-#+ATTR_LaTeX: :width 15cm
-[[./images/empathy-1.png]]
-# =CORTEX= provides a language for describing the sensorimotor
-# experiences of various creatures.
-# Next, I developed an experiment to test the power of =CORTEX='s
-# sensorimotor-centered language for solving recognition problems. As
-# a proof of concept, I wrote routines which enabled a simple
-# worm-like creature to infer the actions of a second worm-like
-# creature, using only its own previous sensorimotor experiences and
-# knowledge of the second worm's joints (figure
-# \ref{worm-recognition-intro-2}). The result of this proof of
-# concept was the program =EMPATH=, described in section
-# \ref{sec-3}. The key results of this
-# Using only first-person sensorimotor experiences and third-person
-# proprioceptive data,
-*** Key results
-- After one-shot supervised training, =EMPATH= was able recognize a
-wide variety of static poses and dynamic actions---ranging from
-curling in a circle to wriggling with a particular frequency ---
-with 95\% accuracy.
-- These results were completely independent of viewing angle
-because the underlying body-centered language fundamentally is
-independent; once an action is learned, it can be recognized
-equally well from any viewing angle.
-- =EMPATH= is surprisingly short; the sensorimotor-centered
-language provided by =CORTEX= resulted in extremely economical
-recognition routines --- about 0000 lines in all --- suggesting
-that such representations are very powerful, and often
-indispensible for the types of recognition tasks considered here.
-- Although for expediency's sake, I relied on direct knowledge of
-joint positions in this proof of concept, it would be
-straightforward to extend =EMPATH= so that it (more
-realistically) infers joint positions from its visual data.
-# because the underlying language is fundamentally orientation-independent
-# recognize the actions of a worm with 95\% accuracy. The
-#      recognition tasks
-[Talk about these results and what you find promising about them]
-** Roadmap
-[I'm going to explain how =CORTEX= works, then break down how
-=EMPATH= does its thing. Because the details reveal such-and-such
-about the approach.]
-# The success of this simple proof-of-concept offers a tantalizing
-# explore the idea
-# The key contribution of this thesis is the idea that body-centered
-# representations (which express
-# the
-# body-centered approach --- in which I try to determine what's
-# happening in a scene by bringing it into registration with my own
-# bodily experiences --- are indispensible for recognizing what
-# creatures are doing in a scene.
-* COMMENT
-# body-centered language
-In this thesis, I'll describe =EMPATH=, which solves a certain
-class of recognition problems
-The key idea is to use self-centered (or first-person) language.
 I have built a system that can express the types of recognition
 problems in a form amenable to computation. It is split into
 four parts:
 data, just as it would if it were actually experiencing the
 scene first-hand. If previous experience has been accurately
 retrieved, and if it is analogous enough to the scene, then
 the creature will correctly identify the action in the scene.
-For example, I think humans are able to label the cat video as
-``drinking'' because they imagine /themselves/ as the cat, and
-imagine putting their face up against a stream of water and
-sticking out their tongue. In that imagined world, they can feel
-the cool water hitting their tongue, and feel the water entering
-their body, and are able to recognize that /feeling/ as drinking.
-So, the label of the action is not really in the pixels of the
-image, but is found clearly in a simulation inspired by those
-pixels. An imaginative system, having been trained on drinking and
-non-drinking examples and learning that the most important
-component of drinking is the feeling of water sliding down one's
-throat, would analyze a video of a cat drinking in the following
-manner:
-1. Create a physical model of the video by putting a ``fuzzy''
-model of its own body in place of the cat. Possibly also create
-a simulation of the stream of water.
-2. Play out this simulated scene and generate imagined sensory
-experience. This will include relevant muscle contractions, a
-close up view of the stream from the cat's perspective, and most
-importantly, the imagined feeling of water entering the
-mouth. The imagined sensory experience can come from a
-simulation of the event, but can also be pattern-matched from
-previous, similar embodied experience.
-3. The action is now easily identified as drinking by the sense of
-taste alone. The other senses (such as the tongue moving in and
-out) help to give plausibility to the simulated action. Note that
-the sense of vision, while critical in creating the simulation,
-is not critical for identifying the action from the simulation.
-For the chair examples, the process is even easier:
-1. Align a model of your body to the person in the image.
-2. Generate proprioceptive sensory data from this alignment.
-3. Use the imagined proprioceptive data as a key to lookup related
-sensory experience associated with that particular proproceptive
-feeling.
-4. Retrieve the feeling of your bottom resting on a surface, your
-knees bent, and your leg muscles relaxed.
-5. This sensory information is consistent with the =sitting?=
-sensory predicate, so you (and the entity in the image) must be
-sitting.
-6. There must be a chair-like object since you are sitting.
-Empathy offers yet another alternative to the age-old AI
-representation question: ``What is a chair?'' --- A chair is the
-feeling of sitting.
 My program, =EMPATH= uses this empathic problem solving technique
 to interpret the actions of a simple, worm-like creature.
 #+caption: The worm performs many actions during free play such as
 #+caption: poses by inferring the complete sensory experience
 #+caption: from proprioceptive data.
 #+name: worm-recognition-intro
 #+ATTR_LaTeX: :width 15cm
 [[./images/worm-poses.png]]
-One powerful advantage of empathic problem solving is that it
+#+caption: From only \emph{proprioceptive} data, =EMPATH= was able to infer
-factors the action recognition problem into two easier problems. To
+#+caption: the complete sensory experience and classify these four poses.
-use empathy, you need an /aligner/, which takes the video and a
+#+caption: The last image is a composite, depicting the intermediate stages
-model of your body, and aligns the model with the video. Then, you
+#+caption: of \emph{wriggling}.
-need a /recognizer/, which uses the aligned model to interpret the
+#+name: worm-recognition-intro-2
-action. The power in this method lies in the fact that you describe
+#+ATTR_LaTeX: :width 15cm
-all actions form a body-centered viewpoint. You are less tied to
+[[./images/empathy-1.png]]
-the particulars of any visual representation of the actions. If you
-teach the system what ``running'' is, and you have a good enough
+Next, I developed an experiment to test the power of =CORTEX='s
-aligner, the system will from then on be able to recognize running
+sensorimotor-centered language for solving recognition problems. As
-from any point of view, even strange points of view like above or
+a proof of concept, I wrote routines which enabled a simple
-underneath the runner. This is in contrast to action recognition
+worm-like creature to infer the actions of a second worm-like
-schemes that try to identify actions using a non-embodied approach.
+creature, using only its own previous sensorimotor experiences and
-If these systems learn about running as viewed from the side, they
+knowledge of the second worm's joints (figure
-will not automatically be able to recognize running from any other
+\ref{worm-recognition-intro-2}). The result of this proof of
-viewpoint.
+concept was the program =EMPATH=, described in section \ref{sec-3}.
-Another powerful advantage is that using the language of multiple
+** =EMPATH= is built on =CORTEX=, en environment for making creatures.
-body-centered rich senses to describe body-centerd actions offers a
-massive boost in descriptive capability. Consider how difficult it
+# =CORTEX= provides a language for describing the sensorimotor
-would be to compose a set of HOG filters to describe the action of
+# experiences of various creatures.
-a simple worm-creature ``curling'' so that its head touches its
-tail, and then behold the simplicity of describing thus action in a
-language designed for the task (listing \ref{grand-circle-intro}):
-#+caption: Body-centerd actions are best expressed in a body-centered
-#+caption: language. This code detects when the worm has curled into a
-#+caption: full circle. Imagine how you would replicate this functionality
-#+caption: using low-level pixel features such as HOG filters!
-#+name: grand-circle-intro
-#+begin_listing clojure
-#+begin_src clojure
-(defn grand-circle?
-"Does the worm form a majestic circle (one end touching the other)?"
-[experiences]
-(and (curled? experiences)
-(let [worm-touch (:touch (peek experiences))
-tail-touch (worm-touch 0)
-head-touch (worm-touch 4)]
-(and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
-(< 0.2 (contact worm-segment-top-tip    head-touch))))))
-#+end_src
-#+end_listing
-**  =CORTEX= is a toolkit for building sensate creatures
 I built =CORTEX= to be a general AI research platform for doing
 experiments involving multiple rich senses and a wide variety and
 number of creatures. I intend it to be useful as a library for many
 more projects than just this thesis. =CORTEX= was necessary to meet
 that I know of that can support multiple entities that can each
 hear the world from their own perspective. Other senses also
 require a small layer of Java code. =CORTEX= also uses =bullet=, a
 physics simulator written in =C=.
-#+caption: Here is the worm from above modeled in Blender, a free
+#+caption: Here is the worm from figure \ref{worm-intro} modeled
-#+caption: 3D-modeling program. Senses and joints are described
+#+caption: in Blender, a free 3D-modeling program. Senses and
-#+caption: using special nodes in Blender.
+#+caption: joints are described using special nodes in Blender.
 #+name: worm-recognition-intro
 #+ATTR_LaTeX: :width 12cm
 [[./images/blender-worm.png]]
 Here are some thing I anticipate that =CORTEX= might be used for:
 its own finger from the eye in its palm, and that it can feel its
 own thumb touching its palm.}
 \end{sidewaysfigure}
 #+END_LaTeX
-** Road map
+** Contributions
-By the end of this thesis, you will have seen a novel approach to
-interpreting video using embodiment and empathy. You will have also
-seen one way to efficiently implement empathy for embodied
-creatures. Finally, you will become familiar with =CORTEX=, a system
-for designing and simulating creatures with rich senses, which you
-may choose to use in your own research.
-This is the core vision of my thesis: That one of the important ways
-in which we understand others is by imagining ourselves in their
-position and emphatically feeling experiences relative to our own
-bodies. By understanding events in terms of our own previous
-corporeal experience, we greatly constrain the possibilities of what
-would otherwise be an unwieldy exponential search. This extra
-constraint can be the difference between easily understanding what
-is happening in a video and being completely lost in a sea of
-incomprehensible color and movement.
 - I built =CORTEX=, a comprehensive platform for embodied AI
 experiments. =CORTEX= supports many features lacking in other
 systems, such proper simulation of hearing. It is easy to create
 new =CORTEX= creatures using Blender, a free 3D modeling program.
 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
 a worm-like creature using a computational model of empathy.
+- After one-shot supervised training, =EMPATH= was able recognize a
+wide variety of static poses and dynamic actions---ranging from
+curling in a circle to wriggling with a particular frequency ---
+with 95\% accuracy.
+- These results were completely independent of viewing angle
+because the underlying body-centered language fundamentally is
+independent; once an action is learned, it can be recognized
+equally well from any viewing angle.
+- =EMPATH= is surprisingly short; the sensorimotor-centered
+language provided by =CORTEX= resulted in extremely economical
+recognition routines --- about 500 lines in all --- suggesting
+that such representations are very powerful, and often
+indispensible for the types of recognition tasks considered here.
+- Although for expediency's sake, I relied on direct knowledge of
+joint positions in this proof of concept, it would be
+straightforward to extend =EMPATH= so that it (more
+realistically) infers joint positions from its visual data.
 * Designing =CORTEX=
 In this section, I outline the design decisions that went into
-making =CORTEX=, along with some details about its
+making =CORTEX=, along with some details about its implementation.
-implementation. (A practical guide to getting started with =CORTEX=,
+(A practical guide to getting started with =CORTEX=, which skips
-which skips over the history and implementation details presented
+over the history and implementation details presented here, is
-here, is provided in an appendix \ref{} at the end of this paper.)
+provided in an appendix at the end of this thesis.)
 Throughout this project, I intended for =CORTEX= to be flexible and
 extensible enough to be useful for other researchers who want to
 test out ideas of their own. To this end, wherver I have had to make
 archetictural choices about =CORTEX=, I have chosen to give as much
 time in the simulated world can be slowed down to accommodate the
 limitations of the character's programming. In terms of cost,
 doing everything in software is far cheaper than building custom
 real-time hardware. All you need is a laptop and some patience.
-** Simulated time enables rapid prototyping and complex scenes
+** Simulated time enables rapid prototyping \& simple programs
 I envision =CORTEX= being used to support rapid prototyping and
 iteration of ideas. Even if I could put together a well constructed
 kit for creating robots, it would still not be enough because of
 the scourge of real-time processing. Anyone who wants to test their

Mercurial > cortex

comparison thesis/cortex.org @ 516:ced955c3c84f