Mercurial > cortex

annotate thesis/org/first-chapter.org @ 401:7ee735a836da

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
incorporate thesis.
author Robert McIntyre <rlm@mit.edu>
date Sun, 16 Mar 2014 23:31:16 -0400
parents
children
rev   line source
rlm@401 1 #+title: =CORTEX=
rlm@401 2 #+author: Robert McIntyre
rlm@401 3 #+email: rlm@mit.edu
rlm@401 4 #+description: Using embodied AI to facilitate Artificial Imagination.
rlm@401 5 #+keywords: AI, clojure, embodiment
rlm@401 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@401 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@401 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@401 9 #+OPTIONS: toc:nil, num:nil
rlm@401 10
rlm@401 11 * Artificial Imagination
rlm@401 12
rlm@401 13 Imagine watching a video of someone skateboarding. When you watch
rlm@401 14 the video, you can imagine yourself skateboarding, and your
rlm@401 15 knowledge of the human body and its dynamics guides your
rlm@401 16 interpretation of the scene. For example, even if the skateboarder
rlm@401 17 is partially occluded, you can infer the positions of his arms and
rlm@401 18 body from your own knowledge of how your body would be positioned if
rlm@401 19 you were skateboarding. If the skateboarder suffers an accident, you
rlm@401 20 wince in sympathy, imagining the pain your own body would experience
rlm@401 21 if it were in the same situation. This empathy with other people
rlm@401 22 guides our understanding of whatever they are doing because it is a
rlm@401 23 powerful constraint on what is probable and possible. In order to
rlm@401 24 make use of this powerful empathy constraint, I need a system that
rlm@401 25 can generate and make sense of sensory data from the many different
rlm@401 26 senses that humans possess. The two key proprieties of such a system
rlm@401 27 are /embodiment/ and /imagination/.
rlm@401 28
rlm@401 29 ** What is imagination?
rlm@401 30
rlm@401 31 One kind of imagination is /sympathetic/ imagination: you imagine
rlm@401 32 yourself in the position of something/someone you are
rlm@401 33 observing. This type of imagination comes into play when you follow
rlm@401 34 along visually when watching someone perform actions, or when you
rlm@401 35 sympathetically grimace when someone hurts themselves. This type of
rlm@401 36 imagination uses the constraints you have learned about your own
rlm@401 37 body to highly constrain the possibilities in whatever you are
rlm@401 38 seeing. It uses all your senses to including your senses of touch,
rlm@401 39 proprioception, etc. Humans are flexible when it comes to "putting
rlm@401 40 themselves in another's shoes," and can sympathetically understand
rlm@401 41 not only other humans, but entities ranging from animals to cartoon
rlm@401 42 characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen!
rlm@401 43
rlm@401 44 Another kind of imagination is /predictive/ imagination: you
rlm@401 45 construct scenes in your mind that are not entirely related to
rlm@401 46 whatever you are observing, but instead are predictions of the
rlm@401 47 future or simply flights of fancy. You use this type of imagination
rlm@401 48 to plan out multi-step actions, or play out dangerous situations in
rlm@401 49 your mind so as to avoid messing them up in reality.
rlm@401 50
rlm@401 51 Of course, sympathetic and predictive imagination blend into each
rlm@401 52 other and are not completely separate concepts. One dimension along
rlm@401 53 which you can distinguish types of imagination is dependence on raw
rlm@401 54 sense data. Sympathetic imagination is highly constrained by your
rlm@401 55 senses, while predictive imagination can be more or less dependent
rlm@401 56 on your senses depending on how far ahead you imagine. Daydreaming
rlm@401 57 is an extreme form of predictive imagination that wanders through
rlm@401 58 different possibilities without concern for whether they are
rlm@401 59 related to whatever is happening in reality.
rlm@401 60
rlm@401 61 For this thesis, I will mostly focus on sympathetic imagination and
rlm@401 62 the constraint it provides for understanding sensory data.
rlm@401 63
rlm@401 64 ** What problems can imagination solve?
rlm@401 65
rlm@401 66 Consider a video of a cat drinking some water.
rlm@401 67
rlm@401 68 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.
rlm@401 69 #+ATTR_LaTeX: width=5cm
rlm@401 70 [[../images/cat-drinking.jpg]]
rlm@401 71
rlm@401 72 It is currently impossible for any computer program to reliably
rlm@401 73 label such an video as "drinking". I think humans are able to label
rlm@401 74 such video as "drinking" because they imagine /themselves/ as the
rlm@401 75 cat, and imagine putting their face up against a stream of water
rlm@401 76 and sticking out their tongue. In that imagined world, they can
rlm@401 77 feel the cool water hitting their tongue, and feel the water
rlm@401 78 entering their body, and are able to recognize that /feeling/ as
rlm@401 79 drinking. So, the label of the action is not really in the pixels
rlm@401 80 of the image, but is found clearly in a simulation inspired by
rlm@401 81 those pixels. An imaginative system, having been trained on
rlm@401 82 drinking and non-drinking examples and learning that the most
rlm@401 83 important component of drinking is the feeling of water sliding
rlm@401 84 down one's throat, would analyze a video of a cat drinking in the
rlm@401 85 following manner:
rlm@401 86
rlm@401 87 - Create a physical model of the video by putting a "fuzzy" model
rlm@401 88 of its own body in place of the cat. Also, create a simulation of
rlm@401 89 the stream of water.
rlm@401 90
rlm@401 91 - Play out this simulated scene and generate imagined sensory
rlm@401 92 experience. This will include relevant muscle contractions, a
rlm@401 93 close up view of the stream from the cat's perspective, and most
rlm@401 94 importantly, the imagined feeling of water entering the mouth.
rlm@401 95
rlm@401 96 - The action is now easily identified as drinking by the sense of
rlm@401 97 taste alone. The other senses (such as the tongue moving in and
rlm@401 98 out) help to give plausibility to the simulated action. Note that
rlm@401 99 the sense of vision, while critical in creating the simulation,
rlm@401 100 is not critical for identifying the action from the simulation.
rlm@401 101
rlm@401 102 More generally, I expect imaginative systems to be particularly
rlm@401 103 good at identifying embodied actions in videos.
rlm@401 104
rlm@401 105 * Cortex
rlm@401 106
rlm@401 107 The previous example involves liquids, the sense of taste, and
rlm@401 108 imagining oneself as a cat. For this thesis I constrain myself to
rlm@401 109 simpler, more easily digitizable senses and situations.
rlm@401 110
rlm@401 111 My system, =CORTEX= performs imagination in two different simplified
rlm@401 112 worlds: /worm world/ and /stick-figure world/. In each of these
rlm@401 113 worlds, entities capable of imagination recognize actions by
rlm@401 114 simulating the experience from their own perspective, and then
rlm@401 115 recognizing the action from a database of examples.
rlm@401 116
rlm@401 117 In order to serve as a framework for experiments in imagination,
rlm@401 118 =CORTEX= requires simulated bodies, worlds, and senses like vision,
rlm@401 119 hearing, touch, proprioception, etc.
rlm@401 120
rlm@401 121 ** A Video Game Engine takes care of some of the groundwork
rlm@401 122
rlm@401 123 When it comes to simulation environments, the engines used to
rlm@401 124 create the worlds in video games offer top-notch physics and
rlm@401 125 graphics support. These engines also have limited support for
rlm@401 126 creating cameras and rendering 3D sound, which can be repurposed
rlm@401 127 for vision and hearing respectively. Physics collision detection
rlm@401 128 can be expanded to create a sense of touch.
rlm@401 129
rlm@401 130 jMonkeyEngine3 is one such engine for creating video games in
rlm@401 131 Java. It uses OpenGL to render to the screen and uses screengraphs
rlm@401 132 to avoid drawing things that do not appear on the screen. It has an
rlm@401 133 active community and several games in the pipeline. The engine was
rlm@401 134 not built to serve any particular game but is instead meant to be
rlm@401 135 used for any 3D game. I chose jMonkeyEngine3 it because it had the
rlm@401 136 most features out of all the open projects I looked at, and because
rlm@401 137 I could then write my code in Clojure, an implementation of LISP
rlm@401 138 that runs on the JVM.
rlm@401 139
rlm@401 140 ** =CORTEX= Extends jMonkeyEngine3 to implement rich senses
rlm@401 141
rlm@401 142 Using the game-making primitives provided by jMonkeyEngine3, I have
rlm@401 143 constructed every major human sense except for smell and
rlm@401 144 taste. =CORTEX= also provides an interface for creating creatures
rlm@401 145 in Blender, a 3D modeling environment, and then "rigging" the
rlm@401 146 creatures with senses using 3D annotations in Blender. A creature
rlm@401 147 can have any number of senses, and there can be any number of
rlm@401 148 creatures in a simulation.
rlm@401 149
rlm@401 150 The senses available in =CORTEX= are:
rlm@401 151
rlm@401 152 - [[../../cortex/html/vision.html][Vision]]
rlm@401 153 - [[../../cortex/html/hearing.html][Hearing]]
rlm@401 154 - [[../../cortex/html/touch.html][Touch]]
rlm@401 155 - [[../../cortex/html/proprioception.html][Proprioception]]
rlm@401 156 - [[../../cortex/html/movement.html][Muscle Tension]]
rlm@401 157
rlm@401 158 * A roadmap for =CORTEX= experiments
rlm@401 159
rlm@401 160 ** Worm World
rlm@401 161
rlm@401 162 Worms in =CORTEX= are segmented creatures which vary in length and
rlm@401 163 number of segments, and have the senses of vision, proprioception,
rlm@401 164 touch, and muscle tension.
rlm@401 165
rlm@401 166 #+attr_html: width=755
rlm@401 167 #+caption: This is the tactile-sensor-profile for the upper segment of a worm. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).
rlm@401 168 [[../images/finger-UV.png]]
rlm@401 169
rlm@401 170
rlm@401 171 #+begin_html
rlm@401 172 <div class="figure">
rlm@401 173 <center>
rlm@401 174 <video controls="controls" width="550">
rlm@401 175 <source src="../video/worm-touch.ogg" type="video/ogg"
rlm@401 176 preload="none" />
rlm@401 177 </video>
rlm@401 178 <br> <a href="http://youtu.be/RHx2wqzNVcU"> YouTube </a>
rlm@401 179 </center>
rlm@401 180 <p>The worm responds to touch.</p>
rlm@401 181 </div>
rlm@401 182 #+end_html
rlm@401 183
rlm@401 184 #+begin_html
rlm@401 185 <div class="figure">
rlm@401 186 <center>
rlm@401 187 <video controls="controls" width="550">
rlm@401 188 <source src="../video/test-proprioception.ogg" type="video/ogg"
rlm@401 189 preload="none" />
rlm@401 190 </video>
rlm@401 191 <br> <a href="http://youtu.be/JjdDmyM8b0w"> YouTube </a>
rlm@401 192 </center>
rlm@401 193 <p>Proprioception in a worm. The proprioceptive readout is
rlm@401 194 in the upper left corner of the screen.</p>
rlm@401 195 </div>
rlm@401 196 #+end_html
rlm@401 197
rlm@401 198 A worm is trained in various actions such as sinusoidal movement,
rlm@401 199 curling, flailing, and spinning by directly playing motor
rlm@401 200 contractions while the worm "feels" the experience. These actions
rlm@401 201 are recorded both as vectors of muscle tension, touch, and
rlm@401 202 proprioceptive data, but also in higher level forms such as
rlm@401 203 frequencies of the various contractions and a symbolic name for the
rlm@401 204 action.
rlm@401 205
rlm@401 206 Then, the worm watches a video of another worm performing one of
rlm@401 207 the actions, and must judge which action was performed. Normally
rlm@401 208 this would be an extremely difficult problem, but the worm is able
rlm@401 209 to greatly diminish the search space through sympathetic
rlm@401 210 imagination. First, it creates an imagined copy of its body which
rlm@401 211 it observes from a third person point of view. Then for each frame
rlm@401 212 of the video, it maneuvers its simulated body to be in registration
rlm@401 213 with the worm depicted in the video. The physical constraints
rlm@401 214 imposed by the physics simulation greatly decrease the number of
rlm@401 215 poses that have to be tried, making the search feasible. As the
rlm@401 216 imaginary worm moves, it generates imaginary muscle tension and
rlm@401 217 proprioceptive sensations. The worm determines the action not by
rlm@401 218 vision, but by matching the imagined proprioceptive data with
rlm@401 219 previous examples.
rlm@401 220
rlm@401 221 By using non-visual sensory data such as touch, the worms can also
rlm@401 222 answer body related questions such as "did your head touch your
rlm@401 223 tail?" and "did worm A touch worm B?"
rlm@401 224
rlm@401 225 The proprioceptive information used for action identification is
rlm@401 226 body-centric, so only the registration step is dependent on point
rlm@401 227 of view, not the identification step. Registration is not specific
rlm@401 228 to any particular action. Thus, action identification can be
rlm@401 229 divided into a point-of-view dependent generic registration step,
rlm@401 230 and a action-specific step that is body-centered and invariant to
rlm@401 231 point of view.
rlm@401 232
rlm@401 233 ** Stick Figure World
rlm@401 234
rlm@401 235 This environment is similar to Worm World, except the creatures are
rlm@401 236 more complicated and the actions and questions more varied. It is
rlm@401 237 an experiment to see how far imagination can go in interpreting
rlm@401 238 actions.