view thesis/cortex.org @ 438:4dcb923c9b16

progress on intro.
author Robert McIntyre <rlm@mit.edu>
date Sun, 23 Mar 2014 22:22:00 -0400
parents c1e6b7221b2f
children 97dc719fd1ac
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
8 * Empathy and Embodiment: problem solving strategies
10 By the end of this thesis, you will have seen a novel approach to
11 interpreting video using embodiment and empathy. You will have also
12 seen one way to efficiently implement empathy for embodied
13 creatures.
15 The core vision of this thesis is that one of the important ways in
16 which we understand others is by imagining ourselves in their
17 posistion and empathicaly feeling experiences based on our own past
18 experiences and imagination.
20 By understanding events in terms of our own previous corperal
21 experience, we greatly constrain the possibilities of what would
22 otherwise be an unweidly exponential search. This extra constraint
23 can be the difference between easily understanding what is happening
24 in a video and being completely lost in a sea of incomprehensible
25 color and movement.
27 ** Recognizing actions in video is extremely difficult
29 Consider for example the problem of determining what is happening in
30 a video of which this is one frame:
32 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.
33 #+ATTR_LaTeX: :width 7cm
34 [[./images/cat-drinking.jpg]]
36 It is currently impossible for any computer program to reliably
37 label such an video as "drinking". And rightly so -- it is a very
38 hard problem! What features can you describe in terms of low level
39 functions of pixels that can even begin to describe what is
40 happening here?
42 Or suppose that you are building a program that recognizes
43 chairs. How could you ``see'' the chair in the following picture?
45 #+caption: When you look at this, do you think ``chair''? I certainly do.
46 #+ATTR_LaTeX: :width 10cm
47 [[./images/invisible-chair.png]]
49 #+caption: The chair in this image is quite obvious to humans, but I doubt any computer program can find it.
50 #+ATTR_LaTeX: :width 10cm
51 [[./images/fat-person-sitting-at-desk.jpg]]
54 I think humans are able to label
55 such video as "drinking" because they imagine /themselves/ as the
56 cat, and imagine putting their face up against a stream of water and
57 sticking out their tongue. In that imagined world, they can feel the
58 cool water hitting their tongue, and feel the water entering their
59 body, and are able to recognize that /feeling/ as drinking. So, the
60 label of the action is not really in the pixels of the image, but is
61 found clearly in a simulation inspired by those pixels. An
62 imaginative system, having been trained on drinking and non-drinking
63 examples and learning that the most important component of drinking
64 is the feeling of water sliding down one's throat, would analyze a
65 video of a cat drinking in the following manner:
67 - Create a physical model of the video by putting a "fuzzy" model
68 of its own body in place of the cat. Also, create a simulation of
69 the stream of water.
71 - Play out this simulated scene and generate imagined sensory
72 experience. This will include relevant muscle contractions, a
73 close up view of the stream from the cat's perspective, and most
74 importantly, the imagined feeling of water entering the mouth.
76 - The action is now easily identified as drinking by the sense of
77 taste alone. The other senses (such as the tongue moving in and
78 out) help to give plausibility to the simulated action. Note that
79 the sense of vision, while critical in creating the simulation,
80 is not critical for identifying the action from the simulation.
88 cat drinking, mimes, leaning, common sense
90 ** =EMPATH= neatly solves recognition problems
92 factorization , right language, etc
94 a new possibility for the question ``what is a chair?'' -- it's the
95 feeling of your butt on something and your knees bent, with your
96 back muscles and legs relaxed.
98 ** =CORTEX= is a toolkit for building sensate creatures
100 Hand integration demo
102 ** Contributions
104 * Building =CORTEX=
106 ** To explore embodiment, we need a world, body, and senses
108 ** Because of Time, simulation is perferable to reality
110 ** Video game engines are a great starting point
112 ** Bodies are composed of segments connected by joints
114 ** Eyes reuse standard video game components
116 ** Hearing is hard; =CORTEX= does it right
118 ** Touch uses hundreds of hair-like elements
120 ** Proprioception is the force that makes everything ``real''
122 ** Muscles are both effectors and sensors
124 ** =CORTEX= brings complex creatures to life!
126 ** =CORTEX= enables many possiblities for further research
128 * Empathy in a simulated worm
130 ** Embodiment factors action recognition into managable parts
132 ** Action recognition is easy with a full gamut of senses
134 ** Digression: bootstrapping touch using free exploration
136 ** \Phi-space describes the worm's experiences
138 ** Empathy is the process of tracing though \Phi-space
140 ** Efficient action recognition via empathy
142 * Contributions
143 - Built =CORTEX=, a comprehensive platform for embodied AI
144 experiments. Has many new features lacking in other systems, such
145 as sound. Easy to model/create new creatures.
146 - created a novel concept for action recognition by using artificial
147 imagination.
149 In the second half of the thesis I develop a computational model of
150 empathy, using =CORTEX= as a base. Empathy in this context is the
151 ability to observe another creature and infer what sorts of sensations
152 that creature is feeling. My empathy algorithm involves multiple
153 phases. First is free-play, where the creature moves around and gains
154 sensory experience. From this experience I construct a representation
155 of the creature's sensory state space, which I call \phi-space. Using
156 \phi-space, I construct an efficient function for enriching the
157 limited data that comes from observing another creature with a full
158 compliment of imagined sensory data based on previous experience. I
159 can then use the imagined sensory data to recognize what the observed
160 creature is doing and feeling, using straightforward embodied action
161 predicates. This is all demonstrated with using a simple worm-like
162 creature, and recognizing worm-actions based on limited data.
164 Embodied representation using multiple senses such as touch,
165 proprioception, and muscle tension turns out be be exceedingly
166 efficient at describing body-centered actions. It is the ``right
167 language for the job''. For example, it takes only around 5 lines of
168 LISP code to describe the action of ``curling'' using embodied
169 primitives. It takes about 8 lines to describe the seemingly
170 complicated action of wiggling.
174 * COMMENT names for cortex
175 - bioland