view thesis/cortex.org @ 450:432f2c4646cb

sleepig.
author Robert McIntyre <rlm@mit.edu>
date Wed, 26 Mar 2014 03:18:57 -0400
parents 09b7c8dd4365
children 0a4362d1f138
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
8 * Empathy and Embodiment as problem solving strategies
10 By the end of this thesis, you will have seen a novel approach to
11 interpreting video using embodiment and empathy. You will have also
12 seen one way to efficiently implement empathy for embodied
13 creatures. Finally, you will become familiar with =CORTEX=, a system
14 for designing and simulating creatures with rich senses, which you
15 may choose to use in your own research.
17 This is the core vision of my thesis: That one of the important ways
18 in which we understand others is by imagining ourselves in their
19 position and emphatically feeling experiences relative to our own
20 bodies. By understanding events in terms of our own previous
21 corporeal experience, we greatly constrain the possibilities of what
22 would otherwise be an unwieldy exponential search. This extra
23 constraint can be the difference between easily understanding what
24 is happening in a video and being completely lost in a sea of
25 incomprehensible color and movement.
27 ** Recognizing actions in video is extremely difficult
29 Consider for example the problem of determining what is happening
30 in a video of which this is one frame:
32 #+caption: A cat drinking some water. Identifying this action is
33 #+caption: beyond the state of the art for computers.
34 #+ATTR_LaTeX: :width 7cm
35 [[./images/cat-drinking.jpg]]
37 It is currently impossible for any computer program to reliably
38 label such a video as ``drinking''. And rightly so -- it is a very
39 hard problem! What features can you describe in terms of low level
40 functions of pixels that can even begin to describe at a high level
41 what is happening here?
43 Or suppose that you are building a program that recognizes chairs.
44 How could you ``see'' the chair in figure \ref{hidden-chair}?
46 #+caption: The chair in this image is quite obvious to humans, but I
47 #+caption: doubt that any modern computer vision program can find it.
48 #+name: hidden-chair
49 #+ATTR_LaTeX: :width 10cm
50 [[./images/fat-person-sitting-at-desk.jpg]]
52 Finally, how is it that you can easily tell the difference between
53 how the girls /muscles/ are working in figure \ref{girl}?
55 #+caption: The mysterious ``common sense'' appears here as you are able
56 #+caption: to discern the difference in how the girl's arm muscles
57 #+caption: are activated between the two images.
58 #+name: girl
59 #+ATTR_LaTeX: :width 7cm
60 [[./images/wall-push.png]]
62 Each of these examples tells us something about what might be going
63 on in our minds as we easily solve these recognition problems.
65 The hidden chairs show us that we are strongly triggered by cues
66 relating to the position of human bodies, and that we can determine
67 the overall physical configuration of a human body even if much of
68 that body is occluded.
70 The picture of the girl pushing against the wall tells us that we
71 have common sense knowledge about the kinetics of our own bodies.
72 We know well how our muscles would have to work to maintain us in
73 most positions, and we can easily project this self-knowledge to
74 imagined positions triggered by images of the human body.
76 ** =EMPATH= neatly solves recognition problems
78 I propose a system that can express the types of recognition
79 problems above in a form amenable to computation. It is split into
80 four parts:
82 - Free/Guided Play :: The creature moves around and experiences the
83 world through its unique perspective. Many otherwise
84 complicated actions are easily described in the language of a
85 full suite of body-centered, rich senses. For example,
86 drinking is the feeling of water sliding down your throat, and
87 cooling your insides. It's often accompanied by bringing your
88 hand close to your face, or bringing your face close to water.
89 Sitting down is the feeling of bending your knees, activating
90 your quadriceps, then feeling a surface with your bottom and
91 relaxing your legs. These body-centered action descriptions
92 can be either learned or hard coded.
93 - Posture Imitation :: When trying to interpret a video or image,
94 the creature takes a model of itself and aligns it with
95 whatever it sees. This alignment can even cross species, as
96 when humans try to align themselves with things like ponies,
97 dogs, or other humans with a different body type.
98 - Empathy :: The alignment triggers associations with
99 sensory data from prior experiences. For example, the
100 alignment itself easily maps to proprioceptive data. Any
101 sounds or obvious skin contact in the video can to a lesser
102 extent trigger previous experience. Segments of previous
103 experiences are stitched together to form a coherent and
104 complete sensory portrait of the scene.
105 - Recognition :: With the scene described in terms of first
106 person sensory events, the creature can now run its
107 action-identification programs on this synthesized sensory
108 data, just as it would if it were actually experiencing the
109 scene first-hand. If previous experience has been accurately
110 retrieved, and if it is analogous enough to the scene, then
111 the creature will correctly identify the action in the scene.
113 For example, I think humans are able to label the cat video as
114 ``drinking'' because they imagine /themselves/ as the cat, and
115 imagine putting their face up against a stream of water and
116 sticking out their tongue. In that imagined world, they can feel
117 the cool water hitting their tongue, and feel the water entering
118 their body, and are able to recognize that /feeling/ as drinking.
119 So, the label of the action is not really in the pixels of the
120 image, but is found clearly in a simulation inspired by those
121 pixels. An imaginative system, having been trained on drinking and
122 non-drinking examples and learning that the most important
123 component of drinking is the feeling of water sliding down one's
124 throat, would analyze a video of a cat drinking in the following
125 manner:
127 1. Create a physical model of the video by putting a ``fuzzy''
128 model of its own body in place of the cat. Possibly also create
129 a simulation of the stream of water.
131 2. Play out this simulated scene and generate imagined sensory
132 experience. This will include relevant muscle contractions, a
133 close up view of the stream from the cat's perspective, and most
134 importantly, the imagined feeling of water entering the
135 mouth. The imagined sensory experience can come from a
136 simulation of the event, but can also be pattern-matched from
137 previous, similar embodied experience.
139 3. The action is now easily identified as drinking by the sense of
140 taste alone. The other senses (such as the tongue moving in and
141 out) help to give plausibility to the simulated action. Note that
142 the sense of vision, while critical in creating the simulation,
143 is not critical for identifying the action from the simulation.
145 For the chair examples, the process is even easier:
147 1. Align a model of your body to the person in the image.
149 2. Generate proprioceptive sensory data from this alignment.
151 3. Use the imagined proprioceptive data as a key to lookup related
152 sensory experience associated with that particular proproceptive
153 feeling.
155 4. Retrieve the feeling of your bottom resting on a surface, your
156 knees bent, and your leg muscles relaxed.
158 5. This sensory information is consistent with the =sitting?=
159 sensory predicate, so you (and the entity in the image) must be
160 sitting.
162 6. There must be a chair-like object since you are sitting.
164 Empathy offers yet another alternative to the age-old AI
165 representation question: ``What is a chair?'' --- A chair is the
166 feeling of sitting.
168 My program, =EMPATH= uses this empathic problem solving technique
169 to interpret the actions of a simple, worm-like creature.
171 #+caption: The worm performs many actions during free play such as
172 #+caption: curling, wiggling, and resting.
173 #+name: worm-intro
174 #+ATTR_LaTeX: :width 15cm
175 [[./images/worm-intro-white.png]]
177 #+caption: =EMPATH= recognized and classified each of these poses by
178 #+caption: inferring the complete sensory experience from
179 #+caption: proprioceptive data.
180 #+name: worm-recognition-intro
181 #+ATTR_LaTeX: :width 15cm
182 [[./images/worm-poses.png]]
184 One powerful advantage of empathic problem solving is that it
185 factors the action recognition problem into two easier problems. To
186 use empathy, you need an /aligner/, which takes the video and a
187 model of your body, and aligns the model with the video. Then, you
188 need a /recognizer/, which uses the aligned model to interpret the
189 action. The power in this method lies in the fact that you describe
190 all actions form a body-centered viewpoint. You are less tied to
191 the particulars of any visual representation of the actions. If you
192 teach the system what ``running'' is, and you have a good enough
193 aligner, the system will from then on be able to recognize running
194 from any point of view, even strange points of view like above or
195 underneath the runner. This is in contrast to action recognition
196 schemes that try to identify actions using a non-embodied approach.
197 If these systems learn about running as viewed from the side, they
198 will not automatically be able to recognize running from any other
199 viewpoint.
201 Another powerful advantage is that using the language of multiple
202 body-centered rich senses to describe body-centerd actions offers a
203 massive boost in descriptive capability. Consider how difficult it
204 would be to compose a set of HOG filters to describe the action of
205 a simple worm-creature ``curling'' so that its head touches its
206 tail, and then behold the simplicity of describing thus action in a
207 language designed for the task (listing \ref{grand-circle-intro}):
209 #+caption: Body-centerd actions are best expressed in a body-centered
210 #+caption: language. This code detects when the worm has curled into a
211 #+caption: full circle. Imagine how you would replicate this functionality
212 #+caption: using low-level pixel features such as HOG filters!
213 #+name: grand-circle-intro
214 #+begin_listing clojure
215 #+begin_src clojure
216 (defn grand-circle?
217 "Does the worm form a majestic circle (one end touching the other)?"
218 [experiences]
219 (and (curled? experiences)
220 (let [worm-touch (:touch (peek experiences))
221 tail-touch (worm-touch 0)
222 head-touch (worm-touch 4)]
223 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
224 (< 0.55 (contact worm-segment-top-tip head-touch))))))
225 #+end_src
226 #+end_listing
229 ** =CORTEX= is a toolkit for building sensate creatures
231 I built =CORTEX= to be a general AI research platform for doing
232 experiments involving multiple rich senses and a wide variety and
233 number of creatures. I intend it to be useful as a library for many
234 more projects than just this one. =CORTEX= was necessary to meet a
235 need among AI researchers at CSAIL and beyond, which is that people
236 often will invent neat ideas that are best expressed in the
237 language of creatures and senses, but in order to explore those
238 ideas they must first build a platform in which they can create
239 simulated creatures with rich senses! There are many ideas that
240 would be simple to execute (such as =EMPATH=), but attached to them
241 is the multi-month effort to make a good creature simulator. Often,
242 that initial investment of time proves to be too much, and the
243 project must make do with a lesser environment.
245 =CORTEX= is well suited as an environment for embodied AI research
246 for three reasons:
248 - You can create new creatures using Blender, a popular 3D modeling
249 program. Each sense can be specified using special blender nodes
250 with biologically inspired paramaters. You need not write any
251 code to create a creature, and can use a wide library of
252 pre-existing blender models as a base for your own creatures.
254 - =CORTEX= implements a wide variety of senses, including touch,
255 proprioception, vision, hearing, and muscle tension. Complicated
256 senses like touch, and vision involve multiple sensory elements
257 embedded in a 2D surface. You have complete control over the
258 distribution of these sensor elements through the use of simple
259 png image files. In particular, =CORTEX= implements more
260 comprehensive hearing than any other creature simulation system
261 available.
263 - =CORTEX= supports any number of creatures and any number of
264 senses. Time in =CORTEX= dialates so that the simulated creatures
265 always precieve a perfectly smooth flow of time, regardless of
266 the actual computational load.
268 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
269 engine designed to create cross-platform 3D desktop games. =CORTEX=
270 is mainly written in clojure, a dialect of =LISP= that runs on the
271 java virtual machine (JVM). The API for creating and simulating
272 creatures and senses is entirely expressed in clojure, though many
273 senses are implemented at the layer of jMonkeyEngine or below. For
274 example, for the sense of hearing I use a layer of clojure code on
275 top of a layer of java JNI bindings that drive a layer of =C++=
276 code which implements a modified version of =OpenAL= to support
277 multiple listeners. =CORTEX= is the only simulation environment
278 that I know of that can support multiple entities that can each
279 hear the world from their own perspective. Other senses also
280 require a small layer of Java code. =CORTEX= also uses =bullet=, a
281 physics simulator written in =C=.
283 #+caption: Here is the worm from above modeled in Blender, a free
284 #+caption: 3D-modeling program. Senses and joints are described
285 #+caption: using special nodes in Blender.
286 #+name: worm-recognition-intro
287 #+ATTR_LaTeX: :width 12cm
288 [[./images/blender-worm.png]]
290 Here are some thing I anticipate that =CORTEX= might be used for:
292 - exploring new ideas about sensory integration
293 - distributed communication among swarm creatures
294 - self-learning using free exploration,
295 - evolutionary algorithms involving creature construction
296 - exploration of exoitic senses and effectors that are not possible
297 in the real world (such as telekenisis or a semantic sense)
298 - imagination using subworlds
300 During one test with =CORTEX=, I created 3,000 entities each with
301 their own independent senses and ran them all at only 1/80 real
302 time. In another test, I created a detailed model of my own hand,
303 equipped with a realistic distribution of touch (more sensitive at
304 the fingertips), as well as eyes and ears, and it ran at around 1/4
305 real time.
307 #+BEGIN_LaTeX
308 \begin{sidewaysfigure}
309 \includegraphics[width=9.5in]{images/full-hand.png}
310 \caption{Here is the worm from above modeled in Blender,
311 a free 3D-modeling program. Senses and joints are described
312 using special nodes in Blender. The senses are displayed on
313 the right, and the simulation is displayed on the left. Notice
314 that the hand is curling its fingers, that it can see its own
315 finger from the eye in its palm, and thta it can feel its own
316 thumb touching its palm.}
317 \end{sidewaysfigure}
318 #+END_LaTeX
320 ** Contributions
322 I built =CORTEX=, a comprehensive platform for embodied AI
323 experiments. =CORTEX= many new features lacking in other systems,
324 such as sound. It is easy to create new creatures using Blender, a
325 free 3D modeling program.
327 I built =EMPATH=, which uses =CORTEX= to identify the actions of a
328 worm-like creature using a computational model of empathy.
330 * Building =CORTEX=
332 ** To explore embodiment, we need a world, body, and senses
334 ** Because of Time, simulation is perferable to reality
336 ** Video game engines are a great starting point
338 ** Bodies are composed of segments connected by joints
340 ** Eyes reuse standard video game components
342 ** Hearing is hard; =CORTEX= does it right
344 ** Touch uses hundreds of hair-like elements
346 ** Proprioception is the sense that makes everything ``real''
348 ** Muscles are both effectors and sensors
350 ** =CORTEX= brings complex creatures to life!
352 ** =CORTEX= enables many possiblities for further research
354 * Empathy in a simulated worm
356 Here I develop a computational model of empathy, using =CORTEX= as a
357 base. Empathy in this context is the ability to observe another
358 creature and infer what sorts of sensations that creature is
359 feeling. My empathy algorithm involves multiple phases. First is
360 free-play, where the creature moves around and gains sensory
361 experience. From this experience I construct a representation of the
362 creature's sensory state space, which I call \Phi-space. Using
363 \Phi-space, I construct an efficient function which takes the
364 limited data that comes from observing another creature and enriches
365 it full compliment of imagined sensory data. I can then use the
366 imagined sensory data to recognize what the observed creature is
367 doing and feeling, using straightforward embodied action predicates.
368 This is all demonstrated with using a simple worm-like creature, and
369 recognizing worm-actions based on limited data.
371 #+caption: Here is the worm with which we will be working.
372 #+caption: It is composed of 5 segments. Each segment has a
373 #+caption: pair of extensor and flexor muscles. Each of the
374 #+caption: worm's four joints is a hinge joint which allows
375 #+caption: 30 degrees of rotation to either side. Each segment
376 #+caption: of the worm is touch-capable and has a uniform
377 #+caption: distribution of touch sensors on each of its faces.
378 #+caption: Each joint has a proprioceptive sense to detect
379 #+caption: relative positions. The worm segments are all the
380 #+caption: same except for the first one, which has a much
381 #+caption: higher weight than the others to allow for easy
382 #+caption: manual motor control.
383 #+name: basic-worm-view
384 #+ATTR_LaTeX: :width 10cm
385 [[./images/basic-worm-view.png]]
387 #+caption: Program for reading a worm from a blender file and
388 #+caption: outfitting it with the senses of proprioception,
389 #+caption: touch, and the ability to move, as specified in the
390 #+caption: blender file.
391 #+name: get-worm
392 #+begin_listing clojure
393 #+begin_src clojure
394 (defn worm []
395 (let [model (load-blender-model "Models/worm/worm.blend")]
396 {:body (doto model (body!))
397 :touch (touch! model)
398 :proprioception (proprioception! model)
399 :muscles (movement! model)}))
400 #+end_src
401 #+end_listing
403 ** Embodiment factors action recognition into managable parts
405 Using empathy, I divide the problem of action recognition into a
406 recognition process expressed in the language of a full compliment
407 of senses, and an imaganitive process that generates full sensory
408 data from partial sensory data. Splitting the action recognition
409 problem in this manner greatly reduces the total amount of work to
410 recognize actions: The imaganitive process is mostly just matching
411 previous experience, and the recognition process gets to use all
412 the senses to directly describe any action.
414 ** Action recognition is easy with a full gamut of senses
416 Embodied representations using multiple senses such as touch,
417 proprioception, and muscle tension turns out be be exceedingly
418 efficient at describing body-centered actions. It is the ``right
419 language for the job''. For example, it takes only around 5 lines
420 of LISP code to describe the action of ``curling'' using embodied
421 primitives. It takes about 8 lines to describe the seemingly
422 complicated action of wiggling.
424 The following action predicates each take a stream of sensory
425 experience, observe however much of it they desire, and decide
426 whether the worm is doing the action they describe. =curled?=
427 relies on proprioception, =resting?= relies on touch, =wiggling?=
428 relies on a fourier analysis of muscle contraction, and
429 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
431 #+caption: Program for detecting whether the worm is curled. This is the
432 #+caption: simplest action predicate, because it only uses the last frame
433 #+caption: of sensory experience, and only uses proprioceptive data. Even
434 #+caption: this simple predicate, however, is automatically frame
435 #+caption: independent and ignores vermopomorphic differences such as
436 #+caption: worm textures and colors.
437 #+name: curled
438 #+begin_listing clojure
439 #+begin_src clojure
440 (defn curled?
441 "Is the worm curled up?"
442 [experiences]
443 (every?
444 (fn [[_ _ bend]]
445 (> (Math/sin bend) 0.64))
446 (:proprioception (peek experiences))))
447 #+end_src
448 #+end_listing
450 #+caption: Program for summarizing the touch information in a patch
451 #+caption: of skin.
452 #+name: touch-summary
453 #+begin_listing clojure
454 #+begin_src clojure
455 (defn contact
456 "Determine how much contact a particular worm segment has with
457 other objects. Returns a value between 0 and 1, where 1 is full
458 contact and 0 is no contact."
459 [touch-region [coords contact :as touch]]
460 (-> (zipmap coords contact)
461 (select-keys touch-region)
462 (vals)
463 (#(map first %))
464 (average)
465 (* 10)
466 (- 1)
467 (Math/abs)))
468 #+end_src
469 #+end_listing
472 #+caption: Program for detecting whether the worm is at rest. This program
473 #+caption: uses a summary of the tactile information from the underbelly
474 #+caption: of the worm, and is only true if every segment is touching the
475 #+caption: floor. Note that this function contains no references to
476 #+caption: proprioction at all.
477 #+name: resting
478 #+begin_listing clojure
479 #+begin_src clojure
480 (def worm-segment-bottom (rect-region [8 15] [14 22]))
482 (defn resting?
483 "Is the worm resting on the ground?"
484 [experiences]
485 (every?
486 (fn [touch-data]
487 (< 0.9 (contact worm-segment-bottom touch-data)))
488 (:touch (peek experiences))))
489 #+end_src
490 #+end_listing
492 #+caption: Program for detecting whether the worm is curled up into a
493 #+caption: full circle. Here the embodied approach begins to shine, as
494 #+caption: I am able to both use a previous action predicate (=curled?=)
495 #+caption: as well as the direct tactile experience of the head and tail.
496 #+name: grand-circle
497 #+begin_listing clojure
498 #+begin_src clojure
499 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
501 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
503 (defn grand-circle?
504 "Does the worm form a majestic circle (one end touching the other)?"
505 [experiences]
506 (and (curled? experiences)
507 (let [worm-touch (:touch (peek experiences))
508 tail-touch (worm-touch 0)
509 head-touch (worm-touch 4)]
510 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
511 (< 0.55 (contact worm-segment-top-tip head-touch))))))
512 #+end_src
513 #+end_listing
516 #+caption: Program for detecting whether the worm has been wiggling for
517 #+caption: the last few frames. It uses a fourier analysis of the muscle
518 #+caption: contractions of the worm's tail to determine wiggling. This is
519 #+caption: signigicant because there is no particular frame that clearly
520 #+caption: indicates that the worm is wiggling --- only when multiple frames
521 #+caption: are analyzed together is the wiggling revealed. Defining
522 #+caption: wiggling this way also gives the worm an opportunity to learn
523 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
524 #+caption: wiggle but can't. Frustrated wiggling is very visually different
525 #+caption: from actual wiggling, but this definition gives it to us for free.
526 #+name: wiggling
527 #+begin_listing clojure
528 #+begin_src clojure
529 (defn fft [nums]
530 (map
531 #(.getReal %)
532 (.transform
533 (FastFourierTransformer. DftNormalization/STANDARD)
534 (double-array nums) TransformType/FORWARD)))
536 (def indexed (partial map-indexed vector))
538 (defn max-indexed [s]
539 (first (sort-by (comp - second) (indexed s))))
541 (defn wiggling?
542 "Is the worm wiggling?"
543 [experiences]
544 (let [analysis-interval 0x40]
545 (when (> (count experiences) analysis-interval)
546 (let [a-flex 3
547 a-ex 2
548 muscle-activity
549 (map :muscle (vector:last-n experiences analysis-interval))
550 base-activity
551 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
552 (= 2
553 (first
554 (max-indexed
555 (map #(Math/abs %)
556 (take 20 (fft base-activity))))))))))
557 #+end_src
558 #+end_listing
560 With these action predicates, I can now recognize the actions of
561 the worm while it is moving under my control and I have access to
562 all the worm's senses.
564 #+caption: Use the action predicates defined earlier to report on
565 #+caption: what the worm is doing while in simulation.
566 #+name: report-worm-activity
567 #+begin_listing clojure
568 #+begin_src clojure
569 (defn debug-experience
570 [experiences text]
571 (cond
572 (grand-circle? experiences) (.setText text "Grand Circle")
573 (curled? experiences) (.setText text "Curled")
574 (wiggling? experiences) (.setText text "Wiggling")
575 (resting? experiences) (.setText text "Resting")))
576 #+end_src
577 #+end_listing
579 #+caption: Using =debug-experience=, the body-centered predicates
580 #+caption: work together to classify the behaviour of the worm.
581 #+caption: while under manual motor control.
582 #+name: basic-worm-view
583 #+ATTR_LaTeX: :width 10cm
584 [[./images/worm-identify-init.png]]
586 These action predicates satisfy the recognition requirement of an
587 empathic recognition system. There is a lot of power in the
588 simplicity of the action predicates. They describe their actions
589 without getting confused in visual details of the worm. Each one is
590 frame independent, but more than that, they are each indepent of
591 irrelevant visual details of the worm and the environment. They
592 will work regardless of whether the worm is a different color or
593 hevaily textured, or of the environment has strange lighting.
595 The trick now is to make the action predicates work even when the
596 sensory data on which they depend is absent. If I can do that, then
597 I will have gained much,
599 ** \Phi-space describes the worm's experiences
601 As a first step towards building empathy, I need to gather all of
602 the worm's experiences during free play. I use a simple vector to
603 store all the experiences.
605 #+caption: Program to gather the worm's experiences into a vector for
606 #+caption: further processing. The =motor-control-program= line uses
607 #+caption: a motor control script that causes the worm to execute a series
608 #+caption: of ``exercices'' that include all the action predicates.
609 #+name: generate-phi-space
610 #+begin_listing clojure
611 #+begin_src clojure
612 (defn generate-phi-space []
613 (let [experiences (atom [])]
614 (run-world
615 (apply-map
616 worm-world
617 (merge
618 (worm-world-defaults)
619 {:end-frame 700
620 :motor-control
621 (motor-control-program worm-muscle-labels do-all-the-things)
622 :experiences experiences})))
623 @experiences))
624 #+end_src
625 #+end_listing
627 Each element of the experience vector exists in the vast space of
628 all possible worm-experiences. Most of this vast space is actually
629 unreachable due to physical constraints of the worm's body. For
630 example, the worm's segments are connected by hinge joints that put
631 a practical limit on the worm's degrees of freedom. Also, the worm
632 can not be bent into a circle so that its ends are touching and at
633 the same time not also experience the sensation of touching itself.
635 As the worm moves around during free play and the vector grows
636 larger, the vector begins to define a subspace which is all the
637 practical experiences the worm can experience during normal
638 operation, which I call \Phi-space, short for physical-space. The
639 vector defines a path through \Phi-space. This path has interesting
640 properties that all derive from embodiment. The proprioceptive
641 components are completely smooth, because in order for the worm to
642 move from one position to another, it must pass through the
643 intermediate positions. The path invariably forms loops as actions
644 are repeated. Finally and most importantly, proprioception actually
645 gives very strong inference about the other senses. For example,
646 when the worm is flat, you can infer that it is touching the ground
647 and that its muscles are not active, because if the muscles were
648 active, the worm would be moving and would not be perfectly flat.
649 In order to stay flat, the worm has to be touching the ground, or
650 it would again be moving out of the flat position due to gravity.
651 If the worm is positioned in such a way that it interacts with
652 itself, then it is very likely to be feeling the same tactile
653 feelings as the last time it was in that position, because it has
654 the same body as then. If you observe multiple frames of
655 proprioceptive data, then you can become increasingly confident
656 about the exact activations of the worm's muscles, because it
657 generally takes a unique combination of muscle contractions to
658 transform the worm's body along a specific path through \Phi-space.
660 There is a simple way of taking \Phi-space and the total ordering
661 provided by an experience vector and reliably infering the rest of
662 the senses.
664 ** Empathy is the process of tracing though \Phi-space
666 Here is the core of a basic empathy algorithm, starting with an
667 experience vector: First, group the experiences into tiered
668 proprioceptive bins. I use powers of 10 and 3 bins, and the
669 smallest bin has and approximate size of 0.001 radians in all
670 proprioceptive dimensions.
672 Then, given a sequence of proprioceptive input, generate a set of
673 matching experience records for each input.
675 Finally, to infer sensory data, select the longest consective chain
676 of experiences as determined by the indexes into the experience
677 vector.
679 This algorithm has three advantages:
681 1. It's simple
683 3. It's very fast -- both tracing through possibilites and
684 retrieving possible interpretations take essentially constant
685 time.
687 2. It protects from wrong interpretations of transient ambiguous
688 proprioceptive data : for example, if the worm is flat for just
689 an instant, this flattness will not be interpreted as implying
690 that the worm has its muscles relaxed, since the flattness is
691 part of a longer chain which includes a distinct pattern of
692 muscle activation. A memoryless statistical model such as a
693 markov model that operates on individual frames may very well
694 make this mistake.
696 #+caption: Program to convert an experience vector into a
697 #+caption: proprioceptively binned lookup function.
698 #+name: bin
699 #+begin_listing clojure
700 #+begin_src clojure
701 (defn bin [digits]
702 (fn [angles]
703 (->> angles
704 (flatten)
705 (map (juxt #(Math/sin %) #(Math/cos %)))
706 (flatten)
707 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
709 (defn gen-phi-scan
710 "Nearest-neighbors with binning. Only returns a result if
711 the propriceptive data is within 10% of a previously recorded
712 result in all dimensions."
713 [phi-space]
714 (let [bin-keys (map bin [3 2 1])
715 bin-maps
716 (map (fn [bin-key]
717 (group-by
718 (comp bin-key :proprioception phi-space)
719 (range (count phi-space)))) bin-keys)
720 lookups (map (fn [bin-key bin-map]
721 (fn [proprio] (bin-map (bin-key proprio))))
722 bin-keys bin-maps)]
723 (fn lookup [proprio-data]
724 (set (some #(% proprio-data) lookups)))))
725 #+end_src
726 #+end_listing
729 #+caption: Program to calculate empathy by tracing though \Phi-space
730 #+caption: and finding the longest (ie. most coherent) interpretation
731 #+caption: of the data.
732 #+name: longest-thread
733 #+begin_listing clojure
734 #+begin_src clojure
735 (defn longest-thread
736 "Find the longest thread from phi-index-sets. The index sets should
737 be ordered from most recent to least recent."
738 [phi-index-sets]
739 (loop [result '()
740 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
741 (if (empty? phi-index-sets)
742 (vec result)
743 (let [threads
744 (for [thread-base thread-bases]
745 (loop [thread (list thread-base)
746 remaining remaining]
747 (let [next-index (dec (first thread))]
748 (cond (empty? remaining) thread
749 (contains? (first remaining) next-index)
750 (recur
751 (cons next-index thread) (rest remaining))
752 :else thread))))
753 longest-thread
754 (reduce (fn [thread-a thread-b]
755 (if (> (count thread-a) (count thread-b))
756 thread-a thread-b))
757 '(nil)
758 threads)]
759 (recur (concat longest-thread result)
760 (drop (count longest-thread) phi-index-sets))))))
761 #+end_src
762 #+end_listing
765 There is one final piece, which is to replace missing sensory data
766 with a best-guess estimate. While I could fill in missing data by
767 using a gradient over the closest known sensory data points, averages
768 can be misleading. It is certainly possible to create an impossible
769 sensory state by averaging two possible sensory states. Therefore, I
770 simply replicate the most recent sensory experience to fill in the
771 gaps.
773 #+caption: Fill in blanks in sensory experience by replicating the most
774 #+caption: recent experience.
775 #+name: infer-nils
776 #+begin_listing clojure
777 #+begin_src clojure
778 (defn infer-nils
779 "Replace nils with the next available non-nil element in the
780 sequence, or barring that, 0."
781 [s]
782 (loop [i (dec (count s))
783 v (transient s)]
784 (if (zero? i) (persistent! v)
785 (if-let [cur (v i)]
786 (if (get v (dec i) 0)
787 (recur (dec i) v)
788 (recur (dec i) (assoc! v (dec i) cur)))
789 (recur i (assoc! v i 0))))))
790 #+end_src
791 #+end_listing
794 ** Efficient action recognition with =EMPATH=
796 In my exploration with the worm, I can generally infer actions from
797 proprioceptive data exactly as well as when I have the complete
798 sensory data. To reach this level, I have to train the worm with
799 verious exercices for about 1 minute.
801 ** Digression: bootstrapping touch using free exploration
803 * Contributions
808 # An anatomical joke:
809 # - Training
810 # - Skeletal imitation
811 # - Sensory fleshing-out
812 # - Classification