view thesis/cortex.org @ 467:ade64947d2bf

s for work at MIT.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 15:30:23 -0400
parents da311eefbb09
children 258078f78b33
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+begin_src clojure
16 #+end_src
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
28 * COMMENT Empathy and Embodiment as problem solving strategies
30 By the end of this thesis, you will have seen a novel approach to
31 interpreting video using embodiment and empathy. You will have also
32 seen one way to efficiently implement empathy for embodied
33 creatures. Finally, you will become familiar with =CORTEX=, a system
34 for designing and simulating creatures with rich senses, which you
35 may choose to use in your own research.
37 This is the core vision of my thesis: That one of the important ways
38 in which we understand others is by imagining ourselves in their
39 position and emphatically feeling experiences relative to our own
40 bodies. By understanding events in terms of our own previous
41 corporeal experience, we greatly constrain the possibilities of what
42 would otherwise be an unwieldy exponential search. This extra
43 constraint can be the difference between easily understanding what
44 is happening in a video and being completely lost in a sea of
45 incomprehensible color and movement.
47 ** Recognizing actions in video is extremely difficult
49 Consider for example the problem of determining what is happening
50 in a video of which this is one frame:
52 #+caption: A cat drinking some water. Identifying this action is
53 #+caption: beyond the state of the art for computers.
54 #+ATTR_LaTeX: :width 7cm
55 [[./images/cat-drinking.jpg]]
57 It is currently impossible for any computer program to reliably
58 label such a video as ``drinking''. And rightly so -- it is a very
59 hard problem! What features can you describe in terms of low level
60 functions of pixels that can even begin to describe at a high level
61 what is happening here?
63 Or suppose that you are building a program that recognizes chairs.
64 How could you ``see'' the chair in figure \ref{hidden-chair}?
66 #+caption: The chair in this image is quite obvious to humans, but I
67 #+caption: doubt that any modern computer vision program can find it.
68 #+name: hidden-chair
69 #+ATTR_LaTeX: :width 10cm
70 [[./images/fat-person-sitting-at-desk.jpg]]
72 Finally, how is it that you can easily tell the difference between
73 how the girls /muscles/ are working in figure \ref{girl}?
75 #+caption: The mysterious ``common sense'' appears here as you are able
76 #+caption: to discern the difference in how the girl's arm muscles
77 #+caption: are activated between the two images.
78 #+name: girl
79 #+ATTR_LaTeX: :width 7cm
80 [[./images/wall-push.png]]
82 Each of these examples tells us something about what might be going
83 on in our minds as we easily solve these recognition problems.
85 The hidden chairs show us that we are strongly triggered by cues
86 relating to the position of human bodies, and that we can determine
87 the overall physical configuration of a human body even if much of
88 that body is occluded.
90 The picture of the girl pushing against the wall tells us that we
91 have common sense knowledge about the kinetics of our own bodies.
92 We know well how our muscles would have to work to maintain us in
93 most positions, and we can easily project this self-knowledge to
94 imagined positions triggered by images of the human body.
96 ** =EMPATH= neatly solves recognition problems
98 I propose a system that can express the types of recognition
99 problems above in a form amenable to computation. It is split into
100 four parts:
102 - Free/Guided Play :: The creature moves around and experiences the
103 world through its unique perspective. Many otherwise
104 complicated actions are easily described in the language of a
105 full suite of body-centered, rich senses. For example,
106 drinking is the feeling of water sliding down your throat, and
107 cooling your insides. It's often accompanied by bringing your
108 hand close to your face, or bringing your face close to water.
109 Sitting down is the feeling of bending your knees, activating
110 your quadriceps, then feeling a surface with your bottom and
111 relaxing your legs. These body-centered action descriptions
112 can be either learned or hard coded.
113 - Posture Imitation :: When trying to interpret a video or image,
114 the creature takes a model of itself and aligns it with
115 whatever it sees. This alignment can even cross species, as
116 when humans try to align themselves with things like ponies,
117 dogs, or other humans with a different body type.
118 - Empathy :: The alignment triggers associations with
119 sensory data from prior experiences. For example, the
120 alignment itself easily maps to proprioceptive data. Any
121 sounds or obvious skin contact in the video can to a lesser
122 extent trigger previous experience. Segments of previous
123 experiences are stitched together to form a coherent and
124 complete sensory portrait of the scene.
125 - Recognition :: With the scene described in terms of first
126 person sensory events, the creature can now run its
127 action-identification programs on this synthesized sensory
128 data, just as it would if it were actually experiencing the
129 scene first-hand. If previous experience has been accurately
130 retrieved, and if it is analogous enough to the scene, then
131 the creature will correctly identify the action in the scene.
133 For example, I think humans are able to label the cat video as
134 ``drinking'' because they imagine /themselves/ as the cat, and
135 imagine putting their face up against a stream of water and
136 sticking out their tongue. In that imagined world, they can feel
137 the cool water hitting their tongue, and feel the water entering
138 their body, and are able to recognize that /feeling/ as drinking.
139 So, the label of the action is not really in the pixels of the
140 image, but is found clearly in a simulation inspired by those
141 pixels. An imaginative system, having been trained on drinking and
142 non-drinking examples and learning that the most important
143 component of drinking is the feeling of water sliding down one's
144 throat, would analyze a video of a cat drinking in the following
145 manner:
147 1. Create a physical model of the video by putting a ``fuzzy''
148 model of its own body in place of the cat. Possibly also create
149 a simulation of the stream of water.
151 2. Play out this simulated scene and generate imagined sensory
152 experience. This will include relevant muscle contractions, a
153 close up view of the stream from the cat's perspective, and most
154 importantly, the imagined feeling of water entering the
155 mouth. The imagined sensory experience can come from a
156 simulation of the event, but can also be pattern-matched from
157 previous, similar embodied experience.
159 3. The action is now easily identified as drinking by the sense of
160 taste alone. The other senses (such as the tongue moving in and
161 out) help to give plausibility to the simulated action. Note that
162 the sense of vision, while critical in creating the simulation,
163 is not critical for identifying the action from the simulation.
165 For the chair examples, the process is even easier:
167 1. Align a model of your body to the person in the image.
169 2. Generate proprioceptive sensory data from this alignment.
171 3. Use the imagined proprioceptive data as a key to lookup related
172 sensory experience associated with that particular proproceptive
173 feeling.
175 4. Retrieve the feeling of your bottom resting on a surface, your
176 knees bent, and your leg muscles relaxed.
178 5. This sensory information is consistent with the =sitting?=
179 sensory predicate, so you (and the entity in the image) must be
180 sitting.
182 6. There must be a chair-like object since you are sitting.
184 Empathy offers yet another alternative to the age-old AI
185 representation question: ``What is a chair?'' --- A chair is the
186 feeling of sitting.
188 My program, =EMPATH= uses this empathic problem solving technique
189 to interpret the actions of a simple, worm-like creature.
191 #+caption: The worm performs many actions during free play such as
192 #+caption: curling, wiggling, and resting.
193 #+name: worm-intro
194 #+ATTR_LaTeX: :width 15cm
195 [[./images/worm-intro-white.png]]
197 #+caption: =EMPATH= recognized and classified each of these
198 #+caption: poses by inferring the complete sensory experience
199 #+caption: from proprioceptive data.
200 #+name: worm-recognition-intro
201 #+ATTR_LaTeX: :width 15cm
202 [[./images/worm-poses.png]]
204 One powerful advantage of empathic problem solving is that it
205 factors the action recognition problem into two easier problems. To
206 use empathy, you need an /aligner/, which takes the video and a
207 model of your body, and aligns the model with the video. Then, you
208 need a /recognizer/, which uses the aligned model to interpret the
209 action. The power in this method lies in the fact that you describe
210 all actions form a body-centered viewpoint. You are less tied to
211 the particulars of any visual representation of the actions. If you
212 teach the system what ``running'' is, and you have a good enough
213 aligner, the system will from then on be able to recognize running
214 from any point of view, even strange points of view like above or
215 underneath the runner. This is in contrast to action recognition
216 schemes that try to identify actions using a non-embodied approach.
217 If these systems learn about running as viewed from the side, they
218 will not automatically be able to recognize running from any other
219 viewpoint.
221 Another powerful advantage is that using the language of multiple
222 body-centered rich senses to describe body-centerd actions offers a
223 massive boost in descriptive capability. Consider how difficult it
224 would be to compose a set of HOG filters to describe the action of
225 a simple worm-creature ``curling'' so that its head touches its
226 tail, and then behold the simplicity of describing thus action in a
227 language designed for the task (listing \ref{grand-circle-intro}):
229 #+caption: Body-centerd actions are best expressed in a body-centered
230 #+caption: language. This code detects when the worm has curled into a
231 #+caption: full circle. Imagine how you would replicate this functionality
232 #+caption: using low-level pixel features such as HOG filters!
233 #+name: grand-circle-intro
234 #+attr_latex: [htpb]
235 #+begin_listing clojure
236 #+begin_src clojure
237 (defn grand-circle?
238 "Does the worm form a majestic circle (one end touching the other)?"
239 [experiences]
240 (and (curled? experiences)
241 (let [worm-touch (:touch (peek experiences))
242 tail-touch (worm-touch 0)
243 head-touch (worm-touch 4)]
244 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
245 (< 0.2 (contact worm-segment-top-tip head-touch))))))
246 #+end_src
247 #+end_listing
250 ** =CORTEX= is a toolkit for building sensate creatures
252 I built =CORTEX= to be a general AI research platform for doing
253 experiments involving multiple rich senses and a wide variety and
254 number of creatures. I intend it to be useful as a library for many
255 more projects than just this thesis. =CORTEX= was necessary to meet
256 a need among AI researchers at CSAIL and beyond, which is that
257 people often will invent neat ideas that are best expressed in the
258 language of creatures and senses, but in order to explore those
259 ideas they must first build a platform in which they can create
260 simulated creatures with rich senses! There are many ideas that
261 would be simple to execute (such as =EMPATH=), but attached to them
262 is the multi-month effort to make a good creature simulator. Often,
263 that initial investment of time proves to be too much, and the
264 project must make do with a lesser environment.
266 =CORTEX= is well suited as an environment for embodied AI research
267 for three reasons:
269 - You can create new creatures using Blender, a popular 3D modeling
270 program. Each sense can be specified using special blender nodes
271 with biologically inspired paramaters. You need not write any
272 code to create a creature, and can use a wide library of
273 pre-existing blender models as a base for your own creatures.
275 - =CORTEX= implements a wide variety of senses, including touch,
276 proprioception, vision, hearing, and muscle tension. Complicated
277 senses like touch, and vision involve multiple sensory elements
278 embedded in a 2D surface. You have complete control over the
279 distribution of these sensor elements through the use of simple
280 png image files. In particular, =CORTEX= implements more
281 comprehensive hearing than any other creature simulation system
282 available.
284 - =CORTEX= supports any number of creatures and any number of
285 senses. Time in =CORTEX= dialates so that the simulated creatures
286 always precieve a perfectly smooth flow of time, regardless of
287 the actual computational load.
289 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
290 engine designed to create cross-platform 3D desktop games. =CORTEX=
291 is mainly written in clojure, a dialect of =LISP= that runs on the
292 java virtual machine (JVM). The API for creating and simulating
293 creatures and senses is entirely expressed in clojure, though many
294 senses are implemented at the layer of jMonkeyEngine or below. For
295 example, for the sense of hearing I use a layer of clojure code on
296 top of a layer of java JNI bindings that drive a layer of =C++=
297 code which implements a modified version of =OpenAL= to support
298 multiple listeners. =CORTEX= is the only simulation environment
299 that I know of that can support multiple entities that can each
300 hear the world from their own perspective. Other senses also
301 require a small layer of Java code. =CORTEX= also uses =bullet=, a
302 physics simulator written in =C=.
304 #+caption: Here is the worm from above modeled in Blender, a free
305 #+caption: 3D-modeling program. Senses and joints are described
306 #+caption: using special nodes in Blender.
307 #+name: worm-recognition-intro
308 #+ATTR_LaTeX: :width 12cm
309 [[./images/blender-worm.png]]
311 Here are some thing I anticipate that =CORTEX= might be used for:
313 - exploring new ideas about sensory integration
314 - distributed communication among swarm creatures
315 - self-learning using free exploration,
316 - evolutionary algorithms involving creature construction
317 - exploration of exoitic senses and effectors that are not possible
318 in the real world (such as telekenisis or a semantic sense)
319 - imagination using subworlds
321 During one test with =CORTEX=, I created 3,000 creatures each with
322 their own independent senses and ran them all at only 1/80 real
323 time. In another test, I created a detailed model of my own hand,
324 equipped with a realistic distribution of touch (more sensitive at
325 the fingertips), as well as eyes and ears, and it ran at around 1/4
326 real time.
328 #+BEGIN_LaTeX
329 \begin{sidewaysfigure}
330 \includegraphics[width=9.5in]{images/full-hand.png}
331 \caption{
332 I modeled my own right hand in Blender and rigged it with all the
333 senses that {\tt CORTEX} supports. My simulated hand has a
334 biologically inspired distribution of touch sensors. The senses are
335 displayed on the right, and the simulation is displayed on the
336 left. Notice that my hand is curling its fingers, that it can see
337 its own finger from the eye in its palm, and that it can feel its
338 own thumb touching its palm.}
339 \end{sidewaysfigure}
340 #+END_LaTeX
342 ** Contributions
344 - I built =CORTEX=, a comprehensive platform for embodied AI
345 experiments. =CORTEX= supports many features lacking in other
346 systems, such proper simulation of hearing. It is easy to create
347 new =CORTEX= creatures using Blender, a free 3D modeling program.
349 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
350 a worm-like creature using a computational model of empathy.
352 * Building =CORTEX=
354 I intend for =CORTEX= to be used as a general purpose library for
355 building creatures and outfitting them with senses, so that it will
356 be useful for other researchers who want to test out ideas of their
357 own. To this end, wherver I have had to make archetictural choices
358 about =CORTEX=, I have chosen to give as much freedom to the user as
359 possible, so that =CORTEX= may be used for things I have not
360 forseen.
362 ** COMMENT Simulation or Reality?
364 The most important archetictural decision of all is the choice to
365 use a computer-simulated environemnt in the first place! The world
366 is a vast and rich place, and for now simulations are a very poor
367 reflection of its complexity. It may be that there is a significant
368 qualatative difference between dealing with senses in the real
369 world and dealing with pale facilimilies of them in a
370 simulation. What are the advantages and disadvantages of a
371 simulation vs. reality?
373 *** Simulation
375 The advantages of virtual reality are that when everything is a
376 simulation, experiments in that simulation are absolutely
377 reproducible. It's also easier to change the character and world
378 to explore new situations and different sensory combinations.
380 If the world is to be simulated on a computer, then not only do
381 you have to worry about whether the character's senses are rich
382 enough to learn from the world, but whether the world itself is
383 rendered with enough detail and realism to give enough working
384 material to the character's senses. To name just a few
385 difficulties facing modern physics simulators: destructibility of
386 the environment, simulation of water/other fluids, large areas,
387 nonrigid bodies, lots of objects, smoke. I don't know of any
388 computer simulation that would allow a character to take a rock
389 and grind it into fine dust, then use that dust to make a clay
390 sculpture, at least not without spending years calculating the
391 interactions of every single small grain of dust. Maybe a
392 simulated world with today's limitations doesn't provide enough
393 richness for real intelligence to evolve.
395 *** Reality
397 The other approach for playing with senses is to hook your
398 software up to real cameras, microphones, robots, etc., and let it
399 loose in the real world. This has the advantage of eliminating
400 concerns about simulating the world at the expense of increasing
401 the complexity of implementing the senses. Instead of just
402 grabbing the current rendered frame for processing, you have to
403 use an actual camera with real lenses and interact with photons to
404 get an image. It is much harder to change the character, which is
405 now partly a physical robot of some sort, since doing so involves
406 changing things around in the real world instead of modifying
407 lines of code. While the real world is very rich and definitely
408 provides enough stimulation for intelligence to develop as
409 evidenced by our own existence, it is also uncontrollable in the
410 sense that a particular situation cannot be recreated perfectly or
411 saved for later use. It is harder to conduct science because it is
412 harder to repeat an experiment. The worst thing about using the
413 real world instead of a simulation is the matter of time. Instead
414 of simulated time you get the constant and unstoppable flow of
415 real time. This severely limits the sorts of software you can use
416 to program the AI because all sense inputs must be handled in real
417 time. Complicated ideas may have to be implemented in hardware or
418 may simply be impossible given the current speed of our
419 processors. Contrast this with a simulation, in which the flow of
420 time in the simulated world can be slowed down to accommodate the
421 limitations of the character's programming. In terms of cost,
422 doing everything in software is far cheaper than building custom
423 real-time hardware. All you need is a laptop and some patience.
425 ** COMMENT Because of Time, simulation is perferable to reality
427 I envision =CORTEX= being used to support rapid prototyping and
428 iteration of ideas. Even if I could put together a well constructed
429 kit for creating robots, it would still not be enough because of
430 the scourge of real-time processing. Anyone who wants to test their
431 ideas in the real world must always worry about getting their
432 algorithms to run fast enough to process information in real time.
433 The need for real time processing only increases if multiple senses
434 are involved. In the extreme case, even simple algorithms will have
435 to be accelerated by ASIC chips or FPGAs, turning what would
436 otherwise be a few lines of code and a 10x speed penality into a
437 multi-month ordeal. For this reason, =CORTEX= supports
438 /time-dialiation/, which scales back the framerate of the
439 simulation in proportion to the amount of processing each frame.
440 From the perspective of the creatures inside the simulation, time
441 always appears to flow at a constant rate, regardless of how
442 complicated the envorimnent becomes or how many creatures are in
443 the simulation. The cost is that =CORTEX= can sometimes run slower
444 than real time. This can also be an advantage, however ---
445 simulations of very simple creatures in =CORTEX= generally run at
446 40x on my machine!
448 ** COMMENT Video game engines are a great starting point
450 I did not need to write my own physics simulation code or shader to
451 build =CORTEX=. Doing so would lead to a system that is impossible
452 for anyone but myself to use anyway. Instead, I use a video game
453 engine as a base and modify it to accomodate the additional needs
454 of =CORTEX=. Video game engines are an ideal starting point to
455 build =CORTEX=, because they are not far from being creature
456 building systems themselves.
458 First off, general purpose video game engines come with a physics
459 engine and lighting / sound system. The physics system provides
460 tools that can be co-opted to serve as touch, proprioception, and
461 muscles. Since some games support split screen views, a good video
462 game engine will allow you to efficiently create multiple cameras
463 in the simulated world that can be used as eyes. Video game systems
464 offer integrated asset management for things like textures and
465 creatures models, providing an avenue for defining creatures.
466 Finally, because video game engines support a large number of
467 users, if I don't stray too far from the base system, other
468 researchers can turn to this community for help when doing their
469 research.
471 ** COMMENT =CORTEX= is based on jMonkeyEngine3
473 While preparing to build =CORTEX= I studied several video game
474 engines to see which would best serve as a base. The top contenders
475 were:
477 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
478 software in 1997. All the source code was released by ID
479 software into the Public Domain several years ago, and as a
480 result it has been ported to many different languages. This
481 engine was famous for its advanced use of realistic shading
482 and had decent and fast physics simulation. The main advantage
483 of the Quake II engine is its simplicity, but I ultimately
484 rejected it because the engine is too tied to the concept of a
485 first-person shooter game. One of the problems I had was that
486 there does not seem to be any easy way to attach multiple
487 cameras to a single character. There are also several physics
488 clipping issues that are corrected in a way that only applies
489 to the main character and do not apply to arbitrary objects.
491 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
492 and Quake I engines and is used by Valve in the Half-Life
493 series of games. The physics simulation in the Source Engine
494 is quite accurate and probably the best out of all the engines
495 I investigated. There is also an extensive community actively
496 working with the engine. However, applications that use the
497 Source Engine must be written in C++, the code is not open, it
498 only runs on Windows, and the tools that come with the SDK to
499 handle models and textures are complicated and awkward to use.
501 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
502 games in Java. It uses OpenGL to render to the screen and uses
503 screengraphs to avoid drawing things that do not appear on the
504 screen. It has an active community and several games in the
505 pipeline. The engine was not built to serve any particular
506 game but is instead meant to be used for any 3D game.
508 I chose jMonkeyEngine3 because it because it had the most features
509 out of all the free projects I looked at, and because I could then
510 write my code in clojure, an implementation of =LISP= that runs on
511 the JVM.
513 ** COMMENT Bodies are composed of segments connected by joints
515 For the simple worm-like creatures I will use later on in this
516 thesis, I could define a simple API in =CORTEX= that would allow
517 one to create boxes, spheres, etc., and leave that API as the sole
518 way to create creatures. However, for =CORTEX= to truly be useful
519 for other projects, it needs to have a way to construct complicated
520 creatures. If possible, it would be nice to leverage work that has
521 already been done by the community of 3D modelers, or at least
522 enable people who are talented at moedling but not programming to
523 design =CORTEX= creatures.
525 Therefore, I use Blender, a free 3D modeling program, as the main
526 way to create creatures in =CORTEX=. However, the creatures modeled
527 in Blender must also be simple to simulate in jMonkeyEngine3's game
528 engine, and must also be easy to rig with =CORTEX='s senses.
530 While trying to find a good compromise for body-design, one option
531 I ultimately rejected is to use blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea
532 would have been to define a mesh which describes the creature's
533 entire body. To this you add a skeleton which deforms this mesh
534 (called rigging). This technique is used extensively to model
535 humans and create realistic animations. It is not a good technique
536 for physical simulation, because deformable surfaces are hard to
537 model. Humans work like a squishy bag with some hard bones to give
538 it shape. The bones are easy to simulate physically, but they
539 interact with thr world though the skin, which is contiguous, but
540 does not have a constant shape. In order to simulate skin you need
541 some way to continuously update the physical model of the skin
542 along with the movement of the bones. Given that bullet is
543 optimized for rigid, solid objects, this leads to unmanagable
544 computation and incorrect simulation.
546 Instead of using the human-like ``deformable bag of bones''
547 approach, I decided to base my body plans on multiple solid objects
548 that are connected by joints, inspired by the robot =EVE= from the
549 movie WALL-E.
551 #+caption: =EVE= from the movie WALL-E. This body plan turns
552 #+caption: out to be much better suited to my purposes than a more
553 #+caption: human-like one.
554 #+ATTR_LaTeX: :width 10cm
555 [[./images/Eve.jpg]]
557 =EVE='s body is composed of several rigid components that are held
558 together by invisible joint constraints. This is what I mean by
559 ``eve-like''. The main reason that I use eve-style bodies is for
560 efficiency, and so that there will be correspondence between the
561 AI's vision and the physical presence of its body. Each individual
562 section is simulated by a separate rigid body that corresponds
563 exactly with its visual representation and does not change.
564 Sections are connected by invisible joints that are well supported
565 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
566 can efficiently simulate hundreds of rigid bodies connected by
567 joints. Sections do not have to stay as one piece forever; they can
568 be dynamically replaced with multiple sections to simulate
569 splitting in two. This could be used to simulate retractable claws
570 or =EVE='s hands, which are able to coalesce into one object in the
571 movie.
573 *** Solidifying/Connecting the body
575 Importing bodies from =CORTEX= into blender involves encoding
576 metadata into the blender file that specifies the mass of each
577 component and the joints by which those components are connected. I
578 do this in Blender in two ways. First is by using the ``metadata''
579 field of each solid object to specify the mass. Second is by using
580 Blender ``empty nodes'' to specify the position and type of each
581 joint. Empty nodes have no mass, physical presence, or appearance,
582 but they can hold metadata and have names. I use a tree structure
583 of empty nodes to specify joints. There is a parent node named
584 ``joints'', and a series of empty child nodes of the ``joints''
585 node that each represent a single joint.
587 #+caption: View of the hand model in Blender showing the main ``joints''
588 #+caption: node (highlighted in yellow) and its children which each
589 #+caption: represent a joint in the hand. Each joint node has metadata
590 #+caption: specifying what sort of joint it is.
591 #+name: blender-hand
592 #+ATTR_LaTeX: :width 10cm
593 [[./images/hand-screenshot1.png]]
596 =CORTEX= creates a creature in two steps: first, it traverses the
597 nodes in the blender file and creates physical representations for
598 any of them that have mass defined.
600 #+caption: Program for iterating through the nodes in a blender file
601 #+caption: and generating physical jMonkeyEngine3 objects with mass
602 #+caption: and a matching physics shape.
603 #+name: name
604 #+begin_listing clojure
605 #+begin_src clojure
606 (defn physical!
607 "Iterate through the nodes in creature and make them real physical
608 objects in the simulation."
609 [#^Node creature]
610 (dorun
611 (map
612 (fn [geom]
613 (let [physics-control
614 (RigidBodyControl.
615 (HullCollisionShape.
616 (.getMesh geom))
617 (if-let [mass (meta-data geom "mass")]
618 (float mass) (float 1)))]
619 (.addControl geom physics-control)))
620 (filter #(isa? (class %) Geometry )
621 (node-seq creature)))))
622 #+end_src
623 #+end_listing
625 The next step to making a proper body is to connect those pieces
626 together with joints. jMonkeyEngine has a large array of joints
627 available via =bullet=, such as Point2Point, Cone, Hinge, and a
628 generic Six Degree of Freedom joint, with or without spring
629 restitution. =CORTEX='s procedure for binding the creature together
630 with joints is as follows:
632 - Find the children of the "joints" node.
633 - Determine the two spatials the joint is meant to connect.
634 - Create the joint based on the meta-data of the empty node.
636 The higher order function =sense-nodes= from =cortex.sense=
637 simplifies finding the joints based on their parent ``joints''
638 node.
640 #+caption: Retrieving the children empty nodes from a single
641 #+caption: named empty node is a common pattern in =CORTEX=
642 #+caption: further instances of this technique for the senses
643 #+caption: will be omitted
644 #+name: get-empty-nodes
645 #+begin_listing clojure
646 #+begin_src clojure
647 (defn sense-nodes
648 "For some senses there is a special empty blender node whose
649 children are considered markers for an instance of that sense. This
650 function generates functions to find those children, given the name
651 of the special parent node."
652 [parent-name]
653 (fn [#^Node creature]
654 (if-let [sense-node (.getChild creature parent-name)]
655 (seq (.getChildren sense-node)) [])))
657 (def
658 ^{:doc "Return the children of the creature's \"joints\" node."
659 :arglists '([creature])}
660 joints
661 (sense-nodes "joints"))
662 #+end_src
663 #+end_listing
665 To find a joint's targets targets, =CORTEX= creates a small cube,
666 centered around the empty-node, and grows the cube exponentially
667 until it intersects two /physical/ objects. The objects are ordered
668 according to the joint's rotation, with the first one being the
669 object that has more negative coordinates in the joint's reference
670 frame. Since the objects must be physical, the empty-node itself
671 escapes detection. Because the objects must be physical,
672 =joint-targets= must be called /after/ =physical!= is called.
674 #+caption: Program to find the targets of a joint node by
675 #+caption: exponentiallly growth of a search cube.
676 #+name: joint-targets
677 #+begin_listing clojure
678 #+begin_src clojure
679 (defn joint-targets
680 "Return the two closest two objects to the joint object, ordered
681 from bottom to top according to the joint's rotation."
682 [#^Node parts #^Node joint]
683 (loop [radius (float 0.01)]
684 (let [results (CollisionResults.)]
685 (.collideWith
686 parts
687 (BoundingBox. (.getWorldTranslation joint)
688 radius radius radius) results)
689 (let [targets
690 (distinct
691 (map #(.getGeometry %) results))]
692 (if (>= (count targets) 2)
693 (sort-by
694 #(let [joint-ref-frame-position
695 (jme-to-blender
696 (.mult
697 (.inverse (.getWorldRotation joint))
698 (.subtract (.getWorldTranslation %)
699 (.getWorldTranslation joint))))]
700 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
701 (take 2 targets))
702 (recur (float (* radius 2))))))))
703 #+end_src
704 #+end_listing
706 Once =CORTEX= finds all joints and targets, it creates them using a
707 simple dispatch on the metadata of the joint node.
709 #+caption: Program to dispatch on blender metadata and create joints
710 #+caption: sutiable for physical simulation.
711 #+name: joint-dispatch
712 #+begin_listing clojure
713 #+begin_src clojure
714 (defmulti joint-dispatch
715 "Translate blender pseudo-joints into real JME joints."
716 (fn [constraints & _]
717 (:type constraints)))
719 (defmethod joint-dispatch :point
720 [constraints control-a control-b pivot-a pivot-b rotation]
721 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
722 (.setLinearLowerLimit Vector3f/ZERO)
723 (.setLinearUpperLimit Vector3f/ZERO)))
725 (defmethod joint-dispatch :hinge
726 [constraints control-a control-b pivot-a pivot-b rotation]
727 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
728 [limit-1 limit-2] (:limit constraints)
729 hinge-axis (.mult rotation (blender-to-jme axis))]
730 (doto (HingeJoint. control-a control-b pivot-a pivot-b
731 hinge-axis hinge-axis)
732 (.setLimit limit-1 limit-2))))
734 (defmethod joint-dispatch :cone
735 [constraints control-a control-b pivot-a pivot-b rotation]
736 (let [limit-xz (:limit-xz constraints)
737 limit-xy (:limit-xy constraints)
738 twist (:twist constraints)]
739 (doto (ConeJoint. control-a control-b pivot-a pivot-b
740 rotation rotation)
741 (.setLimit (float limit-xz) (float limit-xy)
742 (float twist)))))
743 #+end_src
744 #+end_listing
746 All that is left for joints it to combine the above pieces into a
747 something that can operate on the collection of nodes that a
748 blender file represents.
750 #+caption: Program to completely create a joint given information
751 #+caption: from a blender file.
752 #+name: connect
753 #+begin_listing clojure
754 #+begin_src clojure
755 (defn connect
756 "Create a joint between 'obj-a and 'obj-b at the location of
757 'joint. The type of joint is determined by the metadata on 'joint.
759 Here are some examples:
760 {:type :point}
761 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
762 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
764 {:type :cone :limit-xz 0]
765 :limit-xy 0]
766 :twist 0]} (use XZY rotation mode in blender!)"
767 [#^Node obj-a #^Node obj-b #^Node joint]
768 (let [control-a (.getControl obj-a RigidBodyControl)
769 control-b (.getControl obj-b RigidBodyControl)
770 joint-center (.getWorldTranslation joint)
771 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
772 pivot-a (world-to-local obj-a joint-center)
773 pivot-b (world-to-local obj-b joint-center)]
774 (if-let
775 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
776 ;; A side-effect of creating a joint registers
777 ;; it with both physics objects which in turn
778 ;; will register the joint with the physics system
779 ;; when the simulation is started.
780 (joint-dispatch constraints
781 control-a control-b
782 pivot-a pivot-b
783 joint-rotation))))
784 #+end_src
785 #+end_listing
787 In general, whenever =CORTEX= exposes a sense (or in this case
788 physicality), it provides a function of the type =sense!=, which
789 takes in a collection of nodes and augments it to support that
790 sense. The function returns any controlls necessary to use that
791 sense. In this case =body!= cerates a physical body and returns no
792 control functions.
794 #+caption: Program to give joints to a creature.
795 #+name: name
796 #+begin_listing clojure
797 #+begin_src clojure
798 (defn joints!
799 "Connect the solid parts of the creature with physical joints. The
800 joints are taken from the \"joints\" node in the creature."
801 [#^Node creature]
802 (dorun
803 (map
804 (fn [joint]
805 (let [[obj-a obj-b] (joint-targets creature joint)]
806 (connect obj-a obj-b joint)))
807 (joints creature))))
808 (defn body!
809 "Endow the creature with a physical body connected with joints. The
810 particulars of the joints and the masses of each body part are
811 determined in blender."
812 [#^Node creature]
813 (physical! creature)
814 (joints! creature))
815 #+end_src
816 #+end_listing
818 All of the code you have just seen amounts to only 130 lines, yet
819 because it builds on top of Blender and jMonkeyEngine3, those few
820 lines pack quite a punch!
822 The hand from figure \ref{blender-hand}, which was modeled after my
823 own right hand, can now be given joints and simulated as a
824 creature.
826 #+caption: With the ability to create physical creatures from blender,
827 #+caption: =CORTEX= gets one step closer to a full creature simulation
828 #+caption: environment.
829 #+name: name
830 #+ATTR_LaTeX: :width 15cm
831 [[./images/physical-hand.png]]
834 ** Eyes reuse standard video game components
836 ** Hearing is hard; =CORTEX= does it right
838 ** Touch uses hundreds of hair-like elements
840 ** Proprioception is the sense that makes everything ``real''
842 ** Muscles are both effectors and sensors
844 ** =CORTEX= brings complex creatures to life!
846 ** =CORTEX= enables many possiblities for further research
848 * COMMENT Empathy in a simulated worm
850 Here I develop a computational model of empathy, using =CORTEX= as a
851 base. Empathy in this context is the ability to observe another
852 creature and infer what sorts of sensations that creature is
853 feeling. My empathy algorithm involves multiple phases. First is
854 free-play, where the creature moves around and gains sensory
855 experience. From this experience I construct a representation of the
856 creature's sensory state space, which I call \Phi-space. Using
857 \Phi-space, I construct an efficient function which takes the
858 limited data that comes from observing another creature and enriches
859 it full compliment of imagined sensory data. I can then use the
860 imagined sensory data to recognize what the observed creature is
861 doing and feeling, using straightforward embodied action predicates.
862 This is all demonstrated with using a simple worm-like creature, and
863 recognizing worm-actions based on limited data.
865 #+caption: Here is the worm with which we will be working.
866 #+caption: It is composed of 5 segments. Each segment has a
867 #+caption: pair of extensor and flexor muscles. Each of the
868 #+caption: worm's four joints is a hinge joint which allows
869 #+caption: about 30 degrees of rotation to either side. Each segment
870 #+caption: of the worm is touch-capable and has a uniform
871 #+caption: distribution of touch sensors on each of its faces.
872 #+caption: Each joint has a proprioceptive sense to detect
873 #+caption: relative positions. The worm segments are all the
874 #+caption: same except for the first one, which has a much
875 #+caption: higher weight than the others to allow for easy
876 #+caption: manual motor control.
877 #+name: basic-worm-view
878 #+ATTR_LaTeX: :width 10cm
879 [[./images/basic-worm-view.png]]
881 #+caption: Program for reading a worm from a blender file and
882 #+caption: outfitting it with the senses of proprioception,
883 #+caption: touch, and the ability to move, as specified in the
884 #+caption: blender file.
885 #+name: get-worm
886 #+begin_listing clojure
887 #+begin_src clojure
888 (defn worm []
889 (let [model (load-blender-model "Models/worm/worm.blend")]
890 {:body (doto model (body!))
891 :touch (touch! model)
892 :proprioception (proprioception! model)
893 :muscles (movement! model)}))
894 #+end_src
895 #+end_listing
897 ** Embodiment factors action recognition into managable parts
899 Using empathy, I divide the problem of action recognition into a
900 recognition process expressed in the language of a full compliment
901 of senses, and an imaganitive process that generates full sensory
902 data from partial sensory data. Splitting the action recognition
903 problem in this manner greatly reduces the total amount of work to
904 recognize actions: The imaganitive process is mostly just matching
905 previous experience, and the recognition process gets to use all
906 the senses to directly describe any action.
908 ** Action recognition is easy with a full gamut of senses
910 Embodied representations using multiple senses such as touch,
911 proprioception, and muscle tension turns out be be exceedingly
912 efficient at describing body-centered actions. It is the ``right
913 language for the job''. For example, it takes only around 5 lines
914 of LISP code to describe the action of ``curling'' using embodied
915 primitives. It takes about 10 lines to describe the seemingly
916 complicated action of wiggling.
918 The following action predicates each take a stream of sensory
919 experience, observe however much of it they desire, and decide
920 whether the worm is doing the action they describe. =curled?=
921 relies on proprioception, =resting?= relies on touch, =wiggling?=
922 relies on a fourier analysis of muscle contraction, and
923 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
925 #+caption: Program for detecting whether the worm is curled. This is the
926 #+caption: simplest action predicate, because it only uses the last frame
927 #+caption: of sensory experience, and only uses proprioceptive data. Even
928 #+caption: this simple predicate, however, is automatically frame
929 #+caption: independent and ignores vermopomorphic differences such as
930 #+caption: worm textures and colors.
931 #+name: curled
932 #+attr_latex: [htpb]
933 #+begin_listing clojure
934 #+begin_src clojure
935 (defn curled?
936 "Is the worm curled up?"
937 [experiences]
938 (every?
939 (fn [[_ _ bend]]
940 (> (Math/sin bend) 0.64))
941 (:proprioception (peek experiences))))
942 #+end_src
943 #+end_listing
945 #+caption: Program for summarizing the touch information in a patch
946 #+caption: of skin.
947 #+name: touch-summary
948 #+attr_latex: [htpb]
950 #+begin_listing clojure
951 #+begin_src clojure
952 (defn contact
953 "Determine how much contact a particular worm segment has with
954 other objects. Returns a value between 0 and 1, where 1 is full
955 contact and 0 is no contact."
956 [touch-region [coords contact :as touch]]
957 (-> (zipmap coords contact)
958 (select-keys touch-region)
959 (vals)
960 (#(map first %))
961 (average)
962 (* 10)
963 (- 1)
964 (Math/abs)))
965 #+end_src
966 #+end_listing
969 #+caption: Program for detecting whether the worm is at rest. This program
970 #+caption: uses a summary of the tactile information from the underbelly
971 #+caption: of the worm, and is only true if every segment is touching the
972 #+caption: floor. Note that this function contains no references to
973 #+caption: proprioction at all.
974 #+name: resting
975 #+attr_latex: [htpb]
976 #+begin_listing clojure
977 #+begin_src clojure
978 (def worm-segment-bottom (rect-region [8 15] [14 22]))
980 (defn resting?
981 "Is the worm resting on the ground?"
982 [experiences]
983 (every?
984 (fn [touch-data]
985 (< 0.9 (contact worm-segment-bottom touch-data)))
986 (:touch (peek experiences))))
987 #+end_src
988 #+end_listing
990 #+caption: Program for detecting whether the worm is curled up into a
991 #+caption: full circle. Here the embodied approach begins to shine, as
992 #+caption: I am able to both use a previous action predicate (=curled?=)
993 #+caption: as well as the direct tactile experience of the head and tail.
994 #+name: grand-circle
995 #+attr_latex: [htpb]
996 #+begin_listing clojure
997 #+begin_src clojure
998 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
1000 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
1002 (defn grand-circle?
1003 "Does the worm form a majestic circle (one end touching the other)?"
1004 [experiences]
1005 (and (curled? experiences)
1006 (let [worm-touch (:touch (peek experiences))
1007 tail-touch (worm-touch 0)
1008 head-touch (worm-touch 4)]
1009 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
1010 (< 0.55 (contact worm-segment-top-tip head-touch))))))
1011 #+end_src
1012 #+end_listing
1015 #+caption: Program for detecting whether the worm has been wiggling for
1016 #+caption: the last few frames. It uses a fourier analysis of the muscle
1017 #+caption: contractions of the worm's tail to determine wiggling. This is
1018 #+caption: signigicant because there is no particular frame that clearly
1019 #+caption: indicates that the worm is wiggling --- only when multiple frames
1020 #+caption: are analyzed together is the wiggling revealed. Defining
1021 #+caption: wiggling this way also gives the worm an opportunity to learn
1022 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
1023 #+caption: wiggle but can't. Frustrated wiggling is very visually different
1024 #+caption: from actual wiggling, but this definition gives it to us for free.
1025 #+name: wiggling
1026 #+attr_latex: [htpb]
1027 #+begin_listing clojure
1028 #+begin_src clojure
1029 (defn fft [nums]
1030 (map
1031 #(.getReal %)
1032 (.transform
1033 (FastFourierTransformer. DftNormalization/STANDARD)
1034 (double-array nums) TransformType/FORWARD)))
1036 (def indexed (partial map-indexed vector))
1038 (defn max-indexed [s]
1039 (first (sort-by (comp - second) (indexed s))))
1041 (defn wiggling?
1042 "Is the worm wiggling?"
1043 [experiences]
1044 (let [analysis-interval 0x40]
1045 (when (> (count experiences) analysis-interval)
1046 (let [a-flex 3
1047 a-ex 2
1048 muscle-activity
1049 (map :muscle (vector:last-n experiences analysis-interval))
1050 base-activity
1051 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
1052 (= 2
1053 (first
1054 (max-indexed
1055 (map #(Math/abs %)
1056 (take 20 (fft base-activity))))))))))
1057 #+end_src
1058 #+end_listing
1060 With these action predicates, I can now recognize the actions of
1061 the worm while it is moving under my control and I have access to
1062 all the worm's senses.
1064 #+caption: Use the action predicates defined earlier to report on
1065 #+caption: what the worm is doing while in simulation.
1066 #+name: report-worm-activity
1067 #+attr_latex: [htpb]
1068 #+begin_listing clojure
1069 #+begin_src clojure
1070 (defn debug-experience
1071 [experiences text]
1072 (cond
1073 (grand-circle? experiences) (.setText text "Grand Circle")
1074 (curled? experiences) (.setText text "Curled")
1075 (wiggling? experiences) (.setText text "Wiggling")
1076 (resting? experiences) (.setText text "Resting")))
1077 #+end_src
1078 #+end_listing
1080 #+caption: Using =debug-experience=, the body-centered predicates
1081 #+caption: work together to classify the behaviour of the worm.
1082 #+caption: the predicates are operating with access to the worm's
1083 #+caption: full sensory data.
1084 #+name: basic-worm-view
1085 #+ATTR_LaTeX: :width 10cm
1086 [[./images/worm-identify-init.png]]
1088 These action predicates satisfy the recognition requirement of an
1089 empathic recognition system. There is power in the simplicity of
1090 the action predicates. They describe their actions without getting
1091 confused in visual details of the worm. Each one is frame
1092 independent, but more than that, they are each indepent of
1093 irrelevant visual details of the worm and the environment. They
1094 will work regardless of whether the worm is a different color or
1095 hevaily textured, or if the environment has strange lighting.
1097 The trick now is to make the action predicates work even when the
1098 sensory data on which they depend is absent. If I can do that, then
1099 I will have gained much,
1101 ** \Phi-space describes the worm's experiences
1103 As a first step towards building empathy, I need to gather all of
1104 the worm's experiences during free play. I use a simple vector to
1105 store all the experiences.
1107 Each element of the experience vector exists in the vast space of
1108 all possible worm-experiences. Most of this vast space is actually
1109 unreachable due to physical constraints of the worm's body. For
1110 example, the worm's segments are connected by hinge joints that put
1111 a practical limit on the worm's range of motions without limiting
1112 its degrees of freedom. Some groupings of senses are impossible;
1113 the worm can not be bent into a circle so that its ends are
1114 touching and at the same time not also experience the sensation of
1115 touching itself.
1117 As the worm moves around during free play and its experience vector
1118 grows larger, the vector begins to define a subspace which is all
1119 the sensations the worm can practicaly experience during normal
1120 operation. I call this subspace \Phi-space, short for
1121 physical-space. The experience vector defines a path through
1122 \Phi-space. This path has interesting properties that all derive
1123 from physical embodiment. The proprioceptive components are
1124 completely smooth, because in order for the worm to move from one
1125 position to another, it must pass through the intermediate
1126 positions. The path invariably forms loops as actions are repeated.
1127 Finally and most importantly, proprioception actually gives very
1128 strong inference about the other senses. For example, when the worm
1129 is flat, you can infer that it is touching the ground and that its
1130 muscles are not active, because if the muscles were active, the
1131 worm would be moving and would not be perfectly flat. In order to
1132 stay flat, the worm has to be touching the ground, or it would
1133 again be moving out of the flat position due to gravity. If the
1134 worm is positioned in such a way that it interacts with itself,
1135 then it is very likely to be feeling the same tactile feelings as
1136 the last time it was in that position, because it has the same body
1137 as then. If you observe multiple frames of proprioceptive data,
1138 then you can become increasingly confident about the exact
1139 activations of the worm's muscles, because it generally takes a
1140 unique combination of muscle contractions to transform the worm's
1141 body along a specific path through \Phi-space.
1143 There is a simple way of taking \Phi-space and the total ordering
1144 provided by an experience vector and reliably infering the rest of
1145 the senses.
1147 ** Empathy is the process of tracing though \Phi-space
1149 Here is the core of a basic empathy algorithm, starting with an
1150 experience vector:
1152 First, group the experiences into tiered proprioceptive bins. I use
1153 powers of 10 and 3 bins, and the smallest bin has an approximate
1154 size of 0.001 radians in all proprioceptive dimensions.
1156 Then, given a sequence of proprioceptive input, generate a set of
1157 matching experience records for each input, using the tiered
1158 proprioceptive bins.
1160 Finally, to infer sensory data, select the longest consective chain
1161 of experiences. Conecutive experience means that the experiences
1162 appear next to each other in the experience vector.
1164 This algorithm has three advantages:
1166 1. It's simple
1168 3. It's very fast -- retrieving possible interpretations takes
1169 constant time. Tracing through chains of interpretations takes
1170 time proportional to the average number of experiences in a
1171 proprioceptive bin. Redundant experiences in \Phi-space can be
1172 merged to save computation.
1174 2. It protects from wrong interpretations of transient ambiguous
1175 proprioceptive data. For example, if the worm is flat for just
1176 an instant, this flattness will not be interpreted as implying
1177 that the worm has its muscles relaxed, since the flattness is
1178 part of a longer chain which includes a distinct pattern of
1179 muscle activation. Markov chains or other memoryless statistical
1180 models that operate on individual frames may very well make this
1181 mistake.
1183 #+caption: Program to convert an experience vector into a
1184 #+caption: proprioceptively binned lookup function.
1185 #+name: bin
1186 #+attr_latex: [htpb]
1187 #+begin_listing clojure
1188 #+begin_src clojure
1189 (defn bin [digits]
1190 (fn [angles]
1191 (->> angles
1192 (flatten)
1193 (map (juxt #(Math/sin %) #(Math/cos %)))
1194 (flatten)
1195 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
1197 (defn gen-phi-scan
1198 "Nearest-neighbors with binning. Only returns a result if
1199 the propriceptive data is within 10% of a previously recorded
1200 result in all dimensions."
1201 [phi-space]
1202 (let [bin-keys (map bin [3 2 1])
1203 bin-maps
1204 (map (fn [bin-key]
1205 (group-by
1206 (comp bin-key :proprioception phi-space)
1207 (range (count phi-space)))) bin-keys)
1208 lookups (map (fn [bin-key bin-map]
1209 (fn [proprio] (bin-map (bin-key proprio))))
1210 bin-keys bin-maps)]
1211 (fn lookup [proprio-data]
1212 (set (some #(% proprio-data) lookups)))))
1213 #+end_src
1214 #+end_listing
1216 #+caption: =longest-thread= finds the longest path of consecutive
1217 #+caption: experiences to explain proprioceptive worm data.
1218 #+name: phi-space-history-scan
1219 #+ATTR_LaTeX: :width 10cm
1220 [[./images/aurellem-gray.png]]
1222 =longest-thread= infers sensory data by stitching together pieces
1223 from previous experience. It prefers longer chains of previous
1224 experience to shorter ones. For example, during training the worm
1225 might rest on the ground for one second before it performs its
1226 excercises. If during recognition the worm rests on the ground for
1227 five seconds, =longest-thread= will accomodate this five second
1228 rest period by looping the one second rest chain five times.
1230 =longest-thread= takes time proportinal to the average number of
1231 entries in a proprioceptive bin, because for each element in the
1232 starting bin it performes a series of set lookups in the preceeding
1233 bins. If the total history is limited, then this is only a constant
1234 multiple times the number of entries in the starting bin. This
1235 analysis also applies even if the action requires multiple longest
1236 chains -- it's still the average number of entries in a
1237 proprioceptive bin times the desired chain length. Because
1238 =longest-thread= is so efficient and simple, I can interpret
1239 worm-actions in real time.
1241 #+caption: Program to calculate empathy by tracing though \Phi-space
1242 #+caption: and finding the longest (ie. most coherent) interpretation
1243 #+caption: of the data.
1244 #+name: longest-thread
1245 #+attr_latex: [htpb]
1246 #+begin_listing clojure
1247 #+begin_src clojure
1248 (defn longest-thread
1249 "Find the longest thread from phi-index-sets. The index sets should
1250 be ordered from most recent to least recent."
1251 [phi-index-sets]
1252 (loop [result '()
1253 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
1254 (if (empty? phi-index-sets)
1255 (vec result)
1256 (let [threads
1257 (for [thread-base thread-bases]
1258 (loop [thread (list thread-base)
1259 remaining remaining]
1260 (let [next-index (dec (first thread))]
1261 (cond (empty? remaining) thread
1262 (contains? (first remaining) next-index)
1263 (recur
1264 (cons next-index thread) (rest remaining))
1265 :else thread))))
1266 longest-thread
1267 (reduce (fn [thread-a thread-b]
1268 (if (> (count thread-a) (count thread-b))
1269 thread-a thread-b))
1270 '(nil)
1271 threads)]
1272 (recur (concat longest-thread result)
1273 (drop (count longest-thread) phi-index-sets))))))
1274 #+end_src
1275 #+end_listing
1277 There is one final piece, which is to replace missing sensory data
1278 with a best-guess estimate. While I could fill in missing data by
1279 using a gradient over the closest known sensory data points,
1280 averages can be misleading. It is certainly possible to create an
1281 impossible sensory state by averaging two possible sensory states.
1282 Therefore, I simply replicate the most recent sensory experience to
1283 fill in the gaps.
1285 #+caption: Fill in blanks in sensory experience by replicating the most
1286 #+caption: recent experience.
1287 #+name: infer-nils
1288 #+attr_latex: [htpb]
1289 #+begin_listing clojure
1290 #+begin_src clojure
1291 (defn infer-nils
1292 "Replace nils with the next available non-nil element in the
1293 sequence, or barring that, 0."
1294 [s]
1295 (loop [i (dec (count s))
1296 v (transient s)]
1297 (if (zero? i) (persistent! v)
1298 (if-let [cur (v i)]
1299 (if (get v (dec i) 0)
1300 (recur (dec i) v)
1301 (recur (dec i) (assoc! v (dec i) cur)))
1302 (recur i (assoc! v i 0))))))
1303 #+end_src
1304 #+end_listing
1306 ** Efficient action recognition with =EMPATH=
1308 To use =EMPATH= with the worm, I first need to gather a set of
1309 experiences from the worm that includes the actions I want to
1310 recognize. The =generate-phi-space= program (listing
1311 \ref{generate-phi-space} runs the worm through a series of
1312 exercices and gatheres those experiences into a vector. The
1313 =do-all-the-things= program is a routine expressed in a simple
1314 muscle contraction script language for automated worm control. It
1315 causes the worm to rest, curl, and wiggle over about 700 frames
1316 (approx. 11 seconds).
1318 #+caption: Program to gather the worm's experiences into a vector for
1319 #+caption: further processing. The =motor-control-program= line uses
1320 #+caption: a motor control script that causes the worm to execute a series
1321 #+caption: of ``exercices'' that include all the action predicates.
1322 #+name: generate-phi-space
1323 #+attr_latex: [htpb]
1324 #+begin_listing clojure
1325 #+begin_src clojure
1326 (def do-all-the-things
1327 (concat
1328 curl-script
1329 [[300 :d-ex 40]
1330 [320 :d-ex 0]]
1331 (shift-script 280 (take 16 wiggle-script))))
1333 (defn generate-phi-space []
1334 (let [experiences (atom [])]
1335 (run-world
1336 (apply-map
1337 worm-world
1338 (merge
1339 (worm-world-defaults)
1340 {:end-frame 700
1341 :motor-control
1342 (motor-control-program worm-muscle-labels do-all-the-things)
1343 :experiences experiences})))
1344 @experiences))
1345 #+end_src
1346 #+end_listing
1348 #+caption: Use longest thread and a phi-space generated from a short
1349 #+caption: exercise routine to interpret actions during free play.
1350 #+name: empathy-debug
1351 #+attr_latex: [htpb]
1352 #+begin_listing clojure
1353 #+begin_src clojure
1354 (defn init []
1355 (def phi-space (generate-phi-space))
1356 (def phi-scan (gen-phi-scan phi-space)))
1358 (defn empathy-demonstration []
1359 (let [proprio (atom ())]
1360 (fn
1361 [experiences text]
1362 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
1363 (swap! proprio (partial cons phi-indices))
1364 (let [exp-thread (longest-thread (take 300 @proprio))
1365 empathy (mapv phi-space (infer-nils exp-thread))]
1366 (println-repl (vector:last-n exp-thread 22))
1367 (cond
1368 (grand-circle? empathy) (.setText text "Grand Circle")
1369 (curled? empathy) (.setText text "Curled")
1370 (wiggling? empathy) (.setText text "Wiggling")
1371 (resting? empathy) (.setText text "Resting")
1372 :else (.setText text "Unknown")))))))
1374 (defn empathy-experiment [record]
1375 (.start (worm-world :experience-watch (debug-experience-phi)
1376 :record record :worm worm*)))
1377 #+end_src
1378 #+end_listing
1380 The result of running =empathy-experiment= is that the system is
1381 generally able to interpret worm actions using the action-predicates
1382 on simulated sensory data just as well as with actual data. Figure
1383 \ref{empathy-debug-image} was generated using =empathy-experiment=:
1385 #+caption: From only proprioceptive data, =EMPATH= was able to infer
1386 #+caption: the complete sensory experience and classify four poses
1387 #+caption: (The last panel shows a composite image of \emph{wriggling},
1388 #+caption: a dynamic pose.)
1389 #+name: empathy-debug-image
1390 #+ATTR_LaTeX: :width 10cm :placement [H]
1391 [[./images/empathy-1.png]]
1393 One way to measure the performance of =EMPATH= is to compare the
1394 sutiability of the imagined sense experience to trigger the same
1395 action predicates as the real sensory experience.
1397 #+caption: Determine how closely empathy approximates actual
1398 #+caption: sensory data.
1399 #+name: test-empathy-accuracy
1400 #+attr_latex: [htpb]
1401 #+begin_listing clojure
1402 #+begin_src clojure
1403 (def worm-action-label
1404 (juxt grand-circle? curled? wiggling?))
1406 (defn compare-empathy-with-baseline [matches]
1407 (let [proprio (atom ())]
1408 (fn
1409 [experiences text]
1410 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
1411 (swap! proprio (partial cons phi-indices))
1412 (let [exp-thread (longest-thread (take 300 @proprio))
1413 empathy (mapv phi-space (infer-nils exp-thread))
1414 experience-matches-empathy
1415 (= (worm-action-label experiences)
1416 (worm-action-label empathy))]
1417 (println-repl experience-matches-empathy)
1418 (swap! matches #(conj % experience-matches-empathy)))))))
1420 (defn accuracy [v]
1421 (float (/ (count (filter true? v)) (count v))))
1423 (defn test-empathy-accuracy []
1424 (let [res (atom [])]
1425 (run-world
1426 (worm-world :experience-watch
1427 (compare-empathy-with-baseline res)
1428 :worm worm*))
1429 (accuracy @res)))
1430 #+end_src
1431 #+end_listing
1433 Running =test-empathy-accuracy= using the very short exercise
1434 program defined in listing \ref{generate-phi-space}, and then doing
1435 a similar pattern of activity manually yeilds an accuracy of around
1436 73%. This is based on very limited worm experience. By training the
1437 worm for longer, the accuracy dramatically improves.
1439 #+caption: Program to generate \Phi-space using manual training.
1440 #+name: manual-phi-space
1441 #+attr_latex: [htpb]
1442 #+begin_listing clojure
1443 #+begin_src clojure
1444 (defn init-interactive []
1445 (def phi-space
1446 (let [experiences (atom [])]
1447 (run-world
1448 (apply-map
1449 worm-world
1450 (merge
1451 (worm-world-defaults)
1452 {:experiences experiences})))
1453 @experiences))
1454 (def phi-scan (gen-phi-scan phi-space)))
1455 #+end_src
1456 #+end_listing
1458 After about 1 minute of manual training, I was able to achieve 95%
1459 accuracy on manual testing of the worm using =init-interactive= and
1460 =test-empathy-accuracy=. The majority of errors are near the
1461 boundaries of transitioning from one type of action to another.
1462 During these transitions the exact label for the action is more open
1463 to interpretation, and dissaggrement between empathy and experience
1464 is more excusable.
1466 ** Digression: bootstrapping touch using free exploration
1468 In the previous section I showed how to compute actions in terms of
1469 body-centered predicates which relied averate touch activation of
1470 pre-defined regions of the worm's skin. What if, instead of recieving
1471 touch pre-grouped into the six faces of each worm segment, the true
1472 topology of the worm's skin was unknown? This is more similiar to how
1473 a nerve fiber bundle might be arranged. While two fibers that are
1474 close in a nerve bundle /might/ correspond to two touch sensors that
1475 are close together on the skin, the process of taking a complicated
1476 surface and forcing it into essentially a circle requires some cuts
1477 and rerragenments.
1479 In this section I show how to automatically learn the skin-topology of
1480 a worm segment by free exploration. As the worm rolls around on the
1481 floor, large sections of its surface get activated. If the worm has
1482 stopped moving, then whatever region of skin that is touching the
1483 floor is probably an important region, and should be recorded.
1485 #+caption: Program to detect whether the worm is in a resting state
1486 #+caption: with one face touching the floor.
1487 #+name: pure-touch
1488 #+begin_listing clojure
1489 #+begin_src clojure
1490 (def full-contact [(float 0.0) (float 0.1)])
1492 (defn pure-touch?
1493 "This is worm specific code to determine if a large region of touch
1494 sensors is either all on or all off."
1495 [[coords touch :as touch-data]]
1496 (= (set (map first touch)) (set full-contact)))
1497 #+end_src
1498 #+end_listing
1500 After collecting these important regions, there will many nearly
1501 similiar touch regions. While for some purposes the subtle
1502 differences between these regions will be important, for my
1503 purposes I colapse them into mostly non-overlapping sets using
1504 =remove-similiar= in listing \ref{remove-similiar}
1506 #+caption: Program to take a lits of set of points and ``collapse them''
1507 #+caption: so that the remaining sets in the list are siginificantly
1508 #+caption: different from each other. Prefer smaller sets to larger ones.
1509 #+name: remove-similiar
1510 #+begin_listing clojure
1511 #+begin_src clojure
1512 (defn remove-similar
1513 [coll]
1514 (loop [result () coll (sort-by (comp - count) coll)]
1515 (if (empty? coll) result
1516 (let [[x & xs] coll
1517 c (count x)]
1518 (if (some
1519 (fn [other-set]
1520 (let [oc (count other-set)]
1521 (< (- (count (union other-set x)) c) (* oc 0.1))))
1522 xs)
1523 (recur result xs)
1524 (recur (cons x result) xs))))))
1525 #+end_src
1526 #+end_listing
1528 Actually running this simulation is easy given =CORTEX='s facilities.
1530 #+caption: Collect experiences while the worm moves around. Filter the touch
1531 #+caption: sensations by stable ones, collapse similiar ones together,
1532 #+caption: and report the regions learned.
1533 #+name: learn-touch
1534 #+begin_listing clojure
1535 #+begin_src clojure
1536 (defn learn-touch-regions []
1537 (let [experiences (atom [])
1538 world (apply-map
1539 worm-world
1540 (assoc (worm-segment-defaults)
1541 :experiences experiences))]
1542 (run-world world)
1543 (->>
1544 @experiences
1545 (drop 175)
1546 ;; access the single segment's touch data
1547 (map (comp first :touch))
1548 ;; only deal with "pure" touch data to determine surfaces
1549 (filter pure-touch?)
1550 ;; associate coordinates with touch values
1551 (map (partial apply zipmap))
1552 ;; select those regions where contact is being made
1553 (map (partial group-by second))
1554 (map #(get % full-contact))
1555 (map (partial map first))
1556 ;; remove redundant/subset regions
1557 (map set)
1558 remove-similar)))
1560 (defn learn-and-view-touch-regions []
1561 (map view-touch-region
1562 (learn-touch-regions)))
1563 #+end_src
1564 #+end_listing
1566 The only thing remining to define is the particular motion the worm
1567 must take. I accomplish this with a simple motor control program.
1569 #+caption: Motor control program for making the worm roll on the ground.
1570 #+caption: This could also be replaced with random motion.
1571 #+name: worm-roll
1572 #+begin_listing clojure
1573 #+begin_src clojure
1574 (defn touch-kinesthetics []
1575 [[170 :lift-1 40]
1576 [190 :lift-1 19]
1577 [206 :lift-1 0]
1579 [400 :lift-2 40]
1580 [410 :lift-2 0]
1582 [570 :lift-2 40]
1583 [590 :lift-2 21]
1584 [606 :lift-2 0]
1586 [800 :lift-1 30]
1587 [809 :lift-1 0]
1589 [900 :roll-2 40]
1590 [905 :roll-2 20]
1591 [910 :roll-2 0]
1593 [1000 :roll-2 40]
1594 [1005 :roll-2 20]
1595 [1010 :roll-2 0]
1597 [1100 :roll-2 40]
1598 [1105 :roll-2 20]
1599 [1110 :roll-2 0]
1600 ])
1601 #+end_src
1602 #+end_listing
1605 #+caption: The small worm rolls around on the floor, driven
1606 #+caption: by the motor control program in listing \ref{worm-roll}.
1607 #+name: worm-roll
1608 #+ATTR_LaTeX: :width 12cm
1609 [[./images/worm-roll.png]]
1612 #+caption: After completing its adventures, the worm now knows
1613 #+caption: how its touch sensors are arranged along its skin. These
1614 #+caption: are the regions that were deemed important by
1615 #+caption: =learn-touch-regions=. Note that the worm has discovered
1616 #+caption: that it has six sides.
1617 #+name: worm-touch-map
1618 #+ATTR_LaTeX: :width 12cm
1619 [[./images/touch-learn.png]]
1621 While simple, =learn-touch-regions= exploits regularities in both
1622 the worm's physiology and the worm's environment to correctly
1623 deduce that the worm has six sides. Note that =learn-touch-regions=
1624 would work just as well even if the worm's touch sense data were
1625 completely scrambled. The cross shape is just for convienence. This
1626 example justifies the use of pre-defined touch regions in =EMPATH=.
1628 * COMMENT Contributions
1630 In this thesis you have seen the =CORTEX= system, a complete
1631 environment for creating simulated creatures. You have seen how to
1632 implement five senses including touch, proprioception, hearing,
1633 vision, and muscle tension. You have seen how to create new creatues
1634 using blender, a 3D modeling tool. I hope that =CORTEX= will be
1635 useful in further research projects. To this end I have included the
1636 full source to =CORTEX= along with a large suite of tests and
1637 examples. I have also created a user guide for =CORTEX= which is
1638 inculded in an appendix to this thesis.
1640 You have also seen how I used =CORTEX= as a platform to attach the
1641 /action recognition/ problem, which is the problem of recognizing
1642 actions in video. You saw a simple system called =EMPATH= which
1643 ientifies actions by first describing actions in a body-centerd,
1644 rich sense language, then infering a full range of sensory
1645 experience from limited data using previous experience gained from
1646 free play.
1648 As a minor digression, you also saw how I used =CORTEX= to enable a
1649 tiny worm to discover the topology of its skin simply by rolling on
1650 the ground.
1652 In conclusion, the main contributions of this thesis are:
1654 - =CORTEX=, a system for creating simulated creatures with rich
1655 senses.
1656 - =EMPATH=, a program for recognizing actions by imagining sensory
1657 experience.
1659 # An anatomical joke:
1660 # - Training
1661 # - Skeletal imitation
1662 # - Sensory fleshing-out
1663 # - Classification