view thesis/cortex.org @ 472:516a029e0be9

complete first draft of hearing.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 18:14:04 -0400
parents f14fa9e5b67f
children 486ce07f5545
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+end_listing
17 #+caption:
18 #+caption:
19 #+caption:
20 #+name: name
21 #+ATTR_LaTeX: :width 10cm
22 [[./images/aurellem-gray.png]]
24 #+caption:
25 #+caption:
26 #+caption:
27 #+caption:
28 #+name: name
29 #+begin_listing clojure
30 #+end_listing
32 #+caption:
33 #+caption:
34 #+caption:
35 #+name: name
36 #+ATTR_LaTeX: :width 10cm
37 [[./images/aurellem-gray.png]]
40 * COMMENT Empathy and Embodiment as problem solving strategies
42 By the end of this thesis, you will have seen a novel approach to
43 interpreting video using embodiment and empathy. You will have also
44 seen one way to efficiently implement empathy for embodied
45 creatures. Finally, you will become familiar with =CORTEX=, a system
46 for designing and simulating creatures with rich senses, which you
47 may choose to use in your own research.
49 This is the core vision of my thesis: That one of the important ways
50 in which we understand others is by imagining ourselves in their
51 position and emphatically feeling experiences relative to our own
52 bodies. By understanding events in terms of our own previous
53 corporeal experience, we greatly constrain the possibilities of what
54 would otherwise be an unwieldy exponential search. This extra
55 constraint can be the difference between easily understanding what
56 is happening in a video and being completely lost in a sea of
57 incomprehensible color and movement.
59 ** Recognizing actions in video is extremely difficult
61 Consider for example the problem of determining what is happening
62 in a video of which this is one frame:
64 #+caption: A cat drinking some water. Identifying this action is
65 #+caption: beyond the state of the art for computers.
66 #+ATTR_LaTeX: :width 7cm
67 [[./images/cat-drinking.jpg]]
69 It is currently impossible for any computer program to reliably
70 label such a video as ``drinking''. And rightly so -- it is a very
71 hard problem! What features can you describe in terms of low level
72 functions of pixels that can even begin to describe at a high level
73 what is happening here?
75 Or suppose that you are building a program that recognizes chairs.
76 How could you ``see'' the chair in figure \ref{hidden-chair}?
78 #+caption: The chair in this image is quite obvious to humans, but I
79 #+caption: doubt that any modern computer vision program can find it.
80 #+name: hidden-chair
81 #+ATTR_LaTeX: :width 10cm
82 [[./images/fat-person-sitting-at-desk.jpg]]
84 Finally, how is it that you can easily tell the difference between
85 how the girls /muscles/ are working in figure \ref{girl}?
87 #+caption: The mysterious ``common sense'' appears here as you are able
88 #+caption: to discern the difference in how the girl's arm muscles
89 #+caption: are activated between the two images.
90 #+name: girl
91 #+ATTR_LaTeX: :width 7cm
92 [[./images/wall-push.png]]
94 Each of these examples tells us something about what might be going
95 on in our minds as we easily solve these recognition problems.
97 The hidden chairs show us that we are strongly triggered by cues
98 relating to the position of human bodies, and that we can determine
99 the overall physical configuration of a human body even if much of
100 that body is occluded.
102 The picture of the girl pushing against the wall tells us that we
103 have common sense knowledge about the kinetics of our own bodies.
104 We know well how our muscles would have to work to maintain us in
105 most positions, and we can easily project this self-knowledge to
106 imagined positions triggered by images of the human body.
108 ** =EMPATH= neatly solves recognition problems
110 I propose a system that can express the types of recognition
111 problems above in a form amenable to computation. It is split into
112 four parts:
114 - Free/Guided Play :: The creature moves around and experiences the
115 world through its unique perspective. Many otherwise
116 complicated actions are easily described in the language of a
117 full suite of body-centered, rich senses. For example,
118 drinking is the feeling of water sliding down your throat, and
119 cooling your insides. It's often accompanied by bringing your
120 hand close to your face, or bringing your face close to water.
121 Sitting down is the feeling of bending your knees, activating
122 your quadriceps, then feeling a surface with your bottom and
123 relaxing your legs. These body-centered action descriptions
124 can be either learned or hard coded.
125 - Posture Imitation :: When trying to interpret a video or image,
126 the creature takes a model of itself and aligns it with
127 whatever it sees. This alignment can even cross species, as
128 when humans try to align themselves with things like ponies,
129 dogs, or other humans with a different body type.
130 - Empathy :: The alignment triggers associations with
131 sensory data from prior experiences. For example, the
132 alignment itself easily maps to proprioceptive data. Any
133 sounds or obvious skin contact in the video can to a lesser
134 extent trigger previous experience. Segments of previous
135 experiences are stitched together to form a coherent and
136 complete sensory portrait of the scene.
137 - Recognition :: With the scene described in terms of first
138 person sensory events, the creature can now run its
139 action-identification programs on this synthesized sensory
140 data, just as it would if it were actually experiencing the
141 scene first-hand. If previous experience has been accurately
142 retrieved, and if it is analogous enough to the scene, then
143 the creature will correctly identify the action in the scene.
145 For example, I think humans are able to label the cat video as
146 ``drinking'' because they imagine /themselves/ as the cat, and
147 imagine putting their face up against a stream of water and
148 sticking out their tongue. In that imagined world, they can feel
149 the cool water hitting their tongue, and feel the water entering
150 their body, and are able to recognize that /feeling/ as drinking.
151 So, the label of the action is not really in the pixels of the
152 image, but is found clearly in a simulation inspired by those
153 pixels. An imaginative system, having been trained on drinking and
154 non-drinking examples and learning that the most important
155 component of drinking is the feeling of water sliding down one's
156 throat, would analyze a video of a cat drinking in the following
157 manner:
159 1. Create a physical model of the video by putting a ``fuzzy''
160 model of its own body in place of the cat. Possibly also create
161 a simulation of the stream of water.
163 2. Play out this simulated scene and generate imagined sensory
164 experience. This will include relevant muscle contractions, a
165 close up view of the stream from the cat's perspective, and most
166 importantly, the imagined feeling of water entering the
167 mouth. The imagined sensory experience can come from a
168 simulation of the event, but can also be pattern-matched from
169 previous, similar embodied experience.
171 3. The action is now easily identified as drinking by the sense of
172 taste alone. The other senses (such as the tongue moving in and
173 out) help to give plausibility to the simulated action. Note that
174 the sense of vision, while critical in creating the simulation,
175 is not critical for identifying the action from the simulation.
177 For the chair examples, the process is even easier:
179 1. Align a model of your body to the person in the image.
181 2. Generate proprioceptive sensory data from this alignment.
183 3. Use the imagined proprioceptive data as a key to lookup related
184 sensory experience associated with that particular proproceptive
185 feeling.
187 4. Retrieve the feeling of your bottom resting on a surface, your
188 knees bent, and your leg muscles relaxed.
190 5. This sensory information is consistent with the =sitting?=
191 sensory predicate, so you (and the entity in the image) must be
192 sitting.
194 6. There must be a chair-like object since you are sitting.
196 Empathy offers yet another alternative to the age-old AI
197 representation question: ``What is a chair?'' --- A chair is the
198 feeling of sitting.
200 My program, =EMPATH= uses this empathic problem solving technique
201 to interpret the actions of a simple, worm-like creature.
203 #+caption: The worm performs many actions during free play such as
204 #+caption: curling, wiggling, and resting.
205 #+name: worm-intro
206 #+ATTR_LaTeX: :width 15cm
207 [[./images/worm-intro-white.png]]
209 #+caption: =EMPATH= recognized and classified each of these
210 #+caption: poses by inferring the complete sensory experience
211 #+caption: from proprioceptive data.
212 #+name: worm-recognition-intro
213 #+ATTR_LaTeX: :width 15cm
214 [[./images/worm-poses.png]]
216 One powerful advantage of empathic problem solving is that it
217 factors the action recognition problem into two easier problems. To
218 use empathy, you need an /aligner/, which takes the video and a
219 model of your body, and aligns the model with the video. Then, you
220 need a /recognizer/, which uses the aligned model to interpret the
221 action. The power in this method lies in the fact that you describe
222 all actions form a body-centered viewpoint. You are less tied to
223 the particulars of any visual representation of the actions. If you
224 teach the system what ``running'' is, and you have a good enough
225 aligner, the system will from then on be able to recognize running
226 from any point of view, even strange points of view like above or
227 underneath the runner. This is in contrast to action recognition
228 schemes that try to identify actions using a non-embodied approach.
229 If these systems learn about running as viewed from the side, they
230 will not automatically be able to recognize running from any other
231 viewpoint.
233 Another powerful advantage is that using the language of multiple
234 body-centered rich senses to describe body-centerd actions offers a
235 massive boost in descriptive capability. Consider how difficult it
236 would be to compose a set of HOG filters to describe the action of
237 a simple worm-creature ``curling'' so that its head touches its
238 tail, and then behold the simplicity of describing thus action in a
239 language designed for the task (listing \ref{grand-circle-intro}):
241 #+caption: Body-centerd actions are best expressed in a body-centered
242 #+caption: language. This code detects when the worm has curled into a
243 #+caption: full circle. Imagine how you would replicate this functionality
244 #+caption: using low-level pixel features such as HOG filters!
245 #+name: grand-circle-intro
246 #+attr_latex: [htpb]
247 #+begin_listing clojure
248 #+begin_src clojure
249 (defn grand-circle?
250 "Does the worm form a majestic circle (one end touching the other)?"
251 [experiences]
252 (and (curled? experiences)
253 (let [worm-touch (:touch (peek experiences))
254 tail-touch (worm-touch 0)
255 head-touch (worm-touch 4)]
256 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
257 (< 0.2 (contact worm-segment-top-tip head-touch))))))
258 #+end_src
259 #+end_listing
262 ** =CORTEX= is a toolkit for building sensate creatures
264 I built =CORTEX= to be a general AI research platform for doing
265 experiments involving multiple rich senses and a wide variety and
266 number of creatures. I intend it to be useful as a library for many
267 more projects than just this thesis. =CORTEX= was necessary to meet
268 a need among AI researchers at CSAIL and beyond, which is that
269 people often will invent neat ideas that are best expressed in the
270 language of creatures and senses, but in order to explore those
271 ideas they must first build a platform in which they can create
272 simulated creatures with rich senses! There are many ideas that
273 would be simple to execute (such as =EMPATH=), but attached to them
274 is the multi-month effort to make a good creature simulator. Often,
275 that initial investment of time proves to be too much, and the
276 project must make do with a lesser environment.
278 =CORTEX= is well suited as an environment for embodied AI research
279 for three reasons:
281 - You can create new creatures using Blender, a popular 3D modeling
282 program. Each sense can be specified using special blender nodes
283 with biologically inspired paramaters. You need not write any
284 code to create a creature, and can use a wide library of
285 pre-existing blender models as a base for your own creatures.
287 - =CORTEX= implements a wide variety of senses, including touch,
288 proprioception, vision, hearing, and muscle tension. Complicated
289 senses like touch, and vision involve multiple sensory elements
290 embedded in a 2D surface. You have complete control over the
291 distribution of these sensor elements through the use of simple
292 png image files. In particular, =CORTEX= implements more
293 comprehensive hearing than any other creature simulation system
294 available.
296 - =CORTEX= supports any number of creatures and any number of
297 senses. Time in =CORTEX= dialates so that the simulated creatures
298 always precieve a perfectly smooth flow of time, regardless of
299 the actual computational load.
301 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
302 engine designed to create cross-platform 3D desktop games. =CORTEX=
303 is mainly written in clojure, a dialect of =LISP= that runs on the
304 java virtual machine (JVM). The API for creating and simulating
305 creatures and senses is entirely expressed in clojure, though many
306 senses are implemented at the layer of jMonkeyEngine or below. For
307 example, for the sense of hearing I use a layer of clojure code on
308 top of a layer of java JNI bindings that drive a layer of =C++=
309 code which implements a modified version of =OpenAL= to support
310 multiple listeners. =CORTEX= is the only simulation environment
311 that I know of that can support multiple entities that can each
312 hear the world from their own perspective. Other senses also
313 require a small layer of Java code. =CORTEX= also uses =bullet=, a
314 physics simulator written in =C=.
316 #+caption: Here is the worm from above modeled in Blender, a free
317 #+caption: 3D-modeling program. Senses and joints are described
318 #+caption: using special nodes in Blender.
319 #+name: worm-recognition-intro
320 #+ATTR_LaTeX: :width 12cm
321 [[./images/blender-worm.png]]
323 Here are some thing I anticipate that =CORTEX= might be used for:
325 - exploring new ideas about sensory integration
326 - distributed communication among swarm creatures
327 - self-learning using free exploration,
328 - evolutionary algorithms involving creature construction
329 - exploration of exoitic senses and effectors that are not possible
330 in the real world (such as telekenisis or a semantic sense)
331 - imagination using subworlds
333 During one test with =CORTEX=, I created 3,000 creatures each with
334 their own independent senses and ran them all at only 1/80 real
335 time. In another test, I created a detailed model of my own hand,
336 equipped with a realistic distribution of touch (more sensitive at
337 the fingertips), as well as eyes and ears, and it ran at around 1/4
338 real time.
340 #+BEGIN_LaTeX
341 \begin{sidewaysfigure}
342 \includegraphics[width=9.5in]{images/full-hand.png}
343 \caption{
344 I modeled my own right hand in Blender and rigged it with all the
345 senses that {\tt CORTEX} supports. My simulated hand has a
346 biologically inspired distribution of touch sensors. The senses are
347 displayed on the right, and the simulation is displayed on the
348 left. Notice that my hand is curling its fingers, that it can see
349 its own finger from the eye in its palm, and that it can feel its
350 own thumb touching its palm.}
351 \end{sidewaysfigure}
352 #+END_LaTeX
354 ** Contributions
356 - I built =CORTEX=, a comprehensive platform for embodied AI
357 experiments. =CORTEX= supports many features lacking in other
358 systems, such proper simulation of hearing. It is easy to create
359 new =CORTEX= creatures using Blender, a free 3D modeling program.
361 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
362 a worm-like creature using a computational model of empathy.
364 * Building =CORTEX=
366 I intend for =CORTEX= to be used as a general purpose library for
367 building creatures and outfitting them with senses, so that it will
368 be useful for other researchers who want to test out ideas of their
369 own. To this end, wherver I have had to make archetictural choices
370 about =CORTEX=, I have chosen to give as much freedom to the user as
371 possible, so that =CORTEX= may be used for things I have not
372 forseen.
374 ** COMMENT Simulation or Reality?
376 The most important archetictural decision of all is the choice to
377 use a computer-simulated environemnt in the first place! The world
378 is a vast and rich place, and for now simulations are a very poor
379 reflection of its complexity. It may be that there is a significant
380 qualatative difference between dealing with senses in the real
381 world and dealing with pale facilimilies of them in a simulation.
382 What are the advantages and disadvantages of a simulation vs.
383 reality?
385 *** Simulation
387 The advantages of virtual reality are that when everything is a
388 simulation, experiments in that simulation are absolutely
389 reproducible. It's also easier to change the character and world
390 to explore new situations and different sensory combinations.
392 If the world is to be simulated on a computer, then not only do
393 you have to worry about whether the character's senses are rich
394 enough to learn from the world, but whether the world itself is
395 rendered with enough detail and realism to give enough working
396 material to the character's senses. To name just a few
397 difficulties facing modern physics simulators: destructibility of
398 the environment, simulation of water/other fluids, large areas,
399 nonrigid bodies, lots of objects, smoke. I don't know of any
400 computer simulation that would allow a character to take a rock
401 and grind it into fine dust, then use that dust to make a clay
402 sculpture, at least not without spending years calculating the
403 interactions of every single small grain of dust. Maybe a
404 simulated world with today's limitations doesn't provide enough
405 richness for real intelligence to evolve.
407 *** Reality
409 The other approach for playing with senses is to hook your
410 software up to real cameras, microphones, robots, etc., and let it
411 loose in the real world. This has the advantage of eliminating
412 concerns about simulating the world at the expense of increasing
413 the complexity of implementing the senses. Instead of just
414 grabbing the current rendered frame for processing, you have to
415 use an actual camera with real lenses and interact with photons to
416 get an image. It is much harder to change the character, which is
417 now partly a physical robot of some sort, since doing so involves
418 changing things around in the real world instead of modifying
419 lines of code. While the real world is very rich and definitely
420 provides enough stimulation for intelligence to develop as
421 evidenced by our own existence, it is also uncontrollable in the
422 sense that a particular situation cannot be recreated perfectly or
423 saved for later use. It is harder to conduct science because it is
424 harder to repeat an experiment. The worst thing about using the
425 real world instead of a simulation is the matter of time. Instead
426 of simulated time you get the constant and unstoppable flow of
427 real time. This severely limits the sorts of software you can use
428 to program the AI because all sense inputs must be handled in real
429 time. Complicated ideas may have to be implemented in hardware or
430 may simply be impossible given the current speed of our
431 processors. Contrast this with a simulation, in which the flow of
432 time in the simulated world can be slowed down to accommodate the
433 limitations of the character's programming. In terms of cost,
434 doing everything in software is far cheaper than building custom
435 real-time hardware. All you need is a laptop and some patience.
437 ** COMMENT Because of Time, simulation is perferable to reality
439 I envision =CORTEX= being used to support rapid prototyping and
440 iteration of ideas. Even if I could put together a well constructed
441 kit for creating robots, it would still not be enough because of
442 the scourge of real-time processing. Anyone who wants to test their
443 ideas in the real world must always worry about getting their
444 algorithms to run fast enough to process information in real time.
445 The need for real time processing only increases if multiple senses
446 are involved. In the extreme case, even simple algorithms will have
447 to be accelerated by ASIC chips or FPGAs, turning what would
448 otherwise be a few lines of code and a 10x speed penality into a
449 multi-month ordeal. For this reason, =CORTEX= supports
450 /time-dialiation/, which scales back the framerate of the
451 simulation in proportion to the amount of processing each frame.
452 From the perspective of the creatures inside the simulation, time
453 always appears to flow at a constant rate, regardless of how
454 complicated the envorimnent becomes or how many creatures are in
455 the simulation. The cost is that =CORTEX= can sometimes run slower
456 than real time. This can also be an advantage, however ---
457 simulations of very simple creatures in =CORTEX= generally run at
458 40x on my machine!
460 ** COMMENT What is a sense?
462 If =CORTEX= is to support a wide variety of senses, it would help
463 to have a better understanding of what a ``sense'' actually is!
464 While vision, touch, and hearing all seem like they are quite
465 different things, I was supprised to learn during the course of
466 this thesis that they (and all physical senses) can be expressed as
467 exactly the same mathematical object due to a dimensional argument!
469 Human beings are three-dimensional objects, and the nerves that
470 transmit data from our various sense organs to our brain are
471 essentially one-dimensional. This leaves up to two dimensions in
472 which our sensory information may flow. For example, imagine your
473 skin: it is a two-dimensional surface around a three-dimensional
474 object (your body). It has discrete touch sensors embedded at
475 various points, and the density of these sensors corresponds to the
476 sensitivity of that region of skin. Each touch sensor connects to a
477 nerve, all of which eventually are bundled together as they travel
478 up the spinal cord to the brain. Intersect the spinal nerves with a
479 guillotining plane and you will see all of the sensory data of the
480 skin revealed in a roughly circular two-dimensional image which is
481 the cross section of the spinal cord. Points on this image that are
482 close together in this circle represent touch sensors that are
483 /probably/ close together on the skin, although there is of course
484 some cutting and rearrangement that has to be done to transfer the
485 complicated surface of the skin onto a two dimensional image.
487 Most human senses consist of many discrete sensors of various
488 properties distributed along a surface at various densities. For
489 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
490 disks, and Ruffini's endings, which detect pressure and vibration
491 of various intensities. For ears, it is the stereocilia distributed
492 along the basilar membrane inside the cochlea; each one is
493 sensitive to a slightly different frequency of sound. For eyes, it
494 is rods and cones distributed along the surface of the retina. In
495 each case, we can describe the sense with a surface and a
496 distribution of sensors along that surface.
498 The neat idea is that every human sense can be effectively
499 described in terms of a surface containing embedded sensors. If the
500 sense had any more dimensions, then there wouldn't be enough room
501 in the spinal chord to transmit the information!
503 Therefore, =CORTEX= must support the ability to create objects and
504 then be able to ``paint'' points along their surfaces to describe
505 each sense.
507 Fortunately this idea is already a well known computer graphics
508 technique called called /UV-mapping/. The three-dimensional surface
509 of a model is cut and smooshed until it fits on a two-dimensional
510 image. You paint whatever you want on that image, and when the
511 three-dimensional shape is rendered in a game the smooshing and
512 cutting is reversed and the image appears on the three-dimensional
513 object.
515 To make a sense, interpret the UV-image as describing the
516 distribution of that senses sensors. To get different types of
517 sensors, you can either use a different color for each type of
518 sensor, or use multiple UV-maps, each labeled with that sensor
519 type. I generally use a white pixel to mean the presence of a
520 sensor and a black pixel to mean the absence of a sensor, and use
521 one UV-map for each sensor-type within a given sense.
523 #+CAPTION: The UV-map for an elongated icososphere. The white
524 #+caption: dots each represent a touch sensor. They are dense
525 #+caption: in the regions that describe the tip of the finger,
526 #+caption: and less dense along the dorsal side of the finger
527 #+caption: opposite the tip.
528 #+name: finger-UV
529 #+ATTR_latex: :width 10cm
530 [[./images/finger-UV.png]]
532 #+caption: Ventral side of the UV-mapped finger. Notice the
533 #+caption: density of touch sensors at the tip.
534 #+name: finger-side-view
535 #+ATTR_LaTeX: :width 10cm
536 [[./images/finger-1.png]]
538 ** COMMENT Video game engines are a great starting point
540 I did not need to write my own physics simulation code or shader to
541 build =CORTEX=. Doing so would lead to a system that is impossible
542 for anyone but myself to use anyway. Instead, I use a video game
543 engine as a base and modify it to accomodate the additional needs
544 of =CORTEX=. Video game engines are an ideal starting point to
545 build =CORTEX=, because they are not far from being creature
546 building systems themselves.
548 First off, general purpose video game engines come with a physics
549 engine and lighting / sound system. The physics system provides
550 tools that can be co-opted to serve as touch, proprioception, and
551 muscles. Since some games support split screen views, a good video
552 game engine will allow you to efficiently create multiple cameras
553 in the simulated world that can be used as eyes. Video game systems
554 offer integrated asset management for things like textures and
555 creatures models, providing an avenue for defining creatures. They
556 also understand UV-mapping, since this technique is used to apply a
557 texture to a model. Finally, because video game engines support a
558 large number of users, as long as =CORTEX= doesn't stray too far
559 from the base system, other researchers can turn to this community
560 for help when doing their research.
562 ** COMMENT =CORTEX= is based on jMonkeyEngine3
564 While preparing to build =CORTEX= I studied several video game
565 engines to see which would best serve as a base. The top contenders
566 were:
568 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
569 software in 1997. All the source code was released by ID
570 software into the Public Domain several years ago, and as a
571 result it has been ported to many different languages. This
572 engine was famous for its advanced use of realistic shading
573 and had decent and fast physics simulation. The main advantage
574 of the Quake II engine is its simplicity, but I ultimately
575 rejected it because the engine is too tied to the concept of a
576 first-person shooter game. One of the problems I had was that
577 there does not seem to be any easy way to attach multiple
578 cameras to a single character. There are also several physics
579 clipping issues that are corrected in a way that only applies
580 to the main character and do not apply to arbitrary objects.
582 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
583 and Quake I engines and is used by Valve in the Half-Life
584 series of games. The physics simulation in the Source Engine
585 is quite accurate and probably the best out of all the engines
586 I investigated. There is also an extensive community actively
587 working with the engine. However, applications that use the
588 Source Engine must be written in C++, the code is not open, it
589 only runs on Windows, and the tools that come with the SDK to
590 handle models and textures are complicated and awkward to use.
592 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
593 games in Java. It uses OpenGL to render to the screen and uses
594 screengraphs to avoid drawing things that do not appear on the
595 screen. It has an active community and several games in the
596 pipeline. The engine was not built to serve any particular
597 game but is instead meant to be used for any 3D game.
599 I chose jMonkeyEngine3 because it because it had the most features
600 out of all the free projects I looked at, and because I could then
601 write my code in clojure, an implementation of =LISP= that runs on
602 the JVM.
604 ** COMMENT =CORTEX= uses Blender to create creature models
606 For the simple worm-like creatures I will use later on in this
607 thesis, I could define a simple API in =CORTEX= that would allow
608 one to create boxes, spheres, etc., and leave that API as the sole
609 way to create creatures. However, for =CORTEX= to truly be useful
610 for other projects, it needs a way to construct complicated
611 creatures. If possible, it would be nice to leverage work that has
612 already been done by the community of 3D modelers, or at least
613 enable people who are talented at moedling but not programming to
614 design =CORTEX= creatures.
616 Therefore, I use Blender, a free 3D modeling program, as the main
617 way to create creatures in =CORTEX=. However, the creatures modeled
618 in Blender must also be simple to simulate in jMonkeyEngine3's game
619 engine, and must also be easy to rig with =CORTEX='s senses. I
620 accomplish this with extensive use of Blender's ``empty nodes.''
622 Empty nodes have no mass, physical presence, or appearance, but
623 they can hold metadata and have names. I use a tree structure of
624 empty nodes to specify senses in the following manner:
626 - Create a single top-level empty node whose name is the name of
627 the sense.
628 - Add empty nodes which each contain meta-data relevant to the
629 sense, including a UV-map describing the number/distribution of
630 sensors if applicable.
631 - Make each empty-node the child of the top-level node.
633 #+caption: An example of annoting a creature model with empty
634 #+caption: nodes to describe the layout of senses. There are
635 #+caption: multiple empty nodes which each describe the position
636 #+caption: of muscles, ears, eyes, or joints.
637 #+name: sense-nodes
638 #+ATTR_LaTeX: :width 10cm
639 [[./images/empty-sense-nodes.png]]
641 ** COMMENT Bodies are composed of segments connected by joints
643 Blender is a general purpose animation tool, which has been used in
644 the past to create high quality movies such as Sintel
645 \cite{sintel}. Though Blender can model and render even complicated
646 things like water, it is crucual to keep models that are meant to
647 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
648 though jMonkeyEngine3, is a rigid-body physics system. This offers
649 a compromise between the expressiveness of a game level and the
650 speed at which it can be simulated, and it means that creatures
651 should be naturally expressed as rigid components held together by
652 joint constraints.
654 But humans are more like a squishy bag with wrapped around some
655 hard bones which define the overall shape. When we move, our skin
656 bends and stretches to accomodate the new positions of our bones.
658 One way to make bodies composed of rigid pieces connected by joints
659 /seem/ more human-like is to use an /armature/, (or /rigging/)
660 system, which defines a overall ``body mesh'' and defines how the
661 mesh deforms as a function of the position of each ``bone'' which
662 is a standard rigid body. This technique is used extensively to
663 model humans and create realistic animations. It is not a good
664 technique for physical simulation, however because it creates a lie
665 -- the skin is not a physical part of the simulation and does not
666 interact with any objects in the world or itself. Objects will pass
667 right though the skin until they come in contact with the
668 underlying bone, which is a physical object. Whithout simulating
669 the skin, the sense of touch has little meaning, and the creature's
670 own vision will lie to it about the true extent of its body.
671 Simulating the skin as a physical object requires some way to
672 continuously update the physical model of the skin along with the
673 movement of the bones, which is unacceptably slow compared to rigid
674 body simulation.
676 Therefore, instead of using the human-like ``deformable bag of
677 bones'' approach, I decided to base my body plans on multiple solid
678 objects that are connected by joints, inspired by the robot =EVE=
679 from the movie WALL-E.
681 #+caption: =EVE= from the movie WALL-E. This body plan turns
682 #+caption: out to be much better suited to my purposes than a more
683 #+caption: human-like one.
684 #+ATTR_LaTeX: :width 10cm
685 [[./images/Eve.jpg]]
687 =EVE='s body is composed of several rigid components that are held
688 together by invisible joint constraints. This is what I mean by
689 ``eve-like''. The main reason that I use eve-style bodies is for
690 efficiency, and so that there will be correspondence between the
691 AI's semses and the physical presence of its body. Each individual
692 section is simulated by a separate rigid body that corresponds
693 exactly with its visual representation and does not change.
694 Sections are connected by invisible joints that are well supported
695 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
696 can efficiently simulate hundreds of rigid bodies connected by
697 joints. Just because sections are rigid does not mean they have to
698 stay as one piece forever; they can be dynamically replaced with
699 multiple sections to simulate splitting in two. This could be used
700 to simulate retractable claws or =EVE='s hands, which are able to
701 coalesce into one object in the movie.
703 *** Solidifying/Connecting a body
705 =CORTEX= creates a creature in two steps: first, it traverses the
706 nodes in the blender file and creates physical representations for
707 any of them that have mass defined in their blender meta-data.
709 #+caption: Program for iterating through the nodes in a blender file
710 #+caption: and generating physical jMonkeyEngine3 objects with mass
711 #+caption: and a matching physics shape.
712 #+name: name
713 #+begin_listing clojure
714 #+begin_src clojure
715 (defn physical!
716 "Iterate through the nodes in creature and make them real physical
717 objects in the simulation."
718 [#^Node creature]
719 (dorun
720 (map
721 (fn [geom]
722 (let [physics-control
723 (RigidBodyControl.
724 (HullCollisionShape.
725 (.getMesh geom))
726 (if-let [mass (meta-data geom "mass")]
727 (float mass) (float 1)))]
728 (.addControl geom physics-control)))
729 (filter #(isa? (class %) Geometry )
730 (node-seq creature)))))
731 #+end_src
732 #+end_listing
734 The next step to making a proper body is to connect those pieces
735 together with joints. jMonkeyEngine has a large array of joints
736 available via =bullet=, such as Point2Point, Cone, Hinge, and a
737 generic Six Degree of Freedom joint, with or without spring
738 restitution.
740 Joints are treated a lot like proper senses, in that there is a
741 top-level empty node named ``joints'' whose children each
742 represent a joint.
744 #+caption: View of the hand model in Blender showing the main ``joints''
745 #+caption: node (highlighted in yellow) and its children which each
746 #+caption: represent a joint in the hand. Each joint node has metadata
747 #+caption: specifying what sort of joint it is.
748 #+name: blender-hand
749 #+ATTR_LaTeX: :width 10cm
750 [[./images/hand-screenshot1.png]]
753 =CORTEX='s procedure for binding the creature together with joints
754 is as follows:
756 - Find the children of the ``joints'' node.
757 - Determine the two spatials the joint is meant to connect.
758 - Create the joint based on the meta-data of the empty node.
760 The higher order function =sense-nodes= from =cortex.sense=
761 simplifies finding the joints based on their parent ``joints''
762 node.
764 #+caption: Retrieving the children empty nodes from a single
765 #+caption: named empty node is a common pattern in =CORTEX=
766 #+caption: further instances of this technique for the senses
767 #+caption: will be omitted
768 #+name: get-empty-nodes
769 #+begin_listing clojure
770 #+begin_src clojure
771 (defn sense-nodes
772 "For some senses there is a special empty blender node whose
773 children are considered markers for an instance of that sense. This
774 function generates functions to find those children, given the name
775 of the special parent node."
776 [parent-name]
777 (fn [#^Node creature]
778 (if-let [sense-node (.getChild creature parent-name)]
779 (seq (.getChildren sense-node)) [])))
781 (def
782 ^{:doc "Return the children of the creature's \"joints\" node."
783 :arglists '([creature])}
784 joints
785 (sense-nodes "joints"))
786 #+end_src
787 #+end_listing
789 To find a joint's targets, =CORTEX= creates a small cube, centered
790 around the empty-node, and grows the cube exponentially until it
791 intersects two physical objects. The objects are ordered according
792 to the joint's rotation, with the first one being the object that
793 has more negative coordinates in the joint's reference frame.
794 Since the objects must be physical, the empty-node itself escapes
795 detection. Because the objects must be physical, =joint-targets=
796 must be called /after/ =physical!= is called.
798 #+caption: Program to find the targets of a joint node by
799 #+caption: exponentiallly growth of a search cube.
800 #+name: joint-targets
801 #+begin_listing clojure
802 #+begin_src clojure
803 (defn joint-targets
804 "Return the two closest two objects to the joint object, ordered
805 from bottom to top according to the joint's rotation."
806 [#^Node parts #^Node joint]
807 (loop [radius (float 0.01)]
808 (let [results (CollisionResults.)]
809 (.collideWith
810 parts
811 (BoundingBox. (.getWorldTranslation joint)
812 radius radius radius) results)
813 (let [targets
814 (distinct
815 (map #(.getGeometry %) results))]
816 (if (>= (count targets) 2)
817 (sort-by
818 #(let [joint-ref-frame-position
819 (jme-to-blender
820 (.mult
821 (.inverse (.getWorldRotation joint))
822 (.subtract (.getWorldTranslation %)
823 (.getWorldTranslation joint))))]
824 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
825 (take 2 targets))
826 (recur (float (* radius 2))))))))
827 #+end_src
828 #+end_listing
830 Once =CORTEX= finds all joints and targets, it creates them using
831 a dispatch on the metadata of each joint node.
833 #+caption: Program to dispatch on blender metadata and create joints
834 #+caption: sutiable for physical simulation.
835 #+name: joint-dispatch
836 #+begin_listing clojure
837 #+begin_src clojure
838 (defmulti joint-dispatch
839 "Translate blender pseudo-joints into real JME joints."
840 (fn [constraints & _]
841 (:type constraints)))
843 (defmethod joint-dispatch :point
844 [constraints control-a control-b pivot-a pivot-b rotation]
845 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
846 (.setLinearLowerLimit Vector3f/ZERO)
847 (.setLinearUpperLimit Vector3f/ZERO)))
849 (defmethod joint-dispatch :hinge
850 [constraints control-a control-b pivot-a pivot-b rotation]
851 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
852 [limit-1 limit-2] (:limit constraints)
853 hinge-axis (.mult rotation (blender-to-jme axis))]
854 (doto (HingeJoint. control-a control-b pivot-a pivot-b
855 hinge-axis hinge-axis)
856 (.setLimit limit-1 limit-2))))
858 (defmethod joint-dispatch :cone
859 [constraints control-a control-b pivot-a pivot-b rotation]
860 (let [limit-xz (:limit-xz constraints)
861 limit-xy (:limit-xy constraints)
862 twist (:twist constraints)]
863 (doto (ConeJoint. control-a control-b pivot-a pivot-b
864 rotation rotation)
865 (.setLimit (float limit-xz) (float limit-xy)
866 (float twist)))))
867 #+end_src
868 #+end_listing
870 All that is left for joints it to combine the above pieces into a
871 something that can operate on the collection of nodes that a
872 blender file represents.
874 #+caption: Program to completely create a joint given information
875 #+caption: from a blender file.
876 #+name: connect
877 #+begin_listing clojure
878 #+begin_src clojure
879 (defn connect
880 "Create a joint between 'obj-a and 'obj-b at the location of
881 'joint. The type of joint is determined by the metadata on 'joint.
883 Here are some examples:
884 {:type :point}
885 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
886 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
888 {:type :cone :limit-xz 0]
889 :limit-xy 0]
890 :twist 0]} (use XZY rotation mode in blender!)"
891 [#^Node obj-a #^Node obj-b #^Node joint]
892 (let [control-a (.getControl obj-a RigidBodyControl)
893 control-b (.getControl obj-b RigidBodyControl)
894 joint-center (.getWorldTranslation joint)
895 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
896 pivot-a (world-to-local obj-a joint-center)
897 pivot-b (world-to-local obj-b joint-center)]
898 (if-let
899 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
900 ;; A side-effect of creating a joint registers
901 ;; it with both physics objects which in turn
902 ;; will register the joint with the physics system
903 ;; when the simulation is started.
904 (joint-dispatch constraints
905 control-a control-b
906 pivot-a pivot-b
907 joint-rotation))))
908 #+end_src
909 #+end_listing
911 In general, whenever =CORTEX= exposes a sense (or in this case
912 physicality), it provides a function of the type =sense!=, which
913 takes in a collection of nodes and augments it to support that
914 sense. The function returns any controlls necessary to use that
915 sense. In this case =body!= cerates a physical body and returns no
916 control functions.
918 #+caption: Program to give joints to a creature.
919 #+name: name
920 #+begin_listing clojure
921 #+begin_src clojure
922 (defn joints!
923 "Connect the solid parts of the creature with physical joints. The
924 joints are taken from the \"joints\" node in the creature."
925 [#^Node creature]
926 (dorun
927 (map
928 (fn [joint]
929 (let [[obj-a obj-b] (joint-targets creature joint)]
930 (connect obj-a obj-b joint)))
931 (joints creature))))
932 (defn body!
933 "Endow the creature with a physical body connected with joints. The
934 particulars of the joints and the masses of each body part are
935 determined in blender."
936 [#^Node creature]
937 (physical! creature)
938 (joints! creature))
939 #+end_src
940 #+end_listing
942 All of the code you have just seen amounts to only 130 lines, yet
943 because it builds on top of Blender and jMonkeyEngine3, those few
944 lines pack quite a punch!
946 The hand from figure \ref{blender-hand}, which was modeled after
947 my own right hand, can now be given joints and simulated as a
948 creature.
950 #+caption: With the ability to create physical creatures from blender,
951 #+caption: =CORTEX= gets one step closer to becomming a full creature
952 #+caption: simulation environment.
953 #+name: name
954 #+ATTR_LaTeX: :width 15cm
955 [[./images/physical-hand.png]]
957 ** COMMENT Eyes reuse standard video game components
959 Vision is one of the most important senses for humans, so I need to
960 build a simulated sense of vision for my AI. I will do this with
961 simulated eyes. Each eye can be independently moved and should see
962 its own version of the world depending on where it is.
964 Making these simulated eyes a reality is simple because
965 jMonkeyEngine already contains extensive support for multiple views
966 of the same 3D simulated world. The reason jMonkeyEngine has this
967 support is because the support is necessary to create games with
968 split-screen views. Multiple views are also used to create
969 efficient pseudo-reflections by rendering the scene from a certain
970 perspective and then projecting it back onto a surface in the 3D
971 world.
973 #+caption: jMonkeyEngine supports multiple views to enable
974 #+caption: split-screen games, like GoldenEye, which was one of
975 #+caption: the first games to use split-screen views.
976 #+name: name
977 #+ATTR_LaTeX: :width 10cm
978 [[./images/goldeneye-4-player.png]]
980 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
982 jMonkeyEngine allows you to create a =ViewPort=, which represents a
983 view of the simulated world. You can create as many of these as you
984 want. Every frame, the =RenderManager= iterates through each
985 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
986 is a =FrameBuffer= which represents the rendered image in the GPU.
988 #+caption: =ViewPorts= are cameras in the world. During each frame,
989 #+caption: the =RenderManager= records a snapshot of what each view
990 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
991 #+name: name
992 #+ATTR_LaTeX: :width 10cm
993 [[../images/diagram_rendermanager2.png]]
995 Each =ViewPort= can have any number of attached =SceneProcessor=
996 objects, which are called every time a new frame is rendered. A
997 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
998 whatever it wants to the data. Often this consists of invoking GPU
999 specific operations on the rendered image. The =SceneProcessor= can
1000 also copy the GPU image data to RAM and process it with the CPU.
1002 *** Appropriating Views for Vision
1004 Each eye in the simulated creature needs its own =ViewPort= so
1005 that it can see the world from its own perspective. To this
1006 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1007 any arbitrary continuation function for further processing. That
1008 continuation function may perform both CPU and GPU operations on
1009 the data. To make this easy for the continuation function, the
1010 =SceneProcessor= maintains appropriately sized buffers in RAM to
1011 hold the data. It does not do any copying from the GPU to the CPU
1012 itself because it is a slow operation.
1014 #+caption: Function to make the rendered secne in jMonkeyEngine
1015 #+caption: available for further processing.
1016 #+name: pipeline-1
1017 #+begin_listing clojure
1018 #+begin_src clojure
1019 (defn vision-pipeline
1020 "Create a SceneProcessor object which wraps a vision processing
1021 continuation function. The continuation is a function that takes
1022 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1023 each of which has already been appropriately sized."
1024 [continuation]
1025 (let [byte-buffer (atom nil)
1026 renderer (atom nil)
1027 image (atom nil)]
1028 (proxy [SceneProcessor] []
1029 (initialize
1030 [renderManager viewPort]
1031 (let [cam (.getCamera viewPort)
1032 width (.getWidth cam)
1033 height (.getHeight cam)]
1034 (reset! renderer (.getRenderer renderManager))
1035 (reset! byte-buffer
1036 (BufferUtils/createByteBuffer
1037 (* width height 4)))
1038 (reset! image (BufferedImage.
1039 width height
1040 BufferedImage/TYPE_4BYTE_ABGR))))
1041 (isInitialized [] (not (nil? @byte-buffer)))
1042 (reshape [_ _ _])
1043 (preFrame [_])
1044 (postQueue [_])
1045 (postFrame
1046 [#^FrameBuffer fb]
1047 (.clear @byte-buffer)
1048 (continuation @renderer fb @byte-buffer @image))
1049 (cleanup []))))
1050 #+end_src
1051 #+end_listing
1053 The continuation function given to =vision-pipeline= above will be
1054 given a =Renderer= and three containers for image data. The
1055 =FrameBuffer= references the GPU image data, but the pixel data
1056 can not be used directly on the CPU. The =ByteBuffer= and
1057 =BufferedImage= are initially "empty" but are sized to hold the
1058 data in the =FrameBuffer=. I call transferring the GPU image data
1059 to the CPU structures "mixing" the image data.
1061 *** Optical sensor arrays are described with images and referenced with metadata
1063 The vision pipeline described above handles the flow of rendered
1064 images. Now, =CORTEX= needs simulated eyes to serve as the source
1065 of these images.
1067 An eye is described in blender in the same way as a joint. They
1068 are zero dimensional empty objects with no geometry whose local
1069 coordinate system determines the orientation of the resulting eye.
1070 All eyes are children of a parent node named "eyes" just as all
1071 joints have a parent named "joints". An eye binds to the nearest
1072 physical object with =bind-sense=.
1074 #+caption: Here, the camera is created based on metadata on the
1075 #+caption: eye-node and attached to the nearest physical object
1076 #+caption: with =bind-sense=
1077 #+name: add-eye
1078 #+begin_listing clojure
1079 (defn add-eye!
1080 "Create a Camera centered on the current position of 'eye which
1081 follows the closest physical node in 'creature. The camera will
1082 point in the X direction and use the Z vector as up as determined
1083 by the rotation of these vectors in blender coordinate space. Use
1084 XZY rotation for the node in blender."
1085 [#^Node creature #^Spatial eye]
1086 (let [target (closest-node creature eye)
1087 [cam-width cam-height]
1088 ;;[640 480] ;; graphics card on laptop doesn't support
1089 ;; arbitray dimensions.
1090 (eye-dimensions eye)
1091 cam (Camera. cam-width cam-height)
1092 rot (.getWorldRotation eye)]
1093 (.setLocation cam (.getWorldTranslation eye))
1094 (.lookAtDirection
1095 cam ; this part is not a mistake and
1096 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1097 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1098 (.setFrustumPerspective
1099 cam (float 45)
1100 (float (/ (.getWidth cam) (.getHeight cam)))
1101 (float 1)
1102 (float 1000))
1103 (bind-sense target cam) cam))
1104 #+end_listing
1106 *** Simulated Retina
1108 An eye is a surface (the retina) which contains many discrete
1109 sensors to detect light. These sensors can have different
1110 light-sensing properties. In humans, each discrete sensor is
1111 sensitive to red, blue, green, or gray. These different types of
1112 sensors can have different spatial distributions along the retina.
1113 In humans, there is a fovea in the center of the retina which has
1114 a very high density of color sensors, and a blind spot which has
1115 no sensors at all. Sensor density decreases in proportion to
1116 distance from the fovea.
1118 I want to be able to model any retinal configuration, so my
1119 eye-nodes in blender contain metadata pointing to images that
1120 describe the precise position of the individual sensors using
1121 white pixels. The meta-data also describes the precise sensitivity
1122 to light that the sensors described in the image have. An eye can
1123 contain any number of these images. For example, the metadata for
1124 an eye might look like this:
1126 #+begin_src clojure
1127 {0xFF0000 "Models/test-creature/retina-small.png"}
1128 #+end_src
1130 #+caption: An example retinal profile image. White pixels are
1131 #+caption: photo-sensitive elements. The distribution of white
1132 #+caption: pixels is denser in the middle and falls off at the
1133 #+caption: edges and is inspired by the human retina.
1134 #+name: retina
1135 #+ATTR_LaTeX: :width 10cm
1136 [[./images/retina-small.png]]
1138 Together, the number 0xFF0000 and the image image above describe
1139 the placement of red-sensitive sensory elements.
1141 Meta-data to very crudely approximate a human eye might be
1142 something like this:
1144 #+begin_src clojure
1145 (let [retinal-profile "Models/test-creature/retina-small.png"]
1146 {0xFF0000 retinal-profile
1147 0x00FF00 retinal-profile
1148 0x0000FF retinal-profile
1149 0xFFFFFF retinal-profile})
1150 #+end_src
1152 The numbers that serve as keys in the map determine a sensor's
1153 relative sensitivity to the channels red, green, and blue. These
1154 sensitivity values are packed into an integer in the order
1155 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1156 image are added together with these sensitivities as linear
1157 weights. Therefore, 0xFF0000 means sensitive to red only while
1158 0xFFFFFF means sensitive to all colors equally (gray).
1160 #+caption: This is the core of vision in =CORTEX=. A given eye node
1161 #+caption: is converted into a function that returns visual
1162 #+caption: information from the simulation.
1163 #+name: vision-kernel
1164 #+begin_listing clojure
1165 (defn vision-kernel
1166 "Returns a list of functions, each of which will return a color
1167 channel's worth of visual information when called inside a running
1168 simulation."
1169 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1170 (let [retinal-map (retina-sensor-profile eye)
1171 camera (add-eye! creature eye)
1172 vision-image
1173 (atom
1174 (BufferedImage. (.getWidth camera)
1175 (.getHeight camera)
1176 BufferedImage/TYPE_BYTE_BINARY))
1177 register-eye!
1178 (runonce
1179 (fn [world]
1180 (add-camera!
1181 world camera
1182 (let [counter (atom 0)]
1183 (fn [r fb bb bi]
1184 (if (zero? (rem (swap! counter inc) (inc skip)))
1185 (reset! vision-image
1186 (BufferedImage! r fb bb bi))))))))]
1187 (vec
1188 (map
1189 (fn [[key image]]
1190 (let [whites (white-coordinates image)
1191 topology (vec (collapse whites))
1192 sensitivity (sensitivity-presets key key)]
1193 (attached-viewport.
1194 (fn [world]
1195 (register-eye! world)
1196 (vector
1197 topology
1198 (vec
1199 (for [[x y] whites]
1200 (pixel-sense
1201 sensitivity
1202 (.getRGB @vision-image x y))))))
1203 register-eye!)))
1204 retinal-map))))
1205 #+end_listing
1207 Note that since each of the functions generated by =vision-kernel=
1208 shares the same =register-eye!= function, the eye will be
1209 registered only once the first time any of the functions from the
1210 list returned by =vision-kernel= is called. Each of the functions
1211 returned by =vision-kernel= also allows access to the =Viewport=
1212 through which it receives images.
1214 All the hard work has been done; all that remains is to apply
1215 =vision-kernel= to each eye in the creature and gather the results
1216 into one list of functions.
1219 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1220 #+caption: environment for experimenting with different types of
1221 #+caption: eyes.
1222 #+name: vision!
1223 #+begin_listing clojure
1224 (defn vision!
1225 "Returns a list of functions, each of which returns visual sensory
1226 data when called inside a running simulation."
1227 [#^Node creature & {skip :skip :or {skip 0}}]
1228 (reduce
1229 concat
1230 (for [eye (eyes creature)]
1231 (vision-kernel creature eye))))
1232 #+end_listing
1234 #+caption: Simulated vision with a test creature and the
1235 #+caption: human-like eye approximation. Notice how each channel
1236 #+caption: of the eye responds differently to the differently
1237 #+caption: colored balls.
1238 #+name: worm-vision-test.
1239 #+ATTR_LaTeX: :width 13cm
1240 [[./images/worm-vision.png]]
1242 The vision code is not much more complicated than the body code,
1243 and enables multiple further paths for simulated vision. For
1244 example, it is quite easy to create bifocal vision -- you just
1245 make two eyes next to each other in blender! It is also possible
1246 to encode vision transforms in the retinal files. For example, the
1247 human like retina file in figure \ref{retina} approximates a
1248 log-polar transform.
1250 This vision code has already been absorbed by the jMonkeyEngine
1251 community and is now (in modified form) part of a system for
1252 capturing in-game video to a file.
1254 ** Hearing is hard; =CORTEX= does it right
1256 At the end of this section I will have simulated ears that work the
1257 same way as the simulated eyes in the last section. I will be able to
1258 place any number of ear-nodes in a blender file, and they will bind to
1259 the closest physical object and follow it as it moves around. Each ear
1260 will provide access to the sound data it picks up between every frame.
1262 Hearing is one of the more difficult senses to simulate, because there
1263 is less support for obtaining the actual sound data that is processed
1264 by jMonkeyEngine3. There is no "split-screen" support for rendering
1265 sound from different points of view, and there is no way to directly
1266 access the rendered sound data.
1268 =CORTEX='s hearing is unique because it does not have any
1269 limitations compared to other simulation environments. As far as I
1270 know, there is no other system that supports multiple listerers,
1271 and the sound demo at the end of this section is the first time
1272 it's been done in a video game environment.
1274 *** Brief Description of jMonkeyEngine's Sound System
1276 jMonkeyEngine's sound system works as follows:
1278 - jMonkeyEngine uses the =AppSettings= for the particular
1279 application to determine what sort of =AudioRenderer= should be
1280 used.
1281 - Although some support is provided for multiple AudioRendering
1282 backends, jMonkeyEngine at the time of this writing will either
1283 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1284 - jMonkeyEngine tries to figure out what sort of system you're
1285 running and extracts the appropriate native libraries.
1286 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1287 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1288 - =OpenAL= renders the 3D sound and feeds the rendered sound
1289 directly to any of various sound output devices with which it
1290 knows how to communicate.
1292 A consequence of this is that there's no way to access the actual
1293 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1294 one /listener/ (it renders sound data from only one perspective),
1295 which normally isn't a problem for games, but becomes a problem
1296 when trying to make multiple AI creatures that can each hear the
1297 world from a different perspective.
1299 To make many AI creatures in jMonkeyEngine that can each hear the
1300 world from their own perspective, or to make a single creature with
1301 many ears, it is necessary to go all the way back to =OpenAL= and
1302 implement support for simulated hearing there.
1304 *** Extending =OpenAl=
1306 Extending =OpenAL= to support multiple listeners requires 500
1307 lines of =C= code and is too hairy to mention here. Instead, I
1308 will show a small amount of extension code and go over the high
1309 level stragety. Full source is of course available with the
1310 =CORTEX= distribution if you're interested.
1312 =OpenAL= goes to great lengths to support many different systems,
1313 all with different sound capabilities and interfaces. It
1314 accomplishes this difficult task by providing code for many
1315 different sound backends in pseudo-objects called /Devices/.
1316 There's a device for the Linux Open Sound System and the Advanced
1317 Linux Sound Architecture, there's one for Direct Sound on Windows,
1318 and there's even one for Solaris. =OpenAL= solves the problem of
1319 platform independence by providing all these Devices.
1321 Wrapper libraries such as LWJGL are free to examine the system on
1322 which they are running and then select an appropriate device for
1323 that system.
1325 There are also a few "special" devices that don't interface with
1326 any particular system. These include the Null Device, which
1327 doesn't do anything, and the Wave Device, which writes whatever
1328 sound it receives to a file, if everything has been set up
1329 correctly when configuring =OpenAL=.
1331 Actual mixing (doppler shift and distance.environment-based
1332 attenuation) of the sound data happens in the Devices, and they
1333 are the only point in the sound rendering process where this data
1334 is available.
1336 Therefore, in order to support multiple listeners, and get the
1337 sound data in a form that the AIs can use, it is necessary to
1338 create a new Device which supports this feature.
1340 Adding a device to OpenAL is rather tricky -- there are five
1341 separate files in the =OpenAL= source tree that must be modified
1342 to do so. I named my device the "Multiple Audio Send" Device, or
1343 =Send= Device for short, since it sends audio data back to the
1344 calling application like an Aux-Send cable on a mixing board.
1346 The main idea behind the Send device is to take advantage of the
1347 fact that LWJGL only manages one /context/ when using OpenAL. A
1348 /context/ is like a container that holds samples and keeps track
1349 of where the listener is. In order to support multiple listeners,
1350 the Send device identifies the LWJGL context as the master
1351 context, and creates any number of slave contexts to represent
1352 additional listeners. Every time the device renders sound, it
1353 synchronizes every source from the master LWJGL context to the
1354 slave contexts. Then, it renders each context separately, using a
1355 different listener for each one. The rendered sound is made
1356 available via JNI to jMonkeyEngine.
1358 Switching between contexts is not the normal operation of a
1359 Device, and one of the problems with doing so is that a Device
1360 normally keeps around a few pieces of state such as the
1361 =ClickRemoval= array above which will become corrupted if the
1362 contexts are not rendered in parallel. The solution is to create a
1363 copy of this normally global device state for each context, and
1364 copy it back and forth into and out of the actual device state
1365 whenever a context is rendered.
1367 The core of the =Send= device is the =syncSources= function, which
1368 does the job of copying all relevant data from one context to
1369 another.
1371 #+caption: Program for extending =OpenAL= to support multiple
1372 #+caption: listeners via context copying/switching.
1373 #+name: sync-openal-sources
1374 #+begin_listing C
1375 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1376 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1377 ALuint master = masterSource->source;
1378 ALuint slave = slaveSource->source;
1379 ALCcontext *current = alcGetCurrentContext();
1381 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1382 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1383 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1384 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1385 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1386 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1387 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1388 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1389 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1390 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1391 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1392 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1393 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1395 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1396 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1397 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1399 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1400 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1402 alcMakeContextCurrent(masterCtx);
1403 ALint source_type;
1404 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1406 // Only static sources are currently synchronized!
1407 if (AL_STATIC == source_type){
1408 ALint master_buffer;
1409 ALint slave_buffer;
1410 alGetSourcei(master, AL_BUFFER, &master_buffer);
1411 alcMakeContextCurrent(slaveCtx);
1412 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1413 if (master_buffer != slave_buffer){
1414 alSourcei(slave, AL_BUFFER, master_buffer);
1418 // Synchronize the state of the two sources.
1419 alcMakeContextCurrent(masterCtx);
1420 ALint masterState;
1421 ALint slaveState;
1423 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1424 alcMakeContextCurrent(slaveCtx);
1425 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1427 if (masterState != slaveState){
1428 switch (masterState){
1429 case AL_INITIAL : alSourceRewind(slave); break;
1430 case AL_PLAYING : alSourcePlay(slave); break;
1431 case AL_PAUSED : alSourcePause(slave); break;
1432 case AL_STOPPED : alSourceStop(slave); break;
1435 // Restore whatever context was previously active.
1436 alcMakeContextCurrent(current);
1438 #+end_listing
1440 With this special context-switching device, and some ugly JNI
1441 bindings that are not worth mentioning, =CORTEX= gains the ability
1442 to access multiple sound streams from =OpenAL=.
1444 #+caption: Program to create an ear from a blender empty node. The ear
1445 #+caption: follows around the nearest physical object and passes
1446 #+caption: all sensory data to a continuation function.
1447 #+name: add-ear
1448 #+begin_listing clojure
1449 (defn add-ear!
1450 "Create a Listener centered on the current position of 'ear
1451 which follows the closest physical node in 'creature and
1452 sends sound data to 'continuation."
1453 [#^Application world #^Node creature #^Spatial ear continuation]
1454 (let [target (closest-node creature ear)
1455 lis (Listener.)
1456 audio-renderer (.getAudioRenderer world)
1457 sp (hearing-pipeline continuation)]
1458 (.setLocation lis (.getWorldTranslation ear))
1459 (.setRotation lis (.getWorldRotation ear))
1460 (bind-sense target lis)
1461 (update-listener-velocity! target lis)
1462 (.addListener audio-renderer lis)
1463 (.registerSoundProcessor audio-renderer lis sp)))
1464 #+end_listing
1467 The =Send= device, unlike most of the other devices in =OpenAL=,
1468 does not render sound unless asked. This enables the system to
1469 slow down or speed up depending on the needs of the AIs who are
1470 using it to listen. If the device tried to render samples in
1471 real-time, a complicated AI whose mind takes 100 seconds of
1472 computer time to simulate 1 second of AI-time would miss almost
1473 all of the sound in its environment!
1475 #+caption: Program to enable arbitrary hearing in =CORTEX=
1476 #+name: hearing
1477 #+begin_listing clojure
1478 (defn hearing-kernel
1479 "Returns a function which returns auditory sensory data when called
1480 inside a running simulation."
1481 [#^Node creature #^Spatial ear]
1482 (let [hearing-data (atom [])
1483 register-listener!
1484 (runonce
1485 (fn [#^Application world]
1486 (add-ear!
1487 world creature ear
1488 (comp #(reset! hearing-data %)
1489 byteBuffer->pulse-vector))))]
1490 (fn [#^Application world]
1491 (register-listener! world)
1492 (let [data @hearing-data
1493 topology
1494 (vec (map #(vector % 0) (range 0 (count data))))]
1495 [topology data]))))
1497 (defn hearing!
1498 "Endow the creature in a particular world with the sense of
1499 hearing. Will return a sequence of functions, one for each ear,
1500 which when called will return the auditory data from that ear."
1501 [#^Node creature]
1502 (for [ear (ears creature)]
1503 (hearing-kernel creature ear)))
1504 #+end_listing
1506 Armed with these functions, =CORTEX= is able to test possibly the
1507 first ever instance of multiple listeners in a video game engine
1508 based simulation!
1510 #+caption: Here a simple creature responds to sound by changing
1511 #+caption: its color from gray to green when the total volume
1512 #+caption: goes over a threshold.
1513 #+name: sound-test
1514 #+begin_listing java
1515 /**
1516 * Respond to sound! This is the brain of an AI entity that
1517 * hears its surroundings and reacts to them.
1518 */
1519 public void process(ByteBuffer audioSamples,
1520 int numSamples, AudioFormat format) {
1521 audioSamples.clear();
1522 byte[] data = new byte[numSamples];
1523 float[] out = new float[numSamples];
1524 audioSamples.get(data);
1525 FloatSampleTools.
1526 byte2floatInterleaved
1527 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1529 float max = Float.NEGATIVE_INFINITY;
1530 for (float f : out){if (f > max) max = f;}
1531 audioSamples.clear();
1533 if (max > 0.1){
1534 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1536 else {
1537 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1539 #+end_listing
1541 #+caption: First ever simulation of multiple listerners in =CORTEX=.
1542 #+caption: Each cube is a creature which processes sound data with
1543 #+caption: the =process= function from listing \ref{sound-test}.
1544 #+caption: the ball is constantally emiting a pure tone of
1545 #+caption: constant volume. As it approaches the cubes, they each
1546 #+caption: change color in response to the sound.
1547 #+name: sound-cubes.
1548 #+ATTR_LaTeX: :width 10cm
1549 [[./images/aurellem-gray.png]]
1551 This system of hearing has also been co-opted by the
1552 jMonkeyEngine3 community and is used to record audio for demo
1553 videos.
1555 ** Touch uses hundreds of hair-like elements
1557 ** Proprioception is the sense that makes everything ``real''
1559 ** Muscles are both effectors and sensors
1561 ** =CORTEX= brings complex creatures to life!
1563 ** =CORTEX= enables many possiblities for further research
1565 * COMMENT Empathy in a simulated worm
1567 Here I develop a computational model of empathy, using =CORTEX= as a
1568 base. Empathy in this context is the ability to observe another
1569 creature and infer what sorts of sensations that creature is
1570 feeling. My empathy algorithm involves multiple phases. First is
1571 free-play, where the creature moves around and gains sensory
1572 experience. From this experience I construct a representation of the
1573 creature's sensory state space, which I call \Phi-space. Using
1574 \Phi-space, I construct an efficient function which takes the
1575 limited data that comes from observing another creature and enriches
1576 it full compliment of imagined sensory data. I can then use the
1577 imagined sensory data to recognize what the observed creature is
1578 doing and feeling, using straightforward embodied action predicates.
1579 This is all demonstrated with using a simple worm-like creature, and
1580 recognizing worm-actions based on limited data.
1582 #+caption: Here is the worm with which we will be working.
1583 #+caption: It is composed of 5 segments. Each segment has a
1584 #+caption: pair of extensor and flexor muscles. Each of the
1585 #+caption: worm's four joints is a hinge joint which allows
1586 #+caption: about 30 degrees of rotation to either side. Each segment
1587 #+caption: of the worm is touch-capable and has a uniform
1588 #+caption: distribution of touch sensors on each of its faces.
1589 #+caption: Each joint has a proprioceptive sense to detect
1590 #+caption: relative positions. The worm segments are all the
1591 #+caption: same except for the first one, which has a much
1592 #+caption: higher weight than the others to allow for easy
1593 #+caption: manual motor control.
1594 #+name: basic-worm-view
1595 #+ATTR_LaTeX: :width 10cm
1596 [[./images/basic-worm-view.png]]
1598 #+caption: Program for reading a worm from a blender file and
1599 #+caption: outfitting it with the senses of proprioception,
1600 #+caption: touch, and the ability to move, as specified in the
1601 #+caption: blender file.
1602 #+name: get-worm
1603 #+begin_listing clojure
1604 #+begin_src clojure
1605 (defn worm []
1606 (let [model (load-blender-model "Models/worm/worm.blend")]
1607 {:body (doto model (body!))
1608 :touch (touch! model)
1609 :proprioception (proprioception! model)
1610 :muscles (movement! model)}))
1611 #+end_src
1612 #+end_listing
1614 ** Embodiment factors action recognition into managable parts
1616 Using empathy, I divide the problem of action recognition into a
1617 recognition process expressed in the language of a full compliment
1618 of senses, and an imaganitive process that generates full sensory
1619 data from partial sensory data. Splitting the action recognition
1620 problem in this manner greatly reduces the total amount of work to
1621 recognize actions: The imaganitive process is mostly just matching
1622 previous experience, and the recognition process gets to use all
1623 the senses to directly describe any action.
1625 ** Action recognition is easy with a full gamut of senses
1627 Embodied representations using multiple senses such as touch,
1628 proprioception, and muscle tension turns out be be exceedingly
1629 efficient at describing body-centered actions. It is the ``right
1630 language for the job''. For example, it takes only around 5 lines
1631 of LISP code to describe the action of ``curling'' using embodied
1632 primitives. It takes about 10 lines to describe the seemingly
1633 complicated action of wiggling.
1635 The following action predicates each take a stream of sensory
1636 experience, observe however much of it they desire, and decide
1637 whether the worm is doing the action they describe. =curled?=
1638 relies on proprioception, =resting?= relies on touch, =wiggling?=
1639 relies on a fourier analysis of muscle contraction, and
1640 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
1642 #+caption: Program for detecting whether the worm is curled. This is the
1643 #+caption: simplest action predicate, because it only uses the last frame
1644 #+caption: of sensory experience, and only uses proprioceptive data. Even
1645 #+caption: this simple predicate, however, is automatically frame
1646 #+caption: independent and ignores vermopomorphic differences such as
1647 #+caption: worm textures and colors.
1648 #+name: curled
1649 #+attr_latex: [htpb]
1650 #+begin_listing clojure
1651 #+begin_src clojure
1652 (defn curled?
1653 "Is the worm curled up?"
1654 [experiences]
1655 (every?
1656 (fn [[_ _ bend]]
1657 (> (Math/sin bend) 0.64))
1658 (:proprioception (peek experiences))))
1659 #+end_src
1660 #+end_listing
1662 #+caption: Program for summarizing the touch information in a patch
1663 #+caption: of skin.
1664 #+name: touch-summary
1665 #+attr_latex: [htpb]
1667 #+begin_listing clojure
1668 #+begin_src clojure
1669 (defn contact
1670 "Determine how much contact a particular worm segment has with
1671 other objects. Returns a value between 0 and 1, where 1 is full
1672 contact and 0 is no contact."
1673 [touch-region [coords contact :as touch]]
1674 (-> (zipmap coords contact)
1675 (select-keys touch-region)
1676 (vals)
1677 (#(map first %))
1678 (average)
1679 (* 10)
1680 (- 1)
1681 (Math/abs)))
1682 #+end_src
1683 #+end_listing
1686 #+caption: Program for detecting whether the worm is at rest. This program
1687 #+caption: uses a summary of the tactile information from the underbelly
1688 #+caption: of the worm, and is only true if every segment is touching the
1689 #+caption: floor. Note that this function contains no references to
1690 #+caption: proprioction at all.
1691 #+name: resting
1692 #+attr_latex: [htpb]
1693 #+begin_listing clojure
1694 #+begin_src clojure
1695 (def worm-segment-bottom (rect-region [8 15] [14 22]))
1697 (defn resting?
1698 "Is the worm resting on the ground?"
1699 [experiences]
1700 (every?
1701 (fn [touch-data]
1702 (< 0.9 (contact worm-segment-bottom touch-data)))
1703 (:touch (peek experiences))))
1704 #+end_src
1705 #+end_listing
1707 #+caption: Program for detecting whether the worm is curled up into a
1708 #+caption: full circle. Here the embodied approach begins to shine, as
1709 #+caption: I am able to both use a previous action predicate (=curled?=)
1710 #+caption: as well as the direct tactile experience of the head and tail.
1711 #+name: grand-circle
1712 #+attr_latex: [htpb]
1713 #+begin_listing clojure
1714 #+begin_src clojure
1715 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
1717 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
1719 (defn grand-circle?
1720 "Does the worm form a majestic circle (one end touching the other)?"
1721 [experiences]
1722 (and (curled? experiences)
1723 (let [worm-touch (:touch (peek experiences))
1724 tail-touch (worm-touch 0)
1725 head-touch (worm-touch 4)]
1726 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
1727 (< 0.55 (contact worm-segment-top-tip head-touch))))))
1728 #+end_src
1729 #+end_listing
1732 #+caption: Program for detecting whether the worm has been wiggling for
1733 #+caption: the last few frames. It uses a fourier analysis of the muscle
1734 #+caption: contractions of the worm's tail to determine wiggling. This is
1735 #+caption: signigicant because there is no particular frame that clearly
1736 #+caption: indicates that the worm is wiggling --- only when multiple frames
1737 #+caption: are analyzed together is the wiggling revealed. Defining
1738 #+caption: wiggling this way also gives the worm an opportunity to learn
1739 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
1740 #+caption: wiggle but can't. Frustrated wiggling is very visually different
1741 #+caption: from actual wiggling, but this definition gives it to us for free.
1742 #+name: wiggling
1743 #+attr_latex: [htpb]
1744 #+begin_listing clojure
1745 #+begin_src clojure
1746 (defn fft [nums]
1747 (map
1748 #(.getReal %)
1749 (.transform
1750 (FastFourierTransformer. DftNormalization/STANDARD)
1751 (double-array nums) TransformType/FORWARD)))
1753 (def indexed (partial map-indexed vector))
1755 (defn max-indexed [s]
1756 (first (sort-by (comp - second) (indexed s))))
1758 (defn wiggling?
1759 "Is the worm wiggling?"
1760 [experiences]
1761 (let [analysis-interval 0x40]
1762 (when (> (count experiences) analysis-interval)
1763 (let [a-flex 3
1764 a-ex 2
1765 muscle-activity
1766 (map :muscle (vector:last-n experiences analysis-interval))
1767 base-activity
1768 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
1769 (= 2
1770 (first
1771 (max-indexed
1772 (map #(Math/abs %)
1773 (take 20 (fft base-activity))))))))))
1774 #+end_src
1775 #+end_listing
1777 With these action predicates, I can now recognize the actions of
1778 the worm while it is moving under my control and I have access to
1779 all the worm's senses.
1781 #+caption: Use the action predicates defined earlier to report on
1782 #+caption: what the worm is doing while in simulation.
1783 #+name: report-worm-activity
1784 #+attr_latex: [htpb]
1785 #+begin_listing clojure
1786 #+begin_src clojure
1787 (defn debug-experience
1788 [experiences text]
1789 (cond
1790 (grand-circle? experiences) (.setText text "Grand Circle")
1791 (curled? experiences) (.setText text "Curled")
1792 (wiggling? experiences) (.setText text "Wiggling")
1793 (resting? experiences) (.setText text "Resting")))
1794 #+end_src
1795 #+end_listing
1797 #+caption: Using =debug-experience=, the body-centered predicates
1798 #+caption: work together to classify the behaviour of the worm.
1799 #+caption: the predicates are operating with access to the worm's
1800 #+caption: full sensory data.
1801 #+name: basic-worm-view
1802 #+ATTR_LaTeX: :width 10cm
1803 [[./images/worm-identify-init.png]]
1805 These action predicates satisfy the recognition requirement of an
1806 empathic recognition system. There is power in the simplicity of
1807 the action predicates. They describe their actions without getting
1808 confused in visual details of the worm. Each one is frame
1809 independent, but more than that, they are each indepent of
1810 irrelevant visual details of the worm and the environment. They
1811 will work regardless of whether the worm is a different color or
1812 hevaily textured, or if the environment has strange lighting.
1814 The trick now is to make the action predicates work even when the
1815 sensory data on which they depend is absent. If I can do that, then
1816 I will have gained much,
1818 ** \Phi-space describes the worm's experiences
1820 As a first step towards building empathy, I need to gather all of
1821 the worm's experiences during free play. I use a simple vector to
1822 store all the experiences.
1824 Each element of the experience vector exists in the vast space of
1825 all possible worm-experiences. Most of this vast space is actually
1826 unreachable due to physical constraints of the worm's body. For
1827 example, the worm's segments are connected by hinge joints that put
1828 a practical limit on the worm's range of motions without limiting
1829 its degrees of freedom. Some groupings of senses are impossible;
1830 the worm can not be bent into a circle so that its ends are
1831 touching and at the same time not also experience the sensation of
1832 touching itself.
1834 As the worm moves around during free play and its experience vector
1835 grows larger, the vector begins to define a subspace which is all
1836 the sensations the worm can practicaly experience during normal
1837 operation. I call this subspace \Phi-space, short for
1838 physical-space. The experience vector defines a path through
1839 \Phi-space. This path has interesting properties that all derive
1840 from physical embodiment. The proprioceptive components are
1841 completely smooth, because in order for the worm to move from one
1842 position to another, it must pass through the intermediate
1843 positions. The path invariably forms loops as actions are repeated.
1844 Finally and most importantly, proprioception actually gives very
1845 strong inference about the other senses. For example, when the worm
1846 is flat, you can infer that it is touching the ground and that its
1847 muscles are not active, because if the muscles were active, the
1848 worm would be moving and would not be perfectly flat. In order to
1849 stay flat, the worm has to be touching the ground, or it would
1850 again be moving out of the flat position due to gravity. If the
1851 worm is positioned in such a way that it interacts with itself,
1852 then it is very likely to be feeling the same tactile feelings as
1853 the last time it was in that position, because it has the same body
1854 as then. If you observe multiple frames of proprioceptive data,
1855 then you can become increasingly confident about the exact
1856 activations of the worm's muscles, because it generally takes a
1857 unique combination of muscle contractions to transform the worm's
1858 body along a specific path through \Phi-space.
1860 There is a simple way of taking \Phi-space and the total ordering
1861 provided by an experience vector and reliably infering the rest of
1862 the senses.
1864 ** Empathy is the process of tracing though \Phi-space
1866 Here is the core of a basic empathy algorithm, starting with an
1867 experience vector:
1869 First, group the experiences into tiered proprioceptive bins. I use
1870 powers of 10 and 3 bins, and the smallest bin has an approximate
1871 size of 0.001 radians in all proprioceptive dimensions.
1873 Then, given a sequence of proprioceptive input, generate a set of
1874 matching experience records for each input, using the tiered
1875 proprioceptive bins.
1877 Finally, to infer sensory data, select the longest consective chain
1878 of experiences. Conecutive experience means that the experiences
1879 appear next to each other in the experience vector.
1881 This algorithm has three advantages:
1883 1. It's simple
1885 3. It's very fast -- retrieving possible interpretations takes
1886 constant time. Tracing through chains of interpretations takes
1887 time proportional to the average number of experiences in a
1888 proprioceptive bin. Redundant experiences in \Phi-space can be
1889 merged to save computation.
1891 2. It protects from wrong interpretations of transient ambiguous
1892 proprioceptive data. For example, if the worm is flat for just
1893 an instant, this flattness will not be interpreted as implying
1894 that the worm has its muscles relaxed, since the flattness is
1895 part of a longer chain which includes a distinct pattern of
1896 muscle activation. Markov chains or other memoryless statistical
1897 models that operate on individual frames may very well make this
1898 mistake.
1900 #+caption: Program to convert an experience vector into a
1901 #+caption: proprioceptively binned lookup function.
1902 #+name: bin
1903 #+attr_latex: [htpb]
1904 #+begin_listing clojure
1905 #+begin_src clojure
1906 (defn bin [digits]
1907 (fn [angles]
1908 (->> angles
1909 (flatten)
1910 (map (juxt #(Math/sin %) #(Math/cos %)))
1911 (flatten)
1912 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
1914 (defn gen-phi-scan
1915 "Nearest-neighbors with binning. Only returns a result if
1916 the propriceptive data is within 10% of a previously recorded
1917 result in all dimensions."
1918 [phi-space]
1919 (let [bin-keys (map bin [3 2 1])
1920 bin-maps
1921 (map (fn [bin-key]
1922 (group-by
1923 (comp bin-key :proprioception phi-space)
1924 (range (count phi-space)))) bin-keys)
1925 lookups (map (fn [bin-key bin-map]
1926 (fn [proprio] (bin-map (bin-key proprio))))
1927 bin-keys bin-maps)]
1928 (fn lookup [proprio-data]
1929 (set (some #(% proprio-data) lookups)))))
1930 #+end_src
1931 #+end_listing
1933 #+caption: =longest-thread= finds the longest path of consecutive
1934 #+caption: experiences to explain proprioceptive worm data.
1935 #+name: phi-space-history-scan
1936 #+ATTR_LaTeX: :width 10cm
1937 [[./images/aurellem-gray.png]]
1939 =longest-thread= infers sensory data by stitching together pieces
1940 from previous experience. It prefers longer chains of previous
1941 experience to shorter ones. For example, during training the worm
1942 might rest on the ground for one second before it performs its
1943 excercises. If during recognition the worm rests on the ground for
1944 five seconds, =longest-thread= will accomodate this five second
1945 rest period by looping the one second rest chain five times.
1947 =longest-thread= takes time proportinal to the average number of
1948 entries in a proprioceptive bin, because for each element in the
1949 starting bin it performes a series of set lookups in the preceeding
1950 bins. If the total history is limited, then this is only a constant
1951 multiple times the number of entries in the starting bin. This
1952 analysis also applies even if the action requires multiple longest
1953 chains -- it's still the average number of entries in a
1954 proprioceptive bin times the desired chain length. Because
1955 =longest-thread= is so efficient and simple, I can interpret
1956 worm-actions in real time.
1958 #+caption: Program to calculate empathy by tracing though \Phi-space
1959 #+caption: and finding the longest (ie. most coherent) interpretation
1960 #+caption: of the data.
1961 #+name: longest-thread
1962 #+attr_latex: [htpb]
1963 #+begin_listing clojure
1964 #+begin_src clojure
1965 (defn longest-thread
1966 "Find the longest thread from phi-index-sets. The index sets should
1967 be ordered from most recent to least recent."
1968 [phi-index-sets]
1969 (loop [result '()
1970 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
1971 (if (empty? phi-index-sets)
1972 (vec result)
1973 (let [threads
1974 (for [thread-base thread-bases]
1975 (loop [thread (list thread-base)
1976 remaining remaining]
1977 (let [next-index (dec (first thread))]
1978 (cond (empty? remaining) thread
1979 (contains? (first remaining) next-index)
1980 (recur
1981 (cons next-index thread) (rest remaining))
1982 :else thread))))
1983 longest-thread
1984 (reduce (fn [thread-a thread-b]
1985 (if (> (count thread-a) (count thread-b))
1986 thread-a thread-b))
1987 '(nil)
1988 threads)]
1989 (recur (concat longest-thread result)
1990 (drop (count longest-thread) phi-index-sets))))))
1991 #+end_src
1992 #+end_listing
1994 There is one final piece, which is to replace missing sensory data
1995 with a best-guess estimate. While I could fill in missing data by
1996 using a gradient over the closest known sensory data points,
1997 averages can be misleading. It is certainly possible to create an
1998 impossible sensory state by averaging two possible sensory states.
1999 Therefore, I simply replicate the most recent sensory experience to
2000 fill in the gaps.
2002 #+caption: Fill in blanks in sensory experience by replicating the most
2003 #+caption: recent experience.
2004 #+name: infer-nils
2005 #+attr_latex: [htpb]
2006 #+begin_listing clojure
2007 #+begin_src clojure
2008 (defn infer-nils
2009 "Replace nils with the next available non-nil element in the
2010 sequence, or barring that, 0."
2011 [s]
2012 (loop [i (dec (count s))
2013 v (transient s)]
2014 (if (zero? i) (persistent! v)
2015 (if-let [cur (v i)]
2016 (if (get v (dec i) 0)
2017 (recur (dec i) v)
2018 (recur (dec i) (assoc! v (dec i) cur)))
2019 (recur i (assoc! v i 0))))))
2020 #+end_src
2021 #+end_listing
2023 ** Efficient action recognition with =EMPATH=
2025 To use =EMPATH= with the worm, I first need to gather a set of
2026 experiences from the worm that includes the actions I want to
2027 recognize. The =generate-phi-space= program (listing
2028 \ref{generate-phi-space} runs the worm through a series of
2029 exercices and gatheres those experiences into a vector. The
2030 =do-all-the-things= program is a routine expressed in a simple
2031 muscle contraction script language for automated worm control. It
2032 causes the worm to rest, curl, and wiggle over about 700 frames
2033 (approx. 11 seconds).
2035 #+caption: Program to gather the worm's experiences into a vector for
2036 #+caption: further processing. The =motor-control-program= line uses
2037 #+caption: a motor control script that causes the worm to execute a series
2038 #+caption: of ``exercices'' that include all the action predicates.
2039 #+name: generate-phi-space
2040 #+attr_latex: [htpb]
2041 #+begin_listing clojure
2042 #+begin_src clojure
2043 (def do-all-the-things
2044 (concat
2045 curl-script
2046 [[300 :d-ex 40]
2047 [320 :d-ex 0]]
2048 (shift-script 280 (take 16 wiggle-script))))
2050 (defn generate-phi-space []
2051 (let [experiences (atom [])]
2052 (run-world
2053 (apply-map
2054 worm-world
2055 (merge
2056 (worm-world-defaults)
2057 {:end-frame 700
2058 :motor-control
2059 (motor-control-program worm-muscle-labels do-all-the-things)
2060 :experiences experiences})))
2061 @experiences))
2062 #+end_src
2063 #+end_listing
2065 #+caption: Use longest thread and a phi-space generated from a short
2066 #+caption: exercise routine to interpret actions during free play.
2067 #+name: empathy-debug
2068 #+attr_latex: [htpb]
2069 #+begin_listing clojure
2070 #+begin_src clojure
2071 (defn init []
2072 (def phi-space (generate-phi-space))
2073 (def phi-scan (gen-phi-scan phi-space)))
2075 (defn empathy-demonstration []
2076 (let [proprio (atom ())]
2077 (fn
2078 [experiences text]
2079 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2080 (swap! proprio (partial cons phi-indices))
2081 (let [exp-thread (longest-thread (take 300 @proprio))
2082 empathy (mapv phi-space (infer-nils exp-thread))]
2083 (println-repl (vector:last-n exp-thread 22))
2084 (cond
2085 (grand-circle? empathy) (.setText text "Grand Circle")
2086 (curled? empathy) (.setText text "Curled")
2087 (wiggling? empathy) (.setText text "Wiggling")
2088 (resting? empathy) (.setText text "Resting")
2089 :else (.setText text "Unknown")))))))
2091 (defn empathy-experiment [record]
2092 (.start (worm-world :experience-watch (debug-experience-phi)
2093 :record record :worm worm*)))
2094 #+end_src
2095 #+end_listing
2097 The result of running =empathy-experiment= is that the system is
2098 generally able to interpret worm actions using the action-predicates
2099 on simulated sensory data just as well as with actual data. Figure
2100 \ref{empathy-debug-image} was generated using =empathy-experiment=:
2102 #+caption: From only proprioceptive data, =EMPATH= was able to infer
2103 #+caption: the complete sensory experience and classify four poses
2104 #+caption: (The last panel shows a composite image of \emph{wriggling},
2105 #+caption: a dynamic pose.)
2106 #+name: empathy-debug-image
2107 #+ATTR_LaTeX: :width 10cm :placement [H]
2108 [[./images/empathy-1.png]]
2110 One way to measure the performance of =EMPATH= is to compare the
2111 sutiability of the imagined sense experience to trigger the same
2112 action predicates as the real sensory experience.
2114 #+caption: Determine how closely empathy approximates actual
2115 #+caption: sensory data.
2116 #+name: test-empathy-accuracy
2117 #+attr_latex: [htpb]
2118 #+begin_listing clojure
2119 #+begin_src clojure
2120 (def worm-action-label
2121 (juxt grand-circle? curled? wiggling?))
2123 (defn compare-empathy-with-baseline [matches]
2124 (let [proprio (atom ())]
2125 (fn
2126 [experiences text]
2127 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2128 (swap! proprio (partial cons phi-indices))
2129 (let [exp-thread (longest-thread (take 300 @proprio))
2130 empathy (mapv phi-space (infer-nils exp-thread))
2131 experience-matches-empathy
2132 (= (worm-action-label experiences)
2133 (worm-action-label empathy))]
2134 (println-repl experience-matches-empathy)
2135 (swap! matches #(conj % experience-matches-empathy)))))))
2137 (defn accuracy [v]
2138 (float (/ (count (filter true? v)) (count v))))
2140 (defn test-empathy-accuracy []
2141 (let [res (atom [])]
2142 (run-world
2143 (worm-world :experience-watch
2144 (compare-empathy-with-baseline res)
2145 :worm worm*))
2146 (accuracy @res)))
2147 #+end_src
2148 #+end_listing
2150 Running =test-empathy-accuracy= using the very short exercise
2151 program defined in listing \ref{generate-phi-space}, and then doing
2152 a similar pattern of activity manually yeilds an accuracy of around
2153 73%. This is based on very limited worm experience. By training the
2154 worm for longer, the accuracy dramatically improves.
2156 #+caption: Program to generate \Phi-space using manual training.
2157 #+name: manual-phi-space
2158 #+attr_latex: [htpb]
2159 #+begin_listing clojure
2160 #+begin_src clojure
2161 (defn init-interactive []
2162 (def phi-space
2163 (let [experiences (atom [])]
2164 (run-world
2165 (apply-map
2166 worm-world
2167 (merge
2168 (worm-world-defaults)
2169 {:experiences experiences})))
2170 @experiences))
2171 (def phi-scan (gen-phi-scan phi-space)))
2172 #+end_src
2173 #+end_listing
2175 After about 1 minute of manual training, I was able to achieve 95%
2176 accuracy on manual testing of the worm using =init-interactive= and
2177 =test-empathy-accuracy=. The majority of errors are near the
2178 boundaries of transitioning from one type of action to another.
2179 During these transitions the exact label for the action is more open
2180 to interpretation, and dissaggrement between empathy and experience
2181 is more excusable.
2183 ** Digression: bootstrapping touch using free exploration
2185 In the previous section I showed how to compute actions in terms of
2186 body-centered predicates which relied averate touch activation of
2187 pre-defined regions of the worm's skin. What if, instead of recieving
2188 touch pre-grouped into the six faces of each worm segment, the true
2189 topology of the worm's skin was unknown? This is more similiar to how
2190 a nerve fiber bundle might be arranged. While two fibers that are
2191 close in a nerve bundle /might/ correspond to two touch sensors that
2192 are close together on the skin, the process of taking a complicated
2193 surface and forcing it into essentially a circle requires some cuts
2194 and rerragenments.
2196 In this section I show how to automatically learn the skin-topology of
2197 a worm segment by free exploration. As the worm rolls around on the
2198 floor, large sections of its surface get activated. If the worm has
2199 stopped moving, then whatever region of skin that is touching the
2200 floor is probably an important region, and should be recorded.
2202 #+caption: Program to detect whether the worm is in a resting state
2203 #+caption: with one face touching the floor.
2204 #+name: pure-touch
2205 #+begin_listing clojure
2206 #+begin_src clojure
2207 (def full-contact [(float 0.0) (float 0.1)])
2209 (defn pure-touch?
2210 "This is worm specific code to determine if a large region of touch
2211 sensors is either all on or all off."
2212 [[coords touch :as touch-data]]
2213 (= (set (map first touch)) (set full-contact)))
2214 #+end_src
2215 #+end_listing
2217 After collecting these important regions, there will many nearly
2218 similiar touch regions. While for some purposes the subtle
2219 differences between these regions will be important, for my
2220 purposes I colapse them into mostly non-overlapping sets using
2221 =remove-similiar= in listing \ref{remove-similiar}
2223 #+caption: Program to take a lits of set of points and ``collapse them''
2224 #+caption: so that the remaining sets in the list are siginificantly
2225 #+caption: different from each other. Prefer smaller sets to larger ones.
2226 #+name: remove-similiar
2227 #+begin_listing clojure
2228 #+begin_src clojure
2229 (defn remove-similar
2230 [coll]
2231 (loop [result () coll (sort-by (comp - count) coll)]
2232 (if (empty? coll) result
2233 (let [[x & xs] coll
2234 c (count x)]
2235 (if (some
2236 (fn [other-set]
2237 (let [oc (count other-set)]
2238 (< (- (count (union other-set x)) c) (* oc 0.1))))
2239 xs)
2240 (recur result xs)
2241 (recur (cons x result) xs))))))
2242 #+end_src
2243 #+end_listing
2245 Actually running this simulation is easy given =CORTEX='s facilities.
2247 #+caption: Collect experiences while the worm moves around. Filter the touch
2248 #+caption: sensations by stable ones, collapse similiar ones together,
2249 #+caption: and report the regions learned.
2250 #+name: learn-touch
2251 #+begin_listing clojure
2252 #+begin_src clojure
2253 (defn learn-touch-regions []
2254 (let [experiences (atom [])
2255 world (apply-map
2256 worm-world
2257 (assoc (worm-segment-defaults)
2258 :experiences experiences))]
2259 (run-world world)
2260 (->>
2261 @experiences
2262 (drop 175)
2263 ;; access the single segment's touch data
2264 (map (comp first :touch))
2265 ;; only deal with "pure" touch data to determine surfaces
2266 (filter pure-touch?)
2267 ;; associate coordinates with touch values
2268 (map (partial apply zipmap))
2269 ;; select those regions where contact is being made
2270 (map (partial group-by second))
2271 (map #(get % full-contact))
2272 (map (partial map first))
2273 ;; remove redundant/subset regions
2274 (map set)
2275 remove-similar)))
2277 (defn learn-and-view-touch-regions []
2278 (map view-touch-region
2279 (learn-touch-regions)))
2280 #+end_src
2281 #+end_listing
2283 The only thing remining to define is the particular motion the worm
2284 must take. I accomplish this with a simple motor control program.
2286 #+caption: Motor control program for making the worm roll on the ground.
2287 #+caption: This could also be replaced with random motion.
2288 #+name: worm-roll
2289 #+begin_listing clojure
2290 #+begin_src clojure
2291 (defn touch-kinesthetics []
2292 [[170 :lift-1 40]
2293 [190 :lift-1 19]
2294 [206 :lift-1 0]
2296 [400 :lift-2 40]
2297 [410 :lift-2 0]
2299 [570 :lift-2 40]
2300 [590 :lift-2 21]
2301 [606 :lift-2 0]
2303 [800 :lift-1 30]
2304 [809 :lift-1 0]
2306 [900 :roll-2 40]
2307 [905 :roll-2 20]
2308 [910 :roll-2 0]
2310 [1000 :roll-2 40]
2311 [1005 :roll-2 20]
2312 [1010 :roll-2 0]
2314 [1100 :roll-2 40]
2315 [1105 :roll-2 20]
2316 [1110 :roll-2 0]
2317 ])
2318 #+end_src
2319 #+end_listing
2322 #+caption: The small worm rolls around on the floor, driven
2323 #+caption: by the motor control program in listing \ref{worm-roll}.
2324 #+name: worm-roll
2325 #+ATTR_LaTeX: :width 12cm
2326 [[./images/worm-roll.png]]
2329 #+caption: After completing its adventures, the worm now knows
2330 #+caption: how its touch sensors are arranged along its skin. These
2331 #+caption: are the regions that were deemed important by
2332 #+caption: =learn-touch-regions=. Note that the worm has discovered
2333 #+caption: that it has six sides.
2334 #+name: worm-touch-map
2335 #+ATTR_LaTeX: :width 12cm
2336 [[./images/touch-learn.png]]
2338 While simple, =learn-touch-regions= exploits regularities in both
2339 the worm's physiology and the worm's environment to correctly
2340 deduce that the worm has six sides. Note that =learn-touch-regions=
2341 would work just as well even if the worm's touch sense data were
2342 completely scrambled. The cross shape is just for convienence. This
2343 example justifies the use of pre-defined touch regions in =EMPATH=.
2345 * COMMENT Contributions
2347 In this thesis you have seen the =CORTEX= system, a complete
2348 environment for creating simulated creatures. You have seen how to
2349 implement five senses including touch, proprioception, hearing,
2350 vision, and muscle tension. You have seen how to create new creatues
2351 using blender, a 3D modeling tool. I hope that =CORTEX= will be
2352 useful in further research projects. To this end I have included the
2353 full source to =CORTEX= along with a large suite of tests and
2354 examples. I have also created a user guide for =CORTEX= which is
2355 inculded in an appendix to this thesis.
2357 You have also seen how I used =CORTEX= as a platform to attach the
2358 /action recognition/ problem, which is the problem of recognizing
2359 actions in video. You saw a simple system called =EMPATH= which
2360 ientifies actions by first describing actions in a body-centerd,
2361 rich sense language, then infering a full range of sensory
2362 experience from limited data using previous experience gained from
2363 free play.
2365 As a minor digression, you also saw how I used =CORTEX= to enable a
2366 tiny worm to discover the topology of its skin simply by rolling on
2367 the ground.
2369 In conclusion, the main contributions of this thesis are:
2371 - =CORTEX=, a system for creating simulated creatures with rich
2372 senses.
2373 - =EMPATH=, a program for recognizing actions by imagining sensory
2374 experience.
2376 # An anatomical joke:
2377 # - Training
2378 # - Skeletal imitation
2379 # - Sensory fleshing-out
2380 # - Classification