view thesis/cortex.org @ 480:ad76b8b05517

s.
author Robert McIntyre <rlm@mit.edu>
date Fri, 28 Mar 2014 23:17:10 -0400
parents a5480a06d5fe
children 6e68720e1c13
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * COMMENT Empathy and Embodiment as problem solving strategies
46 By the end of this thesis, you will have seen a novel approach to
47 interpreting video using embodiment and empathy. You will have also
48 seen one way to efficiently implement empathy for embodied
49 creatures. Finally, you will become familiar with =CORTEX=, a system
50 for designing and simulating creatures with rich senses, which you
51 may choose to use in your own research.
53 This is the core vision of my thesis: That one of the important ways
54 in which we understand others is by imagining ourselves in their
55 position and emphatically feeling experiences relative to our own
56 bodies. By understanding events in terms of our own previous
57 corporeal experience, we greatly constrain the possibilities of what
58 would otherwise be an unwieldy exponential search. This extra
59 constraint can be the difference between easily understanding what
60 is happening in a video and being completely lost in a sea of
61 incomprehensible color and movement.
63 ** Recognizing actions in video is extremely difficult
65 Consider for example the problem of determining what is happening
66 in a video of which this is one frame:
68 #+caption: A cat drinking some water. Identifying this action is
69 #+caption: beyond the state of the art for computers.
70 #+ATTR_LaTeX: :width 7cm
71 [[./images/cat-drinking.jpg]]
73 It is currently impossible for any computer program to reliably
74 label such a video as ``drinking''. And rightly so -- it is a very
75 hard problem! What features can you describe in terms of low level
76 functions of pixels that can even begin to describe at a high level
77 what is happening here?
79 Or suppose that you are building a program that recognizes chairs.
80 How could you ``see'' the chair in figure \ref{hidden-chair}?
82 #+caption: The chair in this image is quite obvious to humans, but I
83 #+caption: doubt that any modern computer vision program can find it.
84 #+name: hidden-chair
85 #+ATTR_LaTeX: :width 10cm
86 [[./images/fat-person-sitting-at-desk.jpg]]
88 Finally, how is it that you can easily tell the difference between
89 how the girls /muscles/ are working in figure \ref{girl}?
91 #+caption: The mysterious ``common sense'' appears here as you are able
92 #+caption: to discern the difference in how the girl's arm muscles
93 #+caption: are activated between the two images.
94 #+name: girl
95 #+ATTR_LaTeX: :width 7cm
96 [[./images/wall-push.png]]
98 Each of these examples tells us something about what might be going
99 on in our minds as we easily solve these recognition problems.
101 The hidden chairs show us that we are strongly triggered by cues
102 relating to the position of human bodies, and that we can determine
103 the overall physical configuration of a human body even if much of
104 that body is occluded.
106 The picture of the girl pushing against the wall tells us that we
107 have common sense knowledge about the kinetics of our own bodies.
108 We know well how our muscles would have to work to maintain us in
109 most positions, and we can easily project this self-knowledge to
110 imagined positions triggered by images of the human body.
112 ** =EMPATH= neatly solves recognition problems
114 I propose a system that can express the types of recognition
115 problems above in a form amenable to computation. It is split into
116 four parts:
118 - Free/Guided Play :: The creature moves around and experiences the
119 world through its unique perspective. Many otherwise
120 complicated actions are easily described in the language of a
121 full suite of body-centered, rich senses. For example,
122 drinking is the feeling of water sliding down your throat, and
123 cooling your insides. It's often accompanied by bringing your
124 hand close to your face, or bringing your face close to water.
125 Sitting down is the feeling of bending your knees, activating
126 your quadriceps, then feeling a surface with your bottom and
127 relaxing your legs. These body-centered action descriptions
128 can be either learned or hard coded.
129 - Posture Imitation :: When trying to interpret a video or image,
130 the creature takes a model of itself and aligns it with
131 whatever it sees. This alignment can even cross species, as
132 when humans try to align themselves with things like ponies,
133 dogs, or other humans with a different body type.
134 - Empathy :: The alignment triggers associations with
135 sensory data from prior experiences. For example, the
136 alignment itself easily maps to proprioceptive data. Any
137 sounds or obvious skin contact in the video can to a lesser
138 extent trigger previous experience. Segments of previous
139 experiences are stitched together to form a coherent and
140 complete sensory portrait of the scene.
141 - Recognition :: With the scene described in terms of first
142 person sensory events, the creature can now run its
143 action-identification programs on this synthesized sensory
144 data, just as it would if it were actually experiencing the
145 scene first-hand. If previous experience has been accurately
146 retrieved, and if it is analogous enough to the scene, then
147 the creature will correctly identify the action in the scene.
149 For example, I think humans are able to label the cat video as
150 ``drinking'' because they imagine /themselves/ as the cat, and
151 imagine putting their face up against a stream of water and
152 sticking out their tongue. In that imagined world, they can feel
153 the cool water hitting their tongue, and feel the water entering
154 their body, and are able to recognize that /feeling/ as drinking.
155 So, the label of the action is not really in the pixels of the
156 image, but is found clearly in a simulation inspired by those
157 pixels. An imaginative system, having been trained on drinking and
158 non-drinking examples and learning that the most important
159 component of drinking is the feeling of water sliding down one's
160 throat, would analyze a video of a cat drinking in the following
161 manner:
163 1. Create a physical model of the video by putting a ``fuzzy''
164 model of its own body in place of the cat. Possibly also create
165 a simulation of the stream of water.
167 2. Play out this simulated scene and generate imagined sensory
168 experience. This will include relevant muscle contractions, a
169 close up view of the stream from the cat's perspective, and most
170 importantly, the imagined feeling of water entering the
171 mouth. The imagined sensory experience can come from a
172 simulation of the event, but can also be pattern-matched from
173 previous, similar embodied experience.
175 3. The action is now easily identified as drinking by the sense of
176 taste alone. The other senses (such as the tongue moving in and
177 out) help to give plausibility to the simulated action. Note that
178 the sense of vision, while critical in creating the simulation,
179 is not critical for identifying the action from the simulation.
181 For the chair examples, the process is even easier:
183 1. Align a model of your body to the person in the image.
185 2. Generate proprioceptive sensory data from this alignment.
187 3. Use the imagined proprioceptive data as a key to lookup related
188 sensory experience associated with that particular proproceptive
189 feeling.
191 4. Retrieve the feeling of your bottom resting on a surface, your
192 knees bent, and your leg muscles relaxed.
194 5. This sensory information is consistent with the =sitting?=
195 sensory predicate, so you (and the entity in the image) must be
196 sitting.
198 6. There must be a chair-like object since you are sitting.
200 Empathy offers yet another alternative to the age-old AI
201 representation question: ``What is a chair?'' --- A chair is the
202 feeling of sitting.
204 My program, =EMPATH= uses this empathic problem solving technique
205 to interpret the actions of a simple, worm-like creature.
207 #+caption: The worm performs many actions during free play such as
208 #+caption: curling, wiggling, and resting.
209 #+name: worm-intro
210 #+ATTR_LaTeX: :width 15cm
211 [[./images/worm-intro-white.png]]
213 #+caption: =EMPATH= recognized and classified each of these
214 #+caption: poses by inferring the complete sensory experience
215 #+caption: from proprioceptive data.
216 #+name: worm-recognition-intro
217 #+ATTR_LaTeX: :width 15cm
218 [[./images/worm-poses.png]]
220 One powerful advantage of empathic problem solving is that it
221 factors the action recognition problem into two easier problems. To
222 use empathy, you need an /aligner/, which takes the video and a
223 model of your body, and aligns the model with the video. Then, you
224 need a /recognizer/, which uses the aligned model to interpret the
225 action. The power in this method lies in the fact that you describe
226 all actions form a body-centered viewpoint. You are less tied to
227 the particulars of any visual representation of the actions. If you
228 teach the system what ``running'' is, and you have a good enough
229 aligner, the system will from then on be able to recognize running
230 from any point of view, even strange points of view like above or
231 underneath the runner. This is in contrast to action recognition
232 schemes that try to identify actions using a non-embodied approach.
233 If these systems learn about running as viewed from the side, they
234 will not automatically be able to recognize running from any other
235 viewpoint.
237 Another powerful advantage is that using the language of multiple
238 body-centered rich senses to describe body-centerd actions offers a
239 massive boost in descriptive capability. Consider how difficult it
240 would be to compose a set of HOG filters to describe the action of
241 a simple worm-creature ``curling'' so that its head touches its
242 tail, and then behold the simplicity of describing thus action in a
243 language designed for the task (listing \ref{grand-circle-intro}):
245 #+caption: Body-centerd actions are best expressed in a body-centered
246 #+caption: language. This code detects when the worm has curled into a
247 #+caption: full circle. Imagine how you would replicate this functionality
248 #+caption: using low-level pixel features such as HOG filters!
249 #+name: grand-circle-intro
250 #+attr_latex: [htpb]
251 #+begin_listing clojure
252 #+begin_src clojure
253 (defn grand-circle?
254 "Does the worm form a majestic circle (one end touching the other)?"
255 [experiences]
256 (and (curled? experiences)
257 (let [worm-touch (:touch (peek experiences))
258 tail-touch (worm-touch 0)
259 head-touch (worm-touch 4)]
260 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
261 (< 0.2 (contact worm-segment-top-tip head-touch))))))
262 #+end_src
263 #+end_listing
266 ** =CORTEX= is a toolkit for building sensate creatures
268 I built =CORTEX= to be a general AI research platform for doing
269 experiments involving multiple rich senses and a wide variety and
270 number of creatures. I intend it to be useful as a library for many
271 more projects than just this thesis. =CORTEX= was necessary to meet
272 a need among AI researchers at CSAIL and beyond, which is that
273 people often will invent neat ideas that are best expressed in the
274 language of creatures and senses, but in order to explore those
275 ideas they must first build a platform in which they can create
276 simulated creatures with rich senses! There are many ideas that
277 would be simple to execute (such as =EMPATH=), but attached to them
278 is the multi-month effort to make a good creature simulator. Often,
279 that initial investment of time proves to be too much, and the
280 project must make do with a lesser environment.
282 =CORTEX= is well suited as an environment for embodied AI research
283 for three reasons:
285 - You can create new creatures using Blender, a popular 3D modeling
286 program. Each sense can be specified using special blender nodes
287 with biologically inspired paramaters. You need not write any
288 code to create a creature, and can use a wide library of
289 pre-existing blender models as a base for your own creatures.
291 - =CORTEX= implements a wide variety of senses, including touch,
292 proprioception, vision, hearing, and muscle tension. Complicated
293 senses like touch, and vision involve multiple sensory elements
294 embedded in a 2D surface. You have complete control over the
295 distribution of these sensor elements through the use of simple
296 png image files. In particular, =CORTEX= implements more
297 comprehensive hearing than any other creature simulation system
298 available.
300 - =CORTEX= supports any number of creatures and any number of
301 senses. Time in =CORTEX= dialates so that the simulated creatures
302 always precieve a perfectly smooth flow of time, regardless of
303 the actual computational load.
305 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
306 engine designed to create cross-platform 3D desktop games. =CORTEX=
307 is mainly written in clojure, a dialect of =LISP= that runs on the
308 java virtual machine (JVM). The API for creating and simulating
309 creatures and senses is entirely expressed in clojure, though many
310 senses are implemented at the layer of jMonkeyEngine or below. For
311 example, for the sense of hearing I use a layer of clojure code on
312 top of a layer of java JNI bindings that drive a layer of =C++=
313 code which implements a modified version of =OpenAL= to support
314 multiple listeners. =CORTEX= is the only simulation environment
315 that I know of that can support multiple entities that can each
316 hear the world from their own perspective. Other senses also
317 require a small layer of Java code. =CORTEX= also uses =bullet=, a
318 physics simulator written in =C=.
320 #+caption: Here is the worm from above modeled in Blender, a free
321 #+caption: 3D-modeling program. Senses and joints are described
322 #+caption: using special nodes in Blender.
323 #+name: worm-recognition-intro
324 #+ATTR_LaTeX: :width 12cm
325 [[./images/blender-worm.png]]
327 Here are some thing I anticipate that =CORTEX= might be used for:
329 - exploring new ideas about sensory integration
330 - distributed communication among swarm creatures
331 - self-learning using free exploration,
332 - evolutionary algorithms involving creature construction
333 - exploration of exoitic senses and effectors that are not possible
334 in the real world (such as telekenisis or a semantic sense)
335 - imagination using subworlds
337 During one test with =CORTEX=, I created 3,000 creatures each with
338 their own independent senses and ran them all at only 1/80 real
339 time. In another test, I created a detailed model of my own hand,
340 equipped with a realistic distribution of touch (more sensitive at
341 the fingertips), as well as eyes and ears, and it ran at around 1/4
342 real time.
344 #+BEGIN_LaTeX
345 \begin{sidewaysfigure}
346 \includegraphics[width=9.5in]{images/full-hand.png}
347 \caption{
348 I modeled my own right hand in Blender and rigged it with all the
349 senses that {\tt CORTEX} supports. My simulated hand has a
350 biologically inspired distribution of touch sensors. The senses are
351 displayed on the right, and the simulation is displayed on the
352 left. Notice that my hand is curling its fingers, that it can see
353 its own finger from the eye in its palm, and that it can feel its
354 own thumb touching its palm.}
355 \end{sidewaysfigure}
356 #+END_LaTeX
358 ** Contributions
360 - I built =CORTEX=, a comprehensive platform for embodied AI
361 experiments. =CORTEX= supports many features lacking in other
362 systems, such proper simulation of hearing. It is easy to create
363 new =CORTEX= creatures using Blender, a free 3D modeling program.
365 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
366 a worm-like creature using a computational model of empathy.
368 * Building =CORTEX=
370 I intend for =CORTEX= to be used as a general purpose library for
371 building creatures and outfitting them with senses, so that it will
372 be useful for other researchers who want to test out ideas of their
373 own. To this end, wherver I have had to make archetictural choices
374 about =CORTEX=, I have chosen to give as much freedom to the user as
375 possible, so that =CORTEX= may be used for things I have not
376 forseen.
378 ** COMMENT Simulation or Reality?
380 The most important archetictural decision of all is the choice to
381 use a computer-simulated environemnt in the first place! The world
382 is a vast and rich place, and for now simulations are a very poor
383 reflection of its complexity. It may be that there is a significant
384 qualatative difference between dealing with senses in the real
385 world and dealing with pale facilimilies of them in a simulation.
386 What are the advantages and disadvantages of a simulation vs.
387 reality?
389 *** Simulation
391 The advantages of virtual reality are that when everything is a
392 simulation, experiments in that simulation are absolutely
393 reproducible. It's also easier to change the character and world
394 to explore new situations and different sensory combinations.
396 If the world is to be simulated on a computer, then not only do
397 you have to worry about whether the character's senses are rich
398 enough to learn from the world, but whether the world itself is
399 rendered with enough detail and realism to give enough working
400 material to the character's senses. To name just a few
401 difficulties facing modern physics simulators: destructibility of
402 the environment, simulation of water/other fluids, large areas,
403 nonrigid bodies, lots of objects, smoke. I don't know of any
404 computer simulation that would allow a character to take a rock
405 and grind it into fine dust, then use that dust to make a clay
406 sculpture, at least not without spending years calculating the
407 interactions of every single small grain of dust. Maybe a
408 simulated world with today's limitations doesn't provide enough
409 richness for real intelligence to evolve.
411 *** Reality
413 The other approach for playing with senses is to hook your
414 software up to real cameras, microphones, robots, etc., and let it
415 loose in the real world. This has the advantage of eliminating
416 concerns about simulating the world at the expense of increasing
417 the complexity of implementing the senses. Instead of just
418 grabbing the current rendered frame for processing, you have to
419 use an actual camera with real lenses and interact with photons to
420 get an image. It is much harder to change the character, which is
421 now partly a physical robot of some sort, since doing so involves
422 changing things around in the real world instead of modifying
423 lines of code. While the real world is very rich and definitely
424 provides enough stimulation for intelligence to develop as
425 evidenced by our own existence, it is also uncontrollable in the
426 sense that a particular situation cannot be recreated perfectly or
427 saved for later use. It is harder to conduct science because it is
428 harder to repeat an experiment. The worst thing about using the
429 real world instead of a simulation is the matter of time. Instead
430 of simulated time you get the constant and unstoppable flow of
431 real time. This severely limits the sorts of software you can use
432 to program the AI because all sense inputs must be handled in real
433 time. Complicated ideas may have to be implemented in hardware or
434 may simply be impossible given the current speed of our
435 processors. Contrast this with a simulation, in which the flow of
436 time in the simulated world can be slowed down to accommodate the
437 limitations of the character's programming. In terms of cost,
438 doing everything in software is far cheaper than building custom
439 real-time hardware. All you need is a laptop and some patience.
441 ** COMMENT Because of Time, simulation is perferable to reality
443 I envision =CORTEX= being used to support rapid prototyping and
444 iteration of ideas. Even if I could put together a well constructed
445 kit for creating robots, it would still not be enough because of
446 the scourge of real-time processing. Anyone who wants to test their
447 ideas in the real world must always worry about getting their
448 algorithms to run fast enough to process information in real time.
449 The need for real time processing only increases if multiple senses
450 are involved. In the extreme case, even simple algorithms will have
451 to be accelerated by ASIC chips or FPGAs, turning what would
452 otherwise be a few lines of code and a 10x speed penality into a
453 multi-month ordeal. For this reason, =CORTEX= supports
454 /time-dialiation/, which scales back the framerate of the
455 simulation in proportion to the amount of processing each frame.
456 From the perspective of the creatures inside the simulation, time
457 always appears to flow at a constant rate, regardless of how
458 complicated the envorimnent becomes or how many creatures are in
459 the simulation. The cost is that =CORTEX= can sometimes run slower
460 than real time. This can also be an advantage, however ---
461 simulations of very simple creatures in =CORTEX= generally run at
462 40x on my machine!
464 ** COMMENT What is a sense?
466 If =CORTEX= is to support a wide variety of senses, it would help
467 to have a better understanding of what a ``sense'' actually is!
468 While vision, touch, and hearing all seem like they are quite
469 different things, I was supprised to learn during the course of
470 this thesis that they (and all physical senses) can be expressed as
471 exactly the same mathematical object due to a dimensional argument!
473 Human beings are three-dimensional objects, and the nerves that
474 transmit data from our various sense organs to our brain are
475 essentially one-dimensional. This leaves up to two dimensions in
476 which our sensory information may flow. For example, imagine your
477 skin: it is a two-dimensional surface around a three-dimensional
478 object (your body). It has discrete touch sensors embedded at
479 various points, and the density of these sensors corresponds to the
480 sensitivity of that region of skin. Each touch sensor connects to a
481 nerve, all of which eventually are bundled together as they travel
482 up the spinal cord to the brain. Intersect the spinal nerves with a
483 guillotining plane and you will see all of the sensory data of the
484 skin revealed in a roughly circular two-dimensional image which is
485 the cross section of the spinal cord. Points on this image that are
486 close together in this circle represent touch sensors that are
487 /probably/ close together on the skin, although there is of course
488 some cutting and rearrangement that has to be done to transfer the
489 complicated surface of the skin onto a two dimensional image.
491 Most human senses consist of many discrete sensors of various
492 properties distributed along a surface at various densities. For
493 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
494 disks, and Ruffini's endings, which detect pressure and vibration
495 of various intensities. For ears, it is the stereocilia distributed
496 along the basilar membrane inside the cochlea; each one is
497 sensitive to a slightly different frequency of sound. For eyes, it
498 is rods and cones distributed along the surface of the retina. In
499 each case, we can describe the sense with a surface and a
500 distribution of sensors along that surface.
502 The neat idea is that every human sense can be effectively
503 described in terms of a surface containing embedded sensors. If the
504 sense had any more dimensions, then there wouldn't be enough room
505 in the spinal chord to transmit the information!
507 Therefore, =CORTEX= must support the ability to create objects and
508 then be able to ``paint'' points along their surfaces to describe
509 each sense.
511 Fortunately this idea is already a well known computer graphics
512 technique called called /UV-mapping/. The three-dimensional surface
513 of a model is cut and smooshed until it fits on a two-dimensional
514 image. You paint whatever you want on that image, and when the
515 three-dimensional shape is rendered in a game the smooshing and
516 cutting is reversed and the image appears on the three-dimensional
517 object.
519 To make a sense, interpret the UV-image as describing the
520 distribution of that senses sensors. To get different types of
521 sensors, you can either use a different color for each type of
522 sensor, or use multiple UV-maps, each labeled with that sensor
523 type. I generally use a white pixel to mean the presence of a
524 sensor and a black pixel to mean the absence of a sensor, and use
525 one UV-map for each sensor-type within a given sense.
527 #+CAPTION: The UV-map for an elongated icososphere. The white
528 #+caption: dots each represent a touch sensor. They are dense
529 #+caption: in the regions that describe the tip of the finger,
530 #+caption: and less dense along the dorsal side of the finger
531 #+caption: opposite the tip.
532 #+name: finger-UV
533 #+ATTR_latex: :width 10cm
534 [[./images/finger-UV.png]]
536 #+caption: Ventral side of the UV-mapped finger. Notice the
537 #+caption: density of touch sensors at the tip.
538 #+name: finger-side-view
539 #+ATTR_LaTeX: :width 10cm
540 [[./images/finger-1.png]]
542 ** COMMENT Video game engines are a great starting point
544 I did not need to write my own physics simulation code or shader to
545 build =CORTEX=. Doing so would lead to a system that is impossible
546 for anyone but myself to use anyway. Instead, I use a video game
547 engine as a base and modify it to accomodate the additional needs
548 of =CORTEX=. Video game engines are an ideal starting point to
549 build =CORTEX=, because they are not far from being creature
550 building systems themselves.
552 First off, general purpose video game engines come with a physics
553 engine and lighting / sound system. The physics system provides
554 tools that can be co-opted to serve as touch, proprioception, and
555 muscles. Since some games support split screen views, a good video
556 game engine will allow you to efficiently create multiple cameras
557 in the simulated world that can be used as eyes. Video game systems
558 offer integrated asset management for things like textures and
559 creatures models, providing an avenue for defining creatures. They
560 also understand UV-mapping, since this technique is used to apply a
561 texture to a model. Finally, because video game engines support a
562 large number of users, as long as =CORTEX= doesn't stray too far
563 from the base system, other researchers can turn to this community
564 for help when doing their research.
566 ** COMMENT =CORTEX= is based on jMonkeyEngine3
568 While preparing to build =CORTEX= I studied several video game
569 engines to see which would best serve as a base. The top contenders
570 were:
572 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
573 software in 1997. All the source code was released by ID
574 software into the Public Domain several years ago, and as a
575 result it has been ported to many different languages. This
576 engine was famous for its advanced use of realistic shading
577 and had decent and fast physics simulation. The main advantage
578 of the Quake II engine is its simplicity, but I ultimately
579 rejected it because the engine is too tied to the concept of a
580 first-person shooter game. One of the problems I had was that
581 there does not seem to be any easy way to attach multiple
582 cameras to a single character. There are also several physics
583 clipping issues that are corrected in a way that only applies
584 to the main character and do not apply to arbitrary objects.
586 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
587 and Quake I engines and is used by Valve in the Half-Life
588 series of games. The physics simulation in the Source Engine
589 is quite accurate and probably the best out of all the engines
590 I investigated. There is also an extensive community actively
591 working with the engine. However, applications that use the
592 Source Engine must be written in C++, the code is not open, it
593 only runs on Windows, and the tools that come with the SDK to
594 handle models and textures are complicated and awkward to use.
596 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
597 games in Java. It uses OpenGL to render to the screen and uses
598 screengraphs to avoid drawing things that do not appear on the
599 screen. It has an active community and several games in the
600 pipeline. The engine was not built to serve any particular
601 game but is instead meant to be used for any 3D game.
603 I chose jMonkeyEngine3 because it because it had the most features
604 out of all the free projects I looked at, and because I could then
605 write my code in clojure, an implementation of =LISP= that runs on
606 the JVM.
608 ** COMMENT =CORTEX= uses Blender to create creature models
610 For the simple worm-like creatures I will use later on in this
611 thesis, I could define a simple API in =CORTEX= that would allow
612 one to create boxes, spheres, etc., and leave that API as the sole
613 way to create creatures. However, for =CORTEX= to truly be useful
614 for other projects, it needs a way to construct complicated
615 creatures. If possible, it would be nice to leverage work that has
616 already been done by the community of 3D modelers, or at least
617 enable people who are talented at moedling but not programming to
618 design =CORTEX= creatures.
620 Therefore, I use Blender, a free 3D modeling program, as the main
621 way to create creatures in =CORTEX=. However, the creatures modeled
622 in Blender must also be simple to simulate in jMonkeyEngine3's game
623 engine, and must also be easy to rig with =CORTEX='s senses. I
624 accomplish this with extensive use of Blender's ``empty nodes.''
626 Empty nodes have no mass, physical presence, or appearance, but
627 they can hold metadata and have names. I use a tree structure of
628 empty nodes to specify senses in the following manner:
630 - Create a single top-level empty node whose name is the name of
631 the sense.
632 - Add empty nodes which each contain meta-data relevant to the
633 sense, including a UV-map describing the number/distribution of
634 sensors if applicable.
635 - Make each empty-node the child of the top-level node.
637 #+caption: An example of annoting a creature model with empty
638 #+caption: nodes to describe the layout of senses. There are
639 #+caption: multiple empty nodes which each describe the position
640 #+caption: of muscles, ears, eyes, or joints.
641 #+name: sense-nodes
642 #+ATTR_LaTeX: :width 10cm
643 [[./images/empty-sense-nodes.png]]
645 ** COMMENT Bodies are composed of segments connected by joints
647 Blender is a general purpose animation tool, which has been used in
648 the past to create high quality movies such as Sintel
649 \cite{sintel}. Though Blender can model and render even complicated
650 things like water, it is crucual to keep models that are meant to
651 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
652 though jMonkeyEngine3, is a rigid-body physics system. This offers
653 a compromise between the expressiveness of a game level and the
654 speed at which it can be simulated, and it means that creatures
655 should be naturally expressed as rigid components held together by
656 joint constraints.
658 But humans are more like a squishy bag with wrapped around some
659 hard bones which define the overall shape. When we move, our skin
660 bends and stretches to accomodate the new positions of our bones.
662 One way to make bodies composed of rigid pieces connected by joints
663 /seem/ more human-like is to use an /armature/, (or /rigging/)
664 system, which defines a overall ``body mesh'' and defines how the
665 mesh deforms as a function of the position of each ``bone'' which
666 is a standard rigid body. This technique is used extensively to
667 model humans and create realistic animations. It is not a good
668 technique for physical simulation, however because it creates a lie
669 -- the skin is not a physical part of the simulation and does not
670 interact with any objects in the world or itself. Objects will pass
671 right though the skin until they come in contact with the
672 underlying bone, which is a physical object. Whithout simulating
673 the skin, the sense of touch has little meaning, and the creature's
674 own vision will lie to it about the true extent of its body.
675 Simulating the skin as a physical object requires some way to
676 continuously update the physical model of the skin along with the
677 movement of the bones, which is unacceptably slow compared to rigid
678 body simulation.
680 Therefore, instead of using the human-like ``deformable bag of
681 bones'' approach, I decided to base my body plans on multiple solid
682 objects that are connected by joints, inspired by the robot =EVE=
683 from the movie WALL-E.
685 #+caption: =EVE= from the movie WALL-E. This body plan turns
686 #+caption: out to be much better suited to my purposes than a more
687 #+caption: human-like one.
688 #+ATTR_LaTeX: :width 10cm
689 [[./images/Eve.jpg]]
691 =EVE='s body is composed of several rigid components that are held
692 together by invisible joint constraints. This is what I mean by
693 ``eve-like''. The main reason that I use eve-style bodies is for
694 efficiency, and so that there will be correspondence between the
695 AI's semses and the physical presence of its body. Each individual
696 section is simulated by a separate rigid body that corresponds
697 exactly with its visual representation and does not change.
698 Sections are connected by invisible joints that are well supported
699 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
700 can efficiently simulate hundreds of rigid bodies connected by
701 joints. Just because sections are rigid does not mean they have to
702 stay as one piece forever; they can be dynamically replaced with
703 multiple sections to simulate splitting in two. This could be used
704 to simulate retractable claws or =EVE='s hands, which are able to
705 coalesce into one object in the movie.
707 *** Solidifying/Connecting a body
709 =CORTEX= creates a creature in two steps: first, it traverses the
710 nodes in the blender file and creates physical representations for
711 any of them that have mass defined in their blender meta-data.
713 #+caption: Program for iterating through the nodes in a blender file
714 #+caption: and generating physical jMonkeyEngine3 objects with mass
715 #+caption: and a matching physics shape.
716 #+name: name
717 #+begin_listing clojure
718 #+begin_src clojure
719 (defn physical!
720 "Iterate through the nodes in creature and make them real physical
721 objects in the simulation."
722 [#^Node creature]
723 (dorun
724 (map
725 (fn [geom]
726 (let [physics-control
727 (RigidBodyControl.
728 (HullCollisionShape.
729 (.getMesh geom))
730 (if-let [mass (meta-data geom "mass")]
731 (float mass) (float 1)))]
732 (.addControl geom physics-control)))
733 (filter #(isa? (class %) Geometry )
734 (node-seq creature)))))
735 #+end_src
736 #+end_listing
738 The next step to making a proper body is to connect those pieces
739 together with joints. jMonkeyEngine has a large array of joints
740 available via =bullet=, such as Point2Point, Cone, Hinge, and a
741 generic Six Degree of Freedom joint, with or without spring
742 restitution.
744 Joints are treated a lot like proper senses, in that there is a
745 top-level empty node named ``joints'' whose children each
746 represent a joint.
748 #+caption: View of the hand model in Blender showing the main ``joints''
749 #+caption: node (highlighted in yellow) and its children which each
750 #+caption: represent a joint in the hand. Each joint node has metadata
751 #+caption: specifying what sort of joint it is.
752 #+name: blender-hand
753 #+ATTR_LaTeX: :width 10cm
754 [[./images/hand-screenshot1.png]]
757 =CORTEX='s procedure for binding the creature together with joints
758 is as follows:
760 - Find the children of the ``joints'' node.
761 - Determine the two spatials the joint is meant to connect.
762 - Create the joint based on the meta-data of the empty node.
764 The higher order function =sense-nodes= from =cortex.sense=
765 simplifies finding the joints based on their parent ``joints''
766 node.
768 #+caption: Retrieving the children empty nodes from a single
769 #+caption: named empty node is a common pattern in =CORTEX=
770 #+caption: further instances of this technique for the senses
771 #+caption: will be omitted
772 #+name: get-empty-nodes
773 #+begin_listing clojure
774 #+begin_src clojure
775 (defn sense-nodes
776 "For some senses there is a special empty blender node whose
777 children are considered markers for an instance of that sense. This
778 function generates functions to find those children, given the name
779 of the special parent node."
780 [parent-name]
781 (fn [#^Node creature]
782 (if-let [sense-node (.getChild creature parent-name)]
783 (seq (.getChildren sense-node)) [])))
785 (def
786 ^{:doc "Return the children of the creature's \"joints\" node."
787 :arglists '([creature])}
788 joints
789 (sense-nodes "joints"))
790 #+end_src
791 #+end_listing
793 To find a joint's targets, =CORTEX= creates a small cube, centered
794 around the empty-node, and grows the cube exponentially until it
795 intersects two physical objects. The objects are ordered according
796 to the joint's rotation, with the first one being the object that
797 has more negative coordinates in the joint's reference frame.
798 Since the objects must be physical, the empty-node itself escapes
799 detection. Because the objects must be physical, =joint-targets=
800 must be called /after/ =physical!= is called.
802 #+caption: Program to find the targets of a joint node by
803 #+caption: exponentiallly growth of a search cube.
804 #+name: joint-targets
805 #+begin_listing clojure
806 #+begin_src clojure
807 (defn joint-targets
808 "Return the two closest two objects to the joint object, ordered
809 from bottom to top according to the joint's rotation."
810 [#^Node parts #^Node joint]
811 (loop [radius (float 0.01)]
812 (let [results (CollisionResults.)]
813 (.collideWith
814 parts
815 (BoundingBox. (.getWorldTranslation joint)
816 radius radius radius) results)
817 (let [targets
818 (distinct
819 (map #(.getGeometry %) results))]
820 (if (>= (count targets) 2)
821 (sort-by
822 #(let [joint-ref-frame-position
823 (jme-to-blender
824 (.mult
825 (.inverse (.getWorldRotation joint))
826 (.subtract (.getWorldTranslation %)
827 (.getWorldTranslation joint))))]
828 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
829 (take 2 targets))
830 (recur (float (* radius 2))))))))
831 #+end_src
832 #+end_listing
834 Once =CORTEX= finds all joints and targets, it creates them using
835 a dispatch on the metadata of each joint node.
837 #+caption: Program to dispatch on blender metadata and create joints
838 #+caption: sutiable for physical simulation.
839 #+name: joint-dispatch
840 #+begin_listing clojure
841 #+begin_src clojure
842 (defmulti joint-dispatch
843 "Translate blender pseudo-joints into real JME joints."
844 (fn [constraints & _]
845 (:type constraints)))
847 (defmethod joint-dispatch :point
848 [constraints control-a control-b pivot-a pivot-b rotation]
849 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
850 (.setLinearLowerLimit Vector3f/ZERO)
851 (.setLinearUpperLimit Vector3f/ZERO)))
853 (defmethod joint-dispatch :hinge
854 [constraints control-a control-b pivot-a pivot-b rotation]
855 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
856 [limit-1 limit-2] (:limit constraints)
857 hinge-axis (.mult rotation (blender-to-jme axis))]
858 (doto (HingeJoint. control-a control-b pivot-a pivot-b
859 hinge-axis hinge-axis)
860 (.setLimit limit-1 limit-2))))
862 (defmethod joint-dispatch :cone
863 [constraints control-a control-b pivot-a pivot-b rotation]
864 (let [limit-xz (:limit-xz constraints)
865 limit-xy (:limit-xy constraints)
866 twist (:twist constraints)]
867 (doto (ConeJoint. control-a control-b pivot-a pivot-b
868 rotation rotation)
869 (.setLimit (float limit-xz) (float limit-xy)
870 (float twist)))))
871 #+end_src
872 #+end_listing
874 All that is left for joints it to combine the above pieces into a
875 something that can operate on the collection of nodes that a
876 blender file represents.
878 #+caption: Program to completely create a joint given information
879 #+caption: from a blender file.
880 #+name: connect
881 #+begin_listing clojure
882 #+begin_src clojure
883 (defn connect
884 "Create a joint between 'obj-a and 'obj-b at the location of
885 'joint. The type of joint is determined by the metadata on 'joint.
887 Here are some examples:
888 {:type :point}
889 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
890 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
892 {:type :cone :limit-xz 0]
893 :limit-xy 0]
894 :twist 0]} (use XZY rotation mode in blender!)"
895 [#^Node obj-a #^Node obj-b #^Node joint]
896 (let [control-a (.getControl obj-a RigidBodyControl)
897 control-b (.getControl obj-b RigidBodyControl)
898 joint-center (.getWorldTranslation joint)
899 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
900 pivot-a (world-to-local obj-a joint-center)
901 pivot-b (world-to-local obj-b joint-center)]
902 (if-let
903 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
904 ;; A side-effect of creating a joint registers
905 ;; it with both physics objects which in turn
906 ;; will register the joint with the physics system
907 ;; when the simulation is started.
908 (joint-dispatch constraints
909 control-a control-b
910 pivot-a pivot-b
911 joint-rotation))))
912 #+end_src
913 #+end_listing
915 In general, whenever =CORTEX= exposes a sense (or in this case
916 physicality), it provides a function of the type =sense!=, which
917 takes in a collection of nodes and augments it to support that
918 sense. The function returns any controlls necessary to use that
919 sense. In this case =body!= cerates a physical body and returns no
920 control functions.
922 #+caption: Program to give joints to a creature.
923 #+name: name
924 #+begin_listing clojure
925 #+begin_src clojure
926 (defn joints!
927 "Connect the solid parts of the creature with physical joints. The
928 joints are taken from the \"joints\" node in the creature."
929 [#^Node creature]
930 (dorun
931 (map
932 (fn [joint]
933 (let [[obj-a obj-b] (joint-targets creature joint)]
934 (connect obj-a obj-b joint)))
935 (joints creature))))
936 (defn body!
937 "Endow the creature with a physical body connected with joints. The
938 particulars of the joints and the masses of each body part are
939 determined in blender."
940 [#^Node creature]
941 (physical! creature)
942 (joints! creature))
943 #+end_src
944 #+end_listing
946 All of the code you have just seen amounts to only 130 lines, yet
947 because it builds on top of Blender and jMonkeyEngine3, those few
948 lines pack quite a punch!
950 The hand from figure \ref{blender-hand}, which was modeled after
951 my own right hand, can now be given joints and simulated as a
952 creature.
954 #+caption: With the ability to create physical creatures from blender,
955 #+caption: =CORTEX= gets one step closer to becomming a full creature
956 #+caption: simulation environment.
957 #+name: name
958 #+ATTR_LaTeX: :width 15cm
959 [[./images/physical-hand.png]]
961 ** COMMENT Eyes reuse standard video game components
963 Vision is one of the most important senses for humans, so I need to
964 build a simulated sense of vision for my AI. I will do this with
965 simulated eyes. Each eye can be independently moved and should see
966 its own version of the world depending on where it is.
968 Making these simulated eyes a reality is simple because
969 jMonkeyEngine already contains extensive support for multiple views
970 of the same 3D simulated world. The reason jMonkeyEngine has this
971 support is because the support is necessary to create games with
972 split-screen views. Multiple views are also used to create
973 efficient pseudo-reflections by rendering the scene from a certain
974 perspective and then projecting it back onto a surface in the 3D
975 world.
977 #+caption: jMonkeyEngine supports multiple views to enable
978 #+caption: split-screen games, like GoldenEye, which was one of
979 #+caption: the first games to use split-screen views.
980 #+name: name
981 #+ATTR_LaTeX: :width 10cm
982 [[./images/goldeneye-4-player.png]]
984 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
986 jMonkeyEngine allows you to create a =ViewPort=, which represents a
987 view of the simulated world. You can create as many of these as you
988 want. Every frame, the =RenderManager= iterates through each
989 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
990 is a =FrameBuffer= which represents the rendered image in the GPU.
992 #+caption: =ViewPorts= are cameras in the world. During each frame,
993 #+caption: the =RenderManager= records a snapshot of what each view
994 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
995 #+name: name
996 #+ATTR_LaTeX: :width 10cm
997 [[../images/diagram_rendermanager2.png]]
999 Each =ViewPort= can have any number of attached =SceneProcessor=
1000 objects, which are called every time a new frame is rendered. A
1001 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1002 whatever it wants to the data. Often this consists of invoking GPU
1003 specific operations on the rendered image. The =SceneProcessor= can
1004 also copy the GPU image data to RAM and process it with the CPU.
1006 *** Appropriating Views for Vision
1008 Each eye in the simulated creature needs its own =ViewPort= so
1009 that it can see the world from its own perspective. To this
1010 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1011 any arbitrary continuation function for further processing. That
1012 continuation function may perform both CPU and GPU operations on
1013 the data. To make this easy for the continuation function, the
1014 =SceneProcessor= maintains appropriately sized buffers in RAM to
1015 hold the data. It does not do any copying from the GPU to the CPU
1016 itself because it is a slow operation.
1018 #+caption: Function to make the rendered secne in jMonkeyEngine
1019 #+caption: available for further processing.
1020 #+name: pipeline-1
1021 #+begin_listing clojure
1022 #+begin_src clojure
1023 (defn vision-pipeline
1024 "Create a SceneProcessor object which wraps a vision processing
1025 continuation function. The continuation is a function that takes
1026 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1027 each of which has already been appropriately sized."
1028 [continuation]
1029 (let [byte-buffer (atom nil)
1030 renderer (atom nil)
1031 image (atom nil)]
1032 (proxy [SceneProcessor] []
1033 (initialize
1034 [renderManager viewPort]
1035 (let [cam (.getCamera viewPort)
1036 width (.getWidth cam)
1037 height (.getHeight cam)]
1038 (reset! renderer (.getRenderer renderManager))
1039 (reset! byte-buffer
1040 (BufferUtils/createByteBuffer
1041 (* width height 4)))
1042 (reset! image (BufferedImage.
1043 width height
1044 BufferedImage/TYPE_4BYTE_ABGR))))
1045 (isInitialized [] (not (nil? @byte-buffer)))
1046 (reshape [_ _ _])
1047 (preFrame [_])
1048 (postQueue [_])
1049 (postFrame
1050 [#^FrameBuffer fb]
1051 (.clear @byte-buffer)
1052 (continuation @renderer fb @byte-buffer @image))
1053 (cleanup []))))
1054 #+end_src
1055 #+end_listing
1057 The continuation function given to =vision-pipeline= above will be
1058 given a =Renderer= and three containers for image data. The
1059 =FrameBuffer= references the GPU image data, but the pixel data
1060 can not be used directly on the CPU. The =ByteBuffer= and
1061 =BufferedImage= are initially "empty" but are sized to hold the
1062 data in the =FrameBuffer=. I call transferring the GPU image data
1063 to the CPU structures "mixing" the image data.
1065 *** Optical sensor arrays are described with images and referenced with metadata
1067 The vision pipeline described above handles the flow of rendered
1068 images. Now, =CORTEX= needs simulated eyes to serve as the source
1069 of these images.
1071 An eye is described in blender in the same way as a joint. They
1072 are zero dimensional empty objects with no geometry whose local
1073 coordinate system determines the orientation of the resulting eye.
1074 All eyes are children of a parent node named "eyes" just as all
1075 joints have a parent named "joints". An eye binds to the nearest
1076 physical object with =bind-sense=.
1078 #+caption: Here, the camera is created based on metadata on the
1079 #+caption: eye-node and attached to the nearest physical object
1080 #+caption: with =bind-sense=
1081 #+name: add-eye
1082 #+begin_listing clojure
1083 (defn add-eye!
1084 "Create a Camera centered on the current position of 'eye which
1085 follows the closest physical node in 'creature. The camera will
1086 point in the X direction and use the Z vector as up as determined
1087 by the rotation of these vectors in blender coordinate space. Use
1088 XZY rotation for the node in blender."
1089 [#^Node creature #^Spatial eye]
1090 (let [target (closest-node creature eye)
1091 [cam-width cam-height]
1092 ;;[640 480] ;; graphics card on laptop doesn't support
1093 ;; arbitray dimensions.
1094 (eye-dimensions eye)
1095 cam (Camera. cam-width cam-height)
1096 rot (.getWorldRotation eye)]
1097 (.setLocation cam (.getWorldTranslation eye))
1098 (.lookAtDirection
1099 cam ; this part is not a mistake and
1100 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1101 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1102 (.setFrustumPerspective
1103 cam (float 45)
1104 (float (/ (.getWidth cam) (.getHeight cam)))
1105 (float 1)
1106 (float 1000))
1107 (bind-sense target cam) cam))
1108 #+end_listing
1110 *** Simulated Retina
1112 An eye is a surface (the retina) which contains many discrete
1113 sensors to detect light. These sensors can have different
1114 light-sensing properties. In humans, each discrete sensor is
1115 sensitive to red, blue, green, or gray. These different types of
1116 sensors can have different spatial distributions along the retina.
1117 In humans, there is a fovea in the center of the retina which has
1118 a very high density of color sensors, and a blind spot which has
1119 no sensors at all. Sensor density decreases in proportion to
1120 distance from the fovea.
1122 I want to be able to model any retinal configuration, so my
1123 eye-nodes in blender contain metadata pointing to images that
1124 describe the precise position of the individual sensors using
1125 white pixels. The meta-data also describes the precise sensitivity
1126 to light that the sensors described in the image have. An eye can
1127 contain any number of these images. For example, the metadata for
1128 an eye might look like this:
1130 #+begin_src clojure
1131 {0xFF0000 "Models/test-creature/retina-small.png"}
1132 #+end_src
1134 #+caption: An example retinal profile image. White pixels are
1135 #+caption: photo-sensitive elements. The distribution of white
1136 #+caption: pixels is denser in the middle and falls off at the
1137 #+caption: edges and is inspired by the human retina.
1138 #+name: retina
1139 #+ATTR_LaTeX: :width 10cm
1140 [[./images/retina-small.png]]
1142 Together, the number 0xFF0000 and the image image above describe
1143 the placement of red-sensitive sensory elements.
1145 Meta-data to very crudely approximate a human eye might be
1146 something like this:
1148 #+begin_src clojure
1149 (let [retinal-profile "Models/test-creature/retina-small.png"]
1150 {0xFF0000 retinal-profile
1151 0x00FF00 retinal-profile
1152 0x0000FF retinal-profile
1153 0xFFFFFF retinal-profile})
1154 #+end_src
1156 The numbers that serve as keys in the map determine a sensor's
1157 relative sensitivity to the channels red, green, and blue. These
1158 sensitivity values are packed into an integer in the order
1159 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1160 image are added together with these sensitivities as linear
1161 weights. Therefore, 0xFF0000 means sensitive to red only while
1162 0xFFFFFF means sensitive to all colors equally (gray).
1164 #+caption: This is the core of vision in =CORTEX=. A given eye node
1165 #+caption: is converted into a function that returns visual
1166 #+caption: information from the simulation.
1167 #+name: vision-kernel
1168 #+begin_listing clojure
1169 (defn vision-kernel
1170 "Returns a list of functions, each of which will return a color
1171 channel's worth of visual information when called inside a running
1172 simulation."
1173 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1174 (let [retinal-map (retina-sensor-profile eye)
1175 camera (add-eye! creature eye)
1176 vision-image
1177 (atom
1178 (BufferedImage. (.getWidth camera)
1179 (.getHeight camera)
1180 BufferedImage/TYPE_BYTE_BINARY))
1181 register-eye!
1182 (runonce
1183 (fn [world]
1184 (add-camera!
1185 world camera
1186 (let [counter (atom 0)]
1187 (fn [r fb bb bi]
1188 (if (zero? (rem (swap! counter inc) (inc skip)))
1189 (reset! vision-image
1190 (BufferedImage! r fb bb bi))))))))]
1191 (vec
1192 (map
1193 (fn [[key image]]
1194 (let [whites (white-coordinates image)
1195 topology (vec (collapse whites))
1196 sensitivity (sensitivity-presets key key)]
1197 (attached-viewport.
1198 (fn [world]
1199 (register-eye! world)
1200 (vector
1201 topology
1202 (vec
1203 (for [[x y] whites]
1204 (pixel-sense
1205 sensitivity
1206 (.getRGB @vision-image x y))))))
1207 register-eye!)))
1208 retinal-map))))
1209 #+end_listing
1211 Note that since each of the functions generated by =vision-kernel=
1212 shares the same =register-eye!= function, the eye will be
1213 registered only once the first time any of the functions from the
1214 list returned by =vision-kernel= is called. Each of the functions
1215 returned by =vision-kernel= also allows access to the =Viewport=
1216 through which it receives images.
1218 All the hard work has been done; all that remains is to apply
1219 =vision-kernel= to each eye in the creature and gather the results
1220 into one list of functions.
1223 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1224 #+caption: environment for experimenting with different types of
1225 #+caption: eyes.
1226 #+name: vision!
1227 #+begin_listing clojure
1228 (defn vision!
1229 "Returns a list of functions, each of which returns visual sensory
1230 data when called inside a running simulation."
1231 [#^Node creature & {skip :skip :or {skip 0}}]
1232 (reduce
1233 concat
1234 (for [eye (eyes creature)]
1235 (vision-kernel creature eye))))
1236 #+end_listing
1238 #+caption: Simulated vision with a test creature and the
1239 #+caption: human-like eye approximation. Notice how each channel
1240 #+caption: of the eye responds differently to the differently
1241 #+caption: colored balls.
1242 #+name: worm-vision-test.
1243 #+ATTR_LaTeX: :width 13cm
1244 [[./images/worm-vision.png]]
1246 The vision code is not much more complicated than the body code,
1247 and enables multiple further paths for simulated vision. For
1248 example, it is quite easy to create bifocal vision -- you just
1249 make two eyes next to each other in blender! It is also possible
1250 to encode vision transforms in the retinal files. For example, the
1251 human like retina file in figure \ref{retina} approximates a
1252 log-polar transform.
1254 This vision code has already been absorbed by the jMonkeyEngine
1255 community and is now (in modified form) part of a system for
1256 capturing in-game video to a file.
1258 ** COMMENT Hearing is hard; =CORTEX= does it right
1260 At the end of this section I will have simulated ears that work the
1261 same way as the simulated eyes in the last section. I will be able to
1262 place any number of ear-nodes in a blender file, and they will bind to
1263 the closest physical object and follow it as it moves around. Each ear
1264 will provide access to the sound data it picks up between every frame.
1266 Hearing is one of the more difficult senses to simulate, because there
1267 is less support for obtaining the actual sound data that is processed
1268 by jMonkeyEngine3. There is no "split-screen" support for rendering
1269 sound from different points of view, and there is no way to directly
1270 access the rendered sound data.
1272 =CORTEX='s hearing is unique because it does not have any
1273 limitations compared to other simulation environments. As far as I
1274 know, there is no other system that supports multiple listerers,
1275 and the sound demo at the end of this section is the first time
1276 it's been done in a video game environment.
1278 *** Brief Description of jMonkeyEngine's Sound System
1280 jMonkeyEngine's sound system works as follows:
1282 - jMonkeyEngine uses the =AppSettings= for the particular
1283 application to determine what sort of =AudioRenderer= should be
1284 used.
1285 - Although some support is provided for multiple AudioRendering
1286 backends, jMonkeyEngine at the time of this writing will either
1287 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1288 - jMonkeyEngine tries to figure out what sort of system you're
1289 running and extracts the appropriate native libraries.
1290 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1291 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1292 - =OpenAL= renders the 3D sound and feeds the rendered sound
1293 directly to any of various sound output devices with which it
1294 knows how to communicate.
1296 A consequence of this is that there's no way to access the actual
1297 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1298 one /listener/ (it renders sound data from only one perspective),
1299 which normally isn't a problem for games, but becomes a problem
1300 when trying to make multiple AI creatures that can each hear the
1301 world from a different perspective.
1303 To make many AI creatures in jMonkeyEngine that can each hear the
1304 world from their own perspective, or to make a single creature with
1305 many ears, it is necessary to go all the way back to =OpenAL= and
1306 implement support for simulated hearing there.
1308 *** Extending =OpenAl=
1310 Extending =OpenAL= to support multiple listeners requires 500
1311 lines of =C= code and is too hairy to mention here. Instead, I
1312 will show a small amount of extension code and go over the high
1313 level stragety. Full source is of course available with the
1314 =CORTEX= distribution if you're interested.
1316 =OpenAL= goes to great lengths to support many different systems,
1317 all with different sound capabilities and interfaces. It
1318 accomplishes this difficult task by providing code for many
1319 different sound backends in pseudo-objects called /Devices/.
1320 There's a device for the Linux Open Sound System and the Advanced
1321 Linux Sound Architecture, there's one for Direct Sound on Windows,
1322 and there's even one for Solaris. =OpenAL= solves the problem of
1323 platform independence by providing all these Devices.
1325 Wrapper libraries such as LWJGL are free to examine the system on
1326 which they are running and then select an appropriate device for
1327 that system.
1329 There are also a few "special" devices that don't interface with
1330 any particular system. These include the Null Device, which
1331 doesn't do anything, and the Wave Device, which writes whatever
1332 sound it receives to a file, if everything has been set up
1333 correctly when configuring =OpenAL=.
1335 Actual mixing (doppler shift and distance.environment-based
1336 attenuation) of the sound data happens in the Devices, and they
1337 are the only point in the sound rendering process where this data
1338 is available.
1340 Therefore, in order to support multiple listeners, and get the
1341 sound data in a form that the AIs can use, it is necessary to
1342 create a new Device which supports this feature.
1344 Adding a device to OpenAL is rather tricky -- there are five
1345 separate files in the =OpenAL= source tree that must be modified
1346 to do so. I named my device the "Multiple Audio Send" Device, or
1347 =Send= Device for short, since it sends audio data back to the
1348 calling application like an Aux-Send cable on a mixing board.
1350 The main idea behind the Send device is to take advantage of the
1351 fact that LWJGL only manages one /context/ when using OpenAL. A
1352 /context/ is like a container that holds samples and keeps track
1353 of where the listener is. In order to support multiple listeners,
1354 the Send device identifies the LWJGL context as the master
1355 context, and creates any number of slave contexts to represent
1356 additional listeners. Every time the device renders sound, it
1357 synchronizes every source from the master LWJGL context to the
1358 slave contexts. Then, it renders each context separately, using a
1359 different listener for each one. The rendered sound is made
1360 available via JNI to jMonkeyEngine.
1362 Switching between contexts is not the normal operation of a
1363 Device, and one of the problems with doing so is that a Device
1364 normally keeps around a few pieces of state such as the
1365 =ClickRemoval= array above which will become corrupted if the
1366 contexts are not rendered in parallel. The solution is to create a
1367 copy of this normally global device state for each context, and
1368 copy it back and forth into and out of the actual device state
1369 whenever a context is rendered.
1371 The core of the =Send= device is the =syncSources= function, which
1372 does the job of copying all relevant data from one context to
1373 another.
1375 #+caption: Program for extending =OpenAL= to support multiple
1376 #+caption: listeners via context copying/switching.
1377 #+name: sync-openal-sources
1378 #+begin_listing C
1379 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1380 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1381 ALuint master = masterSource->source;
1382 ALuint slave = slaveSource->source;
1383 ALCcontext *current = alcGetCurrentContext();
1385 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1386 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1387 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1388 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1389 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1390 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1391 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1392 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1393 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1394 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1395 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1396 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1397 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1399 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1400 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1401 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1403 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1404 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1406 alcMakeContextCurrent(masterCtx);
1407 ALint source_type;
1408 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1410 // Only static sources are currently synchronized!
1411 if (AL_STATIC == source_type){
1412 ALint master_buffer;
1413 ALint slave_buffer;
1414 alGetSourcei(master, AL_BUFFER, &master_buffer);
1415 alcMakeContextCurrent(slaveCtx);
1416 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1417 if (master_buffer != slave_buffer){
1418 alSourcei(slave, AL_BUFFER, master_buffer);
1422 // Synchronize the state of the two sources.
1423 alcMakeContextCurrent(masterCtx);
1424 ALint masterState;
1425 ALint slaveState;
1427 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1428 alcMakeContextCurrent(slaveCtx);
1429 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1431 if (masterState != slaveState){
1432 switch (masterState){
1433 case AL_INITIAL : alSourceRewind(slave); break;
1434 case AL_PLAYING : alSourcePlay(slave); break;
1435 case AL_PAUSED : alSourcePause(slave); break;
1436 case AL_STOPPED : alSourceStop(slave); break;
1439 // Restore whatever context was previously active.
1440 alcMakeContextCurrent(current);
1442 #+end_listing
1444 With this special context-switching device, and some ugly JNI
1445 bindings that are not worth mentioning, =CORTEX= gains the ability
1446 to access multiple sound streams from =OpenAL=.
1448 #+caption: Program to create an ear from a blender empty node. The ear
1449 #+caption: follows around the nearest physical object and passes
1450 #+caption: all sensory data to a continuation function.
1451 #+name: add-ear
1452 #+begin_listing clojure
1453 (defn add-ear!
1454 "Create a Listener centered on the current position of 'ear
1455 which follows the closest physical node in 'creature and
1456 sends sound data to 'continuation."
1457 [#^Application world #^Node creature #^Spatial ear continuation]
1458 (let [target (closest-node creature ear)
1459 lis (Listener.)
1460 audio-renderer (.getAudioRenderer world)
1461 sp (hearing-pipeline continuation)]
1462 (.setLocation lis (.getWorldTranslation ear))
1463 (.setRotation lis (.getWorldRotation ear))
1464 (bind-sense target lis)
1465 (update-listener-velocity! target lis)
1466 (.addListener audio-renderer lis)
1467 (.registerSoundProcessor audio-renderer lis sp)))
1468 #+end_listing
1471 The =Send= device, unlike most of the other devices in =OpenAL=,
1472 does not render sound unless asked. This enables the system to
1473 slow down or speed up depending on the needs of the AIs who are
1474 using it to listen. If the device tried to render samples in
1475 real-time, a complicated AI whose mind takes 100 seconds of
1476 computer time to simulate 1 second of AI-time would miss almost
1477 all of the sound in its environment!
1479 #+caption: Program to enable arbitrary hearing in =CORTEX=
1480 #+name: hearing
1481 #+begin_listing clojure
1482 (defn hearing-kernel
1483 "Returns a function which returns auditory sensory data when called
1484 inside a running simulation."
1485 [#^Node creature #^Spatial ear]
1486 (let [hearing-data (atom [])
1487 register-listener!
1488 (runonce
1489 (fn [#^Application world]
1490 (add-ear!
1491 world creature ear
1492 (comp #(reset! hearing-data %)
1493 byteBuffer->pulse-vector))))]
1494 (fn [#^Application world]
1495 (register-listener! world)
1496 (let [data @hearing-data
1497 topology
1498 (vec (map #(vector % 0) (range 0 (count data))))]
1499 [topology data]))))
1501 (defn hearing!
1502 "Endow the creature in a particular world with the sense of
1503 hearing. Will return a sequence of functions, one for each ear,
1504 which when called will return the auditory data from that ear."
1505 [#^Node creature]
1506 (for [ear (ears creature)]
1507 (hearing-kernel creature ear)))
1508 #+end_listing
1510 Armed with these functions, =CORTEX= is able to test possibly the
1511 first ever instance of multiple listeners in a video game engine
1512 based simulation!
1514 #+caption: Here a simple creature responds to sound by changing
1515 #+caption: its color from gray to green when the total volume
1516 #+caption: goes over a threshold.
1517 #+name: sound-test
1518 #+begin_listing java
1519 /**
1520 * Respond to sound! This is the brain of an AI entity that
1521 * hears its surroundings and reacts to them.
1522 */
1523 public void process(ByteBuffer audioSamples,
1524 int numSamples, AudioFormat format) {
1525 audioSamples.clear();
1526 byte[] data = new byte[numSamples];
1527 float[] out = new float[numSamples];
1528 audioSamples.get(data);
1529 FloatSampleTools.
1530 byte2floatInterleaved
1531 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1533 float max = Float.NEGATIVE_INFINITY;
1534 for (float f : out){if (f > max) max = f;}
1535 audioSamples.clear();
1537 if (max > 0.1){
1538 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1540 else {
1541 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1543 #+end_listing
1545 #+caption: First ever simulation of multiple listerners in =CORTEX=.
1546 #+caption: Each cube is a creature which processes sound data with
1547 #+caption: the =process= function from listing \ref{sound-test}.
1548 #+caption: the ball is constantally emiting a pure tone of
1549 #+caption: constant volume. As it approaches the cubes, they each
1550 #+caption: change color in response to the sound.
1551 #+name: sound-cubes.
1552 #+ATTR_LaTeX: :width 10cm
1553 [[./images/aurellem-gray.png]]
1555 This system of hearing has also been co-opted by the
1556 jMonkeyEngine3 community and is used to record audio for demo
1557 videos.
1559 ** COMMENT Touch uses hundreds of hair-like elements
1561 Touch is critical to navigation and spatial reasoning and as such I
1562 need a simulated version of it to give to my AI creatures.
1564 Human skin has a wide array of touch sensors, each of which
1565 specialize in detecting different vibrational modes and pressures.
1566 These sensors can integrate a vast expanse of skin (i.e. your
1567 entire palm), or a tiny patch of skin at the tip of your finger.
1568 The hairs of the skin help detect objects before they even come
1569 into contact with the skin proper.
1571 However, touch in my simulated world can not exactly correspond to
1572 human touch because my creatures are made out of completely rigid
1573 segments that don't deform like human skin.
1575 Instead of measuring deformation or vibration, I surround each
1576 rigid part with a plenitude of hair-like objects (/feelers/) which
1577 do not interact with the physical world. Physical objects can pass
1578 through them with no effect. The feelers are able to tell when
1579 other objects pass through them, and they constantly report how
1580 much of their extent is covered. So even though the creature's body
1581 parts do not deform, the feelers create a margin around those body
1582 parts which achieves a sense of touch which is a hybrid between a
1583 human's sense of deformation and sense from hairs.
1585 Implementing touch in jMonkeyEngine follows a different technical
1586 route than vision and hearing. Those two senses piggybacked off
1587 jMonkeyEngine's 3D audio and video rendering subsystems. To
1588 simulate touch, I use jMonkeyEngine's physics system to execute
1589 many small collision detections, one for each feeler. The placement
1590 of the feelers is determined by a UV-mapped image which shows where
1591 each feeler should be on the 3D surface of the body.
1593 *** Defining Touch Meta-Data in Blender
1595 Each geometry can have a single UV map which describes the
1596 position of the feelers which will constitute its sense of touch.
1597 This image path is stored under the ``touch'' key. The image itself
1598 is black and white, with black meaning a feeler length of 0 (no
1599 feeler is present) and white meaning a feeler length of =scale=,
1600 which is a float stored under the key "scale".
1602 #+caption: Touch does not use empty nodes, to store metadata,
1603 #+caption: because the metadata of each solid part of a
1604 #+caption: creature's body is sufficient.
1605 #+name: touch-meta-data
1606 #+begin_listing clojure
1607 #+BEGIN_SRC clojure
1608 (defn tactile-sensor-profile
1609 "Return the touch-sensor distribution image in BufferedImage format,
1610 or nil if it does not exist."
1611 [#^Geometry obj]
1612 (if-let [image-path (meta-data obj "touch")]
1613 (load-image image-path)))
1615 (defn tactile-scale
1616 "Return the length of each feeler. Default scale is 0.01
1617 jMonkeyEngine units."
1618 [#^Geometry obj]
1619 (if-let [scale (meta-data obj "scale")]
1620 scale 0.1))
1621 #+END_SRC
1622 #+end_listing
1624 Here is an example of a UV-map which specifies the position of
1625 touch sensors along the surface of the upper segment of a fingertip.
1627 #+caption: This is the tactile-sensor-profile for the upper segment
1628 #+caption: of a fingertip. It defines regions of high touch sensitivity
1629 #+caption: (where there are many white pixels) and regions of low
1630 #+caption: sensitivity (where white pixels are sparse).
1631 #+name: fimgertip-UV
1632 #+ATTR_LaTeX: :width 13cm
1633 [[./images/finger-UV.png]]
1635 *** Implementation Summary
1637 To simulate touch there are three conceptual steps. For each solid
1638 object in the creature, you first have to get UV image and scale
1639 parameter which define the position and length of the feelers.
1640 Then, you use the triangles which comprise the mesh and the UV
1641 data stored in the mesh to determine the world-space position and
1642 orientation of each feeler. Then once every frame, update these
1643 positions and orientations to match the current position and
1644 orientation of the object, and use physics collision detection to
1645 gather tactile data.
1647 Extracting the meta-data has already been described. The third
1648 step, physics collision detection, is handled in =touch-kernel=.
1649 Translating the positions and orientations of the feelers from the
1650 UV-map to world-space is itself a three-step process.
1652 - Find the triangles which make up the mesh in pixel-space and in
1653 world-space. (=triangles= =pixel-triangles=).
1655 - Find the coordinates of each feeler in world-space. These are
1656 the origins of the feelers. (=feeler-origins=).
1658 - Calculate the normals of the triangles in world space, and add
1659 them to each of the origins of the feelers. These are the
1660 normalized coordinates of the tips of the feelers.
1661 (=feeler-tips=).
1663 *** Triangle Math
1665 The rigid objects which make up a creature have an underlying
1666 =Geometry=, which is a =Mesh= plus a =Material= and other
1667 important data involved with displaying the object.
1669 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1670 vertices which have coordinates in world space and UV space.
1672 Here, =triangles= gets all the world-space triangles which
1673 comprise a mesh, while =pixel-triangles= gets those same triangles
1674 expressed in pixel coordinates (which are UV coordinates scaled to
1675 fit the height and width of the UV image).
1677 #+caption: Programs to extract triangles from a geometry and get
1678 #+caption: their verticies in both world and UV-coordinates.
1679 #+name: get-triangles
1680 #+begin_listing clojure
1681 #+BEGIN_SRC clojure
1682 (defn triangle
1683 "Get the triangle specified by triangle-index from the mesh."
1684 [#^Geometry geo triangle-index]
1685 (triangle-seq
1686 (let [scratch (Triangle.)]
1687 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1689 (defn triangles
1690 "Return a sequence of all the Triangles which comprise a given
1691 Geometry."
1692 [#^Geometry geo]
1693 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1695 (defn triangle-vertex-indices
1696 "Get the triangle vertex indices of a given triangle from a given
1697 mesh."
1698 [#^Mesh mesh triangle-index]
1699 (let [indices (int-array 3)]
1700 (.getTriangle mesh triangle-index indices)
1701 (vec indices)))
1703 (defn vertex-UV-coord
1704 "Get the UV-coordinates of the vertex named by vertex-index"
1705 [#^Mesh mesh vertex-index]
1706 (let [UV-buffer
1707 (.getData
1708 (.getBuffer
1709 mesh
1710 VertexBuffer$Type/TexCoord))]
1711 [(.get UV-buffer (* vertex-index 2))
1712 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1714 (defn pixel-triangle [#^Geometry geo image index]
1715 (let [mesh (.getMesh geo)
1716 width (.getWidth image)
1717 height (.getHeight image)]
1718 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1719 (map (partial vertex-UV-coord mesh)
1720 (triangle-vertex-indices mesh index))))))
1722 (defn pixel-triangles
1723 "The pixel-space triangles of the Geometry, in the same order as
1724 (triangles geo)"
1725 [#^Geometry geo image]
1726 (let [height (.getHeight image)
1727 width (.getWidth image)]
1728 (map (partial pixel-triangle geo image)
1729 (range (.getTriangleCount (.getMesh geo))))))
1730 #+END_SRC
1731 #+end_listing
1733 *** The Affine Transform from one Triangle to Another
1735 =pixel-triangles= gives us the mesh triangles expressed in pixel
1736 coordinates and =triangles= gives us the mesh triangles expressed
1737 in world coordinates. The tactile-sensor-profile gives the
1738 position of each feeler in pixel-space. In order to convert
1739 pixel-space coordinates into world-space coordinates we need
1740 something that takes coordinates on the surface of one triangle
1741 and gives the corresponding coordinates on the surface of another
1742 triangle.
1744 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1745 into any other by a combination of translation, scaling, and
1746 rotation. The affine transformation from one triangle to another
1747 is readily computable if the triangle is expressed in terms of a
1748 $4x4$ matrix.
1750 #+BEGIN_LaTeX
1751 $$
1752 \begin{bmatrix}
1753 x_1 & x_2 & x_3 & n_x \\
1754 y_1 & y_2 & y_3 & n_y \\
1755 z_1 & z_2 & z_3 & n_z \\
1756 1 & 1 & 1 & 1
1757 \end{bmatrix}
1758 $$
1759 #+END_LaTeX
1761 Here, the first three columns of the matrix are the vertices of
1762 the triangle. The last column is the right-handed unit normal of
1763 the triangle.
1765 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1766 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1767 is $T_{2}T_{1}^{-1}$.
1769 The clojure code below recapitulates the formulas above, using
1770 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1771 transformation.
1773 #+caption: Program to interpert triangles as affine transforms.
1774 #+name: triangle-affine
1775 #+begin_listing clojure
1776 #+BEGIN_SRC clojure
1777 (defn triangle->matrix4f
1778 "Converts the triangle into a 4x4 matrix: The first three columns
1779 contain the vertices of the triangle; the last contains the unit
1780 normal of the triangle. The bottom row is filled with 1s."
1781 [#^Triangle t]
1782 (let [mat (Matrix4f.)
1783 [vert-1 vert-2 vert-3]
1784 (mapv #(.get t %) (range 3))
1785 unit-normal (do (.calculateNormal t)(.getNormal t))
1786 vertices [vert-1 vert-2 vert-3 unit-normal]]
1787 (dorun
1788 (for [row (range 4) col (range 3)]
1789 (do
1790 (.set mat col row (.get (vertices row) col))
1791 (.set mat 3 row 1)))) mat))
1793 (defn triangles->affine-transform
1794 "Returns the affine transformation that converts each vertex in the
1795 first triangle into the corresponding vertex in the second
1796 triangle."
1797 [#^Triangle tri-1 #^Triangle tri-2]
1798 (.mult
1799 (triangle->matrix4f tri-2)
1800 (.invert (triangle->matrix4f tri-1))))
1801 #+END_SRC
1802 #+end_listing
1804 *** Triangle Boundaries
1806 For efficiency's sake I will divide the tactile-profile image into
1807 small squares which inscribe each pixel-triangle, then extract the
1808 points which lie inside the triangle and map them to 3D-space using
1809 =triangle-transform= above. To do this I need a function,
1810 =convex-bounds= which finds the smallest box which inscribes a 2D
1811 triangle.
1813 =inside-triangle?= determines whether a point is inside a triangle
1814 in 2D pixel-space.
1816 #+caption: Program to efficiently determine point includion
1817 #+caption: in a triangle.
1818 #+name: in-triangle
1819 #+begin_listing clojure
1820 #+BEGIN_SRC clojure
1821 (defn convex-bounds
1822 "Returns the smallest square containing the given vertices, as a
1823 vector of integers [left top width height]."
1824 [verts]
1825 (let [xs (map first verts)
1826 ys (map second verts)
1827 x0 (Math/floor (apply min xs))
1828 y0 (Math/floor (apply min ys))
1829 x1 (Math/ceil (apply max xs))
1830 y1 (Math/ceil (apply max ys))]
1831 [x0 y0 (- x1 x0) (- y1 y0)]))
1833 (defn same-side?
1834 "Given the points p1 and p2 and the reference point ref, is point p
1835 on the same side of the line that goes through p1 and p2 as ref is?"
1836 [p1 p2 ref p]
1837 (<=
1839 (.dot
1840 (.cross (.subtract p2 p1) (.subtract p p1))
1841 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1843 (defn inside-triangle?
1844 "Is the point inside the triangle?"
1845 {:author "Dylan Holmes"}
1846 [#^Triangle tri #^Vector3f p]
1847 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1848 (and
1849 (same-side? vert-1 vert-2 vert-3 p)
1850 (same-side? vert-2 vert-3 vert-1 p)
1851 (same-side? vert-3 vert-1 vert-2 p))))
1852 #+END_SRC
1853 #+end_listing
1855 *** Feeler Coordinates
1857 The triangle-related functions above make short work of
1858 calculating the positions and orientations of each feeler in
1859 world-space.
1861 #+caption: Program to get the coordinates of ``feelers '' in
1862 #+caption: both world and UV-coordinates.
1863 #+name: feeler-coordinates
1864 #+begin_listing clojure
1865 #+BEGIN_SRC clojure
1866 (defn feeler-pixel-coords
1867 "Returns the coordinates of the feelers in pixel space in lists, one
1868 list for each triangle, ordered in the same way as (triangles) and
1869 (pixel-triangles)."
1870 [#^Geometry geo image]
1871 (map
1872 (fn [pixel-triangle]
1873 (filter
1874 (fn [coord]
1875 (inside-triangle? (->triangle pixel-triangle)
1876 (->vector3f coord)))
1877 (white-coordinates image (convex-bounds pixel-triangle))))
1878 (pixel-triangles geo image)))
1880 (defn feeler-world-coords
1881 "Returns the coordinates of the feelers in world space in lists, one
1882 list for each triangle, ordered in the same way as (triangles) and
1883 (pixel-triangles)."
1884 [#^Geometry geo image]
1885 (let [transforms
1886 (map #(triangles->affine-transform
1887 (->triangle %1) (->triangle %2))
1888 (pixel-triangles geo image)
1889 (triangles geo))]
1890 (map (fn [transform coords]
1891 (map #(.mult transform (->vector3f %)) coords))
1892 transforms (feeler-pixel-coords geo image))))
1893 #+END_SRC
1894 #+end_listing
1896 #+caption: Program to get the position of the base and tip of
1897 #+caption: each ``feeler''
1898 #+name: feeler-tips
1899 #+begin_listing clojure
1900 #+BEGIN_SRC clojure
1901 (defn feeler-origins
1902 "The world space coordinates of the root of each feeler."
1903 [#^Geometry geo image]
1904 (reduce concat (feeler-world-coords geo image)))
1906 (defn feeler-tips
1907 "The world space coordinates of the tip of each feeler."
1908 [#^Geometry geo image]
1909 (let [world-coords (feeler-world-coords geo image)
1910 normals
1911 (map
1912 (fn [triangle]
1913 (.calculateNormal triangle)
1914 (.clone (.getNormal triangle)))
1915 (map ->triangle (triangles geo)))]
1917 (mapcat (fn [origins normal]
1918 (map #(.add % normal) origins))
1919 world-coords normals)))
1921 (defn touch-topology
1922 [#^Geometry geo image]
1923 (collapse (reduce concat (feeler-pixel-coords geo image))))
1924 #+END_SRC
1925 #+end_listing
1927 *** Simulated Touch
1929 Now that the functions to construct feelers are complete,
1930 =touch-kernel= generates functions to be called from within a
1931 simulation that perform the necessary physics collisions to
1932 collect tactile data, and =touch!= recursively applies it to every
1933 node in the creature.
1935 #+caption: Efficient program to transform a ray from
1936 #+caption: one position to another.
1937 #+name: set-ray
1938 #+begin_listing clojure
1939 #+BEGIN_SRC clojure
1940 (defn set-ray [#^Ray ray #^Matrix4f transform
1941 #^Vector3f origin #^Vector3f tip]
1942 ;; Doing everything locally reduces garbage collection by enough to
1943 ;; be worth it.
1944 (.mult transform origin (.getOrigin ray))
1945 (.mult transform tip (.getDirection ray))
1946 (.subtractLocal (.getDirection ray) (.getOrigin ray))
1947 (.normalizeLocal (.getDirection ray)))
1948 #+END_SRC
1949 #+end_listing
1951 #+caption: This is the core of touch in =CORTEX= each feeler
1952 #+caption: follows the object it is bound to, reporting any
1953 #+caption: collisions that may happen.
1954 #+name: touch-kernel
1955 #+begin_listing clojure
1956 #+BEGIN_SRC clojure
1957 (defn touch-kernel
1958 "Constructs a function which will return tactile sensory data from
1959 'geo when called from inside a running simulation"
1960 [#^Geometry geo]
1961 (if-let
1962 [profile (tactile-sensor-profile geo)]
1963 (let [ray-reference-origins (feeler-origins geo profile)
1964 ray-reference-tips (feeler-tips geo profile)
1965 ray-length (tactile-scale geo)
1966 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
1967 topology (touch-topology geo profile)
1968 correction (float (* ray-length -0.2))]
1969 ;; slight tolerance for very close collisions.
1970 (dorun
1971 (map (fn [origin tip]
1972 (.addLocal origin (.mult (.subtract tip origin)
1973 correction)))
1974 ray-reference-origins ray-reference-tips))
1975 (dorun (map #(.setLimit % ray-length) current-rays))
1976 (fn [node]
1977 (let [transform (.getWorldMatrix geo)]
1978 (dorun
1979 (map (fn [ray ref-origin ref-tip]
1980 (set-ray ray transform ref-origin ref-tip))
1981 current-rays ray-reference-origins
1982 ray-reference-tips))
1983 (vector
1984 topology
1985 (vec
1986 (for [ray current-rays]
1987 (do
1988 (let [results (CollisionResults.)]
1989 (.collideWith node ray results)
1990 (let [touch-objects
1991 (filter #(not (= geo (.getGeometry %)))
1992 results)
1993 limit (.getLimit ray)]
1994 [(if (empty? touch-objects)
1995 limit
1996 (let [response
1997 (apply min (map #(.getDistance %)
1998 touch-objects))]
1999 (FastMath/clamp
2000 (float
2001 (if (> response limit) (float 0.0)
2002 (+ response correction)))
2003 (float 0.0)
2004 limit)))
2005 limit])))))))))))
2006 #+END_SRC
2007 #+end_listing
2009 Armed with the =touch!= function, =CORTEX= becomes capable of
2010 giving creatures a sense of touch. A simple test is to create a
2011 cube that is outfitted with a uniform distrubition of touch
2012 sensors. It can feel the ground and any balls that it touches.
2014 #+caption: =CORTEX= interface for creating touch in a simulated
2015 #+caption: creature.
2016 #+name: touch
2017 #+begin_listing clojure
2018 #+BEGIN_SRC clojure
2019 (defn touch!
2020 "Endow the creature with the sense of touch. Returns a sequence of
2021 functions, one for each body part with a tactile-sensor-profile,
2022 each of which when called returns sensory data for that body part."
2023 [#^Node creature]
2024 (filter
2025 (comp not nil?)
2026 (map touch-kernel
2027 (filter #(isa? (class %) Geometry)
2028 (node-seq creature)))))
2029 #+END_SRC
2030 #+end_listing
2032 The tactile-sensor-profile image for the touch cube is a simple
2033 cross with a unifom distribution of touch sensors:
2035 #+caption: The touch profile for the touch-cube. Each pure white
2036 #+caption: pixel defines a touch sensitive feeler.
2037 #+name: touch-cube-uv-map
2038 #+ATTR_LaTeX: :width 10cm
2039 [[./images/touch-profile.png]]
2041 #+caption: The touch cube reacts to canonballs. The black, red,
2042 #+caption: and white cross on the right is a visual display of
2043 #+caption: the creature's touch. White means that it is feeling
2044 #+caption: something strongly, black is not feeling anything,
2045 #+caption: and gray is in-between. The cube can feel both the
2046 #+caption: floor and the ball. Notice that when the ball causes
2047 #+caption: the cube to tip, that the bottom face can still feel
2048 #+caption: part of the ground.
2049 #+name: touch-cube-uv-map
2050 #+ATTR_LaTeX: :width 15cm
2051 [[./images/touch-cube.png]]
2053 ** COMMENT Proprioception is the sense that makes everything ``real''
2055 Close your eyes, and touch your nose with your right index finger.
2056 How did you do it? You could not see your hand, and neither your
2057 hand nor your nose could use the sense of touch to guide the path
2058 of your hand. There are no sound cues, and Taste and Smell
2059 certainly don't provide any help. You know where your hand is
2060 without your other senses because of Proprioception.
2062 Humans can sometimes loose this sense through viral infections or
2063 damage to the spinal cord or brain, and when they do, they loose
2064 the ability to control their own bodies without looking directly at
2065 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2066 Hat]], a woman named Christina looses this sense and has to learn how
2067 to move by carefully watching her arms and legs. She describes
2068 proprioception as the "eyes of the body, the way the body sees
2069 itself".
2071 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2072 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2073 positions of each body part by monitoring muscle strain and length.
2075 It's clear that this is a vital sense for fluid, graceful movement.
2076 It's also particularly easy to implement in jMonkeyEngine.
2078 My simulated proprioception calculates the relative angles of each
2079 joint from the rest position defined in the blender file. This
2080 simulates the muscle-spindles and joint capsules. I will deal with
2081 Golgi tendon organs, which calculate muscle strain, in the next
2082 section.
2084 *** Helper functions
2086 =absolute-angle= calculates the angle between two vectors,
2087 relative to a third axis vector. This angle is the number of
2088 radians you have to move counterclockwise around the axis vector
2089 to get from the first to the second vector. It is not commutative
2090 like a normal dot-product angle is.
2092 The purpose of these functions is to build a system of angle
2093 measurement that is biologically plausable.
2095 #+caption: Program to measure angles along a vector
2096 #+name: helpers
2097 #+begin_listing clojure
2098 #+BEGIN_SRC clojure
2099 (defn right-handed?
2100 "true iff the three vectors form a right handed coordinate
2101 system. The three vectors do not have to be normalized or
2102 orthogonal."
2103 [vec1 vec2 vec3]
2104 (pos? (.dot (.cross vec1 vec2) vec3)))
2106 (defn absolute-angle
2107 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2108 [0 (* 2 Math/PI)]."
2109 [vec1 vec2 axis]
2110 (let [angle (.angleBetween vec1 vec2)]
2111 (if (right-handed? vec1 vec2 axis)
2112 angle (- (* 2 Math/PI) angle))))
2113 #+END_SRC
2114 #+end_listing
2116 *** Proprioception Kernel
2118 Given a joint, =proprioception-kernel= produces a function that
2119 calculates the Euler angles between the the objects the joint
2120 connects. The only tricky part here is making the angles relative
2121 to the joint's initial ``straightness''.
2123 #+caption: Program to return biologially reasonable proprioceptive
2124 #+caption: data for each joint.
2125 #+name: proprioception
2126 #+begin_listing clojure
2127 #+BEGIN_SRC clojure
2128 (defn proprioception-kernel
2129 "Returns a function which returns proprioceptive sensory data when
2130 called inside a running simulation."
2131 [#^Node parts #^Node joint]
2132 (let [[obj-a obj-b] (joint-targets parts joint)
2133 joint-rot (.getWorldRotation joint)
2134 x0 (.mult joint-rot Vector3f/UNIT_X)
2135 y0 (.mult joint-rot Vector3f/UNIT_Y)
2136 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2137 (fn []
2138 (let [rot-a (.clone (.getWorldRotation obj-a))
2139 rot-b (.clone (.getWorldRotation obj-b))
2140 x (.mult rot-a x0)
2141 y (.mult rot-a y0)
2142 z (.mult rot-a z0)
2144 X (.mult rot-b x0)
2145 Y (.mult rot-b y0)
2146 Z (.mult rot-b z0)
2147 heading (Math/atan2 (.dot X z) (.dot X x))
2148 pitch (Math/atan2 (.dot X y) (.dot X x))
2150 ;; rotate x-vector back to origin
2151 reverse
2152 (doto (Quaternion.)
2153 (.fromAngleAxis
2154 (.angleBetween X x)
2155 (let [cross (.normalize (.cross X x))]
2156 (if (= 0 (.length cross)) y cross))))
2157 roll (absolute-angle (.mult reverse Y) y x)]
2158 [heading pitch roll]))))
2160 (defn proprioception!
2161 "Endow the creature with the sense of proprioception. Returns a
2162 sequence of functions, one for each child of the \"joints\" node in
2163 the creature, which each report proprioceptive information about
2164 that joint."
2165 [#^Node creature]
2166 ;; extract the body's joints
2167 (let [senses (map (partial proprioception-kernel creature)
2168 (joints creature))]
2169 (fn []
2170 (map #(%) senses))))
2171 #+END_SRC
2172 #+end_listing
2174 =proprioception!= maps =proprioception-kernel= across all the
2175 joints of the creature. It uses the same list of joints that
2176 =joints= uses. Proprioception is the easiest sense to implement in
2177 =CORTEX=, and it will play a crucial role when efficiently
2178 implementing empathy.
2180 #+caption: In the upper right corner, the three proprioceptive
2181 #+caption: angle measurements are displayed. Red is yaw, Green is
2182 #+caption: pitch, and White is roll.
2183 #+name: proprio
2184 #+ATTR_LaTeX: :width 11cm
2185 [[./images/proprio.png]]
2187 ** Muscles are both effectors and sensors
2189 ** =CORTEX= brings complex creatures to life!
2191 ** =CORTEX= enables many possiblities for further research
2193 * COMMENT Empathy in a simulated worm
2195 Here I develop a computational model of empathy, using =CORTEX= as a
2196 base. Empathy in this context is the ability to observe another
2197 creature and infer what sorts of sensations that creature is
2198 feeling. My empathy algorithm involves multiple phases. First is
2199 free-play, where the creature moves around and gains sensory
2200 experience. From this experience I construct a representation of the
2201 creature's sensory state space, which I call \Phi-space. Using
2202 \Phi-space, I construct an efficient function which takes the
2203 limited data that comes from observing another creature and enriches
2204 it full compliment of imagined sensory data. I can then use the
2205 imagined sensory data to recognize what the observed creature is
2206 doing and feeling, using straightforward embodied action predicates.
2207 This is all demonstrated with using a simple worm-like creature, and
2208 recognizing worm-actions based on limited data.
2210 #+caption: Here is the worm with which we will be working.
2211 #+caption: It is composed of 5 segments. Each segment has a
2212 #+caption: pair of extensor and flexor muscles. Each of the
2213 #+caption: worm's four joints is a hinge joint which allows
2214 #+caption: about 30 degrees of rotation to either side. Each segment
2215 #+caption: of the worm is touch-capable and has a uniform
2216 #+caption: distribution of touch sensors on each of its faces.
2217 #+caption: Each joint has a proprioceptive sense to detect
2218 #+caption: relative positions. The worm segments are all the
2219 #+caption: same except for the first one, which has a much
2220 #+caption: higher weight than the others to allow for easy
2221 #+caption: manual motor control.
2222 #+name: basic-worm-view
2223 #+ATTR_LaTeX: :width 10cm
2224 [[./images/basic-worm-view.png]]
2226 #+caption: Program for reading a worm from a blender file and
2227 #+caption: outfitting it with the senses of proprioception,
2228 #+caption: touch, and the ability to move, as specified in the
2229 #+caption: blender file.
2230 #+name: get-worm
2231 #+begin_listing clojure
2232 #+begin_src clojure
2233 (defn worm []
2234 (let [model (load-blender-model "Models/worm/worm.blend")]
2235 {:body (doto model (body!))
2236 :touch (touch! model)
2237 :proprioception (proprioception! model)
2238 :muscles (movement! model)}))
2239 #+end_src
2240 #+end_listing
2242 ** Embodiment factors action recognition into managable parts
2244 Using empathy, I divide the problem of action recognition into a
2245 recognition process expressed in the language of a full compliment
2246 of senses, and an imaganitive process that generates full sensory
2247 data from partial sensory data. Splitting the action recognition
2248 problem in this manner greatly reduces the total amount of work to
2249 recognize actions: The imaganitive process is mostly just matching
2250 previous experience, and the recognition process gets to use all
2251 the senses to directly describe any action.
2253 ** Action recognition is easy with a full gamut of senses
2255 Embodied representations using multiple senses such as touch,
2256 proprioception, and muscle tension turns out be be exceedingly
2257 efficient at describing body-centered actions. It is the ``right
2258 language for the job''. For example, it takes only around 5 lines
2259 of LISP code to describe the action of ``curling'' using embodied
2260 primitives. It takes about 10 lines to describe the seemingly
2261 complicated action of wiggling.
2263 The following action predicates each take a stream of sensory
2264 experience, observe however much of it they desire, and decide
2265 whether the worm is doing the action they describe. =curled?=
2266 relies on proprioception, =resting?= relies on touch, =wiggling?=
2267 relies on a fourier analysis of muscle contraction, and
2268 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
2270 #+caption: Program for detecting whether the worm is curled. This is the
2271 #+caption: simplest action predicate, because it only uses the last frame
2272 #+caption: of sensory experience, and only uses proprioceptive data. Even
2273 #+caption: this simple predicate, however, is automatically frame
2274 #+caption: independent and ignores vermopomorphic differences such as
2275 #+caption: worm textures and colors.
2276 #+name: curled
2277 #+attr_latex: [htpb]
2278 #+begin_listing clojure
2279 #+begin_src clojure
2280 (defn curled?
2281 "Is the worm curled up?"
2282 [experiences]
2283 (every?
2284 (fn [[_ _ bend]]
2285 (> (Math/sin bend) 0.64))
2286 (:proprioception (peek experiences))))
2287 #+end_src
2288 #+end_listing
2290 #+caption: Program for summarizing the touch information in a patch
2291 #+caption: of skin.
2292 #+name: touch-summary
2293 #+attr_latex: [htpb]
2295 #+begin_listing clojure
2296 #+begin_src clojure
2297 (defn contact
2298 "Determine how much contact a particular worm segment has with
2299 other objects. Returns a value between 0 and 1, where 1 is full
2300 contact and 0 is no contact."
2301 [touch-region [coords contact :as touch]]
2302 (-> (zipmap coords contact)
2303 (select-keys touch-region)
2304 (vals)
2305 (#(map first %))
2306 (average)
2307 (* 10)
2308 (- 1)
2309 (Math/abs)))
2310 #+end_src
2311 #+end_listing
2314 #+caption: Program for detecting whether the worm is at rest. This program
2315 #+caption: uses a summary of the tactile information from the underbelly
2316 #+caption: of the worm, and is only true if every segment is touching the
2317 #+caption: floor. Note that this function contains no references to
2318 #+caption: proprioction at all.
2319 #+name: resting
2320 #+attr_latex: [htpb]
2321 #+begin_listing clojure
2322 #+begin_src clojure
2323 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2325 (defn resting?
2326 "Is the worm resting on the ground?"
2327 [experiences]
2328 (every?
2329 (fn [touch-data]
2330 (< 0.9 (contact worm-segment-bottom touch-data)))
2331 (:touch (peek experiences))))
2332 #+end_src
2333 #+end_listing
2335 #+caption: Program for detecting whether the worm is curled up into a
2336 #+caption: full circle. Here the embodied approach begins to shine, as
2337 #+caption: I am able to both use a previous action predicate (=curled?=)
2338 #+caption: as well as the direct tactile experience of the head and tail.
2339 #+name: grand-circle
2340 #+attr_latex: [htpb]
2341 #+begin_listing clojure
2342 #+begin_src clojure
2343 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2345 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2347 (defn grand-circle?
2348 "Does the worm form a majestic circle (one end touching the other)?"
2349 [experiences]
2350 (and (curled? experiences)
2351 (let [worm-touch (:touch (peek experiences))
2352 tail-touch (worm-touch 0)
2353 head-touch (worm-touch 4)]
2354 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2355 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2356 #+end_src
2357 #+end_listing
2360 #+caption: Program for detecting whether the worm has been wiggling for
2361 #+caption: the last few frames. It uses a fourier analysis of the muscle
2362 #+caption: contractions of the worm's tail to determine wiggling. This is
2363 #+caption: signigicant because there is no particular frame that clearly
2364 #+caption: indicates that the worm is wiggling --- only when multiple frames
2365 #+caption: are analyzed together is the wiggling revealed. Defining
2366 #+caption: wiggling this way also gives the worm an opportunity to learn
2367 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2368 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2369 #+caption: from actual wiggling, but this definition gives it to us for free.
2370 #+name: wiggling
2371 #+attr_latex: [htpb]
2372 #+begin_listing clojure
2373 #+begin_src clojure
2374 (defn fft [nums]
2375 (map
2376 #(.getReal %)
2377 (.transform
2378 (FastFourierTransformer. DftNormalization/STANDARD)
2379 (double-array nums) TransformType/FORWARD)))
2381 (def indexed (partial map-indexed vector))
2383 (defn max-indexed [s]
2384 (first (sort-by (comp - second) (indexed s))))
2386 (defn wiggling?
2387 "Is the worm wiggling?"
2388 [experiences]
2389 (let [analysis-interval 0x40]
2390 (when (> (count experiences) analysis-interval)
2391 (let [a-flex 3
2392 a-ex 2
2393 muscle-activity
2394 (map :muscle (vector:last-n experiences analysis-interval))
2395 base-activity
2396 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2397 (= 2
2398 (first
2399 (max-indexed
2400 (map #(Math/abs %)
2401 (take 20 (fft base-activity))))))))))
2402 #+end_src
2403 #+end_listing
2405 With these action predicates, I can now recognize the actions of
2406 the worm while it is moving under my control and I have access to
2407 all the worm's senses.
2409 #+caption: Use the action predicates defined earlier to report on
2410 #+caption: what the worm is doing while in simulation.
2411 #+name: report-worm-activity
2412 #+attr_latex: [htpb]
2413 #+begin_listing clojure
2414 #+begin_src clojure
2415 (defn debug-experience
2416 [experiences text]
2417 (cond
2418 (grand-circle? experiences) (.setText text "Grand Circle")
2419 (curled? experiences) (.setText text "Curled")
2420 (wiggling? experiences) (.setText text "Wiggling")
2421 (resting? experiences) (.setText text "Resting")))
2422 #+end_src
2423 #+end_listing
2425 #+caption: Using =debug-experience=, the body-centered predicates
2426 #+caption: work together to classify the behaviour of the worm.
2427 #+caption: the predicates are operating with access to the worm's
2428 #+caption: full sensory data.
2429 #+name: basic-worm-view
2430 #+ATTR_LaTeX: :width 10cm
2431 [[./images/worm-identify-init.png]]
2433 These action predicates satisfy the recognition requirement of an
2434 empathic recognition system. There is power in the simplicity of
2435 the action predicates. They describe their actions without getting
2436 confused in visual details of the worm. Each one is frame
2437 independent, but more than that, they are each indepent of
2438 irrelevant visual details of the worm and the environment. They
2439 will work regardless of whether the worm is a different color or
2440 hevaily textured, or if the environment has strange lighting.
2442 The trick now is to make the action predicates work even when the
2443 sensory data on which they depend is absent. If I can do that, then
2444 I will have gained much,
2446 ** \Phi-space describes the worm's experiences
2448 As a first step towards building empathy, I need to gather all of
2449 the worm's experiences during free play. I use a simple vector to
2450 store all the experiences.
2452 Each element of the experience vector exists in the vast space of
2453 all possible worm-experiences. Most of this vast space is actually
2454 unreachable due to physical constraints of the worm's body. For
2455 example, the worm's segments are connected by hinge joints that put
2456 a practical limit on the worm's range of motions without limiting
2457 its degrees of freedom. Some groupings of senses are impossible;
2458 the worm can not be bent into a circle so that its ends are
2459 touching and at the same time not also experience the sensation of
2460 touching itself.
2462 As the worm moves around during free play and its experience vector
2463 grows larger, the vector begins to define a subspace which is all
2464 the sensations the worm can practicaly experience during normal
2465 operation. I call this subspace \Phi-space, short for
2466 physical-space. The experience vector defines a path through
2467 \Phi-space. This path has interesting properties that all derive
2468 from physical embodiment. The proprioceptive components are
2469 completely smooth, because in order for the worm to move from one
2470 position to another, it must pass through the intermediate
2471 positions. The path invariably forms loops as actions are repeated.
2472 Finally and most importantly, proprioception actually gives very
2473 strong inference about the other senses. For example, when the worm
2474 is flat, you can infer that it is touching the ground and that its
2475 muscles are not active, because if the muscles were active, the
2476 worm would be moving and would not be perfectly flat. In order to
2477 stay flat, the worm has to be touching the ground, or it would
2478 again be moving out of the flat position due to gravity. If the
2479 worm is positioned in such a way that it interacts with itself,
2480 then it is very likely to be feeling the same tactile feelings as
2481 the last time it was in that position, because it has the same body
2482 as then. If you observe multiple frames of proprioceptive data,
2483 then you can become increasingly confident about the exact
2484 activations of the worm's muscles, because it generally takes a
2485 unique combination of muscle contractions to transform the worm's
2486 body along a specific path through \Phi-space.
2488 There is a simple way of taking \Phi-space and the total ordering
2489 provided by an experience vector and reliably infering the rest of
2490 the senses.
2492 ** Empathy is the process of tracing though \Phi-space
2494 Here is the core of a basic empathy algorithm, starting with an
2495 experience vector:
2497 First, group the experiences into tiered proprioceptive bins. I use
2498 powers of 10 and 3 bins, and the smallest bin has an approximate
2499 size of 0.001 radians in all proprioceptive dimensions.
2501 Then, given a sequence of proprioceptive input, generate a set of
2502 matching experience records for each input, using the tiered
2503 proprioceptive bins.
2505 Finally, to infer sensory data, select the longest consective chain
2506 of experiences. Conecutive experience means that the experiences
2507 appear next to each other in the experience vector.
2509 This algorithm has three advantages:
2511 1. It's simple
2513 3. It's very fast -- retrieving possible interpretations takes
2514 constant time. Tracing through chains of interpretations takes
2515 time proportional to the average number of experiences in a
2516 proprioceptive bin. Redundant experiences in \Phi-space can be
2517 merged to save computation.
2519 2. It protects from wrong interpretations of transient ambiguous
2520 proprioceptive data. For example, if the worm is flat for just
2521 an instant, this flattness will not be interpreted as implying
2522 that the worm has its muscles relaxed, since the flattness is
2523 part of a longer chain which includes a distinct pattern of
2524 muscle activation. Markov chains or other memoryless statistical
2525 models that operate on individual frames may very well make this
2526 mistake.
2528 #+caption: Program to convert an experience vector into a
2529 #+caption: proprioceptively binned lookup function.
2530 #+name: bin
2531 #+attr_latex: [htpb]
2532 #+begin_listing clojure
2533 #+begin_src clojure
2534 (defn bin [digits]
2535 (fn [angles]
2536 (->> angles
2537 (flatten)
2538 (map (juxt #(Math/sin %) #(Math/cos %)))
2539 (flatten)
2540 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2542 (defn gen-phi-scan
2543 "Nearest-neighbors with binning. Only returns a result if
2544 the propriceptive data is within 10% of a previously recorded
2545 result in all dimensions."
2546 [phi-space]
2547 (let [bin-keys (map bin [3 2 1])
2548 bin-maps
2549 (map (fn [bin-key]
2550 (group-by
2551 (comp bin-key :proprioception phi-space)
2552 (range (count phi-space)))) bin-keys)
2553 lookups (map (fn [bin-key bin-map]
2554 (fn [proprio] (bin-map (bin-key proprio))))
2555 bin-keys bin-maps)]
2556 (fn lookup [proprio-data]
2557 (set (some #(% proprio-data) lookups)))))
2558 #+end_src
2559 #+end_listing
2561 #+caption: =longest-thread= finds the longest path of consecutive
2562 #+caption: experiences to explain proprioceptive worm data.
2563 #+name: phi-space-history-scan
2564 #+ATTR_LaTeX: :width 10cm
2565 [[./images/aurellem-gray.png]]
2567 =longest-thread= infers sensory data by stitching together pieces
2568 from previous experience. It prefers longer chains of previous
2569 experience to shorter ones. For example, during training the worm
2570 might rest on the ground for one second before it performs its
2571 excercises. If during recognition the worm rests on the ground for
2572 five seconds, =longest-thread= will accomodate this five second
2573 rest period by looping the one second rest chain five times.
2575 =longest-thread= takes time proportinal to the average number of
2576 entries in a proprioceptive bin, because for each element in the
2577 starting bin it performes a series of set lookups in the preceeding
2578 bins. If the total history is limited, then this is only a constant
2579 multiple times the number of entries in the starting bin. This
2580 analysis also applies even if the action requires multiple longest
2581 chains -- it's still the average number of entries in a
2582 proprioceptive bin times the desired chain length. Because
2583 =longest-thread= is so efficient and simple, I can interpret
2584 worm-actions in real time.
2586 #+caption: Program to calculate empathy by tracing though \Phi-space
2587 #+caption: and finding the longest (ie. most coherent) interpretation
2588 #+caption: of the data.
2589 #+name: longest-thread
2590 #+attr_latex: [htpb]
2591 #+begin_listing clojure
2592 #+begin_src clojure
2593 (defn longest-thread
2594 "Find the longest thread from phi-index-sets. The index sets should
2595 be ordered from most recent to least recent."
2596 [phi-index-sets]
2597 (loop [result '()
2598 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2599 (if (empty? phi-index-sets)
2600 (vec result)
2601 (let [threads
2602 (for [thread-base thread-bases]
2603 (loop [thread (list thread-base)
2604 remaining remaining]
2605 (let [next-index (dec (first thread))]
2606 (cond (empty? remaining) thread
2607 (contains? (first remaining) next-index)
2608 (recur
2609 (cons next-index thread) (rest remaining))
2610 :else thread))))
2611 longest-thread
2612 (reduce (fn [thread-a thread-b]
2613 (if (> (count thread-a) (count thread-b))
2614 thread-a thread-b))
2615 '(nil)
2616 threads)]
2617 (recur (concat longest-thread result)
2618 (drop (count longest-thread) phi-index-sets))))))
2619 #+end_src
2620 #+end_listing
2622 There is one final piece, which is to replace missing sensory data
2623 with a best-guess estimate. While I could fill in missing data by
2624 using a gradient over the closest known sensory data points,
2625 averages can be misleading. It is certainly possible to create an
2626 impossible sensory state by averaging two possible sensory states.
2627 Therefore, I simply replicate the most recent sensory experience to
2628 fill in the gaps.
2630 #+caption: Fill in blanks in sensory experience by replicating the most
2631 #+caption: recent experience.
2632 #+name: infer-nils
2633 #+attr_latex: [htpb]
2634 #+begin_listing clojure
2635 #+begin_src clojure
2636 (defn infer-nils
2637 "Replace nils with the next available non-nil element in the
2638 sequence, or barring that, 0."
2639 [s]
2640 (loop [i (dec (count s))
2641 v (transient s)]
2642 (if (zero? i) (persistent! v)
2643 (if-let [cur (v i)]
2644 (if (get v (dec i) 0)
2645 (recur (dec i) v)
2646 (recur (dec i) (assoc! v (dec i) cur)))
2647 (recur i (assoc! v i 0))))))
2648 #+end_src
2649 #+end_listing
2651 ** Efficient action recognition with =EMPATH=
2653 To use =EMPATH= with the worm, I first need to gather a set of
2654 experiences from the worm that includes the actions I want to
2655 recognize. The =generate-phi-space= program (listing
2656 \ref{generate-phi-space} runs the worm through a series of
2657 exercices and gatheres those experiences into a vector. The
2658 =do-all-the-things= program is a routine expressed in a simple
2659 muscle contraction script language for automated worm control. It
2660 causes the worm to rest, curl, and wiggle over about 700 frames
2661 (approx. 11 seconds).
2663 #+caption: Program to gather the worm's experiences into a vector for
2664 #+caption: further processing. The =motor-control-program= line uses
2665 #+caption: a motor control script that causes the worm to execute a series
2666 #+caption: of ``exercices'' that include all the action predicates.
2667 #+name: generate-phi-space
2668 #+attr_latex: [htpb]
2669 #+begin_listing clojure
2670 #+begin_src clojure
2671 (def do-all-the-things
2672 (concat
2673 curl-script
2674 [[300 :d-ex 40]
2675 [320 :d-ex 0]]
2676 (shift-script 280 (take 16 wiggle-script))))
2678 (defn generate-phi-space []
2679 (let [experiences (atom [])]
2680 (run-world
2681 (apply-map
2682 worm-world
2683 (merge
2684 (worm-world-defaults)
2685 {:end-frame 700
2686 :motor-control
2687 (motor-control-program worm-muscle-labels do-all-the-things)
2688 :experiences experiences})))
2689 @experiences))
2690 #+end_src
2691 #+end_listing
2693 #+caption: Use longest thread and a phi-space generated from a short
2694 #+caption: exercise routine to interpret actions during free play.
2695 #+name: empathy-debug
2696 #+attr_latex: [htpb]
2697 #+begin_listing clojure
2698 #+begin_src clojure
2699 (defn init []
2700 (def phi-space (generate-phi-space))
2701 (def phi-scan (gen-phi-scan phi-space)))
2703 (defn empathy-demonstration []
2704 (let [proprio (atom ())]
2705 (fn
2706 [experiences text]
2707 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2708 (swap! proprio (partial cons phi-indices))
2709 (let [exp-thread (longest-thread (take 300 @proprio))
2710 empathy (mapv phi-space (infer-nils exp-thread))]
2711 (println-repl (vector:last-n exp-thread 22))
2712 (cond
2713 (grand-circle? empathy) (.setText text "Grand Circle")
2714 (curled? empathy) (.setText text "Curled")
2715 (wiggling? empathy) (.setText text "Wiggling")
2716 (resting? empathy) (.setText text "Resting")
2717 :else (.setText text "Unknown")))))))
2719 (defn empathy-experiment [record]
2720 (.start (worm-world :experience-watch (debug-experience-phi)
2721 :record record :worm worm*)))
2722 #+end_src
2723 #+end_listing
2725 The result of running =empathy-experiment= is that the system is
2726 generally able to interpret worm actions using the action-predicates
2727 on simulated sensory data just as well as with actual data. Figure
2728 \ref{empathy-debug-image} was generated using =empathy-experiment=:
2730 #+caption: From only proprioceptive data, =EMPATH= was able to infer
2731 #+caption: the complete sensory experience and classify four poses
2732 #+caption: (The last panel shows a composite image of \emph{wriggling},
2733 #+caption: a dynamic pose.)
2734 #+name: empathy-debug-image
2735 #+ATTR_LaTeX: :width 10cm :placement [H]
2736 [[./images/empathy-1.png]]
2738 One way to measure the performance of =EMPATH= is to compare the
2739 sutiability of the imagined sense experience to trigger the same
2740 action predicates as the real sensory experience.
2742 #+caption: Determine how closely empathy approximates actual
2743 #+caption: sensory data.
2744 #+name: test-empathy-accuracy
2745 #+attr_latex: [htpb]
2746 #+begin_listing clojure
2747 #+begin_src clojure
2748 (def worm-action-label
2749 (juxt grand-circle? curled? wiggling?))
2751 (defn compare-empathy-with-baseline [matches]
2752 (let [proprio (atom ())]
2753 (fn
2754 [experiences text]
2755 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2756 (swap! proprio (partial cons phi-indices))
2757 (let [exp-thread (longest-thread (take 300 @proprio))
2758 empathy (mapv phi-space (infer-nils exp-thread))
2759 experience-matches-empathy
2760 (= (worm-action-label experiences)
2761 (worm-action-label empathy))]
2762 (println-repl experience-matches-empathy)
2763 (swap! matches #(conj % experience-matches-empathy)))))))
2765 (defn accuracy [v]
2766 (float (/ (count (filter true? v)) (count v))))
2768 (defn test-empathy-accuracy []
2769 (let [res (atom [])]
2770 (run-world
2771 (worm-world :experience-watch
2772 (compare-empathy-with-baseline res)
2773 :worm worm*))
2774 (accuracy @res)))
2775 #+end_src
2776 #+end_listing
2778 Running =test-empathy-accuracy= using the very short exercise
2779 program defined in listing \ref{generate-phi-space}, and then doing
2780 a similar pattern of activity manually yeilds an accuracy of around
2781 73%. This is based on very limited worm experience. By training the
2782 worm for longer, the accuracy dramatically improves.
2784 #+caption: Program to generate \Phi-space using manual training.
2785 #+name: manual-phi-space
2786 #+attr_latex: [htpb]
2787 #+begin_listing clojure
2788 #+begin_src clojure
2789 (defn init-interactive []
2790 (def phi-space
2791 (let [experiences (atom [])]
2792 (run-world
2793 (apply-map
2794 worm-world
2795 (merge
2796 (worm-world-defaults)
2797 {:experiences experiences})))
2798 @experiences))
2799 (def phi-scan (gen-phi-scan phi-space)))
2800 #+end_src
2801 #+end_listing
2803 After about 1 minute of manual training, I was able to achieve 95%
2804 accuracy on manual testing of the worm using =init-interactive= and
2805 =test-empathy-accuracy=. The majority of errors are near the
2806 boundaries of transitioning from one type of action to another.
2807 During these transitions the exact label for the action is more open
2808 to interpretation, and dissaggrement between empathy and experience
2809 is more excusable.
2811 ** Digression: bootstrapping touch using free exploration
2813 In the previous section I showed how to compute actions in terms of
2814 body-centered predicates which relied averate touch activation of
2815 pre-defined regions of the worm's skin. What if, instead of recieving
2816 touch pre-grouped into the six faces of each worm segment, the true
2817 topology of the worm's skin was unknown? This is more similiar to how
2818 a nerve fiber bundle might be arranged. While two fibers that are
2819 close in a nerve bundle /might/ correspond to two touch sensors that
2820 are close together on the skin, the process of taking a complicated
2821 surface and forcing it into essentially a circle requires some cuts
2822 and rerragenments.
2824 In this section I show how to automatically learn the skin-topology of
2825 a worm segment by free exploration. As the worm rolls around on the
2826 floor, large sections of its surface get activated. If the worm has
2827 stopped moving, then whatever region of skin that is touching the
2828 floor is probably an important region, and should be recorded.
2830 #+caption: Program to detect whether the worm is in a resting state
2831 #+caption: with one face touching the floor.
2832 #+name: pure-touch
2833 #+begin_listing clojure
2834 #+begin_src clojure
2835 (def full-contact [(float 0.0) (float 0.1)])
2837 (defn pure-touch?
2838 "This is worm specific code to determine if a large region of touch
2839 sensors is either all on or all off."
2840 [[coords touch :as touch-data]]
2841 (= (set (map first touch)) (set full-contact)))
2842 #+end_src
2843 #+end_listing
2845 After collecting these important regions, there will many nearly
2846 similiar touch regions. While for some purposes the subtle
2847 differences between these regions will be important, for my
2848 purposes I colapse them into mostly non-overlapping sets using
2849 =remove-similiar= in listing \ref{remove-similiar}
2851 #+caption: Program to take a lits of set of points and ``collapse them''
2852 #+caption: so that the remaining sets in the list are siginificantly
2853 #+caption: different from each other. Prefer smaller sets to larger ones.
2854 #+name: remove-similiar
2855 #+begin_listing clojure
2856 #+begin_src clojure
2857 (defn remove-similar
2858 [coll]
2859 (loop [result () coll (sort-by (comp - count) coll)]
2860 (if (empty? coll) result
2861 (let [[x & xs] coll
2862 c (count x)]
2863 (if (some
2864 (fn [other-set]
2865 (let [oc (count other-set)]
2866 (< (- (count (union other-set x)) c) (* oc 0.1))))
2867 xs)
2868 (recur result xs)
2869 (recur (cons x result) xs))))))
2870 #+end_src
2871 #+end_listing
2873 Actually running this simulation is easy given =CORTEX='s facilities.
2875 #+caption: Collect experiences while the worm moves around. Filter the touch
2876 #+caption: sensations by stable ones, collapse similiar ones together,
2877 #+caption: and report the regions learned.
2878 #+name: learn-touch
2879 #+begin_listing clojure
2880 #+begin_src clojure
2881 (defn learn-touch-regions []
2882 (let [experiences (atom [])
2883 world (apply-map
2884 worm-world
2885 (assoc (worm-segment-defaults)
2886 :experiences experiences))]
2887 (run-world world)
2888 (->>
2889 @experiences
2890 (drop 175)
2891 ;; access the single segment's touch data
2892 (map (comp first :touch))
2893 ;; only deal with "pure" touch data to determine surfaces
2894 (filter pure-touch?)
2895 ;; associate coordinates with touch values
2896 (map (partial apply zipmap))
2897 ;; select those regions where contact is being made
2898 (map (partial group-by second))
2899 (map #(get % full-contact))
2900 (map (partial map first))
2901 ;; remove redundant/subset regions
2902 (map set)
2903 remove-similar)))
2905 (defn learn-and-view-touch-regions []
2906 (map view-touch-region
2907 (learn-touch-regions)))
2908 #+end_src
2909 #+end_listing
2911 The only thing remining to define is the particular motion the worm
2912 must take. I accomplish this with a simple motor control program.
2914 #+caption: Motor control program for making the worm roll on the ground.
2915 #+caption: This could also be replaced with random motion.
2916 #+name: worm-roll
2917 #+begin_listing clojure
2918 #+begin_src clojure
2919 (defn touch-kinesthetics []
2920 [[170 :lift-1 40]
2921 [190 :lift-1 19]
2922 [206 :lift-1 0]
2924 [400 :lift-2 40]
2925 [410 :lift-2 0]
2927 [570 :lift-2 40]
2928 [590 :lift-2 21]
2929 [606 :lift-2 0]
2931 [800 :lift-1 30]
2932 [809 :lift-1 0]
2934 [900 :roll-2 40]
2935 [905 :roll-2 20]
2936 [910 :roll-2 0]
2938 [1000 :roll-2 40]
2939 [1005 :roll-2 20]
2940 [1010 :roll-2 0]
2942 [1100 :roll-2 40]
2943 [1105 :roll-2 20]
2944 [1110 :roll-2 0]
2945 ])
2946 #+end_src
2947 #+end_listing
2950 #+caption: The small worm rolls around on the floor, driven
2951 #+caption: by the motor control program in listing \ref{worm-roll}.
2952 #+name: worm-roll
2953 #+ATTR_LaTeX: :width 12cm
2954 [[./images/worm-roll.png]]
2957 #+caption: After completing its adventures, the worm now knows
2958 #+caption: how its touch sensors are arranged along its skin. These
2959 #+caption: are the regions that were deemed important by
2960 #+caption: =learn-touch-regions=. Note that the worm has discovered
2961 #+caption: that it has six sides.
2962 #+name: worm-touch-map
2963 #+ATTR_LaTeX: :width 12cm
2964 [[./images/touch-learn.png]]
2966 While simple, =learn-touch-regions= exploits regularities in both
2967 the worm's physiology and the worm's environment to correctly
2968 deduce that the worm has six sides. Note that =learn-touch-regions=
2969 would work just as well even if the worm's touch sense data were
2970 completely scrambled. The cross shape is just for convienence. This
2971 example justifies the use of pre-defined touch regions in =EMPATH=.
2973 * COMMENT Contributions
2975 In this thesis you have seen the =CORTEX= system, a complete
2976 environment for creating simulated creatures. You have seen how to
2977 implement five senses including touch, proprioception, hearing,
2978 vision, and muscle tension. You have seen how to create new creatues
2979 using blender, a 3D modeling tool. I hope that =CORTEX= will be
2980 useful in further research projects. To this end I have included the
2981 full source to =CORTEX= along with a large suite of tests and
2982 examples. I have also created a user guide for =CORTEX= which is
2983 inculded in an appendix to this thesis.
2985 You have also seen how I used =CORTEX= as a platform to attach the
2986 /action recognition/ problem, which is the problem of recognizing
2987 actions in video. You saw a simple system called =EMPATH= which
2988 ientifies actions by first describing actions in a body-centerd,
2989 rich sense language, then infering a full range of sensory
2990 experience from limited data using previous experience gained from
2991 free play.
2993 As a minor digression, you also saw how I used =CORTEX= to enable a
2994 tiny worm to discover the topology of its skin simply by rolling on
2995 the ground.
2997 In conclusion, the main contributions of this thesis are:
2999 - =CORTEX=, a system for creating simulated creatures with rich
3000 senses.
3001 - =EMPATH=, a program for recognizing actions by imagining sensory
3002 experience.
3004 # An anatomical joke:
3005 # - Training
3006 # - Skeletal imitation
3007 # - Sensory fleshing-out
3008 # - Classification