view thesis/cortex.org @ 504:609356a82760

prepare to fix listings.
author Robert McIntyre <rlm@mit.edu>
date Sat, 29 Mar 2014 23:57:29 -0400
parents c816594fbbe6
children c15e24d24396
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * COMMENT Empathy and Embodiment as problem solving strategies
46 By the end of this thesis, you will have seen a novel approach to
47 interpreting video using embodiment and empathy. You will have also
48 seen one way to efficiently implement empathy for embodied
49 creatures. Finally, you will become familiar with =CORTEX=, a system
50 for designing and simulating creatures with rich senses, which you
51 may choose to use in your own research.
53 This is the core vision of my thesis: That one of the important ways
54 in which we understand others is by imagining ourselves in their
55 position and emphatically feeling experiences relative to our own
56 bodies. By understanding events in terms of our own previous
57 corporeal experience, we greatly constrain the possibilities of what
58 would otherwise be an unwieldy exponential search. This extra
59 constraint can be the difference between easily understanding what
60 is happening in a video and being completely lost in a sea of
61 incomprehensible color and movement.
63 ** Recognizing actions in video is extremely difficult
65 Consider for example the problem of determining what is happening
66 in a video of which this is one frame:
68 #+caption: A cat drinking some water. Identifying this action is
69 #+caption: beyond the state of the art for computers.
70 #+ATTR_LaTeX: :width 7cm
71 [[./images/cat-drinking.jpg]]
73 It is currently impossible for any computer program to reliably
74 label such a video as ``drinking''. And rightly so -- it is a very
75 hard problem! What features can you describe in terms of low level
76 functions of pixels that can even begin to describe at a high level
77 what is happening here?
79 Or suppose that you are building a program that recognizes chairs.
80 How could you ``see'' the chair in figure \ref{hidden-chair}?
82 #+caption: The chair in this image is quite obvious to humans, but I
83 #+caption: doubt that any modern computer vision program can find it.
84 #+name: hidden-chair
85 #+ATTR_LaTeX: :width 10cm
86 [[./images/fat-person-sitting-at-desk.jpg]]
88 Finally, how is it that you can easily tell the difference between
89 how the girls /muscles/ are working in figure \ref{girl}?
91 #+caption: The mysterious ``common sense'' appears here as you are able
92 #+caption: to discern the difference in how the girl's arm muscles
93 #+caption: are activated between the two images.
94 #+name: girl
95 #+ATTR_LaTeX: :width 7cm
96 [[./images/wall-push.png]]
98 Each of these examples tells us something about what might be going
99 on in our minds as we easily solve these recognition problems.
101 The hidden chairs show us that we are strongly triggered by cues
102 relating to the position of human bodies, and that we can determine
103 the overall physical configuration of a human body even if much of
104 that body is occluded.
106 The picture of the girl pushing against the wall tells us that we
107 have common sense knowledge about the kinetics of our own bodies.
108 We know well how our muscles would have to work to maintain us in
109 most positions, and we can easily project this self-knowledge to
110 imagined positions triggered by images of the human body.
112 ** =EMPATH= neatly solves recognition problems
114 I propose a system that can express the types of recognition
115 problems above in a form amenable to computation. It is split into
116 four parts:
118 - Free/Guided Play :: The creature moves around and experiences the
119 world through its unique perspective. Many otherwise
120 complicated actions are easily described in the language of a
121 full suite of body-centered, rich senses. For example,
122 drinking is the feeling of water sliding down your throat, and
123 cooling your insides. It's often accompanied by bringing your
124 hand close to your face, or bringing your face close to water.
125 Sitting down is the feeling of bending your knees, activating
126 your quadriceps, then feeling a surface with your bottom and
127 relaxing your legs. These body-centered action descriptions
128 can be either learned or hard coded.
129 - Posture Imitation :: When trying to interpret a video or image,
130 the creature takes a model of itself and aligns it with
131 whatever it sees. This alignment can even cross species, as
132 when humans try to align themselves with things like ponies,
133 dogs, or other humans with a different body type.
134 - Empathy :: The alignment triggers associations with
135 sensory data from prior experiences. For example, the
136 alignment itself easily maps to proprioceptive data. Any
137 sounds or obvious skin contact in the video can to a lesser
138 extent trigger previous experience. Segments of previous
139 experiences are stitched together to form a coherent and
140 complete sensory portrait of the scene.
141 - Recognition :: With the scene described in terms of first
142 person sensory events, the creature can now run its
143 action-identification programs on this synthesized sensory
144 data, just as it would if it were actually experiencing the
145 scene first-hand. If previous experience has been accurately
146 retrieved, and if it is analogous enough to the scene, then
147 the creature will correctly identify the action in the scene.
149 For example, I think humans are able to label the cat video as
150 ``drinking'' because they imagine /themselves/ as the cat, and
151 imagine putting their face up against a stream of water and
152 sticking out their tongue. In that imagined world, they can feel
153 the cool water hitting their tongue, and feel the water entering
154 their body, and are able to recognize that /feeling/ as drinking.
155 So, the label of the action is not really in the pixels of the
156 image, but is found clearly in a simulation inspired by those
157 pixels. An imaginative system, having been trained on drinking and
158 non-drinking examples and learning that the most important
159 component of drinking is the feeling of water sliding down one's
160 throat, would analyze a video of a cat drinking in the following
161 manner:
163 1. Create a physical model of the video by putting a ``fuzzy''
164 model of its own body in place of the cat. Possibly also create
165 a simulation of the stream of water.
167 2. Play out this simulated scene and generate imagined sensory
168 experience. This will include relevant muscle contractions, a
169 close up view of the stream from the cat's perspective, and most
170 importantly, the imagined feeling of water entering the
171 mouth. The imagined sensory experience can come from a
172 simulation of the event, but can also be pattern-matched from
173 previous, similar embodied experience.
175 3. The action is now easily identified as drinking by the sense of
176 taste alone. The other senses (such as the tongue moving in and
177 out) help to give plausibility to the simulated action. Note that
178 the sense of vision, while critical in creating the simulation,
179 is not critical for identifying the action from the simulation.
181 For the chair examples, the process is even easier:
183 1. Align a model of your body to the person in the image.
185 2. Generate proprioceptive sensory data from this alignment.
187 3. Use the imagined proprioceptive data as a key to lookup related
188 sensory experience associated with that particular proproceptive
189 feeling.
191 4. Retrieve the feeling of your bottom resting on a surface, your
192 knees bent, and your leg muscles relaxed.
194 5. This sensory information is consistent with the =sitting?=
195 sensory predicate, so you (and the entity in the image) must be
196 sitting.
198 6. There must be a chair-like object since you are sitting.
200 Empathy offers yet another alternative to the age-old AI
201 representation question: ``What is a chair?'' --- A chair is the
202 feeling of sitting.
204 My program, =EMPATH= uses this empathic problem solving technique
205 to interpret the actions of a simple, worm-like creature.
207 #+caption: The worm performs many actions during free play such as
208 #+caption: curling, wiggling, and resting.
209 #+name: worm-intro
210 #+ATTR_LaTeX: :width 15cm
211 [[./images/worm-intro-white.png]]
213 #+caption: =EMPATH= recognized and classified each of these
214 #+caption: poses by inferring the complete sensory experience
215 #+caption: from proprioceptive data.
216 #+name: worm-recognition-intro
217 #+ATTR_LaTeX: :width 15cm
218 [[./images/worm-poses.png]]
220 One powerful advantage of empathic problem solving is that it
221 factors the action recognition problem into two easier problems. To
222 use empathy, you need an /aligner/, which takes the video and a
223 model of your body, and aligns the model with the video. Then, you
224 need a /recognizer/, which uses the aligned model to interpret the
225 action. The power in this method lies in the fact that you describe
226 all actions form a body-centered viewpoint. You are less tied to
227 the particulars of any visual representation of the actions. If you
228 teach the system what ``running'' is, and you have a good enough
229 aligner, the system will from then on be able to recognize running
230 from any point of view, even strange points of view like above or
231 underneath the runner. This is in contrast to action recognition
232 schemes that try to identify actions using a non-embodied approach.
233 If these systems learn about running as viewed from the side, they
234 will not automatically be able to recognize running from any other
235 viewpoint.
237 Another powerful advantage is that using the language of multiple
238 body-centered rich senses to describe body-centerd actions offers a
239 massive boost in descriptive capability. Consider how difficult it
240 would be to compose a set of HOG filters to describe the action of
241 a simple worm-creature ``curling'' so that its head touches its
242 tail, and then behold the simplicity of describing thus action in a
243 language designed for the task (listing \ref{grand-circle-intro}):
245 #+caption: Body-centerd actions are best expressed in a body-centered
246 #+caption: language. This code detects when the worm has curled into a
247 #+caption: full circle. Imagine how you would replicate this functionality
248 #+caption: using low-level pixel features such as HOG filters!
249 #+name: grand-circle-intro
250 #+attr_latex: [htpb]
251 #+begin_listing clojure
252 #+begin_src clojure
253 (defn grand-circle?
254 "Does the worm form a majestic circle (one end touching the other)?"
255 [experiences]
256 (and (curled? experiences)
257 (let [worm-touch (:touch (peek experiences))
258 tail-touch (worm-touch 0)
259 head-touch (worm-touch 4)]
260 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
261 (< 0.2 (contact worm-segment-top-tip head-touch))))))
262 #+end_src
263 #+end_listing
265 ** =CORTEX= is a toolkit for building sensate creatures
267 I built =CORTEX= to be a general AI research platform for doing
268 experiments involving multiple rich senses and a wide variety and
269 number of creatures. I intend it to be useful as a library for many
270 more projects than just this thesis. =CORTEX= was necessary to meet
271 a need among AI researchers at CSAIL and beyond, which is that
272 people often will invent neat ideas that are best expressed in the
273 language of creatures and senses, but in order to explore those
274 ideas they must first build a platform in which they can create
275 simulated creatures with rich senses! There are many ideas that
276 would be simple to execute (such as =EMPATH=), but attached to them
277 is the multi-month effort to make a good creature simulator. Often,
278 that initial investment of time proves to be too much, and the
279 project must make do with a lesser environment.
281 =CORTEX= is well suited as an environment for embodied AI research
282 for three reasons:
284 - You can create new creatures using Blender, a popular 3D modeling
285 program. Each sense can be specified using special blender nodes
286 with biologically inspired paramaters. You need not write any
287 code to create a creature, and can use a wide library of
288 pre-existing blender models as a base for your own creatures.
290 - =CORTEX= implements a wide variety of senses, including touch,
291 proprioception, vision, hearing, and muscle tension. Complicated
292 senses like touch, and vision involve multiple sensory elements
293 embedded in a 2D surface. You have complete control over the
294 distribution of these sensor elements through the use of simple
295 png image files. In particular, =CORTEX= implements more
296 comprehensive hearing than any other creature simulation system
297 available.
299 - =CORTEX= supports any number of creatures and any number of
300 senses. Time in =CORTEX= dialates so that the simulated creatures
301 always precieve a perfectly smooth flow of time, regardless of
302 the actual computational load.
304 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
305 engine designed to create cross-platform 3D desktop games. =CORTEX=
306 is mainly written in clojure, a dialect of =LISP= that runs on the
307 java virtual machine (JVM). The API for creating and simulating
308 creatures and senses is entirely expressed in clojure, though many
309 senses are implemented at the layer of jMonkeyEngine or below. For
310 example, for the sense of hearing I use a layer of clojure code on
311 top of a layer of java JNI bindings that drive a layer of =C++=
312 code which implements a modified version of =OpenAL= to support
313 multiple listeners. =CORTEX= is the only simulation environment
314 that I know of that can support multiple entities that can each
315 hear the world from their own perspective. Other senses also
316 require a small layer of Java code. =CORTEX= also uses =bullet=, a
317 physics simulator written in =C=.
319 #+caption: Here is the worm from above modeled in Blender, a free
320 #+caption: 3D-modeling program. Senses and joints are described
321 #+caption: using special nodes in Blender.
322 #+name: worm-recognition-intro
323 #+ATTR_LaTeX: :width 12cm
324 [[./images/blender-worm.png]]
326 Here are some thing I anticipate that =CORTEX= might be used for:
328 - exploring new ideas about sensory integration
329 - distributed communication among swarm creatures
330 - self-learning using free exploration,
331 - evolutionary algorithms involving creature construction
332 - exploration of exoitic senses and effectors that are not possible
333 in the real world (such as telekenisis or a semantic sense)
334 - imagination using subworlds
336 During one test with =CORTEX=, I created 3,000 creatures each with
337 their own independent senses and ran them all at only 1/80 real
338 time. In another test, I created a detailed model of my own hand,
339 equipped with a realistic distribution of touch (more sensitive at
340 the fingertips), as well as eyes and ears, and it ran at around 1/4
341 real time.
343 #+BEGIN_LaTeX
344 \begin{sidewaysfigure}
345 \includegraphics[width=9.5in]{images/full-hand.png}
346 \caption{
347 I modeled my own right hand in Blender and rigged it with all the
348 senses that {\tt CORTEX} supports. My simulated hand has a
349 biologically inspired distribution of touch sensors. The senses are
350 displayed on the right, and the simulation is displayed on the
351 left. Notice that my hand is curling its fingers, that it can see
352 its own finger from the eye in its palm, and that it can feel its
353 own thumb touching its palm.}
354 \end{sidewaysfigure}
355 #+END_LaTeX
357 ** Contributions
359 - I built =CORTEX=, a comprehensive platform for embodied AI
360 experiments. =CORTEX= supports many features lacking in other
361 systems, such proper simulation of hearing. It is easy to create
362 new =CORTEX= creatures using Blender, a free 3D modeling program.
364 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
365 a worm-like creature using a computational model of empathy.
367 * Building =CORTEX=
369 I intend for =CORTEX= to be used as a general-purpose library for
370 building creatures and outfitting them with senses, so that it will
371 be useful for other researchers who want to test out ideas of their
372 own. To this end, wherver I have had to make archetictural choices
373 about =CORTEX=, I have chosen to give as much freedom to the user as
374 possible, so that =CORTEX= may be used for things I have not
375 forseen.
377 ** COMMENT Simulation or Reality?
379 The most important archetictural decision of all is the choice to
380 use a computer-simulated environemnt in the first place! The world
381 is a vast and rich place, and for now simulations are a very poor
382 reflection of its complexity. It may be that there is a significant
383 qualatative difference between dealing with senses in the real
384 world and dealing with pale facilimilies of them in a simulation.
385 What are the advantages and disadvantages of a simulation vs.
386 reality?
388 *** Simulation
390 The advantages of virtual reality are that when everything is a
391 simulation, experiments in that simulation are absolutely
392 reproducible. It's also easier to change the character and world
393 to explore new situations and different sensory combinations.
395 If the world is to be simulated on a computer, then not only do
396 you have to worry about whether the character's senses are rich
397 enough to learn from the world, but whether the world itself is
398 rendered with enough detail and realism to give enough working
399 material to the character's senses. To name just a few
400 difficulties facing modern physics simulators: destructibility of
401 the environment, simulation of water/other fluids, large areas,
402 nonrigid bodies, lots of objects, smoke. I don't know of any
403 computer simulation that would allow a character to take a rock
404 and grind it into fine dust, then use that dust to make a clay
405 sculpture, at least not without spending years calculating the
406 interactions of every single small grain of dust. Maybe a
407 simulated world with today's limitations doesn't provide enough
408 richness for real intelligence to evolve.
410 *** Reality
412 The other approach for playing with senses is to hook your
413 software up to real cameras, microphones, robots, etc., and let it
414 loose in the real world. This has the advantage of eliminating
415 concerns about simulating the world at the expense of increasing
416 the complexity of implementing the senses. Instead of just
417 grabbing the current rendered frame for processing, you have to
418 use an actual camera with real lenses and interact with photons to
419 get an image. It is much harder to change the character, which is
420 now partly a physical robot of some sort, since doing so involves
421 changing things around in the real world instead of modifying
422 lines of code. While the real world is very rich and definitely
423 provides enough stimulation for intelligence to develop as
424 evidenced by our own existence, it is also uncontrollable in the
425 sense that a particular situation cannot be recreated perfectly or
426 saved for later use. It is harder to conduct science because it is
427 harder to repeat an experiment. The worst thing about using the
428 real world instead of a simulation is the matter of time. Instead
429 of simulated time you get the constant and unstoppable flow of
430 real time. This severely limits the sorts of software you can use
431 to program the AI because all sense inputs must be handled in real
432 time. Complicated ideas may have to be implemented in hardware or
433 may simply be impossible given the current speed of our
434 processors. Contrast this with a simulation, in which the flow of
435 time in the simulated world can be slowed down to accommodate the
436 limitations of the character's programming. In terms of cost,
437 doing everything in software is far cheaper than building custom
438 real-time hardware. All you need is a laptop and some patience.
440 ** COMMENT Because of Time, simulation is perferable to reality
442 I envision =CORTEX= being used to support rapid prototyping and
443 iteration of ideas. Even if I could put together a well constructed
444 kit for creating robots, it would still not be enough because of
445 the scourge of real-time processing. Anyone who wants to test their
446 ideas in the real world must always worry about getting their
447 algorithms to run fast enough to process information in real time.
448 The need for real time processing only increases if multiple senses
449 are involved. In the extreme case, even simple algorithms will have
450 to be accelerated by ASIC chips or FPGAs, turning what would
451 otherwise be a few lines of code and a 10x speed penality into a
452 multi-month ordeal. For this reason, =CORTEX= supports
453 /time-dialiation/, which scales back the framerate of the
454 simulation in proportion to the amount of processing each frame.
455 From the perspective of the creatures inside the simulation, time
456 always appears to flow at a constant rate, regardless of how
457 complicated the envorimnent becomes or how many creatures are in
458 the simulation. The cost is that =CORTEX= can sometimes run slower
459 than real time. This can also be an advantage, however ---
460 simulations of very simple creatures in =CORTEX= generally run at
461 40x on my machine!
463 ** COMMENT What is a sense?
465 If =CORTEX= is to support a wide variety of senses, it would help
466 to have a better understanding of what a ``sense'' actually is!
467 While vision, touch, and hearing all seem like they are quite
468 different things, I was supprised to learn during the course of
469 this thesis that they (and all physical senses) can be expressed as
470 exactly the same mathematical object due to a dimensional argument!
472 Human beings are three-dimensional objects, and the nerves that
473 transmit data from our various sense organs to our brain are
474 essentially one-dimensional. This leaves up to two dimensions in
475 which our sensory information may flow. For example, imagine your
476 skin: it is a two-dimensional surface around a three-dimensional
477 object (your body). It has discrete touch sensors embedded at
478 various points, and the density of these sensors corresponds to the
479 sensitivity of that region of skin. Each touch sensor connects to a
480 nerve, all of which eventually are bundled together as they travel
481 up the spinal cord to the brain. Intersect the spinal nerves with a
482 guillotining plane and you will see all of the sensory data of the
483 skin revealed in a roughly circular two-dimensional image which is
484 the cross section of the spinal cord. Points on this image that are
485 close together in this circle represent touch sensors that are
486 /probably/ close together on the skin, although there is of course
487 some cutting and rearrangement that has to be done to transfer the
488 complicated surface of the skin onto a two dimensional image.
490 Most human senses consist of many discrete sensors of various
491 properties distributed along a surface at various densities. For
492 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
493 disks, and Ruffini's endings, which detect pressure and vibration
494 of various intensities. For ears, it is the stereocilia distributed
495 along the basilar membrane inside the cochlea; each one is
496 sensitive to a slightly different frequency of sound. For eyes, it
497 is rods and cones distributed along the surface of the retina. In
498 each case, we can describe the sense with a surface and a
499 distribution of sensors along that surface.
501 The neat idea is that every human sense can be effectively
502 described in terms of a surface containing embedded sensors. If the
503 sense had any more dimensions, then there wouldn't be enough room
504 in the spinal chord to transmit the information!
506 Therefore, =CORTEX= must support the ability to create objects and
507 then be able to ``paint'' points along their surfaces to describe
508 each sense.
510 Fortunately this idea is already a well known computer graphics
511 technique called called /UV-mapping/. The three-dimensional surface
512 of a model is cut and smooshed until it fits on a two-dimensional
513 image. You paint whatever you want on that image, and when the
514 three-dimensional shape is rendered in a game the smooshing and
515 cutting is reversed and the image appears on the three-dimensional
516 object.
518 To make a sense, interpret the UV-image as describing the
519 distribution of that senses sensors. To get different types of
520 sensors, you can either use a different color for each type of
521 sensor, or use multiple UV-maps, each labeled with that sensor
522 type. I generally use a white pixel to mean the presence of a
523 sensor and a black pixel to mean the absence of a sensor, and use
524 one UV-map for each sensor-type within a given sense.
526 #+CAPTION: The UV-map for an elongated icososphere. The white
527 #+caption: dots each represent a touch sensor. They are dense
528 #+caption: in the regions that describe the tip of the finger,
529 #+caption: and less dense along the dorsal side of the finger
530 #+caption: opposite the tip.
531 #+name: finger-UV
532 #+ATTR_latex: :width 10cm
533 [[./images/finger-UV.png]]
535 #+caption: Ventral side of the UV-mapped finger. Notice the
536 #+caption: density of touch sensors at the tip.
537 #+name: finger-side-view
538 #+ATTR_LaTeX: :width 10cm
539 [[./images/finger-1.png]]
541 ** COMMENT Video game engines provide ready-made physics and shading
543 I did not need to write my own physics simulation code or shader to
544 build =CORTEX=. Doing so would lead to a system that is impossible
545 for anyone but myself to use anyway. Instead, I use a video game
546 engine as a base and modify it to accomodate the additional needs
547 of =CORTEX=. Video game engines are an ideal starting point to
548 build =CORTEX=, because they are not far from being creature
549 building systems themselves.
551 First off, general purpose video game engines come with a physics
552 engine and lighting / sound system. The physics system provides
553 tools that can be co-opted to serve as touch, proprioception, and
554 muscles. Since some games support split screen views, a good video
555 game engine will allow you to efficiently create multiple cameras
556 in the simulated world that can be used as eyes. Video game systems
557 offer integrated asset management for things like textures and
558 creatures models, providing an avenue for defining creatures. They
559 also understand UV-mapping, since this technique is used to apply a
560 texture to a model. Finally, because video game engines support a
561 large number of users, as long as =CORTEX= doesn't stray too far
562 from the base system, other researchers can turn to this community
563 for help when doing their research.
565 ** COMMENT =CORTEX= is based on jMonkeyEngine3
567 While preparing to build =CORTEX= I studied several video game
568 engines to see which would best serve as a base. The top contenders
569 were:
571 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
572 software in 1997. All the source code was released by ID
573 software into the Public Domain several years ago, and as a
574 result it has been ported to many different languages. This
575 engine was famous for its advanced use of realistic shading
576 and had decent and fast physics simulation. The main advantage
577 of the Quake II engine is its simplicity, but I ultimately
578 rejected it because the engine is too tied to the concept of a
579 first-person shooter game. One of the problems I had was that
580 there does not seem to be any easy way to attach multiple
581 cameras to a single character. There are also several physics
582 clipping issues that are corrected in a way that only applies
583 to the main character and do not apply to arbitrary objects.
585 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
586 and Quake I engines and is used by Valve in the Half-Life
587 series of games. The physics simulation in the Source Engine
588 is quite accurate and probably the best out of all the engines
589 I investigated. There is also an extensive community actively
590 working with the engine. However, applications that use the
591 Source Engine must be written in C++, the code is not open, it
592 only runs on Windows, and the tools that come with the SDK to
593 handle models and textures are complicated and awkward to use.
595 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
596 games in Java. It uses OpenGL to render to the screen and uses
597 screengraphs to avoid drawing things that do not appear on the
598 screen. It has an active community and several games in the
599 pipeline. The engine was not built to serve any particular
600 game but is instead meant to be used for any 3D game.
602 I chose jMonkeyEngine3 because it because it had the most features
603 out of all the free projects I looked at, and because I could then
604 write my code in clojure, an implementation of =LISP= that runs on
605 the JVM.
607 ** COMMENT =CORTEX= uses Blender to create creature models
609 For the simple worm-like creatures I will use later on in this
610 thesis, I could define a simple API in =CORTEX= that would allow
611 one to create boxes, spheres, etc., and leave that API as the sole
612 way to create creatures. However, for =CORTEX= to truly be useful
613 for other projects, it needs a way to construct complicated
614 creatures. If possible, it would be nice to leverage work that has
615 already been done by the community of 3D modelers, or at least
616 enable people who are talented at moedling but not programming to
617 design =CORTEX= creatures.
619 Therefore, I use Blender, a free 3D modeling program, as the main
620 way to create creatures in =CORTEX=. However, the creatures modeled
621 in Blender must also be simple to simulate in jMonkeyEngine3's game
622 engine, and must also be easy to rig with =CORTEX='s senses. I
623 accomplish this with extensive use of Blender's ``empty nodes.''
625 Empty nodes have no mass, physical presence, or appearance, but
626 they can hold metadata and have names. I use a tree structure of
627 empty nodes to specify senses in the following manner:
629 - Create a single top-level empty node whose name is the name of
630 the sense.
631 - Add empty nodes which each contain meta-data relevant to the
632 sense, including a UV-map describing the number/distribution of
633 sensors if applicable.
634 - Make each empty-node the child of the top-level node.
636 #+caption: An example of annoting a creature model with empty
637 #+caption: nodes to describe the layout of senses. There are
638 #+caption: multiple empty nodes which each describe the position
639 #+caption: of muscles, ears, eyes, or joints.
640 #+name: sense-nodes
641 #+ATTR_LaTeX: :width 10cm
642 [[./images/empty-sense-nodes.png]]
644 ** COMMENT Bodies are composed of segments connected by joints
646 Blender is a general purpose animation tool, which has been used in
647 the past to create high quality movies such as Sintel
648 \cite{sintel}. Though Blender can model and render even complicated
649 things like water, it is crucual to keep models that are meant to
650 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
651 though jMonkeyEngine3, is a rigid-body physics system. This offers
652 a compromise between the expressiveness of a game level and the
653 speed at which it can be simulated, and it means that creatures
654 should be naturally expressed as rigid components held together by
655 joint constraints.
657 But humans are more like a squishy bag with wrapped around some
658 hard bones which define the overall shape. When we move, our skin
659 bends and stretches to accomodate the new positions of our bones.
661 One way to make bodies composed of rigid pieces connected by joints
662 /seem/ more human-like is to use an /armature/, (or /rigging/)
663 system, which defines a overall ``body mesh'' and defines how the
664 mesh deforms as a function of the position of each ``bone'' which
665 is a standard rigid body. This technique is used extensively to
666 model humans and create realistic animations. It is not a good
667 technique for physical simulation, however because it creates a lie
668 -- the skin is not a physical part of the simulation and does not
669 interact with any objects in the world or itself. Objects will pass
670 right though the skin until they come in contact with the
671 underlying bone, which is a physical object. Whithout simulating
672 the skin, the sense of touch has little meaning, and the creature's
673 own vision will lie to it about the true extent of its body.
674 Simulating the skin as a physical object requires some way to
675 continuously update the physical model of the skin along with the
676 movement of the bones, which is unacceptably slow compared to rigid
677 body simulation.
679 Therefore, instead of using the human-like ``deformable bag of
680 bones'' approach, I decided to base my body plans on multiple solid
681 objects that are connected by joints, inspired by the robot =EVE=
682 from the movie WALL-E.
684 #+caption: =EVE= from the movie WALL-E. This body plan turns
685 #+caption: out to be much better suited to my purposes than a more
686 #+caption: human-like one.
687 #+ATTR_LaTeX: :width 10cm
688 [[./images/Eve.jpg]]
690 =EVE='s body is composed of several rigid components that are held
691 together by invisible joint constraints. This is what I mean by
692 ``eve-like''. The main reason that I use eve-style bodies is for
693 efficiency, and so that there will be correspondence between the
694 AI's semses and the physical presence of its body. Each individual
695 section is simulated by a separate rigid body that corresponds
696 exactly with its visual representation and does not change.
697 Sections are connected by invisible joints that are well supported
698 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
699 can efficiently simulate hundreds of rigid bodies connected by
700 joints. Just because sections are rigid does not mean they have to
701 stay as one piece forever; they can be dynamically replaced with
702 multiple sections to simulate splitting in two. This could be used
703 to simulate retractable claws or =EVE='s hands, which are able to
704 coalesce into one object in the movie.
706 *** Solidifying/Connecting a body
708 =CORTEX= creates a creature in two steps: first, it traverses the
709 nodes in the blender file and creates physical representations for
710 any of them that have mass defined in their blender meta-data.
712 #+caption: Program for iterating through the nodes in a blender file
713 #+caption: and generating physical jMonkeyEngine3 objects with mass
714 #+caption: and a matching physics shape.
715 #+name: name
716 #+begin_listing clojure
717 #+begin_src clojure
718 (defn physical!
719 "Iterate through the nodes in creature and make them real physical
720 objects in the simulation."
721 [#^Node creature]
722 (dorun
723 (map
724 (fn [geom]
725 (let [physics-control
726 (RigidBodyControl.
727 (HullCollisionShape.
728 (.getMesh geom))
729 (if-let [mass (meta-data geom "mass")]
730 (float mass) (float 1)))]
731 (.addControl geom physics-control)))
732 (filter #(isa? (class %) Geometry )
733 (node-seq creature)))))
734 #+end_src
735 #+end_listing
737 The next step to making a proper body is to connect those pieces
738 together with joints. jMonkeyEngine has a large array of joints
739 available via =bullet=, such as Point2Point, Cone, Hinge, and a
740 generic Six Degree of Freedom joint, with or without spring
741 restitution.
743 Joints are treated a lot like proper senses, in that there is a
744 top-level empty node named ``joints'' whose children each
745 represent a joint.
747 #+caption: View of the hand model in Blender showing the main ``joints''
748 #+caption: node (highlighted in yellow) and its children which each
749 #+caption: represent a joint in the hand. Each joint node has metadata
750 #+caption: specifying what sort of joint it is.
751 #+name: blender-hand
752 #+ATTR_LaTeX: :width 10cm
753 [[./images/hand-screenshot1.png]]
756 =CORTEX='s procedure for binding the creature together with joints
757 is as follows:
759 - Find the children of the ``joints'' node.
760 - Determine the two spatials the joint is meant to connect.
761 - Create the joint based on the meta-data of the empty node.
763 The higher order function =sense-nodes= from =cortex.sense=
764 simplifies finding the joints based on their parent ``joints''
765 node.
767 #+caption: Retrieving the children empty nodes from a single
768 #+caption: named empty node is a common pattern in =CORTEX=
769 #+caption: further instances of this technique for the senses
770 #+caption: will be omitted
771 #+name: get-empty-nodes
772 #+begin_listing clojure
773 #+begin_src clojure
774 (defn sense-nodes
775 "For some senses there is a special empty blender node whose
776 children are considered markers for an instance of that sense. This
777 function generates functions to find those children, given the name
778 of the special parent node."
779 [parent-name]
780 (fn [#^Node creature]
781 (if-let [sense-node (.getChild creature parent-name)]
782 (seq (.getChildren sense-node)) [])))
784 (def
785 ^{:doc "Return the children of the creature's \"joints\" node."
786 :arglists '([creature])}
787 joints
788 (sense-nodes "joints"))
789 #+end_src
790 #+end_listing
792 To find a joint's targets, =CORTEX= creates a small cube, centered
793 around the empty-node, and grows the cube exponentially until it
794 intersects two physical objects. The objects are ordered according
795 to the joint's rotation, with the first one being the object that
796 has more negative coordinates in the joint's reference frame.
797 Since the objects must be physical, the empty-node itself escapes
798 detection. Because the objects must be physical, =joint-targets=
799 must be called /after/ =physical!= is called.
801 #+caption: Program to find the targets of a joint node by
802 #+caption: exponentiallly growth of a search cube.
803 #+name: joint-targets
804 #+begin_listing clojure
805 #+begin_src clojure
806 (defn joint-targets
807 "Return the two closest two objects to the joint object, ordered
808 from bottom to top according to the joint's rotation."
809 [#^Node parts #^Node joint]
810 (loop [radius (float 0.01)]
811 (let [results (CollisionResults.)]
812 (.collideWith
813 parts
814 (BoundingBox. (.getWorldTranslation joint)
815 radius radius radius) results)
816 (let [targets
817 (distinct
818 (map #(.getGeometry %) results))]
819 (if (>= (count targets) 2)
820 (sort-by
821 #(let [joint-ref-frame-position
822 (jme-to-blender
823 (.mult
824 (.inverse (.getWorldRotation joint))
825 (.subtract (.getWorldTranslation %)
826 (.getWorldTranslation joint))))]
827 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
828 (take 2 targets))
829 (recur (float (* radius 2))))))))
830 #+end_src
831 #+end_listing
833 Once =CORTEX= finds all joints and targets, it creates them using
834 a dispatch on the metadata of each joint node.
836 #+caption: Program to dispatch on blender metadata and create joints
837 #+caption: sutiable for physical simulation.
838 #+name: joint-dispatch
839 #+begin_listing clojure
840 #+begin_src clojure
841 (defmulti joint-dispatch
842 "Translate blender pseudo-joints into real JME joints."
843 (fn [constraints & _]
844 (:type constraints)))
846 (defmethod joint-dispatch :point
847 [constraints control-a control-b pivot-a pivot-b rotation]
848 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
849 (.setLinearLowerLimit Vector3f/ZERO)
850 (.setLinearUpperLimit Vector3f/ZERO)))
852 (defmethod joint-dispatch :hinge
853 [constraints control-a control-b pivot-a pivot-b rotation]
854 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
855 [limit-1 limit-2] (:limit constraints)
856 hinge-axis (.mult rotation (blender-to-jme axis))]
857 (doto (HingeJoint. control-a control-b pivot-a pivot-b
858 hinge-axis hinge-axis)
859 (.setLimit limit-1 limit-2))))
861 (defmethod joint-dispatch :cone
862 [constraints control-a control-b pivot-a pivot-b rotation]
863 (let [limit-xz (:limit-xz constraints)
864 limit-xy (:limit-xy constraints)
865 twist (:twist constraints)]
866 (doto (ConeJoint. control-a control-b pivot-a pivot-b
867 rotation rotation)
868 (.setLimit (float limit-xz) (float limit-xy)
869 (float twist)))))
870 #+end_src
871 #+end_listing
873 All that is left for joints it to combine the above pieces into a
874 something that can operate on the collection of nodes that a
875 blender file represents.
877 #+caption: Program to completely create a joint given information
878 #+caption: from a blender file.
879 #+name: connect
880 #+begin_listing clojure
881 #+begin_src clojure
882 (defn connect
883 "Create a joint between 'obj-a and 'obj-b at the location of
884 'joint. The type of joint is determined by the metadata on 'joint.
886 Here are some examples:
887 {:type :point}
888 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
889 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
891 {:type :cone :limit-xz 0]
892 :limit-xy 0]
893 :twist 0]} (use XZY rotation mode in blender!)"
894 [#^Node obj-a #^Node obj-b #^Node joint]
895 (let [control-a (.getControl obj-a RigidBodyControl)
896 control-b (.getControl obj-b RigidBodyControl)
897 joint-center (.getWorldTranslation joint)
898 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
899 pivot-a (world-to-local obj-a joint-center)
900 pivot-b (world-to-local obj-b joint-center)]
901 (if-let
902 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
903 ;; A side-effect of creating a joint registers
904 ;; it with both physics objects which in turn
905 ;; will register the joint with the physics system
906 ;; when the simulation is started.
907 (joint-dispatch constraints
908 control-a control-b
909 pivot-a pivot-b
910 joint-rotation))))
911 #+end_src
912 #+end_listing
914 In general, whenever =CORTEX= exposes a sense (or in this case
915 physicality), it provides a function of the type =sense!=, which
916 takes in a collection of nodes and augments it to support that
917 sense. The function returns any controlls necessary to use that
918 sense. In this case =body!= cerates a physical body and returns no
919 control functions.
921 #+caption: Program to give joints to a creature.
922 #+name: name
923 #+begin_listing clojure
924 #+begin_src clojure
925 (defn joints!
926 "Connect the solid parts of the creature with physical joints. The
927 joints are taken from the \"joints\" node in the creature."
928 [#^Node creature]
929 (dorun
930 (map
931 (fn [joint]
932 (let [[obj-a obj-b] (joint-targets creature joint)]
933 (connect obj-a obj-b joint)))
934 (joints creature))))
935 (defn body!
936 "Endow the creature with a physical body connected with joints. The
937 particulars of the joints and the masses of each body part are
938 determined in blender."
939 [#^Node creature]
940 (physical! creature)
941 (joints! creature))
942 #+end_src
943 #+end_listing
945 All of the code you have just seen amounts to only 130 lines, yet
946 because it builds on top of Blender and jMonkeyEngine3, those few
947 lines pack quite a punch!
949 The hand from figure \ref{blender-hand}, which was modeled after
950 my own right hand, can now be given joints and simulated as a
951 creature.
953 #+caption: With the ability to create physical creatures from blender,
954 #+caption: =CORTEX= gets one step closer to becomming a full creature
955 #+caption: simulation environment.
956 #+name: name
957 #+ATTR_LaTeX: :width 15cm
958 [[./images/physical-hand.png]]
960 ** COMMENT Eyes reuse standard video game components
962 Vision is one of the most important senses for humans, so I need to
963 build a simulated sense of vision for my AI. I will do this with
964 simulated eyes. Each eye can be independently moved and should see
965 its own version of the world depending on where it is.
967 Making these simulated eyes a reality is simple because
968 jMonkeyEngine already contains extensive support for multiple views
969 of the same 3D simulated world. The reason jMonkeyEngine has this
970 support is because the support is necessary to create games with
971 split-screen views. Multiple views are also used to create
972 efficient pseudo-reflections by rendering the scene from a certain
973 perspective and then projecting it back onto a surface in the 3D
974 world.
976 #+caption: jMonkeyEngine supports multiple views to enable
977 #+caption: split-screen games, like GoldenEye, which was one of
978 #+caption: the first games to use split-screen views.
979 #+name: name
980 #+ATTR_LaTeX: :width 10cm
981 [[./images/goldeneye-4-player.png]]
983 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
985 jMonkeyEngine allows you to create a =ViewPort=, which represents a
986 view of the simulated world. You can create as many of these as you
987 want. Every frame, the =RenderManager= iterates through each
988 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
989 is a =FrameBuffer= which represents the rendered image in the GPU.
991 #+caption: =ViewPorts= are cameras in the world. During each frame,
992 #+caption: the =RenderManager= records a snapshot of what each view
993 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
994 #+name: name
995 #+ATTR_LaTeX: :width 10cm
996 [[../images/diagram_rendermanager2.png]]
998 Each =ViewPort= can have any number of attached =SceneProcessor=
999 objects, which are called every time a new frame is rendered. A
1000 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1001 whatever it wants to the data. Often this consists of invoking GPU
1002 specific operations on the rendered image. The =SceneProcessor= can
1003 also copy the GPU image data to RAM and process it with the CPU.
1005 *** Appropriating Views for Vision
1007 Each eye in the simulated creature needs its own =ViewPort= so
1008 that it can see the world from its own perspective. To this
1009 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1010 any arbitrary continuation function for further processing. That
1011 continuation function may perform both CPU and GPU operations on
1012 the data. To make this easy for the continuation function, the
1013 =SceneProcessor= maintains appropriately sized buffers in RAM to
1014 hold the data. It does not do any copying from the GPU to the CPU
1015 itself because it is a slow operation.
1017 #+caption: Function to make the rendered secne in jMonkeyEngine
1018 #+caption: available for further processing.
1019 #+name: pipeline-1
1020 #+begin_listing clojure
1021 #+begin_src clojure
1022 (defn vision-pipeline
1023 "Create a SceneProcessor object which wraps a vision processing
1024 continuation function. The continuation is a function that takes
1025 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1026 each of which has already been appropriately sized."
1027 [continuation]
1028 (let [byte-buffer (atom nil)
1029 renderer (atom nil)
1030 image (atom nil)]
1031 (proxy [SceneProcessor] []
1032 (initialize
1033 [renderManager viewPort]
1034 (let [cam (.getCamera viewPort)
1035 width (.getWidth cam)
1036 height (.getHeight cam)]
1037 (reset! renderer (.getRenderer renderManager))
1038 (reset! byte-buffer
1039 (BufferUtils/createByteBuffer
1040 (* width height 4)))
1041 (reset! image (BufferedImage.
1042 width height
1043 BufferedImage/TYPE_4BYTE_ABGR))))
1044 (isInitialized [] (not (nil? @byte-buffer)))
1045 (reshape [_ _ _])
1046 (preFrame [_])
1047 (postQueue [_])
1048 (postFrame
1049 [#^FrameBuffer fb]
1050 (.clear @byte-buffer)
1051 (continuation @renderer fb @byte-buffer @image))
1052 (cleanup []))))
1053 #+end_src
1054 #+end_listing
1056 The continuation function given to =vision-pipeline= above will be
1057 given a =Renderer= and three containers for image data. The
1058 =FrameBuffer= references the GPU image data, but the pixel data
1059 can not be used directly on the CPU. The =ByteBuffer= and
1060 =BufferedImage= are initially "empty" but are sized to hold the
1061 data in the =FrameBuffer=. I call transferring the GPU image data
1062 to the CPU structures "mixing" the image data.
1064 *** Optical sensor arrays are described with images and referenced with metadata
1066 The vision pipeline described above handles the flow of rendered
1067 images. Now, =CORTEX= needs simulated eyes to serve as the source
1068 of these images.
1070 An eye is described in blender in the same way as a joint. They
1071 are zero dimensional empty objects with no geometry whose local
1072 coordinate system determines the orientation of the resulting eye.
1073 All eyes are children of a parent node named "eyes" just as all
1074 joints have a parent named "joints". An eye binds to the nearest
1075 physical object with =bind-sense=.
1077 #+caption: Here, the camera is created based on metadata on the
1078 #+caption: eye-node and attached to the nearest physical object
1079 #+caption: with =bind-sense=
1080 #+name: add-eye
1081 #+begin_listing clojure
1082 (defn add-eye!
1083 "Create a Camera centered on the current position of 'eye which
1084 follows the closest physical node in 'creature. The camera will
1085 point in the X direction and use the Z vector as up as determined
1086 by the rotation of these vectors in blender coordinate space. Use
1087 XZY rotation for the node in blender."
1088 [#^Node creature #^Spatial eye]
1089 (let [target (closest-node creature eye)
1090 [cam-width cam-height]
1091 ;;[640 480] ;; graphics card on laptop doesn't support
1092 ;; arbitray dimensions.
1093 (eye-dimensions eye)
1094 cam (Camera. cam-width cam-height)
1095 rot (.getWorldRotation eye)]
1096 (.setLocation cam (.getWorldTranslation eye))
1097 (.lookAtDirection
1098 cam ; this part is not a mistake and
1099 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1100 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1101 (.setFrustumPerspective
1102 cam (float 45)
1103 (float (/ (.getWidth cam) (.getHeight cam)))
1104 (float 1)
1105 (float 1000))
1106 (bind-sense target cam) cam))
1107 #+end_listing
1109 *** Simulated Retina
1111 An eye is a surface (the retina) which contains many discrete
1112 sensors to detect light. These sensors can have different
1113 light-sensing properties. In humans, each discrete sensor is
1114 sensitive to red, blue, green, or gray. These different types of
1115 sensors can have different spatial distributions along the retina.
1116 In humans, there is a fovea in the center of the retina which has
1117 a very high density of color sensors, and a blind spot which has
1118 no sensors at all. Sensor density decreases in proportion to
1119 distance from the fovea.
1121 I want to be able to model any retinal configuration, so my
1122 eye-nodes in blender contain metadata pointing to images that
1123 describe the precise position of the individual sensors using
1124 white pixels. The meta-data also describes the precise sensitivity
1125 to light that the sensors described in the image have. An eye can
1126 contain any number of these images. For example, the metadata for
1127 an eye might look like this:
1129 #+begin_src clojure
1130 {0xFF0000 "Models/test-creature/retina-small.png"}
1131 #+end_src
1133 #+caption: An example retinal profile image. White pixels are
1134 #+caption: photo-sensitive elements. The distribution of white
1135 #+caption: pixels is denser in the middle and falls off at the
1136 #+caption: edges and is inspired by the human retina.
1137 #+name: retina
1138 #+ATTR_LaTeX: :width 10cm
1139 [[./images/retina-small.png]]
1141 Together, the number 0xFF0000 and the image image above describe
1142 the placement of red-sensitive sensory elements.
1144 Meta-data to very crudely approximate a human eye might be
1145 something like this:
1147 #+begin_src clojure
1148 (let [retinal-profile "Models/test-creature/retina-small.png"]
1149 {0xFF0000 retinal-profile
1150 0x00FF00 retinal-profile
1151 0x0000FF retinal-profile
1152 0xFFFFFF retinal-profile})
1153 #+end_src
1155 The numbers that serve as keys in the map determine a sensor's
1156 relative sensitivity to the channels red, green, and blue. These
1157 sensitivity values are packed into an integer in the order
1158 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1159 image are added together with these sensitivities as linear
1160 weights. Therefore, 0xFF0000 means sensitive to red only while
1161 0xFFFFFF means sensitive to all colors equally (gray).
1163 #+caption: This is the core of vision in =CORTEX=. A given eye node
1164 #+caption: is converted into a function that returns visual
1165 #+caption: information from the simulation.
1166 #+name: vision-kernel
1167 #+begin_listing clojure
1168 (defn vision-kernel
1169 "Returns a list of functions, each of which will return a color
1170 channel's worth of visual information when called inside a running
1171 simulation."
1172 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1173 (let [retinal-map (retina-sensor-profile eye)
1174 camera (add-eye! creature eye)
1175 vision-image
1176 (atom
1177 (BufferedImage. (.getWidth camera)
1178 (.getHeight camera)
1179 BufferedImage/TYPE_BYTE_BINARY))
1180 register-eye!
1181 (runonce
1182 (fn [world]
1183 (add-camera!
1184 world camera
1185 (let [counter (atom 0)]
1186 (fn [r fb bb bi]
1187 (if (zero? (rem (swap! counter inc) (inc skip)))
1188 (reset! vision-image
1189 (BufferedImage! r fb bb bi))))))))]
1190 (vec
1191 (map
1192 (fn [[key image]]
1193 (let [whites (white-coordinates image)
1194 topology (vec (collapse whites))
1195 sensitivity (sensitivity-presets key key)]
1196 (attached-viewport.
1197 (fn [world]
1198 (register-eye! world)
1199 (vector
1200 topology
1201 (vec
1202 (for [[x y] whites]
1203 (pixel-sense
1204 sensitivity
1205 (.getRGB @vision-image x y))))))
1206 register-eye!)))
1207 retinal-map))))
1208 #+end_listing
1210 Note that since each of the functions generated by =vision-kernel=
1211 shares the same =register-eye!= function, the eye will be
1212 registered only once the first time any of the functions from the
1213 list returned by =vision-kernel= is called. Each of the functions
1214 returned by =vision-kernel= also allows access to the =Viewport=
1215 through which it receives images.
1217 All the hard work has been done; all that remains is to apply
1218 =vision-kernel= to each eye in the creature and gather the results
1219 into one list of functions.
1222 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1223 #+caption: environment for experimenting with different types of
1224 #+caption: eyes.
1225 #+name: vision!
1226 #+begin_listing clojure
1227 (defn vision!
1228 "Returns a list of functions, each of which returns visual sensory
1229 data when called inside a running simulation."
1230 [#^Node creature & {skip :skip :or {skip 0}}]
1231 (reduce
1232 concat
1233 (for [eye (eyes creature)]
1234 (vision-kernel creature eye))))
1235 #+end_listing
1237 #+caption: Simulated vision with a test creature and the
1238 #+caption: human-like eye approximation. Notice how each channel
1239 #+caption: of the eye responds differently to the differently
1240 #+caption: colored balls.
1241 #+name: worm-vision-test.
1242 #+ATTR_LaTeX: :width 13cm
1243 [[./images/worm-vision.png]]
1245 The vision code is not much more complicated than the body code,
1246 and enables multiple further paths for simulated vision. For
1247 example, it is quite easy to create bifocal vision -- you just
1248 make two eyes next to each other in blender! It is also possible
1249 to encode vision transforms in the retinal files. For example, the
1250 human like retina file in figure \ref{retina} approximates a
1251 log-polar transform.
1253 This vision code has already been absorbed by the jMonkeyEngine
1254 community and is now (in modified form) part of a system for
1255 capturing in-game video to a file.
1257 ** COMMENT Hearing is hard; =CORTEX= does it right
1259 At the end of this section I will have simulated ears that work the
1260 same way as the simulated eyes in the last section. I will be able to
1261 place any number of ear-nodes in a blender file, and they will bind to
1262 the closest physical object and follow it as it moves around. Each ear
1263 will provide access to the sound data it picks up between every frame.
1265 Hearing is one of the more difficult senses to simulate, because there
1266 is less support for obtaining the actual sound data that is processed
1267 by jMonkeyEngine3. There is no "split-screen" support for rendering
1268 sound from different points of view, and there is no way to directly
1269 access the rendered sound data.
1271 =CORTEX='s hearing is unique because it does not have any
1272 limitations compared to other simulation environments. As far as I
1273 know, there is no other system that supports multiple listerers,
1274 and the sound demo at the end of this section is the first time
1275 it's been done in a video game environment.
1277 *** Brief Description of jMonkeyEngine's Sound System
1279 jMonkeyEngine's sound system works as follows:
1281 - jMonkeyEngine uses the =AppSettings= for the particular
1282 application to determine what sort of =AudioRenderer= should be
1283 used.
1284 - Although some support is provided for multiple AudioRendering
1285 backends, jMonkeyEngine at the time of this writing will either
1286 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1287 - jMonkeyEngine tries to figure out what sort of system you're
1288 running and extracts the appropriate native libraries.
1289 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1290 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1291 - =OpenAL= renders the 3D sound and feeds the rendered sound
1292 directly to any of various sound output devices with which it
1293 knows how to communicate.
1295 A consequence of this is that there's no way to access the actual
1296 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1297 one /listener/ (it renders sound data from only one perspective),
1298 which normally isn't a problem for games, but becomes a problem
1299 when trying to make multiple AI creatures that can each hear the
1300 world from a different perspective.
1302 To make many AI creatures in jMonkeyEngine that can each hear the
1303 world from their own perspective, or to make a single creature with
1304 many ears, it is necessary to go all the way back to =OpenAL= and
1305 implement support for simulated hearing there.
1307 *** Extending =OpenAl=
1309 Extending =OpenAL= to support multiple listeners requires 500
1310 lines of =C= code and is too hairy to mention here. Instead, I
1311 will show a small amount of extension code and go over the high
1312 level stragety. Full source is of course available with the
1313 =CORTEX= distribution if you're interested.
1315 =OpenAL= goes to great lengths to support many different systems,
1316 all with different sound capabilities and interfaces. It
1317 accomplishes this difficult task by providing code for many
1318 different sound backends in pseudo-objects called /Devices/.
1319 There's a device for the Linux Open Sound System and the Advanced
1320 Linux Sound Architecture, there's one for Direct Sound on Windows,
1321 and there's even one for Solaris. =OpenAL= solves the problem of
1322 platform independence by providing all these Devices.
1324 Wrapper libraries such as LWJGL are free to examine the system on
1325 which they are running and then select an appropriate device for
1326 that system.
1328 There are also a few "special" devices that don't interface with
1329 any particular system. These include the Null Device, which
1330 doesn't do anything, and the Wave Device, which writes whatever
1331 sound it receives to a file, if everything has been set up
1332 correctly when configuring =OpenAL=.
1334 Actual mixing (doppler shift and distance.environment-based
1335 attenuation) of the sound data happens in the Devices, and they
1336 are the only point in the sound rendering process where this data
1337 is available.
1339 Therefore, in order to support multiple listeners, and get the
1340 sound data in a form that the AIs can use, it is necessary to
1341 create a new Device which supports this feature.
1343 Adding a device to OpenAL is rather tricky -- there are five
1344 separate files in the =OpenAL= source tree that must be modified
1345 to do so. I named my device the "Multiple Audio Send" Device, or
1346 =Send= Device for short, since it sends audio data back to the
1347 calling application like an Aux-Send cable on a mixing board.
1349 The main idea behind the Send device is to take advantage of the
1350 fact that LWJGL only manages one /context/ when using OpenAL. A
1351 /context/ is like a container that holds samples and keeps track
1352 of where the listener is. In order to support multiple listeners,
1353 the Send device identifies the LWJGL context as the master
1354 context, and creates any number of slave contexts to represent
1355 additional listeners. Every time the device renders sound, it
1356 synchronizes every source from the master LWJGL context to the
1357 slave contexts. Then, it renders each context separately, using a
1358 different listener for each one. The rendered sound is made
1359 available via JNI to jMonkeyEngine.
1361 Switching between contexts is not the normal operation of a
1362 Device, and one of the problems with doing so is that a Device
1363 normally keeps around a few pieces of state such as the
1364 =ClickRemoval= array above which will become corrupted if the
1365 contexts are not rendered in parallel. The solution is to create a
1366 copy of this normally global device state for each context, and
1367 copy it back and forth into and out of the actual device state
1368 whenever a context is rendered.
1370 The core of the =Send= device is the =syncSources= function, which
1371 does the job of copying all relevant data from one context to
1372 another.
1374 #+caption: Program for extending =OpenAL= to support multiple
1375 #+caption: listeners via context copying/switching.
1376 #+name: sync-openal-sources
1377 #+begin_listing C
1378 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1379 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1380 ALuint master = masterSource->source;
1381 ALuint slave = slaveSource->source;
1382 ALCcontext *current = alcGetCurrentContext();
1384 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1385 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1386 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1387 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1388 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1389 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1390 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1391 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1392 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1393 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1394 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1395 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1396 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1398 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1399 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1400 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1402 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1403 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1405 alcMakeContextCurrent(masterCtx);
1406 ALint source_type;
1407 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1409 // Only static sources are currently synchronized!
1410 if (AL_STATIC == source_type){
1411 ALint master_buffer;
1412 ALint slave_buffer;
1413 alGetSourcei(master, AL_BUFFER, &master_buffer);
1414 alcMakeContextCurrent(slaveCtx);
1415 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1416 if (master_buffer != slave_buffer){
1417 alSourcei(slave, AL_BUFFER, master_buffer);
1421 // Synchronize the state of the two sources.
1422 alcMakeContextCurrent(masterCtx);
1423 ALint masterState;
1424 ALint slaveState;
1426 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1427 alcMakeContextCurrent(slaveCtx);
1428 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1430 if (masterState != slaveState){
1431 switch (masterState){
1432 case AL_INITIAL : alSourceRewind(slave); break;
1433 case AL_PLAYING : alSourcePlay(slave); break;
1434 case AL_PAUSED : alSourcePause(slave); break;
1435 case AL_STOPPED : alSourceStop(slave); break;
1438 // Restore whatever context was previously active.
1439 alcMakeContextCurrent(current);
1441 #+end_listing
1443 With this special context-switching device, and some ugly JNI
1444 bindings that are not worth mentioning, =CORTEX= gains the ability
1445 to access multiple sound streams from =OpenAL=.
1447 #+caption: Program to create an ear from a blender empty node. The ear
1448 #+caption: follows around the nearest physical object and passes
1449 #+caption: all sensory data to a continuation function.
1450 #+name: add-ear
1451 #+begin_listing clojure
1452 (defn add-ear!
1453 "Create a Listener centered on the current position of 'ear
1454 which follows the closest physical node in 'creature and
1455 sends sound data to 'continuation."
1456 [#^Application world #^Node creature #^Spatial ear continuation]
1457 (let [target (closest-node creature ear)
1458 lis (Listener.)
1459 audio-renderer (.getAudioRenderer world)
1460 sp (hearing-pipeline continuation)]
1461 (.setLocation lis (.getWorldTranslation ear))
1462 (.setRotation lis (.getWorldRotation ear))
1463 (bind-sense target lis)
1464 (update-listener-velocity! target lis)
1465 (.addListener audio-renderer lis)
1466 (.registerSoundProcessor audio-renderer lis sp)))
1467 #+end_listing
1470 The =Send= device, unlike most of the other devices in =OpenAL=,
1471 does not render sound unless asked. This enables the system to
1472 slow down or speed up depending on the needs of the AIs who are
1473 using it to listen. If the device tried to render samples in
1474 real-time, a complicated AI whose mind takes 100 seconds of
1475 computer time to simulate 1 second of AI-time would miss almost
1476 all of the sound in its environment!
1478 #+caption: Program to enable arbitrary hearing in =CORTEX=
1479 #+name: hearing
1480 #+begin_listing clojure
1481 (defn hearing-kernel
1482 "Returns a function which returns auditory sensory data when called
1483 inside a running simulation."
1484 [#^Node creature #^Spatial ear]
1485 (let [hearing-data (atom [])
1486 register-listener!
1487 (runonce
1488 (fn [#^Application world]
1489 (add-ear!
1490 world creature ear
1491 (comp #(reset! hearing-data %)
1492 byteBuffer->pulse-vector))))]
1493 (fn [#^Application world]
1494 (register-listener! world)
1495 (let [data @hearing-data
1496 topology
1497 (vec (map #(vector % 0) (range 0 (count data))))]
1498 [topology data]))))
1500 (defn hearing!
1501 "Endow the creature in a particular world with the sense of
1502 hearing. Will return a sequence of functions, one for each ear,
1503 which when called will return the auditory data from that ear."
1504 [#^Node creature]
1505 (for [ear (ears creature)]
1506 (hearing-kernel creature ear)))
1507 #+end_listing
1509 Armed with these functions, =CORTEX= is able to test possibly the
1510 first ever instance of multiple listeners in a video game engine
1511 based simulation!
1513 #+caption: Here a simple creature responds to sound by changing
1514 #+caption: its color from gray to green when the total volume
1515 #+caption: goes over a threshold.
1516 #+name: sound-test
1517 #+begin_listing java
1518 /**
1519 * Respond to sound! This is the brain of an AI entity that
1520 * hears its surroundings and reacts to them.
1521 */
1522 public void process(ByteBuffer audioSamples,
1523 int numSamples, AudioFormat format) {
1524 audioSamples.clear();
1525 byte[] data = new byte[numSamples];
1526 float[] out = new float[numSamples];
1527 audioSamples.get(data);
1528 FloatSampleTools.
1529 byte2floatInterleaved
1530 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1532 float max = Float.NEGATIVE_INFINITY;
1533 for (float f : out){if (f > max) max = f;}
1534 audioSamples.clear();
1536 if (max > 0.1){
1537 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1539 else {
1540 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1542 #+end_listing
1544 #+caption: First ever simulation of multiple listerners in =CORTEX=.
1545 #+caption: Each cube is a creature which processes sound data with
1546 #+caption: the =process= function from listing \ref{sound-test}.
1547 #+caption: the ball is constantally emiting a pure tone of
1548 #+caption: constant volume. As it approaches the cubes, they each
1549 #+caption: change color in response to the sound.
1550 #+name: sound-cubes.
1551 #+ATTR_LaTeX: :width 10cm
1552 [[./images/aurellem-gray.png]]
1554 This system of hearing has also been co-opted by the
1555 jMonkeyEngine3 community and is used to record audio for demo
1556 videos.
1558 ** COMMENT Touch uses hundreds of hair-like elements
1560 Touch is critical to navigation and spatial reasoning and as such I
1561 need a simulated version of it to give to my AI creatures.
1563 Human skin has a wide array of touch sensors, each of which
1564 specialize in detecting different vibrational modes and pressures.
1565 These sensors can integrate a vast expanse of skin (i.e. your
1566 entire palm), or a tiny patch of skin at the tip of your finger.
1567 The hairs of the skin help detect objects before they even come
1568 into contact with the skin proper.
1570 However, touch in my simulated world can not exactly correspond to
1571 human touch because my creatures are made out of completely rigid
1572 segments that don't deform like human skin.
1574 Instead of measuring deformation or vibration, I surround each
1575 rigid part with a plenitude of hair-like objects (/feelers/) which
1576 do not interact with the physical world. Physical objects can pass
1577 through them with no effect. The feelers are able to tell when
1578 other objects pass through them, and they constantly report how
1579 much of their extent is covered. So even though the creature's body
1580 parts do not deform, the feelers create a margin around those body
1581 parts which achieves a sense of touch which is a hybrid between a
1582 human's sense of deformation and sense from hairs.
1584 Implementing touch in jMonkeyEngine follows a different technical
1585 route than vision and hearing. Those two senses piggybacked off
1586 jMonkeyEngine's 3D audio and video rendering subsystems. To
1587 simulate touch, I use jMonkeyEngine's physics system to execute
1588 many small collision detections, one for each feeler. The placement
1589 of the feelers is determined by a UV-mapped image which shows where
1590 each feeler should be on the 3D surface of the body.
1592 *** Defining Touch Meta-Data in Blender
1594 Each geometry can have a single UV map which describes the
1595 position of the feelers which will constitute its sense of touch.
1596 This image path is stored under the ``touch'' key. The image itself
1597 is black and white, with black meaning a feeler length of 0 (no
1598 feeler is present) and white meaning a feeler length of =scale=,
1599 which is a float stored under the key "scale".
1601 #+caption: Touch does not use empty nodes, to store metadata,
1602 #+caption: because the metadata of each solid part of a
1603 #+caption: creature's body is sufficient.
1604 #+name: touch-meta-data
1605 #+begin_listing clojure
1606 #+BEGIN_SRC clojure
1607 (defn tactile-sensor-profile
1608 "Return the touch-sensor distribution image in BufferedImage format,
1609 or nil if it does not exist."
1610 [#^Geometry obj]
1611 (if-let [image-path (meta-data obj "touch")]
1612 (load-image image-path)))
1614 (defn tactile-scale
1615 "Return the length of each feeler. Default scale is 0.01
1616 jMonkeyEngine units."
1617 [#^Geometry obj]
1618 (if-let [scale (meta-data obj "scale")]
1619 scale 0.1))
1620 #+END_SRC
1621 #+end_listing
1623 Here is an example of a UV-map which specifies the position of
1624 touch sensors along the surface of the upper segment of a fingertip.
1626 #+caption: This is the tactile-sensor-profile for the upper segment
1627 #+caption: of a fingertip. It defines regions of high touch sensitivity
1628 #+caption: (where there are many white pixels) and regions of low
1629 #+caption: sensitivity (where white pixels are sparse).
1630 #+name: fingertip-UV
1631 #+ATTR_LaTeX: :width 13cm
1632 [[./images/finger-UV.png]]
1634 *** Implementation Summary
1636 To simulate touch there are three conceptual steps. For each solid
1637 object in the creature, you first have to get UV image and scale
1638 parameter which define the position and length of the feelers.
1639 Then, you use the triangles which comprise the mesh and the UV
1640 data stored in the mesh to determine the world-space position and
1641 orientation of each feeler. Then once every frame, update these
1642 positions and orientations to match the current position and
1643 orientation of the object, and use physics collision detection to
1644 gather tactile data.
1646 Extracting the meta-data has already been described. The third
1647 step, physics collision detection, is handled in =touch-kernel=.
1648 Translating the positions and orientations of the feelers from the
1649 UV-map to world-space is itself a three-step process.
1651 - Find the triangles which make up the mesh in pixel-space and in
1652 world-space. (=triangles= =pixel-triangles=).
1654 - Find the coordinates of each feeler in world-space. These are
1655 the origins of the feelers. (=feeler-origins=).
1657 - Calculate the normals of the triangles in world space, and add
1658 them to each of the origins of the feelers. These are the
1659 normalized coordinates of the tips of the feelers.
1660 (=feeler-tips=).
1662 *** Triangle Math
1664 The rigid objects which make up a creature have an underlying
1665 =Geometry=, which is a =Mesh= plus a =Material= and other
1666 important data involved with displaying the object.
1668 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1669 vertices which have coordinates in world space and UV space.
1671 Here, =triangles= gets all the world-space triangles which
1672 comprise a mesh, while =pixel-triangles= gets those same triangles
1673 expressed in pixel coordinates (which are UV coordinates scaled to
1674 fit the height and width of the UV image).
1676 #+caption: Programs to extract triangles from a geometry and get
1677 #+caption: their verticies in both world and UV-coordinates.
1678 #+name: get-triangles
1679 #+begin_listing clojure
1680 #+BEGIN_SRC clojure
1681 (defn triangle
1682 "Get the triangle specified by triangle-index from the mesh."
1683 [#^Geometry geo triangle-index]
1684 (triangle-seq
1685 (let [scratch (Triangle.)]
1686 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1688 (defn triangles
1689 "Return a sequence of all the Triangles which comprise a given
1690 Geometry."
1691 [#^Geometry geo]
1692 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1694 (defn triangle-vertex-indices
1695 "Get the triangle vertex indices of a given triangle from a given
1696 mesh."
1697 [#^Mesh mesh triangle-index]
1698 (let [indices (int-array 3)]
1699 (.getTriangle mesh triangle-index indices)
1700 (vec indices)))
1702 (defn vertex-UV-coord
1703 "Get the UV-coordinates of the vertex named by vertex-index"
1704 [#^Mesh mesh vertex-index]
1705 (let [UV-buffer
1706 (.getData
1707 (.getBuffer
1708 mesh
1709 VertexBuffer$Type/TexCoord))]
1710 [(.get UV-buffer (* vertex-index 2))
1711 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1713 (defn pixel-triangle [#^Geometry geo image index]
1714 (let [mesh (.getMesh geo)
1715 width (.getWidth image)
1716 height (.getHeight image)]
1717 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1718 (map (partial vertex-UV-coord mesh)
1719 (triangle-vertex-indices mesh index))))))
1721 (defn pixel-triangles
1722 "The pixel-space triangles of the Geometry, in the same order as
1723 (triangles geo)"
1724 [#^Geometry geo image]
1725 (let [height (.getHeight image)
1726 width (.getWidth image)]
1727 (map (partial pixel-triangle geo image)
1728 (range (.getTriangleCount (.getMesh geo))))))
1729 #+END_SRC
1730 #+end_listing
1732 *** The Affine Transform from one Triangle to Another
1734 =pixel-triangles= gives us the mesh triangles expressed in pixel
1735 coordinates and =triangles= gives us the mesh triangles expressed
1736 in world coordinates. The tactile-sensor-profile gives the
1737 position of each feeler in pixel-space. In order to convert
1738 pixel-space coordinates into world-space coordinates we need
1739 something that takes coordinates on the surface of one triangle
1740 and gives the corresponding coordinates on the surface of another
1741 triangle.
1743 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1744 into any other by a combination of translation, scaling, and
1745 rotation. The affine transformation from one triangle to another
1746 is readily computable if the triangle is expressed in terms of a
1747 $4x4$ matrix.
1749 #+BEGIN_LaTeX
1750 $$
1751 \begin{bmatrix}
1752 x_1 & x_2 & x_3 & n_x \\
1753 y_1 & y_2 & y_3 & n_y \\
1754 z_1 & z_2 & z_3 & n_z \\
1755 1 & 1 & 1 & 1
1756 \end{bmatrix}
1757 $$
1758 #+END_LaTeX
1760 Here, the first three columns of the matrix are the vertices of
1761 the triangle. The last column is the right-handed unit normal of
1762 the triangle.
1764 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1765 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1766 is $T_{2}T_{1}^{-1}$.
1768 The clojure code below recapitulates the formulas above, using
1769 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1770 transformation.
1772 #+caption: Program to interpert triangles as affine transforms.
1773 #+name: triangle-affine
1774 #+begin_listing clojure
1775 #+BEGIN_SRC clojure
1776 (defn triangle->matrix4f
1777 "Converts the triangle into a 4x4 matrix: The first three columns
1778 contain the vertices of the triangle; the last contains the unit
1779 normal of the triangle. The bottom row is filled with 1s."
1780 [#^Triangle t]
1781 (let [mat (Matrix4f.)
1782 [vert-1 vert-2 vert-3]
1783 (mapv #(.get t %) (range 3))
1784 unit-normal (do (.calculateNormal t)(.getNormal t))
1785 vertices [vert-1 vert-2 vert-3 unit-normal]]
1786 (dorun
1787 (for [row (range 4) col (range 3)]
1788 (do
1789 (.set mat col row (.get (vertices row) col))
1790 (.set mat 3 row 1)))) mat))
1792 (defn triangles->affine-transform
1793 "Returns the affine transformation that converts each vertex in the
1794 first triangle into the corresponding vertex in the second
1795 triangle."
1796 [#^Triangle tri-1 #^Triangle tri-2]
1797 (.mult
1798 (triangle->matrix4f tri-2)
1799 (.invert (triangle->matrix4f tri-1))))
1800 #+END_SRC
1801 #+end_listing
1803 *** Triangle Boundaries
1805 For efficiency's sake I will divide the tactile-profile image into
1806 small squares which inscribe each pixel-triangle, then extract the
1807 points which lie inside the triangle and map them to 3D-space using
1808 =triangle-transform= above. To do this I need a function,
1809 =convex-bounds= which finds the smallest box which inscribes a 2D
1810 triangle.
1812 =inside-triangle?= determines whether a point is inside a triangle
1813 in 2D pixel-space.
1815 #+caption: Program to efficiently determine point includion
1816 #+caption: in a triangle.
1817 #+name: in-triangle
1818 #+begin_listing clojure
1819 #+BEGIN_SRC clojure
1820 (defn convex-bounds
1821 "Returns the smallest square containing the given vertices, as a
1822 vector of integers [left top width height]."
1823 [verts]
1824 (let [xs (map first verts)
1825 ys (map second verts)
1826 x0 (Math/floor (apply min xs))
1827 y0 (Math/floor (apply min ys))
1828 x1 (Math/ceil (apply max xs))
1829 y1 (Math/ceil (apply max ys))]
1830 [x0 y0 (- x1 x0) (- y1 y0)]))
1832 (defn same-side?
1833 "Given the points p1 and p2 and the reference point ref, is point p
1834 on the same side of the line that goes through p1 and p2 as ref is?"
1835 [p1 p2 ref p]
1836 (<=
1838 (.dot
1839 (.cross (.subtract p2 p1) (.subtract p p1))
1840 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1842 (defn inside-triangle?
1843 "Is the point inside the triangle?"
1844 {:author "Dylan Holmes"}
1845 [#^Triangle tri #^Vector3f p]
1846 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1847 (and
1848 (same-side? vert-1 vert-2 vert-3 p)
1849 (same-side? vert-2 vert-3 vert-1 p)
1850 (same-side? vert-3 vert-1 vert-2 p))))
1851 #+END_SRC
1852 #+end_listing
1854 *** Feeler Coordinates
1856 The triangle-related functions above make short work of
1857 calculating the positions and orientations of each feeler in
1858 world-space.
1860 #+caption: Program to get the coordinates of ``feelers '' in
1861 #+caption: both world and UV-coordinates.
1862 #+name: feeler-coordinates
1863 #+begin_listing clojure
1864 #+BEGIN_SRC clojure
1865 (defn feeler-pixel-coords
1866 "Returns the coordinates of the feelers in pixel space in lists, one
1867 list for each triangle, ordered in the same way as (triangles) and
1868 (pixel-triangles)."
1869 [#^Geometry geo image]
1870 (map
1871 (fn [pixel-triangle]
1872 (filter
1873 (fn [coord]
1874 (inside-triangle? (->triangle pixel-triangle)
1875 (->vector3f coord)))
1876 (white-coordinates image (convex-bounds pixel-triangle))))
1877 (pixel-triangles geo image)))
1879 (defn feeler-world-coords
1880 "Returns the coordinates of the feelers in world space in lists, one
1881 list for each triangle, ordered in the same way as (triangles) and
1882 (pixel-triangles)."
1883 [#^Geometry geo image]
1884 (let [transforms
1885 (map #(triangles->affine-transform
1886 (->triangle %1) (->triangle %2))
1887 (pixel-triangles geo image)
1888 (triangles geo))]
1889 (map (fn [transform coords]
1890 (map #(.mult transform (->vector3f %)) coords))
1891 transforms (feeler-pixel-coords geo image))))
1892 #+END_SRC
1893 #+end_listing
1895 #+caption: Program to get the position of the base and tip of
1896 #+caption: each ``feeler''
1897 #+name: feeler-tips
1898 #+begin_listing clojure
1899 #+BEGIN_SRC clojure
1900 (defn feeler-origins
1901 "The world space coordinates of the root of each feeler."
1902 [#^Geometry geo image]
1903 (reduce concat (feeler-world-coords geo image)))
1905 (defn feeler-tips
1906 "The world space coordinates of the tip of each feeler."
1907 [#^Geometry geo image]
1908 (let [world-coords (feeler-world-coords geo image)
1909 normals
1910 (map
1911 (fn [triangle]
1912 (.calculateNormal triangle)
1913 (.clone (.getNormal triangle)))
1914 (map ->triangle (triangles geo)))]
1916 (mapcat (fn [origins normal]
1917 (map #(.add % normal) origins))
1918 world-coords normals)))
1920 (defn touch-topology
1921 [#^Geometry geo image]
1922 (collapse (reduce concat (feeler-pixel-coords geo image))))
1923 #+END_SRC
1924 #+end_listing
1926 *** Simulated Touch
1928 Now that the functions to construct feelers are complete,
1929 =touch-kernel= generates functions to be called from within a
1930 simulation that perform the necessary physics collisions to
1931 collect tactile data, and =touch!= recursively applies it to every
1932 node in the creature.
1934 #+caption: Efficient program to transform a ray from
1935 #+caption: one position to another.
1936 #+name: set-ray
1937 #+begin_listing clojure
1938 #+BEGIN_SRC clojure
1939 (defn set-ray [#^Ray ray #^Matrix4f transform
1940 #^Vector3f origin #^Vector3f tip]
1941 ;; Doing everything locally reduces garbage collection by enough to
1942 ;; be worth it.
1943 (.mult transform origin (.getOrigin ray))
1944 (.mult transform tip (.getDirection ray))
1945 (.subtractLocal (.getDirection ray) (.getOrigin ray))
1946 (.normalizeLocal (.getDirection ray)))
1947 #+END_SRC
1948 #+end_listing
1950 #+caption: This is the core of touch in =CORTEX= each feeler
1951 #+caption: follows the object it is bound to, reporting any
1952 #+caption: collisions that may happen.
1953 #+name: touch-kernel
1954 #+begin_listing clojure
1955 #+BEGIN_SRC clojure
1956 (defn touch-kernel
1957 "Constructs a function which will return tactile sensory data from
1958 'geo when called from inside a running simulation"
1959 [#^Geometry geo]
1960 (if-let
1961 [profile (tactile-sensor-profile geo)]
1962 (let [ray-reference-origins (feeler-origins geo profile)
1963 ray-reference-tips (feeler-tips geo profile)
1964 ray-length (tactile-scale geo)
1965 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
1966 topology (touch-topology geo profile)
1967 correction (float (* ray-length -0.2))]
1968 ;; slight tolerance for very close collisions.
1969 (dorun
1970 (map (fn [origin tip]
1971 (.addLocal origin (.mult (.subtract tip origin)
1972 correction)))
1973 ray-reference-origins ray-reference-tips))
1974 (dorun (map #(.setLimit % ray-length) current-rays))
1975 (fn [node]
1976 (let [transform (.getWorldMatrix geo)]
1977 (dorun
1978 (map (fn [ray ref-origin ref-tip]
1979 (set-ray ray transform ref-origin ref-tip))
1980 current-rays ray-reference-origins
1981 ray-reference-tips))
1982 (vector
1983 topology
1984 (vec
1985 (for [ray current-rays]
1986 (do
1987 (let [results (CollisionResults.)]
1988 (.collideWith node ray results)
1989 (let [touch-objects
1990 (filter #(not (= geo (.getGeometry %)))
1991 results)
1992 limit (.getLimit ray)]
1993 [(if (empty? touch-objects)
1994 limit
1995 (let [response
1996 (apply min (map #(.getDistance %)
1997 touch-objects))]
1998 (FastMath/clamp
1999 (float
2000 (if (> response limit) (float 0.0)
2001 (+ response correction)))
2002 (float 0.0)
2003 limit)))
2004 limit])))))))))))
2005 #+END_SRC
2006 #+end_listing
2008 Armed with the =touch!= function, =CORTEX= becomes capable of
2009 giving creatures a sense of touch. A simple test is to create a
2010 cube that is outfitted with a uniform distrubition of touch
2011 sensors. It can feel the ground and any balls that it touches.
2013 #+caption: =CORTEX= interface for creating touch in a simulated
2014 #+caption: creature.
2015 #+name: touch
2016 #+begin_listing clojure
2017 #+BEGIN_SRC clojure
2018 (defn touch!
2019 "Endow the creature with the sense of touch. Returns a sequence of
2020 functions, one for each body part with a tactile-sensor-profile,
2021 each of which when called returns sensory data for that body part."
2022 [#^Node creature]
2023 (filter
2024 (comp not nil?)
2025 (map touch-kernel
2026 (filter #(isa? (class %) Geometry)
2027 (node-seq creature)))))
2028 #+END_SRC
2029 #+end_listing
2031 The tactile-sensor-profile image for the touch cube is a simple
2032 cross with a unifom distribution of touch sensors:
2034 #+caption: The touch profile for the touch-cube. Each pure white
2035 #+caption: pixel defines a touch sensitive feeler.
2036 #+name: touch-cube-uv-map
2037 #+ATTR_LaTeX: :width 7cm
2038 [[./images/touch-profile.png]]
2040 #+caption: The touch cube reacts to canonballs. The black, red,
2041 #+caption: and white cross on the right is a visual display of
2042 #+caption: the creature's touch. White means that it is feeling
2043 #+caption: something strongly, black is not feeling anything,
2044 #+caption: and gray is in-between. The cube can feel both the
2045 #+caption: floor and the ball. Notice that when the ball causes
2046 #+caption: the cube to tip, that the bottom face can still feel
2047 #+caption: part of the ground.
2048 #+name: touch-cube-uv-map
2049 #+ATTR_LaTeX: :width 15cm
2050 [[./images/touch-cube.png]]
2052 ** COMMENT Proprioception is the sense that makes everything ``real''
2054 Close your eyes, and touch your nose with your right index finger.
2055 How did you do it? You could not see your hand, and neither your
2056 hand nor your nose could use the sense of touch to guide the path
2057 of your hand. There are no sound cues, and Taste and Smell
2058 certainly don't provide any help. You know where your hand is
2059 without your other senses because of Proprioception.
2061 Humans can sometimes loose this sense through viral infections or
2062 damage to the spinal cord or brain, and when they do, they loose
2063 the ability to control their own bodies without looking directly at
2064 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2065 Hat]], a woman named Christina looses this sense and has to learn how
2066 to move by carefully watching her arms and legs. She describes
2067 proprioception as the "eyes of the body, the way the body sees
2068 itself".
2070 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2071 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2072 positions of each body part by monitoring muscle strain and length.
2074 It's clear that this is a vital sense for fluid, graceful movement.
2075 It's also particularly easy to implement in jMonkeyEngine.
2077 My simulated proprioception calculates the relative angles of each
2078 joint from the rest position defined in the blender file. This
2079 simulates the muscle-spindles and joint capsules. I will deal with
2080 Golgi tendon organs, which calculate muscle strain, in the next
2081 section.
2083 *** Helper functions
2085 =absolute-angle= calculates the angle between two vectors,
2086 relative to a third axis vector. This angle is the number of
2087 radians you have to move counterclockwise around the axis vector
2088 to get from the first to the second vector. It is not commutative
2089 like a normal dot-product angle is.
2091 The purpose of these functions is to build a system of angle
2092 measurement that is biologically plausable.
2094 #+caption: Program to measure angles along a vector
2095 #+name: helpers
2096 #+begin_listing clojure
2097 #+BEGIN_SRC clojure
2098 (defn right-handed?
2099 "true iff the three vectors form a right handed coordinate
2100 system. The three vectors do not have to be normalized or
2101 orthogonal."
2102 [vec1 vec2 vec3]
2103 (pos? (.dot (.cross vec1 vec2) vec3)))
2105 (defn absolute-angle
2106 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2107 [0 (* 2 Math/PI)]."
2108 [vec1 vec2 axis]
2109 (let [angle (.angleBetween vec1 vec2)]
2110 (if (right-handed? vec1 vec2 axis)
2111 angle (- (* 2 Math/PI) angle))))
2112 #+END_SRC
2113 #+end_listing
2115 *** Proprioception Kernel
2117 Given a joint, =proprioception-kernel= produces a function that
2118 calculates the Euler angles between the the objects the joint
2119 connects. The only tricky part here is making the angles relative
2120 to the joint's initial ``straightness''.
2122 #+caption: Program to return biologially reasonable proprioceptive
2123 #+caption: data for each joint.
2124 #+name: proprioception
2125 #+begin_listing clojure
2126 #+BEGIN_SRC clojure
2127 (defn proprioception-kernel
2128 "Returns a function which returns proprioceptive sensory data when
2129 called inside a running simulation."
2130 [#^Node parts #^Node joint]
2131 (let [[obj-a obj-b] (joint-targets parts joint)
2132 joint-rot (.getWorldRotation joint)
2133 x0 (.mult joint-rot Vector3f/UNIT_X)
2134 y0 (.mult joint-rot Vector3f/UNIT_Y)
2135 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2136 (fn []
2137 (let [rot-a (.clone (.getWorldRotation obj-a))
2138 rot-b (.clone (.getWorldRotation obj-b))
2139 x (.mult rot-a x0)
2140 y (.mult rot-a y0)
2141 z (.mult rot-a z0)
2143 X (.mult rot-b x0)
2144 Y (.mult rot-b y0)
2145 Z (.mult rot-b z0)
2146 heading (Math/atan2 (.dot X z) (.dot X x))
2147 pitch (Math/atan2 (.dot X y) (.dot X x))
2149 ;; rotate x-vector back to origin
2150 reverse
2151 (doto (Quaternion.)
2152 (.fromAngleAxis
2153 (.angleBetween X x)
2154 (let [cross (.normalize (.cross X x))]
2155 (if (= 0 (.length cross)) y cross))))
2156 roll (absolute-angle (.mult reverse Y) y x)]
2157 [heading pitch roll]))))
2159 (defn proprioception!
2160 "Endow the creature with the sense of proprioception. Returns a
2161 sequence of functions, one for each child of the \"joints\" node in
2162 the creature, which each report proprioceptive information about
2163 that joint."
2164 [#^Node creature]
2165 ;; extract the body's joints
2166 (let [senses (map (partial proprioception-kernel creature)
2167 (joints creature))]
2168 (fn []
2169 (map #(%) senses))))
2170 #+END_SRC
2171 #+end_listing
2173 =proprioception!= maps =proprioception-kernel= across all the
2174 joints of the creature. It uses the same list of joints that
2175 =joints= uses. Proprioception is the easiest sense to implement in
2176 =CORTEX=, and it will play a crucial role when efficiently
2177 implementing empathy.
2179 #+caption: In the upper right corner, the three proprioceptive
2180 #+caption: angle measurements are displayed. Red is yaw, Green is
2181 #+caption: pitch, and White is roll.
2182 #+name: proprio
2183 #+ATTR_LaTeX: :width 11cm
2184 [[./images/proprio.png]]
2186 ** COMMENT Muscles are both effectors and sensors
2188 Surprisingly enough, terrestrial creatures only move by using
2189 torque applied about their joints. There's not a single straight
2190 line of force in the human body at all! (A straight line of force
2191 would correspond to some sort of jet or rocket propulsion.)
2193 In humans, muscles are composed of muscle fibers which can contract
2194 to exert force. The muscle fibers which compose a muscle are
2195 partitioned into discrete groups which are each controlled by a
2196 single alpha motor neuron. A single alpha motor neuron might
2197 control as little as three or as many as one thousand muscle
2198 fibers. When the alpha motor neuron is engaged by the spinal cord,
2199 it activates all of the muscle fibers to which it is attached. The
2200 spinal cord generally engages the alpha motor neurons which control
2201 few muscle fibers before the motor neurons which control many
2202 muscle fibers. This recruitment strategy allows for precise
2203 movements at low strength. The collection of all motor neurons that
2204 control a muscle is called the motor pool. The brain essentially
2205 says "activate 30% of the motor pool" and the spinal cord recruits
2206 motor neurons until 30% are activated. Since the distribution of
2207 power among motor neurons is unequal and recruitment goes from
2208 weakest to strongest, the first 30% of the motor pool might be 5%
2209 of the strength of the muscle.
2211 My simulated muscles follow a similar design: Each muscle is
2212 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2213 the array represents a motor neuron which controls a number of
2214 muscle fibers equal to the value of the entry. Each muscle has a
2215 scalar strength factor which determines the total force the muscle
2216 can exert when all motor neurons are activated. The effector
2217 function for a muscle takes a number to index into the motor pool,
2218 and then "activates" all the motor neurons whose index is lower or
2219 equal to the number. Each motor-neuron will apply force in
2220 proportion to its value in the array. Lower values cause less
2221 force. The lower values can be put at the "beginning" of the 1-D
2222 array to simulate the layout of actual human muscles, which are
2223 capable of more precise movements when exerting less force. Or, the
2224 motor pool can simulate more exotic recruitment strategies which do
2225 not correspond to human muscles.
2227 This 1D array is defined in an image file for ease of
2228 creation/visualization. Here is an example muscle profile image.
2230 #+caption: A muscle profile image that describes the strengths
2231 #+caption: of each motor neuron in a muscle. White is weakest
2232 #+caption: and dark red is strongest. This particular pattern
2233 #+caption: has weaker motor neurons at the beginning, just
2234 #+caption: like human muscle.
2235 #+name: muscle-recruit
2236 #+ATTR_LaTeX: :width 7cm
2237 [[./images/basic-muscle.png]]
2239 *** Muscle meta-data
2241 #+caption: Program to deal with loading muscle data from a blender
2242 #+caption: file's metadata.
2243 #+name: motor-pool
2244 #+begin_listing clojure
2245 #+BEGIN_SRC clojure
2246 (defn muscle-profile-image
2247 "Get the muscle-profile image from the node's blender meta-data."
2248 [#^Node muscle]
2249 (if-let [image (meta-data muscle "muscle")]
2250 (load-image image)))
2252 (defn muscle-strength
2253 "Return the strength of this muscle, or 1 if it is not defined."
2254 [#^Node muscle]
2255 (if-let [strength (meta-data muscle "strength")]
2256 strength 1))
2258 (defn motor-pool
2259 "Return a vector where each entry is the strength of the \"motor
2260 neuron\" at that part in the muscle."
2261 [#^Node muscle]
2262 (let [profile (muscle-profile-image muscle)]
2263 (vec
2264 (let [width (.getWidth profile)]
2265 (for [x (range width)]
2266 (- 255
2267 (bit-and
2268 0x0000FF
2269 (.getRGB profile x 0))))))))
2270 #+END_SRC
2271 #+end_listing
2273 Of note here is =motor-pool= which interprets the muscle-profile
2274 image in a way that allows me to use gradients between white and
2275 red, instead of shades of gray as I've been using for all the
2276 other senses. This is purely an aesthetic touch.
2278 *** Creating muscles
2280 #+caption: This is the core movement functoion in =CORTEX=, which
2281 #+caption: implements muscles that report on their activation.
2282 #+name: muscle-kernel
2283 #+begin_listing clojure
2284 #+BEGIN_SRC clojure
2285 (defn movement-kernel
2286 "Returns a function which when called with a integer value inside a
2287 running simulation will cause movement in the creature according
2288 to the muscle's position and strength profile. Each function
2289 returns the amount of force applied / max force."
2290 [#^Node creature #^Node muscle]
2291 (let [target (closest-node creature muscle)
2292 axis
2293 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2294 strength (muscle-strength muscle)
2296 pool (motor-pool muscle)
2297 pool-integral (reductions + pool)
2298 forces
2299 (vec (map #(float (* strength (/ % (last pool-integral))))
2300 pool-integral))
2301 control (.getControl target RigidBodyControl)]
2302 ;;(println-repl (.getName target) axis)
2303 (fn [n]
2304 (let [pool-index (max 0 (min n (dec (count pool))))
2305 force (forces pool-index)]
2306 (.applyTorque control (.mult axis force))
2307 (float (/ force strength))))))
2309 (defn movement!
2310 "Endow the creature with the power of movement. Returns a sequence
2311 of functions, each of which accept an integer value and will
2312 activate their corresponding muscle."
2313 [#^Node creature]
2314 (for [muscle (muscles creature)]
2315 (movement-kernel creature muscle)))
2316 #+END_SRC
2317 #+end_listing
2320 =movement-kernel= creates a function that will move the nearest
2321 physical object to the muscle node. The muscle exerts a rotational
2322 force dependent on it's orientation to the object in the blender
2323 file. The function returned by =movement-kernel= is also a sense
2324 function: it returns the percent of the total muscle strength that
2325 is currently being employed. This is analogous to muscle tension
2326 in humans and completes the sense of proprioception begun in the
2327 last section.
2329 ** COMMENT =CORTEX= brings complex creatures to life!
2331 The ultimate test of =CORTEX= is to create a creature with the full
2332 gamut of senses and put it though its paces.
2334 With all senses enabled, my right hand model looks like an
2335 intricate marionette hand with several strings for each finger:
2337 #+caption: View of the hand model with all sense nodes. You can see
2338 #+caption: the joint, muscle, ear, and eye nodess here.
2339 #+name: hand-nodes-1
2340 #+ATTR_LaTeX: :width 11cm
2341 [[./images/hand-with-all-senses2.png]]
2343 #+caption: An alternate view of the hand.
2344 #+name: hand-nodes-2
2345 #+ATTR_LaTeX: :width 15cm
2346 [[./images/hand-with-all-senses3.png]]
2348 With the hand fully rigged with senses, I can run it though a test
2349 that will test everything.
2351 #+caption: A full test of the hand with all senses. Note expecially
2352 #+caption: the interactions the hand has with itself: it feels
2353 #+caption: its own palm and fingers, and when it curls its fingers,
2354 #+caption: it sees them with its eye (which is located in the center
2355 #+caption: of the palm. The red block appears with a pure tone sound.
2356 #+caption: The hand then uses its muscles to launch the cube!
2357 #+name: integration
2358 #+ATTR_LaTeX: :width 16cm
2359 [[./images/integration.png]]
2361 ** COMMENT =CORTEX= enables many possiblities for further research
2363 Often times, the hardest part of building a system involving
2364 creatures is dealing with physics and graphics. =CORTEX= removes
2365 much of this initial difficulty and leaves researchers free to
2366 directly pursue their ideas. I hope that even undergrads with a
2367 passing curiosity about simulated touch or creature evolution will
2368 be able to use cortex for experimentation. =CORTEX= is a completely
2369 simulated world, and far from being a disadvantage, its simulated
2370 nature enables you to create senses and creatures that would be
2371 impossible to make in the real world.
2373 While not by any means a complete list, here are some paths
2374 =CORTEX= is well suited to help you explore:
2376 - Empathy :: my empathy program leaves many areas for
2377 improvement, among which are using vision to infer
2378 proprioception and looking up sensory experience with imagined
2379 vision, touch, and sound.
2380 - Evolution :: Karl Sims created a rich environment for
2381 simulating the evolution of creatures on a connection
2382 machine. Today, this can be redone and expanded with =CORTEX=
2383 on an ordinary computer.
2384 - Exotic senses :: Cortex enables many fascinating senses that are
2385 not possible to build in the real world. For example,
2386 telekinesis is an interesting avenue to explore. You can also
2387 make a ``semantic'' sense which looks up metadata tags on
2388 objects in the environment the metadata tags might contain
2389 other sensory information.
2390 - Imagination via subworlds :: this would involve a creature with
2391 an effector which creates an entire new sub-simulation where
2392 the creature has direct control over placement/creation of
2393 objects via simulated telekinesis. The creature observes this
2394 sub-world through it's normal senses and uses its observations
2395 to make predictions about its top level world.
2396 - Simulated prescience :: step the simulation forward a few ticks,
2397 gather sensory data, then supply this data for the creature as
2398 one of its actual senses. The cost of prescience is slowing
2399 the simulation down by a factor proportional to however far
2400 you want the entities to see into the future. What happens
2401 when two evolved creatures that can each see into the future
2402 fight each other?
2403 - Swarm creatures :: Program a group of creatures that cooperate
2404 with each other. Because the creatures would be simulated, you
2405 could investigate computationally complex rules of behavior
2406 which still, from the group's point of view, would happen in
2407 ``real time''. Interactions could be as simple as cellular
2408 organisms communicating via flashing lights, or as complex as
2409 humanoids completing social tasks, etc.
2410 - =HACKER= for writing muscle-control programs :: Presented with
2411 low-level muscle control/ sense API, generate higher level
2412 programs for accomplishing various stated goals. Example goals
2413 might be "extend all your fingers" or "move your hand into the
2414 area with blue light" or "decrease the angle of this joint".
2415 It would be like Sussman's HACKER, except it would operate
2416 with much more data in a more realistic world. Start off with
2417 "calisthenics" to develop subroutines over the motor control
2418 API. This would be the "spinal chord" of a more intelligent
2419 creature. The low level programming code might be a turning
2420 machine that could develop programs to iterate over a "tape"
2421 where each entry in the tape could control recruitment of the
2422 fibers in a muscle.
2423 - Sense fusion :: There is much work to be done on sense
2424 integration -- building up a coherent picture of the world and
2425 the things in it with =CORTEX= as a base, you can explore
2426 concepts like self-organizing maps or cross modal clustering
2427 in ways that have never before been tried.
2428 - Inverse kinematics :: experiments in sense guided motor control
2429 are easy given =CORTEX='s support -- you can get right to the
2430 hard control problems without worrying about physics or
2431 senses.
2433 * COMMENT Empathy in a simulated worm
2435 Here I develop a computational model of empathy, using =CORTEX= as a
2436 base. Empathy in this context is the ability to observe another
2437 creature and infer what sorts of sensations that creature is
2438 feeling. My empathy algorithm involves multiple phases. First is
2439 free-play, where the creature moves around and gains sensory
2440 experience. From this experience I construct a representation of the
2441 creature's sensory state space, which I call \Phi-space. Using
2442 \Phi-space, I construct an efficient function which takes the
2443 limited data that comes from observing another creature and enriches
2444 it full compliment of imagined sensory data. I can then use the
2445 imagined sensory data to recognize what the observed creature is
2446 doing and feeling, using straightforward embodied action predicates.
2447 This is all demonstrated with using a simple worm-like creature, and
2448 recognizing worm-actions based on limited data.
2450 #+caption: Here is the worm with which we will be working.
2451 #+caption: It is composed of 5 segments. Each segment has a
2452 #+caption: pair of extensor and flexor muscles. Each of the
2453 #+caption: worm's four joints is a hinge joint which allows
2454 #+caption: about 30 degrees of rotation to either side. Each segment
2455 #+caption: of the worm is touch-capable and has a uniform
2456 #+caption: distribution of touch sensors on each of its faces.
2457 #+caption: Each joint has a proprioceptive sense to detect
2458 #+caption: relative positions. The worm segments are all the
2459 #+caption: same except for the first one, which has a much
2460 #+caption: higher weight than the others to allow for easy
2461 #+caption: manual motor control.
2462 #+name: basic-worm-view
2463 #+ATTR_LaTeX: :width 10cm
2464 [[./images/basic-worm-view.png]]
2466 #+caption: Program for reading a worm from a blender file and
2467 #+caption: outfitting it with the senses of proprioception,
2468 #+caption: touch, and the ability to move, as specified in the
2469 #+caption: blender file.
2470 #+name: get-worm
2471 #+begin_listing clojure
2472 #+begin_src clojure
2473 (defn worm []
2474 (let [model (load-blender-model "Models/worm/worm.blend")]
2475 {:body (doto model (body!))
2476 :touch (touch! model)
2477 :proprioception (proprioception! model)
2478 :muscles (movement! model)}))
2479 #+end_src
2480 #+end_listing
2482 ** Embodiment factors action recognition into managable parts
2484 Using empathy, I divide the problem of action recognition into a
2485 recognition process expressed in the language of a full compliment
2486 of senses, and an imaganitive process that generates full sensory
2487 data from partial sensory data. Splitting the action recognition
2488 problem in this manner greatly reduces the total amount of work to
2489 recognize actions: The imaganitive process is mostly just matching
2490 previous experience, and the recognition process gets to use all
2491 the senses to directly describe any action.
2493 ** Action recognition is easy with a full gamut of senses
2495 Embodied representations using multiple senses such as touch,
2496 proprioception, and muscle tension turns out be be exceedingly
2497 efficient at describing body-centered actions. It is the ``right
2498 language for the job''. For example, it takes only around 5 lines
2499 of LISP code to describe the action of ``curling'' using embodied
2500 primitives. It takes about 10 lines to describe the seemingly
2501 complicated action of wiggling.
2503 The following action predicates each take a stream of sensory
2504 experience, observe however much of it they desire, and decide
2505 whether the worm is doing the action they describe. =curled?=
2506 relies on proprioception, =resting?= relies on touch, =wiggling?=
2507 relies on a fourier analysis of muscle contraction, and
2508 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
2510 #+caption: Program for detecting whether the worm is curled. This is the
2511 #+caption: simplest action predicate, because it only uses the last frame
2512 #+caption: of sensory experience, and only uses proprioceptive data. Even
2513 #+caption: this simple predicate, however, is automatically frame
2514 #+caption: independent and ignores vermopomorphic differences such as
2515 #+caption: worm textures and colors.
2516 #+name: curled
2517 #+attr_latex: [htpb]
2518 #+begin_listing clojure
2519 #+begin_src clojure
2520 (defn curled?
2521 "Is the worm curled up?"
2522 [experiences]
2523 (every?
2524 (fn [[_ _ bend]]
2525 (> (Math/sin bend) 0.64))
2526 (:proprioception (peek experiences))))
2527 #+end_src
2528 #+end_listing
2530 #+caption: Program for summarizing the touch information in a patch
2531 #+caption: of skin.
2532 #+name: touch-summary
2533 #+attr_latex: [htpb]
2535 #+begin_listing clojure
2536 #+begin_src clojure
2537 (defn contact
2538 "Determine how much contact a particular worm segment has with
2539 other objects. Returns a value between 0 and 1, where 1 is full
2540 contact and 0 is no contact."
2541 [touch-region [coords contact :as touch]]
2542 (-> (zipmap coords contact)
2543 (select-keys touch-region)
2544 (vals)
2545 (#(map first %))
2546 (average)
2547 (* 10)
2548 (- 1)
2549 (Math/abs)))
2550 #+end_src
2551 #+end_listing
2554 #+caption: Program for detecting whether the worm is at rest. This program
2555 #+caption: uses a summary of the tactile information from the underbelly
2556 #+caption: of the worm, and is only true if every segment is touching the
2557 #+caption: floor. Note that this function contains no references to
2558 #+caption: proprioction at all.
2559 #+name: resting
2560 #+attr_latex: [htpb]
2561 #+begin_listing clojure
2562 #+begin_src clojure
2563 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2565 (defn resting?
2566 "Is the worm resting on the ground?"
2567 [experiences]
2568 (every?
2569 (fn [touch-data]
2570 (< 0.9 (contact worm-segment-bottom touch-data)))
2571 (:touch (peek experiences))))
2572 #+end_src
2573 #+end_listing
2575 #+caption: Program for detecting whether the worm is curled up into a
2576 #+caption: full circle. Here the embodied approach begins to shine, as
2577 #+caption: I am able to both use a previous action predicate (=curled?=)
2578 #+caption: as well as the direct tactile experience of the head and tail.
2579 #+name: grand-circle
2580 #+attr_latex: [htpb]
2581 #+begin_listing clojure
2582 #+begin_src clojure
2583 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2585 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2587 (defn grand-circle?
2588 "Does the worm form a majestic circle (one end touching the other)?"
2589 [experiences]
2590 (and (curled? experiences)
2591 (let [worm-touch (:touch (peek experiences))
2592 tail-touch (worm-touch 0)
2593 head-touch (worm-touch 4)]
2594 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2595 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2596 #+end_src
2597 #+end_listing
2600 #+caption: Program for detecting whether the worm has been wiggling for
2601 #+caption: the last few frames. It uses a fourier analysis of the muscle
2602 #+caption: contractions of the worm's tail to determine wiggling. This is
2603 #+caption: signigicant because there is no particular frame that clearly
2604 #+caption: indicates that the worm is wiggling --- only when multiple frames
2605 #+caption: are analyzed together is the wiggling revealed. Defining
2606 #+caption: wiggling this way also gives the worm an opportunity to learn
2607 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2608 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2609 #+caption: from actual wiggling, but this definition gives it to us for free.
2610 #+name: wiggling
2611 #+attr_latex: [htpb]
2612 #+begin_listing clojure
2613 #+begin_src clojure
2614 (defn fft [nums]
2615 (map
2616 #(.getReal %)
2617 (.transform
2618 (FastFourierTransformer. DftNormalization/STANDARD)
2619 (double-array nums) TransformType/FORWARD)))
2621 (def indexed (partial map-indexed vector))
2623 (defn max-indexed [s]
2624 (first (sort-by (comp - second) (indexed s))))
2626 (defn wiggling?
2627 "Is the worm wiggling?"
2628 [experiences]
2629 (let [analysis-interval 0x40]
2630 (when (> (count experiences) analysis-interval)
2631 (let [a-flex 3
2632 a-ex 2
2633 muscle-activity
2634 (map :muscle (vector:last-n experiences analysis-interval))
2635 base-activity
2636 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2637 (= 2
2638 (first
2639 (max-indexed
2640 (map #(Math/abs %)
2641 (take 20 (fft base-activity))))))))))
2642 #+end_src
2643 #+end_listing
2645 With these action predicates, I can now recognize the actions of
2646 the worm while it is moving under my control and I have access to
2647 all the worm's senses.
2649 #+caption: Use the action predicates defined earlier to report on
2650 #+caption: what the worm is doing while in simulation.
2651 #+name: report-worm-activity
2652 #+attr_latex: [htpb]
2653 #+begin_listing clojure
2654 #+begin_src clojure
2655 (defn debug-experience
2656 [experiences text]
2657 (cond
2658 (grand-circle? experiences) (.setText text "Grand Circle")
2659 (curled? experiences) (.setText text "Curled")
2660 (wiggling? experiences) (.setText text "Wiggling")
2661 (resting? experiences) (.setText text "Resting")))
2662 #+end_src
2663 #+end_listing
2665 #+caption: Using =debug-experience=, the body-centered predicates
2666 #+caption: work together to classify the behaviour of the worm.
2667 #+caption: the predicates are operating with access to the worm's
2668 #+caption: full sensory data.
2669 #+name: basic-worm-view
2670 #+ATTR_LaTeX: :width 10cm
2671 [[./images/worm-identify-init.png]]
2673 These action predicates satisfy the recognition requirement of an
2674 empathic recognition system. There is power in the simplicity of
2675 the action predicates. They describe their actions without getting
2676 confused in visual details of the worm. Each one is frame
2677 independent, but more than that, they are each indepent of
2678 irrelevant visual details of the worm and the environment. They
2679 will work regardless of whether the worm is a different color or
2680 hevaily textured, or if the environment has strange lighting.
2682 The trick now is to make the action predicates work even when the
2683 sensory data on which they depend is absent. If I can do that, then
2684 I will have gained much,
2686 ** \Phi-space describes the worm's experiences
2688 As a first step towards building empathy, I need to gather all of
2689 the worm's experiences during free play. I use a simple vector to
2690 store all the experiences.
2692 Each element of the experience vector exists in the vast space of
2693 all possible worm-experiences. Most of this vast space is actually
2694 unreachable due to physical constraints of the worm's body. For
2695 example, the worm's segments are connected by hinge joints that put
2696 a practical limit on the worm's range of motions without limiting
2697 its degrees of freedom. Some groupings of senses are impossible;
2698 the worm can not be bent into a circle so that its ends are
2699 touching and at the same time not also experience the sensation of
2700 touching itself.
2702 As the worm moves around during free play and its experience vector
2703 grows larger, the vector begins to define a subspace which is all
2704 the sensations the worm can practicaly experience during normal
2705 operation. I call this subspace \Phi-space, short for
2706 physical-space. The experience vector defines a path through
2707 \Phi-space. This path has interesting properties that all derive
2708 from physical embodiment. The proprioceptive components are
2709 completely smooth, because in order for the worm to move from one
2710 position to another, it must pass through the intermediate
2711 positions. The path invariably forms loops as actions are repeated.
2712 Finally and most importantly, proprioception actually gives very
2713 strong inference about the other senses. For example, when the worm
2714 is flat, you can infer that it is touching the ground and that its
2715 muscles are not active, because if the muscles were active, the
2716 worm would be moving and would not be perfectly flat. In order to
2717 stay flat, the worm has to be touching the ground, or it would
2718 again be moving out of the flat position due to gravity. If the
2719 worm is positioned in such a way that it interacts with itself,
2720 then it is very likely to be feeling the same tactile feelings as
2721 the last time it was in that position, because it has the same body
2722 as then. If you observe multiple frames of proprioceptive data,
2723 then you can become increasingly confident about the exact
2724 activations of the worm's muscles, because it generally takes a
2725 unique combination of muscle contractions to transform the worm's
2726 body along a specific path through \Phi-space.
2728 There is a simple way of taking \Phi-space and the total ordering
2729 provided by an experience vector and reliably infering the rest of
2730 the senses.
2732 ** Empathy is the process of tracing though \Phi-space
2734 Here is the core of a basic empathy algorithm, starting with an
2735 experience vector:
2737 First, group the experiences into tiered proprioceptive bins. I use
2738 powers of 10 and 3 bins, and the smallest bin has an approximate
2739 size of 0.001 radians in all proprioceptive dimensions.
2741 Then, given a sequence of proprioceptive input, generate a set of
2742 matching experience records for each input, using the tiered
2743 proprioceptive bins.
2745 Finally, to infer sensory data, select the longest consective chain
2746 of experiences. Conecutive experience means that the experiences
2747 appear next to each other in the experience vector.
2749 This algorithm has three advantages:
2751 1. It's simple
2753 3. It's very fast -- retrieving possible interpretations takes
2754 constant time. Tracing through chains of interpretations takes
2755 time proportional to the average number of experiences in a
2756 proprioceptive bin. Redundant experiences in \Phi-space can be
2757 merged to save computation.
2759 2. It protects from wrong interpretations of transient ambiguous
2760 proprioceptive data. For example, if the worm is flat for just
2761 an instant, this flattness will not be interpreted as implying
2762 that the worm has its muscles relaxed, since the flattness is
2763 part of a longer chain which includes a distinct pattern of
2764 muscle activation. Markov chains or other memoryless statistical
2765 models that operate on individual frames may very well make this
2766 mistake.
2768 #+caption: Program to convert an experience vector into a
2769 #+caption: proprioceptively binned lookup function.
2770 #+name: bin
2771 #+attr_latex: [htpb]
2772 #+begin_listing clojure
2773 #+begin_src clojure
2774 (defn bin [digits]
2775 (fn [angles]
2776 (->> angles
2777 (flatten)
2778 (map (juxt #(Math/sin %) #(Math/cos %)))
2779 (flatten)
2780 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2782 (defn gen-phi-scan
2783 "Nearest-neighbors with binning. Only returns a result if
2784 the propriceptive data is within 10% of a previously recorded
2785 result in all dimensions."
2786 [phi-space]
2787 (let [bin-keys (map bin [3 2 1])
2788 bin-maps
2789 (map (fn [bin-key]
2790 (group-by
2791 (comp bin-key :proprioception phi-space)
2792 (range (count phi-space)))) bin-keys)
2793 lookups (map (fn [bin-key bin-map]
2794 (fn [proprio] (bin-map (bin-key proprio))))
2795 bin-keys bin-maps)]
2796 (fn lookup [proprio-data]
2797 (set (some #(% proprio-data) lookups)))))
2798 #+end_src
2799 #+end_listing
2801 #+caption: =longest-thread= finds the longest path of consecutive
2802 #+caption: experiences to explain proprioceptive worm data.
2803 #+name: phi-space-history-scan
2804 #+ATTR_LaTeX: :width 10cm
2805 [[./images/aurellem-gray.png]]
2807 =longest-thread= infers sensory data by stitching together pieces
2808 from previous experience. It prefers longer chains of previous
2809 experience to shorter ones. For example, during training the worm
2810 might rest on the ground for one second before it performs its
2811 excercises. If during recognition the worm rests on the ground for
2812 five seconds, =longest-thread= will accomodate this five second
2813 rest period by looping the one second rest chain five times.
2815 =longest-thread= takes time proportinal to the average number of
2816 entries in a proprioceptive bin, because for each element in the
2817 starting bin it performes a series of set lookups in the preceeding
2818 bins. If the total history is limited, then this is only a constant
2819 multiple times the number of entries in the starting bin. This
2820 analysis also applies even if the action requires multiple longest
2821 chains -- it's still the average number of entries in a
2822 proprioceptive bin times the desired chain length. Because
2823 =longest-thread= is so efficient and simple, I can interpret
2824 worm-actions in real time.
2826 #+caption: Program to calculate empathy by tracing though \Phi-space
2827 #+caption: and finding the longest (ie. most coherent) interpretation
2828 #+caption: of the data.
2829 #+name: longest-thread
2830 #+attr_latex: [htpb]
2831 #+begin_listing clojure
2832 #+begin_src clojure
2833 (defn longest-thread
2834 "Find the longest thread from phi-index-sets. The index sets should
2835 be ordered from most recent to least recent."
2836 [phi-index-sets]
2837 (loop [result '()
2838 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2839 (if (empty? phi-index-sets)
2840 (vec result)
2841 (let [threads
2842 (for [thread-base thread-bases]
2843 (loop [thread (list thread-base)
2844 remaining remaining]
2845 (let [next-index (dec (first thread))]
2846 (cond (empty? remaining) thread
2847 (contains? (first remaining) next-index)
2848 (recur
2849 (cons next-index thread) (rest remaining))
2850 :else thread))))
2851 longest-thread
2852 (reduce (fn [thread-a thread-b]
2853 (if (> (count thread-a) (count thread-b))
2854 thread-a thread-b))
2855 '(nil)
2856 threads)]
2857 (recur (concat longest-thread result)
2858 (drop (count longest-thread) phi-index-sets))))))
2859 #+end_src
2860 #+end_listing
2862 There is one final piece, which is to replace missing sensory data
2863 with a best-guess estimate. While I could fill in missing data by
2864 using a gradient over the closest known sensory data points,
2865 averages can be misleading. It is certainly possible to create an
2866 impossible sensory state by averaging two possible sensory states.
2867 Therefore, I simply replicate the most recent sensory experience to
2868 fill in the gaps.
2870 #+caption: Fill in blanks in sensory experience by replicating the most
2871 #+caption: recent experience.
2872 #+name: infer-nils
2873 #+attr_latex: [htpb]
2874 #+begin_listing clojure
2875 #+begin_src clojure
2876 (defn infer-nils
2877 "Replace nils with the next available non-nil element in the
2878 sequence, or barring that, 0."
2879 [s]
2880 (loop [i (dec (count s))
2881 v (transient s)]
2882 (if (zero? i) (persistent! v)
2883 (if-let [cur (v i)]
2884 (if (get v (dec i) 0)
2885 (recur (dec i) v)
2886 (recur (dec i) (assoc! v (dec i) cur)))
2887 (recur i (assoc! v i 0))))))
2888 #+end_src
2889 #+end_listing
2891 ** Efficient action recognition with =EMPATH=
2893 To use =EMPATH= with the worm, I first need to gather a set of
2894 experiences from the worm that includes the actions I want to
2895 recognize. The =generate-phi-space= program (listing
2896 \ref{generate-phi-space} runs the worm through a series of
2897 exercices and gatheres those experiences into a vector. The
2898 =do-all-the-things= program is a routine expressed in a simple
2899 muscle contraction script language for automated worm control. It
2900 causes the worm to rest, curl, and wiggle over about 700 frames
2901 (approx. 11 seconds).
2903 #+caption: Program to gather the worm's experiences into a vector for
2904 #+caption: further processing. The =motor-control-program= line uses
2905 #+caption: a motor control script that causes the worm to execute a series
2906 #+caption: of ``exercices'' that include all the action predicates.
2907 #+name: generate-phi-space
2908 #+attr_latex: [htpb]
2909 #+begin_listing clojure
2910 #+begin_src clojure
2911 (def do-all-the-things
2912 (concat
2913 curl-script
2914 [[300 :d-ex 40]
2915 [320 :d-ex 0]]
2916 (shift-script 280 (take 16 wiggle-script))))
2918 (defn generate-phi-space []
2919 (let [experiences (atom [])]
2920 (run-world
2921 (apply-map
2922 worm-world
2923 (merge
2924 (worm-world-defaults)
2925 {:end-frame 700
2926 :motor-control
2927 (motor-control-program worm-muscle-labels do-all-the-things)
2928 :experiences experiences})))
2929 @experiences))
2930 #+end_src
2931 #+end_listing
2933 #+caption: Use longest thread and a phi-space generated from a short
2934 #+caption: exercise routine to interpret actions during free play.
2935 #+name: empathy-debug
2936 #+attr_latex: [htpb]
2937 #+begin_listing clojure
2938 #+begin_src clojure
2939 (defn init []
2940 (def phi-space (generate-phi-space))
2941 (def phi-scan (gen-phi-scan phi-space)))
2943 (defn empathy-demonstration []
2944 (let [proprio (atom ())]
2945 (fn
2946 [experiences text]
2947 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2948 (swap! proprio (partial cons phi-indices))
2949 (let [exp-thread (longest-thread (take 300 @proprio))
2950 empathy (mapv phi-space (infer-nils exp-thread))]
2951 (println-repl (vector:last-n exp-thread 22))
2952 (cond
2953 (grand-circle? empathy) (.setText text "Grand Circle")
2954 (curled? empathy) (.setText text "Curled")
2955 (wiggling? empathy) (.setText text "Wiggling")
2956 (resting? empathy) (.setText text "Resting")
2957 :else (.setText text "Unknown")))))))
2959 (defn empathy-experiment [record]
2960 (.start (worm-world :experience-watch (debug-experience-phi)
2961 :record record :worm worm*)))
2962 #+end_src
2963 #+end_listing
2965 The result of running =empathy-experiment= is that the system is
2966 generally able to interpret worm actions using the action-predicates
2967 on simulated sensory data just as well as with actual data. Figure
2968 \ref{empathy-debug-image} was generated using =empathy-experiment=:
2970 #+caption: From only proprioceptive data, =EMPATH= was able to infer
2971 #+caption: the complete sensory experience and classify four poses
2972 #+caption: (The last panel shows a composite image of \emph{wriggling},
2973 #+caption: a dynamic pose.)
2974 #+name: empathy-debug-image
2975 #+ATTR_LaTeX: :width 10cm :placement [H]
2976 [[./images/empathy-1.png]]
2978 One way to measure the performance of =EMPATH= is to compare the
2979 sutiability of the imagined sense experience to trigger the same
2980 action predicates as the real sensory experience.
2982 #+caption: Determine how closely empathy approximates actual
2983 #+caption: sensory data.
2984 #+name: test-empathy-accuracy
2985 #+attr_latex: [htpb]
2986 #+begin_listing clojure
2987 #+begin_src clojure
2988 (def worm-action-label
2989 (juxt grand-circle? curled? wiggling?))
2991 (defn compare-empathy-with-baseline [matches]
2992 (let [proprio (atom ())]
2993 (fn
2994 [experiences text]
2995 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2996 (swap! proprio (partial cons phi-indices))
2997 (let [exp-thread (longest-thread (take 300 @proprio))
2998 empathy (mapv phi-space (infer-nils exp-thread))
2999 experience-matches-empathy
3000 (= (worm-action-label experiences)
3001 (worm-action-label empathy))]
3002 (println-repl experience-matches-empathy)
3003 (swap! matches #(conj % experience-matches-empathy)))))))
3005 (defn accuracy [v]
3006 (float (/ (count (filter true? v)) (count v))))
3008 (defn test-empathy-accuracy []
3009 (let [res (atom [])]
3010 (run-world
3011 (worm-world :experience-watch
3012 (compare-empathy-with-baseline res)
3013 :worm worm*))
3014 (accuracy @res)))
3015 #+end_src
3016 #+end_listing
3018 Running =test-empathy-accuracy= using the very short exercise
3019 program defined in listing \ref{generate-phi-space}, and then doing
3020 a similar pattern of activity manually yeilds an accuracy of around
3021 73%. This is based on very limited worm experience. By training the
3022 worm for longer, the accuracy dramatically improves.
3024 #+caption: Program to generate \Phi-space using manual training.
3025 #+name: manual-phi-space
3026 #+attr_latex: [htpb]
3027 #+begin_listing clojure
3028 #+begin_src clojure
3029 (defn init-interactive []
3030 (def phi-space
3031 (let [experiences (atom [])]
3032 (run-world
3033 (apply-map
3034 worm-world
3035 (merge
3036 (worm-world-defaults)
3037 {:experiences experiences})))
3038 @experiences))
3039 (def phi-scan (gen-phi-scan phi-space)))
3040 #+end_src
3041 #+end_listing
3043 After about 1 minute of manual training, I was able to achieve 95%
3044 accuracy on manual testing of the worm using =init-interactive= and
3045 =test-empathy-accuracy=. The majority of errors are near the
3046 boundaries of transitioning from one type of action to another.
3047 During these transitions the exact label for the action is more open
3048 to interpretation, and dissaggrement between empathy and experience
3049 is more excusable.
3051 ** Digression: bootstrapping touch using free exploration
3053 In the previous section I showed how to compute actions in terms of
3054 body-centered predicates which relied averate touch activation of
3055 pre-defined regions of the worm's skin. What if, instead of recieving
3056 touch pre-grouped into the six faces of each worm segment, the true
3057 topology of the worm's skin was unknown? This is more similiar to how
3058 a nerve fiber bundle might be arranged. While two fibers that are
3059 close in a nerve bundle /might/ correspond to two touch sensors that
3060 are close together on the skin, the process of taking a complicated
3061 surface and forcing it into essentially a circle requires some cuts
3062 and rerragenments.
3064 In this section I show how to automatically learn the skin-topology of
3065 a worm segment by free exploration. As the worm rolls around on the
3066 floor, large sections of its surface get activated. If the worm has
3067 stopped moving, then whatever region of skin that is touching the
3068 floor is probably an important region, and should be recorded.
3070 #+caption: Program to detect whether the worm is in a resting state
3071 #+caption: with one face touching the floor.
3072 #+name: pure-touch
3073 #+begin_listing clojure
3074 #+begin_src clojure
3075 (def full-contact [(float 0.0) (float 0.1)])
3077 (defn pure-touch?
3078 "This is worm specific code to determine if a large region of touch
3079 sensors is either all on or all off."
3080 [[coords touch :as touch-data]]
3081 (= (set (map first touch)) (set full-contact)))
3082 #+end_src
3083 #+end_listing
3085 After collecting these important regions, there will many nearly
3086 similiar touch regions. While for some purposes the subtle
3087 differences between these regions will be important, for my
3088 purposes I colapse them into mostly non-overlapping sets using
3089 =remove-similiar= in listing \ref{remove-similiar}
3091 #+caption: Program to take a lits of set of points and ``collapse them''
3092 #+caption: so that the remaining sets in the list are siginificantly
3093 #+caption: different from each other. Prefer smaller sets to larger ones.
3094 #+name: remove-similiar
3095 #+begin_listing clojure
3096 #+begin_src clojure
3097 (defn remove-similar
3098 [coll]
3099 (loop [result () coll (sort-by (comp - count) coll)]
3100 (if (empty? coll) result
3101 (let [[x & xs] coll
3102 c (count x)]
3103 (if (some
3104 (fn [other-set]
3105 (let [oc (count other-set)]
3106 (< (- (count (union other-set x)) c) (* oc 0.1))))
3107 xs)
3108 (recur result xs)
3109 (recur (cons x result) xs))))))
3110 #+end_src
3111 #+end_listing
3113 Actually running this simulation is easy given =CORTEX='s facilities.
3115 #+caption: Collect experiences while the worm moves around. Filter the touch
3116 #+caption: sensations by stable ones, collapse similiar ones together,
3117 #+caption: and report the regions learned.
3118 #+name: learn-touch
3119 #+begin_listing clojure
3120 #+begin_src clojure
3121 (defn learn-touch-regions []
3122 (let [experiences (atom [])
3123 world (apply-map
3124 worm-world
3125 (assoc (worm-segment-defaults)
3126 :experiences experiences))]
3127 (run-world world)
3128 (->>
3129 @experiences
3130 (drop 175)
3131 ;; access the single segment's touch data
3132 (map (comp first :touch))
3133 ;; only deal with "pure" touch data to determine surfaces
3134 (filter pure-touch?)
3135 ;; associate coordinates with touch values
3136 (map (partial apply zipmap))
3137 ;; select those regions where contact is being made
3138 (map (partial group-by second))
3139 (map #(get % full-contact))
3140 (map (partial map first))
3141 ;; remove redundant/subset regions
3142 (map set)
3143 remove-similar)))
3145 (defn learn-and-view-touch-regions []
3146 (map view-touch-region
3147 (learn-touch-regions)))
3148 #+end_src
3149 #+end_listing
3151 The only thing remining to define is the particular motion the worm
3152 must take. I accomplish this with a simple motor control program.
3154 #+caption: Motor control program for making the worm roll on the ground.
3155 #+caption: This could also be replaced with random motion.
3156 #+name: worm-roll
3157 #+begin_listing clojure
3158 #+begin_src clojure
3159 (defn touch-kinesthetics []
3160 [[170 :lift-1 40]
3161 [190 :lift-1 19]
3162 [206 :lift-1 0]
3164 [400 :lift-2 40]
3165 [410 :lift-2 0]
3167 [570 :lift-2 40]
3168 [590 :lift-2 21]
3169 [606 :lift-2 0]
3171 [800 :lift-1 30]
3172 [809 :lift-1 0]
3174 [900 :roll-2 40]
3175 [905 :roll-2 20]
3176 [910 :roll-2 0]
3178 [1000 :roll-2 40]
3179 [1005 :roll-2 20]
3180 [1010 :roll-2 0]
3182 [1100 :roll-2 40]
3183 [1105 :roll-2 20]
3184 [1110 :roll-2 0]
3185 ])
3186 #+end_src
3187 #+end_listing
3190 #+caption: The small worm rolls around on the floor, driven
3191 #+caption: by the motor control program in listing \ref{worm-roll}.
3192 #+name: worm-roll
3193 #+ATTR_LaTeX: :width 12cm
3194 [[./images/worm-roll.png]]
3197 #+caption: After completing its adventures, the worm now knows
3198 #+caption: how its touch sensors are arranged along its skin. These
3199 #+caption: are the regions that were deemed important by
3200 #+caption: =learn-touch-regions=. Note that the worm has discovered
3201 #+caption: that it has six sides.
3202 #+name: worm-touch-map
3203 #+ATTR_LaTeX: :width 12cm
3204 [[./images/touch-learn.png]]
3206 While simple, =learn-touch-regions= exploits regularities in both
3207 the worm's physiology and the worm's environment to correctly
3208 deduce that the worm has six sides. Note that =learn-touch-regions=
3209 would work just as well even if the worm's touch sense data were
3210 completely scrambled. The cross shape is just for convienence. This
3211 example justifies the use of pre-defined touch regions in =EMPATH=.
3213 * COMMENT Contributions
3215 In this thesis you have seen the =CORTEX= system, a complete
3216 environment for creating simulated creatures. You have seen how to
3217 implement five senses including touch, proprioception, hearing,
3218 vision, and muscle tension. You have seen how to create new creatues
3219 using blender, a 3D modeling tool. I hope that =CORTEX= will be
3220 useful in further research projects. To this end I have included the
3221 full source to =CORTEX= along with a large suite of tests and
3222 examples. I have also created a user guide for =CORTEX= which is
3223 inculded in an appendix to this thesis.
3225 You have also seen how I used =CORTEX= as a platform to attach the
3226 /action recognition/ problem, which is the problem of recognizing
3227 actions in video. You saw a simple system called =EMPATH= which
3228 ientifies actions by first describing actions in a body-centerd,
3229 rich sense language, then infering a full range of sensory
3230 experience from limited data using previous experience gained from
3231 free play.
3233 As a minor digression, you also saw how I used =CORTEX= to enable a
3234 tiny worm to discover the topology of its skin simply by rolling on
3235 the ground.
3237 In conclusion, the main contributions of this thesis are:
3239 - =CORTEX=, a system for creating simulated creatures with rich
3240 senses.
3241 - =EMPATH=, a program for recognizing actions by imagining sensory
3242 experience.
3244 # An anatomical joke:
3245 # - Training
3246 # - Skeletal imitation
3247 # - Sensory fleshing-out
3248 # - Classification
3249 #+BEGIN_LaTeX
3250 \appendix
3251 #+END_LaTeX
3252 * COMMENT Appendix: =CORTEX= User Guide
3254 Those who write a thesis should endeavor to make their code not only
3255 accessable, but actually useable, as a way to pay back the community
3256 that made the thesis possible in the first place. This thesis would
3257 not be possible without Free Software such as jMonkeyEngine3,
3258 Blender, clojure, emacs, ffmpeg, and many other tools. That is why I
3259 have included this user guide, in the hope that someone else might
3260 find =CORTEX= useful.
3262 ** Obtaining =CORTEX=
3264 You can get cortex from its mercurial repository at
3265 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3266 releases at http://aurellem.org/cortex/releases/. As a condition of
3267 making this thesis, I have also provided Professor Winston the
3268 =CORTEX= source, and he knows how to run the demos and get started.
3269 You may also email me at =cortex@aurellem.org= and I may help where
3270 I can.
3272 ** Running =CORTEX=
3274 =CORTEX= comes with README and INSTALL files that will guide you
3275 through installation and running the test suite. In particular you
3276 should look at test =cortex.test= which contains test suites that
3277 run through all senses and multiple creatures.
3279 ** Creating creatures
3281 Creatures are created using /Blender/, a free 3D modeling program.
3282 You will need Blender version 2.6 when using the =CORTEX= included
3283 in this thesis. You create a =CORTEX= creature in a similiar manner
3284 to modeling anything in Blender, except that you also create
3285 several trees of empty nodes which define the creature's senses.
3287 *** Mass
3289 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3290 to the object with the mass in jMonkeyEngine units. Note that
3291 setting the mass to 0 causes the object to be immovable.
3293 *** Joints
3295 Joints are created by creating an empty node named =joints= and
3296 then creating any number of empty child nodes to represent your
3297 creature's joints. The joint will automatically connect the
3298 closest two physical objects. It will help to set the empty node's
3299 display mode to ``Arrows'' so that you can clearly see the
3300 direction of the axes.
3302 Joint nodes should have the following metadata under the ``joint''
3303 label:
3305 #+BEGIN_SRC clojure
3306 ;; ONE OF the following, under the label "joint":
3307 {:type :point}
3309 ;; OR
3311 {:type :hinge
3312 :limit [<limit-low> <limit-high>]
3313 :axis (Vector3f. <x> <y> <z>)}
3314 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3316 ;; OR
3318 {:type :cone
3319 :limit-xz <lim-xz>
3320 :limit-xy <lim-xy>
3321 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
3322 #+END_SRC
3324 *** Eyes
3326 Eyes are created by creating an empty node named =eyes= and then
3327 creating any number of empty child nodes to represent your
3328 creature's eyes.
3330 Eye nodes should have the following metadata under the ``eye''
3331 label:
3333 #+BEGIN_SRC clojure
3334 {:red <red-retina-definition>
3335 :blue <blue-retina-definition>
3336 :green <green-retina-definition>
3337 :all <all-retina-definition>
3338 (<0xrrggbb> <custom-retina-image>)...
3340 #+END_SRC
3342 Any of the color channels may be omitted. You may also include
3343 your own color selectors, and in fact :red is equivalent to
3344 0xFF0000 and so forth. The eye will be placed at the same position
3345 as the empty node and will bind to the neatest physical object.
3346 The eye will point outward from the X-axis of the node, and ``up''
3347 will be in the direction of the X-axis of the node. It will help
3348 to set the empty node's display mode to ``Arrows'' so that you can
3349 clearly see the direction of the axes.
3351 Each retina file should contain white pixels whever you want to be
3352 sensitive to your chosen color. If you want the entire field of
3353 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3354 white.
3356 Here is a sample retinal map:
3358 #+caption: An example retinal profile image. White pixels are
3359 #+caption: photo-sensitive elements. The distribution of white
3360 #+caption: pixels is denser in the middle and falls off at the
3361 #+caption: edges and is inspired by the human retina.
3362 #+name: retina
3363 #+ATTR_LaTeX: :width 7cm :placement [H]
3364 [[./images/retina-small.png]]
3366 *** Hearing
3368 Ears are created by creating an empty node named =ears= and then
3369 creating any number of empty child nodes to represent your
3370 creature's ears.
3372 Ear nodes do not require any metadata.
3374 The ear will bind to and follow the closest physical node.
3376 *** Touch
3378 Touch is handled similarly to mass. To make a particular object
3379 touch sensitive, add metadata of the following form under the
3380 object's ``touch'' metadata field:
3382 #+BEGIN_EXAMPLE
3383 <touch-UV-map-file-name>
3384 #+END_EXAMPLE
3386 You may also include an optional ``scale'' metadata number to
3387 specifiy the length of the touch feelers. The default is $0.1$,
3388 and this is generally sufficient.
3390 The touch UV should contain white pixels for each touch sensor.
3392 Here is an example touch-uv map that approximates a human finger,
3393 and its corresponding model.
3395 #+caption: This is the tactile-sensor-profile for the upper segment
3396 #+caption: of a fingertip. It defines regions of high touch sensitivity
3397 #+caption: (where there are many white pixels) and regions of low
3398 #+caption: sensitivity (where white pixels are sparse).
3399 #+name: guide-fingertip-UV
3400 #+ATTR_LaTeX: :width 9cm :placement [H]
3401 [[./images/finger-UV.png]]
3403 #+caption: The fingertip UV-image form above applied to a simple
3404 #+caption: model of a fingertip.
3405 #+name: guide-fingertip
3406 #+ATTR_LaTeX: :width 9cm :placement [H]
3407 [[./images/finger-2.png]]
3409 *** Propriocepotion
3411 Proprioception is tied to each joint node -- nothing special must
3412 be done in a blender model to enable proprioception other than
3413 creating joint nodes.
3415 *** Muscles
3417 Muscles are created by creating an empty node named =muscles= and
3418 then creating any number of empty child nodes to represent your
3419 creature's muscles.
3422 Muscle nodes should have the following metadata under the
3423 ``muscle'' label:
3425 #+BEGIN_EXAMPLE
3426 <muscle-profile-file-name>
3427 #+END_EXAMPLE
3429 Muscles should also have a ``strength'' metadata entry describing
3430 the muscle's total strength at full activation.
3432 Muscle profiles are simple images that contain the relative amount
3433 of muscle power in each simulated alpha motor neuron. The width of
3434 the image is the total size of the motor pool, and the redness of
3435 each neuron is the relative power of that motor pool.
3437 While the profile image can have any dimensions, only the first
3438 line of pixels is used to define the muscle. Here is a sample
3439 muscle profile image that defines a human-like muscle.
3441 #+caption: A muscle profile image that describes the strengths
3442 #+caption: of each motor neuron in a muscle. White is weakest
3443 #+caption: and dark red is strongest. This particular pattern
3444 #+caption: has weaker motor neurons at the beginning, just
3445 #+caption: like human muscle.
3446 #+name: muscle-recruit
3447 #+ATTR_LaTeX: :width 7cm :placement [H]
3448 [[./images/basic-muscle.png]]
3450 Muscles twist the nearest physical object about the muscle node's
3451 Z-axis. I recommend using the ``Single Arrow'' display mode for
3452 muscles and using the right hand rule to determine which way the
3453 muscle will twist. To make a segment that can twist in multiple
3454 directions, create multiple, differently aligned muscles.
3456 ** =CORTEX= API
3458 These are the some functions exposed by =CORTEX= for creating
3459 worlds and simulating creatures. These are in addition to
3460 jMonkeyEngine3's extensive library, which is documented elsewhere.
3462 *** Simulation
3463 - =(world root-node key-map setup-fn update-fn)= :: create
3464 a simulation.
3465 - /root-node/ :: a =com.jme3.scene.Node= object which
3466 contains all of the objects that should be in the
3467 simulation.
3469 - /key-map/ :: a map from strings describing keys to
3470 functions that should be executed whenever that key is
3471 pressed. the functions should take a SimpleApplication
3472 object and a boolean value. The SimpleApplication is the
3473 current simulation that is running, and the boolean is true
3474 if the key is being pressed, and false if it is being
3475 released. As an example,
3476 #+BEGIN_SRC clojure
3477 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3478 #+END_SRC
3479 is a valid key-map which will cause the simulation to print
3480 a message whenever the 'j' key on the keyboard is pressed.
3482 - /setup-fn/ :: a function that takes a =SimpleApplication=
3483 object. It is called once when initializing the simulation.
3484 Use it to create things like lights, change the gravity,
3485 initialize debug nodes, etc.
3487 - /update-fn/ :: this function takes a =SimpleApplication=
3488 object and a float and is called every frame of the
3489 simulation. The float tells how many seconds is has been
3490 since the last frame was rendered, according to whatever
3491 clock jme is currently using. The default is to use IsoTimer
3492 which will result in this value always being the same.
3494 - =(position-camera world position rotation)= :: set the position
3495 of the simulation's main camera.
3497 - =(enable-debug world)= :: turn on debug wireframes for each
3498 simulated object.
3500 - =(set-gravity world gravity)= :: set the gravity of a running
3501 simulation.
3503 - =(box length width height & {options})= :: create a box in the
3504 simulation. Options is a hash map specifying texture, mass,
3505 etc. Possible options are =:name=, =:color=, =:mass=,
3506 =:friction=, =:texture=, =:material=, =:position=,
3507 =:rotation=, =:shape=, and =:physical?=.
3509 - =(sphere radius & {options})= :: create a sphere in the simulation.
3510 Options are the same as in =box=.
3512 - =(load-blender-model file-name)= :: create a node structure
3513 representing that described in a blender file.
3515 - =(light-up-everything world)= :: distribute a standard compliment
3516 of lights throught the simulation. Should be adequate for most
3517 purposes.
3519 - =(node-seq node)= :: return a recursuve list of the node's
3520 children.
3522 - =(nodify name children)= :: construct a node given a node-name and
3523 desired children.
3525 - =(add-element world element)= :: add an object to a running world
3526 simulation.
3528 - =(set-accuracy world accuracy)= :: change the accuracy of the
3529 world's physics simulator.
3531 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3532 construct that is useful for loading textures and is required
3533 for smooth interaction with jMonkeyEngine library functions.
3535 - =(load-bullet)= :: unpack native libraries and initialize
3536 blender. This function is required before other world building
3537 functions are called.
3540 *** Creature Manipulation / Import
3542 - =(body! creature)= :: give the creature a physical body.
3544 - =(vision! creature)= :: give the creature a sense of vision.
3545 Returns a list of functions which will each, when called
3546 during a simulation, return the vision data for the channel of
3547 one of the eyes. The functions are ordered depending on the
3548 alphabetical order of the names of the eye nodes in the
3549 blender file. The data returned by the functions is a vector
3550 containing the eye's /topology/, a vector of coordinates, and
3551 the eye's /data/, a vector of RGB values filtered by the eye's
3552 sensitivity.
3554 - =(hearing! creature)= :: give the creature a sense of hearing.
3555 Returns a list of functions, one for each ear, that when
3556 called will return a frame's worth of hearing data for that
3557 ear. The functions are ordered depending on the alphabetical
3558 order of the names of the ear nodes in the blender file. The
3559 data returned by the functions is an array PCM encoded wav
3560 data.
3562 - =(touch! creature)= :: give the creature a sense of touch. Returns
3563 a single function that must be called with the /root node/ of
3564 the world, and which will return a vector of /touch-data/
3565 one entry for each touch sensitive component, each entry of
3566 which contains a /topology/ that specifies the distribution of
3567 touch sensors, and the /data/, which is a vector of
3568 =[activation, length]= pairs for each touch hair.
3570 - =(proprioception! creature)= :: give the creature the sense of
3571 proprioception. Returns a list of functions, one for each
3572 joint, that when called during a running simulation will
3573 report the =[headnig, pitch, roll]= of the joint.
3575 - =(movement! creature)= :: give the creature the power of movement.
3576 Creates a list of functions, one for each muscle, that when
3577 called with an integer, will set the recruitment of that
3578 muscle to that integer, and will report the current power
3579 being exerted by the muscle. Order of muscles is determined by
3580 the alphabetical sort order of the names of the muscle nodes.
3582 *** Visualization/Debug
3584 - =(view-vision)= :: create a function that when called with a list
3585 of visual data returned from the functions made by =vision!=,
3586 will display that visual data on the screen.
3588 - =(view-hearing)= :: same as =view-vision= but for hearing.
3590 - =(view-touch)= :: same as =view-vision= but for touch.
3592 - =(view-proprioception)= :: same as =view-vision= but for
3593 proprioception.
3595 - =(view-movement)= :: same as =view-vision= but for
3596 proprioception.
3598 - =(view anything)= :: =view= is a polymorphic function that allows
3599 you to inspect almost anything you could reasonably expect to
3600 be able to ``see'' in =CORTEX=.
3602 - =(text anything)= :: =text= is a polymorphic function that allows
3603 you to convert practically anything into a text string.
3605 - =(println-repl anything)= :: print messages to clojure's repl
3606 instead of the simulation's terminal window.
3608 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3609 function will import all jMonkeyEngine3 classes for immediate
3610 use.
3612 - =(display-dialated-time world timer)= :: Shows the time as it is
3613 flowing in the simulation on a HUD display.