view thesis/cortex.org @ 536:0b0fef5e817b

more clarification.
author Robert McIntyre <rlm@mit.edu>
date Sun, 27 Apr 2014 20:39:33 -0400
parents 8a5abd51cd4f
children bc5eb476693a
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * COMMENT Empathy \& Embodiment: problem solving strategies
46 By the end of this thesis, you will have seen a novel approach to
47 interpreting video using embodiment and empathy. You will also see
48 one way to efficiently implement physical empathy for embodied
49 creatures. Finally, you will become familiar with =CORTEX=, a system
50 for designing and simulating creatures with rich senses, which I
51 have designed as a library that you can use in your own research.
52 Note that I /do not/ process video directly --- I start with
53 knowledge of the positions of a creature's body parts and works from
54 there.
56 This is the core vision of my thesis: That one of the important ways
57 in which we understand others is by imagining ourselves in their
58 position and emphatically feeling experiences relative to our own
59 bodies. By understanding events in terms of our own previous
60 corporeal experience, we greatly constrain the possibilities of what
61 would otherwise be an unwieldy exponential search. This extra
62 constraint can be the difference between easily understanding what
63 is happening in a video and being completely lost in a sea of
64 incomprehensible color and movement.
66 ** The problem: recognizing actions is hard!
68 Examine the following image. What is happening? As you, and indeed
69 very young children, can easily determine, this is an image of
70 drinking.
72 #+caption: A cat drinking some water. Identifying this action is
73 #+caption: beyond the capabilities of existing computer vision systems.
74 #+ATTR_LaTeX: :width 7cm
75 [[./images/cat-drinking.jpg]]
77 Nevertheless, it is beyond the state of the art for a computer
78 vision program to describe what's happening in this image. Part of
79 the problem is that many computer vision systems focus on
80 pixel-level details or comparisons to example images (such as
81 \cite{volume-action-recognition}), but the 3D world is so variable
82 that it is hard to describe the world in terms of possible images.
84 In fact, the contents of scene may have much less to do with pixel
85 probabilities than with recognizing various affordances: things you
86 can move, objects you can grasp, spaces that can be filled . For
87 example, what processes might enable you to see the chair in figure
88 \ref{hidden-chair}?
90 #+caption: The chair in this image is quite obvious to humans, but
91 #+caption: it can't be found by any modern computer vision program.
92 #+name: hidden-chair
93 #+ATTR_LaTeX: :width 10cm
94 [[./images/fat-person-sitting-at-desk.jpg]]
96 Finally, how is it that you can easily tell the difference between
97 how the girls /muscles/ are working in figure \ref{girl}?
99 #+caption: The mysterious ``common sense'' appears here as you are able
100 #+caption: to discern the difference in how the girl's arm muscles
101 #+caption: are activated between the two images.
102 #+name: girl
103 #+ATTR_LaTeX: :width 7cm
104 [[./images/wall-push.png]]
106 Each of these examples tells us something about what might be going
107 on in our minds as we easily solve these recognition problems:
109 The hidden chair shows us that we are strongly triggered by cues
110 relating to the position of human bodies, and that we can determine
111 the overall physical configuration of a human body even if much of
112 that body is occluded.
114 The picture of the girl pushing against the wall tells us that we
115 have common sense knowledge about the kinetics of our own bodies.
116 We know well how our muscles would have to work to maintain us in
117 most positions, and we can easily project this self-knowledge to
118 imagined positions triggered by images of the human body.
120 The cat tells us that imagination of some kind plays an important
121 role in understanding actions. The question is: Can we be more
122 precise about what sort of imagination is required to understand
123 these actions?
125 ** A step forward: the sensorimotor-centered approach
127 In this thesis, I explore the idea that our knowledge of our own
128 bodies, combined with our own rich senses, enables us to recognize
129 the actions of others.
131 For example, I think humans are able to label the cat video as
132 ``drinking'' because they imagine /themselves/ as the cat, and
133 imagine putting their face up against a stream of water and
134 sticking out their tongue. In that imagined world, they can feel
135 the cool water hitting their tongue, and feel the water entering
136 their body, and are able to recognize that /feeling/ as drinking.
137 So, the label of the action is not really in the pixels of the
138 image, but is found clearly in a simulation inspired by those
139 pixels. An imaginative system, having been trained on drinking and
140 non-drinking examples and learning that the most important
141 component of drinking is the feeling of water sliding down one's
142 throat, would analyze a video of a cat drinking in the following
143 manner:
145 1. Create a physical model of the video by putting a ``fuzzy''
146 model of its own body in place of the cat. Possibly also create
147 a simulation of the stream of water.
149 2. ``Play out'' this simulated scene and generate imagined sensory
150 experience. This will include relevant muscle contractions, a
151 close up view of the stream from the cat's perspective, and most
152 importantly, the imagined feeling of water entering the mouth.
153 The imagined sensory experience can come from a simulation of
154 the event, but can also be pattern-matched from previous,
155 similar embodied experience.
157 3. The action is now easily identified as drinking by the sense of
158 taste alone. The other senses (such as the tongue moving in and
159 out) help to give plausibility to the simulated action. Note that
160 the sense of vision, while critical in creating the simulation,
161 is not critical for identifying the action from the simulation.
163 For the chair examples, the process is even easier:
165 1. Align a model of your body to the person in the image.
167 2. Generate proprioceptive sensory data from this alignment.
169 3. Use the imagined proprioceptive data as a key to lookup related
170 sensory experience associated with that particular proprioceptive
171 feeling.
173 4. Retrieve the feeling of your bottom resting on a surface, your
174 knees bent, and your leg muscles relaxed.
176 5. This sensory information is consistent with your =sitting?=
177 sensory predicate, so you (and the entity in the image) must be
178 sitting.
180 6. There must be a chair-like object since you are sitting.
182 Empathy offers yet another alternative to the age-old AI
183 representation question: ``What is a chair?'' --- A chair is the
184 feeling of sitting!
186 One powerful advantage of empathic problem solving is that it
187 factors the action recognition problem into two easier problems. To
188 use empathy, you need an /aligner/, which takes the video and a
189 model of your body, and aligns the model with the video. Then, you
190 need a /recognizer/, which uses the aligned model to interpret the
191 action. The power in this method lies in the fact that you describe
192 all actions from a body-centered viewpoint. You are less tied to
193 the particulars of any visual representation of the actions. If you
194 teach the system what ``running'' is, and you have a good enough
195 aligner, the system will from then on be able to recognize running
196 from any point of view, even strange points of view like above or
197 underneath the runner. This is in contrast to action recognition
198 schemes that try to identify actions using a non-embodied approach.
199 If these systems learn about running as viewed from the side, they
200 will not automatically be able to recognize running from any other
201 viewpoint.
203 Another powerful advantage is that using the language of multiple
204 body-centered rich senses to describe body-centered actions offers a
205 massive boost in descriptive capability. Consider how difficult it
206 would be to compose a set of HOG filters to describe the action of
207 a simple worm-creature ``curling'' so that its head touches its
208 tail, and then behold the simplicity of describing thus action in a
209 language designed for the task (listing \ref{grand-circle-intro}):
211 #+caption: Body-centered actions are best expressed in a body-centered
212 #+caption: language. This code detects when the worm has curled into a
213 #+caption: full circle. Imagine how you would replicate this functionality
214 #+caption: using low-level pixel features such as HOG filters!
215 #+name: grand-circle-intro
216 #+begin_listing clojure
217 #+begin_src clojure
218 (defn grand-circle?
219 "Does the worm form a majestic circle (one end touching the other)?"
220 [experiences]
221 (and (curled? experiences)
222 (let [worm-touch (:touch (peek experiences))
223 tail-touch (worm-touch 0)
224 head-touch (worm-touch 4)]
225 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
226 (< 0.2 (contact worm-segment-top-tip head-touch))))))
227 #+end_src
228 #+end_listing
230 ** =EMPATH= recognizes actions using empathy
232 Exploring these ideas further demands a concrete implementation, so
233 first, I built a system for constructing virtual creatures with
234 physiologically plausible sensorimotor systems and detailed
235 environments. The result is =CORTEX=, which is described in section
236 \ref{sec-2}.
238 Next, I wrote routines which enabled a simple worm-like creature to
239 infer the actions of a second worm-like creature, using only its
240 own prior sensorimotor experiences and knowledge of the second
241 worm's joint positions. This program, =EMPATH=, is described in
242 section \ref{sec-3}. It's main components are:
244 - Embodied Action Definitions :: Many otherwise complicated actions
245 are easily described in the language of a full suite of
246 body-centered, rich senses and experiences. For example,
247 drinking is the feeling of water sliding down your throat, and
248 cooling your insides. It's often accompanied by bringing your
249 hand close to your face, or bringing your face close to water.
250 Sitting down is the feeling of bending your knees, activating
251 your quadriceps, then feeling a surface with your bottom and
252 relaxing your legs. These body-centered action descriptions
253 can be either learned or hard coded.
255 - Guided Play :: The creature moves around and experiences the
256 world through its unique perspective. As the creature moves,
257 it gathers experiences that satisfy the embodied action
258 definitions.
260 - Posture imitation :: When trying to interpret a video or image,
261 the creature takes a model of itself and aligns it with
262 whatever it sees. This alignment might even cross species, as
263 when humans try to align themselves with things like ponies,
264 dogs, or other humans with a different body type.
266 - Empathy :: The alignment triggers associations with
267 sensory data from prior experiences. For example, the
268 alignment itself easily maps to proprioceptive data. Any
269 sounds or obvious skin contact in the video can to a lesser
270 extent trigger previous experience keyed to hearing or touch.
271 Segments of previous experiences gained from play are stitched
272 together to form a coherent and complete sensory portrait of
273 the scene.
275 - Recognition :: With the scene described in terms of
276 remembered first person sensory events, the creature can now
277 run its action-identified programs (such as the one in listing
278 \ref{grand-circle-intro} on this synthesized sensory data,
279 just as it would if it were actually experiencing the scene
280 first-hand. If previous experience has been accurately
281 retrieved, and if it is analogous enough to the scene, then
282 the creature will correctly identify the action in the scene.
284 My program, =EMPATH= uses this empathic problem solving technique
285 to interpret the actions of a simple, worm-like creature.
287 #+caption: The worm performs many actions during free play such as
288 #+caption: curling, wiggling, and resting.
289 #+name: worm-intro
290 #+ATTR_LaTeX: :width 15cm
291 [[./images/worm-intro-white.png]]
293 #+caption: =EMPATH= recognized and classified each of these
294 #+caption: poses by inferring the complete sensory experience
295 #+caption: from proprioceptive data.
296 #+name: worm-recognition-intro
297 #+ATTR_LaTeX: :width 15cm
298 [[./images/worm-poses.png]]
300 *** Main Results
302 - After one-shot supervised training, =EMPATH= was able to
303 recognize a wide variety of static poses and dynamic
304 actions---ranging from curling in a circle to wiggling with a
305 particular frequency --- with 95\% accuracy.
307 - These results were completely independent of viewing angle
308 because the underlying body-centered language fundamentally is
309 independent; once an action is learned, it can be recognized
310 equally well from any viewing angle.
312 - =EMPATH= is surprisingly short; the sensorimotor-centered
313 language provided by =CORTEX= resulted in extremely economical
314 recognition routines --- about 500 lines in all --- suggesting
315 that such representations are very powerful, and often
316 indispensable for the types of recognition tasks considered here.
318 - Although for expediency's sake, I relied on direct knowledge of
319 joint positions in this proof of concept, it would be
320 straightforward to extend =EMPATH= so that it (more
321 realistically) infers joint positions from its visual data.
323 ** =EMPATH= is built on =CORTEX=, a creature builder.
325 I built =CORTEX= to be a general AI research platform for doing
326 experiments involving multiple rich senses and a wide variety and
327 number of creatures. I intend it to be useful as a library for many
328 more projects than just this thesis. =CORTEX= was necessary to meet
329 a need among AI researchers at CSAIL and beyond, which is that
330 people often will invent neat ideas that are best expressed in the
331 language of creatures and senses, but in order to explore those
332 ideas they must first build a platform in which they can create
333 simulated creatures with rich senses! There are many ideas that
334 would be simple to execute (such as =EMPATH= or
335 \cite{larson-symbols}), but attached to them is the multi-month
336 effort to make a good creature simulator. Often, that initial
337 investment of time proves to be too much, and the project must make
338 do with a lesser environment.
340 =CORTEX= is well suited as an environment for embodied AI research
341 for three reasons:
343 - You can create new creatures using Blender (\cite{blender}), a
344 popular 3D modeling program. Each sense can be specified using
345 special blender nodes with biologically inspired parameters. You
346 need not write any code to create a creature, and can use a wide
347 library of pre-existing blender models as a base for your own
348 creatures.
350 - =CORTEX= implements a wide variety of senses: touch,
351 proprioception, vision, hearing, and muscle tension. Complicated
352 senses like touch, and vision involve multiple sensory elements
353 embedded in a 2D surface. You have complete control over the
354 distribution of these sensor elements through the use of simple
355 png image files. In particular, =CORTEX= implements more
356 comprehensive hearing than any other creature simulation system
357 available.
359 - =CORTEX= supports any number of creatures and any number of
360 senses. Time in =CORTEX= dilates so that the simulated creatures
361 always perceive a perfectly smooth flow of time, regardless of
362 the actual computational load.
364 =CORTEX= is built on top of =jMonkeyEngine3=
365 (\cite{jmonkeyengine}), which is a video game engine designed to
366 create cross-platform 3D desktop games. =CORTEX= is mainly written
367 in clojure, a dialect of =LISP= that runs on the java virtual
368 machine (JVM). The API for creating and simulating creatures and
369 senses is entirely expressed in clojure, though many senses are
370 implemented at the layer of jMonkeyEngine or below. For example,
371 for the sense of hearing I use a layer of clojure code on top of a
372 layer of java JNI bindings that drive a layer of =C++= code which
373 implements a modified version of =OpenAL= to support multiple
374 listeners. =CORTEX= is the only simulation environment that I know
375 of that can support multiple entities that can each hear the world
376 from their own perspective. Other senses also require a small layer
377 of Java code. =CORTEX= also uses =bullet=, a physics simulator
378 written in =C=.
380 #+caption: Here is the worm from figure \ref{worm-intro} modeled
381 #+caption: in Blender, a free 3D-modeling program. Senses and
382 #+caption: joints are described using special nodes in Blender.
383 #+name: worm-recognition-intro-2
384 #+ATTR_LaTeX: :width 12cm
385 [[./images/blender-worm.png]]
387 Here are some things I anticipate that =CORTEX= might be used for:
389 - exploring new ideas about sensory integration
390 - distributed communication among swarm creatures
391 - self-learning using free exploration,
392 - evolutionary algorithms involving creature construction
393 - exploration of exotic senses and effectors that are not possible
394 in the real world (such as telekinesis or a semantic sense)
395 - imagination using subworlds
397 During one test with =CORTEX=, I created 3,000 creatures each with
398 their own independent senses and ran them all at only 1/80 real
399 time. In another test, I created a detailed model of my own hand,
400 equipped with a realistic distribution of touch (more sensitive at
401 the fingertips), as well as eyes and ears, and it ran at around 1/4
402 real time.
404 #+BEGIN_LaTeX
405 \begin{sidewaysfigure}
406 \includegraphics[width=9.5in]{images/full-hand.png}
407 \caption{
408 I modeled my own right hand in Blender and rigged it with all the
409 senses that {\tt CORTEX} supports. My simulated hand has a
410 biologically inspired distribution of touch sensors. The senses are
411 displayed on the right, and the simulation is displayed on the
412 left. Notice that my hand is curling its fingers, that it can see
413 its own finger from the eye in its palm, and that it can feel its
414 own thumb touching its palm.}
415 \end{sidewaysfigure}
416 #+END_LaTeX
418 * COMMENT Designing =CORTEX=
420 In this section, I outline the design decisions that went into
421 making =CORTEX=, along with some details about its implementation.
422 (A practical guide to getting started with =CORTEX=, which skips
423 over the history and implementation details presented here, is
424 provided in an appendix at the end of this thesis.)
426 Throughout this project, I intended for =CORTEX= to be flexible and
427 extensible enough to be useful for other researchers who want to
428 test out ideas of their own. To this end, wherever I have had to make
429 architectural choices about =CORTEX=, I have chosen to give as much
430 freedom to the user as possible, so that =CORTEX= may be used for
431 things I have not foreseen.
433 ** Building in simulation versus reality
434 The most important architectural decision of all is the choice to
435 use a computer-simulated environment in the first place! The world
436 is a vast and rich place, and for now simulations are a very poor
437 reflection of its complexity. It may be that there is a significant
438 qualitative difference between dealing with senses in the real
439 world and dealing with pale facsimiles of them in a simulation
440 \cite{brooks-representation}. What are the advantages and
441 disadvantages of a simulation vs. reality?
443 *** Simulation
445 The advantages of virtual reality are that when everything is a
446 simulation, experiments in that simulation are absolutely
447 reproducible. It's also easier to change the character and world
448 to explore new situations and different sensory combinations.
450 If the world is to be simulated on a computer, then not only do
451 you have to worry about whether the character's senses are rich
452 enough to learn from the world, but whether the world itself is
453 rendered with enough detail and realism to give enough working
454 material to the character's senses. To name just a few
455 difficulties facing modern physics simulators: destructibility of
456 the environment, simulation of water/other fluids, large areas,
457 nonrigid bodies, lots of objects, smoke. I don't know of any
458 computer simulation that would allow a character to take a rock
459 and grind it into fine dust, then use that dust to make a clay
460 sculpture, at least not without spending years calculating the
461 interactions of every single small grain of dust. Maybe a
462 simulated world with today's limitations doesn't provide enough
463 richness for real intelligence to evolve.
465 *** Reality
467 The other approach for playing with senses is to hook your
468 software up to real cameras, microphones, robots, etc., and let it
469 loose in the real world. This has the advantage of eliminating
470 concerns about simulating the world at the expense of increasing
471 the complexity of implementing the senses. Instead of just
472 grabbing the current rendered frame for processing, you have to
473 use an actual camera with real lenses and interact with photons to
474 get an image. It is much harder to change the character, which is
475 now partly a physical robot of some sort, since doing so involves
476 changing things around in the real world instead of modifying
477 lines of code. While the real world is very rich and definitely
478 provides enough stimulation for intelligence to develop as
479 evidenced by our own existence, it is also uncontrollable in the
480 sense that a particular situation cannot be recreated perfectly or
481 saved for later use. It is harder to conduct science because it is
482 harder to repeat an experiment. The worst thing about using the
483 real world instead of a simulation is the matter of time. Instead
484 of simulated time you get the constant and unstoppable flow of
485 real time. This severely limits the sorts of software you can use
486 to program an AI, because all sense inputs must be handled in real
487 time. Complicated ideas may have to be implemented in hardware or
488 may simply be impossible given the current speed of our
489 processors. Contrast this with a simulation, in which the flow of
490 time in the simulated world can be slowed down to accommodate the
491 limitations of the character's programming. In terms of cost,
492 doing everything in software is far cheaper than building custom
493 real-time hardware. All you need is a laptop and some patience.
495 ** Simulated time enables rapid prototyping \& simple programs
497 I envision =CORTEX= being used to support rapid prototyping and
498 iteration of ideas. Even if I could put together a well constructed
499 kit for creating robots, it would still not be enough because of
500 the scourge of real-time processing. Anyone who wants to test their
501 ideas in the real world must always worry about getting their
502 algorithms to run fast enough to process information in real time.
503 The need for real time processing only increases if multiple senses
504 are involved. In the extreme case, even simple algorithms will have
505 to be accelerated by ASIC chips or FPGAs, turning what would
506 otherwise be a few lines of code and a 10x speed penalty into a
507 multi-month ordeal. For this reason, =CORTEX= supports
508 /time-dilation/, which scales back the framerate of the
509 simulation in proportion to the amount of processing each frame.
510 From the perspective of the creatures inside the simulation, time
511 always appears to flow at a constant rate, regardless of how
512 complicated the environment becomes or how many creatures are in
513 the simulation. The cost is that =CORTEX= can sometimes run slower
514 than real time. This can also be an advantage, however ---
515 simulations of very simple creatures in =CORTEX= generally run at
516 40x on my machine!
518 ** All sense organs are two-dimensional surfaces
520 If =CORTEX= is to support a wide variety of senses, it would help
521 to have a better understanding of what a ``sense'' actually is!
522 While vision, touch, and hearing all seem like they are quite
523 different things, I was surprised to learn during the course of
524 this thesis that they (and all physical senses) can be expressed as
525 exactly the same mathematical object due to a dimensional argument!
527 Human beings are three-dimensional objects, and the nerves that
528 transmit data from our various sense organs to our brain are
529 essentially one-dimensional. This leaves up to two dimensions in
530 which our sensory information may flow. For example, imagine your
531 skin: it is a two-dimensional surface around a three-dimensional
532 object (your body). It has discrete touch sensors embedded at
533 various points, and the density of these sensors corresponds to the
534 sensitivity of that region of skin. Each touch sensor connects to a
535 nerve, all of which eventually are bundled together as they travel
536 up the spinal cord to the brain. Intersect the spinal nerves with a
537 guillotining plane and you will see all of the sensory data of the
538 skin revealed in a roughly circular two-dimensional image which is
539 the cross section of the spinal cord. Points on this image that are
540 close together in this circle represent touch sensors that are
541 /probably/ close together on the skin, although there is of course
542 some cutting and rearrangement that has to be done to transfer the
543 complicated surface of the skin onto a two dimensional image.
545 Most human senses consist of many discrete sensors of various
546 properties distributed along a surface at various densities. For
547 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
548 disks, and Ruffini's endings \cite{textbook901}, which detect
549 pressure and vibration of various intensities. For ears, it is the
550 stereocilia distributed along the basilar membrane inside the
551 cochlea; each one is sensitive to a slightly different frequency of
552 sound. For eyes, it is rods and cones distributed along the surface
553 of the retina. In each case, we can describe the sense with a
554 surface and a distribution of sensors along that surface.
556 In fact, almost every human sense can be effectively described in
557 terms of a surface containing embedded sensors. If the sense had
558 any more dimensions, then there wouldn't be enough room in the
559 spinal chord to transmit the information!
561 Therefore, =CORTEX= must support the ability to create objects and
562 then be able to ``paint'' points along their surfaces to describe
563 each sense.
565 Fortunately this idea is already a well known computer graphics
566 technique called /UV-mapping/. The three-dimensional surface of a
567 model is cut and smooshed until it fits on a two-dimensional
568 image. You paint whatever you want on that image, and when the
569 three-dimensional shape is rendered in a game the smooshing and
570 cutting is reversed and the image appears on the three-dimensional
571 object.
573 To make a sense, interpret the UV-image as describing the
574 distribution of that senses sensors. To get different types of
575 sensors, you can either use a different color for each type of
576 sensor, or use multiple UV-maps, each labeled with that sensor
577 type. I generally use a white pixel to mean the presence of a
578 sensor and a black pixel to mean the absence of a sensor, and use
579 one UV-map for each sensor-type within a given sense.
581 #+CAPTION: The UV-map for an elongated icososphere. The white
582 #+caption: dots each represent a touch sensor. They are dense
583 #+caption: in the regions that describe the tip of the finger,
584 #+caption: and less dense along the dorsal side of the finger
585 #+caption: opposite the tip.
586 #+name: finger-UV
587 #+ATTR_latex: :width 10cm
588 [[./images/finger-UV.png]]
590 #+caption: Ventral side of the UV-mapped finger. Notice the
591 #+caption: density of touch sensors at the tip.
592 #+name: finger-side-view
593 #+ATTR_LaTeX: :width 10cm
594 [[./images/finger-1.png]]
596 ** Video game engines provide ready-made physics and shading
598 I did not need to write my own physics simulation code or shader to
599 build =CORTEX=. Doing so would lead to a system that is impossible
600 for anyone but myself to use anyway. Instead, I use a video game
601 engine as a base and modify it to accommodate the additional needs
602 of =CORTEX=. Video game engines are an ideal starting point to
603 build =CORTEX=, because they are not far from being creature
604 building systems themselves.
606 First off, general purpose video game engines come with a physics
607 engine and lighting / sound system. The physics system provides
608 tools that can be co-opted to serve as touch, proprioception, and
609 muscles. Since some games support split screen views, a good video
610 game engine will allow you to efficiently create multiple cameras
611 in the simulated world that can be used as eyes. Video game systems
612 offer integrated asset management for things like textures and
613 creatures models, providing an avenue for defining creatures. They
614 also understand UV-mapping, since this technique is used to apply a
615 texture to a model. Finally, because video game engines support a
616 large number of users, as long as =CORTEX= doesn't stray too far
617 from the base system, other researchers can turn to this community
618 for help when doing their research.
620 ** =CORTEX= is based on jMonkeyEngine3
622 While preparing to build =CORTEX= I studied several video game
623 engines to see which would best serve as a base. The top contenders
624 were:
626 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
627 software in 1997. All the source code was released by ID
628 software into the Public Domain several years ago, and as a
629 result it has been ported to many different languages. This
630 engine was famous for its advanced use of realistic shading
631 and had decent and fast physics simulation. The main advantage
632 of the Quake II engine is its simplicity, but I ultimately
633 rejected it because the engine is too tied to the concept of a
634 first-person shooter game. One of the problems I had was that
635 there does not seem to be any easy way to attach multiple
636 cameras to a single character. There are also several physics
637 clipping issues that are corrected in a way that only applies
638 to the main character and do not apply to arbitrary objects.
640 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
641 and Quake I engines and is used by Valve in the Half-Life
642 series of games. The physics simulation in the Source Engine
643 is quite accurate and probably the best out of all the engines
644 I investigated. There is also an extensive community actively
645 working with the engine. However, applications that use the
646 Source Engine must be written in C++, the code is not open, it
647 only runs on Windows, and the tools that come with the SDK to
648 handle models and textures are complicated and awkward to use.
650 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
651 games in Java. It uses OpenGL to render to the screen and uses
652 screengraphs to avoid drawing things that do not appear on the
653 screen. It has an active community and several games in the
654 pipeline. The engine was not built to serve any particular
655 game but is instead meant to be used for any 3D game.
657 I chose jMonkeyEngine3 because it had the most features out of all
658 the free projects I looked at, and because I could then write my
659 code in clojure, an implementation of =LISP= that runs on the JVM.
661 ** =CORTEX= uses Blender to create creature models
663 For the simple worm-like creatures I will use later on in this
664 thesis, I could define a simple API in =CORTEX= that would allow
665 one to create boxes, spheres, etc., and leave that API as the sole
666 way to create creatures. However, for =CORTEX= to truly be useful
667 for other projects, it needs a way to construct complicated
668 creatures. If possible, it would be nice to leverage work that has
669 already been done by the community of 3D modelers, or at least
670 enable people who are talented at modeling but not programming to
671 design =CORTEX= creatures.
673 Therefore, I use Blender, a free 3D modeling program, as the main
674 way to create creatures in =CORTEX=. However, the creatures modeled
675 in Blender must also be simple to simulate in jMonkeyEngine3's game
676 engine, and must also be easy to rig with =CORTEX='s senses. I
677 accomplish this with extensive use of Blender's ``empty nodes.''
679 Empty nodes have no mass, physical presence, or appearance, but
680 they can hold metadata and have names. I use a tree structure of
681 empty nodes to specify senses in the following manner:
683 - Create a single top-level empty node whose name is the name of
684 the sense.
685 - Add empty nodes which each contain meta-data relevant to the
686 sense, including a UV-map describing the number/distribution of
687 sensors if applicable.
688 - Make each empty-node the child of the top-level node.
690 #+caption: An example of annotating a creature model with empty
691 #+caption: nodes to describe the layout of senses. There are
692 #+caption: multiple empty nodes which each describe the position
693 #+caption: of muscles, ears, eyes, or joints.
694 #+name: sense-nodes
695 #+ATTR_LaTeX: :width 10cm
696 [[./images/empty-sense-nodes.png]]
698 ** Bodies are composed of segments connected by joints
700 Blender is a general purpose animation tool, which has been used in
701 the past to create high quality movies such as Sintel
702 \cite{blender}. Though Blender can model and render even complicated
703 things like water, it is crucial to keep models that are meant to
704 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
705 though jMonkeyEngine3, is a rigid-body physics system. This offers
706 a compromise between the expressiveness of a game level and the
707 speed at which it can be simulated, and it means that creatures
708 should be naturally expressed as rigid components held together by
709 joint constraints.
711 But humans are more like a squishy bag wrapped around some hard
712 bones which define the overall shape. When we move, our skin bends
713 and stretches to accommodate the new positions of our bones.
715 One way to make bodies composed of rigid pieces connected by joints
716 /seem/ more human-like is to use an /armature/, (or /rigging/)
717 system, which defines a overall ``body mesh'' and defines how the
718 mesh deforms as a function of the position of each ``bone'' which
719 is a standard rigid body. This technique is used extensively to
720 model humans and create realistic animations. It is not a good
721 technique for physical simulation because it is a lie -- the skin
722 is not a physical part of the simulation and does not interact with
723 any objects in the world or itself. Objects will pass right though
724 the skin until they come in contact with the underlying bone, which
725 is a physical object. Without simulating the skin, the sense of
726 touch has little meaning, and the creature's own vision will lie to
727 it about the true extent of its body. Simulating the skin as a
728 physical object requires some way to continuously update the
729 physical model of the skin along with the movement of the bones,
730 which is unacceptably slow compared to rigid body simulation.
732 Therefore, instead of using the human-like ``deformable bag of
733 bones'' approach, I decided to base my body plans on multiple solid
734 objects that are connected by joints, inspired by the robot =EVE=
735 from the movie WALL-E.
737 #+caption: =EVE= from the movie WALL-E. This body plan turns
738 #+caption: out to be much better suited to my purposes than a more
739 #+caption: human-like one.
740 #+ATTR_LaTeX: :width 10cm
741 [[./images/Eve.jpg]]
743 =EVE='s body is composed of several rigid components that are held
744 together by invisible joint constraints. This is what I mean by
745 ``eve-like''. The main reason that I use eve-style bodies is for
746 efficiency, and so that there will be correspondence between the
747 AI's senses and the physical presence of its body. Each individual
748 section is simulated by a separate rigid body that corresponds
749 exactly with its visual representation and does not change.
750 Sections are connected by invisible joints that are well supported
751 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
752 can efficiently simulate hundreds of rigid bodies connected by
753 joints. Just because sections are rigid does not mean they have to
754 stay as one piece forever; they can be dynamically replaced with
755 multiple sections to simulate splitting in two. This could be used
756 to simulate retractable claws or =EVE='s hands, which are able to
757 coalesce into one object in the movie.
759 *** Solidifying/Connecting a body
761 =CORTEX= creates a creature in two steps: first, it traverses the
762 nodes in the blender file and creates physical representations for
763 any of them that have mass defined in their blender meta-data.
765 #+caption: Program for iterating through the nodes in a blender file
766 #+caption: and generating physical jMonkeyEngine3 objects with mass
767 #+caption: and a matching physics shape.
768 #+name: physical
769 #+begin_listing clojure
770 #+begin_src clojure
771 (defn physical!
772 "Iterate through the nodes in creature and make them real physical
773 objects in the simulation."
774 [#^Node creature]
775 (dorun
776 (map
777 (fn [geom]
778 (let [physics-control
779 (RigidBodyControl.
780 (HullCollisionShape.
781 (.getMesh geom))
782 (if-let [mass (meta-data geom "mass")]
783 (float mass) (float 1)))]
784 (.addControl geom physics-control)))
785 (filter #(isa? (class %) Geometry )
786 (node-seq creature)))))
787 #+end_src
788 #+end_listing
790 The next step to making a proper body is to connect those pieces
791 together with joints. jMonkeyEngine has a large array of joints
792 available via =bullet=, such as Point2Point, Cone, Hinge, and a
793 generic Six Degree of Freedom joint, with or without spring
794 restitution.
796 Joints are treated a lot like proper senses, in that there is a
797 top-level empty node named ``joints'' whose children each
798 represent a joint.
800 #+caption: View of the hand model in Blender showing the main ``joints''
801 #+caption: node (highlighted in yellow) and its children which each
802 #+caption: represent a joint in the hand. Each joint node has metadata
803 #+caption: specifying what sort of joint it is.
804 #+name: blender-hand
805 #+ATTR_LaTeX: :width 10cm
806 [[./images/hand-screenshot1.png]]
809 =CORTEX='s procedure for binding the creature together with joints
810 is as follows:
812 - Find the children of the ``joints'' node.
813 - Determine the two spatials the joint is meant to connect.
814 - Create the joint based on the meta-data of the empty node.
816 The higher order function =sense-nodes= from =cortex.sense=
817 simplifies finding the joints based on their parent ``joints''
818 node.
820 #+caption: Retrieving the children empty nodes from a single
821 #+caption: named empty node is a common pattern in =CORTEX=
822 #+caption: further instances of this technique for the senses
823 #+caption: will be omitted
824 #+name: get-empty-nodes
825 #+begin_listing clojure
826 #+begin_src clojure
827 (defn sense-nodes
828 "For some senses there is a special empty blender node whose
829 children are considered markers for an instance of that sense. This
830 function generates functions to find those children, given the name
831 of the special parent node."
832 [parent-name]
833 (fn [#^Node creature]
834 (if-let [sense-node (.getChild creature parent-name)]
835 (seq (.getChildren sense-node)) [])))
837 (def
838 ^{:doc "Return the children of the creature's \"joints\" node."
839 :arglists '([creature])}
840 joints
841 (sense-nodes "joints"))
842 #+end_src
843 #+end_listing
845 To find a joint's targets, =CORTEX= creates a small cube, centered
846 around the empty-node, and grows the cube exponentially until it
847 intersects two physical objects. The objects are ordered according
848 to the joint's rotation, with the first one being the object that
849 has more negative coordinates in the joint's reference frame.
850 Since the objects must be physical, the empty-node itself escapes
851 detection. Because the objects must be physical, =joint-targets=
852 must be called /after/ =physical!= is called.
854 #+caption: Program to find the targets of a joint node by
855 #+caption: exponentially growth of a search cube.
856 #+name: joint-targets
857 #+begin_listing clojure
858 #+begin_src clojure
859 (defn joint-targets
860 "Return the two closest two objects to the joint object, ordered
861 from bottom to top according to the joint's rotation."
862 [#^Node parts #^Node joint]
863 (loop [radius (float 0.01)]
864 (let [results (CollisionResults.)]
865 (.collideWith
866 parts
867 (BoundingBox. (.getWorldTranslation joint)
868 radius radius radius) results)
869 (let [targets
870 (distinct
871 (map #(.getGeometry %) results))]
872 (if (>= (count targets) 2)
873 (sort-by
874 #(let [joint-ref-frame-position
875 (jme-to-blender
876 (.mult
877 (.inverse (.getWorldRotation joint))
878 (.subtract (.getWorldTranslation %)
879 (.getWorldTranslation joint))))]
880 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
881 (take 2 targets))
882 (recur (float (* radius 2))))))))
883 #+end_src
884 #+end_listing
886 Once =CORTEX= finds all joints and targets, it creates them using
887 a dispatch on the metadata of each joint node.
889 #+caption: Program to dispatch on blender metadata and create joints
890 #+caption: suitable for physical simulation.
891 #+name: joint-dispatch
892 #+begin_listing clojure
893 #+begin_src clojure
894 (defmulti joint-dispatch
895 "Translate blender pseudo-joints into real JME joints."
896 (fn [constraints & _]
897 (:type constraints)))
899 (defmethod joint-dispatch :point
900 [constraints control-a control-b pivot-a pivot-b rotation]
901 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
902 (.setLinearLowerLimit Vector3f/ZERO)
903 (.setLinearUpperLimit Vector3f/ZERO)))
905 (defmethod joint-dispatch :hinge
906 [constraints control-a control-b pivot-a pivot-b rotation]
907 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
908 [limit-1 limit-2] (:limit constraints)
909 hinge-axis (.mult rotation (blender-to-jme axis))]
910 (doto (HingeJoint. control-a control-b pivot-a pivot-b
911 hinge-axis hinge-axis)
912 (.setLimit limit-1 limit-2))))
914 (defmethod joint-dispatch :cone
915 [constraints control-a control-b pivot-a pivot-b rotation]
916 (let [limit-xz (:limit-xz constraints)
917 limit-xy (:limit-xy constraints)
918 twist (:twist constraints)]
919 (doto (ConeJoint. control-a control-b pivot-a pivot-b
920 rotation rotation)
921 (.setLimit (float limit-xz) (float limit-xy)
922 (float twist)))))
923 #+end_src
924 #+end_listing
926 All that is left for joints is to combine the above pieces into
927 something that can operate on the collection of nodes that a
928 blender file represents.
930 #+caption: Program to completely create a joint given information
931 #+caption: from a blender file.
932 #+name: connect
933 #+begin_listing clojure
934 #+begin_src clojure
935 (defn connect
936 "Create a joint between 'obj-a and 'obj-b at the location of
937 'joint. The type of joint is determined by the metadata on 'joint.
939 Here are some examples:
940 {:type :point}
941 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
942 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
944 {:type :cone :limit-xz 0]
945 :limit-xy 0]
946 :twist 0]} (use XZY rotation mode in blender!)"
947 [#^Node obj-a #^Node obj-b #^Node joint]
948 (let [control-a (.getControl obj-a RigidBodyControl)
949 control-b (.getControl obj-b RigidBodyControl)
950 joint-center (.getWorldTranslation joint)
951 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
952 pivot-a (world-to-local obj-a joint-center)
953 pivot-b (world-to-local obj-b joint-center)]
954 (if-let
955 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
956 ;; A side-effect of creating a joint registers
957 ;; it with both physics objects which in turn
958 ;; will register the joint with the physics system
959 ;; when the simulation is started.
960 (joint-dispatch constraints
961 control-a control-b
962 pivot-a pivot-b
963 joint-rotation))))
964 #+end_src
965 #+end_listing
967 In general, whenever =CORTEX= exposes a sense (or in this case
968 physicality), it provides a function of the type =sense!=, which
969 takes in a collection of nodes and augments it to support that
970 sense. The function returns any controls necessary to use that
971 sense. In this case =body!= creates a physical body and returns no
972 control functions.
974 #+caption: Program to give joints to a creature.
975 #+name: joints
976 #+begin_listing clojure
977 #+begin_src clojure
978 (defn joints!
979 "Connect the solid parts of the creature with physical joints. The
980 joints are taken from the \"joints\" node in the creature."
981 [#^Node creature]
982 (dorun
983 (map
984 (fn [joint]
985 (let [[obj-a obj-b] (joint-targets creature joint)]
986 (connect obj-a obj-b joint)))
987 (joints creature))))
988 (defn body!
989 "Endow the creature with a physical body connected with joints. The
990 particulars of the joints and the masses of each body part are
991 determined in blender."
992 [#^Node creature]
993 (physical! creature)
994 (joints! creature))
995 #+end_src
996 #+end_listing
998 All of the code you have just seen amounts to only 130 lines, yet
999 because it builds on top of Blender and jMonkeyEngine3, those few
1000 lines pack quite a punch!
1002 The hand from figure \ref{blender-hand}, which was modeled after
1003 my own right hand, can now be given joints and simulated as a
1004 creature.
1006 #+caption: With the ability to create physical creatures from blender,
1007 #+caption: =CORTEX= gets one step closer to becoming a full creature
1008 #+caption: simulation environment.
1009 #+name: physical-hand
1010 #+ATTR_LaTeX: :width 15cm
1011 [[./images/physical-hand.png]]
1013 ** Sight reuses standard video game components...
1015 Vision is one of the most important senses for humans, so I need to
1016 build a simulated sense of vision for my AI. I will do this with
1017 simulated eyes. Each eye can be independently moved and should see
1018 its own version of the world depending on where it is.
1020 Making these simulated eyes a reality is simple because
1021 jMonkeyEngine already contains extensive support for multiple views
1022 of the same 3D simulated world. The reason jMonkeyEngine has this
1023 support is because the support is necessary to create games with
1024 split-screen views. Multiple views are also used to create
1025 efficient pseudo-reflections by rendering the scene from a certain
1026 perspective and then projecting it back onto a surface in the 3D
1027 world.
1029 #+caption: jMonkeyEngine supports multiple views to enable
1030 #+caption: split-screen games, like GoldenEye, which was one of
1031 #+caption: the first games to use split-screen views.
1032 #+name: goldeneye
1033 #+ATTR_LaTeX: :width 10cm
1034 [[./images/goldeneye-4-player.png]]
1036 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
1038 jMonkeyEngine allows you to create a =ViewPort=, which represents a
1039 view of the simulated world. You can create as many of these as you
1040 want. Every frame, the =RenderManager= iterates through each
1041 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
1042 is a =FrameBuffer= which represents the rendered image in the GPU.
1044 #+caption: =ViewPorts= are cameras in the world. During each frame,
1045 #+caption: the =RenderManager= records a snapshot of what each view
1046 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
1047 #+name: rendermanagers
1048 #+ATTR_LaTeX: :width 10cm
1049 [[./images/diagram_rendermanager2.png]]
1051 Each =ViewPort= can have any number of attached =SceneProcessor=
1052 objects, which are called every time a new frame is rendered. A
1053 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1054 whatever it wants to the data. Often this consists of invoking GPU
1055 specific operations on the rendered image. The =SceneProcessor= can
1056 also copy the GPU image data to RAM and process it with the CPU.
1058 *** Appropriating Views for Vision
1060 Each eye in the simulated creature needs its own =ViewPort= so
1061 that it can see the world from its own perspective. To this
1062 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1063 any arbitrary continuation function for further processing. That
1064 continuation function may perform both CPU and GPU operations on
1065 the data. To make this easy for the continuation function, the
1066 =SceneProcessor= maintains appropriately sized buffers in RAM to
1067 hold the data. It does not do any copying from the GPU to the CPU
1068 itself because it is a slow operation.
1070 #+caption: Function to make the rendered scene in jMonkeyEngine
1071 #+caption: available for further processing.
1072 #+name: pipeline-1
1073 #+begin_listing clojure
1074 #+begin_src clojure
1075 (defn vision-pipeline
1076 "Create a SceneProcessor object which wraps a vision processing
1077 continuation function. The continuation is a function that takes
1078 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1079 each of which has already been appropriately sized."
1080 [continuation]
1081 (let [byte-buffer (atom nil)
1082 renderer (atom nil)
1083 image (atom nil)]
1084 (proxy [SceneProcessor] []
1085 (initialize
1086 [renderManager viewPort]
1087 (let [cam (.getCamera viewPort)
1088 width (.getWidth cam)
1089 height (.getHeight cam)]
1090 (reset! renderer (.getRenderer renderManager))
1091 (reset! byte-buffer
1092 (BufferUtils/createByteBuffer
1093 (* width height 4)))
1094 (reset! image (BufferedImage.
1095 width height
1096 BufferedImage/TYPE_4BYTE_ABGR))))
1097 (isInitialized [] (not (nil? @byte-buffer)))
1098 (reshape [_ _ _])
1099 (preFrame [_])
1100 (postQueue [_])
1101 (postFrame
1102 [#^FrameBuffer fb]
1103 (.clear @byte-buffer)
1104 (continuation @renderer fb @byte-buffer @image))
1105 (cleanup []))))
1106 #+end_src
1107 #+end_listing
1109 The continuation function given to =vision-pipeline= above will be
1110 given a =Renderer= and three containers for image data. The
1111 =FrameBuffer= references the GPU image data, but the pixel data
1112 can not be used directly on the CPU. The =ByteBuffer= and
1113 =BufferedImage= are initially "empty" but are sized to hold the
1114 data in the =FrameBuffer=. I call transferring the GPU image data
1115 to the CPU structures "mixing" the image data.
1117 *** Optical sensor arrays are described with images and referenced with metadata
1119 The vision pipeline described above handles the flow of rendered
1120 images. Now, =CORTEX= needs simulated eyes to serve as the source
1121 of these images.
1123 An eye is described in blender in the same way as a joint. They
1124 are zero dimensional empty objects with no geometry whose local
1125 coordinate system determines the orientation of the resulting eye.
1126 All eyes are children of a parent node named "eyes" just as all
1127 joints have a parent named "joints". An eye binds to the nearest
1128 physical object with =bind-sense=.
1130 #+caption: Here, the camera is created based on metadata on the
1131 #+caption: eye-node and attached to the nearest physical object
1132 #+caption: with =bind-sense=
1133 #+name: add-eye
1134 #+begin_listing clojure
1135 (defn add-eye!
1136 "Create a Camera centered on the current position of 'eye which
1137 follows the closest physical node in 'creature. The camera will
1138 point in the X direction and use the Z vector as up as determined
1139 by the rotation of these vectors in blender coordinate space. Use
1140 XZY rotation for the node in blender."
1141 [#^Node creature #^Spatial eye]
1142 (let [target (closest-node creature eye)
1143 [cam-width cam-height]
1144 ;;[640 480] ;; graphics card on laptop doesn't support
1145 ;; arbitrary dimensions.
1146 (eye-dimensions eye)
1147 cam (Camera. cam-width cam-height)
1148 rot (.getWorldRotation eye)]
1149 (.setLocation cam (.getWorldTranslation eye))
1150 (.lookAtDirection
1151 cam ; this part is not a mistake and
1152 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1153 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1154 (.setFrustumPerspective
1155 cam (float 45)
1156 (float (/ (.getWidth cam) (.getHeight cam)))
1157 (float 1)
1158 (float 1000))
1159 (bind-sense target cam) cam))
1160 #+end_listing
1162 *** Simulated Retina
1164 An eye is a surface (the retina) which contains many discrete
1165 sensors to detect light. These sensors can have different
1166 light-sensing properties. In humans, each discrete sensor is
1167 sensitive to red, blue, green, or gray. These different types of
1168 sensors can have different spatial distributions along the retina.
1169 In humans, there is a fovea in the center of the retina which has
1170 a very high density of color sensors, and a blind spot which has
1171 no sensors at all. Sensor density decreases in proportion to
1172 distance from the fovea.
1174 I want to be able to model any retinal configuration, so my
1175 eye-nodes in blender contain metadata pointing to images that
1176 describe the precise position of the individual sensors using
1177 white pixels. The meta-data also describes the precise sensitivity
1178 to light that the sensors described in the image have. An eye can
1179 contain any number of these images. For example, the metadata for
1180 an eye might look like this:
1182 #+begin_src clojure
1183 {0xFF0000 "Models/test-creature/retina-small.png"}
1184 #+end_src
1186 #+caption: An example retinal profile image. White pixels are
1187 #+caption: photo-sensitive elements. The distribution of white
1188 #+caption: pixels is denser in the middle and falls off at the
1189 #+caption: edges and is inspired by the human retina.
1190 #+name: retina
1191 #+ATTR_LaTeX: :width 7cm
1192 [[./images/retina-small.png]]
1194 Together, the number 0xFF0000 and the image image above describe
1195 the placement of red-sensitive sensory elements.
1197 Meta-data to very crudely approximate a human eye might be
1198 something like this:
1200 #+begin_src clojure
1201 (let [retinal-profile "Models/test-creature/retina-small.png"]
1202 {0xFF0000 retinal-profile
1203 0x00FF00 retinal-profile
1204 0x0000FF retinal-profile
1205 0xFFFFFF retinal-profile})
1206 #+end_src
1208 The numbers that serve as keys in the map determine a sensor's
1209 relative sensitivity to the channels red, green, and blue. These
1210 sensitivity values are packed into an integer in the order
1211 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1212 image are added together with these sensitivities as linear
1213 weights. Therefore, 0xFF0000 means sensitive to red only while
1214 0xFFFFFF means sensitive to all colors equally (gray).
1216 #+caption: This is the core of vision in =CORTEX=. A given eye node
1217 #+caption: is converted into a function that returns visual
1218 #+caption: information from the simulation.
1219 #+name: vision-kernel
1220 #+begin_listing clojure
1221 #+BEGIN_SRC clojure
1222 (defn vision-kernel
1223 "Returns a list of functions, each of which will return a color
1224 channel's worth of visual information when called inside a running
1225 simulation."
1226 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1227 (let [retinal-map (retina-sensor-profile eye)
1228 camera (add-eye! creature eye)
1229 vision-image
1230 (atom
1231 (BufferedImage. (.getWidth camera)
1232 (.getHeight camera)
1233 BufferedImage/TYPE_BYTE_BINARY))
1234 register-eye!
1235 (runonce
1236 (fn [world]
1237 (add-camera!
1238 world camera
1239 (let [counter (atom 0)]
1240 (fn [r fb bb bi]
1241 (if (zero? (rem (swap! counter inc) (inc skip)))
1242 (reset! vision-image
1243 (BufferedImage! r fb bb bi))))))))]
1244 (vec
1245 (map
1246 (fn [[key image]]
1247 (let [whites (white-coordinates image)
1248 topology (vec (collapse whites))
1249 sensitivity (sensitivity-presets key key)]
1250 (attached-viewport.
1251 (fn [world]
1252 (register-eye! world)
1253 (vector
1254 topology
1255 (vec
1256 (for [[x y] whites]
1257 (pixel-sense
1258 sensitivity
1259 (.getRGB @vision-image x y))))))
1260 register-eye!)))
1261 retinal-map))))
1262 #+END_SRC
1263 #+end_listing
1265 Note that since each of the functions generated by =vision-kernel=
1266 shares the same =register-eye!= function, the eye will be
1267 registered only once the first time any of the functions from the
1268 list returned by =vision-kernel= is called. Each of the functions
1269 returned by =vision-kernel= also allows access to the =Viewport=
1270 through which it receives images.
1272 All the hard work has been done; all that remains is to apply
1273 =vision-kernel= to each eye in the creature and gather the results
1274 into one list of functions.
1277 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1278 #+caption: environment for experimenting with different types of
1279 #+caption: eyes.
1280 #+name: vision!
1281 #+begin_listing clojure
1282 #+BEGIN_SRC clojure
1283 (defn vision!
1284 "Returns a list of functions, each of which returns visual sensory
1285 data when called inside a running simulation."
1286 [#^Node creature & {skip :skip :or {skip 0}}]
1287 (reduce
1288 concat
1289 (for [eye (eyes creature)]
1290 (vision-kernel creature eye))))
1291 #+END_SRC
1292 #+end_listing
1294 #+caption: Simulated vision with a test creature and the
1295 #+caption: human-like eye approximation. Notice how each channel
1296 #+caption: of the eye responds differently to the differently
1297 #+caption: colored balls.
1298 #+name: worm-vision-test.
1299 #+ATTR_LaTeX: :width 13cm
1300 [[./images/worm-vision.png]]
1302 The vision code is not much more complicated than the body code,
1303 and enables multiple further paths for simulated vision. For
1304 example, it is quite easy to create bifocal vision -- you just
1305 make two eyes next to each other in blender! It is also possible
1306 to encode vision transforms in the retinal files. For example, the
1307 human like retina file in figure \ref{retina} approximates a
1308 log-polar transform.
1310 This vision code has already been absorbed by the jMonkeyEngine
1311 community and is now (in modified form) part of a system for
1312 capturing in-game video to a file.
1314 ** ...but hearing must be built from scratch
1316 At the end of this section I will have simulated ears that work the
1317 same way as the simulated eyes in the last section. I will be able to
1318 place any number of ear-nodes in a blender file, and they will bind to
1319 the closest physical object and follow it as it moves around. Each ear
1320 will provide access to the sound data it picks up between every frame.
1322 Hearing is one of the more difficult senses to simulate, because there
1323 is less support for obtaining the actual sound data that is processed
1324 by jMonkeyEngine3. There is no "split-screen" support for rendering
1325 sound from different points of view, and there is no way to directly
1326 access the rendered sound data.
1328 =CORTEX='s hearing is unique because it does not have any
1329 limitations compared to other simulation environments. As far as I
1330 know, there is no other system that supports multiple listeners,
1331 and the sound demo at the end of this section is the first time
1332 it's been done in a video game environment.
1334 *** Brief Description of jMonkeyEngine's Sound System
1336 jMonkeyEngine's sound system works as follows:
1338 - jMonkeyEngine uses the =AppSettings= for the particular
1339 application to determine what sort of =AudioRenderer= should be
1340 used.
1341 - Although some support is provided for multiple AudioRendering
1342 backends, jMonkeyEngine at the time of this writing will either
1343 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1344 - jMonkeyEngine tries to figure out what sort of system you're
1345 running and extracts the appropriate native libraries.
1346 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1347 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1348 - =OpenAL= renders the 3D sound and feeds the rendered sound
1349 directly to any of various sound output devices with which it
1350 knows how to communicate.
1352 A consequence of this is that there's no way to access the actual
1353 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1354 one /listener/ (it renders sound data from only one perspective),
1355 which normally isn't a problem for games, but becomes a problem
1356 when trying to make multiple AI creatures that can each hear the
1357 world from a different perspective.
1359 To make many AI creatures in jMonkeyEngine that can each hear the
1360 world from their own perspective, or to make a single creature with
1361 many ears, it is necessary to go all the way back to =OpenAL= and
1362 implement support for simulated hearing there.
1364 *** Extending =OpenAl=
1366 Extending =OpenAL= to support multiple listeners requires 500
1367 lines of =C= code and is too hairy to mention here. Instead, I
1368 will show a small amount of extension code and go over the high
1369 level strategy. Full source is of course available with the
1370 =CORTEX= distribution if you're interested.
1372 =OpenAL= goes to great lengths to support many different systems,
1373 all with different sound capabilities and interfaces. It
1374 accomplishes this difficult task by providing code for many
1375 different sound backends in pseudo-objects called /Devices/.
1376 There's a device for the Linux Open Sound System and the Advanced
1377 Linux Sound Architecture, there's one for Direct Sound on Windows,
1378 and there's even one for Solaris. =OpenAL= solves the problem of
1379 platform independence by providing all these Devices.
1381 Wrapper libraries such as LWJGL are free to examine the system on
1382 which they are running and then select an appropriate device for
1383 that system.
1385 There are also a few "special" devices that don't interface with
1386 any particular system. These include the Null Device, which
1387 doesn't do anything, and the Wave Device, which writes whatever
1388 sound it receives to a file, if everything has been set up
1389 correctly when configuring =OpenAL=.
1391 Actual mixing (Doppler shift and distance.environment-based
1392 attenuation) of the sound data happens in the Devices, and they
1393 are the only point in the sound rendering process where this data
1394 is available.
1396 Therefore, in order to support multiple listeners, and get the
1397 sound data in a form that the AIs can use, it is necessary to
1398 create a new Device which supports this feature.
1400 Adding a device to OpenAL is rather tricky -- there are five
1401 separate files in the =OpenAL= source tree that must be modified
1402 to do so. I named my device the "Multiple Audio Send" Device, or
1403 =Send= Device for short, since it sends audio data back to the
1404 calling application like an Aux-Send cable on a mixing board.
1406 The main idea behind the Send device is to take advantage of the
1407 fact that LWJGL only manages one /context/ when using OpenAL. A
1408 /context/ is like a container that holds samples and keeps track
1409 of where the listener is. In order to support multiple listeners,
1410 the Send device identifies the LWJGL context as the master
1411 context, and creates any number of slave contexts to represent
1412 additional listeners. Every time the device renders sound, it
1413 synchronizes every source from the master LWJGL context to the
1414 slave contexts. Then, it renders each context separately, using a
1415 different listener for each one. The rendered sound is made
1416 available via JNI to jMonkeyEngine.
1418 Switching between contexts is not the normal operation of a
1419 Device, and one of the problems with doing so is that a Device
1420 normally keeps around a few pieces of state such as the
1421 =ClickRemoval= array above which will become corrupted if the
1422 contexts are not rendered in parallel. The solution is to create a
1423 copy of this normally global device state for each context, and
1424 copy it back and forth into and out of the actual device state
1425 whenever a context is rendered.
1427 The core of the =Send= device is the =syncSources= function, which
1428 does the job of copying all relevant data from one context to
1429 another.
1431 #+caption: Program for extending =OpenAL= to support multiple
1432 #+caption: listeners via context copying/switching.
1433 #+name: sync-openal-sources
1434 #+begin_listing c
1435 #+BEGIN_SRC c
1436 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1437 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1438 ALuint master = masterSource->source;
1439 ALuint slave = slaveSource->source;
1440 ALCcontext *current = alcGetCurrentContext();
1442 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1443 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1444 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1445 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1446 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1447 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1448 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1449 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1450 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1451 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1452 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1453 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1454 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1456 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1457 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1458 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1460 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1461 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1463 alcMakeContextCurrent(masterCtx);
1464 ALint source_type;
1465 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1467 // Only static sources are currently synchronized!
1468 if (AL_STATIC == source_type){
1469 ALint master_buffer;
1470 ALint slave_buffer;
1471 alGetSourcei(master, AL_BUFFER, &master_buffer);
1472 alcMakeContextCurrent(slaveCtx);
1473 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1474 if (master_buffer != slave_buffer){
1475 alSourcei(slave, AL_BUFFER, master_buffer);
1479 // Synchronize the state of the two sources.
1480 alcMakeContextCurrent(masterCtx);
1481 ALint masterState;
1482 ALint slaveState;
1484 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1485 alcMakeContextCurrent(slaveCtx);
1486 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1488 if (masterState != slaveState){
1489 switch (masterState){
1490 case AL_INITIAL : alSourceRewind(slave); break;
1491 case AL_PLAYING : alSourcePlay(slave); break;
1492 case AL_PAUSED : alSourcePause(slave); break;
1493 case AL_STOPPED : alSourceStop(slave); break;
1496 // Restore whatever context was previously active.
1497 alcMakeContextCurrent(current);
1499 #+END_SRC
1500 #+end_listing
1502 With this special context-switching device, and some ugly JNI
1503 bindings that are not worth mentioning, =CORTEX= gains the ability
1504 to access multiple sound streams from =OpenAL=.
1506 #+caption: Program to create an ear from a blender empty node. The ear
1507 #+caption: follows around the nearest physical object and passes
1508 #+caption: all sensory data to a continuation function.
1509 #+name: add-ear
1510 #+begin_listing clojure
1511 #+BEGIN_SRC clojure
1512 (defn add-ear!
1513 "Create a Listener centered on the current position of 'ear
1514 which follows the closest physical node in 'creature and
1515 sends sound data to 'continuation."
1516 [#^Application world #^Node creature #^Spatial ear continuation]
1517 (let [target (closest-node creature ear)
1518 lis (Listener.)
1519 audio-renderer (.getAudioRenderer world)
1520 sp (hearing-pipeline continuation)]
1521 (.setLocation lis (.getWorldTranslation ear))
1522 (.setRotation lis (.getWorldRotation ear))
1523 (bind-sense target lis)
1524 (update-listener-velocity! target lis)
1525 (.addListener audio-renderer lis)
1526 (.registerSoundProcessor audio-renderer lis sp)))
1527 #+END_SRC
1528 #+end_listing
1530 The =Send= device, unlike most of the other devices in =OpenAL=,
1531 does not render sound unless asked. This enables the system to
1532 slow down or speed up depending on the needs of the AIs who are
1533 using it to listen. If the device tried to render samples in
1534 real-time, a complicated AI whose mind takes 100 seconds of
1535 computer time to simulate 1 second of AI-time would miss almost
1536 all of the sound in its environment!
1538 #+caption: Program to enable arbitrary hearing in =CORTEX=
1539 #+name: hearing
1540 #+begin_listing clojure
1541 #+BEGIN_SRC clojure
1542 (defn hearing-kernel
1543 "Returns a function which returns auditory sensory data when called
1544 inside a running simulation."
1545 [#^Node creature #^Spatial ear]
1546 (let [hearing-data (atom [])
1547 register-listener!
1548 (runonce
1549 (fn [#^Application world]
1550 (add-ear!
1551 world creature ear
1552 (comp #(reset! hearing-data %)
1553 byteBuffer->pulse-vector))))]
1554 (fn [#^Application world]
1555 (register-listener! world)
1556 (let [data @hearing-data
1557 topology
1558 (vec (map #(vector % 0) (range 0 (count data))))]
1559 [topology data]))))
1561 (defn hearing!
1562 "Endow the creature in a particular world with the sense of
1563 hearing. Will return a sequence of functions, one for each ear,
1564 which when called will return the auditory data from that ear."
1565 [#^Node creature]
1566 (for [ear (ears creature)]
1567 (hearing-kernel creature ear)))
1568 #+END_SRC
1569 #+end_listing
1571 Armed with these functions, =CORTEX= is able to test possibly the
1572 first ever instance of multiple listeners in a video game engine
1573 based simulation!
1575 #+caption: Here a simple creature responds to sound by changing
1576 #+caption: its color from gray to green when the total volume
1577 #+caption: goes over a threshold.
1578 #+name: sound-test
1579 #+begin_listing java
1580 #+BEGIN_SRC java
1581 /**
1582 * Respond to sound! This is the brain of an AI entity that
1583 * hears its surroundings and reacts to them.
1584 */
1585 public void process(ByteBuffer audioSamples,
1586 int numSamples, AudioFormat format) {
1587 audioSamples.clear();
1588 byte[] data = new byte[numSamples];
1589 float[] out = new float[numSamples];
1590 audioSamples.get(data);
1591 FloatSampleTools.
1592 byte2floatInterleaved
1593 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1595 float max = Float.NEGATIVE_INFINITY;
1596 for (float f : out){if (f > max) max = f;}
1597 audioSamples.clear();
1599 if (max > 0.1){
1600 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1602 else {
1603 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1605 #+END_SRC
1606 #+end_listing
1608 #+caption: First ever simulation of multiple listeners in =CORTEX=.
1609 #+caption: Each cube is a creature which processes sound data with
1610 #+caption: the =process= function from listing \ref{sound-test}.
1611 #+caption: the ball is constantly emitting a pure tone of
1612 #+caption: constant volume. As it approaches the cubes, they each
1613 #+caption: change color in response to the sound.
1614 #+name: sound-cubes.
1615 #+ATTR_LaTeX: :width 10cm
1616 [[./images/java-hearing-test.png]]
1618 This system of hearing has also been co-opted by the
1619 jMonkeyEngine3 community and is used to record audio for demo
1620 videos.
1622 ** Hundreds of hair-like elements provide a sense of touch
1624 Touch is critical to navigation and spatial reasoning and as such I
1625 need a simulated version of it to give to my AI creatures.
1627 Human skin has a wide array of touch sensors, each of which
1628 specialize in detecting different vibrational modes and pressures.
1629 These sensors can integrate a vast expanse of skin (i.e. your
1630 entire palm), or a tiny patch of skin at the tip of your finger.
1631 The hairs of the skin help detect objects before they even come
1632 into contact with the skin proper.
1634 However, touch in my simulated world can not exactly correspond to
1635 human touch because my creatures are made out of completely rigid
1636 segments that don't deform like human skin.
1638 Instead of measuring deformation or vibration, I surround each
1639 rigid part with a plenitude of hair-like objects (/feelers/) which
1640 do not interact with the physical world. Physical objects can pass
1641 through them with no effect. The feelers are able to tell when
1642 other objects pass through them, and they constantly report how
1643 much of their extent is covered. So even though the creature's body
1644 parts do not deform, the feelers create a margin around those body
1645 parts which achieves a sense of touch which is a hybrid between a
1646 human's sense of deformation and sense from hairs.
1648 Implementing touch in jMonkeyEngine follows a different technical
1649 route than vision and hearing. Those two senses piggybacked off
1650 jMonkeyEngine's 3D audio and video rendering subsystems. To
1651 simulate touch, I use jMonkeyEngine's physics system to execute
1652 many small collision detections, one for each feeler. The placement
1653 of the feelers is determined by a UV-mapped image which shows where
1654 each feeler should be on the 3D surface of the body.
1656 *** Defining Touch Meta-Data in Blender
1658 Each geometry can have a single UV map which describes the
1659 position of the feelers which will constitute its sense of touch.
1660 This image path is stored under the ``touch'' key. The image itself
1661 is black and white, with black meaning a feeler length of 0 (no
1662 feeler is present) and white meaning a feeler length of =scale=,
1663 which is a float stored under the key "scale".
1665 #+caption: Touch does not use empty nodes, to store metadata,
1666 #+caption: because the metadata of each solid part of a
1667 #+caption: creature's body is sufficient.
1668 #+name: touch-meta-data
1669 #+begin_listing clojure
1670 #+BEGIN_SRC clojure
1671 (defn tactile-sensor-profile
1672 "Return the touch-sensor distribution image in BufferedImage format,
1673 or nil if it does not exist."
1674 [#^Geometry obj]
1675 (if-let [image-path (meta-data obj "touch")]
1676 (load-image image-path)))
1678 (defn tactile-scale
1679 "Return the length of each feeler. Default scale is 0.01
1680 jMonkeyEngine units."
1681 [#^Geometry obj]
1682 (if-let [scale (meta-data obj "scale")]
1683 scale 0.1))
1684 #+END_SRC
1685 #+end_listing
1687 Here is an example of a UV-map which specifies the position of
1688 touch sensors along the surface of the upper segment of a fingertip.
1690 #+caption: This is the tactile-sensor-profile for the upper segment
1691 #+caption: of a fingertip. It defines regions of high touch sensitivity
1692 #+caption: (where there are many white pixels) and regions of low
1693 #+caption: sensitivity (where white pixels are sparse).
1694 #+name: fingertip-UV
1695 #+ATTR_LaTeX: :width 13cm
1696 [[./images/finger-UV.png]]
1698 *** Implementation Summary
1700 To simulate touch there are three conceptual steps. For each solid
1701 object in the creature, you first have to get UV image and scale
1702 parameter which define the position and length of the feelers.
1703 Then, you use the triangles which comprise the mesh and the UV
1704 data stored in the mesh to determine the world-space position and
1705 orientation of each feeler. Then once every frame, update these
1706 positions and orientations to match the current position and
1707 orientation of the object, and use physics collision detection to
1708 gather tactile data.
1710 Extracting the meta-data has already been described. The third
1711 step, physics collision detection, is handled in =touch-kernel=.
1712 Translating the positions and orientations of the feelers from the
1713 UV-map to world-space is itself a three-step process.
1715 - Find the triangles which make up the mesh in pixel-space and in
1716 world-space. \\(=triangles=, =pixel-triangles=).
1718 - Find the coordinates of each feeler in world-space. These are
1719 the origins of the feelers. (=feeler-origins=).
1721 - Calculate the normals of the triangles in world space, and add
1722 them to each of the origins of the feelers. These are the
1723 normalized coordinates of the tips of the feelers.
1724 (=feeler-tips=).
1726 *** Triangle Math
1728 The rigid objects which make up a creature have an underlying
1729 =Geometry=, which is a =Mesh= plus a =Material= and other
1730 important data involved with displaying the object.
1732 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1733 vertices which have coordinates in world space and UV space.
1735 Here, =triangles= gets all the world-space triangles which
1736 comprise a mesh, while =pixel-triangles= gets those same triangles
1737 expressed in pixel coordinates (which are UV coordinates scaled to
1738 fit the height and width of the UV image).
1740 #+caption: Programs to extract triangles from a geometry and get
1741 #+caption: their vertices in both world and UV-coordinates.
1742 #+name: get-triangles
1743 #+begin_listing clojure
1744 #+BEGIN_SRC clojure
1745 (defn triangle
1746 "Get the triangle specified by triangle-index from the mesh."
1747 [#^Geometry geo triangle-index]
1748 (triangle-seq
1749 (let [scratch (Triangle.)]
1750 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1752 (defn triangles
1753 "Return a sequence of all the Triangles which comprise a given
1754 Geometry."
1755 [#^Geometry geo]
1756 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1758 (defn triangle-vertex-indices
1759 "Get the triangle vertex indices of a given triangle from a given
1760 mesh."
1761 [#^Mesh mesh triangle-index]
1762 (let [indices (int-array 3)]
1763 (.getTriangle mesh triangle-index indices)
1764 (vec indices)))
1766 (defn vertex-UV-coord
1767 "Get the UV-coordinates of the vertex named by vertex-index"
1768 [#^Mesh mesh vertex-index]
1769 (let [UV-buffer
1770 (.getData
1771 (.getBuffer
1772 mesh
1773 VertexBuffer$Type/TexCoord))]
1774 [(.get UV-buffer (* vertex-index 2))
1775 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1777 (defn pixel-triangle [#^Geometry geo image index]
1778 (let [mesh (.getMesh geo)
1779 width (.getWidth image)
1780 height (.getHeight image)]
1781 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1782 (map (partial vertex-UV-coord mesh)
1783 (triangle-vertex-indices mesh index))))))
1785 (defn pixel-triangles
1786 "The pixel-space triangles of the Geometry, in the same order as
1787 (triangles geo)"
1788 [#^Geometry geo image]
1789 (let [height (.getHeight image)
1790 width (.getWidth image)]
1791 (map (partial pixel-triangle geo image)
1792 (range (.getTriangleCount (.getMesh geo))))))
1793 #+END_SRC
1794 #+end_listing
1796 *** The Affine Transform from one Triangle to Another
1798 =pixel-triangles= gives us the mesh triangles expressed in pixel
1799 coordinates and =triangles= gives us the mesh triangles expressed
1800 in world coordinates. The tactile-sensor-profile gives the
1801 position of each feeler in pixel-space. In order to convert
1802 pixel-space coordinates into world-space coordinates we need
1803 something that takes coordinates on the surface of one triangle
1804 and gives the corresponding coordinates on the surface of another
1805 triangle.
1807 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1808 into any other by a combination of translation, scaling, and
1809 rotation. The affine transformation from one triangle to another
1810 is readily computable if the triangle is expressed in terms of a
1811 $4x4$ matrix.
1813 #+BEGIN_LaTeX
1814 $$
1815 \begin{bmatrix}
1816 x_1 & x_2 & x_3 & n_x \\
1817 y_1 & y_2 & y_3 & n_y \\
1818 z_1 & z_2 & z_3 & n_z \\
1819 1 & 1 & 1 & 1
1820 \end{bmatrix}
1821 $$
1822 #+END_LaTeX
1824 Here, the first three columns of the matrix are the vertices of
1825 the triangle. The last column is the right-handed unit normal of
1826 the triangle.
1828 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1829 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1830 is $T_{2}T_{1}^{-1}$.
1832 The clojure code below recapitulates the formulas above, using
1833 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1834 transformation.
1836 #+caption: Program to interpret triangles as affine transforms.
1837 #+name: triangle-affine
1838 #+begin_listing clojure
1839 #+BEGIN_SRC clojure
1840 (defn triangle->matrix4f
1841 "Converts the triangle into a 4x4 matrix: The first three columns
1842 contain the vertices of the triangle; the last contains the unit
1843 normal of the triangle. The bottom row is filled with 1s."
1844 [#^Triangle t]
1845 (let [mat (Matrix4f.)
1846 [vert-1 vert-2 vert-3]
1847 (mapv #(.get t %) (range 3))
1848 unit-normal (do (.calculateNormal t)(.getNormal t))
1849 vertices [vert-1 vert-2 vert-3 unit-normal]]
1850 (dorun
1851 (for [row (range 4) col (range 3)]
1852 (do
1853 (.set mat col row (.get (vertices row) col))
1854 (.set mat 3 row 1)))) mat))
1856 (defn triangles->affine-transform
1857 "Returns the affine transformation that converts each vertex in the
1858 first triangle into the corresponding vertex in the second
1859 triangle."
1860 [#^Triangle tri-1 #^Triangle tri-2]
1861 (.mult
1862 (triangle->matrix4f tri-2)
1863 (.invert (triangle->matrix4f tri-1))))
1864 #+END_SRC
1865 #+end_listing
1867 *** Triangle Boundaries
1869 For efficiency's sake I will divide the tactile-profile image into
1870 small squares which inscribe each pixel-triangle, then extract the
1871 points which lie inside the triangle and map them to 3D-space using
1872 =triangle-transform= above. To do this I need a function,
1873 =convex-bounds= which finds the smallest box which inscribes a 2D
1874 triangle.
1876 =inside-triangle?= determines whether a point is inside a triangle
1877 in 2D pixel-space.
1879 #+caption: Program to efficiently determine point inclusion
1880 #+caption: in a triangle.
1881 #+name: in-triangle
1882 #+begin_listing clojure
1883 #+BEGIN_SRC clojure
1884 (defn convex-bounds
1885 "Returns the smallest square containing the given vertices, as a
1886 vector of integers [left top width height]."
1887 [verts]
1888 (let [xs (map first verts)
1889 ys (map second verts)
1890 x0 (Math/floor (apply min xs))
1891 y0 (Math/floor (apply min ys))
1892 x1 (Math/ceil (apply max xs))
1893 y1 (Math/ceil (apply max ys))]
1894 [x0 y0 (- x1 x0) (- y1 y0)]))
1896 (defn same-side?
1897 "Given the points p1 and p2 and the reference point ref, is point p
1898 on the same side of the line that goes through p1 and p2 as ref is?"
1899 [p1 p2 ref p]
1900 (<=
1902 (.dot
1903 (.cross (.subtract p2 p1) (.subtract p p1))
1904 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1906 (defn inside-triangle?
1907 "Is the point inside the triangle?"
1908 {:author "Dylan Holmes"}
1909 [#^Triangle tri #^Vector3f p]
1910 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1911 (and
1912 (same-side? vert-1 vert-2 vert-3 p)
1913 (same-side? vert-2 vert-3 vert-1 p)
1914 (same-side? vert-3 vert-1 vert-2 p))))
1915 #+END_SRC
1916 #+end_listing
1918 *** Feeler Coordinates
1920 The triangle-related functions above make short work of
1921 calculating the positions and orientations of each feeler in
1922 world-space.
1924 #+caption: Program to get the coordinates of ``feelers '' in
1925 #+caption: both world and UV-coordinates.
1926 #+name: feeler-coordinates
1927 #+begin_listing clojure
1928 #+BEGIN_SRC clojure
1929 (defn feeler-pixel-coords
1930 "Returns the coordinates of the feelers in pixel space in lists, one
1931 list for each triangle, ordered in the same way as (triangles) and
1932 (pixel-triangles)."
1933 [#^Geometry geo image]
1934 (map
1935 (fn [pixel-triangle]
1936 (filter
1937 (fn [coord]
1938 (inside-triangle? (->triangle pixel-triangle)
1939 (->vector3f coord)))
1940 (white-coordinates image (convex-bounds pixel-triangle))))
1941 (pixel-triangles geo image)))
1943 (defn feeler-world-coords
1944 "Returns the coordinates of the feelers in world space in lists, one
1945 list for each triangle, ordered in the same way as (triangles) and
1946 (pixel-triangles)."
1947 [#^Geometry geo image]
1948 (let [transforms
1949 (map #(triangles->affine-transform
1950 (->triangle %1) (->triangle %2))
1951 (pixel-triangles geo image)
1952 (triangles geo))]
1953 (map (fn [transform coords]
1954 (map #(.mult transform (->vector3f %)) coords))
1955 transforms (feeler-pixel-coords geo image))))
1956 #+END_SRC
1957 #+end_listing
1959 #+caption: Program to get the position of the base and tip of
1960 #+caption: each ``feeler''
1961 #+name: feeler-tips
1962 #+begin_listing clojure
1963 #+BEGIN_SRC clojure
1964 (defn feeler-origins
1965 "The world space coordinates of the root of each feeler."
1966 [#^Geometry geo image]
1967 (reduce concat (feeler-world-coords geo image)))
1969 (defn feeler-tips
1970 "The world space coordinates of the tip of each feeler."
1971 [#^Geometry geo image]
1972 (let [world-coords (feeler-world-coords geo image)
1973 normals
1974 (map
1975 (fn [triangle]
1976 (.calculateNormal triangle)
1977 (.clone (.getNormal triangle)))
1978 (map ->triangle (triangles geo)))]
1980 (mapcat (fn [origins normal]
1981 (map #(.add % normal) origins))
1982 world-coords normals)))
1984 (defn touch-topology
1985 [#^Geometry geo image]
1986 (collapse (reduce concat (feeler-pixel-coords geo image))))
1987 #+END_SRC
1988 #+end_listing
1990 *** Simulated Touch
1992 Now that the functions to construct feelers are complete,
1993 =touch-kernel= generates functions to be called from within a
1994 simulation that perform the necessary physics collisions to
1995 collect tactile data, and =touch!= recursively applies it to every
1996 node in the creature.
1998 #+caption: Efficient program to transform a ray from
1999 #+caption: one position to another.
2000 #+name: set-ray
2001 #+begin_listing clojure
2002 #+BEGIN_SRC clojure
2003 (defn set-ray [#^Ray ray #^Matrix4f transform
2004 #^Vector3f origin #^Vector3f tip]
2005 ;; Doing everything locally reduces garbage collection by enough to
2006 ;; be worth it.
2007 (.mult transform origin (.getOrigin ray))
2008 (.mult transform tip (.getDirection ray))
2009 (.subtractLocal (.getDirection ray) (.getOrigin ray))
2010 (.normalizeLocal (.getDirection ray)))
2011 #+END_SRC
2012 #+end_listing
2014 #+caption: This is the core of touch in =CORTEX= each feeler
2015 #+caption: follows the object it is bound to, reporting any
2016 #+caption: collisions that may happen.
2017 #+name: touch-kernel
2018 #+begin_listing clojure
2019 #+BEGIN_SRC clojure
2020 (defn touch-kernel
2021 "Constructs a function which will return tactile sensory data from
2022 'geo when called from inside a running simulation"
2023 [#^Geometry geo]
2024 (if-let
2025 [profile (tactile-sensor-profile geo)]
2026 (let [ray-reference-origins (feeler-origins geo profile)
2027 ray-reference-tips (feeler-tips geo profile)
2028 ray-length (tactile-scale geo)
2029 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
2030 topology (touch-topology geo profile)
2031 correction (float (* ray-length -0.2))]
2032 ;; slight tolerance for very close collisions.
2033 (dorun
2034 (map (fn [origin tip]
2035 (.addLocal origin (.mult (.subtract tip origin)
2036 correction)))
2037 ray-reference-origins ray-reference-tips))
2038 (dorun (map #(.setLimit % ray-length) current-rays))
2039 (fn [node]
2040 (let [transform (.getWorldMatrix geo)]
2041 (dorun
2042 (map (fn [ray ref-origin ref-tip]
2043 (set-ray ray transform ref-origin ref-tip))
2044 current-rays ray-reference-origins
2045 ray-reference-tips))
2046 (vector
2047 topology
2048 (vec
2049 (for [ray current-rays]
2050 (do
2051 (let [results (CollisionResults.)]
2052 (.collideWith node ray results)
2053 (let [touch-objects
2054 (filter #(not (= geo (.getGeometry %)))
2055 results)
2056 limit (.getLimit ray)]
2057 [(if (empty? touch-objects)
2058 limit
2059 (let [response
2060 (apply min (map #(.getDistance %)
2061 touch-objects))]
2062 (FastMath/clamp
2063 (float
2064 (if (> response limit) (float 0.0)
2065 (+ response correction)))
2066 (float 0.0)
2067 limit)))
2068 limit])))))))))))
2069 #+END_SRC
2070 #+end_listing
2072 Armed with the =touch!= function, =CORTEX= becomes capable of
2073 giving creatures a sense of touch. A simple test is to create a
2074 cube that is outfitted with a uniform distribution of touch
2075 sensors. It can feel the ground and any balls that it touches.
2077 #+caption: =CORTEX= interface for creating touch in a simulated
2078 #+caption: creature.
2079 #+name: touch
2080 #+begin_listing clojure
2081 #+BEGIN_SRC clojure
2082 (defn touch!
2083 "Endow the creature with the sense of touch. Returns a sequence of
2084 functions, one for each body part with a tactile-sensor-profile,
2085 each of which when called returns sensory data for that body part."
2086 [#^Node creature]
2087 (filter
2088 (comp not nil?)
2089 (map touch-kernel
2090 (filter #(isa? (class %) Geometry)
2091 (node-seq creature)))))
2092 #+END_SRC
2093 #+end_listing
2095 The tactile-sensor-profile image for the touch cube is a simple
2096 cross with a uniform distribution of touch sensors:
2098 #+caption: The touch profile for the touch-cube. Each pure white
2099 #+caption: pixel defines a touch sensitive feeler.
2100 #+name: touch-cube-uv-map
2101 #+ATTR_LaTeX: :width 7cm
2102 [[./images/touch-profile.png]]
2104 #+caption: The touch cube reacts to cannonballs. The black, red,
2105 #+caption: and white cross on the right is a visual display of
2106 #+caption: the creature's touch. White means that it is feeling
2107 #+caption: something strongly, black is not feeling anything,
2108 #+caption: and gray is in-between. The cube can feel both the
2109 #+caption: floor and the ball. Notice that when the ball causes
2110 #+caption: the cube to tip, that the bottom face can still feel
2111 #+caption: part of the ground.
2112 #+name: touch-cube-uv-map-2
2113 #+ATTR_LaTeX: :width 15cm
2114 [[./images/touch-cube.png]]
2116 ** Proprioception provides knowledge of your own body's position
2118 Close your eyes, and touch your nose with your right index finger.
2119 How did you do it? You could not see your hand, and neither your
2120 hand nor your nose could use the sense of touch to guide the path
2121 of your hand. There are no sound cues, and Taste and Smell
2122 certainly don't provide any help. You know where your hand is
2123 without your other senses because of Proprioception.
2125 Humans can sometimes loose this sense through viral infections or
2126 damage to the spinal cord or brain, and when they do, they loose
2127 the ability to control their own bodies without looking directly at
2128 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2129 Hat]] (\cite{man-wife-hat}), a woman named Christina looses this
2130 sense and has to learn how to move by carefully watching her arms
2131 and legs. She describes proprioception as the "eyes of the body,
2132 the way the body sees itself".
2134 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2135 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2136 positions of each body part by monitoring muscle strain and length.
2138 It's clear that this is a vital sense for fluid, graceful movement.
2139 It's also particularly easy to implement in jMonkeyEngine.
2141 My simulated proprioception calculates the relative angles of each
2142 joint from the rest position defined in the blender file. This
2143 simulates the muscle-spindles and joint capsules. I will deal with
2144 Golgi tendon organs, which calculate muscle strain, in the next
2145 section.
2147 *** Helper functions
2149 =absolute-angle= calculates the angle between two vectors,
2150 relative to a third axis vector. This angle is the number of
2151 radians you have to move counterclockwise around the axis vector
2152 to get from the first to the second vector. It is not commutative
2153 like a normal dot-product angle is.
2155 The purpose of these functions is to build a system of angle
2156 measurement that is biologically plausible.
2158 #+caption: Program to measure angles along a vector
2159 #+name: helpers
2160 #+begin_listing clojure
2161 #+BEGIN_SRC clojure
2162 (defn right-handed?
2163 "true iff the three vectors form a right handed coordinate
2164 system. The three vectors do not have to be normalized or
2165 orthogonal."
2166 [vec1 vec2 vec3]
2167 (pos? (.dot (.cross vec1 vec2) vec3)))
2169 (defn absolute-angle
2170 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2171 [0 (* 2 Math/PI)]."
2172 [vec1 vec2 axis]
2173 (let [angle (.angleBetween vec1 vec2)]
2174 (if (right-handed? vec1 vec2 axis)
2175 angle (- (* 2 Math/PI) angle))))
2176 #+END_SRC
2177 #+end_listing
2179 *** Proprioception Kernel
2181 Given a joint, =proprioception-kernel= produces a function that
2182 calculates the Euler angles between the the objects the joint
2183 connects. The only tricky part here is making the angles relative
2184 to the joint's initial ``straightness''.
2186 #+caption: Program to return biologically reasonable proprioceptive
2187 #+caption: data for each joint.
2188 #+name: proprioception
2189 #+begin_listing clojure
2190 #+BEGIN_SRC clojure
2191 (defn proprioception-kernel
2192 "Returns a function which returns proprioceptive sensory data when
2193 called inside a running simulation."
2194 [#^Node parts #^Node joint]
2195 (let [[obj-a obj-b] (joint-targets parts joint)
2196 joint-rot (.getWorldRotation joint)
2197 x0 (.mult joint-rot Vector3f/UNIT_X)
2198 y0 (.mult joint-rot Vector3f/UNIT_Y)
2199 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2200 (fn []
2201 (let [rot-a (.clone (.getWorldRotation obj-a))
2202 rot-b (.clone (.getWorldRotation obj-b))
2203 x (.mult rot-a x0)
2204 y (.mult rot-a y0)
2205 z (.mult rot-a z0)
2207 X (.mult rot-b x0)
2208 Y (.mult rot-b y0)
2209 Z (.mult rot-b z0)
2210 heading (Math/atan2 (.dot X z) (.dot X x))
2211 pitch (Math/atan2 (.dot X y) (.dot X x))
2213 ;; rotate x-vector back to origin
2214 reverse
2215 (doto (Quaternion.)
2216 (.fromAngleAxis
2217 (.angleBetween X x)
2218 (let [cross (.normalize (.cross X x))]
2219 (if (= 0 (.length cross)) y cross))))
2220 roll (absolute-angle (.mult reverse Y) y x)]
2221 [heading pitch roll]))))
2223 (defn proprioception!
2224 "Endow the creature with the sense of proprioception. Returns a
2225 sequence of functions, one for each child of the \"joints\" node in
2226 the creature, which each report proprioceptive information about
2227 that joint."
2228 [#^Node creature]
2229 ;; extract the body's joints
2230 (let [senses (map (partial proprioception-kernel creature)
2231 (joints creature))]
2232 (fn []
2233 (map #(%) senses))))
2234 #+END_SRC
2235 #+end_listing
2237 =proprioception!= maps =proprioception-kernel= across all the
2238 joints of the creature. It uses the same list of joints that
2239 =joints= uses. Proprioception is the easiest sense to implement in
2240 =CORTEX=, and it will play a crucial role when efficiently
2241 implementing empathy.
2243 #+caption: In the upper right corner, the three proprioceptive
2244 #+caption: angle measurements are displayed. Red is yaw, Green is
2245 #+caption: pitch, and White is roll.
2246 #+name: proprio
2247 #+ATTR_LaTeX: :width 11cm
2248 [[./images/proprio.png]]
2250 ** Muscles contain both sensors and effectors
2252 Surprisingly enough, terrestrial creatures only move by using
2253 torque applied about their joints. There's not a single straight
2254 line of force in the human body at all! (A straight line of force
2255 would correspond to some sort of jet or rocket propulsion.)
2257 In humans, muscles are composed of muscle fibers which can contract
2258 to exert force. The muscle fibers which compose a muscle are
2259 partitioned into discrete groups which are each controlled by a
2260 single alpha motor neuron. A single alpha motor neuron might
2261 control as little as three or as many as one thousand muscle
2262 fibers. When the alpha motor neuron is engaged by the spinal cord,
2263 it activates all of the muscle fibers to which it is attached. The
2264 spinal cord generally engages the alpha motor neurons which control
2265 few muscle fibers before the motor neurons which control many
2266 muscle fibers. This recruitment strategy allows for precise
2267 movements at low strength. The collection of all motor neurons that
2268 control a muscle is called the motor pool. The brain essentially
2269 says "activate 30% of the motor pool" and the spinal cord recruits
2270 motor neurons until 30% are activated. Since the distribution of
2271 power among motor neurons is unequal and recruitment goes from
2272 weakest to strongest, the first 30% of the motor pool might be 5%
2273 of the strength of the muscle.
2275 My simulated muscles follow a similar design: Each muscle is
2276 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2277 the array represents a motor neuron which controls a number of
2278 muscle fibers equal to the value of the entry. Each muscle has a
2279 scalar strength factor which determines the total force the muscle
2280 can exert when all motor neurons are activated. The effector
2281 function for a muscle takes a number to index into the motor pool,
2282 and then "activates" all the motor neurons whose index is lower or
2283 equal to the number. Each motor-neuron will apply force in
2284 proportion to its value in the array. Lower values cause less
2285 force. The lower values can be put at the "beginning" of the 1-D
2286 array to simulate the layout of actual human muscles, which are
2287 capable of more precise movements when exerting less force. Or, the
2288 motor pool can simulate more exotic recruitment strategies which do
2289 not correspond to human muscles.
2291 This 1D array is defined in an image file for ease of
2292 creation/visualization. Here is an example muscle profile image.
2294 #+caption: A muscle profile image that describes the strengths
2295 #+caption: of each motor neuron in a muscle. White is weakest
2296 #+caption: and dark red is strongest. This particular pattern
2297 #+caption: has weaker motor neurons at the beginning, just
2298 #+caption: like human muscle.
2299 #+name: muscle-recruit
2300 #+ATTR_LaTeX: :width 7cm
2301 [[./images/basic-muscle.png]]
2303 *** Muscle meta-data
2305 #+caption: Program to deal with loading muscle data from a blender
2306 #+caption: file's metadata.
2307 #+name: motor-pool
2308 #+begin_listing clojure
2309 #+BEGIN_SRC clojure
2310 (defn muscle-profile-image
2311 "Get the muscle-profile image from the node's blender meta-data."
2312 [#^Node muscle]
2313 (if-let [image (meta-data muscle "muscle")]
2314 (load-image image)))
2316 (defn muscle-strength
2317 "Return the strength of this muscle, or 1 if it is not defined."
2318 [#^Node muscle]
2319 (if-let [strength (meta-data muscle "strength")]
2320 strength 1))
2322 (defn motor-pool
2323 "Return a vector where each entry is the strength of the \"motor
2324 neuron\" at that part in the muscle."
2325 [#^Node muscle]
2326 (let [profile (muscle-profile-image muscle)]
2327 (vec
2328 (let [width (.getWidth profile)]
2329 (for [x (range width)]
2330 (- 255
2331 (bit-and
2332 0x0000FF
2333 (.getRGB profile x 0))))))))
2334 #+END_SRC
2335 #+end_listing
2337 Of note here is =motor-pool= which interprets the muscle-profile
2338 image in a way that allows me to use gradients between white and
2339 red, instead of shades of gray as I've been using for all the
2340 other senses. This is purely an aesthetic touch.
2342 *** Creating muscles
2344 #+caption: This is the core movement function in =CORTEX=, which
2345 #+caption: implements muscles that report on their activation.
2346 #+name: muscle-kernel
2347 #+begin_listing clojure
2348 #+BEGIN_SRC clojure
2349 (defn movement-kernel
2350 "Returns a function which when called with a integer value inside a
2351 running simulation will cause movement in the creature according
2352 to the muscle's position and strength profile. Each function
2353 returns the amount of force applied / max force."
2354 [#^Node creature #^Node muscle]
2355 (let [target (closest-node creature muscle)
2356 axis
2357 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2358 strength (muscle-strength muscle)
2360 pool (motor-pool muscle)
2361 pool-integral (reductions + pool)
2362 forces
2363 (vec (map #(float (* strength (/ % (last pool-integral))))
2364 pool-integral))
2365 control (.getControl target RigidBodyControl)]
2366 ;;(println-repl (.getName target) axis)
2367 (fn [n]
2368 (let [pool-index (max 0 (min n (dec (count pool))))
2369 force (forces pool-index)]
2370 (.applyTorque control (.mult axis force))
2371 (float (/ force strength))))))
2373 (defn movement!
2374 "Endow the creature with the power of movement. Returns a sequence
2375 of functions, each of which accept an integer value and will
2376 activate their corresponding muscle."
2377 [#^Node creature]
2378 (for [muscle (muscles creature)]
2379 (movement-kernel creature muscle)))
2380 #+END_SRC
2381 #+end_listing
2384 =movement-kernel= creates a function that controls the movement
2385 of the nearest physical node to the muscle node. The muscle exerts
2386 a rotational force dependent on it's orientation to the object in
2387 the blender file. The function returned by =movement-kernel= is
2388 also a sense function: it returns the percent of the total muscle
2389 strength that is currently being employed. This is analogous to
2390 muscle tension in humans and completes the sense of proprioception
2391 begun in the last section.
2393 ** =CORTEX= brings complex creatures to life!
2395 The ultimate test of =CORTEX= is to create a creature with the full
2396 gamut of senses and put it though its paces.
2398 With all senses enabled, my right hand model looks like an
2399 intricate marionette hand with several strings for each finger:
2401 #+caption: View of the hand model with all sense nodes. You can see
2402 #+caption: the joint, muscle, ear, and eye nodes here.
2403 #+name: hand-nodes-1
2404 #+ATTR_LaTeX: :width 11cm
2405 [[./images/hand-with-all-senses2.png]]
2407 #+caption: An alternate view of the hand.
2408 #+name: hand-nodes-2
2409 #+ATTR_LaTeX: :width 15cm
2410 [[./images/hand-with-all-senses3.png]]
2412 With the hand fully rigged with senses, I can run it though a test
2413 that will test everything.
2415 #+caption: A full test of the hand with all senses. Note especially
2416 #+caption: the interactions the hand has with itself: it feels
2417 #+caption: its own palm and fingers, and when it curls its fingers,
2418 #+caption: it sees them with its eye (which is located in the center
2419 #+caption: of the palm. The red block appears with a pure tone sound.
2420 #+caption: The hand then uses its muscles to launch the cube!
2421 #+name: integration
2422 #+ATTR_LaTeX: :width 16cm
2423 [[./images/integration.png]]
2425 ** =CORTEX= enables many possibilities for further research
2427 Often times, the hardest part of building a system involving
2428 creatures is dealing with physics and graphics. =CORTEX= removes
2429 much of this initial difficulty and leaves researchers free to
2430 directly pursue their ideas. I hope that even undergrads with a
2431 passing curiosity about simulated touch or creature evolution will
2432 be able to use cortex for experimentation. =CORTEX= is a completely
2433 simulated world, and far from being a disadvantage, its simulated
2434 nature enables you to create senses and creatures that would be
2435 impossible to make in the real world.
2437 While not by any means a complete list, here are some paths
2438 =CORTEX= is well suited to help you explore:
2440 - Empathy :: my empathy program leaves many areas for
2441 improvement, among which are using vision to infer
2442 proprioception and looking up sensory experience with imagined
2443 vision, touch, and sound.
2444 - Evolution :: Karl Sims created a rich environment for
2445 simulating the evolution of creatures on a connection
2446 machine. Today, this can be redone and expanded with =CORTEX=
2447 on an ordinary computer.
2448 - Exotic senses :: Cortex enables many fascinating senses that are
2449 not possible to build in the real world. For example,
2450 telekinesis is an interesting avenue to explore. You can also
2451 make a ``semantic'' sense which looks up metadata tags on
2452 objects in the environment the metadata tags might contain
2453 other sensory information.
2454 - Imagination via subworlds :: this would involve a creature with
2455 an effector which creates an entire new sub-simulation where
2456 the creature has direct control over placement/creation of
2457 objects via simulated telekinesis. The creature observes this
2458 sub-world through it's normal senses and uses its observations
2459 to make predictions about its top level world.
2460 - Simulated prescience :: step the simulation forward a few ticks,
2461 gather sensory data, then supply this data for the creature as
2462 one of its actual senses. The cost of prescience is slowing
2463 the simulation down by a factor proportional to however far
2464 you want the entities to see into the future. What happens
2465 when two evolved creatures that can each see into the future
2466 fight each other?
2467 - Swarm creatures :: Program a group of creatures that cooperate
2468 with each other. Because the creatures would be simulated, you
2469 could investigate computationally complex rules of behavior
2470 which still, from the group's point of view, would happen in
2471 ``real time''. Interactions could be as simple as cellular
2472 organisms communicating via flashing lights, or as complex as
2473 humanoids completing social tasks, etc.
2474 - =HACKER= for writing muscle-control programs :: Presented with
2475 low-level muscle control/ sense API, generate higher level
2476 programs for accomplishing various stated goals. Example goals
2477 might be "extend all your fingers" or "move your hand into the
2478 area with blue light" or "decrease the angle of this joint".
2479 It would be like Sussman's HACKER, except it would operate
2480 with much more data in a more realistic world. Start off with
2481 "calisthenics" to develop subroutines over the motor control
2482 API. This would be the "spinal chord" of a more intelligent
2483 creature. The low level programming code might be a turning
2484 machine that could develop programs to iterate over a "tape"
2485 where each entry in the tape could control recruitment of the
2486 fibers in a muscle.
2487 - Sense fusion :: There is much work to be done on sense
2488 integration -- building up a coherent picture of the world and
2489 the things in it with =CORTEX= as a base, you can explore
2490 concepts like self-organizing maps or cross modal clustering
2491 in ways that have never before been tried.
2492 - Inverse kinematics :: experiments in sense guided motor control
2493 are easy given =CORTEX='s support -- you can get right to the
2494 hard control problems without worrying about physics or
2495 senses.
2497 \newpage
2499 * =EMPATH=: action recognition in a simulated worm
2501 Here I develop a computational model of empathy, using =CORTEX= as a
2502 base. Empathy in this context is the ability to observe another
2503 creature and infer what sorts of sensations that creature is
2504 feeling. My empathy algorithm involves multiple phases. First is
2505 free-play, where the creature moves around and gains sensory
2506 experience. From this experience I construct a representation of the
2507 creature's sensory state space, which I call \Phi-space. Using
2508 \Phi-space, I construct an efficient function which takes the
2509 limited data that comes from observing another creature and enriches
2510 it with a full compliment of imagined sensory data. I can then use
2511 the imagined sensory data to recognize what the observed creature is
2512 doing and feeling, using straightforward embodied action predicates.
2513 This is all demonstrated with using a simple worm-like creature, and
2514 recognizing worm-actions based on limited data.
2516 #+caption: Here is the worm with which we will be working.
2517 #+caption: It is composed of 5 segments. Each segment has a
2518 #+caption: pair of extensor and flexor muscles. Each of the
2519 #+caption: worm's four joints is a hinge joint which allows
2520 #+caption: about 30 degrees of rotation to either side. Each segment
2521 #+caption: of the worm is touch-capable and has a uniform
2522 #+caption: distribution of touch sensors on each of its faces.
2523 #+caption: Each joint has a proprioceptive sense to detect
2524 #+caption: relative positions. The worm segments are all the
2525 #+caption: same except for the first one, which has a much
2526 #+caption: higher weight than the others to allow for easy
2527 #+caption: manual motor control.
2528 #+name: basic-worm-view
2529 #+ATTR_LaTeX: :width 10cm
2530 [[./images/basic-worm-view.png]]
2532 #+caption: Program for reading a worm from a blender file and
2533 #+caption: outfitting it with the senses of proprioception,
2534 #+caption: touch, and the ability to move, as specified in the
2535 #+caption: blender file.
2536 #+name: get-worm
2537 #+begin_listing clojure
2538 #+begin_src clojure
2539 (defn worm []
2540 (let [model (load-blender-model "Models/worm/worm.blend")]
2541 {:body (doto model (body!))
2542 :touch (touch! model)
2543 :proprioception (proprioception! model)
2544 :muscles (movement! model)}))
2545 #+end_src
2546 #+end_listing
2548 ** Embodiment factors action recognition into manageable parts
2550 Using empathy, I divide the problem of action recognition into a
2551 recognition process expressed in the language of a full compliment
2552 of senses, and an imaginative process that generates full sensory
2553 data from partial sensory data. Splitting the action recognition
2554 problem in this manner greatly reduces the total amount of work to
2555 recognize actions: The imaginative process is mostly just matching
2556 previous experience, and the recognition process gets to use all
2557 the senses to directly describe any action.
2559 ** Action recognition is easy with a full gamut of senses
2561 Embodied representations using multiple senses such as touch,
2562 proprioception, and muscle tension turns out be be exceedingly
2563 efficient at describing body-centered actions. It is the right
2564 language for the job. For example, it takes only around 5 lines of
2565 LISP code to describe the action of curling using embodied
2566 primitives. It takes about 10 lines to describe the seemingly
2567 complicated action of wiggling.
2569 The following action predicates each take a stream of sensory
2570 experience, observe however much of it they desire, and decide
2571 whether the worm is doing the action they describe. =curled?=
2572 relies on proprioception, =resting?= relies on touch, =wiggling?=
2573 relies on a Fourier analysis of muscle contraction, and
2574 =grand-circle?= relies on touch and reuses =curled?= in its
2575 definition, showing how embodied predicates can be composed.
2578 #+caption: Program for detecting whether the worm is curled. This is the
2579 #+caption: simplest action predicate, because it only uses the last frame
2580 #+caption: of sensory experience, and only uses proprioceptive data. Even
2581 #+caption: this simple predicate, however, is automatically frame
2582 #+caption: independent and ignores vermopomorphic\protect\footnotemark
2583 #+caption: \space differences such as worm textures and colors.
2584 #+name: curled
2585 #+begin_listing clojure
2586 #+begin_src clojure
2587 (defn curled?
2588 "Is the worm curled up?"
2589 [experiences]
2590 (every?
2591 (fn [[_ _ bend]]
2592 (> (Math/sin bend) 0.64))
2593 (:proprioception (peek experiences))))
2594 #+end_src
2595 #+end_listing
2597 #+BEGIN_LaTeX
2598 \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
2599 #+END_LaTeX
2601 #+caption: Program for summarizing the touch information in a patch
2602 #+caption: of skin.
2603 #+name: touch-summary
2604 #+begin_listing clojure
2605 #+begin_src clojure
2606 (defn contact
2607 "Determine how much contact a particular worm segment has with
2608 other objects. Returns a value between 0 and 1, where 1 is full
2609 contact and 0 is no contact."
2610 [touch-region [coords contact :as touch]]
2611 (-> (zipmap coords contact)
2612 (select-keys touch-region)
2613 (vals)
2614 (#(map first %))
2615 (average)
2616 (* 10)
2617 (- 1)
2618 (Math/abs)))
2619 #+end_src
2620 #+end_listing
2623 #+caption: Program for detecting whether the worm is at rest. This program
2624 #+caption: uses a summary of the tactile information from the underbelly
2625 #+caption: of the worm, and is only true if every segment is touching the
2626 #+caption: floor. Note that this function contains no references to
2627 #+caption: proprioception at all.
2628 #+name: resting
2629 #+begin_listing clojure
2630 #+begin_src clojure
2631 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2633 (defn resting?
2634 "Is the worm resting on the ground?"
2635 [experiences]
2636 (every?
2637 (fn [touch-data]
2638 (< 0.9 (contact worm-segment-bottom touch-data)))
2639 (:touch (peek experiences))))
2640 #+end_src
2641 #+end_listing
2643 #+caption: Program for detecting whether the worm is curled up into a
2644 #+caption: full circle. Here the embodied approach begins to shine, as
2645 #+caption: I am able to both use a previous action predicate (=curled?=)
2646 #+caption: as well as the direct tactile experience of the head and tail.
2647 #+name: grand-circle
2648 #+begin_listing clojure
2649 #+begin_src clojure
2650 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2652 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2654 (defn grand-circle?
2655 "Does the worm form a majestic circle (one end touching the other)?"
2656 [experiences]
2657 (and (curled? experiences)
2658 (let [worm-touch (:touch (peek experiences))
2659 tail-touch (worm-touch 0)
2660 head-touch (worm-touch 4)]
2661 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2662 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2663 #+end_src
2664 #+end_listing
2667 #+caption: Program for detecting whether the worm has been wiggling for
2668 #+caption: the last few frames. It uses a Fourier analysis of the muscle
2669 #+caption: contractions of the worm's tail to determine wiggling. This is
2670 #+caption: significant because there is no particular frame that clearly
2671 #+caption: indicates that the worm is wiggling --- only when multiple frames
2672 #+caption: are analyzed together is the wiggling revealed. Defining
2673 #+caption: wiggling this way also gives the worm an opportunity to learn
2674 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2675 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2676 #+caption: from actual wiggling, but this definition gives it to us for free.
2677 #+name: wiggling
2678 #+begin_listing clojure
2679 #+begin_src clojure
2680 (defn fft [nums]
2681 (map
2682 #(.getReal %)
2683 (.transform
2684 (FastFourierTransformer. DftNormalization/STANDARD)
2685 (double-array nums) TransformType/FORWARD)))
2687 (def indexed (partial map-indexed vector))
2689 (defn max-indexed [s]
2690 (first (sort-by (comp - second) (indexed s))))
2692 (defn wiggling?
2693 "Is the worm wiggling?"
2694 [experiences]
2695 (let [analysis-interval 0x40]
2696 (when (> (count experiences) analysis-interval)
2697 (let [a-flex 3
2698 a-ex 2
2699 muscle-activity
2700 (map :muscle (vector:last-n experiences analysis-interval))
2701 base-activity
2702 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2703 (= 2
2704 (first
2705 (max-indexed
2706 (map #(Math/abs %)
2707 (take 20 (fft base-activity))))))))))
2708 #+end_src
2709 #+end_listing
2711 With these action predicates, I can now recognize the actions of
2712 the worm while it is moving under my control and I have access to
2713 all the worm's senses.
2715 #+caption: Use the action predicates defined earlier to report on
2716 #+caption: what the worm is doing while in simulation.
2717 #+name: report-worm-activity
2718 #+begin_listing clojure
2719 #+begin_src clojure
2720 (defn debug-experience
2721 [experiences text]
2722 (cond
2723 (grand-circle? experiences) (.setText text "Grand Circle")
2724 (curled? experiences) (.setText text "Curled")
2725 (wiggling? experiences) (.setText text "Wiggling")
2726 (resting? experiences) (.setText text "Resting")))
2727 #+end_src
2728 #+end_listing
2730 #+caption: Using =debug-experience=, the body-centered predicates
2731 #+caption: work together to classify the behavior of the worm.
2732 #+caption: the predicates are operating with access to the worm's
2733 #+caption: full sensory data.
2734 #+name: basic-worm-view
2735 #+ATTR_LaTeX: :width 10cm
2736 [[./images/worm-identify-init.png]]
2738 These action predicates satisfy the recognition requirement of an
2739 empathic recognition system. There is power in the simplicity of
2740 the action predicates. They describe their actions without getting
2741 confused in visual details of the worm. Each one is independent of
2742 position and rotation, but more than that, they are each
2743 independent of irrelevant visual details of the worm and the
2744 environment. They will work regardless of whether the worm is a
2745 different color or heavily textured, or if the environment has
2746 strange lighting.
2748 Consider how the human act of jumping might be described with
2749 body-centered action predicates: You might specify that jumping is
2750 mainly the feeling of your knees bending, your thigh muscles
2751 contracting, and your inner ear experiencing a certain sort of back
2752 and forth acceleration. This representation is a very concrete
2753 description of jumping, couched in terms of muscles and senses, but
2754 it also has the ability to describe almost all kinds of jumping, a
2755 generality that you might think could only be achieved by a very
2756 abstract description. The body centered jumping predicate does not
2757 have terms that consider the color of a person's skin or whether
2758 they are male or female, instead it gets right to the meat of what
2759 jumping actually /is/.
2761 Of course, the action predicates are not directly applicable to
2762 video data which lacks the advanced sensory information which they
2763 require!
2765 The trick now is to make the action predicates work even when the
2766 sensory data on which they depend is absent. If I can do that, then
2767 I will have gained much.
2769 ** \Phi-space describes the worm's experiences
2771 As a first step towards building empathy, I need to gather all of
2772 the worm's experiences during free play. I use a simple vector to
2773 store all the experiences.
2775 Each element of the experience vector exists in the vast space of
2776 all possible worm-experiences. Most of this vast space is actually
2777 unreachable due to physical constraints of the worm's body. For
2778 example, the worm's segments are connected by hinge joints that put
2779 a practical limit on the worm's range of motions without limiting
2780 its degrees of freedom. Some groupings of senses are impossible;
2781 the worm can not be bent into a circle so that its ends are
2782 touching and at the same time not also experience the sensation of
2783 touching itself.
2785 As the worm moves around during free play and its experience vector
2786 grows larger, the vector begins to define a subspace which is all
2787 the sensations the worm can practically experience during normal
2788 operation. I call this subspace \Phi-space, short for
2789 physical-space. The experience vector defines a path through
2790 \Phi-space. This path has interesting properties that all derive
2791 from physical embodiment. The proprioceptive components of the path
2792 vary smoothly, because in order for the worm to move from one
2793 position to another, it must pass through the intermediate
2794 positions. The path invariably forms loops as common actions are
2795 repeated. Finally and most importantly, proprioception alone
2796 actually gives very strong inference about the other senses. For
2797 example, when the worm is proprioceptively flat over several
2798 frames, you can infer that it is touching the ground and that its
2799 muscles are not active, because if the muscles were active, the
2800 worm would be moving and would not remain perfectly flat. In order
2801 to stay flat, the worm has to be touching the ground, or it would
2802 again be moving out of the flat position due to gravity. If the
2803 worm is positioned in such a way that it interacts with itself,
2804 then it is very likely to be feeling the same tactile feelings as
2805 the last time it was in that position, because it has the same body
2806 as then. As you observe multiple frames of proprioceptive data, you
2807 can become increasingly confident about the exact activations of
2808 the worm's muscles, because it generally takes a unique combination
2809 of muscle contractions to transform the worm's body along a
2810 specific path through \Phi-space.
2812 The worm's total life experience is a long looping path through
2813 \Phi-space. I will now introduce simple way of taking that
2814 experiece path and building a function that can infer complete
2815 sensory experience given only a stream of proprioceptive data. This
2816 /empathy/ function will provide a bridge to use the body centered
2817 action predicates on video-like streams of information.
2819 ** Empathy is the process of building paths in \Phi-space
2821 Here is the core of a basic empathy algorithm, starting with an
2822 experience vector:
2824 An /experience-index/ is an index into the grand experience vector
2825 that defines the worm's life. It is a time-stamp for each set of
2826 sensations the worm has experienced.
2828 First, group the experience-indices into bins according to the
2829 similarity of their proprioceptive data. I organize my bins into a
2830 3 level hierarchy. The smallest bins have an approximate size of
2831 0.001 radians in all proprioceptive dimensions. Each higher level
2832 is 10x bigger than the level below it.
2834 The bins serve as a hashing function for proprioceptive data. Given
2835 a single piece of proprioceptive experience, the bins allow us to
2836 rapidly find all other similar experience-indices of past
2837 experience that had a very similar proprioceptive configuration.
2838 When looking up a proprioceptive experience, if the smallest bin
2839 does not match any previous experience, then successively larger
2840 bins are used until a match is found or we reach the largest bin.
2842 Given a sequence of proprioceptive input, I use the bins to
2843 generate a set of similar experiences for each input using the
2844 tiered proprioceptive bins.
2846 Finally, to infer sensory data, I select the longest consecutive
2847 chain of experiences that threads through the sets of similar
2848 experiences, starting with the current moment as a root and going
2849 backwards. Consecutive experience means that the experiences appear
2850 next to each other in the experience vector.
2852 A stream of proprioceptive input might be:
2854 #+BEGIN_EXAMPLE
2855 [ flat, flat, flat, flat, flat, flat, lift-head ]
2856 #+END_EXAMPLE
2858 The worm's previous experience of lying on the ground and lifting
2859 its head generates possible interpretations for each frame:
2861 #+BEGIN_EXAMPLE
2862 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2863 1 1 1 1 1 1 1 4
2864 2 2 2 2 2 2 2
2865 3 3 3 3 3 3 3
2866 7 7 7 7 7 7 7
2867 8 8 8 8 8 8 8
2868 9 9 9 9 9 9 9
2869 #+END_EXAMPLE
2871 These interpretations suggest a new path through phi space:
2873 #+BEGIN_EXAMPLE
2874 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2875 6 7 8 9 1 2 3 4
2876 #+END_EXAMPLE
2878 The new path through \Phi-space is synthesized from two actual
2879 paths that the creature actually experiences, the "1-2-3-4" chain
2880 and the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because
2881 it ends with the worm lifting its head. It originated from a short
2882 training session where the worm rested on the floor for a brief
2883 while and then raised its head. The "6-7-8-9" chain is part of a
2884 longer chain of inactivity where the worm simply rested on the
2885 floor without moving. It is preferred over a "1-2-3" chain (which
2886 also describes inactivity) because it is longer. The main ideas
2887 again:
2889 - Imagined \Phi-space paths are synthesized by looping and mixing
2890 previous experiences.
2892 - Longer experience paths (less edits) are preferred.
2894 - The present is more important than the past --- more recent
2895 events take precedence in interpretation.
2897 This algorithm has three advantages:
2899 1. It's simple
2901 3. It's very fast -- retrieving possible interpretations takes
2902 constant time. Tracing through chains of interpretations takes
2903 time proportional to the average number of experiences in a
2904 proprioceptive bin. Redundant experiences in \Phi-space can be
2905 merged to save computation.
2907 2. It protects from wrong interpretations of transient ambiguous
2908 proprioceptive data. For example, if the worm is flat for just
2909 an instant, this flatness will not be interpreted as implying
2910 that the worm has its muscles relaxed, since the flatness is
2911 part of a longer chain which includes a distinct pattern of
2912 muscle activation. Markov chains or other memoryless statistical
2913 models that operate on individual frames may very well make this
2914 mistake.
2916 #+caption: Program to convert an experience vector into a
2917 #+caption: proprioceptively binned lookup function.
2918 #+name: bin
2919 #+begin_listing clojure
2920 #+begin_src clojure
2921 (defn bin [digits]
2922 (fn [angles]
2923 (->> angles
2924 (flatten)
2925 (map (juxt #(Math/sin %) #(Math/cos %)))
2926 (flatten)
2927 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2929 (defn gen-phi-scan
2930 "Nearest-neighbors with binning. Only returns a result if
2931 the proprioceptive data is within 10% of a previously recorded
2932 result in all dimensions."
2933 [phi-space]
2934 (let [bin-keys (map bin [3 2 1])
2935 bin-maps
2936 (map (fn [bin-key]
2937 (group-by
2938 (comp bin-key :proprioception phi-space)
2939 (range (count phi-space)))) bin-keys)
2940 lookups (map (fn [bin-key bin-map]
2941 (fn [proprio] (bin-map (bin-key proprio))))
2942 bin-keys bin-maps)]
2943 (fn lookup [proprio-data]
2944 (set (some #(% proprio-data) lookups)))))
2945 #+end_src
2946 #+end_listing
2948 #+caption: =longest-thread= finds the longest path of consecutive
2949 #+caption: past experiences to explain proprioceptive worm data from
2950 #+caption: previous data. Here, the film strip represents the
2951 #+caption: creature's previous experience. Sort sequences of
2952 #+caption: memories are spliced together to match the
2953 #+caption: proprioceptive data. Their carry the other senses
2954 #+caption: along with them.
2955 #+name: phi-space-history-scan
2956 #+ATTR_LaTeX: :width 10cm
2957 [[./images/film-of-imagination.png]]
2959 =longest-thread= infers sensory data by stitching together pieces
2960 from previous experience. It prefers longer chains of previous
2961 experience to shorter ones. For example, during training the worm
2962 might rest on the ground for one second before it performs its
2963 exercises. If during recognition the worm rests on the ground for
2964 five seconds, =longest-thread= will accommodate this five second
2965 rest period by looping the one second rest chain five times.
2967 =longest-thread= takes time proportional to the average number of
2968 entries in a proprioceptive bin, because for each element in the
2969 starting bin it performs a series of set lookups in the preceding
2970 bins. If the total history is limited, then this takes time
2971 proprotional to a only a constant multiple of the number of entries
2972 in the starting bin. This analysis also applies, even if the action
2973 requires multiple longest chains -- it's still the average number
2974 of entries in a proprioceptive bin times the desired chain length.
2975 Because =longest-thread= is so efficient and simple, I can
2976 interpret worm-actions in real time.
2978 #+caption: Program to calculate empathy by tracing though \Phi-space
2979 #+caption: and finding the longest (ie. most coherent) interpretation
2980 #+caption: of the data.
2981 #+name: longest-thread
2982 #+begin_listing clojure
2983 #+begin_src clojure
2984 (defn longest-thread
2985 "Find the longest thread from phi-index-sets. The index sets should
2986 be ordered from most recent to least recent."
2987 [phi-index-sets]
2988 (loop [result '()
2989 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2990 (if (empty? phi-index-sets)
2991 (vec result)
2992 (let [threads
2993 (for [thread-base thread-bases]
2994 (loop [thread (list thread-base)
2995 remaining remaining]
2996 (let [next-index (dec (first thread))]
2997 (cond (empty? remaining) thread
2998 (contains? (first remaining) next-index)
2999 (recur
3000 (cons next-index thread) (rest remaining))
3001 :else thread))))
3002 longest-thread
3003 (reduce (fn [thread-a thread-b]
3004 (if (> (count thread-a) (count thread-b))
3005 thread-a thread-b))
3006 '(nil)
3007 threads)]
3008 (recur (concat longest-thread result)
3009 (drop (count longest-thread) phi-index-sets))))))
3010 #+end_src
3011 #+end_listing
3013 There is one final piece, which is to replace missing sensory data
3014 with a best-guess estimate. While I could fill in missing data by
3015 using a gradient over the closest known sensory data points,
3016 averages can be misleading. It is certainly possible to create an
3017 impossible sensory state by averaging two possible sensory states.
3018 For example, consider moving your hand in an arc over your head. If
3019 for some reason you only have the initial and final positions of
3020 this movement in your \Phi-space, averaging them together will
3021 produce the proprioceptive sensation of having your hand /inside/
3022 your head, which is physically impossible to ever experience
3023 (barring motor adaption illusions). Therefore I simply replicate
3024 the most recent sensory experience to fill in the gaps.
3026 #+caption: Fill in blanks in sensory experience by replicating the most
3027 #+caption: recent experience.
3028 #+name: infer-nils
3029 #+begin_listing clojure
3030 #+begin_src clojure
3031 (defn infer-nils
3032 "Replace nils with the next available non-nil element in the
3033 sequence, or barring that, 0."
3034 [s]
3035 (loop [i (dec (count s))
3036 v (transient s)]
3037 (if (zero? i) (persistent! v)
3038 (if-let [cur (v i)]
3039 (if (get v (dec i) 0)
3040 (recur (dec i) v)
3041 (recur (dec i) (assoc! v (dec i) cur)))
3042 (recur i (assoc! v i 0))))))
3043 #+end_src
3044 #+end_listing
3046 ** COMMENT =EMPATH= recognizes actions efficiently
3048 To use =EMPATH= with the worm, I first need to gather a set of
3049 experiences from the worm that includes the actions I want to
3050 recognize. The =generate-phi-space= program (listing
3051 \ref{generate-phi-space} runs the worm through a series of
3052 exercises and gatherers those experiences into a vector. The
3053 =do-all-the-things= program is a routine expressed in a simple
3054 muscle contraction script language for automated worm control. It
3055 causes the worm to rest, curl, and wiggle over about 700 frames
3056 (approx. 11 seconds).
3058 #+caption: Program to gather the worm's experiences into a vector for
3059 #+caption: further processing. The =motor-control-program= line uses
3060 #+caption: a motor control script that causes the worm to execute a series
3061 #+caption: of ``exercises'' that include all the action predicates.
3062 #+name: generate-phi-space
3063 #+begin_listing clojure
3064 #+begin_src clojure
3065 (def do-all-the-things
3066 (concat
3067 curl-script
3068 [[300 :d-ex 40]
3069 [320 :d-ex 0]]
3070 (shift-script 280 (take 16 wiggle-script))))
3072 (defn generate-phi-space []
3073 (let [experiences (atom [])]
3074 (run-world
3075 (apply-map
3076 worm-world
3077 (merge
3078 (worm-world-defaults)
3079 {:end-frame 700
3080 :motor-control
3081 (motor-control-program worm-muscle-labels do-all-the-things)
3082 :experiences experiences})))
3083 @experiences))
3084 #+end_src
3085 #+end_listing
3087 #+caption: Use =longest-thread= and a \Phi-space generated from a short
3088 #+caption: exercise routine to interpret actions during free play.
3089 #+name: empathy-debug
3090 #+begin_listing clojure
3091 #+begin_src clojure
3092 (defn init []
3093 (def phi-space (generate-phi-space))
3094 (def phi-scan (gen-phi-scan phi-space)))
3096 (defn empathy-demonstration []
3097 (let [proprio (atom ())]
3098 (fn
3099 [experiences text]
3100 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3101 (swap! proprio (partial cons phi-indices))
3102 (let [exp-thread (longest-thread (take 300 @proprio))
3103 empathy (mapv phi-space (infer-nils exp-thread))]
3104 (println-repl (vector:last-n exp-thread 22))
3105 (cond
3106 (grand-circle? empathy) (.setText text "Grand Circle")
3107 (curled? empathy) (.setText text "Curled")
3108 (wiggling? empathy) (.setText text "Wiggling")
3109 (resting? empathy) (.setText text "Resting")
3110 :else (.setText text "Unknown")))))))
3112 (defn empathy-experiment [record]
3113 (.start (worm-world :experience-watch (debug-experience-phi)
3114 :record record :worm worm*)))
3115 #+end_src
3116 #+end_listing
3118 These programs create a test for the empathy system. First, the
3119 worm's \Phi-space is generated from a simple motor script. Then the
3120 worm is re-created in an environment almost exactly identical to
3121 the testing environment for the action-predicates, with one major
3122 difference : the only sensory information available to the system
3123 is proprioception. From just the proprioception data and
3124 \Phi-space, =longest-thread= synthesises a complete record the last
3125 300 sensory experiences of the worm. These synthesized experiences
3126 are fed directly into the action predicates =grand-circle?=,
3127 =curled?=, =wiggling?=, and =resting?= from before and their output
3128 is printed to the screen at each frame.
3130 The result of running =empathy-experiment= is that the system is
3131 generally able to interpret worm actions using the action-predicates
3132 on simulated sensory data just as well as with actual data. Figure
3133 \ref{empathy-debug-image} was generated using =empathy-experiment=:
3135 #+caption: From only proprioceptive data, =EMPATH= was able to infer
3136 #+caption: the complete sensory experience and classify four poses
3137 #+caption: (The last panel shows a composite image of /wiggling/,
3138 #+caption: a dynamic pose.)
3139 #+name: empathy-debug-image
3140 #+ATTR_LaTeX: :width 10cm :placement [H]
3141 [[./images/empathy-1.png]]
3143 One way to measure the performance of =EMPATH= is to compare the
3144 suitability of the imagined sense experience to trigger the same
3145 action predicates as the real sensory experience.
3147 #+caption: Determine how closely empathy approximates actual
3148 #+caption: sensory data.
3149 #+name: test-empathy-accuracy
3150 #+begin_listing clojure
3151 #+begin_src clojure
3152 (def worm-action-label
3153 (juxt grand-circle? curled? wiggling?))
3155 (defn compare-empathy-with-baseline [matches]
3156 (let [proprio (atom ())]
3157 (fn
3158 [experiences text]
3159 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3160 (swap! proprio (partial cons phi-indices))
3161 (let [exp-thread (longest-thread (take 300 @proprio))
3162 empathy (mapv phi-space (infer-nils exp-thread))
3163 experience-matches-empathy
3164 (= (worm-action-label experiences)
3165 (worm-action-label empathy))]
3166 (println-repl experience-matches-empathy)
3167 (swap! matches #(conj % experience-matches-empathy)))))))
3169 (defn accuracy [v]
3170 (float (/ (count (filter true? v)) (count v))))
3172 (defn test-empathy-accuracy []
3173 (let [res (atom [])]
3174 (run-world
3175 (worm-world :experience-watch
3176 (compare-empathy-with-baseline res)
3177 :worm worm*))
3178 (accuracy @res)))
3179 #+end_src
3180 #+end_listing
3182 Running =test-empathy-accuracy= using the very short exercise
3183 program defined in listing \ref{generate-phi-space}, and then doing
3184 a similar pattern of activity manually yields an accuracy of around
3185 73%. This is based on very limited worm experience. By training the
3186 worm for longer, the accuracy dramatically improves.
3188 #+caption: Program to generate \Phi-space using manual training.
3189 #+name: manual-phi-space
3190 #+begin_listing clojure
3191 #+begin_src clojure
3192 (defn init-interactive []
3193 (def phi-space
3194 (let [experiences (atom [])]
3195 (run-world
3196 (apply-map
3197 worm-world
3198 (merge
3199 (worm-world-defaults)
3200 {:experiences experiences})))
3201 @experiences))
3202 (def phi-scan (gen-phi-scan phi-space)))
3203 #+end_src
3204 #+end_listing
3206 After about 1 minute of manual training, I was able to achieve 95%
3207 accuracy on manual testing of the worm using =init-interactive= and
3208 =test-empathy-accuracy=. The majority of errors are near the
3209 boundaries of transitioning from one type of action to another.
3210 During these transitions the exact label for the action is more open
3211 to interpretation, and disagreement between empathy and experience
3212 is essentially irrelevant at this point, giving a practical
3213 identification accuracy of even higher than 95%. When I watch this
3214 system myself, I generally see no errors in action identification.
3216 ** COMMENT Digression: Learning touch sensor layout through free play
3218 In the previous section I showed how to compute actions in terms of
3219 body-centered predicates which relied on the average touch
3220 activation of pre-defined regions of the worm's skin. What if,
3221 instead of receiving touch pre-grouped into the six faces of each
3222 worm segment, the true topology of the worm's skin was unknown?
3223 This is more similar to how a nerve fiber bundle might be
3224 arranged. While two fibers that are close in a nerve bundle /might/
3225 correspond to two touch sensors that are close together on the
3226 skin, the process of taking a complicated surface and forcing it
3227 into essentially a circle requires some cuts and rearrangements.
3229 In this section I show how to automatically learn the skin-topology of
3230 a worm segment by free exploration. As the worm rolls around on the
3231 floor, large sections of its surface get activated. If the worm has
3232 stopped moving, then whatever region of skin that is touching the
3233 floor is probably an important region, and should be recorded.
3235 #+caption: Program to detect whether the worm is in a resting state
3236 #+caption: with one face touching the floor.
3237 #+name: pure-touch
3238 #+begin_listing clojure
3239 #+begin_src clojure
3240 (def full-contact [(float 0.0) (float 0.1)])
3242 (defn pure-touch?
3243 "This is worm specific code to determine if a large region of touch
3244 sensors is either all on or all off."
3245 [[coords touch :as touch-data]]
3246 (= (set (map first touch)) (set full-contact)))
3247 #+end_src
3248 #+end_listing
3250 After collecting these important regions, there will many nearly
3251 similar touch regions. While for some purposes the subtle
3252 differences between these regions will be important, for my
3253 purposes I collapse them into mostly non-overlapping sets using
3254 =remove-similar= in listing \ref{remove-similar}
3256 #+caption: Program to take a list of sets of points and ``collapse them''
3257 #+caption: so that the remaining sets in the list are significantly
3258 #+caption: different from each other. Prefer smaller sets to larger ones.
3259 #+name: remove-similar
3260 #+begin_listing clojure
3261 #+begin_src clojure
3262 (defn remove-similar
3263 [coll]
3264 (loop [result () coll (sort-by (comp - count) coll)]
3265 (if (empty? coll) result
3266 (let [[x & xs] coll
3267 c (count x)]
3268 (if (some
3269 (fn [other-set]
3270 (let [oc (count other-set)]
3271 (< (- (count (union other-set x)) c) (* oc 0.1))))
3272 xs)
3273 (recur result xs)
3274 (recur (cons x result) xs))))))
3275 #+end_src
3276 #+end_listing
3278 Actually running this simulation is easy given =CORTEX='s facilities.
3280 #+caption: Collect experiences while the worm moves around. Filter the touch
3281 #+caption: sensations by stable ones, collapse similar ones together,
3282 #+caption: and report the regions learned.
3283 #+name: learn-touch
3284 #+begin_listing clojure
3285 #+begin_src clojure
3286 (defn learn-touch-regions []
3287 (let [experiences (atom [])
3288 world (apply-map
3289 worm-world
3290 (assoc (worm-segment-defaults)
3291 :experiences experiences))]
3292 (run-world world)
3293 (->>
3294 @experiences
3295 (drop 175)
3296 ;; access the single segment's touch data
3297 (map (comp first :touch))
3298 ;; only deal with "pure" touch data to determine surfaces
3299 (filter pure-touch?)
3300 ;; associate coordinates with touch values
3301 (map (partial apply zipmap))
3302 ;; select those regions where contact is being made
3303 (map (partial group-by second))
3304 (map #(get % full-contact))
3305 (map (partial map first))
3306 ;; remove redundant/subset regions
3307 (map set)
3308 remove-similar)))
3310 (defn learn-and-view-touch-regions []
3311 (map view-touch-region
3312 (learn-touch-regions)))
3313 #+end_src
3314 #+end_listing
3316 The only thing remaining to define is the particular motion the worm
3317 must take. I accomplish this with a simple motor control program.
3319 #+caption: Motor control program for making the worm roll on the ground.
3320 #+caption: This could also be replaced with random motion.
3321 #+name: worm-roll
3322 #+begin_listing clojure
3323 #+begin_src clojure
3324 (defn touch-kinesthetics []
3325 [[170 :lift-1 40]
3326 [190 :lift-1 19]
3327 [206 :lift-1 0]
3329 [400 :lift-2 40]
3330 [410 :lift-2 0]
3332 [570 :lift-2 40]
3333 [590 :lift-2 21]
3334 [606 :lift-2 0]
3336 [800 :lift-1 30]
3337 [809 :lift-1 0]
3339 [900 :roll-2 40]
3340 [905 :roll-2 20]
3341 [910 :roll-2 0]
3343 [1000 :roll-2 40]
3344 [1005 :roll-2 20]
3345 [1010 :roll-2 0]
3347 [1100 :roll-2 40]
3348 [1105 :roll-2 20]
3349 [1110 :roll-2 0]
3350 ])
3351 #+end_src
3352 #+end_listing
3355 #+caption: The small worm rolls around on the floor, driven
3356 #+caption: by the motor control program in listing \ref{worm-roll}.
3357 #+name: worm-roll
3358 #+ATTR_LaTeX: :width 12cm
3359 [[./images/worm-roll.png]]
3362 #+caption: After completing its adventures, the worm now knows
3363 #+caption: how its touch sensors are arranged along its skin. These
3364 #+caption: are the regions that were deemed important by
3365 #+caption: =learn-touch-regions=. Note that the worm has discovered
3366 #+caption: that it has six sides.
3367 #+name: worm-touch-map
3368 #+ATTR_LaTeX: :width 12cm
3369 [[./images/touch-learn.png]]
3371 While simple, =learn-touch-regions= exploits regularities in both
3372 the worm's physiology and the worm's environment to correctly
3373 deduce that the worm has six sides. Note that =learn-touch-regions=
3374 would work just as well even if the worm's touch sense data were
3375 completely scrambled. The cross shape is just for convenience. This
3376 example justifies the use of pre-defined touch regions in =EMPATH=.
3378 * Contributions
3380 In this thesis you have seen the =CORTEX= system, a complete
3381 environment for creating simulated creatures. You have seen how to
3382 implement five senses: touch, proprioception, hearing, vision, and
3383 muscle tension. You have seen how to create new creatures using
3384 blender, a 3D modeling tool. I hope that =CORTEX= will be useful in
3385 further research projects. To this end I have included the full
3386 source to =CORTEX= along with a large suite of tests and examples. I
3387 have also created a user guide for =CORTEX= which is included in an
3388 appendix to this thesis.
3390 You have also seen how I used =CORTEX= as a platform to attach the
3391 /action recognition/ problem, which is the problem of recognizing
3392 actions in video. You saw a simple system called =EMPATH= which
3393 identifies actions by first describing actions in a body-centered,
3394 rich sense language, then inferring a full range of sensory
3395 experience from limited data using previous experience gained from
3396 free play.
3398 As a minor digression, you also saw how I used =CORTEX= to enable a
3399 tiny worm to discover the topology of its skin simply by rolling on
3400 the ground.
3402 In conclusion, the main contributions of this thesis are:
3404 - =CORTEX=, a comprehensive platform for embodied AI experiments.
3405 =CORTEX= supports many features lacking in other systems, such
3406 proper simulation of hearing. It is easy to create new =CORTEX=
3407 creatures using Blender, a free 3D modeling program.
3409 - =EMPATH=, which uses =CORTEX= to identify the actions of a
3410 worm-like creature using a computational model of empathy.
3412 #+BEGIN_LaTeX
3413 \appendix
3414 #+END_LaTeX
3416 \newpage
3418 * COMMENT Appendix: =CORTEX= User Guide
3420 Those who write a thesis should endeavor to make their code not only
3421 accessible, but actually usable, as a way to pay back the community
3422 that made the thesis possible in the first place. This thesis would
3423 not be possible without Free Software such as jMonkeyEngine3,
3424 Blender, clojure, emacs, ffmpeg, and many other tools. That is why I
3425 have included this user guide, in the hope that someone else might
3426 find =CORTEX= useful.
3428 ** Obtaining =CORTEX=
3430 You can get cortex from its mercurial repository at
3431 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3432 releases at http://aurellem.org/cortex/releases/. As a condition of
3433 making this thesis, I have also provided Professor Winston the
3434 =CORTEX= source, and he knows how to run the demos and get started.
3435 You may also email me at =cortex@aurellem.org= and I may help where
3436 I can.
3438 ** Running =CORTEX=
3440 =CORTEX= comes with README and INSTALL files that will guide you
3441 through installation and running the test suite. In particular you
3442 should look at test =cortex.test= which contains test suites that
3443 run through all senses and multiple creatures.
3445 ** Creating creatures
3447 Creatures are created using /Blender/, a free 3D modeling program.
3448 You will need Blender version 2.6 when using the =CORTEX= included
3449 in this thesis. You create a =CORTEX= creature in a similar manner
3450 to modeling anything in Blender, except that you also create
3451 several trees of empty nodes which define the creature's senses.
3453 *** Mass
3455 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3456 to the object with the mass in jMonkeyEngine units. Note that
3457 setting the mass to 0 causes the object to be immovable.
3459 *** Joints
3461 Joints are created by creating an empty node named =joints= and
3462 then creating any number of empty child nodes to represent your
3463 creature's joints. The joint will automatically connect the
3464 closest two physical objects. It will help to set the empty node's
3465 display mode to ``Arrows'' so that you can clearly see the
3466 direction of the axes.
3468 Joint nodes should have the following metadata under the ``joint''
3469 label:
3471 #+BEGIN_SRC clojure
3472 ;; ONE OF the following, under the label "joint":
3473 {:type :point}
3475 ;; OR
3477 {:type :hinge
3478 :limit [<limit-low> <limit-high>]
3479 :axis (Vector3f. <x> <y> <z>)}
3480 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3482 ;; OR
3484 {:type :cone
3485 :limit-xz <lim-xz>
3486 :limit-xy <lim-xy>
3487 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
3488 #+END_SRC
3490 *** Eyes
3492 Eyes are created by creating an empty node named =eyes= and then
3493 creating any number of empty child nodes to represent your
3494 creature's eyes.
3496 Eye nodes should have the following metadata under the ``eye''
3497 label:
3499 #+BEGIN_SRC clojure
3500 {:red <red-retina-definition>
3501 :blue <blue-retina-definition>
3502 :green <green-retina-definition>
3503 :all <all-retina-definition>
3504 (<0xrrggbb> <custom-retina-image>)...
3506 #+END_SRC
3508 Any of the color channels may be omitted. You may also include
3509 your own color selectors, and in fact :red is equivalent to
3510 0xFF0000 and so forth. The eye will be placed at the same position
3511 as the empty node and will bind to the neatest physical object.
3512 The eye will point outward from the X-axis of the node, and ``up''
3513 will be in the direction of the X-axis of the node. It will help
3514 to set the empty node's display mode to ``Arrows'' so that you can
3515 clearly see the direction of the axes.
3517 Each retina file should contain white pixels wherever you want to be
3518 sensitive to your chosen color. If you want the entire field of
3519 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3520 white.
3522 Here is a sample retinal map:
3524 #+caption: An example retinal profile image. White pixels are
3525 #+caption: photo-sensitive elements. The distribution of white
3526 #+caption: pixels is denser in the middle and falls off at the
3527 #+caption: edges and is inspired by the human retina.
3528 #+name: retina
3529 #+ATTR_LaTeX: :width 7cm :placement [H]
3530 [[./images/retina-small.png]]
3532 *** Hearing
3534 Ears are created by creating an empty node named =ears= and then
3535 creating any number of empty child nodes to represent your
3536 creature's ears.
3538 Ear nodes do not require any metadata.
3540 The ear will bind to and follow the closest physical node.
3542 *** Touch
3544 Touch is handled similarly to mass. To make a particular object
3545 touch sensitive, add metadata of the following form under the
3546 object's ``touch'' metadata field:
3548 #+BEGIN_EXAMPLE
3549 <touch-UV-map-file-name>
3550 #+END_EXAMPLE
3552 You may also include an optional ``scale'' metadata number to
3553 specify the length of the touch feelers. The default is $0.1$,
3554 and this is generally sufficient.
3556 The touch UV should contain white pixels for each touch sensor.
3558 Here is an example touch-uv map that approximates a human finger,
3559 and its corresponding model.
3561 #+caption: This is the tactile-sensor-profile for the upper segment
3562 #+caption: of a fingertip. It defines regions of high touch sensitivity
3563 #+caption: (where there are many white pixels) and regions of low
3564 #+caption: sensitivity (where white pixels are sparse).
3565 #+name: guide-fingertip-UV
3566 #+ATTR_LaTeX: :width 9cm :placement [H]
3567 [[./images/finger-UV.png]]
3569 #+caption: The fingertip UV-image form above applied to a simple
3570 #+caption: model of a fingertip.
3571 #+name: guide-fingertip
3572 #+ATTR_LaTeX: :width 9cm :placement [H]
3573 [[./images/finger-2.png]]
3575 *** Proprioception
3577 Proprioception is tied to each joint node -- nothing special must
3578 be done in a blender model to enable proprioception other than
3579 creating joint nodes.
3581 *** Muscles
3583 Muscles are created by creating an empty node named =muscles= and
3584 then creating any number of empty child nodes to represent your
3585 creature's muscles.
3588 Muscle nodes should have the following metadata under the
3589 ``muscle'' label:
3591 #+BEGIN_EXAMPLE
3592 <muscle-profile-file-name>
3593 #+END_EXAMPLE
3595 Muscles should also have a ``strength'' metadata entry describing
3596 the muscle's total strength at full activation.
3598 Muscle profiles are simple images that contain the relative amount
3599 of muscle power in each simulated alpha motor neuron. The width of
3600 the image is the total size of the motor pool, and the redness of
3601 each neuron is the relative power of that motor pool.
3603 While the profile image can have any dimensions, only the first
3604 line of pixels is used to define the muscle. Here is a sample
3605 muscle profile image that defines a human-like muscle.
3607 #+caption: A muscle profile image that describes the strengths
3608 #+caption: of each motor neuron in a muscle. White is weakest
3609 #+caption: and dark red is strongest. This particular pattern
3610 #+caption: has weaker motor neurons at the beginning, just
3611 #+caption: like human muscle.
3612 #+name: muscle-recruit
3613 #+ATTR_LaTeX: :width 7cm :placement [H]
3614 [[./images/basic-muscle.png]]
3616 Muscles twist the nearest physical object about the muscle node's
3617 Z-axis. I recommend using the ``Single Arrow'' display mode for
3618 muscles and using the right hand rule to determine which way the
3619 muscle will twist. To make a segment that can twist in multiple
3620 directions, create multiple, differently aligned muscles.
3622 ** =CORTEX= API
3624 These are the some functions exposed by =CORTEX= for creating
3625 worlds and simulating creatures. These are in addition to
3626 jMonkeyEngine3's extensive library, which is documented elsewhere.
3628 *** Simulation
3629 - =(world root-node key-map setup-fn update-fn)= :: create
3630 a simulation.
3631 - /root-node/ :: a =com.jme3.scene.Node= object which
3632 contains all of the objects that should be in the
3633 simulation.
3635 - /key-map/ :: a map from strings describing keys to
3636 functions that should be executed whenever that key is
3637 pressed. the functions should take a SimpleApplication
3638 object and a boolean value. The SimpleApplication is the
3639 current simulation that is running, and the boolean is true
3640 if the key is being pressed, and false if it is being
3641 released. As an example,
3642 #+BEGIN_SRC clojure
3643 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3644 #+END_SRC
3645 is a valid key-map which will cause the simulation to print
3646 a message whenever the 'j' key on the keyboard is pressed.
3648 - /setup-fn/ :: a function that takes a =SimpleApplication=
3649 object. It is called once when initializing the simulation.
3650 Use it to create things like lights, change the gravity,
3651 initialize debug nodes, etc.
3653 - /update-fn/ :: this function takes a =SimpleApplication=
3654 object and a float and is called every frame of the
3655 simulation. The float tells how many seconds is has been
3656 since the last frame was rendered, according to whatever
3657 clock jme is currently using. The default is to use IsoTimer
3658 which will result in this value always being the same.
3660 - =(position-camera world position rotation)= :: set the position
3661 of the simulation's main camera.
3663 - =(enable-debug world)= :: turn on debug wireframes for each
3664 simulated object.
3666 - =(set-gravity world gravity)= :: set the gravity of a running
3667 simulation.
3669 - =(box length width height & {options})= :: create a box in the
3670 simulation. Options is a hash map specifying texture, mass,
3671 etc. Possible options are =:name=, =:color=, =:mass=,
3672 =:friction=, =:texture=, =:material=, =:position=,
3673 =:rotation=, =:shape=, and =:physical?=.
3675 - =(sphere radius & {options})= :: create a sphere in the simulation.
3676 Options are the same as in =box=.
3678 - =(load-blender-model file-name)= :: create a node structure
3679 representing that described in a blender file.
3681 - =(light-up-everything world)= :: distribute a standard compliment
3682 of lights throughout the simulation. Should be adequate for most
3683 purposes.
3685 - =(node-seq node)= :: return a recursive list of the node's
3686 children.
3688 - =(nodify name children)= :: construct a node given a node-name and
3689 desired children.
3691 - =(add-element world element)= :: add an object to a running world
3692 simulation.
3694 - =(set-accuracy world accuracy)= :: change the accuracy of the
3695 world's physics simulator.
3697 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3698 construct that is useful for loading textures and is required
3699 for smooth interaction with jMonkeyEngine library functions.
3701 - =(load-bullet)= :: unpack native libraries and initialize
3702 blender. This function is required before other world building
3703 functions are called.
3705 *** Creature Manipulation / Import
3707 - =(body! creature)= :: give the creature a physical body.
3709 - =(vision! creature)= :: give the creature a sense of vision.
3710 Returns a list of functions which will each, when called
3711 during a simulation, return the vision data for the channel of
3712 one of the eyes. The functions are ordered depending on the
3713 alphabetical order of the names of the eye nodes in the
3714 blender file. The data returned by the functions is a vector
3715 containing the eye's /topology/, a vector of coordinates, and
3716 the eye's /data/, a vector of RGB values filtered by the eye's
3717 sensitivity.
3719 - =(hearing! creature)= :: give the creature a sense of hearing.
3720 Returns a list of functions, one for each ear, that when
3721 called will return a frame's worth of hearing data for that
3722 ear. The functions are ordered depending on the alphabetical
3723 order of the names of the ear nodes in the blender file. The
3724 data returned by the functions is an array PCM encoded wav
3725 data.
3727 - =(touch! creature)= :: give the creature a sense of touch. Returns
3728 a single function that must be called with the /root node/ of
3729 the world, and which will return a vector of /touch-data/
3730 one entry for each touch sensitive component, each entry of
3731 which contains a /topology/ that specifies the distribution of
3732 touch sensors, and the /data/, which is a vector of
3733 =[activation, length]= pairs for each touch hair.
3735 - =(proprioception! creature)= :: give the creature the sense of
3736 proprioception. Returns a list of functions, one for each
3737 joint, that when called during a running simulation will
3738 report the =[heading, pitch, roll]= of the joint.
3740 - =(movement! creature)= :: give the creature the power of movement.
3741 Creates a list of functions, one for each muscle, that when
3742 called with an integer, will set the recruitment of that
3743 muscle to that integer, and will report the current power
3744 being exerted by the muscle. Order of muscles is determined by
3745 the alphabetical sort order of the names of the muscle nodes.
3747 *** Visualization/Debug
3749 - =(view-vision)= :: create a function that when called with a list
3750 of visual data returned from the functions made by =vision!=,
3751 will display that visual data on the screen.
3753 - =(view-hearing)= :: same as =view-vision= but for hearing.
3755 - =(view-touch)= :: same as =view-vision= but for touch.
3757 - =(view-proprioception)= :: same as =view-vision= but for
3758 proprioception.
3760 - =(view-movement)= :: same as =view-vision= but for
3761 proprioception.
3763 - =(view anything)= :: =view= is a polymorphic function that allows
3764 you to inspect almost anything you could reasonably expect to
3765 be able to ``see'' in =CORTEX=.
3767 - =(text anything)= :: =text= is a polymorphic function that allows
3768 you to convert practically anything into a text string.
3770 - =(println-repl anything)= :: print messages to clojure's repl
3771 instead of the simulation's terminal window.
3773 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3774 function will import all jMonkeyEngine3 classes for immediate
3775 use.
3777 - =(display-dilated-time world timer)= :: Shows the time as it is
3778 flowing in the simulation on a HUD display.