view thesis/cortex.org @ 518:d78f5102d693

latex error so mad.
author Robert McIntyre <rlm@mit.edu>
date Mon, 31 Mar 2014 00:35:31 -0400
parents 68665d2c32a7
children 1803144ec9ae
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * Empathy \& Embodiment: problem solving strategies
46 By the end of this thesis, you will have seen a novel approach to
47 interpreting video using embodiment and empathy. You will have also
48 seen one way to efficiently implement empathy for embodied
49 creatures. Finally, you will become familiar with =CORTEX=, a system
50 for designing and simulating creatures with rich senses, which you
51 may choose to use in your own research.
53 This is the core vision of my thesis: That one of the important ways
54 in which we understand others is by imagining ourselves in their
55 position and emphatically feeling experiences relative to our own
56 bodies. By understanding events in terms of our own previous
57 corporeal experience, we greatly constrain the possibilities of what
58 would otherwise be an unwieldy exponential search. This extra
59 constraint can be the difference between easily understanding what
60 is happening in a video and being completely lost in a sea of
61 incomprehensible color and movement.
63 ** The problem: recognizing actions in video is hard!
65 Examine the following image. What is happening? As you, and indeed
66 very young children, can easily determine, this is an image of
67 drinking.
69 #+caption: A cat drinking some water. Identifying this action is
70 #+caption: beyond the capabilities of existing computer vision systems.
71 #+ATTR_LaTeX: :width 7cm
72 [[./images/cat-drinking.jpg]]
74 Nevertheless, it is beyond the state of the art for a computer
75 vision program to describe what's happening in this image. Part of
76 the problem is that many computer vision systems focus on
77 pixel-level details or comparisons to example images (such as
78 \cite{volume-action-recognition}), but the 3D world is so variable
79 that it is hard to describe the world in terms of possible images.
81 In fact, the contents of scene may have much less to do with pixel
82 probabilities than with recognizing various affordances: things you
83 can move, objects you can grasp, spaces that can be filled . For
84 example, what processes might enable you to see the chair in figure
85 \ref{hidden-chair}?
87 #+caption: The chair in this image is quite obvious to humans, but I
88 #+caption: doubt that any modern computer vision program can find it.
89 #+name: hidden-chair
90 #+ATTR_LaTeX: :width 10cm
91 [[./images/fat-person-sitting-at-desk.jpg]]
93 Finally, how is it that you can easily tell the difference between
94 how the girls /muscles/ are working in figure \ref{girl}?
96 #+caption: The mysterious ``common sense'' appears here as you are able
97 #+caption: to discern the difference in how the girl's arm muscles
98 #+caption: are activated between the two images.
99 #+name: girl
100 #+ATTR_LaTeX: :width 7cm
101 [[./images/wall-push.png]]
103 Each of these examples tells us something about what might be going
104 on in our minds as we easily solve these recognition problems:
106 The hidden chair shows us that we are strongly triggered by cues
107 relating to the position of human bodies, and that we can determine
108 the overall physical configuration of a human body even if much of
109 that body is occluded.
111 The picture of the girl pushing against the wall tells us that we
112 have common sense knowledge about the kinetics of our own bodies.
113 We know well how our muscles would have to work to maintain us in
114 most positions, and we can easily project this self-knowledge to
115 imagined positions triggered by images of the human body.
117 The cat tells us that imagination of some kind plays an important
118 role in understanding actions. The question is: Can we be more
119 precise about what sort of imagination is required to understand
120 these actions?
122 ** A step forward: the sensorimotor-centered approach
124 In this thesis, I explore the idea that our knowledge of our own
125 bodies, combined with our own rich senses, enables us to recognize
126 the actions of others.
128 For example, I think humans are able to label the cat video as
129 ``drinking'' because they imagine /themselves/ as the cat, and
130 imagine putting their face up against a stream of water and
131 sticking out their tongue. In that imagined world, they can feel
132 the cool water hitting their tongue, and feel the water entering
133 their body, and are able to recognize that /feeling/ as drinking.
134 So, the label of the action is not really in the pixels of the
135 image, but is found clearly in a simulation inspired by those
136 pixels. An imaginative system, having been trained on drinking and
137 non-drinking examples and learning that the most important
138 component of drinking is the feeling of water sliding down one's
139 throat, would analyze a video of a cat drinking in the following
140 manner:
142 1. Create a physical model of the video by putting a ``fuzzy''
143 model of its own body in place of the cat. Possibly also create
144 a simulation of the stream of water.
146 2. ``Play out'' this simulated scene and generate imagined sensory
147 experience. This will include relevant muscle contractions, a
148 close up view of the stream from the cat's perspective, and most
149 importantly, the imagined feeling of water entering the mouth.
150 The imagined sensory experience can come from a simulation of
151 the event, but can also be pattern-matched from previous,
152 similar embodied experience.
154 3. The action is now easily identified as drinking by the sense of
155 taste alone. The other senses (such as the tongue moving in and
156 out) help to give plausibility to the simulated action. Note that
157 the sense of vision, while critical in creating the simulation,
158 is not critical for identifying the action from the simulation.
160 For the chair examples, the process is even easier:
162 1. Align a model of your body to the person in the image.
164 2. Generate proprioceptive sensory data from this alignment.
166 3. Use the imagined proprioceptive data as a key to lookup related
167 sensory experience associated with that particular proprioceptive
168 feeling.
170 4. Retrieve the feeling of your bottom resting on a surface, your
171 knees bent, and your leg muscles relaxed.
173 5. This sensory information is consistent with your =sitting?=
174 sensory predicate, so you (and the entity in the image) must be
175 sitting.
177 6. There must be a chair-like object since you are sitting.
179 Empathy offers yet another alternative to the age-old AI
180 representation question: ``What is a chair?'' --- A chair is the
181 feeling of sitting!
183 One powerful advantage of empathic problem solving is that it
184 factors the action recognition problem into two easier problems. To
185 use empathy, you need an /aligner/, which takes the video and a
186 model of your body, and aligns the model with the video. Then, you
187 need a /recognizer/, which uses the aligned model to interpret the
188 action. The power in this method lies in the fact that you describe
189 all actions form a body-centered viewpoint. You are less tied to
190 the particulars of any visual representation of the actions. If you
191 teach the system what ``running'' is, and you have a good enough
192 aligner, the system will from then on be able to recognize running
193 from any point of view, even strange points of view like above or
194 underneath the runner. This is in contrast to action recognition
195 schemes that try to identify actions using a non-embodied approach.
196 If these systems learn about running as viewed from the side, they
197 will not automatically be able to recognize running from any other
198 viewpoint.
200 Another powerful advantage is that using the language of multiple
201 body-centered rich senses to describe body-centered actions offers a
202 massive boost in descriptive capability. Consider how difficult it
203 would be to compose a set of HOG filters to describe the action of
204 a simple worm-creature ``curling'' so that its head touches its
205 tail, and then behold the simplicity of describing thus action in a
206 language designed for the task (listing \ref{grand-circle-intro}):
208 #+caption: Body-centered actions are best expressed in a body-centered
209 #+caption: language. This code detects when the worm has curled into a
210 #+caption: full circle. Imagine how you would replicate this functionality
211 #+caption: using low-level pixel features such as HOG filters!
212 #+name: grand-circle-intro
213 #+begin_listing clojure
214 #+begin_src clojure
215 (defn grand-circle?
216 "Does the worm form a majestic circle (one end touching the other)?"
217 [experiences]
218 (and (curled? experiences)
219 (let [worm-touch (:touch (peek experiences))
220 tail-touch (worm-touch 0)
221 head-touch (worm-touch 4)]
222 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
223 (< 0.2 (contact worm-segment-top-tip head-touch))))))
224 #+end_src
225 #+end_listing
227 ** =EMPATH= recognizes actions using empathy
229 Exploring these ideas further demands a concrete implementation, so
230 first, I built a system for constructing virtual creatures with
231 physiologically plausible sensorimotor systems and detailed
232 environments. The result is =CORTEX=, which is described in section
233 \ref{sec-2}.
235 Next, I wrote routines which enabled a simple worm-like creature to
236 infer the actions of a second worm-like creature, using only its
237 own prior sensorimotor experiences and knowledge of the second
238 worm's joint positions. This program, =EMPATH=, is described in
239 section \ref{sec-3}. It's main components are:
241 - Embodied Action Definitions :: Many otherwise complicated actions
242 are easily described in the language of a full suite of
243 body-centered, rich senses and experiences. For example,
244 drinking is the feeling of water sliding down your throat, and
245 cooling your insides. It's often accompanied by bringing your
246 hand close to your face, or bringing your face close to water.
247 Sitting down is the feeling of bending your knees, activating
248 your quadriceps, then feeling a surface with your bottom and
249 relaxing your legs. These body-centered action descriptions
250 can be either learned or hard coded.
252 - Guided Play :: The creature moves around and experiences the
253 world through its unique perspective. As the creature moves,
254 it gathers experiences that satisfy the embodied action
255 definitions.
257 - Posture imitation :: When trying to interpret a video or image,
258 the creature takes a model of itself and aligns it with
259 whatever it sees. This alignment might even cross species, as
260 when humans try to align themselves with things like ponies,
261 dogs, or other humans with a different body type.
263 - Empathy :: The alignment triggers associations with
264 sensory data from prior experiences. For example, the
265 alignment itself easily maps to proprioceptive data. Any
266 sounds or obvious skin contact in the video can to a lesser
267 extent trigger previous experience keyed to hearing or touch.
268 Segments of previous experiences gained from play are stitched
269 together to form a coherent and complete sensory portrait of
270 the scene.
272 - Recognition :: With the scene described in terms of
273 remembered first person sensory events, the creature can now
274 run its action-identified programs (such as the one in listing
275 \ref{grand-circle-intro} on this synthesized sensory data,
276 just as it would if it were actually experiencing the scene
277 first-hand. If previous experience has been accurately
278 retrieved, and if it is analogous enough to the scene, then
279 the creature will correctly identify the action in the scene.
281 My program, =EMPATH= uses this empathic problem solving technique
282 to interpret the actions of a simple, worm-like creature.
284 #+caption: The worm performs many actions during free play such as
285 #+caption: curling, wiggling, and resting.
286 #+name: worm-intro
287 #+ATTR_LaTeX: :width 15cm
288 [[./images/worm-intro-white.png]]
290 #+caption: =EMPATH= recognized and classified each of these
291 #+caption: poses by inferring the complete sensory experience
292 #+caption: from proprioceptive data.
293 #+name: worm-recognition-intro
294 #+ATTR_LaTeX: :width 15cm
295 [[./images/worm-poses.png]]
297 *** Main Results
299 - After one-shot supervised training, =EMPATH= was able recognize a
300 wide variety of static poses and dynamic actions---ranging from
301 curling in a circle to wiggling with a particular frequency ---
302 with 95\% accuracy.
304 - These results were completely independent of viewing angle
305 because the underlying body-centered language fundamentally is
306 independent; once an action is learned, it can be recognized
307 equally well from any viewing angle.
309 - =EMPATH= is surprisingly short; the sensorimotor-centered
310 language provided by =CORTEX= resulted in extremely economical
311 recognition routines --- about 500 lines in all --- suggesting
312 that such representations are very powerful, and often
313 indispensable for the types of recognition tasks considered here.
315 - Although for expediency's sake, I relied on direct knowledge of
316 joint positions in this proof of concept, it would be
317 straightforward to extend =EMPATH= so that it (more
318 realistically) infers joint positions from its visual data.
320 ** =EMPATH= is built on =CORTEX=, a creature builder.
322 I built =CORTEX= to be a general AI research platform for doing
323 experiments involving multiple rich senses and a wide variety and
324 number of creatures. I intend it to be useful as a library for many
325 more projects than just this thesis. =CORTEX= was necessary to meet
326 a need among AI researchers at CSAIL and beyond, which is that
327 people often will invent neat ideas that are best expressed in the
328 language of creatures and senses, but in order to explore those
329 ideas they must first build a platform in which they can create
330 simulated creatures with rich senses! There are many ideas that
331 would be simple to execute (such as =EMPATH= or
332 \cite{larson-symbols}), but attached to them is the multi-month
333 effort to make a good creature simulator. Often, that initial
334 investment of time proves to be too much, and the project must make
335 do with a lesser environment.
337 =CORTEX= is well suited as an environment for embodied AI research
338 for three reasons:
340 - You can create new creatures using Blender (\cite{blender}), a
341 popular 3D modeling program. Each sense can be specified using
342 special blender nodes with biologically inspired parameters. You
343 need not write any code to create a creature, and can use a wide
344 library of pre-existing blender models as a base for your own
345 creatures.
347 - =CORTEX= implements a wide variety of senses: touch,
348 proprioception, vision, hearing, and muscle tension. Complicated
349 senses like touch, and vision involve multiple sensory elements
350 embedded in a 2D surface. You have complete control over the
351 distribution of these sensor elements through the use of simple
352 png image files. In particular, =CORTEX= implements more
353 comprehensive hearing than any other creature simulation system
354 available.
356 - =CORTEX= supports any number of creatures and any number of
357 senses. Time in =CORTEX= dilates so that the simulated creatures
358 always perceive a perfectly smooth flow of time, regardless of
359 the actual computational load.
361 =CORTEX= is built on top of =jMonkeyEngine3=
362 (\cite{jmonkeyengine}), which is a video game engine designed to
363 create cross-platform 3D desktop games. =CORTEX= is mainly written
364 in clojure, a dialect of =LISP= that runs on the java virtual
365 machine (JVM). The API for creating and simulating creatures and
366 senses is entirely expressed in clojure, though many senses are
367 implemented at the layer of jMonkeyEngine or below. For example,
368 for the sense of hearing I use a layer of clojure code on top of a
369 layer of java JNI bindings that drive a layer of =C++= code which
370 implements a modified version of =OpenAL= to support multiple
371 listeners. =CORTEX= is the only simulation environment that I know
372 of that can support multiple entities that can each hear the world
373 from their own perspective. Other senses also require a small layer
374 of Java code. =CORTEX= also uses =bullet=, a physics simulator
375 written in =C=.
377 #+caption: Here is the worm from figure \ref{worm-intro} modeled
378 #+caption: in Blender, a free 3D-modeling program. Senses and
379 #+caption: joints are described using special nodes in Blender.
380 #+name: worm-recognition-intro-2
381 #+ATTR_LaTeX: :width 12cm
382 [[./images/blender-worm.png]]
384 Here are some thing I anticipate that =CORTEX= might be used for:
386 - exploring new ideas about sensory integration
387 - distributed communication among swarm creatures
388 - self-learning using free exploration,
389 - evolutionary algorithms involving creature construction
390 - exploration of exotic senses and effectors that are not possible
391 in the real world (such as telekinesis or a semantic sense)
392 - imagination using subworlds
394 During one test with =CORTEX=, I created 3,000 creatures each with
395 their own independent senses and ran them all at only 1/80 real
396 time. In another test, I created a detailed model of my own hand,
397 equipped with a realistic distribution of touch (more sensitive at
398 the fingertips), as well as eyes and ears, and it ran at around 1/4
399 real time.
401 #+BEGIN_LaTeX
402 \begin{sidewaysfigure}
403 \includegraphics[width=9.5in]{images/full-hand.png}
404 \caption{
405 I modeled my own right hand in Blender and rigged it with all the
406 senses that {\tt CORTEX} supports. My simulated hand has a
407 biologically inspired distribution of touch sensors. The senses are
408 displayed on the right, and the simulation is displayed on the
409 left. Notice that my hand is curling its fingers, that it can see
410 its own finger from the eye in its palm, and that it can feel its
411 own thumb touching its palm.}
412 \end{sidewaysfigure}
413 #+END_LaTeX
415 * Designing =CORTEX=
417 In this section, I outline the design decisions that went into
418 making =CORTEX=, along with some details about its implementation.
419 (A practical guide to getting started with =CORTEX=, which skips
420 over the history and implementation details presented here, is
421 provided in an appendix at the end of this thesis.)
423 Throughout this project, I intended for =CORTEX= to be flexible and
424 extensible enough to be useful for other researchers who want to
425 test out ideas of their own. To this end, wherever I have had to make
426 architectural choices about =CORTEX=, I have chosen to give as much
427 freedom to the user as possible, so that =CORTEX= may be used for
428 things I have not foreseen.
430 ** Building in simulation versus reality
431 The most important architectural decision of all is the choice to
432 use a computer-simulated environment in the first place! The world
433 is a vast and rich place, and for now simulations are a very poor
434 reflection of its complexity. It may be that there is a significant
435 qualitative difference between dealing with senses in the real
436 world and dealing with pale facsimiles of them in a simulation
437 \cite{brooks-representation}. What are the advantages and
438 disadvantages of a simulation vs. reality?
440 *** Simulation
442 The advantages of virtual reality are that when everything is a
443 simulation, experiments in that simulation are absolutely
444 reproducible. It's also easier to change the character and world
445 to explore new situations and different sensory combinations.
447 If the world is to be simulated on a computer, then not only do
448 you have to worry about whether the character's senses are rich
449 enough to learn from the world, but whether the world itself is
450 rendered with enough detail and realism to give enough working
451 material to the character's senses. To name just a few
452 difficulties facing modern physics simulators: destructibility of
453 the environment, simulation of water/other fluids, large areas,
454 nonrigid bodies, lots of objects, smoke. I don't know of any
455 computer simulation that would allow a character to take a rock
456 and grind it into fine dust, then use that dust to make a clay
457 sculpture, at least not without spending years calculating the
458 interactions of every single small grain of dust. Maybe a
459 simulated world with today's limitations doesn't provide enough
460 richness for real intelligence to evolve.
462 *** Reality
464 The other approach for playing with senses is to hook your
465 software up to real cameras, microphones, robots, etc., and let it
466 loose in the real world. This has the advantage of eliminating
467 concerns about simulating the world at the expense of increasing
468 the complexity of implementing the senses. Instead of just
469 grabbing the current rendered frame for processing, you have to
470 use an actual camera with real lenses and interact with photons to
471 get an image. It is much harder to change the character, which is
472 now partly a physical robot of some sort, since doing so involves
473 changing things around in the real world instead of modifying
474 lines of code. While the real world is very rich and definitely
475 provides enough stimulation for intelligence to develop as
476 evidenced by our own existence, it is also uncontrollable in the
477 sense that a particular situation cannot be recreated perfectly or
478 saved for later use. It is harder to conduct science because it is
479 harder to repeat an experiment. The worst thing about using the
480 real world instead of a simulation is the matter of time. Instead
481 of simulated time you get the constant and unstoppable flow of
482 real time. This severely limits the sorts of software you can use
483 to program the AI because all sense inputs must be handled in real
484 time. Complicated ideas may have to be implemented in hardware or
485 may simply be impossible given the current speed of our
486 processors. Contrast this with a simulation, in which the flow of
487 time in the simulated world can be slowed down to accommodate the
488 limitations of the character's programming. In terms of cost,
489 doing everything in software is far cheaper than building custom
490 real-time hardware. All you need is a laptop and some patience.
492 ** Simulated time enables rapid prototyping \& simple programs
494 I envision =CORTEX= being used to support rapid prototyping and
495 iteration of ideas. Even if I could put together a well constructed
496 kit for creating robots, it would still not be enough because of
497 the scourge of real-time processing. Anyone who wants to test their
498 ideas in the real world must always worry about getting their
499 algorithms to run fast enough to process information in real time.
500 The need for real time processing only increases if multiple senses
501 are involved. In the extreme case, even simple algorithms will have
502 to be accelerated by ASIC chips or FPGAs, turning what would
503 otherwise be a few lines of code and a 10x speed penalty into a
504 multi-month ordeal. For this reason, =CORTEX= supports
505 /time-dilation/, which scales back the framerate of the
506 simulation in proportion to the amount of processing each frame.
507 From the perspective of the creatures inside the simulation, time
508 always appears to flow at a constant rate, regardless of how
509 complicated the environment becomes or how many creatures are in
510 the simulation. The cost is that =CORTEX= can sometimes run slower
511 than real time. This can also be an advantage, however ---
512 simulations of very simple creatures in =CORTEX= generally run at
513 40x on my machine!
515 ** All sense organs are two-dimensional surfaces
517 If =CORTEX= is to support a wide variety of senses, it would help
518 to have a better understanding of what a ``sense'' actually is!
519 While vision, touch, and hearing all seem like they are quite
520 different things, I was surprised to learn during the course of
521 this thesis that they (and all physical senses) can be expressed as
522 exactly the same mathematical object due to a dimensional argument!
524 Human beings are three-dimensional objects, and the nerves that
525 transmit data from our various sense organs to our brain are
526 essentially one-dimensional. This leaves up to two dimensions in
527 which our sensory information may flow. For example, imagine your
528 skin: it is a two-dimensional surface around a three-dimensional
529 object (your body). It has discrete touch sensors embedded at
530 various points, and the density of these sensors corresponds to the
531 sensitivity of that region of skin. Each touch sensor connects to a
532 nerve, all of which eventually are bundled together as they travel
533 up the spinal cord to the brain. Intersect the spinal nerves with a
534 guillotining plane and you will see all of the sensory data of the
535 skin revealed in a roughly circular two-dimensional image which is
536 the cross section of the spinal cord. Points on this image that are
537 close together in this circle represent touch sensors that are
538 /probably/ close together on the skin, although there is of course
539 some cutting and rearrangement that has to be done to transfer the
540 complicated surface of the skin onto a two dimensional image.
542 Most human senses consist of many discrete sensors of various
543 properties distributed along a surface at various densities. For
544 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
545 disks, and Ruffini's endings (\cite{9.01-textbook), which detect
546 pressure and vibration of various intensities. For ears, it is the
547 stereocilia distributed along the basilar membrane inside the
548 cochlea; each one is sensitive to a slightly different frequency of
549 sound. For eyes, it is rods and cones distributed along the surface
550 of the retina. In each case, we can describe the sense with a
551 surface and a distribution of sensors along that surface.
553 The neat idea is that every human sense can be effectively
554 described in terms of a surface containing embedded sensors. If the
555 sense had any more dimensions, then there wouldn't be enough room
556 in the spinal chord to transmit the information!
558 Therefore, =CORTEX= must support the ability to create objects and
559 then be able to ``paint'' points along their surfaces to describe
560 each sense.
562 Fortunately this idea is already a well known computer graphics
563 technique called called /UV-mapping/. The three-dimensional surface
564 of a model is cut and smooshed until it fits on a two-dimensional
565 image. You paint whatever you want on that image, and when the
566 three-dimensional shape is rendered in a game the smooshing and
567 cutting is reversed and the image appears on the three-dimensional
568 object.
570 To make a sense, interpret the UV-image as describing the
571 distribution of that senses sensors. To get different types of
572 sensors, you can either use a different color for each type of
573 sensor, or use multiple UV-maps, each labeled with that sensor
574 type. I generally use a white pixel to mean the presence of a
575 sensor and a black pixel to mean the absence of a sensor, and use
576 one UV-map for each sensor-type within a given sense.
578 #+CAPTION: The UV-map for an elongated icososphere. The white
579 #+caption: dots each represent a touch sensor. They are dense
580 #+caption: in the regions that describe the tip of the finger,
581 #+caption: and less dense along the dorsal side of the finger
582 #+caption: opposite the tip.
583 #+name: finger-UV
584 #+ATTR_latex: :width 10cm
585 [[./images/finger-UV.png]]
587 #+caption: Ventral side of the UV-mapped finger. Notice the
588 #+caption: density of touch sensors at the tip.
589 #+name: finger-side-view
590 #+ATTR_LaTeX: :width 10cm
591 [[./images/finger-1.png]]
593 ** Video game engines provide ready-made physics and shading
595 I did not need to write my own physics simulation code or shader to
596 build =CORTEX=. Doing so would lead to a system that is impossible
597 for anyone but myself to use anyway. Instead, I use a video game
598 engine as a base and modify it to accommodate the additional needs
599 of =CORTEX=. Video game engines are an ideal starting point to
600 build =CORTEX=, because they are not far from being creature
601 building systems themselves.
603 First off, general purpose video game engines come with a physics
604 engine and lighting / sound system. The physics system provides
605 tools that can be co-opted to serve as touch, proprioception, and
606 muscles. Since some games support split screen views, a good video
607 game engine will allow you to efficiently create multiple cameras
608 in the simulated world that can be used as eyes. Video game systems
609 offer integrated asset management for things like textures and
610 creatures models, providing an avenue for defining creatures. They
611 also understand UV-mapping, since this technique is used to apply a
612 texture to a model. Finally, because video game engines support a
613 large number of users, as long as =CORTEX= doesn't stray too far
614 from the base system, other researchers can turn to this community
615 for help when doing their research.
617 ** =CORTEX= is based on jMonkeyEngine3
619 While preparing to build =CORTEX= I studied several video game
620 engines to see which would best serve as a base. The top contenders
621 were:
623 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
624 software in 1997. All the source code was released by ID
625 software into the Public Domain several years ago, and as a
626 result it has been ported to many different languages. This
627 engine was famous for its advanced use of realistic shading
628 and had decent and fast physics simulation. The main advantage
629 of the Quake II engine is its simplicity, but I ultimately
630 rejected it because the engine is too tied to the concept of a
631 first-person shooter game. One of the problems I had was that
632 there does not seem to be any easy way to attach multiple
633 cameras to a single character. There are also several physics
634 clipping issues that are corrected in a way that only applies
635 to the main character and do not apply to arbitrary objects.
637 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
638 and Quake I engines and is used by Valve in the Half-Life
639 series of games. The physics simulation in the Source Engine
640 is quite accurate and probably the best out of all the engines
641 I investigated. There is also an extensive community actively
642 working with the engine. However, applications that use the
643 Source Engine must be written in C++, the code is not open, it
644 only runs on Windows, and the tools that come with the SDK to
645 handle models and textures are complicated and awkward to use.
647 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
648 games in Java. It uses OpenGL to render to the screen and uses
649 screengraphs to avoid drawing things that do not appear on the
650 screen. It has an active community and several games in the
651 pipeline. The engine was not built to serve any particular
652 game but is instead meant to be used for any 3D game.
654 I chose jMonkeyEngine3 because it because it had the most features
655 out of all the free projects I looked at, and because I could then
656 write my code in clojure, an implementation of =LISP= that runs on
657 the JVM.
659 ** =CORTEX= uses Blender to create creature models
661 For the simple worm-like creatures I will use later on in this
662 thesis, I could define a simple API in =CORTEX= that would allow
663 one to create boxes, spheres, etc., and leave that API as the sole
664 way to create creatures. However, for =CORTEX= to truly be useful
665 for other projects, it needs a way to construct complicated
666 creatures. If possible, it would be nice to leverage work that has
667 already been done by the community of 3D modelers, or at least
668 enable people who are talented at modeling but not programming to
669 design =CORTEX= creatures.
671 Therefore, I use Blender, a free 3D modeling program, as the main
672 way to create creatures in =CORTEX=. However, the creatures modeled
673 in Blender must also be simple to simulate in jMonkeyEngine3's game
674 engine, and must also be easy to rig with =CORTEX='s senses. I
675 accomplish this with extensive use of Blender's ``empty nodes.''
677 Empty nodes have no mass, physical presence, or appearance, but
678 they can hold metadata and have names. I use a tree structure of
679 empty nodes to specify senses in the following manner:
681 - Create a single top-level empty node whose name is the name of
682 the sense.
683 - Add empty nodes which each contain meta-data relevant to the
684 sense, including a UV-map describing the number/distribution of
685 sensors if applicable.
686 - Make each empty-node the child of the top-level node.
688 #+caption: An example of annotating a creature model with empty
689 #+caption: nodes to describe the layout of senses. There are
690 #+caption: multiple empty nodes which each describe the position
691 #+caption: of muscles, ears, eyes, or joints.
692 #+name: sense-nodes
693 #+ATTR_LaTeX: :width 10cm
694 [[./images/empty-sense-nodes.png]]
696 ** Bodies are composed of segments connected by joints
698 Blender is a general purpose animation tool, which has been used in
699 the past to create high quality movies such as Sintel
700 \cite{blender}. Though Blender can model and render even complicated
701 things like water, it is crucial to keep models that are meant to
702 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
703 though jMonkeyEngine3, is a rigid-body physics system. This offers
704 a compromise between the expressiveness of a game level and the
705 speed at which it can be simulated, and it means that creatures
706 should be naturally expressed as rigid components held together by
707 joint constraints.
709 But humans are more like a squishy bag wrapped around some hard
710 bones which define the overall shape. When we move, our skin bends
711 and stretches to accommodate the new positions of our bones.
713 One way to make bodies composed of rigid pieces connected by joints
714 /seem/ more human-like is to use an /armature/, (or /rigging/)
715 system, which defines a overall ``body mesh'' and defines how the
716 mesh deforms as a function of the position of each ``bone'' which
717 is a standard rigid body. This technique is used extensively to
718 model humans and create realistic animations. It is not a good
719 technique for physical simulation because it is a lie -- the skin
720 is not a physical part of the simulation and does not interact with
721 any objects in the world or itself. Objects will pass right though
722 the skin until they come in contact with the underlying bone, which
723 is a physical object. Without simulating the skin, the sense of
724 touch has little meaning, and the creature's own vision will lie to
725 it about the true extent of its body. Simulating the skin as a
726 physical object requires some way to continuously update the
727 physical model of the skin along with the movement of the bones,
728 which is unacceptably slow compared to rigid body simulation.
730 Therefore, instead of using the human-like ``deformable bag of
731 bones'' approach, I decided to base my body plans on multiple solid
732 objects that are connected by joints, inspired by the robot =EVE=
733 from the movie WALL-E.
735 #+caption: =EVE= from the movie WALL-E. This body plan turns
736 #+caption: out to be much better suited to my purposes than a more
737 #+caption: human-like one.
738 #+ATTR_LaTeX: :width 10cm
739 [[./images/Eve.jpg]]
741 =EVE='s body is composed of several rigid components that are held
742 together by invisible joint constraints. This is what I mean by
743 ``eve-like''. The main reason that I use eve-style bodies is for
744 efficiency, and so that there will be correspondence between the
745 AI's senses and the physical presence of its body. Each individual
746 section is simulated by a separate rigid body that corresponds
747 exactly with its visual representation and does not change.
748 Sections are connected by invisible joints that are well supported
749 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
750 can efficiently simulate hundreds of rigid bodies connected by
751 joints. Just because sections are rigid does not mean they have to
752 stay as one piece forever; they can be dynamically replaced with
753 multiple sections to simulate splitting in two. This could be used
754 to simulate retractable claws or =EVE='s hands, which are able to
755 coalesce into one object in the movie.
757 *** Solidifying/Connecting a body
759 =CORTEX= creates a creature in two steps: first, it traverses the
760 nodes in the blender file and creates physical representations for
761 any of them that have mass defined in their blender meta-data.
763 #+caption: Program for iterating through the nodes in a blender file
764 #+caption: and generating physical jMonkeyEngine3 objects with mass
765 #+caption: and a matching physics shape.
766 #+name: physical
767 #+begin_listing clojure
768 #+begin_src clojure
769 (defn physical!
770 "Iterate through the nodes in creature and make them real physical
771 objects in the simulation."
772 [#^Node creature]
773 (dorun
774 (map
775 (fn [geom]
776 (let [physics-control
777 (RigidBodyControl.
778 (HullCollisionShape.
779 (.getMesh geom))
780 (if-let [mass (meta-data geom "mass")]
781 (float mass) (float 1)))]
782 (.addControl geom physics-control)))
783 (filter #(isa? (class %) Geometry )
784 (node-seq creature)))))
785 #+end_src
786 #+end_listing
788 The next step to making a proper body is to connect those pieces
789 together with joints. jMonkeyEngine has a large array of joints
790 available via =bullet=, such as Point2Point, Cone, Hinge, and a
791 generic Six Degree of Freedom joint, with or without spring
792 restitution.
794 Joints are treated a lot like proper senses, in that there is a
795 top-level empty node named ``joints'' whose children each
796 represent a joint.
798 #+caption: View of the hand model in Blender showing the main ``joints''
799 #+caption: node (highlighted in yellow) and its children which each
800 #+caption: represent a joint in the hand. Each joint node has metadata
801 #+caption: specifying what sort of joint it is.
802 #+name: blender-hand
803 #+ATTR_LaTeX: :width 10cm
804 [[./images/hand-screenshot1.png]]
807 =CORTEX='s procedure for binding the creature together with joints
808 is as follows:
810 - Find the children of the ``joints'' node.
811 - Determine the two spatials the joint is meant to connect.
812 - Create the joint based on the meta-data of the empty node.
814 The higher order function =sense-nodes= from =cortex.sense=
815 simplifies finding the joints based on their parent ``joints''
816 node.
818 #+caption: Retrieving the children empty nodes from a single
819 #+caption: named empty node is a common pattern in =CORTEX=
820 #+caption: further instances of this technique for the senses
821 #+caption: will be omitted
822 #+name: get-empty-nodes
823 #+begin_listing clojure
824 #+begin_src clojure
825 (defn sense-nodes
826 "For some senses there is a special empty blender node whose
827 children are considered markers for an instance of that sense. This
828 function generates functions to find those children, given the name
829 of the special parent node."
830 [parent-name]
831 (fn [#^Node creature]
832 (if-let [sense-node (.getChild creature parent-name)]
833 (seq (.getChildren sense-node)) [])))
835 (def
836 ^{:doc "Return the children of the creature's \"joints\" node."
837 :arglists '([creature])}
838 joints
839 (sense-nodes "joints"))
840 #+end_src
841 #+end_listing
843 To find a joint's targets, =CORTEX= creates a small cube, centered
844 around the empty-node, and grows the cube exponentially until it
845 intersects two physical objects. The objects are ordered according
846 to the joint's rotation, with the first one being the object that
847 has more negative coordinates in the joint's reference frame.
848 Since the objects must be physical, the empty-node itself escapes
849 detection. Because the objects must be physical, =joint-targets=
850 must be called /after/ =physical!= is called.
852 #+caption: Program to find the targets of a joint node by
853 #+caption: exponentially growth of a search cube.
854 #+name: joint-targets
855 #+begin_listing clojure
856 #+begin_src clojure
857 (defn joint-targets
858 "Return the two closest two objects to the joint object, ordered
859 from bottom to top according to the joint's rotation."
860 [#^Node parts #^Node joint]
861 (loop [radius (float 0.01)]
862 (let [results (CollisionResults.)]
863 (.collideWith
864 parts
865 (BoundingBox. (.getWorldTranslation joint)
866 radius radius radius) results)
867 (let [targets
868 (distinct
869 (map #(.getGeometry %) results))]
870 (if (>= (count targets) 2)
871 (sort-by
872 #(let [joint-ref-frame-position
873 (jme-to-blender
874 (.mult
875 (.inverse (.getWorldRotation joint))
876 (.subtract (.getWorldTranslation %)
877 (.getWorldTranslation joint))))]
878 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
879 (take 2 targets))
880 (recur (float (* radius 2))))))))
881 #+end_src
882 #+end_listing
884 Once =CORTEX= finds all joints and targets, it creates them using
885 a dispatch on the metadata of each joint node.
887 #+caption: Program to dispatch on blender metadata and create joints
888 #+caption: suitable for physical simulation.
889 #+name: joint-dispatch
890 #+begin_listing clojure
891 #+begin_src clojure
892 (defmulti joint-dispatch
893 "Translate blender pseudo-joints into real JME joints."
894 (fn [constraints & _]
895 (:type constraints)))
897 (defmethod joint-dispatch :point
898 [constraints control-a control-b pivot-a pivot-b rotation]
899 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
900 (.setLinearLowerLimit Vector3f/ZERO)
901 (.setLinearUpperLimit Vector3f/ZERO)))
903 (defmethod joint-dispatch :hinge
904 [constraints control-a control-b pivot-a pivot-b rotation]
905 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
906 [limit-1 limit-2] (:limit constraints)
907 hinge-axis (.mult rotation (blender-to-jme axis))]
908 (doto (HingeJoint. control-a control-b pivot-a pivot-b
909 hinge-axis hinge-axis)
910 (.setLimit limit-1 limit-2))))
912 (defmethod joint-dispatch :cone
913 [constraints control-a control-b pivot-a pivot-b rotation]
914 (let [limit-xz (:limit-xz constraints)
915 limit-xy (:limit-xy constraints)
916 twist (:twist constraints)]
917 (doto (ConeJoint. control-a control-b pivot-a pivot-b
918 rotation rotation)
919 (.setLimit (float limit-xz) (float limit-xy)
920 (float twist)))))
921 #+end_src
922 #+end_listing
924 All that is left for joints it to combine the above pieces into a
925 something that can operate on the collection of nodes that a
926 blender file represents.
928 #+caption: Program to completely create a joint given information
929 #+caption: from a blender file.
930 #+name: connect
931 #+begin_listing clojure
932 #+begin_src clojure
933 (defn connect
934 "Create a joint between 'obj-a and 'obj-b at the location of
935 'joint. The type of joint is determined by the metadata on 'joint.
937 Here are some examples:
938 {:type :point}
939 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
940 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
942 {:type :cone :limit-xz 0]
943 :limit-xy 0]
944 :twist 0]} (use XZY rotation mode in blender!)"
945 [#^Node obj-a #^Node obj-b #^Node joint]
946 (let [control-a (.getControl obj-a RigidBodyControl)
947 control-b (.getControl obj-b RigidBodyControl)
948 joint-center (.getWorldTranslation joint)
949 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
950 pivot-a (world-to-local obj-a joint-center)
951 pivot-b (world-to-local obj-b joint-center)]
952 (if-let
953 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
954 ;; A side-effect of creating a joint registers
955 ;; it with both physics objects which in turn
956 ;; will register the joint with the physics system
957 ;; when the simulation is started.
958 (joint-dispatch constraints
959 control-a control-b
960 pivot-a pivot-b
961 joint-rotation))))
962 #+end_src
963 #+end_listing
965 In general, whenever =CORTEX= exposes a sense (or in this case
966 physicality), it provides a function of the type =sense!=, which
967 takes in a collection of nodes and augments it to support that
968 sense. The function returns any controls necessary to use that
969 sense. In this case =body!= creates a physical body and returns no
970 control functions.
972 #+caption: Program to give joints to a creature.
973 #+name: joints
974 #+begin_listing clojure
975 #+begin_src clojure
976 (defn joints!
977 "Connect the solid parts of the creature with physical joints. The
978 joints are taken from the \"joints\" node in the creature."
979 [#^Node creature]
980 (dorun
981 (map
982 (fn [joint]
983 (let [[obj-a obj-b] (joint-targets creature joint)]
984 (connect obj-a obj-b joint)))
985 (joints creature))))
986 (defn body!
987 "Endow the creature with a physical body connected with joints. The
988 particulars of the joints and the masses of each body part are
989 determined in blender."
990 [#^Node creature]
991 (physical! creature)
992 (joints! creature))
993 #+end_src
994 #+end_listing
996 All of the code you have just seen amounts to only 130 lines, yet
997 because it builds on top of Blender and jMonkeyEngine3, those few
998 lines pack quite a punch!
1000 The hand from figure \ref{blender-hand}, which was modeled after
1001 my own right hand, can now be given joints and simulated as a
1002 creature.
1004 #+caption: With the ability to create physical creatures from blender,
1005 #+caption: =CORTEX= gets one step closer to becoming a full creature
1006 #+caption: simulation environment.
1007 #+name: physical-hand
1008 #+ATTR_LaTeX: :width 15cm
1009 [[./images/physical-hand.png]]
1011 ** Sight reuses standard video game components...
1013 Vision is one of the most important senses for humans, so I need to
1014 build a simulated sense of vision for my AI. I will do this with
1015 simulated eyes. Each eye can be independently moved and should see
1016 its own version of the world depending on where it is.
1018 Making these simulated eyes a reality is simple because
1019 jMonkeyEngine already contains extensive support for multiple views
1020 of the same 3D simulated world. The reason jMonkeyEngine has this
1021 support is because the support is necessary to create games with
1022 split-screen views. Multiple views are also used to create
1023 efficient pseudo-reflections by rendering the scene from a certain
1024 perspective and then projecting it back onto a surface in the 3D
1025 world.
1027 #+caption: jMonkeyEngine supports multiple views to enable
1028 #+caption: split-screen games, like GoldenEye, which was one of
1029 #+caption: the first games to use split-screen views.
1030 #+name: goldeneye
1031 #+ATTR_LaTeX: :width 10cm
1032 [[./images/goldeneye-4-player.png]]
1034 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
1036 jMonkeyEngine allows you to create a =ViewPort=, which represents a
1037 view of the simulated world. You can create as many of these as you
1038 want. Every frame, the =RenderManager= iterates through each
1039 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
1040 is a =FrameBuffer= which represents the rendered image in the GPU.
1042 #+caption: =ViewPorts= are cameras in the world. During each frame,
1043 #+caption: the =RenderManager= records a snapshot of what each view
1044 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
1045 #+name: rendermanagers
1046 #+ATTR_LaTeX: :width 10cm
1047 [[./images/diagram_rendermanager2.png]]
1049 Each =ViewPort= can have any number of attached =SceneProcessor=
1050 objects, which are called every time a new frame is rendered. A
1051 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1052 whatever it wants to the data. Often this consists of invoking GPU
1053 specific operations on the rendered image. The =SceneProcessor= can
1054 also copy the GPU image data to RAM and process it with the CPU.
1056 *** Appropriating Views for Vision
1058 Each eye in the simulated creature needs its own =ViewPort= so
1059 that it can see the world from its own perspective. To this
1060 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1061 any arbitrary continuation function for further processing. That
1062 continuation function may perform both CPU and GPU operations on
1063 the data. To make this easy for the continuation function, the
1064 =SceneProcessor= maintains appropriately sized buffers in RAM to
1065 hold the data. It does not do any copying from the GPU to the CPU
1066 itself because it is a slow operation.
1068 #+caption: Function to make the rendered scene in jMonkeyEngine
1069 #+caption: available for further processing.
1070 #+name: pipeline-1
1071 #+begin_listing clojure
1072 #+begin_src clojure
1073 (defn vision-pipeline
1074 "Create a SceneProcessor object which wraps a vision processing
1075 continuation function. The continuation is a function that takes
1076 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1077 each of which has already been appropriately sized."
1078 [continuation]
1079 (let [byte-buffer (atom nil)
1080 renderer (atom nil)
1081 image (atom nil)]
1082 (proxy [SceneProcessor] []
1083 (initialize
1084 [renderManager viewPort]
1085 (let [cam (.getCamera viewPort)
1086 width (.getWidth cam)
1087 height (.getHeight cam)]
1088 (reset! renderer (.getRenderer renderManager))
1089 (reset! byte-buffer
1090 (BufferUtils/createByteBuffer
1091 (* width height 4)))
1092 (reset! image (BufferedImage.
1093 width height
1094 BufferedImage/TYPE_4BYTE_ABGR))))
1095 (isInitialized [] (not (nil? @byte-buffer)))
1096 (reshape [_ _ _])
1097 (preFrame [_])
1098 (postQueue [_])
1099 (postFrame
1100 [#^FrameBuffer fb]
1101 (.clear @byte-buffer)
1102 (continuation @renderer fb @byte-buffer @image))
1103 (cleanup []))))
1104 #+end_src
1105 #+end_listing
1107 The continuation function given to =vision-pipeline= above will be
1108 given a =Renderer= and three containers for image data. The
1109 =FrameBuffer= references the GPU image data, but the pixel data
1110 can not be used directly on the CPU. The =ByteBuffer= and
1111 =BufferedImage= are initially "empty" but are sized to hold the
1112 data in the =FrameBuffer=. I call transferring the GPU image data
1113 to the CPU structures "mixing" the image data.
1115 *** Optical sensor arrays are described with images and referenced with metadata
1117 The vision pipeline described above handles the flow of rendered
1118 images. Now, =CORTEX= needs simulated eyes to serve as the source
1119 of these images.
1121 An eye is described in blender in the same way as a joint. They
1122 are zero dimensional empty objects with no geometry whose local
1123 coordinate system determines the orientation of the resulting eye.
1124 All eyes are children of a parent node named "eyes" just as all
1125 joints have a parent named "joints". An eye binds to the nearest
1126 physical object with =bind-sense=.
1128 #+caption: Here, the camera is created based on metadata on the
1129 #+caption: eye-node and attached to the nearest physical object
1130 #+caption: with =bind-sense=
1131 #+name: add-eye
1132 #+begin_listing clojure
1133 (defn add-eye!
1134 "Create a Camera centered on the current position of 'eye which
1135 follows the closest physical node in 'creature. The camera will
1136 point in the X direction and use the Z vector as up as determined
1137 by the rotation of these vectors in blender coordinate space. Use
1138 XZY rotation for the node in blender."
1139 [#^Node creature #^Spatial eye]
1140 (let [target (closest-node creature eye)
1141 [cam-width cam-height]
1142 ;;[640 480] ;; graphics card on laptop doesn't support
1143 ;; arbitrary dimensions.
1144 (eye-dimensions eye)
1145 cam (Camera. cam-width cam-height)
1146 rot (.getWorldRotation eye)]
1147 (.setLocation cam (.getWorldTranslation eye))
1148 (.lookAtDirection
1149 cam ; this part is not a mistake and
1150 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1151 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1152 (.setFrustumPerspective
1153 cam (float 45)
1154 (float (/ (.getWidth cam) (.getHeight cam)))
1155 (float 1)
1156 (float 1000))
1157 (bind-sense target cam) cam))
1158 #+end_listing
1160 *** Simulated Retina
1162 An eye is a surface (the retina) which contains many discrete
1163 sensors to detect light. These sensors can have different
1164 light-sensing properties. In humans, each discrete sensor is
1165 sensitive to red, blue, green, or gray. These different types of
1166 sensors can have different spatial distributions along the retina.
1167 In humans, there is a fovea in the center of the retina which has
1168 a very high density of color sensors, and a blind spot which has
1169 no sensors at all. Sensor density decreases in proportion to
1170 distance from the fovea.
1172 I want to be able to model any retinal configuration, so my
1173 eye-nodes in blender contain metadata pointing to images that
1174 describe the precise position of the individual sensors using
1175 white pixels. The meta-data also describes the precise sensitivity
1176 to light that the sensors described in the image have. An eye can
1177 contain any number of these images. For example, the metadata for
1178 an eye might look like this:
1180 #+begin_src clojure
1181 {0xFF0000 "Models/test-creature/retina-small.png"}
1182 #+end_src
1184 #+caption: An example retinal profile image. White pixels are
1185 #+caption: photo-sensitive elements. The distribution of white
1186 #+caption: pixels is denser in the middle and falls off at the
1187 #+caption: edges and is inspired by the human retina.
1188 #+name: retina
1189 #+ATTR_LaTeX: :width 7cm
1190 [[./images/retina-small.png]]
1192 Together, the number 0xFF0000 and the image image above describe
1193 the placement of red-sensitive sensory elements.
1195 Meta-data to very crudely approximate a human eye might be
1196 something like this:
1198 #+begin_src clojure
1199 (let [retinal-profile "Models/test-creature/retina-small.png"]
1200 {0xFF0000 retinal-profile
1201 0x00FF00 retinal-profile
1202 0x0000FF retinal-profile
1203 0xFFFFFF retinal-profile})
1204 #+end_src
1206 The numbers that serve as keys in the map determine a sensor's
1207 relative sensitivity to the channels red, green, and blue. These
1208 sensitivity values are packed into an integer in the order
1209 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1210 image are added together with these sensitivities as linear
1211 weights. Therefore, 0xFF0000 means sensitive to red only while
1212 0xFFFFFF means sensitive to all colors equally (gray).
1214 #+caption: This is the core of vision in =CORTEX=. A given eye node
1215 #+caption: is converted into a function that returns visual
1216 #+caption: information from the simulation.
1217 #+name: vision-kernel
1218 #+begin_listing clojure
1219 #+BEGIN_SRC clojure
1220 (defn vision-kernel
1221 "Returns a list of functions, each of which will return a color
1222 channel's worth of visual information when called inside a running
1223 simulation."
1224 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1225 (let [retinal-map (retina-sensor-profile eye)
1226 camera (add-eye! creature eye)
1227 vision-image
1228 (atom
1229 (BufferedImage. (.getWidth camera)
1230 (.getHeight camera)
1231 BufferedImage/TYPE_BYTE_BINARY))
1232 register-eye!
1233 (runonce
1234 (fn [world]
1235 (add-camera!
1236 world camera
1237 (let [counter (atom 0)]
1238 (fn [r fb bb bi]
1239 (if (zero? (rem (swap! counter inc) (inc skip)))
1240 (reset! vision-image
1241 (BufferedImage! r fb bb bi))))))))]
1242 (vec
1243 (map
1244 (fn [[key image]]
1245 (let [whites (white-coordinates image)
1246 topology (vec (collapse whites))
1247 sensitivity (sensitivity-presets key key)]
1248 (attached-viewport.
1249 (fn [world]
1250 (register-eye! world)
1251 (vector
1252 topology
1253 (vec
1254 (for [[x y] whites]
1255 (pixel-sense
1256 sensitivity
1257 (.getRGB @vision-image x y))))))
1258 register-eye!)))
1259 retinal-map))))
1260 #+END_SRC
1261 #+end_listing
1263 Note that since each of the functions generated by =vision-kernel=
1264 shares the same =register-eye!= function, the eye will be
1265 registered only once the first time any of the functions from the
1266 list returned by =vision-kernel= is called. Each of the functions
1267 returned by =vision-kernel= also allows access to the =Viewport=
1268 through which it receives images.
1270 All the hard work has been done; all that remains is to apply
1271 =vision-kernel= to each eye in the creature and gather the results
1272 into one list of functions.
1275 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1276 #+caption: environment for experimenting with different types of
1277 #+caption: eyes.
1278 #+name: vision!
1279 #+begin_listing clojure
1280 #+BEGIN_SRC clojure
1281 (defn vision!
1282 "Returns a list of functions, each of which returns visual sensory
1283 data when called inside a running simulation."
1284 [#^Node creature & {skip :skip :or {skip 0}}]
1285 (reduce
1286 concat
1287 (for [eye (eyes creature)]
1288 (vision-kernel creature eye))))
1289 #+END_SRC
1290 #+end_listing
1292 #+caption: Simulated vision with a test creature and the
1293 #+caption: human-like eye approximation. Notice how each channel
1294 #+caption: of the eye responds differently to the differently
1295 #+caption: colored balls.
1296 #+name: worm-vision-test.
1297 #+ATTR_LaTeX: :width 13cm
1298 [[./images/worm-vision.png]]
1300 The vision code is not much more complicated than the body code,
1301 and enables multiple further paths for simulated vision. For
1302 example, it is quite easy to create bifocal vision -- you just
1303 make two eyes next to each other in blender! It is also possible
1304 to encode vision transforms in the retinal files. For example, the
1305 human like retina file in figure \ref{retina} approximates a
1306 log-polar transform.
1308 This vision code has already been absorbed by the jMonkeyEngine
1309 community and is now (in modified form) part of a system for
1310 capturing in-game video to a file.
1312 ** ...but hearing must be built from scratch
1314 At the end of this section I will have simulated ears that work the
1315 same way as the simulated eyes in the last section. I will be able to
1316 place any number of ear-nodes in a blender file, and they will bind to
1317 the closest physical object and follow it as it moves around. Each ear
1318 will provide access to the sound data it picks up between every frame.
1320 Hearing is one of the more difficult senses to simulate, because there
1321 is less support for obtaining the actual sound data that is processed
1322 by jMonkeyEngine3. There is no "split-screen" support for rendering
1323 sound from different points of view, and there is no way to directly
1324 access the rendered sound data.
1326 =CORTEX='s hearing is unique because it does not have any
1327 limitations compared to other simulation environments. As far as I
1328 know, there is no other system that supports multiple listeners,
1329 and the sound demo at the end of this section is the first time
1330 it's been done in a video game environment.
1332 *** Brief Description of jMonkeyEngine's Sound System
1334 jMonkeyEngine's sound system works as follows:
1336 - jMonkeyEngine uses the =AppSettings= for the particular
1337 application to determine what sort of =AudioRenderer= should be
1338 used.
1339 - Although some support is provided for multiple AudioRendering
1340 backends, jMonkeyEngine at the time of this writing will either
1341 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1342 - jMonkeyEngine tries to figure out what sort of system you're
1343 running and extracts the appropriate native libraries.
1344 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1345 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1346 - =OpenAL= renders the 3D sound and feeds the rendered sound
1347 directly to any of various sound output devices with which it
1348 knows how to communicate.
1350 A consequence of this is that there's no way to access the actual
1351 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1352 one /listener/ (it renders sound data from only one perspective),
1353 which normally isn't a problem for games, but becomes a problem
1354 when trying to make multiple AI creatures that can each hear the
1355 world from a different perspective.
1357 To make many AI creatures in jMonkeyEngine that can each hear the
1358 world from their own perspective, or to make a single creature with
1359 many ears, it is necessary to go all the way back to =OpenAL= and
1360 implement support for simulated hearing there.
1362 *** Extending =OpenAl=
1364 Extending =OpenAL= to support multiple listeners requires 500
1365 lines of =C= code and is too hairy to mention here. Instead, I
1366 will show a small amount of extension code and go over the high
1367 level strategy. Full source is of course available with the
1368 =CORTEX= distribution if you're interested.
1370 =OpenAL= goes to great lengths to support many different systems,
1371 all with different sound capabilities and interfaces. It
1372 accomplishes this difficult task by providing code for many
1373 different sound backends in pseudo-objects called /Devices/.
1374 There's a device for the Linux Open Sound System and the Advanced
1375 Linux Sound Architecture, there's one for Direct Sound on Windows,
1376 and there's even one for Solaris. =OpenAL= solves the problem of
1377 platform independence by providing all these Devices.
1379 Wrapper libraries such as LWJGL are free to examine the system on
1380 which they are running and then select an appropriate device for
1381 that system.
1383 There are also a few "special" devices that don't interface with
1384 any particular system. These include the Null Device, which
1385 doesn't do anything, and the Wave Device, which writes whatever
1386 sound it receives to a file, if everything has been set up
1387 correctly when configuring =OpenAL=.
1389 Actual mixing (Doppler shift and distance.environment-based
1390 attenuation) of the sound data happens in the Devices, and they
1391 are the only point in the sound rendering process where this data
1392 is available.
1394 Therefore, in order to support multiple listeners, and get the
1395 sound data in a form that the AIs can use, it is necessary to
1396 create a new Device which supports this feature.
1398 Adding a device to OpenAL is rather tricky -- there are five
1399 separate files in the =OpenAL= source tree that must be modified
1400 to do so. I named my device the "Multiple Audio Send" Device, or
1401 =Send= Device for short, since it sends audio data back to the
1402 calling application like an Aux-Send cable on a mixing board.
1404 The main idea behind the Send device is to take advantage of the
1405 fact that LWJGL only manages one /context/ when using OpenAL. A
1406 /context/ is like a container that holds samples and keeps track
1407 of where the listener is. In order to support multiple listeners,
1408 the Send device identifies the LWJGL context as the master
1409 context, and creates any number of slave contexts to represent
1410 additional listeners. Every time the device renders sound, it
1411 synchronizes every source from the master LWJGL context to the
1412 slave contexts. Then, it renders each context separately, using a
1413 different listener for each one. The rendered sound is made
1414 available via JNI to jMonkeyEngine.
1416 Switching between contexts is not the normal operation of a
1417 Device, and one of the problems with doing so is that a Device
1418 normally keeps around a few pieces of state such as the
1419 =ClickRemoval= array above which will become corrupted if the
1420 contexts are not rendered in parallel. The solution is to create a
1421 copy of this normally global device state for each context, and
1422 copy it back and forth into and out of the actual device state
1423 whenever a context is rendered.
1425 The core of the =Send= device is the =syncSources= function, which
1426 does the job of copying all relevant data from one context to
1427 another.
1429 #+caption: Program for extending =OpenAL= to support multiple
1430 #+caption: listeners via context copying/switching.
1431 #+name: sync-openal-sources
1432 #+begin_listing c
1433 #+BEGIN_SRC c
1434 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1435 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1436 ALuint master = masterSource->source;
1437 ALuint slave = slaveSource->source;
1438 ALCcontext *current = alcGetCurrentContext();
1440 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1441 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1442 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1443 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1444 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1445 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1446 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1447 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1448 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1449 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1450 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1451 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1452 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1454 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1455 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1456 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1458 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1459 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1461 alcMakeContextCurrent(masterCtx);
1462 ALint source_type;
1463 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1465 // Only static sources are currently synchronized!
1466 if (AL_STATIC == source_type){
1467 ALint master_buffer;
1468 ALint slave_buffer;
1469 alGetSourcei(master, AL_BUFFER, &master_buffer);
1470 alcMakeContextCurrent(slaveCtx);
1471 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1472 if (master_buffer != slave_buffer){
1473 alSourcei(slave, AL_BUFFER, master_buffer);
1477 // Synchronize the state of the two sources.
1478 alcMakeContextCurrent(masterCtx);
1479 ALint masterState;
1480 ALint slaveState;
1482 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1483 alcMakeContextCurrent(slaveCtx);
1484 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1486 if (masterState != slaveState){
1487 switch (masterState){
1488 case AL_INITIAL : alSourceRewind(slave); break;
1489 case AL_PLAYING : alSourcePlay(slave); break;
1490 case AL_PAUSED : alSourcePause(slave); break;
1491 case AL_STOPPED : alSourceStop(slave); break;
1494 // Restore whatever context was previously active.
1495 alcMakeContextCurrent(current);
1497 #+END_SRC
1498 #+end_listing
1500 With this special context-switching device, and some ugly JNI
1501 bindings that are not worth mentioning, =CORTEX= gains the ability
1502 to access multiple sound streams from =OpenAL=.
1504 #+caption: Program to create an ear from a blender empty node. The ear
1505 #+caption: follows around the nearest physical object and passes
1506 #+caption: all sensory data to a continuation function.
1507 #+name: add-ear
1508 #+begin_listing clojure
1509 #+BEGIN_SRC clojure
1510 (defn add-ear!
1511 "Create a Listener centered on the current position of 'ear
1512 which follows the closest physical node in 'creature and
1513 sends sound data to 'continuation."
1514 [#^Application world #^Node creature #^Spatial ear continuation]
1515 (let [target (closest-node creature ear)
1516 lis (Listener.)
1517 audio-renderer (.getAudioRenderer world)
1518 sp (hearing-pipeline continuation)]
1519 (.setLocation lis (.getWorldTranslation ear))
1520 (.setRotation lis (.getWorldRotation ear))
1521 (bind-sense target lis)
1522 (update-listener-velocity! target lis)
1523 (.addListener audio-renderer lis)
1524 (.registerSoundProcessor audio-renderer lis sp)))
1525 #+END_SRC
1526 #+end_listing
1528 The =Send= device, unlike most of the other devices in =OpenAL=,
1529 does not render sound unless asked. This enables the system to
1530 slow down or speed up depending on the needs of the AIs who are
1531 using it to listen. If the device tried to render samples in
1532 real-time, a complicated AI whose mind takes 100 seconds of
1533 computer time to simulate 1 second of AI-time would miss almost
1534 all of the sound in its environment!
1536 #+caption: Program to enable arbitrary hearing in =CORTEX=
1537 #+name: hearing
1538 #+begin_listing clojure
1539 #+BEGIN_SRC clojure
1540 (defn hearing-kernel
1541 "Returns a function which returns auditory sensory data when called
1542 inside a running simulation."
1543 [#^Node creature #^Spatial ear]
1544 (let [hearing-data (atom [])
1545 register-listener!
1546 (runonce
1547 (fn [#^Application world]
1548 (add-ear!
1549 world creature ear
1550 (comp #(reset! hearing-data %)
1551 byteBuffer->pulse-vector))))]
1552 (fn [#^Application world]
1553 (register-listener! world)
1554 (let [data @hearing-data
1555 topology
1556 (vec (map #(vector % 0) (range 0 (count data))))]
1557 [topology data]))))
1559 (defn hearing!
1560 "Endow the creature in a particular world with the sense of
1561 hearing. Will return a sequence of functions, one for each ear,
1562 which when called will return the auditory data from that ear."
1563 [#^Node creature]
1564 (for [ear (ears creature)]
1565 (hearing-kernel creature ear)))
1566 #+END_SRC
1567 #+end_listing
1569 Armed with these functions, =CORTEX= is able to test possibly the
1570 first ever instance of multiple listeners in a video game engine
1571 based simulation!
1573 #+caption: Here a simple creature responds to sound by changing
1574 #+caption: its color from gray to green when the total volume
1575 #+caption: goes over a threshold.
1576 #+name: sound-test
1577 #+begin_listing java
1578 #+BEGIN_SRC java
1579 /**
1580 * Respond to sound! This is the brain of an AI entity that
1581 * hears its surroundings and reacts to them.
1582 */
1583 public void process(ByteBuffer audioSamples,
1584 int numSamples, AudioFormat format) {
1585 audioSamples.clear();
1586 byte[] data = new byte[numSamples];
1587 float[] out = new float[numSamples];
1588 audioSamples.get(data);
1589 FloatSampleTools.
1590 byte2floatInterleaved
1591 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1593 float max = Float.NEGATIVE_INFINITY;
1594 for (float f : out){if (f > max) max = f;}
1595 audioSamples.clear();
1597 if (max > 0.1){
1598 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1600 else {
1601 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1603 #+END_SRC
1604 #+end_listing
1606 #+caption: First ever simulation of multiple listeners in =CORTEX=.
1607 #+caption: Each cube is a creature which processes sound data with
1608 #+caption: the =process= function from listing \ref{sound-test}.
1609 #+caption: the ball is constantly emitting a pure tone of
1610 #+caption: constant volume. As it approaches the cubes, they each
1611 #+caption: change color in response to the sound.
1612 #+name: sound-cubes.
1613 #+ATTR_LaTeX: :width 10cm
1614 [[./images/java-hearing-test.png]]
1616 This system of hearing has also been co-opted by the
1617 jMonkeyEngine3 community and is used to record audio for demo
1618 videos.
1620 ** Hundreds of hair-like elements provide a sense of touch
1622 Touch is critical to navigation and spatial reasoning and as such I
1623 need a simulated version of it to give to my AI creatures.
1625 Human skin has a wide array of touch sensors, each of which
1626 specialize in detecting different vibrational modes and pressures.
1627 These sensors can integrate a vast expanse of skin (i.e. your
1628 entire palm), or a tiny patch of skin at the tip of your finger.
1629 The hairs of the skin help detect objects before they even come
1630 into contact with the skin proper.
1632 However, touch in my simulated world can not exactly correspond to
1633 human touch because my creatures are made out of completely rigid
1634 segments that don't deform like human skin.
1636 Instead of measuring deformation or vibration, I surround each
1637 rigid part with a plenitude of hair-like objects (/feelers/) which
1638 do not interact with the physical world. Physical objects can pass
1639 through them with no effect. The feelers are able to tell when
1640 other objects pass through them, and they constantly report how
1641 much of their extent is covered. So even though the creature's body
1642 parts do not deform, the feelers create a margin around those body
1643 parts which achieves a sense of touch which is a hybrid between a
1644 human's sense of deformation and sense from hairs.
1646 Implementing touch in jMonkeyEngine follows a different technical
1647 route than vision and hearing. Those two senses piggybacked off
1648 jMonkeyEngine's 3D audio and video rendering subsystems. To
1649 simulate touch, I use jMonkeyEngine's physics system to execute
1650 many small collision detections, one for each feeler. The placement
1651 of the feelers is determined by a UV-mapped image which shows where
1652 each feeler should be on the 3D surface of the body.
1654 *** Defining Touch Meta-Data in Blender
1656 Each geometry can have a single UV map which describes the
1657 position of the feelers which will constitute its sense of touch.
1658 This image path is stored under the ``touch'' key. The image itself
1659 is black and white, with black meaning a feeler length of 0 (no
1660 feeler is present) and white meaning a feeler length of =scale=,
1661 which is a float stored under the key "scale".
1663 #+caption: Touch does not use empty nodes, to store metadata,
1664 #+caption: because the metadata of each solid part of a
1665 #+caption: creature's body is sufficient.
1666 #+name: touch-meta-data
1667 #+begin_listing clojure
1668 #+BEGIN_SRC clojure
1669 (defn tactile-sensor-profile
1670 "Return the touch-sensor distribution image in BufferedImage format,
1671 or nil if it does not exist."
1672 [#^Geometry obj]
1673 (if-let [image-path (meta-data obj "touch")]
1674 (load-image image-path)))
1676 (defn tactile-scale
1677 "Return the length of each feeler. Default scale is 0.01
1678 jMonkeyEngine units."
1679 [#^Geometry obj]
1680 (if-let [scale (meta-data obj "scale")]
1681 scale 0.1))
1682 #+END_SRC
1683 #+end_listing
1685 Here is an example of a UV-map which specifies the position of
1686 touch sensors along the surface of the upper segment of a fingertip.
1688 #+caption: This is the tactile-sensor-profile for the upper segment
1689 #+caption: of a fingertip. It defines regions of high touch sensitivity
1690 #+caption: (where there are many white pixels) and regions of low
1691 #+caption: sensitivity (where white pixels are sparse).
1692 #+name: fingertip-UV
1693 #+ATTR_LaTeX: :width 13cm
1694 [[./images/finger-UV.png]]
1696 *** Implementation Summary
1698 To simulate touch there are three conceptual steps. For each solid
1699 object in the creature, you first have to get UV image and scale
1700 parameter which define the position and length of the feelers.
1701 Then, you use the triangles which comprise the mesh and the UV
1702 data stored in the mesh to determine the world-space position and
1703 orientation of each feeler. Then once every frame, update these
1704 positions and orientations to match the current position and
1705 orientation of the object, and use physics collision detection to
1706 gather tactile data.
1708 Extracting the meta-data has already been described. The third
1709 step, physics collision detection, is handled in =touch-kernel=.
1710 Translating the positions and orientations of the feelers from the
1711 UV-map to world-space is itself a three-step process.
1713 - Find the triangles which make up the mesh in pixel-space and in
1714 world-space. \\(=triangles=, =pixel-triangles=).
1716 - Find the coordinates of each feeler in world-space. These are
1717 the origins of the feelers. (=feeler-origins=).
1719 - Calculate the normals of the triangles in world space, and add
1720 them to each of the origins of the feelers. These are the
1721 normalized coordinates of the tips of the feelers.
1722 (=feeler-tips=).
1724 *** Triangle Math
1726 The rigid objects which make up a creature have an underlying
1727 =Geometry=, which is a =Mesh= plus a =Material= and other
1728 important data involved with displaying the object.
1730 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1731 vertices which have coordinates in world space and UV space.
1733 Here, =triangles= gets all the world-space triangles which
1734 comprise a mesh, while =pixel-triangles= gets those same triangles
1735 expressed in pixel coordinates (which are UV coordinates scaled to
1736 fit the height and width of the UV image).
1738 #+caption: Programs to extract triangles from a geometry and get
1739 #+caption: their vertices in both world and UV-coordinates.
1740 #+name: get-triangles
1741 #+begin_listing clojure
1742 #+BEGIN_SRC clojure
1743 (defn triangle
1744 "Get the triangle specified by triangle-index from the mesh."
1745 [#^Geometry geo triangle-index]
1746 (triangle-seq
1747 (let [scratch (Triangle.)]
1748 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1750 (defn triangles
1751 "Return a sequence of all the Triangles which comprise a given
1752 Geometry."
1753 [#^Geometry geo]
1754 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1756 (defn triangle-vertex-indices
1757 "Get the triangle vertex indices of a given triangle from a given
1758 mesh."
1759 [#^Mesh mesh triangle-index]
1760 (let [indices (int-array 3)]
1761 (.getTriangle mesh triangle-index indices)
1762 (vec indices)))
1764 (defn vertex-UV-coord
1765 "Get the UV-coordinates of the vertex named by vertex-index"
1766 [#^Mesh mesh vertex-index]
1767 (let [UV-buffer
1768 (.getData
1769 (.getBuffer
1770 mesh
1771 VertexBuffer$Type/TexCoord))]
1772 [(.get UV-buffer (* vertex-index 2))
1773 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1775 (defn pixel-triangle [#^Geometry geo image index]
1776 (let [mesh (.getMesh geo)
1777 width (.getWidth image)
1778 height (.getHeight image)]
1779 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1780 (map (partial vertex-UV-coord mesh)
1781 (triangle-vertex-indices mesh index))))))
1783 (defn pixel-triangles
1784 "The pixel-space triangles of the Geometry, in the same order as
1785 (triangles geo)"
1786 [#^Geometry geo image]
1787 (let [height (.getHeight image)
1788 width (.getWidth image)]
1789 (map (partial pixel-triangle geo image)
1790 (range (.getTriangleCount (.getMesh geo))))))
1791 #+END_SRC
1792 #+end_listing
1794 *** The Affine Transform from one Triangle to Another
1796 =pixel-triangles= gives us the mesh triangles expressed in pixel
1797 coordinates and =triangles= gives us the mesh triangles expressed
1798 in world coordinates. The tactile-sensor-profile gives the
1799 position of each feeler in pixel-space. In order to convert
1800 pixel-space coordinates into world-space coordinates we need
1801 something that takes coordinates on the surface of one triangle
1802 and gives the corresponding coordinates on the surface of another
1803 triangle.
1805 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1806 into any other by a combination of translation, scaling, and
1807 rotation. The affine transformation from one triangle to another
1808 is readily computable if the triangle is expressed in terms of a
1809 $4x4$ matrix.
1811 #+BEGIN_LaTeX
1812 $$
1813 \begin{bmatrix}
1814 x_1 & x_2 & x_3 & n_x \\
1815 y_1 & y_2 & y_3 & n_y \\
1816 z_1 & z_2 & z_3 & n_z \\
1817 1 & 1 & 1 & 1
1818 \end{bmatrix}
1819 $$
1820 #+END_LaTeX
1822 Here, the first three columns of the matrix are the vertices of
1823 the triangle. The last column is the right-handed unit normal of
1824 the triangle.
1826 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1827 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1828 is $T_{2}T_{1}^{-1}$.
1830 The clojure code below recapitulates the formulas above, using
1831 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1832 transformation.
1834 #+caption: Program to interpret triangles as affine transforms.
1835 #+name: triangle-affine
1836 #+begin_listing clojure
1837 #+BEGIN_SRC clojure
1838 (defn triangle->matrix4f
1839 "Converts the triangle into a 4x4 matrix: The first three columns
1840 contain the vertices of the triangle; the last contains the unit
1841 normal of the triangle. The bottom row is filled with 1s."
1842 [#^Triangle t]
1843 (let [mat (Matrix4f.)
1844 [vert-1 vert-2 vert-3]
1845 (mapv #(.get t %) (range 3))
1846 unit-normal (do (.calculateNormal t)(.getNormal t))
1847 vertices [vert-1 vert-2 vert-3 unit-normal]]
1848 (dorun
1849 (for [row (range 4) col (range 3)]
1850 (do
1851 (.set mat col row (.get (vertices row) col))
1852 (.set mat 3 row 1)))) mat))
1854 (defn triangles->affine-transform
1855 "Returns the affine transformation that converts each vertex in the
1856 first triangle into the corresponding vertex in the second
1857 triangle."
1858 [#^Triangle tri-1 #^Triangle tri-2]
1859 (.mult
1860 (triangle->matrix4f tri-2)
1861 (.invert (triangle->matrix4f tri-1))))
1862 #+END_SRC
1863 #+end_listing
1865 *** Triangle Boundaries
1867 For efficiency's sake I will divide the tactile-profile image into
1868 small squares which inscribe each pixel-triangle, then extract the
1869 points which lie inside the triangle and map them to 3D-space using
1870 =triangle-transform= above. To do this I need a function,
1871 =convex-bounds= which finds the smallest box which inscribes a 2D
1872 triangle.
1874 =inside-triangle?= determines whether a point is inside a triangle
1875 in 2D pixel-space.
1877 #+caption: Program to efficiently determine point inclusion
1878 #+caption: in a triangle.
1879 #+name: in-triangle
1880 #+begin_listing clojure
1881 #+BEGIN_SRC clojure
1882 (defn convex-bounds
1883 "Returns the smallest square containing the given vertices, as a
1884 vector of integers [left top width height]."
1885 [verts]
1886 (let [xs (map first verts)
1887 ys (map second verts)
1888 x0 (Math/floor (apply min xs))
1889 y0 (Math/floor (apply min ys))
1890 x1 (Math/ceil (apply max xs))
1891 y1 (Math/ceil (apply max ys))]
1892 [x0 y0 (- x1 x0) (- y1 y0)]))
1894 (defn same-side?
1895 "Given the points p1 and p2 and the reference point ref, is point p
1896 on the same side of the line that goes through p1 and p2 as ref is?"
1897 [p1 p2 ref p]
1898 (<=
1900 (.dot
1901 (.cross (.subtract p2 p1) (.subtract p p1))
1902 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1904 (defn inside-triangle?
1905 "Is the point inside the triangle?"
1906 {:author "Dylan Holmes"}
1907 [#^Triangle tri #^Vector3f p]
1908 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1909 (and
1910 (same-side? vert-1 vert-2 vert-3 p)
1911 (same-side? vert-2 vert-3 vert-1 p)
1912 (same-side? vert-3 vert-1 vert-2 p))))
1913 #+END_SRC
1914 #+end_listing
1916 *** Feeler Coordinates
1918 The triangle-related functions above make short work of
1919 calculating the positions and orientations of each feeler in
1920 world-space.
1922 #+caption: Program to get the coordinates of ``feelers '' in
1923 #+caption: both world and UV-coordinates.
1924 #+name: feeler-coordinates
1925 #+begin_listing clojure
1926 #+BEGIN_SRC clojure
1927 (defn feeler-pixel-coords
1928 "Returns the coordinates of the feelers in pixel space in lists, one
1929 list for each triangle, ordered in the same way as (triangles) and
1930 (pixel-triangles)."
1931 [#^Geometry geo image]
1932 (map
1933 (fn [pixel-triangle]
1934 (filter
1935 (fn [coord]
1936 (inside-triangle? (->triangle pixel-triangle)
1937 (->vector3f coord)))
1938 (white-coordinates image (convex-bounds pixel-triangle))))
1939 (pixel-triangles geo image)))
1941 (defn feeler-world-coords
1942 "Returns the coordinates of the feelers in world space in lists, one
1943 list for each triangle, ordered in the same way as (triangles) and
1944 (pixel-triangles)."
1945 [#^Geometry geo image]
1946 (let [transforms
1947 (map #(triangles->affine-transform
1948 (->triangle %1) (->triangle %2))
1949 (pixel-triangles geo image)
1950 (triangles geo))]
1951 (map (fn [transform coords]
1952 (map #(.mult transform (->vector3f %)) coords))
1953 transforms (feeler-pixel-coords geo image))))
1954 #+END_SRC
1955 #+end_listing
1957 #+caption: Program to get the position of the base and tip of
1958 #+caption: each ``feeler''
1959 #+name: feeler-tips
1960 #+begin_listing clojure
1961 #+BEGIN_SRC clojure
1962 (defn feeler-origins
1963 "The world space coordinates of the root of each feeler."
1964 [#^Geometry geo image]
1965 (reduce concat (feeler-world-coords geo image)))
1967 (defn feeler-tips
1968 "The world space coordinates of the tip of each feeler."
1969 [#^Geometry geo image]
1970 (let [world-coords (feeler-world-coords geo image)
1971 normals
1972 (map
1973 (fn [triangle]
1974 (.calculateNormal triangle)
1975 (.clone (.getNormal triangle)))
1976 (map ->triangle (triangles geo)))]
1978 (mapcat (fn [origins normal]
1979 (map #(.add % normal) origins))
1980 world-coords normals)))
1982 (defn touch-topology
1983 [#^Geometry geo image]
1984 (collapse (reduce concat (feeler-pixel-coords geo image))))
1985 #+END_SRC
1986 #+end_listing
1988 *** Simulated Touch
1990 Now that the functions to construct feelers are complete,
1991 =touch-kernel= generates functions to be called from within a
1992 simulation that perform the necessary physics collisions to
1993 collect tactile data, and =touch!= recursively applies it to every
1994 node in the creature.
1996 #+caption: Efficient program to transform a ray from
1997 #+caption: one position to another.
1998 #+name: set-ray
1999 #+begin_listing clojure
2000 #+BEGIN_SRC clojure
2001 (defn set-ray [#^Ray ray #^Matrix4f transform
2002 #^Vector3f origin #^Vector3f tip]
2003 ;; Doing everything locally reduces garbage collection by enough to
2004 ;; be worth it.
2005 (.mult transform origin (.getOrigin ray))
2006 (.mult transform tip (.getDirection ray))
2007 (.subtractLocal (.getDirection ray) (.getOrigin ray))
2008 (.normalizeLocal (.getDirection ray)))
2009 #+END_SRC
2010 #+end_listing
2012 #+caption: This is the core of touch in =CORTEX= each feeler
2013 #+caption: follows the object it is bound to, reporting any
2014 #+caption: collisions that may happen.
2015 #+name: touch-kernel
2016 #+begin_listing clojure
2017 #+BEGIN_SRC clojure
2018 (defn touch-kernel
2019 "Constructs a function which will return tactile sensory data from
2020 'geo when called from inside a running simulation"
2021 [#^Geometry geo]
2022 (if-let
2023 [profile (tactile-sensor-profile geo)]
2024 (let [ray-reference-origins (feeler-origins geo profile)
2025 ray-reference-tips (feeler-tips geo profile)
2026 ray-length (tactile-scale geo)
2027 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
2028 topology (touch-topology geo profile)
2029 correction (float (* ray-length -0.2))]
2030 ;; slight tolerance for very close collisions.
2031 (dorun
2032 (map (fn [origin tip]
2033 (.addLocal origin (.mult (.subtract tip origin)
2034 correction)))
2035 ray-reference-origins ray-reference-tips))
2036 (dorun (map #(.setLimit % ray-length) current-rays))
2037 (fn [node]
2038 (let [transform (.getWorldMatrix geo)]
2039 (dorun
2040 (map (fn [ray ref-origin ref-tip]
2041 (set-ray ray transform ref-origin ref-tip))
2042 current-rays ray-reference-origins
2043 ray-reference-tips))
2044 (vector
2045 topology
2046 (vec
2047 (for [ray current-rays]
2048 (do
2049 (let [results (CollisionResults.)]
2050 (.collideWith node ray results)
2051 (let [touch-objects
2052 (filter #(not (= geo (.getGeometry %)))
2053 results)
2054 limit (.getLimit ray)]
2055 [(if (empty? touch-objects)
2056 limit
2057 (let [response
2058 (apply min (map #(.getDistance %)
2059 touch-objects))]
2060 (FastMath/clamp
2061 (float
2062 (if (> response limit) (float 0.0)
2063 (+ response correction)))
2064 (float 0.0)
2065 limit)))
2066 limit])))))))))))
2067 #+END_SRC
2068 #+end_listing
2070 Armed with the =touch!= function, =CORTEX= becomes capable of
2071 giving creatures a sense of touch. A simple test is to create a
2072 cube that is outfitted with a uniform distribution of touch
2073 sensors. It can feel the ground and any balls that it touches.
2075 #+caption: =CORTEX= interface for creating touch in a simulated
2076 #+caption: creature.
2077 #+name: touch
2078 #+begin_listing clojure
2079 #+BEGIN_SRC clojure
2080 (defn touch!
2081 "Endow the creature with the sense of touch. Returns a sequence of
2082 functions, one for each body part with a tactile-sensor-profile,
2083 each of which when called returns sensory data for that body part."
2084 [#^Node creature]
2085 (filter
2086 (comp not nil?)
2087 (map touch-kernel
2088 (filter #(isa? (class %) Geometry)
2089 (node-seq creature)))))
2090 #+END_SRC
2091 #+end_listing
2093 The tactile-sensor-profile image for the touch cube is a simple
2094 cross with a uniform distribution of touch sensors:
2096 #+caption: The touch profile for the touch-cube. Each pure white
2097 #+caption: pixel defines a touch sensitive feeler.
2098 #+name: touch-cube-uv-map
2099 #+ATTR_LaTeX: :width 7cm
2100 [[./images/touch-profile.png]]
2102 #+caption: The touch cube reacts to cannonballs. The black, red,
2103 #+caption: and white cross on the right is a visual display of
2104 #+caption: the creature's touch. White means that it is feeling
2105 #+caption: something strongly, black is not feeling anything,
2106 #+caption: and gray is in-between. The cube can feel both the
2107 #+caption: floor and the ball. Notice that when the ball causes
2108 #+caption: the cube to tip, that the bottom face can still feel
2109 #+caption: part of the ground.
2110 #+name: touch-cube-uv-map-2
2111 #+ATTR_LaTeX: :width 15cm
2112 [[./images/touch-cube.png]]
2114 ** Proprioception provides knowledge of your own body's position
2116 Close your eyes, and touch your nose with your right index finger.
2117 How did you do it? You could not see your hand, and neither your
2118 hand nor your nose could use the sense of touch to guide the path
2119 of your hand. There are no sound cues, and Taste and Smell
2120 certainly don't provide any help. You know where your hand is
2121 without your other senses because of Proprioception.
2123 Humans can sometimes loose this sense through viral infections or
2124 damage to the spinal cord or brain, and when they do, they loose
2125 the ability to control their own bodies without looking directly at
2126 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2127 Hat]] (\cite{man-wife-hat}), a woman named Christina looses this
2128 sense and has to learn how to move by carefully watching her arms
2129 and legs. She describes proprioception as the "eyes of the body,
2130 the way the body sees itself".
2132 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2133 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2134 positions of each body part by monitoring muscle strain and length.
2136 It's clear that this is a vital sense for fluid, graceful movement.
2137 It's also particularly easy to implement in jMonkeyEngine.
2139 My simulated proprioception calculates the relative angles of each
2140 joint from the rest position defined in the blender file. This
2141 simulates the muscle-spindles and joint capsules. I will deal with
2142 Golgi tendon organs, which calculate muscle strain, in the next
2143 section.
2145 *** Helper functions
2147 =absolute-angle= calculates the angle between two vectors,
2148 relative to a third axis vector. This angle is the number of
2149 radians you have to move counterclockwise around the axis vector
2150 to get from the first to the second vector. It is not commutative
2151 like a normal dot-product angle is.
2153 The purpose of these functions is to build a system of angle
2154 measurement that is biologically plausible.
2156 #+caption: Program to measure angles along a vector
2157 #+name: helpers
2158 #+begin_listing clojure
2159 #+BEGIN_SRC clojure
2160 (defn right-handed?
2161 "true iff the three vectors form a right handed coordinate
2162 system. The three vectors do not have to be normalized or
2163 orthogonal."
2164 [vec1 vec2 vec3]
2165 (pos? (.dot (.cross vec1 vec2) vec3)))
2167 (defn absolute-angle
2168 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2169 [0 (* 2 Math/PI)]."
2170 [vec1 vec2 axis]
2171 (let [angle (.angleBetween vec1 vec2)]
2172 (if (right-handed? vec1 vec2 axis)
2173 angle (- (* 2 Math/PI) angle))))
2174 #+END_SRC
2175 #+end_listing
2177 *** Proprioception Kernel
2179 Given a joint, =proprioception-kernel= produces a function that
2180 calculates the Euler angles between the the objects the joint
2181 connects. The only tricky part here is making the angles relative
2182 to the joint's initial ``straightness''.
2184 #+caption: Program to return biologically reasonable proprioceptive
2185 #+caption: data for each joint.
2186 #+name: proprioception
2187 #+begin_listing clojure
2188 #+BEGIN_SRC clojure
2189 (defn proprioception-kernel
2190 "Returns a function which returns proprioceptive sensory data when
2191 called inside a running simulation."
2192 [#^Node parts #^Node joint]
2193 (let [[obj-a obj-b] (joint-targets parts joint)
2194 joint-rot (.getWorldRotation joint)
2195 x0 (.mult joint-rot Vector3f/UNIT_X)
2196 y0 (.mult joint-rot Vector3f/UNIT_Y)
2197 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2198 (fn []
2199 (let [rot-a (.clone (.getWorldRotation obj-a))
2200 rot-b (.clone (.getWorldRotation obj-b))
2201 x (.mult rot-a x0)
2202 y (.mult rot-a y0)
2203 z (.mult rot-a z0)
2205 X (.mult rot-b x0)
2206 Y (.mult rot-b y0)
2207 Z (.mult rot-b z0)
2208 heading (Math/atan2 (.dot X z) (.dot X x))
2209 pitch (Math/atan2 (.dot X y) (.dot X x))
2211 ;; rotate x-vector back to origin
2212 reverse
2213 (doto (Quaternion.)
2214 (.fromAngleAxis
2215 (.angleBetween X x)
2216 (let [cross (.normalize (.cross X x))]
2217 (if (= 0 (.length cross)) y cross))))
2218 roll (absolute-angle (.mult reverse Y) y x)]
2219 [heading pitch roll]))))
2221 (defn proprioception!
2222 "Endow the creature with the sense of proprioception. Returns a
2223 sequence of functions, one for each child of the \"joints\" node in
2224 the creature, which each report proprioceptive information about
2225 that joint."
2226 [#^Node creature]
2227 ;; extract the body's joints
2228 (let [senses (map (partial proprioception-kernel creature)
2229 (joints creature))]
2230 (fn []
2231 (map #(%) senses))))
2232 #+END_SRC
2233 #+end_listing
2235 =proprioception!= maps =proprioception-kernel= across all the
2236 joints of the creature. It uses the same list of joints that
2237 =joints= uses. Proprioception is the easiest sense to implement in
2238 =CORTEX=, and it will play a crucial role when efficiently
2239 implementing empathy.
2241 #+caption: In the upper right corner, the three proprioceptive
2242 #+caption: angle measurements are displayed. Red is yaw, Green is
2243 #+caption: pitch, and White is roll.
2244 #+name: proprio
2245 #+ATTR_LaTeX: :width 11cm
2246 [[./images/proprio.png]]
2248 ** Muscles contain both sensors and effectors
2250 Surprisingly enough, terrestrial creatures only move by using
2251 torque applied about their joints. There's not a single straight
2252 line of force in the human body at all! (A straight line of force
2253 would correspond to some sort of jet or rocket propulsion.)
2255 In humans, muscles are composed of muscle fibers which can contract
2256 to exert force. The muscle fibers which compose a muscle are
2257 partitioned into discrete groups which are each controlled by a
2258 single alpha motor neuron. A single alpha motor neuron might
2259 control as little as three or as many as one thousand muscle
2260 fibers. When the alpha motor neuron is engaged by the spinal cord,
2261 it activates all of the muscle fibers to which it is attached. The
2262 spinal cord generally engages the alpha motor neurons which control
2263 few muscle fibers before the motor neurons which control many
2264 muscle fibers. This recruitment strategy allows for precise
2265 movements at low strength. The collection of all motor neurons that
2266 control a muscle is called the motor pool. The brain essentially
2267 says "activate 30% of the motor pool" and the spinal cord recruits
2268 motor neurons until 30% are activated. Since the distribution of
2269 power among motor neurons is unequal and recruitment goes from
2270 weakest to strongest, the first 30% of the motor pool might be 5%
2271 of the strength of the muscle.
2273 My simulated muscles follow a similar design: Each muscle is
2274 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2275 the array represents a motor neuron which controls a number of
2276 muscle fibers equal to the value of the entry. Each muscle has a
2277 scalar strength factor which determines the total force the muscle
2278 can exert when all motor neurons are activated. The effector
2279 function for a muscle takes a number to index into the motor pool,
2280 and then "activates" all the motor neurons whose index is lower or
2281 equal to the number. Each motor-neuron will apply force in
2282 proportion to its value in the array. Lower values cause less
2283 force. The lower values can be put at the "beginning" of the 1-D
2284 array to simulate the layout of actual human muscles, which are
2285 capable of more precise movements when exerting less force. Or, the
2286 motor pool can simulate more exotic recruitment strategies which do
2287 not correspond to human muscles.
2289 This 1D array is defined in an image file for ease of
2290 creation/visualization. Here is an example muscle profile image.
2292 #+caption: A muscle profile image that describes the strengths
2293 #+caption: of each motor neuron in a muscle. White is weakest
2294 #+caption: and dark red is strongest. This particular pattern
2295 #+caption: has weaker motor neurons at the beginning, just
2296 #+caption: like human muscle.
2297 #+name: muscle-recruit
2298 #+ATTR_LaTeX: :width 7cm
2299 [[./images/basic-muscle.png]]
2301 *** Muscle meta-data
2303 #+caption: Program to deal with loading muscle data from a blender
2304 #+caption: file's metadata.
2305 #+name: motor-pool
2306 #+begin_listing clojure
2307 #+BEGIN_SRC clojure
2308 (defn muscle-profile-image
2309 "Get the muscle-profile image from the node's blender meta-data."
2310 [#^Node muscle]
2311 (if-let [image (meta-data muscle "muscle")]
2312 (load-image image)))
2314 (defn muscle-strength
2315 "Return the strength of this muscle, or 1 if it is not defined."
2316 [#^Node muscle]
2317 (if-let [strength (meta-data muscle "strength")]
2318 strength 1))
2320 (defn motor-pool
2321 "Return a vector where each entry is the strength of the \"motor
2322 neuron\" at that part in the muscle."
2323 [#^Node muscle]
2324 (let [profile (muscle-profile-image muscle)]
2325 (vec
2326 (let [width (.getWidth profile)]
2327 (for [x (range width)]
2328 (- 255
2329 (bit-and
2330 0x0000FF
2331 (.getRGB profile x 0))))))))
2332 #+END_SRC
2333 #+end_listing
2335 Of note here is =motor-pool= which interprets the muscle-profile
2336 image in a way that allows me to use gradients between white and
2337 red, instead of shades of gray as I've been using for all the
2338 other senses. This is purely an aesthetic touch.
2340 *** Creating muscles
2342 #+caption: This is the core movement function in =CORTEX=, which
2343 #+caption: implements muscles that report on their activation.
2344 #+name: muscle-kernel
2345 #+begin_listing clojure
2346 #+BEGIN_SRC clojure
2347 (defn movement-kernel
2348 "Returns a function which when called with a integer value inside a
2349 running simulation will cause movement in the creature according
2350 to the muscle's position and strength profile. Each function
2351 returns the amount of force applied / max force."
2352 [#^Node creature #^Node muscle]
2353 (let [target (closest-node creature muscle)
2354 axis
2355 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2356 strength (muscle-strength muscle)
2358 pool (motor-pool muscle)
2359 pool-integral (reductions + pool)
2360 forces
2361 (vec (map #(float (* strength (/ % (last pool-integral))))
2362 pool-integral))
2363 control (.getControl target RigidBodyControl)]
2364 ;;(println-repl (.getName target) axis)
2365 (fn [n]
2366 (let [pool-index (max 0 (min n (dec (count pool))))
2367 force (forces pool-index)]
2368 (.applyTorque control (.mult axis force))
2369 (float (/ force strength))))))
2371 (defn movement!
2372 "Endow the creature with the power of movement. Returns a sequence
2373 of functions, each of which accept an integer value and will
2374 activate their corresponding muscle."
2375 [#^Node creature]
2376 (for [muscle (muscles creature)]
2377 (movement-kernel creature muscle)))
2378 #+END_SRC
2379 #+end_listing
2382 =movement-kernel= creates a function that will move the nearest
2383 physical object to the muscle node. The muscle exerts a rotational
2384 force dependent on it's orientation to the object in the blender
2385 file. The function returned by =movement-kernel= is also a sense
2386 function: it returns the percent of the total muscle strength that
2387 is currently being employed. This is analogous to muscle tension
2388 in humans and completes the sense of proprioception begun in the
2389 last section.
2391 ** =CORTEX= brings complex creatures to life!
2393 The ultimate test of =CORTEX= is to create a creature with the full
2394 gamut of senses and put it though its paces.
2396 With all senses enabled, my right hand model looks like an
2397 intricate marionette hand with several strings for each finger:
2399 #+caption: View of the hand model with all sense nodes. You can see
2400 #+caption: the joint, muscle, ear, and eye nodes here.
2401 #+name: hand-nodes-1
2402 #+ATTR_LaTeX: :width 11cm
2403 [[./images/hand-with-all-senses2.png]]
2405 #+caption: An alternate view of the hand.
2406 #+name: hand-nodes-2
2407 #+ATTR_LaTeX: :width 15cm
2408 [[./images/hand-with-all-senses3.png]]
2410 With the hand fully rigged with senses, I can run it though a test
2411 that will test everything.
2413 #+caption: A full test of the hand with all senses. Note especially
2414 #+caption: the interactions the hand has with itself: it feels
2415 #+caption: its own palm and fingers, and when it curls its fingers,
2416 #+caption: it sees them with its eye (which is located in the center
2417 #+caption: of the palm. The red block appears with a pure tone sound.
2418 #+caption: The hand then uses its muscles to launch the cube!
2419 #+name: integration
2420 #+ATTR_LaTeX: :width 16cm
2421 [[./images/integration.png]]
2423 ** =CORTEX= enables many possibilities for further research
2425 Often times, the hardest part of building a system involving
2426 creatures is dealing with physics and graphics. =CORTEX= removes
2427 much of this initial difficulty and leaves researchers free to
2428 directly pursue their ideas. I hope that even undergrads with a
2429 passing curiosity about simulated touch or creature evolution will
2430 be able to use cortex for experimentation. =CORTEX= is a completely
2431 simulated world, and far from being a disadvantage, its simulated
2432 nature enables you to create senses and creatures that would be
2433 impossible to make in the real world.
2435 While not by any means a complete list, here are some paths
2436 =CORTEX= is well suited to help you explore:
2438 - Empathy :: my empathy program leaves many areas for
2439 improvement, among which are using vision to infer
2440 proprioception and looking up sensory experience with imagined
2441 vision, touch, and sound.
2442 - Evolution :: Karl Sims created a rich environment for
2443 simulating the evolution of creatures on a connection
2444 machine. Today, this can be redone and expanded with =CORTEX=
2445 on an ordinary computer.
2446 - Exotic senses :: Cortex enables many fascinating senses that are
2447 not possible to build in the real world. For example,
2448 telekinesis is an interesting avenue to explore. You can also
2449 make a ``semantic'' sense which looks up metadata tags on
2450 objects in the environment the metadata tags might contain
2451 other sensory information.
2452 - Imagination via subworlds :: this would involve a creature with
2453 an effector which creates an entire new sub-simulation where
2454 the creature has direct control over placement/creation of
2455 objects via simulated telekinesis. The creature observes this
2456 sub-world through it's normal senses and uses its observations
2457 to make predictions about its top level world.
2458 - Simulated prescience :: step the simulation forward a few ticks,
2459 gather sensory data, then supply this data for the creature as
2460 one of its actual senses. The cost of prescience is slowing
2461 the simulation down by a factor proportional to however far
2462 you want the entities to see into the future. What happens
2463 when two evolved creatures that can each see into the future
2464 fight each other?
2465 - Swarm creatures :: Program a group of creatures that cooperate
2466 with each other. Because the creatures would be simulated, you
2467 could investigate computationally complex rules of behavior
2468 which still, from the group's point of view, would happen in
2469 ``real time''. Interactions could be as simple as cellular
2470 organisms communicating via flashing lights, or as complex as
2471 humanoids completing social tasks, etc.
2472 - =HACKER= for writing muscle-control programs :: Presented with
2473 low-level muscle control/ sense API, generate higher level
2474 programs for accomplishing various stated goals. Example goals
2475 might be "extend all your fingers" or "move your hand into the
2476 area with blue light" or "decrease the angle of this joint".
2477 It would be like Sussman's HACKER, except it would operate
2478 with much more data in a more realistic world. Start off with
2479 "calisthenics" to develop subroutines over the motor control
2480 API. This would be the "spinal chord" of a more intelligent
2481 creature. The low level programming code might be a turning
2482 machine that could develop programs to iterate over a "tape"
2483 where each entry in the tape could control recruitment of the
2484 fibers in a muscle.
2485 - Sense fusion :: There is much work to be done on sense
2486 integration -- building up a coherent picture of the world and
2487 the things in it with =CORTEX= as a base, you can explore
2488 concepts like self-organizing maps or cross modal clustering
2489 in ways that have never before been tried.
2490 - Inverse kinematics :: experiments in sense guided motor control
2491 are easy given =CORTEX='s support -- you can get right to the
2492 hard control problems without worrying about physics or
2493 senses.
2495 * =EMPATH=: action recognition in a simulated worm
2497 Here I develop a computational model of empathy, using =CORTEX= as a
2498 base. Empathy in this context is the ability to observe another
2499 creature and infer what sorts of sensations that creature is
2500 feeling. My empathy algorithm involves multiple phases. First is
2501 free-play, where the creature moves around and gains sensory
2502 experience. From this experience I construct a representation of the
2503 creature's sensory state space, which I call \Phi-space. Using
2504 \Phi-space, I construct an efficient function which takes the
2505 limited data that comes from observing another creature and enriches
2506 it full compliment of imagined sensory data. I can then use the
2507 imagined sensory data to recognize what the observed creature is
2508 doing and feeling, using straightforward embodied action predicates.
2509 This is all demonstrated with using a simple worm-like creature, and
2510 recognizing worm-actions based on limited data.
2512 #+caption: Here is the worm with which we will be working.
2513 #+caption: It is composed of 5 segments. Each segment has a
2514 #+caption: pair of extensor and flexor muscles. Each of the
2515 #+caption: worm's four joints is a hinge joint which allows
2516 #+caption: about 30 degrees of rotation to either side. Each segment
2517 #+caption: of the worm is touch-capable and has a uniform
2518 #+caption: distribution of touch sensors on each of its faces.
2519 #+caption: Each joint has a proprioceptive sense to detect
2520 #+caption: relative positions. The worm segments are all the
2521 #+caption: same except for the first one, which has a much
2522 #+caption: higher weight than the others to allow for easy
2523 #+caption: manual motor control.
2524 #+name: basic-worm-view
2525 #+ATTR_LaTeX: :width 10cm
2526 [[./images/basic-worm-view.png]]
2528 #+caption: Program for reading a worm from a blender file and
2529 #+caption: outfitting it with the senses of proprioception,
2530 #+caption: touch, and the ability to move, as specified in the
2531 #+caption: blender file.
2532 #+name: get-worm
2533 #+begin_listing clojure
2534 #+begin_src clojure
2535 (defn worm []
2536 (let [model (load-blender-model "Models/worm/worm.blend")]
2537 {:body (doto model (body!))
2538 :touch (touch! model)
2539 :proprioception (proprioception! model)
2540 :muscles (movement! model)}))
2541 #+end_src
2542 #+end_listing
2544 ** Embodiment factors action recognition into manageable parts
2546 Using empathy, I divide the problem of action recognition into a
2547 recognition process expressed in the language of a full compliment
2548 of senses, and an imaginative process that generates full sensory
2549 data from partial sensory data. Splitting the action recognition
2550 problem in this manner greatly reduces the total amount of work to
2551 recognize actions: The imaginative process is mostly just matching
2552 previous experience, and the recognition process gets to use all
2553 the senses to directly describe any action.
2555 ** Action recognition is easy with a full gamut of senses
2557 Embodied representations using multiple senses such as touch,
2558 proprioception, and muscle tension turns out be be exceedingly
2559 efficient at describing body-centered actions. It is the ``right
2560 language for the job''. For example, it takes only around 5 lines
2561 of LISP code to describe the action of ``curling'' using embodied
2562 primitives. It takes about 10 lines to describe the seemingly
2563 complicated action of wiggling.
2565 The following action predicates each take a stream of sensory
2566 experience, observe however much of it they desire, and decide
2567 whether the worm is doing the action they describe. =curled?=
2568 relies on proprioception, =resting?= relies on touch, =wiggling?=
2569 relies on a Fourier analysis of muscle contraction, and
2570 =grand-circle?= relies on touch and reuses =curled?= as a guard.
2572 #+caption: Program for detecting whether the worm is curled. This is the
2573 #+caption: simplest action predicate, because it only uses the last frame
2574 #+caption: of sensory experience, and only uses proprioceptive data. Even
2575 #+caption: this simple predicate, however, is automatically frame
2576 #+caption: independent and ignores vermopomorphic differences such as
2577 #+caption: worm textures and colors.
2578 #+name: curled
2579 #+begin_listing clojure
2580 #+begin_src clojure
2581 (defn curled?
2582 "Is the worm curled up?"
2583 [experiences]
2584 (every?
2585 (fn [[_ _ bend]]
2586 (> (Math/sin bend) 0.64))
2587 (:proprioception (peek experiences))))
2588 #+end_src
2589 #+end_listing
2591 #+caption: Program for summarizing the touch information in a patch
2592 #+caption: of skin.
2593 #+name: touch-summary
2594 #+begin_listing clojure
2595 #+begin_src clojure
2596 (defn contact
2597 "Determine how much contact a particular worm segment has with
2598 other objects. Returns a value between 0 and 1, where 1 is full
2599 contact and 0 is no contact."
2600 [touch-region [coords contact :as touch]]
2601 (-> (zipmap coords contact)
2602 (select-keys touch-region)
2603 (vals)
2604 (#(map first %))
2605 (average)
2606 (* 10)
2607 (- 1)
2608 (Math/abs)))
2609 #+end_src
2610 #+end_listing
2613 #+caption: Program for detecting whether the worm is at rest. This program
2614 #+caption: uses a summary of the tactile information from the underbelly
2615 #+caption: of the worm, and is only true if every segment is touching the
2616 #+caption: floor. Note that this function contains no references to
2617 #+caption: proprioception at all.
2618 #+name: resting
2619 #+begin_listing clojure
2620 #+begin_src clojure
2621 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2623 (defn resting?
2624 "Is the worm resting on the ground?"
2625 [experiences]
2626 (every?
2627 (fn [touch-data]
2628 (< 0.9 (contact worm-segment-bottom touch-data)))
2629 (:touch (peek experiences))))
2630 #+end_src
2631 #+end_listing
2633 #+caption: Program for detecting whether the worm is curled up into a
2634 #+caption: full circle. Here the embodied approach begins to shine, as
2635 #+caption: I am able to both use a previous action predicate (=curled?=)
2636 #+caption: as well as the direct tactile experience of the head and tail.
2637 #+name: grand-circle
2638 #+begin_listing clojure
2639 #+begin_src clojure
2640 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2642 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2644 (defn grand-circle?
2645 "Does the worm form a majestic circle (one end touching the other)?"
2646 [experiences]
2647 (and (curled? experiences)
2648 (let [worm-touch (:touch (peek experiences))
2649 tail-touch (worm-touch 0)
2650 head-touch (worm-touch 4)]
2651 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2652 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2653 #+end_src
2654 #+end_listing
2657 #+caption: Program for detecting whether the worm has been wiggling for
2658 #+caption: the last few frames. It uses a Fourier analysis of the muscle
2659 #+caption: contractions of the worm's tail to determine wiggling. This is
2660 #+caption: significant because there is no particular frame that clearly
2661 #+caption: indicates that the worm is wiggling --- only when multiple frames
2662 #+caption: are analyzed together is the wiggling revealed. Defining
2663 #+caption: wiggling this way also gives the worm an opportunity to learn
2664 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2665 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2666 #+caption: from actual wiggling, but this definition gives it to us for free.
2667 #+name: wiggling
2668 #+begin_listing clojure
2669 #+begin_src clojure
2670 (defn fft [nums]
2671 (map
2672 #(.getReal %)
2673 (.transform
2674 (FastFourierTransformer. DftNormalization/STANDARD)
2675 (double-array nums) TransformType/FORWARD)))
2677 (def indexed (partial map-indexed vector))
2679 (defn max-indexed [s]
2680 (first (sort-by (comp - second) (indexed s))))
2682 (defn wiggling?
2683 "Is the worm wiggling?"
2684 [experiences]
2685 (let [analysis-interval 0x40]
2686 (when (> (count experiences) analysis-interval)
2687 (let [a-flex 3
2688 a-ex 2
2689 muscle-activity
2690 (map :muscle (vector:last-n experiences analysis-interval))
2691 base-activity
2692 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2693 (= 2
2694 (first
2695 (max-indexed
2696 (map #(Math/abs %)
2697 (take 20 (fft base-activity))))))))))
2698 #+end_src
2699 #+end_listing
2701 With these action predicates, I can now recognize the actions of
2702 the worm while it is moving under my control and I have access to
2703 all the worm's senses.
2705 #+caption: Use the action predicates defined earlier to report on
2706 #+caption: what the worm is doing while in simulation.
2707 #+name: report-worm-activity
2708 #+begin_listing clojure
2709 #+begin_src clojure
2710 (defn debug-experience
2711 [experiences text]
2712 (cond
2713 (grand-circle? experiences) (.setText text "Grand Circle")
2714 (curled? experiences) (.setText text "Curled")
2715 (wiggling? experiences) (.setText text "Wiggling")
2716 (resting? experiences) (.setText text "Resting")))
2717 #+end_src
2718 #+end_listing
2720 #+caption: Using =debug-experience=, the body-centered predicates
2721 #+caption: work together to classify the behavior of the worm.
2722 #+caption: the predicates are operating with access to the worm's
2723 #+caption: full sensory data.
2724 #+name: basic-worm-view
2725 #+ATTR_LaTeX: :width 10cm
2726 [[./images/worm-identify-init.png]]
2728 These action predicates satisfy the recognition requirement of an
2729 empathic recognition system. There is power in the simplicity of
2730 the action predicates. They describe their actions without getting
2731 confused in visual details of the worm. Each one is frame
2732 independent, but more than that, they are each independent of
2733 irrelevant visual details of the worm and the environment. They
2734 will work regardless of whether the worm is a different color or
2735 heavily textured, or if the environment has strange lighting.
2737 The trick now is to make the action predicates work even when the
2738 sensory data on which they depend is absent. If I can do that, then
2739 I will have gained much,
2741 ** \Phi-space describes the worm's experiences
2743 As a first step towards building empathy, I need to gather all of
2744 the worm's experiences during free play. I use a simple vector to
2745 store all the experiences.
2747 Each element of the experience vector exists in the vast space of
2748 all possible worm-experiences. Most of this vast space is actually
2749 unreachable due to physical constraints of the worm's body. For
2750 example, the worm's segments are connected by hinge joints that put
2751 a practical limit on the worm's range of motions without limiting
2752 its degrees of freedom. Some groupings of senses are impossible;
2753 the worm can not be bent into a circle so that its ends are
2754 touching and at the same time not also experience the sensation of
2755 touching itself.
2757 As the worm moves around during free play and its experience vector
2758 grows larger, the vector begins to define a subspace which is all
2759 the sensations the worm can practically experience during normal
2760 operation. I call this subspace \Phi-space, short for
2761 physical-space. The experience vector defines a path through
2762 \Phi-space. This path has interesting properties that all derive
2763 from physical embodiment. The proprioceptive components are
2764 completely smooth, because in order for the worm to move from one
2765 position to another, it must pass through the intermediate
2766 positions. The path invariably forms loops as actions are repeated.
2767 Finally and most importantly, proprioception actually gives very
2768 strong inference about the other senses. For example, when the worm
2769 is flat, you can infer that it is touching the ground and that its
2770 muscles are not active, because if the muscles were active, the
2771 worm would be moving and would not be perfectly flat. In order to
2772 stay flat, the worm has to be touching the ground, or it would
2773 again be moving out of the flat position due to gravity. If the
2774 worm is positioned in such a way that it interacts with itself,
2775 then it is very likely to be feeling the same tactile feelings as
2776 the last time it was in that position, because it has the same body
2777 as then. If you observe multiple frames of proprioceptive data,
2778 then you can become increasingly confident about the exact
2779 activations of the worm's muscles, because it generally takes a
2780 unique combination of muscle contractions to transform the worm's
2781 body along a specific path through \Phi-space.
2783 There is a simple way of taking \Phi-space and the total ordering
2784 provided by an experience vector and reliably inferring the rest of
2785 the senses.
2787 ** Empathy is the process of tracing though \Phi-space
2789 Here is the core of a basic empathy algorithm, starting with an
2790 experience vector:
2792 First, group the experiences into tiered proprioceptive bins. I use
2793 powers of 10 and 3 bins, and the smallest bin has an approximate
2794 size of 0.001 radians in all proprioceptive dimensions.
2796 Then, given a sequence of proprioceptive input, generate a set of
2797 matching experience records for each input, using the tiered
2798 proprioceptive bins.
2800 Finally, to infer sensory data, select the longest consecutive chain
2801 of experiences. Consecutive experience means that the experiences
2802 appear next to each other in the experience vector.
2804 This algorithm has three advantages:
2806 1. It's simple
2808 3. It's very fast -- retrieving possible interpretations takes
2809 constant time. Tracing through chains of interpretations takes
2810 time proportional to the average number of experiences in a
2811 proprioceptive bin. Redundant experiences in \Phi-space can be
2812 merged to save computation.
2814 2. It protects from wrong interpretations of transient ambiguous
2815 proprioceptive data. For example, if the worm is flat for just
2816 an instant, this flatness will not be interpreted as implying
2817 that the worm has its muscles relaxed, since the flatness is
2818 part of a longer chain which includes a distinct pattern of
2819 muscle activation. Markov chains or other memoryless statistical
2820 models that operate on individual frames may very well make this
2821 mistake.
2823 #+caption: Program to convert an experience vector into a
2824 #+caption: proprioceptively binned lookup function.
2825 #+name: bin
2826 #+begin_listing clojure
2827 #+begin_src clojure
2828 (defn bin [digits]
2829 (fn [angles]
2830 (->> angles
2831 (flatten)
2832 (map (juxt #(Math/sin %) #(Math/cos %)))
2833 (flatten)
2834 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2836 (defn gen-phi-scan
2837 "Nearest-neighbors with binning. Only returns a result if
2838 the proprioceptive data is within 10% of a previously recorded
2839 result in all dimensions."
2840 [phi-space]
2841 (let [bin-keys (map bin [3 2 1])
2842 bin-maps
2843 (map (fn [bin-key]
2844 (group-by
2845 (comp bin-key :proprioception phi-space)
2846 (range (count phi-space)))) bin-keys)
2847 lookups (map (fn [bin-key bin-map]
2848 (fn [proprio] (bin-map (bin-key proprio))))
2849 bin-keys bin-maps)]
2850 (fn lookup [proprio-data]
2851 (set (some #(% proprio-data) lookups)))))
2852 #+end_src
2853 #+end_listing
2855 #+caption: =longest-thread= finds the longest path of consecutive
2856 #+caption: experiences to explain proprioceptive worm data.
2857 #+name: phi-space-history-scan
2858 #+ATTR_LaTeX: :width 10cm
2859 [[./images/aurellem-gray.png]]
2861 =longest-thread= infers sensory data by stitching together pieces
2862 from previous experience. It prefers longer chains of previous
2863 experience to shorter ones. For example, during training the worm
2864 might rest on the ground for one second before it performs its
2865 exercises. If during recognition the worm rests on the ground for
2866 five seconds, =longest-thread= will accommodate this five second
2867 rest period by looping the one second rest chain five times.
2869 =longest-thread= takes time proportional to the average number of
2870 entries in a proprioceptive bin, because for each element in the
2871 starting bin it performs a series of set lookups in the preceding
2872 bins. If the total history is limited, then this is only a constant
2873 multiple times the number of entries in the starting bin. This
2874 analysis also applies even if the action requires multiple longest
2875 chains -- it's still the average number of entries in a
2876 proprioceptive bin times the desired chain length. Because
2877 =longest-thread= is so efficient and simple, I can interpret
2878 worm-actions in real time.
2880 #+caption: Program to calculate empathy by tracing though \Phi-space
2881 #+caption: and finding the longest (ie. most coherent) interpretation
2882 #+caption: of the data.
2883 #+name: longest-thread
2884 #+begin_listing clojure
2885 #+begin_src clojure
2886 (defn longest-thread
2887 "Find the longest thread from phi-index-sets. The index sets should
2888 be ordered from most recent to least recent."
2889 [phi-index-sets]
2890 (loop [result '()
2891 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2892 (if (empty? phi-index-sets)
2893 (vec result)
2894 (let [threads
2895 (for [thread-base thread-bases]
2896 (loop [thread (list thread-base)
2897 remaining remaining]
2898 (let [next-index (dec (first thread))]
2899 (cond (empty? remaining) thread
2900 (contains? (first remaining) next-index)
2901 (recur
2902 (cons next-index thread) (rest remaining))
2903 :else thread))))
2904 longest-thread
2905 (reduce (fn [thread-a thread-b]
2906 (if (> (count thread-a) (count thread-b))
2907 thread-a thread-b))
2908 '(nil)
2909 threads)]
2910 (recur (concat longest-thread result)
2911 (drop (count longest-thread) phi-index-sets))))))
2912 #+end_src
2913 #+end_listing
2915 There is one final piece, which is to replace missing sensory data
2916 with a best-guess estimate. While I could fill in missing data by
2917 using a gradient over the closest known sensory data points,
2918 averages can be misleading. It is certainly possible to create an
2919 impossible sensory state by averaging two possible sensory states.
2920 Therefore, I simply replicate the most recent sensory experience to
2921 fill in the gaps.
2923 #+caption: Fill in blanks in sensory experience by replicating the most
2924 #+caption: recent experience.
2925 #+name: infer-nils
2926 #+begin_listing clojure
2927 #+begin_src clojure
2928 (defn infer-nils
2929 "Replace nils with the next available non-nil element in the
2930 sequence, or barring that, 0."
2931 [s]
2932 (loop [i (dec (count s))
2933 v (transient s)]
2934 (if (zero? i) (persistent! v)
2935 (if-let [cur (v i)]
2936 (if (get v (dec i) 0)
2937 (recur (dec i) v)
2938 (recur (dec i) (assoc! v (dec i) cur)))
2939 (recur i (assoc! v i 0))))))
2940 #+end_src
2941 #+end_listing
2943 ** =EMPATH= recognizes actions efficiently
2945 To use =EMPATH= with the worm, I first need to gather a set of
2946 experiences from the worm that includes the actions I want to
2947 recognize. The =generate-phi-space= program (listing
2948 \ref{generate-phi-space} runs the worm through a series of
2949 exercises and gatherers those experiences into a vector. The
2950 =do-all-the-things= program is a routine expressed in a simple
2951 muscle contraction script language for automated worm control. It
2952 causes the worm to rest, curl, and wiggle over about 700 frames
2953 (approx. 11 seconds).
2955 #+caption: Program to gather the worm's experiences into a vector for
2956 #+caption: further processing. The =motor-control-program= line uses
2957 #+caption: a motor control script that causes the worm to execute a series
2958 #+caption: of ``exercises'' that include all the action predicates.
2959 #+name: generate-phi-space
2960 #+begin_listing clojure
2961 #+begin_src clojure
2962 (def do-all-the-things
2963 (concat
2964 curl-script
2965 [[300 :d-ex 40]
2966 [320 :d-ex 0]]
2967 (shift-script 280 (take 16 wiggle-script))))
2969 (defn generate-phi-space []
2970 (let [experiences (atom [])]
2971 (run-world
2972 (apply-map
2973 worm-world
2974 (merge
2975 (worm-world-defaults)
2976 {:end-frame 700
2977 :motor-control
2978 (motor-control-program worm-muscle-labels do-all-the-things)
2979 :experiences experiences})))
2980 @experiences))
2981 #+end_src
2982 #+end_listing
2984 #+caption: Use longest thread and a phi-space generated from a short
2985 #+caption: exercise routine to interpret actions during free play.
2986 #+name: empathy-debug
2987 #+begin_listing clojure
2988 #+begin_src clojure
2989 (defn init []
2990 (def phi-space (generate-phi-space))
2991 (def phi-scan (gen-phi-scan phi-space)))
2993 (defn empathy-demonstration []
2994 (let [proprio (atom ())]
2995 (fn
2996 [experiences text]
2997 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
2998 (swap! proprio (partial cons phi-indices))
2999 (let [exp-thread (longest-thread (take 300 @proprio))
3000 empathy (mapv phi-space (infer-nils exp-thread))]
3001 (println-repl (vector:last-n exp-thread 22))
3002 (cond
3003 (grand-circle? empathy) (.setText text "Grand Circle")
3004 (curled? empathy) (.setText text "Curled")
3005 (wiggling? empathy) (.setText text "Wiggling")
3006 (resting? empathy) (.setText text "Resting")
3007 :else (.setText text "Unknown")))))))
3009 (defn empathy-experiment [record]
3010 (.start (worm-world :experience-watch (debug-experience-phi)
3011 :record record :worm worm*)))
3012 #+end_src
3013 #+end_listing
3015 The result of running =empathy-experiment= is that the system is
3016 generally able to interpret worm actions using the action-predicates
3017 on simulated sensory data just as well as with actual data. Figure
3018 \ref{empathy-debug-image} was generated using =empathy-experiment=:
3020 #+caption: From only proprioceptive data, =EMPATH= was able to infer
3021 #+caption: the complete sensory experience and classify four poses
3022 #+caption: (The last panel shows a composite image of /wiggling/,
3023 #+caption: a dynamic pose.)
3024 #+name: empathy-debug-image
3025 #+ATTR_LaTeX: :width 10cm :placement [H]
3026 [[./images/empathy-1.png]]
3028 One way to measure the performance of =EMPATH= is to compare the
3029 suitability of the imagined sense experience to trigger the same
3030 action predicates as the real sensory experience.
3032 #+caption: Determine how closely empathy approximates actual
3033 #+caption: sensory data.
3034 #+name: test-empathy-accuracy
3035 #+begin_listing clojure
3036 #+begin_src clojure
3037 (def worm-action-label
3038 (juxt grand-circle? curled? wiggling?))
3040 (defn compare-empathy-with-baseline [matches]
3041 (let [proprio (atom ())]
3042 (fn
3043 [experiences text]
3044 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3045 (swap! proprio (partial cons phi-indices))
3046 (let [exp-thread (longest-thread (take 300 @proprio))
3047 empathy (mapv phi-space (infer-nils exp-thread))
3048 experience-matches-empathy
3049 (= (worm-action-label experiences)
3050 (worm-action-label empathy))]
3051 (println-repl experience-matches-empathy)
3052 (swap! matches #(conj % experience-matches-empathy)))))))
3054 (defn accuracy [v]
3055 (float (/ (count (filter true? v)) (count v))))
3057 (defn test-empathy-accuracy []
3058 (let [res (atom [])]
3059 (run-world
3060 (worm-world :experience-watch
3061 (compare-empathy-with-baseline res)
3062 :worm worm*))
3063 (accuracy @res)))
3064 #+end_src
3065 #+end_listing
3067 Running =test-empathy-accuracy= using the very short exercise
3068 program defined in listing \ref{generate-phi-space}, and then doing
3069 a similar pattern of activity manually yields an accuracy of around
3070 73%. This is based on very limited worm experience. By training the
3071 worm for longer, the accuracy dramatically improves.
3073 #+caption: Program to generate \Phi-space using manual training.
3074 #+name: manual-phi-space
3075 #+begin_listing clojure
3076 #+begin_src clojure
3077 (defn init-interactive []
3078 (def phi-space
3079 (let [experiences (atom [])]
3080 (run-world
3081 (apply-map
3082 worm-world
3083 (merge
3084 (worm-world-defaults)
3085 {:experiences experiences})))
3086 @experiences))
3087 (def phi-scan (gen-phi-scan phi-space)))
3088 #+end_src
3089 #+end_listing
3091 After about 1 minute of manual training, I was able to achieve 95%
3092 accuracy on manual testing of the worm using =init-interactive= and
3093 =test-empathy-accuracy=. The majority of errors are near the
3094 boundaries of transitioning from one type of action to another.
3095 During these transitions the exact label for the action is more open
3096 to interpretation, and disagreement between empathy and experience
3097 is more excusable.
3099 ** Digression: Learn touch sensor layout through free play
3101 In the previous section I showed how to compute actions in terms of
3102 body-centered predicates which relied on the average touch
3103 activation of pre-defined regions of the worm's skin. What if,
3104 instead of receiving touch pre-grouped into the six faces of each
3105 worm segment, the true topology of the worm's skin was unknown?
3106 This is more similar to how a nerve fiber bundle might be
3107 arranged. While two fibers that are close in a nerve bundle /might/
3108 correspond to two touch sensors that are close together on the
3109 skin, the process of taking a complicated surface and forcing it
3110 into essentially a circle requires some cuts and rearrangements.
3112 In this section I show how to automatically learn the skin-topology of
3113 a worm segment by free exploration. As the worm rolls around on the
3114 floor, large sections of its surface get activated. If the worm has
3115 stopped moving, then whatever region of skin that is touching the
3116 floor is probably an important region, and should be recorded.
3118 #+caption: Program to detect whether the worm is in a resting state
3119 #+caption: with one face touching the floor.
3120 #+name: pure-touch
3121 #+begin_listing clojure
3122 #+begin_src clojure
3123 (def full-contact [(float 0.0) (float 0.1)])
3125 (defn pure-touch?
3126 "This is worm specific code to determine if a large region of touch
3127 sensors is either all on or all off."
3128 [[coords touch :as touch-data]]
3129 (= (set (map first touch)) (set full-contact)))
3130 #+end_src
3131 #+end_listing
3133 After collecting these important regions, there will many nearly
3134 similar touch regions. While for some purposes the subtle
3135 differences between these regions will be important, for my
3136 purposes I collapse them into mostly non-overlapping sets using
3137 =remove-similar= in listing \ref{remove-similar}
3139 #+caption: Program to take a list of sets of points and ``collapse them''
3140 #+caption: so that the remaining sets in the list are significantly
3141 #+caption: different from each other. Prefer smaller sets to larger ones.
3142 #+name: remove-similar
3143 #+begin_listing clojure
3144 #+begin_src clojure
3145 (defn remove-similar
3146 [coll]
3147 (loop [result () coll (sort-by (comp - count) coll)]
3148 (if (empty? coll) result
3149 (let [[x & xs] coll
3150 c (count x)]
3151 (if (some
3152 (fn [other-set]
3153 (let [oc (count other-set)]
3154 (< (- (count (union other-set x)) c) (* oc 0.1))))
3155 xs)
3156 (recur result xs)
3157 (recur (cons x result) xs))))))
3158 #+end_src
3159 #+end_listing
3161 Actually running this simulation is easy given =CORTEX='s facilities.
3163 #+caption: Collect experiences while the worm moves around. Filter the touch
3164 #+caption: sensations by stable ones, collapse similar ones together,
3165 #+caption: and report the regions learned.
3166 #+name: learn-touch
3167 #+begin_listing clojure
3168 #+begin_src clojure
3169 (defn learn-touch-regions []
3170 (let [experiences (atom [])
3171 world (apply-map
3172 worm-world
3173 (assoc (worm-segment-defaults)
3174 :experiences experiences))]
3175 (run-world world)
3176 (->>
3177 @experiences
3178 (drop 175)
3179 ;; access the single segment's touch data
3180 (map (comp first :touch))
3181 ;; only deal with "pure" touch data to determine surfaces
3182 (filter pure-touch?)
3183 ;; associate coordinates with touch values
3184 (map (partial apply zipmap))
3185 ;; select those regions where contact is being made
3186 (map (partial group-by second))
3187 (map #(get % full-contact))
3188 (map (partial map first))
3189 ;; remove redundant/subset regions
3190 (map set)
3191 remove-similar)))
3193 (defn learn-and-view-touch-regions []
3194 (map view-touch-region
3195 (learn-touch-regions)))
3196 #+end_src
3197 #+end_listing
3199 The only thing remaining to define is the particular motion the worm
3200 must take. I accomplish this with a simple motor control program.
3202 #+caption: Motor control program for making the worm roll on the ground.
3203 #+caption: This could also be replaced with random motion.
3204 #+name: worm-roll
3205 #+begin_listing clojure
3206 #+begin_src clojure
3207 (defn touch-kinesthetics []
3208 [[170 :lift-1 40]
3209 [190 :lift-1 19]
3210 [206 :lift-1 0]
3212 [400 :lift-2 40]
3213 [410 :lift-2 0]
3215 [570 :lift-2 40]
3216 [590 :lift-2 21]
3217 [606 :lift-2 0]
3219 [800 :lift-1 30]
3220 [809 :lift-1 0]
3222 [900 :roll-2 40]
3223 [905 :roll-2 20]
3224 [910 :roll-2 0]
3226 [1000 :roll-2 40]
3227 [1005 :roll-2 20]
3228 [1010 :roll-2 0]
3230 [1100 :roll-2 40]
3231 [1105 :roll-2 20]
3232 [1110 :roll-2 0]
3233 ])
3234 #+end_src
3235 #+end_listing
3238 #+caption: The small worm rolls around on the floor, driven
3239 #+caption: by the motor control program in listing \ref{worm-roll}.
3240 #+name: worm-roll
3241 #+ATTR_LaTeX: :width 12cm
3242 [[./images/worm-roll.png]]
3245 #+caption: After completing its adventures, the worm now knows
3246 #+caption: how its touch sensors are arranged along its skin. These
3247 #+caption: are the regions that were deemed important by
3248 #+caption: =learn-touch-regions=. Note that the worm has discovered
3249 #+caption: that it has six sides.
3250 #+name: worm-touch-map
3251 #+ATTR_LaTeX: :width 12cm
3252 [[./images/touch-learn.png]]
3254 While simple, =learn-touch-regions= exploits regularities in both
3255 the worm's physiology and the worm's environment to correctly
3256 deduce that the worm has six sides. Note that =learn-touch-regions=
3257 would work just as well even if the worm's touch sense data were
3258 completely scrambled. The cross shape is just for convenience. This
3259 example justifies the use of pre-defined touch regions in =EMPATH=.
3261 * Contributions
3263 In this thesis you have seen the =CORTEX= system, a complete
3264 environment for creating simulated creatures. You have seen how to
3265 implement five senses: touch, proprioception, hearing, vision, and
3266 muscle tension. You have seen how to create new creatures using
3267 blender, a 3D modeling tool. I hope that =CORTEX= will be useful in
3268 further research projects. To this end I have included the full
3269 source to =CORTEX= along with a large suite of tests and examples. I
3270 have also created a user guide for =CORTEX= which is included in an
3271 appendix to this thesis.
3273 You have also seen how I used =CORTEX= as a platform to attach the
3274 /action recognition/ problem, which is the problem of recognizing
3275 actions in video. You saw a simple system called =EMPATH= which
3276 identifies actions by first describing actions in a body-centered,
3277 rich sense language, then inferring a full range of sensory
3278 experience from limited data using previous experience gained from
3279 free play.
3281 As a minor digression, you also saw how I used =CORTEX= to enable a
3282 tiny worm to discover the topology of its skin simply by rolling on
3283 the ground.
3285 In conclusion, the main contributions of this thesis are:
3287 - =CORTEX=, a comprehensive platform for embodied AI experiments.
3288 =CORTEX= supports many features lacking in other systems, such
3289 proper simulation of hearing. It is easy to create new =CORTEX=
3290 creatures using Blender, a free 3D modeling program.
3292 - =EMPATH=, which uses =CORTEX= to identify the actions of a
3293 worm-like creature using a computational model of empathy.
3295 #+BEGIN_LaTeX
3296 \appendix
3297 #+END_LaTeX
3299 * Appendix: =CORTEX= User Guide
3301 Those who write a thesis should endeavor to make their code not only
3302 accessible, but actually usable, as a way to pay back the community
3303 that made the thesis possible in the first place. This thesis would
3304 not be possible without Free Software such as jMonkeyEngine3,
3305 Blender, clojure, emacs, ffmpeg, and many other tools. That is why I
3306 have included this user guide, in the hope that someone else might
3307 find =CORTEX= useful.
3309 ** Obtaining =CORTEX=
3311 You can get cortex from its mercurial repository at
3312 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3313 releases at http://aurellem.org/cortex/releases/. As a condition of
3314 making this thesis, I have also provided Professor Winston the
3315 =CORTEX= source, and he knows how to run the demos and get started.
3316 You may also email me at =cortex@aurellem.org= and I may help where
3317 I can.
3319 ** Running =CORTEX=
3321 =CORTEX= comes with README and INSTALL files that will guide you
3322 through installation and running the test suite. In particular you
3323 should look at test =cortex.test= which contains test suites that
3324 run through all senses and multiple creatures.
3326 ** Creating creatures
3328 Creatures are created using /Blender/, a free 3D modeling program.
3329 You will need Blender version 2.6 when using the =CORTEX= included
3330 in this thesis. You create a =CORTEX= creature in a similar manner
3331 to modeling anything in Blender, except that you also create
3332 several trees of empty nodes which define the creature's senses.
3334 *** Mass
3336 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3337 to the object with the mass in jMonkeyEngine units. Note that
3338 setting the mass to 0 causes the object to be immovable.
3340 *** Joints
3342 Joints are created by creating an empty node named =joints= and
3343 then creating any number of empty child nodes to represent your
3344 creature's joints. The joint will automatically connect the
3345 closest two physical objects. It will help to set the empty node's
3346 display mode to ``Arrows'' so that you can clearly see the
3347 direction of the axes.
3349 Joint nodes should have the following metadata under the ``joint''
3350 label:
3352 #+BEGIN_SRC clojure
3353 ;; ONE OF the following, under the label "joint":
3354 {:type :point}
3356 ;; OR
3358 {:type :hinge
3359 :limit [<limit-low> <limit-high>]
3360 :axis (Vector3f. <x> <y> <z>)}
3361 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3363 ;; OR
3365 {:type :cone
3366 :limit-xz <lim-xz>
3367 :limit-xy <lim-xy>
3368 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
3369 #+END_SRC
3371 *** Eyes
3373 Eyes are created by creating an empty node named =eyes= and then
3374 creating any number of empty child nodes to represent your
3375 creature's eyes.
3377 Eye nodes should have the following metadata under the ``eye''
3378 label:
3380 #+BEGIN_SRC clojure
3381 {:red <red-retina-definition>
3382 :blue <blue-retina-definition>
3383 :green <green-retina-definition>
3384 :all <all-retina-definition>
3385 (<0xrrggbb> <custom-retina-image>)...
3387 #+END_SRC
3389 Any of the color channels may be omitted. You may also include
3390 your own color selectors, and in fact :red is equivalent to
3391 0xFF0000 and so forth. The eye will be placed at the same position
3392 as the empty node and will bind to the neatest physical object.
3393 The eye will point outward from the X-axis of the node, and ``up''
3394 will be in the direction of the X-axis of the node. It will help
3395 to set the empty node's display mode to ``Arrows'' so that you can
3396 clearly see the direction of the axes.
3398 Each retina file should contain white pixels wherever you want to be
3399 sensitive to your chosen color. If you want the entire field of
3400 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3401 white.
3403 Here is a sample retinal map:
3405 #+caption: An example retinal profile image. White pixels are
3406 #+caption: photo-sensitive elements. The distribution of white
3407 #+caption: pixels is denser in the middle and falls off at the
3408 #+caption: edges and is inspired by the human retina.
3409 #+name: retina
3410 #+ATTR_LaTeX: :width 7cm :placement [H]
3411 [[./images/retina-small.png]]
3413 *** Hearing
3415 Ears are created by creating an empty node named =ears= and then
3416 creating any number of empty child nodes to represent your
3417 creature's ears.
3419 Ear nodes do not require any metadata.
3421 The ear will bind to and follow the closest physical node.
3423 *** Touch
3425 Touch is handled similarly to mass. To make a particular object
3426 touch sensitive, add metadata of the following form under the
3427 object's ``touch'' metadata field:
3429 #+BEGIN_EXAMPLE
3430 <touch-UV-map-file-name>
3431 #+END_EXAMPLE
3433 You may also include an optional ``scale'' metadata number to
3434 specify the length of the touch feelers. The default is $0.1$,
3435 and this is generally sufficient.
3437 The touch UV should contain white pixels for each touch sensor.
3439 Here is an example touch-uv map that approximates a human finger,
3440 and its corresponding model.
3442 #+caption: This is the tactile-sensor-profile for the upper segment
3443 #+caption: of a fingertip. It defines regions of high touch sensitivity
3444 #+caption: (where there are many white pixels) and regions of low
3445 #+caption: sensitivity (where white pixels are sparse).
3446 #+name: guide-fingertip-UV
3447 #+ATTR_LaTeX: :width 9cm :placement [H]
3448 [[./images/finger-UV.png]]
3450 #+caption: The fingertip UV-image form above applied to a simple
3451 #+caption: model of a fingertip.
3452 #+name: guide-fingertip
3453 #+ATTR_LaTeX: :width 9cm :placement [H]
3454 [[./images/finger-2.png]]
3456 *** Proprioception
3458 Proprioception is tied to each joint node -- nothing special must
3459 be done in a blender model to enable proprioception other than
3460 creating joint nodes.
3462 *** Muscles
3464 Muscles are created by creating an empty node named =muscles= and
3465 then creating any number of empty child nodes to represent your
3466 creature's muscles.
3469 Muscle nodes should have the following metadata under the
3470 ``muscle'' label:
3472 #+BEGIN_EXAMPLE
3473 <muscle-profile-file-name>
3474 #+END_EXAMPLE
3476 Muscles should also have a ``strength'' metadata entry describing
3477 the muscle's total strength at full activation.
3479 Muscle profiles are simple images that contain the relative amount
3480 of muscle power in each simulated alpha motor neuron. The width of
3481 the image is the total size of the motor pool, and the redness of
3482 each neuron is the relative power of that motor pool.
3484 While the profile image can have any dimensions, only the first
3485 line of pixels is used to define the muscle. Here is a sample
3486 muscle profile image that defines a human-like muscle.
3488 #+caption: A muscle profile image that describes the strengths
3489 #+caption: of each motor neuron in a muscle. White is weakest
3490 #+caption: and dark red is strongest. This particular pattern
3491 #+caption: has weaker motor neurons at the beginning, just
3492 #+caption: like human muscle.
3493 #+name: muscle-recruit
3494 #+ATTR_LaTeX: :width 7cm :placement [H]
3495 [[./images/basic-muscle.png]]
3497 Muscles twist the nearest physical object about the muscle node's
3498 Z-axis. I recommend using the ``Single Arrow'' display mode for
3499 muscles and using the right hand rule to determine which way the
3500 muscle will twist. To make a segment that can twist in multiple
3501 directions, create multiple, differently aligned muscles.
3503 ** =CORTEX= API
3505 These are the some functions exposed by =CORTEX= for creating
3506 worlds and simulating creatures. These are in addition to
3507 jMonkeyEngine3's extensive library, which is documented elsewhere.
3509 *** Simulation
3510 - =(world root-node key-map setup-fn update-fn)= :: create
3511 a simulation.
3512 - /root-node/ :: a =com.jme3.scene.Node= object which
3513 contains all of the objects that should be in the
3514 simulation.
3516 - /key-map/ :: a map from strings describing keys to
3517 functions that should be executed whenever that key is
3518 pressed. the functions should take a SimpleApplication
3519 object and a boolean value. The SimpleApplication is the
3520 current simulation that is running, and the boolean is true
3521 if the key is being pressed, and false if it is being
3522 released. As an example,
3523 #+BEGIN_SRC clojure
3524 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3525 #+END_SRC
3526 is a valid key-map which will cause the simulation to print
3527 a message whenever the 'j' key on the keyboard is pressed.
3529 - /setup-fn/ :: a function that takes a =SimpleApplication=
3530 object. It is called once when initializing the simulation.
3531 Use it to create things like lights, change the gravity,
3532 initialize debug nodes, etc.
3534 - /update-fn/ :: this function takes a =SimpleApplication=
3535 object and a float and is called every frame of the
3536 simulation. The float tells how many seconds is has been
3537 since the last frame was rendered, according to whatever
3538 clock jme is currently using. The default is to use IsoTimer
3539 which will result in this value always being the same.
3541 - =(position-camera world position rotation)= :: set the position
3542 of the simulation's main camera.
3544 - =(enable-debug world)= :: turn on debug wireframes for each
3545 simulated object.
3547 - =(set-gravity world gravity)= :: set the gravity of a running
3548 simulation.
3550 - =(box length width height & {options})= :: create a box in the
3551 simulation. Options is a hash map specifying texture, mass,
3552 etc. Possible options are =:name=, =:color=, =:mass=,
3553 =:friction=, =:texture=, =:material=, =:position=,
3554 =:rotation=, =:shape=, and =:physical?=.
3556 - =(sphere radius & {options})= :: create a sphere in the simulation.
3557 Options are the same as in =box=.
3559 - =(load-blender-model file-name)= :: create a node structure
3560 representing that described in a blender file.
3562 - =(light-up-everything world)= :: distribute a standard compliment
3563 of lights throughout the simulation. Should be adequate for most
3564 purposes.
3566 - =(node-seq node)= :: return a recursive list of the node's
3567 children.
3569 - =(nodify name children)= :: construct a node given a node-name and
3570 desired children.
3572 - =(add-element world element)= :: add an object to a running world
3573 simulation.
3575 - =(set-accuracy world accuracy)= :: change the accuracy of the
3576 world's physics simulator.
3578 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3579 construct that is useful for loading textures and is required
3580 for smooth interaction with jMonkeyEngine library functions.
3582 - =(load-bullet)= :: unpack native libraries and initialize
3583 blender. This function is required before other world building
3584 functions are called.
3586 *** Creature Manipulation / Import
3588 - =(body! creature)= :: give the creature a physical body.
3590 - =(vision! creature)= :: give the creature a sense of vision.
3591 Returns a list of functions which will each, when called
3592 during a simulation, return the vision data for the channel of
3593 one of the eyes. The functions are ordered depending on the
3594 alphabetical order of the names of the eye nodes in the
3595 blender file. The data returned by the functions is a vector
3596 containing the eye's /topology/, a vector of coordinates, and
3597 the eye's /data/, a vector of RGB values filtered by the eye's
3598 sensitivity.
3600 - =(hearing! creature)= :: give the creature a sense of hearing.
3601 Returns a list of functions, one for each ear, that when
3602 called will return a frame's worth of hearing data for that
3603 ear. The functions are ordered depending on the alphabetical
3604 order of the names of the ear nodes in the blender file. The
3605 data returned by the functions is an array PCM encoded wav
3606 data.
3608 - =(touch! creature)= :: give the creature a sense of touch. Returns
3609 a single function that must be called with the /root node/ of
3610 the world, and which will return a vector of /touch-data/
3611 one entry for each touch sensitive component, each entry of
3612 which contains a /topology/ that specifies the distribution of
3613 touch sensors, and the /data/, which is a vector of
3614 =[activation, length]= pairs for each touch hair.
3616 - =(proprioception! creature)= :: give the creature the sense of
3617 proprioception. Returns a list of functions, one for each
3618 joint, that when called during a running simulation will
3619 report the =[heading, pitch, roll]= of the joint.
3621 - =(movement! creature)= :: give the creature the power of movement.
3622 Creates a list of functions, one for each muscle, that when
3623 called with an integer, will set the recruitment of that
3624 muscle to that integer, and will report the current power
3625 being exerted by the muscle. Order of muscles is determined by
3626 the alphabetical sort order of the names of the muscle nodes.
3628 *** Visualization/Debug
3630 - =(view-vision)= :: create a function that when called with a list
3631 of visual data returned from the functions made by =vision!=,
3632 will display that visual data on the screen.
3634 - =(view-hearing)= :: same as =view-vision= but for hearing.
3636 - =(view-touch)= :: same as =view-vision= but for touch.
3638 - =(view-proprioception)= :: same as =view-vision= but for
3639 proprioception.
3641 - =(view-movement)= :: same as =view-vision= but for
3642 proprioception.
3644 - =(view anything)= :: =view= is a polymorphic function that allows
3645 you to inspect almost anything you could reasonably expect to
3646 be able to ``see'' in =CORTEX=.
3648 - =(text anything)= :: =text= is a polymorphic function that allows
3649 you to convert practically anything into a text string.
3651 - =(println-repl anything)= :: print messages to clojure's repl
3652 instead of the simulation's terminal window.
3654 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3655 function will import all jMonkeyEngine3 classes for immediate
3656 use.
3658 - =(display-dilated-time world timer)= :: Shows the time as it is
3659 flowing in the simulation on a HUD display.