view thesis/cortex.org @ 564:ecae29320b00

even more changes from winston.
author Robert McIntyre <rlm@mit.edu>
date Mon, 12 May 2014 13:31:55 -0400
parents 3c3dc4dbb973
children b9b8567c14ee
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * Empathy \& Embodiment: problem solving strategies
46 By the time you have read this thesis, you will understand a novel
47 approach to representing and recognizing physical actions using
48 embodiment and empathy. You will also see one way to efficiently
49 implement physical empathy for embodied creatures. Finally, you will
50 become familiar with =CORTEX=, a system for designing and simulating
51 creatures with rich senses, which I have designed as a library that
52 you can use in your own research. Note that I /do not/ process video
53 directly --- I start with knowledge of the positions of a creature's
54 body parts and work from there.
56 This is the core vision of my thesis: That one of the important ways
57 in which we understand others is by imagining ourselves in their
58 position and empathically feeling experiences relative to our own
59 bodies. By understanding events in terms of our own previous
60 corporeal experience, we greatly constrain the possibilities of what
61 would otherwise be an unwieldy exponential search. This extra
62 constraint can be the difference between easily understanding what
63 is happening in a video and being completely lost in a sea of
64 incomprehensible color and movement.
66 ** The problem: recognizing actions is hard!
68 Examine figure \ref{cat-drink}. What is happening? As you, and
69 indeed very young children, can easily determine, this is an image
70 of drinking.
72 #+caption: A cat drinking some water. Identifying this action is
73 #+caption: beyond the capabilities of existing computer vision systems.
74 #+name: cat-drink
75 #+ATTR_LaTeX: :width 7cm
76 [[./images/cat-drinking.jpg]]
78 Nevertheless, it is beyond the state of the art for a computer
79 vision program to describe what's happening in this image. Part of
80 the problem is that many computer vision systems focus on
81 pixel-level details or comparisons to example images (such as
82 \cite{volume-action-recognition}), but the 3D world is so variable
83 that it is hard to describe the world in terms of possible images.
85 In fact, the contents of a scene may have much less to do with
86 pixel probabilities than with recognizing various affordances:
87 things you can move, objects you can grasp, spaces that can be
88 filled . For example, what processes might enable you to see the
89 chair in figure \ref{hidden-chair}?
91 #+caption: The chair in this image is quite obvious to humans, but
92 #+caption: it can't be found by any modern computer vision program.
93 #+name: hidden-chair
94 #+ATTR_LaTeX: :width 10cm
95 [[./images/fat-person-sitting-at-desk.jpg]]
97 Finally, how is it that you can easily tell the difference between
98 how the girl's /muscles/ are working in figure \ref{girl}?
100 #+caption: The mysterious ``common sense'' appears here as you are able
101 #+caption: to discern the difference in how the girl's arm muscles
102 #+caption: are activated between the two images. When you compare
103 #+caption: these two images, do you feel something in your own arm
104 #+caption: muscles?
105 #+name: girl
106 #+ATTR_LaTeX: :width 7cm
107 [[./images/wall-push.png]]
109 Each of these examples tells us something about what might be going
110 on in our minds as we easily solve these recognition problems:
112 - The hidden chair shows us that we are strongly triggered by cues
113 relating to the position of human bodies, and that we can
114 determine the overall physical configuration of a human body even
115 if much of that body is occluded.
117 - The picture of the girl pushing against the wall tells us that we
118 have common sense knowledge about the kinetics of our own bodies.
119 We know well how our muscles would have to work to maintain us in
120 most positions, and we can easily project this self-knowledge to
121 imagined positions triggered by images of the human body.
123 - The cat tells us that imagination of some kind plays an important
124 role in understanding actions. The question is: Can we be more
125 precise about what sort of imagination is required to understand
126 these actions?
128 ** A step forward: the sensorimotor-centered approach
130 In this thesis, I explore the idea that our knowledge of our own
131 bodies, combined with our own rich senses, enables us to recognize
132 the actions of others.
134 For example, I think humans are able to label the cat video as
135 ``drinking'' because they imagine /themselves/ as the cat, and
136 imagine putting their face up against a stream of water and
137 sticking out their tongue. In that imagined world, they can feel
138 the cool water hitting their tongue, and feel the water entering
139 their body, and are able to recognize that /feeling/ as drinking.
140 So, the label of the action is not really in the pixels of the
141 image, but is found clearly in a simulation / recollection inspired
142 by those pixels. An imaginative system, having been trained on
143 drinking and non-drinking examples and learning that the most
144 important component of drinking is the feeling of water flowing
145 down one's throat, would analyze a video of a cat drinking in the
146 following manner:
148 1. Create a physical model of the video by putting a ``fuzzy''
149 model of its own body in place of the cat. Possibly also create
150 a simulation of the stream of water.
152 2. Play out this simulated scene and generate imagined sensory
153 experience. This will include relevant muscle contractions, a
154 close up view of the stream from the cat's perspective, and most
155 importantly, the imagined feeling of water entering the mouth.
156 The imagined sensory experience can come from a simulation of
157 the event, but can also be pattern-matched from previous,
158 similar embodied experience.
160 3. The action is now easily identified as drinking by the sense of
161 taste alone. The other senses (such as the tongue moving in and
162 out) help to give plausibility to the simulated action. Note that
163 the sense of vision, while critical in creating the simulation,
164 is not critical for identifying the action from the simulation.
166 For the chair examples, the process is even easier:
168 1. Align a model of your body to the person in the image.
170 2. Generate proprioceptive sensory data from this alignment.
172 3. Use the imagined proprioceptive data as a key to lookup related
173 sensory experience associated with that particular proprioceptive
174 feeling.
176 4. Retrieve the feeling of your bottom resting on a surface, your
177 knees bent, and your leg muscles relaxed.
179 5. This sensory information is consistent with your =sitting?=
180 sensory predicate, so you (and the entity in the image) must be
181 sitting.
183 6. There must be a chair-like object since you are sitting.
185 Empathy offers yet another alternative to the age-old AI
186 representation question: ``What is a chair?'' --- A chair is the
187 feeling of sitting!
189 One powerful advantage of empathic problem solving is that it
190 factors the action recognition problem into two easier problems. To
191 use empathy, you need an /aligner/, which takes the video and a
192 model of your body, and aligns the model with the video. Then, you
193 need a /recognizer/, which uses the aligned model to interpret the
194 action. The power in this method lies in the fact that you describe
195 all actions from a body-centered viewpoint. You are less tied to
196 the particulars of any visual representation of the actions. If you
197 teach the system what ``running'' is, and you have a good enough
198 aligner, the system will from then on be able to recognize running
199 from any point of view -- even strange points of view like above or
200 underneath the runner. This is in contrast to action recognition
201 schemes that try to identify actions using a non-embodied approach.
202 If these systems learn about running as viewed from the side, they
203 will not automatically be able to recognize running from any other
204 viewpoint.
206 Another powerful advantage is that using the language of multiple
207 body-centered rich senses to describe body-centered actions offers
208 a massive boost in descriptive capability. Consider how difficult
209 it would be to compose a set of HOG (Histogram of Oriented
210 Gradients) filters to describe the action of a simple worm-creature
211 ``curling'' so that its head touches its tail, and then behold the
212 simplicity of describing thus action in a language designed for the
213 task (listing \ref{grand-circle-intro}):
215 #+caption: Body-centered actions are best expressed in a body-centered
216 #+caption: language. This code detects when the worm has curled into a
217 #+caption: full circle. Imagine how you would replicate this functionality
218 #+caption: using low-level pixel features such as HOG filters!
219 #+name: grand-circle-intro
220 #+begin_listing clojure
221 #+begin_src clojure
222 (defn grand-circle?
223 "Does the worm form a majestic circle (one end touching the other)?"
224 [experiences]
225 (and (curled? experiences)
226 (let [worm-touch (:touch (peek experiences))
227 tail-touch (worm-touch 0)
228 head-touch (worm-touch 4)]
229 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
230 (< 0.2 (contact worm-segment-top-tip head-touch))))))
231 #+end_src
232 #+end_listing
234 ** =EMPATH= recognizes actions using empathy
236 Exploring these ideas further demands a concrete implementation, so
237 first, I built a system for constructing virtual creatures with
238 physiologically plausible sensorimotor systems and detailed
239 environments. The result is =CORTEX=, which I describe in chapter
240 \ref{sec-2}.
242 Next, I wrote routines which enabled a simple worm-like creature to
243 infer the actions of a second worm-like creature, using only its
244 own prior sensorimotor experiences and knowledge of the second
245 worm's joint positions. This program, =EMPATH=, is described in
246 chapter \ref{sec-3}. It's main components are:
248 - Embodied Action Definitions :: Many otherwise complicated actions
249 are easily described in the language of a full suite of
250 body-centered, rich senses and experiences. For example,
251 drinking is the feeling of water flowing down your throat, and
252 cooling your insides. It's often accompanied by bringing your
253 hand close to your face, or bringing your face close to water.
254 Sitting down is the feeling of bending your knees, activating
255 your quadriceps, then feeling a surface with your bottom and
256 relaxing your legs. These body-centered action descriptions
257 can be either learned or hard coded.
259 - Guided Play :: The creature moves around and experiences the
260 world through its unique perspective. As the creature moves,
261 it gathers experiences that satisfy the embodied action
262 definitions.
264 - Posture Imitation :: When trying to interpret a video or image,
265 the creature takes a model of itself and aligns it with
266 whatever it sees. This alignment might even cross species, as
267 when humans try to align themselves with things like ponies,
268 dogs, or other humans with a different body type.
270 - Empathy :: The alignment triggers associations with
271 sensory data from prior experiences. For example, the
272 alignment itself easily maps to proprioceptive data. Any
273 sounds or obvious skin contact in the video can to a lesser
274 extent trigger previous experience keyed to hearing or touch.
275 Segments of previous experiences gained from play are stitched
276 together to form a coherent and complete sensory portrait of
277 the scene.
279 - Recognition :: With the scene described in terms of remembered
280 first person sensory events, the creature can now run its
281 action-definition programs (such as the one in listing
282 \ref{grand-circle-intro}) on this synthesized sensory data,
283 just as it would if it were actually experiencing the scene
284 first-hand. If previous experience has been accurately
285 retrieved, and if it is analogous enough to the scene, then
286 the creature will correctly identify the action in the scene.
288 My program =EMPATH= uses this empathic problem solving technique
289 to interpret the actions of a simple, worm-like creature.
291 #+caption: The worm performs many actions during free play such as
292 #+caption: curling, wiggling, and resting.
293 #+name: worm-intro
294 #+ATTR_LaTeX: :width 15cm
295 [[./images/worm-intro-white.png]]
297 #+caption: =EMPATH= recognized and classified each of these
298 #+caption: poses by inferring the complete sensory experience
299 #+caption: from proprioceptive data.
300 #+name: worm-recognition-intro
301 #+ATTR_LaTeX: :width 15cm
302 [[./images/worm-poses.png]]
304 *** Main Results
306 - After one-shot supervised training, =EMPATH= was able to
307 recognize a wide variety of static poses and dynamic
308 actions---ranging from curling in a circle to wiggling with a
309 particular frequency --- with 95\% accuracy.
311 - These results were completely independent of viewing angle
312 because the underlying body-centered language fundamentally is
313 independent; once an action is learned, it can be recognized
314 equally well from any viewing angle.
316 - =EMPATH= is surprisingly short; the sensorimotor-centered
317 language provided by =CORTEX= resulted in extremely economical
318 recognition routines --- about 500 lines in all --- suggesting
319 that such representations are very powerful, and often
320 indispensable for the types of recognition tasks considered here.
322 - For expediency's sake, I relied on direct knowledge of joint
323 positions in this proof of concept. However, I believe that the
324 structure of =EMPATH= and =CORTEX= will make future work to
325 enable video analysis much easier than it would otherwise be.
327 ** =EMPATH= is built on =CORTEX=, a creature builder.
329 I built =CORTEX= to be a general AI research platform for doing
330 experiments involving multiple rich senses and a wide variety and
331 number of creatures. I intend it to be useful as a library for many
332 more projects than just this thesis. =CORTEX= was necessary to meet
333 a need among AI researchers at CSAIL and beyond, which is that
334 people often will invent wonderful ideas that are best expressed in
335 the language of creatures and senses, but in order to explore those
336 ideas they must first build a platform in which they can create
337 simulated creatures with rich senses! There are many ideas that
338 would be simple to execute (such as =EMPATH= or Larson's
339 self-organizing maps (\cite{larson-symbols})), but attached to them
340 is the multi-month effort to make a good creature simulator. Often,
341 that initial investment of time proves to be too much, and the
342 project must make do with a lesser environment or be abandoned
343 entirely.
345 =CORTEX= is well suited as an environment for embodied AI research
346 for three reasons:
348 - You can design new creatures using Blender (\cite{blender}), a
349 popular, free 3D modeling program. Each sense can be specified
350 using special Blender nodes with biologically inspired
351 parameters. You need not write any code to create a creature, and
352 can use a wide library of pre-existing blender models as a base
353 for your own creatures.
355 - =CORTEX= implements a wide variety of senses: touch,
356 proprioception, vision, hearing, and muscle tension. Complicated
357 senses like touch and vision involve multiple sensory elements
358 embedded in a 2D surface. You have complete control over the
359 distribution of these sensor elements through the use of simple
360 image files. =CORTEX= implements more comprehensive hearing than
361 any other creature simulation system available.
363 - =CORTEX= supports any number of creatures and any number of
364 senses. Time in =CORTEX= dilates so that the simulated creatures
365 always perceive a perfectly smooth flow of time, regardless of
366 the actual computational load.
368 =CORTEX= is built on top of =jMonkeyEngine3=
369 (\cite{jmonkeyengine}), which is a video game engine designed to
370 create cross-platform 3D desktop games. =CORTEX= is mainly written
371 in clojure, a dialect of =LISP= that runs on the Java Virtual
372 Machine (JVM). The API for creating and simulating creatures and
373 senses is entirely expressed in clojure, though many senses are
374 implemented at the layer of jMonkeyEngine or below. For example,
375 for the sense of hearing I use a layer of clojure code on top of a
376 layer of java JNI bindings that drive a layer of =C++= code which
377 implements a modified version of =OpenAL= to support multiple
378 listeners. =CORTEX= is the only simulation environment that I know
379 of that can support multiple entities that can each hear the world
380 from their own perspective. Other senses also require a small layer
381 of Java code. =CORTEX= also uses =bullet=, a physics simulator
382 written in =C=.
384 #+caption: Here is the worm from figure \ref{worm-intro} modeled
385 #+caption: in Blender, a free 3D-modeling program. Senses and
386 #+caption: joints are described using special nodes in Blender.
387 #+name: worm-recognition-intro-2
388 #+ATTR_LaTeX: :width 12cm
389 [[./images/blender-worm.png]]
391 Here are some things I anticipate that =CORTEX= might be used for:
393 - exploring new ideas about sensory integration
394 - distributed communication among swarm creatures
395 - self-learning using free exploration,
396 - evolutionary algorithms involving creature construction
397 - exploration of exotic senses and effectors that are not possible
398 in the real world (such as telekinesis or a semantic sense)
399 - imagination using subworlds
401 During one test with =CORTEX=, I created 3,000 creatures each with
402 its own independent senses and ran them all at only 1/80 real time.
403 In another test, I created a detailed model of my own hand,
404 equipped with a realistic distribution of touch (more sensitive at
405 the fingertips), as well as eyes and ears, and it ran at around 1/4
406 real time.
408 #+BEGIN_LaTeX
409 \begin{sidewaysfigure}
410 \includegraphics[width=9.5in]{images/full-hand.png}
411 \caption{
412 I modeled my own right hand in Blender and rigged it with all the
413 senses that {\tt CORTEX} supports. My simulated hand has a
414 biologically inspired distribution of touch sensors. The senses are
415 displayed on the right (the red/black squares are raw sensory output),
416 and the simulation is displayed on the
417 left. Notice that my hand is curling its fingers, that it can see
418 its own finger from the eye in its palm, and that it can feel its
419 own thumb touching its palm.}
420 \end{sidewaysfigure}
421 #+END_LaTeX
423 * Designing =CORTEX=
425 In this chapter, I outline the design decisions that went into
426 making =CORTEX=, along with some details about its implementation.
427 (A practical guide to getting started with =CORTEX=, which skips
428 over the history and implementation details presented here, is
429 provided in an appendix at the end of this thesis.)
431 Throughout this project, I intended for =CORTEX= to be flexible and
432 extensible enough to be useful for other researchers who want to
433 test ideas of their own. To this end, wherever I have had to make
434 architectural choices about =CORTEX=, I have chosen to give as much
435 freedom to the user as possible, so that =CORTEX= may be used for
436 things I have not foreseen.
438 ** Building in simulation versus reality
439 The most important architectural decision of all is the choice to
440 use a computer-simulated environment in the first place! The world
441 is a vast and rich place, and for now simulations are a very poor
442 reflection of its complexity. It may be that there is a significant
443 qualitative difference between dealing with senses in the real
444 world and dealing with pale facsimiles of them in a simulation
445 (\cite{brooks-representation}). What are the advantages and
446 disadvantages of a simulation vs. reality?
448 *** Simulation
450 The advantages of virtual reality are that when everything is a
451 simulation, experiments in that simulation are absolutely
452 reproducible. It's also easier to change the creature and
453 environment to explore new situations and different sensory
454 combinations.
456 If the world is to be simulated on a computer, then not only do
457 you have to worry about whether the creature's senses are rich
458 enough to learn from the world, but whether the world itself is
459 rendered with enough detail and realism to give enough working
460 material to the creature's senses. To name just a few
461 difficulties facing modern physics simulators: destructibility of
462 the environment, simulation of water/other fluids, large areas,
463 nonrigid bodies, lots of objects, smoke. I don't know of any
464 computer simulation that would allow a creature to take a rock
465 and grind it into fine dust, then use that dust to make a clay
466 sculpture, at least not without spending years calculating the
467 interactions of every single small grain of dust. Maybe a
468 simulated world with today's limitations doesn't provide enough
469 richness for real intelligence to evolve.
471 *** Reality
473 The other approach for playing with senses is to hook your
474 software up to real cameras, microphones, robots, etc., and let it
475 loose in the real world. This has the advantage of eliminating
476 concerns about simulating the world at the expense of increasing
477 the complexity of implementing the senses. Instead of just
478 grabbing the current rendered frame for processing, you have to
479 use an actual camera with real lenses and interact with photons to
480 get an image. It is much harder to change the creature, which is
481 now partly a physical robot of some sort, since doing so involves
482 changing things around in the real world instead of modifying
483 lines of code. While the real world is very rich and definitely
484 provides enough stimulation for intelligence to develop (as
485 evidenced by our own existence), it is also uncontrollable in the
486 sense that a particular situation cannot be recreated perfectly or
487 saved for later use. It is harder to conduct Science because it is
488 harder to repeat an experiment. The worst thing about using the
489 real world instead of a simulation is the matter of time. Instead
490 of simulated time you get the constant and unstoppable flow of
491 real time. This severely limits the sorts of software you can use
492 to program an AI, because all sense inputs must be handled in real
493 time. Complicated ideas may have to be implemented in hardware or
494 may simply be impossible given the current speed of our
495 processors. Contrast this with a simulation, in which the flow of
496 time in the simulated world can be slowed down to accommodate the
497 limitations of the creature's programming. In terms of cost, doing
498 everything in software is far cheaper than building custom
499 real-time hardware. All you need is a laptop and some patience.
501 ** Simulated time enables rapid prototyping \& simple programs
503 I envision =CORTEX= being used to support rapid prototyping and
504 iteration of ideas. Even if I could put together a well constructed
505 kit for creating robots, it would still not be enough because of
506 the scourge of real-time processing. Anyone who wants to test their
507 ideas in the real world must always worry about getting their
508 algorithms to run fast enough to process information in real time.
509 The need for real time processing only increases if multiple senses
510 are involved. In the extreme case, even simple algorithms will have
511 to be accelerated by ASIC chips or FPGAs, turning what would
512 otherwise be a few lines of code and a 10x speed penalty into a
513 multi-month ordeal. For this reason, =CORTEX= supports
514 /time-dilation/, which scales back the framerate of the simulation
515 in proportion to the amount of processing each frame. From the
516 perspective of the creatures inside the simulation, time always
517 appears to flow at a constant rate, regardless of how complicated
518 the environment becomes or how many creatures are in the
519 simulation. The cost is that =CORTEX= can sometimes run slower than
520 real time. Time dilation works both ways, however --- simulations
521 of very simple creatures in =CORTEX= generally run at 40x real-time
522 on my machine!
524 ** All sense organs are two-dimensional surfaces
526 If =CORTEX= is to support a wide variety of senses, it would help
527 to have a better understanding of what a sense actually is! While
528 vision, touch, and hearing all seem like they are quite different
529 things, I was surprised to learn during the course of this thesis
530 that they (and all physical senses) can be expressed as exactly the
531 same mathematical object!
533 Human beings are three-dimensional objects, and the nerves that
534 transmit data from our various sense organs to our brain are
535 essentially one-dimensional. This leaves up to two dimensions in
536 which our sensory information may flow. For example, imagine your
537 skin: it is a two-dimensional surface around a three-dimensional
538 object (your body). It has discrete touch sensors embedded at
539 various points, and the density of these sensors corresponds to the
540 sensitivity of that region of skin. Each touch sensor connects to a
541 nerve, all of which eventually are bundled together as they travel
542 up the spinal cord to the brain. Intersect the spinal nerves with a
543 guillotining plane and you will see all of the sensory data of the
544 skin revealed in a roughly circular two-dimensional image which is
545 the cross section of the spinal cord. Points on this image that are
546 close together in this circle represent touch sensors that are
547 /probably/ close together on the skin, although there is of course
548 some cutting and rearrangement that has to be done to transfer the
549 complicated surface of the skin onto a two dimensional image.
551 Most human senses consist of many discrete sensors of various
552 properties distributed along a surface at various densities. For
553 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
554 disks, and Ruffini's endings (\cite{textbook901}), which detect
555 pressure and vibration of various intensities. For ears, it is the
556 stereocilia distributed along the basilar membrane inside the
557 cochlea; each one is sensitive to a slightly different frequency of
558 sound. For eyes, it is rods and cones distributed along the surface
559 of the retina. In each case, we can describe the sense with a
560 surface and a distribution of sensors along that surface.
562 In fact, almost every human sense can be effectively described in
563 terms of a surface containing embedded sensors. If the sense had
564 any more dimensions, then there wouldn't be enough room in the
565 spinal cord to transmit the information!
567 Therefore, =CORTEX= must support the ability to create objects and
568 then be able to ``paint'' points along their surfaces to describe
569 each sense.
571 Fortunately this idea is already a well known computer graphics
572 technique called /UV-mapping/. In UV-mapping, the three-dimensional
573 surface of a model is cut and smooshed until it fits on a
574 two-dimensional image. You paint whatever you want on that image,
575 and when the three-dimensional shape is rendered in a game the
576 smooshing and cutting is reversed and the image appears on the
577 three-dimensional object.
579 To make a sense, interpret the UV-image as describing the
580 distribution of that senses' sensors. To get different types of
581 sensors, you can either use a different color for each type of
582 sensor, or use multiple UV-maps, each labeled with that sensor
583 type. I generally use a white pixel to mean the presence of a
584 sensor and a black pixel to mean the absence of a sensor, and use
585 one UV-map for each sensor-type within a given sense.
587 #+CAPTION: The UV-map for an elongated icososphere. The white
588 #+caption: dots each represent a touch sensor. They are dense
589 #+caption: in the regions that describe the tip of the finger,
590 #+caption: and less dense along the dorsal side of the finger
591 #+caption: opposite the tip.
592 #+name: finger-UV
593 #+ATTR_latex: :width 10cm
594 [[./images/finger-UV.png]]
596 #+caption: Ventral side of the UV-mapped finger. Note the
597 #+caption: density of touch sensors at the tip.
598 #+name: finger-side-view
599 #+ATTR_LaTeX: :width 10cm
600 [[./images/finger-1.png]]
602 ** Video game engines provide ready-made physics and shading
604 I did not need to write my own physics simulation code or shader to
605 build =CORTEX=. Doing so would lead to a system that is impossible
606 for anyone but myself to use anyway. Instead, I use a video game
607 engine as a base and modify it to accommodate the additional needs
608 of =CORTEX=. Video game engines are an ideal starting point to
609 build =CORTEX=, because they are not far from being creature
610 building systems themselves.
612 First off, general purpose video game engines come with a physics
613 engine and lighting / sound system. The physics system provides
614 tools that can be co-opted to serve as touch, proprioception, and
615 muscles. Because some games support split screen views, a good
616 video game engine will allow you to efficiently create multiple
617 cameras in the simulated world that can be used as eyes. Video game
618 systems offer integrated asset management for things like textures
619 and creature models, providing an avenue for defining creatures.
620 They also understand UV-mapping, because this technique is used to
621 apply a texture to a model. Finally, because video game engines
622 support a large number of developers, as long as =CORTEX= doesn't
623 stray too far from the base system, other researchers can turn to
624 this community for help when doing their research.
626 ** =CORTEX= is based on jMonkeyEngine3
628 While preparing to build =CORTEX= I studied several video game
629 engines to see which would best serve as a base. The top contenders
630 were:
632 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID software
633 in 1997. All the source code was released by ID software into
634 the Public Domain several years ago, and as a result it has
635 been ported to many different languages. This engine was
636 famous for its advanced use of realistic shading and it had
637 decent and fast physics simulation. The main advantage of the
638 Quake II engine is its simplicity, but I ultimately rejected
639 it because the engine is too tied to the concept of a
640 first-person shooter game. One of the problems I had was that
641 there does not seem to be any easy way to attach multiple
642 cameras to a single character. There are also several physics
643 clipping issues that are corrected in a way that only applies
644 to the main character and do not apply to arbitrary objects.
646 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
647 and Quake I engines and is used by Valve in the Half-Life
648 series of games. The physics simulation in the Source Engine
649 is quite accurate and probably the best out of all the engines
650 I investigated. There is also an extensive community actively
651 working with the engine. However, applications that use the
652 Source Engine must be written in C++, the code is not open, it
653 only runs on Windows, and the tools that come with the SDK to
654 handle models and textures are complicated and awkward to use.
656 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
657 games in Java. It uses OpenGL to render to the screen and uses
658 screengraphs to avoid drawing things that do not appear on the
659 screen. It has an active community and several games in the
660 pipeline. The engine was not built to serve any particular
661 game but is instead meant to be used for any 3D game.
663 I chose jMonkeyEngine3 because it had the most features out of all
664 the free projects I looked at, and because I could then write my
665 code in clojure, an implementation of =LISP= that runs on the JVM.
667 ** =CORTEX= uses Blender to create creature models
669 For the simple worm-like creatures I will use later on in this
670 thesis, I could define a simple API in =CORTEX= that would allow
671 one to create boxes, spheres, etc., and leave that API as the sole
672 way to create creatures. However, for =CORTEX= to truly be useful
673 for other projects, it needs a way to construct complicated
674 creatures. If possible, it would be nice to leverage work that has
675 already been done by the community of 3D modelers, or at least
676 enable people who are talented at modeling but not programming to
677 design =CORTEX= creatures.
679 Therefore I use Blender, a free 3D modeling program, as the main
680 way to create creatures in =CORTEX=. However, the creatures modeled
681 in Blender must also be simple to simulate in jMonkeyEngine3's game
682 engine, and must also be easy to rig with =CORTEX='s senses. I
683 accomplish this with extensive use of Blender's ``empty nodes.''
685 Empty nodes have no mass, physical presence, or appearance, but
686 they can hold metadata and have names. I use a tree structure of
687 empty nodes to specify senses in the following manner:
689 - Create a single top-level empty node whose name is the name of
690 the sense.
691 - Add empty nodes which each contain meta-data relevant to the
692 sense, including a UV-map describing the number/distribution of
693 sensors if applicable.
694 - Make each empty-node the child of the top-level node.
696 #+caption: An example of annotating a creature model with empty
697 #+caption: nodes to describe the layout of senses. There are
698 #+caption: multiple empty nodes which each describe the position
699 #+caption: of muscles, ears, eyes, or joints.
700 #+name: sense-nodes
701 #+ATTR_LaTeX: :width 10cm
702 [[./images/empty-sense-nodes.png]]
704 ** Bodies are composed of segments connected by joints
706 Blender is a general purpose animation tool, which has been used in
707 the past to create high quality movies such as Sintel
708 (\cite{blender}). Though Blender can model and render even
709 complicated things like water, it is crucial to keep models that
710 are meant to be simulated as creatures simple. =Bullet=, which
711 =CORTEX= uses though jMonkeyEngine3, is a rigid-body physics
712 system. This offers a compromise between the expressiveness of a
713 game level and the speed at which it can be simulated, and it means
714 that creatures should be naturally expressed as rigid components
715 held together by joint constraints.
717 But humans are more like a squishy bag wrapped around some hard
718 bones which define the overall shape. When we move, our skin bends
719 and stretches to accommodate the new positions of our bones.
721 One way to make bodies composed of rigid pieces connected by joints
722 /seem/ more human-like is to use an /armature/, (or /rigging/)
723 system, which defines a overall ``body mesh'' and defines how the
724 mesh deforms as a function of the position of each ``bone'' which
725 is a standard rigid body. This technique is used extensively to
726 model humans and create realistic animations. It is not a good
727 technique for physical simulation because it is a lie -- the skin
728 is not a physical part of the simulation and does not interact with
729 any objects in the world or itself. Objects will pass right though
730 the skin until they come in contact with the underlying bone, which
731 is a physical object. Without simulating the skin, the sense of
732 touch has little meaning, and the creature's own vision will lie to
733 it about the true extent of its body. Simulating the skin as a
734 physical object requires some way to continuously update the
735 physical model of the skin along with the movement of the bones,
736 which is unacceptably slow compared to rigid body simulation.
738 Therefore, instead of using the human-like ``bony meatbag''
739 approach, I decided to base my body plans on multiple solid objects
740 that are connected by joints, inspired by the robot =EVE= from the
741 movie WALL-E.
743 #+caption: =EVE= from the movie WALL-E. This body plan turns
744 #+caption: out to be much better suited to my purposes than a more
745 #+caption: human-like one.
746 #+ATTR_LaTeX: :width 10cm
747 [[./images/Eve.jpg]]
749 =EVE='s body is composed of several rigid components that are held
750 together by invisible joint constraints. This is what I mean by
751 /eve-like/. The main reason that I use eve-like bodies is for
752 simulation efficiency, and so that there will be correspondence
753 between the AI's senses and the physical presence of its body. Each
754 individual section is simulated by a separate rigid body that
755 corresponds exactly with its visual representation and does not
756 change. Sections are connected by invisible joints that are well
757 supported in jMonkeyEngine3. Bullet, the physics backend for
758 jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
759 connected by joints. Just because sections are rigid does not mean
760 they have to stay as one piece forever; they can be dynamically
761 replaced with multiple sections to simulate splitting in two. This
762 could be used to simulate retractable claws or =EVE='s hands, which
763 are able to coalesce into one object in the movie.
765 *** Solidifying/Connecting a body
767 =CORTEX= creates a creature in two steps: first, it traverses the
768 nodes in the Blender file and creates physical representations for
769 any of them that have mass defined in their Blender meta-data.
771 #+caption: Program for iterating through the nodes in a blender file
772 #+caption: and generating physical jMonkeyEngine3 objects with mass
773 #+caption: and a matching physics shape.
774 #+name: physical
775 #+begin_listing clojure
776 #+begin_src clojure
777 (defn physical!
778 "Iterate through the nodes in creature and make them real physical
779 objects in the simulation."
780 [#^Node creature]
781 (dorun
782 (map
783 (fn [geom]
784 (let [physics-control
785 (RigidBodyControl.
786 (HullCollisionShape.
787 (.getMesh geom))
788 (if-let [mass (meta-data geom "mass")]
789 (float mass) (float 1)))]
790 (.addControl geom physics-control)))
791 (filter #(isa? (class %) Geometry )
792 (node-seq creature)))))
793 #+end_src
794 #+end_listing
796 The next step to making a proper body is to connect those pieces
797 together with joints. jMonkeyEngine has a large array of joints
798 available via =bullet=, such as Point2Point, Cone, Hinge, and a
799 generic Six Degree of Freedom joint, with or without spring
800 restitution.
802 Joints are treated a lot like proper senses, in that there is a
803 top-level empty node named ``joints'' whose children each
804 represent a joint.
806 #+caption: View of the hand model in Blender showing the main ``joints''
807 #+caption: node (highlighted in yellow) and its children which each
808 #+caption: represent a joint in the hand. Each joint node has metadata
809 #+caption: specifying what sort of joint it is.
810 #+name: blender-hand
811 #+ATTR_LaTeX: :width 10cm
812 [[./images/hand-screenshot1.png]]
815 =CORTEX='s procedure for binding the creature together with joints
816 is as follows:
818 - Find the children of the ``joints'' node.
819 - Determine the two spatials the joint is meant to connect.
820 - Create the joint based on the meta-data of the empty node.
822 The higher order function =sense-nodes= from =cortex.sense=
823 simplifies finding the joints based on their parent ``joints''
824 node.
826 #+caption: Retrieving the children empty nodes from a single
827 #+caption: named empty node is a common pattern in =CORTEX=.
828 #+caption: Further instances of this technique for the senses
829 #+caption: will be omitted
830 #+name: get-empty-nodes
831 #+begin_listing clojure
832 #+begin_src clojure
833 (defn sense-nodes
834 "For some senses there is a special empty Blender node whose
835 children are considered markers for an instance of that sense. This
836 function generates functions to find those children, given the name
837 of the special parent node."
838 [parent-name]
839 (fn [#^Node creature]
840 (if-let [sense-node (.getChild creature parent-name)]
841 (seq (.getChildren sense-node)) [])))
843 (def
844 ^{:doc "Return the children of the creature's \"joints\" node."
845 :arglists '([creature])}
846 joints
847 (sense-nodes "joints"))
848 #+end_src
849 #+end_listing
851 To find a joint's targets, =CORTEX= creates a small cube, centered
852 around the empty-node, and grows the cube exponentially until it
853 intersects two physical objects. The objects are ordered according
854 to the joint's rotation, with the first one being the object that
855 has more negative coordinates in the joint's reference frame.
856 Because the objects must be physical, the empty-node itself
857 escapes detection. Because the objects must be physical,
858 =joint-targets= must be called /after/ =physical!= is called.
860 #+caption: Program to find the targets of a joint node by
861 #+caption: exponentially growth of a search cube.
862 #+name: joint-targets
863 #+begin_listing clojure
864 #+begin_src clojure
865 (defn joint-targets
866 "Return the two closest two objects to the joint object, ordered
867 from bottom to top according to the joint's rotation."
868 [#^Node parts #^Node joint]
869 (loop [radius (float 0.01)]
870 (let [results (CollisionResults.)]
871 (.collideWith
872 parts
873 (BoundingBox. (.getWorldTranslation joint)
874 radius radius radius) results)
875 (let [targets
876 (distinct
877 (map #(.getGeometry %) results))]
878 (if (>= (count targets) 2)
879 (sort-by
880 #(let [joint-ref-frame-position
881 (jme-to-blender
882 (.mult
883 (.inverse (.getWorldRotation joint))
884 (.subtract (.getWorldTranslation %)
885 (.getWorldTranslation joint))))]
886 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
887 (take 2 targets))
888 (recur (float (* radius 2))))))))
889 #+end_src
890 #+end_listing
892 Once =CORTEX= finds all joints and targets, it creates them using
893 a dispatch on the metadata of each joint node.
895 #+caption: Program to dispatch on Blender metadata and create joints
896 #+caption: suitable for physical simulation.
897 #+name: joint-dispatch
898 #+begin_listing clojure
899 #+begin_src clojure
900 (defmulti joint-dispatch
901 "Translate Blender pseudo-joints into real JME joints."
902 (fn [constraints & _]
903 (:type constraints)))
905 (defmethod joint-dispatch :point
906 [constraints control-a control-b pivot-a pivot-b rotation]
907 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
908 (.setLinearLowerLimit Vector3f/ZERO)
909 (.setLinearUpperLimit Vector3f/ZERO)))
911 (defmethod joint-dispatch :hinge
912 [constraints control-a control-b pivot-a pivot-b rotation]
913 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
914 [limit-1 limit-2] (:limit constraints)
915 hinge-axis (.mult rotation (blender-to-jme axis))]
916 (doto (HingeJoint. control-a control-b pivot-a pivot-b
917 hinge-axis hinge-axis)
918 (.setLimit limit-1 limit-2))))
920 (defmethod joint-dispatch :cone
921 [constraints control-a control-b pivot-a pivot-b rotation]
922 (let [limit-xz (:limit-xz constraints)
923 limit-xy (:limit-xy constraints)
924 twist (:twist constraints)]
925 (doto (ConeJoint. control-a control-b pivot-a pivot-b
926 rotation rotation)
927 (.setLimit (float limit-xz) (float limit-xy)
928 (float twist)))))
929 #+end_src
930 #+end_listing
932 All that is left for joints is to combine the above pieces into
933 something that can operate on the collection of nodes that a
934 Blender file represents.
936 #+caption: Program to completely create a joint given information
937 #+caption: from a Blender file.
938 #+name: connect
939 #+begin_listing clojure
940 #+begin_src clojure
941 (defn connect
942 "Create a joint between 'obj-a and 'obj-b at the location of
943 'joint. The type of joint is determined by the metadata on 'joint.
945 Here are some examples:
946 {:type :point}
947 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
948 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
950 {:type :cone :limit-xz 0]
951 :limit-xy 0]
952 :twist 0]} (use XZY rotation mode in Blender!)"
953 [#^Node obj-a #^Node obj-b #^Node joint]
954 (let [control-a (.getControl obj-a RigidBodyControl)
955 control-b (.getControl obj-b RigidBodyControl)
956 joint-center (.getWorldTranslation joint)
957 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
958 pivot-a (world-to-local obj-a joint-center)
959 pivot-b (world-to-local obj-b joint-center)]
960 (if-let
961 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
962 ;; A side-effect of creating a joint registers
963 ;; it with both physics objects which in turn
964 ;; will register the joint with the physics system
965 ;; when the simulation is started.
966 (joint-dispatch constraints
967 control-a control-b
968 pivot-a pivot-b
969 joint-rotation))))
970 #+end_src
971 #+end_listing
973 In general, whenever =CORTEX= exposes a sense (or in this case
974 physicality), it provides a function of the type =sense!=, which
975 takes in a collection of nodes and augments it to support that
976 sense. The function returns any controls necessary to use that
977 sense. In this case =body!= creates a physical body and returns no
978 control functions.
980 #+caption: Program to give joints to a creature.
981 #+name: joints
982 #+begin_listing clojure
983 #+begin_src clojure
984 (defn joints!
985 "Connect the solid parts of the creature with physical joints. The
986 joints are taken from the \"joints\" node in the creature."
987 [#^Node creature]
988 (dorun
989 (map
990 (fn [joint]
991 (let [[obj-a obj-b] (joint-targets creature joint)]
992 (connect obj-a obj-b joint)))
993 (joints creature))))
994 (defn body!
995 "Endow the creature with a physical body connected with joints. The
996 particulars of the joints and the masses of each body part are
997 determined in Blender."
998 [#^Node creature]
999 (physical! creature)
1000 (joints! creature))
1001 #+end_src
1002 #+end_listing
1004 All of the code you have just seen amounts to only 130 lines, yet
1005 because it builds on top of Blender and jMonkeyEngine3, those few
1006 lines pack quite a punch!
1008 The hand from figure \ref{blender-hand}, which was modeled after
1009 my own right hand, can now be given joints and simulated as a
1010 creature.
1012 #+caption: With the ability to create physical creatures from Blender,
1013 #+caption: =CORTEX= gets one step closer to becoming a full creature
1014 #+caption: simulation environment.
1015 #+name: physical-hand
1016 #+ATTR_LaTeX: :width 15cm
1017 [[./images/physical-hand.png]]
1019 ** Sight reuses standard video game components...
1021 Vision is one of the most important senses for humans, so I need to
1022 build a simulated sense of vision for my AI. I will do this with
1023 simulated eyes. Each eye can be independently moved and should see
1024 its own version of the world depending on where it is.
1026 Making these simulated eyes a reality is simple because
1027 jMonkeyEngine already contains extensive support for multiple views
1028 of the same 3D simulated world. The reason jMonkeyEngine has this
1029 support is because the support is necessary to create games with
1030 split-screen views. Multiple views are also used to create
1031 efficient pseudo-reflections by rendering the scene from a certain
1032 perspective and then projecting it back onto a surface in the 3D
1033 world.
1035 #+caption: jMonkeyEngine supports multiple views to enable
1036 #+caption: split-screen games, like GoldenEye, which was one of
1037 #+caption: the first games to use split-screen views.
1038 #+name: goldeneye
1039 #+ATTR_LaTeX: :width 10cm
1040 [[./images/goldeneye-4-player.png]]
1042 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
1044 jMonkeyEngine allows you to create a =ViewPort=, which represents a
1045 view of the simulated world. You can create as many of these as you
1046 want. Every frame, the =RenderManager= iterates through each
1047 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
1048 is a =FrameBuffer= which represents the rendered image in the GPU.
1050 #+caption: =ViewPorts= are cameras in the world. During each frame,
1051 #+caption: the =RenderManager= records a snapshot of what each view
1052 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
1053 #+name: rendermanagers
1054 #+ATTR_LaTeX: :width 10cm
1055 [[./images/diagram_rendermanager2.png]]
1057 Each =ViewPort= can have any number of attached =SceneProcessor=
1058 objects, which are called every time a new frame is rendered. A
1059 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1060 whatever it wants to the data. Often this consists of invoking GPU
1061 specific operations on the rendered image. The =SceneProcessor= can
1062 also copy the GPU image data to RAM and process it with the CPU.
1064 *** Appropriating Views for Vision
1066 Each eye in the simulated creature needs its own =ViewPort= so
1067 that it can see the world from its own perspective. To this
1068 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1069 any arbitrary continuation function for further processing. That
1070 continuation function may perform both CPU and GPU operations on
1071 the data. To make this easy for the continuation function, the
1072 =SceneProcessor= maintains appropriately sized buffers in RAM to
1073 hold the data. It does not do any copying from the GPU to the CPU
1074 itself because it is a slow operation.
1076 #+caption: Function to make the rendered scene in jMonkeyEngine
1077 #+caption: available for further processing.
1078 #+name: pipeline-1
1079 #+begin_listing clojure
1080 #+begin_src clojure
1081 (defn vision-pipeline
1082 "Create a SceneProcessor object which wraps a vision processing
1083 continuation function. The continuation is a function that takes
1084 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1085 each of which has already been appropriately sized."
1086 [continuation]
1087 (let [byte-buffer (atom nil)
1088 renderer (atom nil)
1089 image (atom nil)]
1090 (proxy [SceneProcessor] []
1091 (initialize
1092 [renderManager viewPort]
1093 (let [cam (.getCamera viewPort)
1094 width (.getWidth cam)
1095 height (.getHeight cam)]
1096 (reset! renderer (.getRenderer renderManager))
1097 (reset! byte-buffer
1098 (BufferUtils/createByteBuffer
1099 (* width height 4)))
1100 (reset! image (BufferedImage.
1101 width height
1102 BufferedImage/TYPE_4BYTE_ABGR))))
1103 (isInitialized [] (not (nil? @byte-buffer)))
1104 (reshape [_ _ _])
1105 (preFrame [_])
1106 (postQueue [_])
1107 (postFrame
1108 [#^FrameBuffer fb]
1109 (.clear @byte-buffer)
1110 (continuation @renderer fb @byte-buffer @image))
1111 (cleanup []))))
1112 #+end_src
1113 #+end_listing
1115 The continuation function given to =vision-pipeline= above will be
1116 given a =Renderer= and three containers for image data. The
1117 =FrameBuffer= references the GPU image data, but the pixel data
1118 can not be used directly on the CPU. The =ByteBuffer= and
1119 =BufferedImage= are initially "empty" but are sized to hold the
1120 data in the =FrameBuffer=. I call transferring the GPU image data
1121 to the CPU structures "mixing" the image data.
1123 *** Optical sensor arrays are described with images and referenced with metadata
1125 The vision pipeline described above handles the flow of rendered
1126 images. Now, =CORTEX= needs simulated eyes to serve as the source
1127 of these images.
1129 An eye is described in Blender in the same way as a joint. They
1130 are zero dimensional empty objects with no geometry whose local
1131 coordinate system determines the orientation of the resulting eye.
1132 All eyes are children of a parent node named "eyes" just as all
1133 joints have a parent named "joints". An eye binds to the nearest
1134 physical object with =bind-sense=.
1136 #+caption: Here, the camera is created based on metadata on the
1137 #+caption: eye-node and attached to the nearest physical object
1138 #+caption: with =bind-sense=
1139 #+name: add-eye
1140 #+begin_listing clojure
1141 #+begin_src clojure
1142 (defn add-eye!
1143 "Create a Camera centered on the current position of 'eye which
1144 follows the closest physical node in 'creature. The camera will
1145 point in the X direction and use the Z vector as up as determined
1146 by the rotation of these vectors in Blender coordinate space. Use
1147 XZY rotation for the node in blender."
1148 [#^Node creature #^Spatial eye]
1149 (let [target (closest-node creature eye)
1150 [cam-width cam-height]
1151 ;;[640 480] ;; graphics card on laptop doesn't support
1152 ;; arbitrary dimensions.
1153 (eye-dimensions eye)
1154 cam (Camera. cam-width cam-height)
1155 rot (.getWorldRotation eye)]
1156 (.setLocation cam (.getWorldTranslation eye))
1157 (.lookAtDirection
1158 cam ; this part is not a mistake and
1159 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1160 (.mult rot Vector3f/UNIT_Y)) ; Blender as the UP vector.
1161 (.setFrustumPerspective
1162 cam (float 45)
1163 (float (/ (.getWidth cam) (.getHeight cam)))
1164 (float 1)
1165 (float 1000))
1166 (bind-sense target cam) cam))
1167 #+end_src
1168 #+end_listing
1170 *** Simulated Retina
1172 An eye is a surface (the retina) which contains many discrete
1173 sensors to detect light. These sensors can have different
1174 light-sensing properties. In humans, each discrete sensor is
1175 sensitive to red, blue, green, or gray. These different types of
1176 sensors can have different spatial distributions along the retina.
1177 In humans, there is a fovea in the center of the retina which has
1178 a very high density of color sensors, and a blind spot which has
1179 no sensors at all. Sensor density decreases in proportion to
1180 distance from the fovea.
1182 I want to be able to model any retinal configuration, so my
1183 eye-nodes in Blender contain metadata pointing to images that
1184 describe the precise position of the individual sensors using
1185 white pixels. The meta-data also describes the precise sensitivity
1186 to light that the sensors described in the image have. An eye can
1187 contain any number of these images. For example, the metadata for
1188 an eye might look like this:
1190 #+begin_src clojure
1191 {0xFF0000 "Models/test-creature/retina-small.png"}
1192 #+end_src
1194 #+caption: An example retinal profile image. White pixels are
1195 #+caption: photo-sensitive elements. The distribution of white
1196 #+caption: pixels is denser in the middle and falls off at the
1197 #+caption: edges and is inspired by the human retina.
1198 #+name: retina
1199 #+ATTR_LaTeX: :width 7cm
1200 [[./images/retina-small.png]]
1202 Together, the number 0xFF0000 and the image above describe the
1203 placement of red-sensitive sensory elements.
1205 Meta-data to very crudely approximate a human eye might be
1206 something like this:
1208 #+begin_src clojure
1209 (let [retinal-profile "Models/test-creature/retina-small.png"]
1210 {0xFF0000 retinal-profile
1211 0x00FF00 retinal-profile
1212 0x0000FF retinal-profile
1213 0xFFFFFF retinal-profile})
1214 #+end_src
1216 The numbers that serve as keys in the map determine a sensor's
1217 relative sensitivity to the channels red, green, and blue. These
1218 sensitivity values are packed into an integer in the order
1219 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1220 image are added together with these sensitivities as linear
1221 weights. Therefore, 0xFF0000 means sensitive to red only while
1222 0xFFFFFF means sensitive to all colors equally (gray).
1224 #+caption: This is the core of vision in =CORTEX=. A given eye node
1225 #+caption: is converted into a function that returns visual
1226 #+caption: information from the simulation.
1227 #+name: vision-kernel
1228 #+begin_listing clojure
1229 #+BEGIN_SRC clojure
1230 (defn vision-kernel
1231 "Returns a list of functions, each of which will return a color
1232 channel's worth of visual information when called inside a running
1233 simulation."
1234 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1235 (let [retinal-map (retina-sensor-profile eye)
1236 camera (add-eye! creature eye)
1237 vision-image
1238 (atom
1239 (BufferedImage. (.getWidth camera)
1240 (.getHeight camera)
1241 BufferedImage/TYPE_BYTE_BINARY))
1242 register-eye!
1243 (runonce
1244 (fn [world]
1245 (add-camera!
1246 world camera
1247 (let [counter (atom 0)]
1248 (fn [r fb bb bi]
1249 (if (zero? (rem (swap! counter inc) (inc skip)))
1250 (reset! vision-image
1251 (BufferedImage! r fb bb bi))))))))]
1252 (vec
1253 (map
1254 (fn [[key image]]
1255 (let [whites (white-coordinates image)
1256 topology (vec (collapse whites))
1257 sensitivity (sensitivity-presets key key)]
1258 (attached-viewport.
1259 (fn [world]
1260 (register-eye! world)
1261 (vector
1262 topology
1263 (vec
1264 (for [[x y] whites]
1265 (pixel-sense
1266 sensitivity
1267 (.getRGB @vision-image x y))))))
1268 register-eye!)))
1269 retinal-map))))
1270 #+END_SRC
1271 #+end_listing
1273 Note that because each of the functions generated by
1274 =vision-kernel= shares the same =register-eye!= function, the eye
1275 will be registered only once the first time any of the functions
1276 from the list returned by =vision-kernel= is called. Each of the
1277 functions returned by =vision-kernel= also allows access to the
1278 =Viewport= through which it receives images.
1280 All the hard work has been done; all that remains is to apply
1281 =vision-kernel= to each eye in the creature and gather the results
1282 into one list of functions.
1285 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1286 #+caption: environment for experimenting with different types of
1287 #+caption: eyes.
1288 #+name: vision!
1289 #+begin_listing clojure
1290 #+BEGIN_SRC clojure
1291 (defn vision!
1292 "Returns a list of functions, each of which returns visual sensory
1293 data when called inside a running simulation."
1294 [#^Node creature & {skip :skip :or {skip 0}}]
1295 (reduce
1296 concat
1297 (for [eye (eyes creature)]
1298 (vision-kernel creature eye))))
1299 #+END_SRC
1300 #+end_listing
1302 #+caption: Simulated vision with a test creature and the
1303 #+caption: human-like eye approximation. Notice how each channel
1304 #+caption: of the eye responds differently to the differently
1305 #+caption: colored balls.
1306 #+name: worm-vision-test.
1307 #+ATTR_LaTeX: :width 13cm
1308 [[./images/worm-vision.png]]
1310 The vision code is not much more complicated than the body code,
1311 and enables multiple further paths for simulated vision. For
1312 example, it is quite easy to create bifocal vision -- you just
1313 make two eyes next to each other in blender! It is also possible
1314 to encode vision transforms in the retinal files. For example, the
1315 human like retina file in figure \ref{retina} approximates a
1316 log-polar transform.
1318 This vision code has already been absorbed by the jMonkeyEngine
1319 community and is now (in modified form) part of a system for
1320 capturing in-game video to a file.
1322 ** ...but hearing must be built from scratch
1324 At the end of this chapter I will have simulated ears that work the
1325 same way as the simulated eyes in the last chapter. I will be able to
1326 place any number of ear-nodes in a Blender file, and they will bind to
1327 the closest physical object and follow it as it moves around. Each ear
1328 will provide access to the sound data it picks up between every frame.
1330 Hearing is one of the more difficult senses to simulate, because there
1331 is less support for obtaining the actual sound data that is processed
1332 by jMonkeyEngine3. There is no "split-screen" support for rendering
1333 sound from different points of view, and there is no way to directly
1334 access the rendered sound data.
1336 =CORTEX='s hearing is unique because it does not have any
1337 limitations compared to other simulation environments. As far as I
1338 know, there is no other system that supports multiple listeners,
1339 and the sound demo at the end of this chapter is the first time
1340 it's been done in a video game environment.
1342 *** Brief Description of jMonkeyEngine's Sound System
1344 jMonkeyEngine's sound system works as follows:
1346 - jMonkeyEngine uses the =AppSettings= for the particular
1347 application to determine what sort of =AudioRenderer= should be
1348 used.
1349 - Although some support is provided for multiple AudioRenderer
1350 backends, jMonkeyEngine at the time of this writing will either
1351 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1352 - jMonkeyEngine tries to figure out what sort of system you're
1353 running and extracts the appropriate native libraries.
1354 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1355 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1356 - =OpenAL= renders the 3D sound and feeds the rendered sound
1357 directly to any of various sound output devices with which it
1358 knows how to communicate.
1360 A consequence of this is that there's no way to access the actual
1361 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1362 one /listener/ (it renders sound data from only one perspective),
1363 which normally isn't a problem for games, but becomes a problem
1364 when trying to make multiple AI creatures that can each hear the
1365 world from a different perspective.
1367 To make many AI creatures in jMonkeyEngine that can each hear the
1368 world from their own perspective, or to make a single creature with
1369 many ears, it is necessary to go all the way back to =OpenAL= and
1370 implement support for simulated hearing there.
1372 *** Extending =OpenAl=
1374 Extending =OpenAL= to support multiple listeners requires 500
1375 lines of =C= code and is too complicated to mention here. Instead,
1376 I will show a small amount of extension code and go over the high
1377 level strategy. Full source is of course available with the
1378 =CORTEX= distribution if you're interested.
1380 =OpenAL= goes to great lengths to support many different systems,
1381 all with different sound capabilities and interfaces. It
1382 accomplishes this difficult task by providing code for many
1383 different sound backends in pseudo-objects called /Devices/.
1384 There's a device for the Linux Open Sound System and the Advanced
1385 Linux Sound Architecture, there's one for Direct Sound on Windows,
1386 and there's even one for Solaris. =OpenAL= solves the problem of
1387 platform independence by providing all these Devices.
1389 Wrapper libraries such as LWJGL are free to examine the system on
1390 which they are running and then select an appropriate device for
1391 that system.
1393 There are also a few "special" devices that don't interface with
1394 any particular system. These include the Null Device, which
1395 doesn't do anything, and the Wave Device, which writes whatever
1396 sound it receives to a file, if everything has been set up
1397 correctly when configuring =OpenAL=.
1399 Actual mixing (Doppler shift and distance.environment-based
1400 attenuation) of the sound data happens in the Devices, and they
1401 are the only point in the sound rendering process where this data
1402 is available.
1404 Therefore, in order to support multiple listeners, and get the
1405 sound data in a form that the AIs can use, it is necessary to
1406 create a new Device which supports this feature.
1408 Adding a device to OpenAL is rather tricky -- there are five
1409 separate files in the =OpenAL= source tree that must be modified
1410 to do so. I named my device the "Multiple Audio Send" Device, or
1411 =Send= Device for short, since it sends audio data back to the
1412 calling application like an Aux-Send cable on a mixing board.
1414 The main idea behind the Send device is to take advantage of the
1415 fact that LWJGL only manages one /context/ when using OpenAL. A
1416 /context/ is like a container that holds samples and keeps track
1417 of where the listener is. In order to support multiple listeners,
1418 the Send device identifies the LWJGL context as the master
1419 context, and creates any number of slave contexts to represent
1420 additional listeners. Every time the device renders sound, it
1421 synchronizes every source from the master LWJGL context to the
1422 slave contexts. Then, it renders each context separately, using a
1423 different listener for each one. The rendered sound is made
1424 available via JNI to jMonkeyEngine.
1426 Switching between contexts is not the normal operation of a
1427 Device, and one of the problems with doing so is that a Device
1428 normally keeps around a few pieces of state such as the
1429 =ClickRemoval= array above which will become corrupted if the
1430 contexts are not rendered in parallel. The solution is to create a
1431 copy of this normally global device state for each context, and
1432 copy it back and forth into and out of the actual device state
1433 whenever a context is rendered.
1435 The core of the =Send= device is the =syncSources= function, which
1436 does the job of copying all relevant data from one context to
1437 another.
1439 #+caption: Program for extending =OpenAL= to support multiple
1440 #+caption: listeners via context copying/switching.
1441 #+name: sync-openal-sources
1442 #+begin_listing c
1443 #+BEGIN_SRC c
1444 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1445 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1446 ALuint master = masterSource->source;
1447 ALuint slave = slaveSource->source;
1448 ALCcontext *current = alcGetCurrentContext();
1450 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1451 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1452 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1453 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1454 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1455 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1456 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1457 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1458 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1459 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1460 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1461 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1462 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1464 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1465 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1466 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1468 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1469 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1471 alcMakeContextCurrent(masterCtx);
1472 ALint source_type;
1473 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1475 // Only static sources are currently synchronized!
1476 if (AL_STATIC == source_type){
1477 ALint master_buffer;
1478 ALint slave_buffer;
1479 alGetSourcei(master, AL_BUFFER, &master_buffer);
1480 alcMakeContextCurrent(slaveCtx);
1481 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1482 if (master_buffer != slave_buffer){
1483 alSourcei(slave, AL_BUFFER, master_buffer);
1487 // Synchronize the state of the two sources.
1488 alcMakeContextCurrent(masterCtx);
1489 ALint masterState;
1490 ALint slaveState;
1492 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1493 alcMakeContextCurrent(slaveCtx);
1494 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1496 if (masterState != slaveState){
1497 switch (masterState){
1498 case AL_INITIAL : alSourceRewind(slave); break;
1499 case AL_PLAYING : alSourcePlay(slave); break;
1500 case AL_PAUSED : alSourcePause(slave); break;
1501 case AL_STOPPED : alSourceStop(slave); break;
1504 // Restore whatever context was previously active.
1505 alcMakeContextCurrent(current);
1507 #+END_SRC
1508 #+end_listing
1510 With this special context-switching device, and some ugly JNI
1511 bindings that are not worth mentioning, =CORTEX= gains the ability
1512 to access multiple sound streams from =OpenAL=.
1514 #+caption: Program to create an ear from a blender empty node. The ear
1515 #+caption: follows around the nearest physical object and passes
1516 #+caption: all sensory data to a continuation function.
1517 #+name: add-ear
1518 #+begin_listing clojure
1519 #+BEGIN_SRC clojure
1520 (defn add-ear!
1521 "Create a Listener centered on the current position of 'ear
1522 which follows the closest physical node in 'creature and
1523 sends sound data to 'continuation."
1524 [#^Application world #^Node creature #^Spatial ear continuation]
1525 (let [target (closest-node creature ear)
1526 lis (Listener.)
1527 audio-renderer (.getAudioRenderer world)
1528 sp (hearing-pipeline continuation)]
1529 (.setLocation lis (.getWorldTranslation ear))
1530 (.setRotation lis (.getWorldRotation ear))
1531 (bind-sense target lis)
1532 (update-listener-velocity! target lis)
1533 (.addListener audio-renderer lis)
1534 (.registerSoundProcessor audio-renderer lis sp)))
1535 #+END_SRC
1536 #+end_listing
1538 The =Send= device, unlike most of the other devices in =OpenAL=,
1539 does not render sound unless asked. This enables the system to
1540 slow down or speed up depending on the needs of the AIs who are
1541 using it to listen. If the device tried to render samples in
1542 real-time, a complicated AI whose mind takes 100 seconds of
1543 computer time to simulate 1 second of AI-time would miss almost
1544 all of the sound in its environment!
1546 #+caption: Program to enable arbitrary hearing in =CORTEX=
1547 #+name: hearing
1548 #+begin_listing clojure
1549 #+BEGIN_SRC clojure
1550 (defn hearing-kernel
1551 "Returns a function which returns auditory sensory data when called
1552 inside a running simulation."
1553 [#^Node creature #^Spatial ear]
1554 (let [hearing-data (atom [])
1555 register-listener!
1556 (runonce
1557 (fn [#^Application world]
1558 (add-ear!
1559 world creature ear
1560 (comp #(reset! hearing-data %)
1561 byteBuffer->pulse-vector))))]
1562 (fn [#^Application world]
1563 (register-listener! world)
1564 (let [data @hearing-data
1565 topology
1566 (vec (map #(vector % 0) (range 0 (count data))))]
1567 [topology data]))))
1569 (defn hearing!
1570 "Endow the creature in a particular world with the sense of
1571 hearing. Will return a sequence of functions, one for each ear,
1572 which when called will return the auditory data from that ear."
1573 [#^Node creature]
1574 (for [ear (ears creature)]
1575 (hearing-kernel creature ear)))
1576 #+END_SRC
1577 #+end_listing
1579 Armed with these functions, =CORTEX= is able to test possibly the
1580 first ever instance of multiple listeners in a video game engine
1581 based simulation!
1583 #+caption: Here a simple creature responds to sound by changing
1584 #+caption: its color from gray to green when the total volume
1585 #+caption: goes over a threshold.
1586 #+name: sound-test
1587 #+begin_listing java
1588 #+BEGIN_SRC java
1589 /**
1590 * Respond to sound! This is the brain of an AI entity that
1591 * hears its surroundings and reacts to them.
1592 */
1593 public void process(ByteBuffer audioSamples,
1594 int numSamples, AudioFormat format) {
1595 audioSamples.clear();
1596 byte[] data = new byte[numSamples];
1597 float[] out = new float[numSamples];
1598 audioSamples.get(data);
1599 FloatSampleTools.
1600 byte2floatInterleaved
1601 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1603 float max = Float.NEGATIVE_INFINITY;
1604 for (float f : out){if (f > max) max = f;}
1605 audioSamples.clear();
1607 if (max > 0.1){
1608 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1610 else {
1611 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1613 #+END_SRC
1614 #+end_listing
1616 #+caption: First ever simulation of multiple listeners in =CORTEX=.
1617 #+caption: Each cube is a creature which processes sound data with
1618 #+caption: the =process= function from listing \ref{sound-test}.
1619 #+caption: the ball is constantly emitting a pure tone of
1620 #+caption: constant volume. As it approaches the cubes, they each
1621 #+caption: change color in response to the sound.
1622 #+name: sound-cubes.
1623 #+ATTR_LaTeX: :width 10cm
1624 [[./images/java-hearing-test.png]]
1626 This system of hearing has also been co-opted by the
1627 jMonkeyEngine3 community and is used to record audio for demo
1628 videos.
1630 ** Hundreds of hair-like elements provide a sense of touch
1632 Touch is critical to navigation and spatial reasoning and as such I
1633 need a simulated version of it to give to my AI creatures.
1635 Human skin has a wide array of touch sensors, each of which
1636 specialize in detecting different vibrational modes and pressures.
1637 These sensors can integrate a vast expanse of skin (i.e. your
1638 entire palm), or a tiny patch of skin at the tip of your finger.
1639 The hairs of the skin help detect objects before they even come
1640 into contact with the skin proper.
1642 However, touch in my simulated world can not exactly correspond to
1643 human touch because my creatures are made out of completely rigid
1644 segments that don't deform like human skin.
1646 Instead of measuring deformation or vibration, I surround each
1647 rigid part with a plenitude of hair-like objects (/feelers/) which
1648 do not interact with the physical world. Physical objects can pass
1649 through them with no effect. The feelers are able to tell when
1650 other objects pass through them, and they constantly report how
1651 much of their extent is covered. So even though the creature's body
1652 parts do not deform, the feelers create a margin around those body
1653 parts which achieves a sense of touch which is a hybrid between a
1654 human's sense of deformation and sense from hairs.
1656 Implementing touch in jMonkeyEngine follows a different technical
1657 route than vision and hearing. Those two senses piggybacked off
1658 jMonkeyEngine's 3D audio and video rendering subsystems. To
1659 simulate touch, I use jMonkeyEngine's physics system to execute
1660 many small collision detections, one for each feeler. The placement
1661 of the feelers is determined by a UV-mapped image which shows where
1662 each feeler should be on the 3D surface of the body.
1664 *** Defining Touch Meta-Data in Blender
1666 Each geometry can have a single UV map which describes the
1667 position of the feelers which will constitute its sense of touch.
1668 This image path is stored under the ``touch'' key. The image itself
1669 is black and white, with black meaning a feeler length of 0 (no
1670 feeler is present) and white meaning a feeler length of =scale=,
1671 which is a float stored under the key "scale".
1673 #+caption: Touch does not use empty nodes, to store metadata,
1674 #+caption: because the metadata of each solid part of a
1675 #+caption: creature's body is sufficient.
1676 #+name: touch-meta-data
1677 #+begin_listing clojure
1678 #+BEGIN_SRC clojure
1679 (defn tactile-sensor-profile
1680 "Return the touch-sensor distribution image in BufferedImage format,
1681 or nil if it does not exist."
1682 [#^Geometry obj]
1683 (if-let [image-path (meta-data obj "touch")]
1684 (load-image image-path)))
1686 (defn tactile-scale
1687 "Return the length of each feeler. Default scale is 0.01
1688 jMonkeyEngine units."
1689 [#^Geometry obj]
1690 (if-let [scale (meta-data obj "scale")]
1691 scale 0.1))
1692 #+END_SRC
1693 #+end_listing
1695 Here is an example of a UV-map which specifies the position of
1696 touch sensors along the surface of the upper segment of a fingertip.
1698 #+caption: This is the tactile-sensor-profile for the upper segment
1699 #+caption: of a fingertip. It defines regions of high touch sensitivity
1700 #+caption: (where there are many white pixels) and regions of low
1701 #+caption: sensitivity (where white pixels are sparse).
1702 #+name: fingertip-UV
1703 #+ATTR_LaTeX: :width 13cm
1704 [[./images/finger-UV.png]]
1706 *** Implementation Summary
1708 To simulate touch there are three conceptual steps. For each solid
1709 object in the creature, you first have to get UV image and scale
1710 parameter which define the position and length of the feelers.
1711 Then, you use the triangles which comprise the mesh and the UV
1712 data stored in the mesh to determine the world-space position and
1713 orientation of each feeler. Then once every frame, update these
1714 positions and orientations to match the current position and
1715 orientation of the object, and use physics collision detection to
1716 gather tactile data.
1718 Extracting the meta-data has already been described. The third
1719 step, physics collision detection, is handled in =touch-kernel=.
1720 Translating the positions and orientations of the feelers from the
1721 UV-map to world-space is itself a three-step process.
1723 - Find the triangles which make up the mesh in pixel-space and in
1724 world-space. \\(=triangles=, =pixel-triangles=).
1726 - Find the coordinates of each feeler in world-space. These are
1727 the origins of the feelers. (=feeler-origins=).
1729 - Calculate the normals of the triangles in world space, and add
1730 them to each of the origins of the feelers. These are the
1731 normalized coordinates of the tips of the feelers.
1732 (=feeler-tips=).
1734 *** Triangle Math
1736 The rigid objects which make up a creature have an underlying
1737 =Geometry=, which is a =Mesh= plus a =Material= and other
1738 important data involved with displaying the object.
1740 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1741 vertices which have coordinates in world space and UV space.
1743 Here, =triangles= gets all the world-space triangles which
1744 comprise a mesh, while =pixel-triangles= gets those same triangles
1745 expressed in pixel coordinates (which are UV coordinates scaled to
1746 fit the height and width of the UV image).
1748 #+caption: Programs to extract triangles from a geometry and get
1749 #+caption: their vertices in both world and UV-coordinates.
1750 #+name: get-triangles
1751 #+begin_listing clojure
1752 #+BEGIN_SRC clojure
1753 (defn triangle
1754 "Get the triangle specified by triangle-index from the mesh."
1755 [#^Geometry geo triangle-index]
1756 (triangle-seq
1757 (let [scratch (Triangle.)]
1758 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1760 (defn triangles
1761 "Return a sequence of all the Triangles which comprise a given
1762 Geometry."
1763 [#^Geometry geo]
1764 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1766 (defn triangle-vertex-indices
1767 "Get the triangle vertex indices of a given triangle from a given
1768 mesh."
1769 [#^Mesh mesh triangle-index]
1770 (let [indices (int-array 3)]
1771 (.getTriangle mesh triangle-index indices)
1772 (vec indices)))
1774 (defn vertex-UV-coord
1775 "Get the UV-coordinates of the vertex named by vertex-index"
1776 [#^Mesh mesh vertex-index]
1777 (let [UV-buffer
1778 (.getData
1779 (.getBuffer
1780 mesh
1781 VertexBuffer$Type/TexCoord))]
1782 [(.get UV-buffer (* vertex-index 2))
1783 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1785 (defn pixel-triangle [#^Geometry geo image index]
1786 (let [mesh (.getMesh geo)
1787 width (.getWidth image)
1788 height (.getHeight image)]
1789 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1790 (map (partial vertex-UV-coord mesh)
1791 (triangle-vertex-indices mesh index))))))
1793 (defn pixel-triangles
1794 "The pixel-space triangles of the Geometry, in the same order as
1795 (triangles geo)"
1796 [#^Geometry geo image]
1797 (let [height (.getHeight image)
1798 width (.getWidth image)]
1799 (map (partial pixel-triangle geo image)
1800 (range (.getTriangleCount (.getMesh geo))))))
1801 #+END_SRC
1802 #+end_listing
1804 *** The Affine Transform from one Triangle to Another
1806 =pixel-triangles= gives us the mesh triangles expressed in pixel
1807 coordinates and =triangles= gives us the mesh triangles expressed
1808 in world coordinates. The tactile-sensor-profile gives the
1809 position of each feeler in pixel-space. In order to convert
1810 pixel-space coordinates into world-space coordinates we need
1811 something that takes coordinates on the surface of one triangle
1812 and gives the corresponding coordinates on the surface of another
1813 triangle.
1815 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1816 into any other by a combination of translation, scaling, and
1817 rotation. The affine transformation from one triangle to another
1818 is readily computable if the triangle is expressed in terms of a
1819 $4x4$ matrix.
1821 #+BEGIN_LaTeX
1822 $$
1823 \begin{bmatrix}
1824 x_1 & x_2 & x_3 & n_x \\
1825 y_1 & y_2 & y_3 & n_y \\
1826 z_1 & z_2 & z_3 & n_z \\
1827 1 & 1 & 1 & 1
1828 \end{bmatrix}
1829 $$
1830 #+END_LaTeX
1832 Here, the first three columns of the matrix are the vertices of
1833 the triangle. The last column is the right-handed unit normal of
1834 the triangle.
1836 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1837 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1838 is $T_{2}T_{1}^{-1}$.
1840 The clojure code below recapitulates the formulas above, using
1841 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1842 transformation.
1844 #+caption: Program to interpret triangles as affine transforms.
1845 #+name: triangle-affine
1846 #+begin_listing clojure
1847 #+BEGIN_SRC clojure
1848 (defn triangle->matrix4f
1849 "Converts the triangle into a 4x4 matrix: The first three columns
1850 contain the vertices of the triangle; the last contains the unit
1851 normal of the triangle. The bottom row is filled with 1s."
1852 [#^Triangle t]
1853 (let [mat (Matrix4f.)
1854 [vert-1 vert-2 vert-3]
1855 (mapv #(.get t %) (range 3))
1856 unit-normal (do (.calculateNormal t)(.getNormal t))
1857 vertices [vert-1 vert-2 vert-3 unit-normal]]
1858 (dorun
1859 (for [row (range 4) col (range 3)]
1860 (do
1861 (.set mat col row (.get (vertices row) col))
1862 (.set mat 3 row 1)))) mat))
1864 (defn triangles->affine-transform
1865 "Returns the affine transformation that converts each vertex in the
1866 first triangle into the corresponding vertex in the second
1867 triangle."
1868 [#^Triangle tri-1 #^Triangle tri-2]
1869 (.mult
1870 (triangle->matrix4f tri-2)
1871 (.invert (triangle->matrix4f tri-1))))
1872 #+END_SRC
1873 #+end_listing
1875 *** Triangle Boundaries
1877 For efficiency's sake I will divide the tactile-profile image into
1878 small squares which inscribe each pixel-triangle, then extract the
1879 points which lie inside the triangle and map them to 3D-space using
1880 =triangle-transform= above. To do this I need a function,
1881 =convex-bounds= which finds the smallest box which inscribes a 2D
1882 triangle.
1884 =inside-triangle?= determines whether a point is inside a triangle
1885 in 2D pixel-space.
1887 #+caption: Program to efficiently determine point inclusion
1888 #+caption: in a triangle.
1889 #+name: in-triangle
1890 #+begin_listing clojure
1891 #+BEGIN_SRC clojure
1892 (defn convex-bounds
1893 "Returns the smallest square containing the given vertices, as a
1894 vector of integers [left top width height]."
1895 [verts]
1896 (let [xs (map first verts)
1897 ys (map second verts)
1898 x0 (Math/floor (apply min xs))
1899 y0 (Math/floor (apply min ys))
1900 x1 (Math/ceil (apply max xs))
1901 y1 (Math/ceil (apply max ys))]
1902 [x0 y0 (- x1 x0) (- y1 y0)]))
1904 (defn same-side?
1905 "Given the points p1 and p2 and the reference point ref, is point p
1906 on the same side of the line that goes through p1 and p2 as ref is?"
1907 [p1 p2 ref p]
1908 (<=
1910 (.dot
1911 (.cross (.subtract p2 p1) (.subtract p p1))
1912 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1914 (defn inside-triangle?
1915 "Is the point inside the triangle?"
1916 {:author "Dylan Holmes"}
1917 [#^Triangle tri #^Vector3f p]
1918 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1919 (and
1920 (same-side? vert-1 vert-2 vert-3 p)
1921 (same-side? vert-2 vert-3 vert-1 p)
1922 (same-side? vert-3 vert-1 vert-2 p))))
1923 #+END_SRC
1924 #+end_listing
1926 *** Feeler Coordinates
1928 The triangle-related functions above make short work of
1929 calculating the positions and orientations of each feeler in
1930 world-space.
1932 #+caption: Program to get the coordinates of ``feelers '' in
1933 #+caption: both world and UV-coordinates.
1934 #+name: feeler-coordinates
1935 #+begin_listing clojure
1936 #+BEGIN_SRC clojure
1937 (defn feeler-pixel-coords
1938 "Returns the coordinates of the feelers in pixel space in lists, one
1939 list for each triangle, ordered in the same way as (triangles) and
1940 (pixel-triangles)."
1941 [#^Geometry geo image]
1942 (map
1943 (fn [pixel-triangle]
1944 (filter
1945 (fn [coord]
1946 (inside-triangle? (->triangle pixel-triangle)
1947 (->vector3f coord)))
1948 (white-coordinates image (convex-bounds pixel-triangle))))
1949 (pixel-triangles geo image)))
1951 (defn feeler-world-coords
1952 "Returns the coordinates of the feelers in world space in lists, one
1953 list for each triangle, ordered in the same way as (triangles) and
1954 (pixel-triangles)."
1955 [#^Geometry geo image]
1956 (let [transforms
1957 (map #(triangles->affine-transform
1958 (->triangle %1) (->triangle %2))
1959 (pixel-triangles geo image)
1960 (triangles geo))]
1961 (map (fn [transform coords]
1962 (map #(.mult transform (->vector3f %)) coords))
1963 transforms (feeler-pixel-coords geo image))))
1964 #+END_SRC
1965 #+end_listing
1967 #+caption: Program to get the position of the base and tip of
1968 #+caption: each ``feeler''
1969 #+name: feeler-tips
1970 #+begin_listing clojure
1971 #+BEGIN_SRC clojure
1972 (defn feeler-origins
1973 "The world space coordinates of the root of each feeler."
1974 [#^Geometry geo image]
1975 (reduce concat (feeler-world-coords geo image)))
1977 (defn feeler-tips
1978 "The world space coordinates of the tip of each feeler."
1979 [#^Geometry geo image]
1980 (let [world-coords (feeler-world-coords geo image)
1981 normals
1982 (map
1983 (fn [triangle]
1984 (.calculateNormal triangle)
1985 (.clone (.getNormal triangle)))
1986 (map ->triangle (triangles geo)))]
1988 (mapcat (fn [origins normal]
1989 (map #(.add % normal) origins))
1990 world-coords normals)))
1992 (defn touch-topology
1993 [#^Geometry geo image]
1994 (collapse (reduce concat (feeler-pixel-coords geo image))))
1995 #+END_SRC
1996 #+end_listing
1998 *** Simulated Touch
2000 Now that the functions to construct feelers are complete,
2001 =touch-kernel= generates functions to be called from within a
2002 simulation that perform the necessary physics collisions to
2003 collect tactile data, and =touch!= recursively applies it to every
2004 node in the creature.
2006 #+caption: Efficient program to transform a ray from
2007 #+caption: one position to another.
2008 #+name: set-ray
2009 #+begin_listing clojure
2010 #+BEGIN_SRC clojure
2011 (defn set-ray [#^Ray ray #^Matrix4f transform
2012 #^Vector3f origin #^Vector3f tip]
2013 ;; Doing everything locally reduces garbage collection by enough to
2014 ;; be worth it.
2015 (.mult transform origin (.getOrigin ray))
2016 (.mult transform tip (.getDirection ray))
2017 (.subtractLocal (.getDirection ray) (.getOrigin ray))
2018 (.normalizeLocal (.getDirection ray)))
2019 #+END_SRC
2020 #+end_listing
2022 #+caption: This is the core of touch in =CORTEX= each feeler
2023 #+caption: follows the object it is bound to, reporting any
2024 #+caption: collisions that may happen.
2025 #+name: touch-kernel
2026 #+begin_listing clojure
2027 #+BEGIN_SRC clojure
2028 (defn touch-kernel
2029 "Constructs a function which will return tactile sensory data from
2030 'geo when called from inside a running simulation"
2031 [#^Geometry geo]
2032 (if-let
2033 [profile (tactile-sensor-profile geo)]
2034 (let [ray-reference-origins (feeler-origins geo profile)
2035 ray-reference-tips (feeler-tips geo profile)
2036 ray-length (tactile-scale geo)
2037 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
2038 topology (touch-topology geo profile)
2039 correction (float (* ray-length -0.2))]
2040 ;; slight tolerance for very close collisions.
2041 (dorun
2042 (map (fn [origin tip]
2043 (.addLocal origin (.mult (.subtract tip origin)
2044 correction)))
2045 ray-reference-origins ray-reference-tips))
2046 (dorun (map #(.setLimit % ray-length) current-rays))
2047 (fn [node]
2048 (let [transform (.getWorldMatrix geo)]
2049 (dorun
2050 (map (fn [ray ref-origin ref-tip]
2051 (set-ray ray transform ref-origin ref-tip))
2052 current-rays ray-reference-origins
2053 ray-reference-tips))
2054 (vector
2055 topology
2056 (vec
2057 (for [ray current-rays]
2058 (do
2059 (let [results (CollisionResults.)]
2060 (.collideWith node ray results)
2061 (let [touch-objects
2062 (filter #(not (= geo (.getGeometry %)))
2063 results)
2064 limit (.getLimit ray)]
2065 [(if (empty? touch-objects)
2066 limit
2067 (let [response
2068 (apply min (map #(.getDistance %)
2069 touch-objects))]
2070 (FastMath/clamp
2071 (float
2072 (if (> response limit) (float 0.0)
2073 (+ response correction)))
2074 (float 0.0)
2075 limit)))
2076 limit])))))))))))
2077 #+END_SRC
2078 #+end_listing
2080 Armed with the =touch!= function, =CORTEX= becomes capable of
2081 giving creatures a sense of touch. A simple test is to create a
2082 cube that is outfitted with a uniform distribution of touch
2083 sensors. It can feel the ground and any balls that it touches.
2085 #+caption: =CORTEX= interface for creating touch in a simulated
2086 #+caption: creature.
2087 #+name: touch
2088 #+begin_listing clojure
2089 #+BEGIN_SRC clojure
2090 (defn touch!
2091 "Endow the creature with the sense of touch. Returns a sequence of
2092 functions, one for each body part with a tactile-sensor-profile,
2093 each of which when called returns sensory data for that body part."
2094 [#^Node creature]
2095 (filter
2096 (comp not nil?)
2097 (map touch-kernel
2098 (filter #(isa? (class %) Geometry)
2099 (node-seq creature)))))
2100 #+END_SRC
2101 #+end_listing
2103 The tactile-sensor-profile image for the touch cube is a simple
2104 cross with a uniform distribution of touch sensors:
2106 #+caption: The touch profile for the touch-cube. Each pure white
2107 #+caption: pixel defines a touch sensitive feeler.
2108 #+name: touch-cube-uv-map
2109 #+ATTR_LaTeX: :width 7cm
2110 [[./images/touch-profile.png]]
2112 #+caption: The touch cube reacts to cannonballs. The black, red,
2113 #+caption: and white cross on the right is a visual display of
2114 #+caption: the creature's touch. White means that it is feeling
2115 #+caption: something strongly, black is not feeling anything,
2116 #+caption: and gray is in-between. The cube can feel both the
2117 #+caption: floor and the ball. Notice that when the ball causes
2118 #+caption: the cube to tip, that the bottom face can still feel
2119 #+caption: part of the ground.
2120 #+name: touch-cube-uv-map-2
2121 #+ATTR_LaTeX: :width 15cm
2122 [[./images/touch-cube.png]]
2124 ** Proprioception provides knowledge of your own body's position
2126 Close your eyes, and touch your nose with your right index finger.
2127 How did you do it? You could not see your hand, and neither your
2128 hand nor your nose could use the sense of touch to guide the path
2129 of your hand. There are no sound cues, and Taste and Smell
2130 certainly don't provide any help. You know where your hand is
2131 without your other senses because of Proprioception.
2133 Humans can sometimes loose this sense through viral infections or
2134 damage to the spinal cord or brain, and when they do, they loose
2135 the ability to control their own bodies without looking directly at
2136 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2137 Hat]] (\cite{man-wife-hat}), a woman named Christina looses this
2138 sense and has to learn how to move by carefully watching her arms
2139 and legs. She describes proprioception as the "eyes of the body,
2140 the way the body sees itself".
2142 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2143 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2144 positions of each body part by monitoring muscle strain and length.
2146 It's clear that this is a vital sense for fluid, graceful movement.
2147 It's also particularly easy to implement in jMonkeyEngine.
2149 My simulated proprioception calculates the relative angles of each
2150 joint from the rest position defined in the Blender file. This
2151 simulates the muscle-spindles and joint capsules. I will deal with
2152 Golgi tendon organs, which calculate muscle strain, in the next
2153 chapter.
2155 *** Helper functions
2157 =absolute-angle= calculates the angle between two vectors,
2158 relative to a third axis vector. This angle is the number of
2159 radians you have to move counterclockwise around the axis vector
2160 to get from the first to the second vector. It is not commutative
2161 like a normal dot-product angle is.
2163 The purpose of these functions is to build a system of angle
2164 measurement that is biologically plausible.
2166 #+caption: Program to measure angles along a vector
2167 #+name: helpers
2168 #+begin_listing clojure
2169 #+BEGIN_SRC clojure
2170 (defn right-handed?
2171 "true iff the three vectors form a right handed coordinate
2172 system. The three vectors do not have to be normalized or
2173 orthogonal."
2174 [vec1 vec2 vec3]
2175 (pos? (.dot (.cross vec1 vec2) vec3)))
2177 (defn absolute-angle
2178 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2179 [0 (* 2 Math/PI)]."
2180 [vec1 vec2 axis]
2181 (let [angle (.angleBetween vec1 vec2)]
2182 (if (right-handed? vec1 vec2 axis)
2183 angle (- (* 2 Math/PI) angle))))
2184 #+END_SRC
2185 #+end_listing
2187 *** Proprioception Kernel
2189 Given a joint, =proprioception-kernel= produces a function that
2190 calculates the Euler angles between the objects the joint
2191 connects. The only tricky part here is making the angles relative
2192 to the joint's initial ``straightness''.
2194 #+caption: Program to return biologically reasonable proprioceptive
2195 #+caption: data for each joint.
2196 #+name: proprioception
2197 #+begin_listing clojure
2198 #+BEGIN_SRC clojure
2199 (defn proprioception-kernel
2200 "Returns a function which returns proprioceptive sensory data when
2201 called inside a running simulation."
2202 [#^Node parts #^Node joint]
2203 (let [[obj-a obj-b] (joint-targets parts joint)
2204 joint-rot (.getWorldRotation joint)
2205 x0 (.mult joint-rot Vector3f/UNIT_X)
2206 y0 (.mult joint-rot Vector3f/UNIT_Y)
2207 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2208 (fn []
2209 (let [rot-a (.clone (.getWorldRotation obj-a))
2210 rot-b (.clone (.getWorldRotation obj-b))
2211 x (.mult rot-a x0)
2212 y (.mult rot-a y0)
2213 z (.mult rot-a z0)
2215 X (.mult rot-b x0)
2216 Y (.mult rot-b y0)
2217 Z (.mult rot-b z0)
2218 heading (Math/atan2 (.dot X z) (.dot X x))
2219 pitch (Math/atan2 (.dot X y) (.dot X x))
2221 ;; rotate x-vector back to origin
2222 reverse
2223 (doto (Quaternion.)
2224 (.fromAngleAxis
2225 (.angleBetween X x)
2226 (let [cross (.normalize (.cross X x))]
2227 (if (= 0 (.length cross)) y cross))))
2228 roll (absolute-angle (.mult reverse Y) y x)]
2229 [heading pitch roll]))))
2231 (defn proprioception!
2232 "Endow the creature with the sense of proprioception. Returns a
2233 sequence of functions, one for each child of the \"joints\" node in
2234 the creature, which each report proprioceptive information about
2235 that joint."
2236 [#^Node creature]
2237 ;; extract the body's joints
2238 (let [senses (map (partial proprioception-kernel creature)
2239 (joints creature))]
2240 (fn []
2241 (map #(%) senses))))
2242 #+END_SRC
2243 #+end_listing
2245 =proprioception!= maps =proprioception-kernel= across all the
2246 joints of the creature. It uses the same list of joints that
2247 =joints= uses. Proprioception is the easiest sense to implement in
2248 =CORTEX=, and it will play a crucial role when efficiently
2249 implementing empathy.
2251 #+caption: In the upper right corner, the three proprioceptive
2252 #+caption: angle measurements are displayed. Red is yaw, Green is
2253 #+caption: pitch, and White is roll.
2254 #+name: proprio
2255 #+ATTR_LaTeX: :width 11cm
2256 [[./images/proprio.png]]
2258 ** Muscles contain both sensors and effectors
2260 Surprisingly enough, terrestrial creatures only move by using
2261 torque applied about their joints. There's not a single straight
2262 line of force in the human body at all! (A straight line of force
2263 would correspond to some sort of jet or rocket propulsion.)
2265 In humans, muscles are composed of muscle fibers which can contract
2266 to exert force. The muscle fibers which compose a muscle are
2267 partitioned into discrete groups which are each controlled by a
2268 single alpha motor neuron. A single alpha motor neuron might
2269 control as little as three or as many as one thousand muscle
2270 fibers. When the alpha motor neuron is engaged by the spinal cord,
2271 it activates all of the muscle fibers to which it is attached. The
2272 spinal cord generally engages the alpha motor neurons which control
2273 few muscle fibers before the motor neurons which control many
2274 muscle fibers. This recruitment strategy allows for precise
2275 movements at low strength. The collection of all motor neurons that
2276 control a muscle is called the motor pool. The brain essentially
2277 says "activate 30% of the motor pool" and the spinal cord recruits
2278 motor neurons until 30% are activated. Since the distribution of
2279 power among motor neurons is unequal and recruitment goes from
2280 weakest to strongest, the first 30% of the motor pool might be 5%
2281 of the strength of the muscle.
2283 My simulated muscles follow a similar design: Each muscle is
2284 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2285 the array represents a motor neuron which controls a number of
2286 muscle fibers equal to the value of the entry. Each muscle has a
2287 scalar strength factor which determines the total force the muscle
2288 can exert when all motor neurons are activated. The effector
2289 function for a muscle takes a number to index into the motor pool,
2290 and then "activates" all the motor neurons whose index is lower or
2291 equal to the number. Each motor-neuron will apply force in
2292 proportion to its value in the array. Lower values cause less
2293 force. The lower values can be put at the "beginning" of the 1-D
2294 array to simulate the layout of actual human muscles, which are
2295 capable of more precise movements when exerting less force. Or, the
2296 motor pool can simulate more exotic recruitment strategies which do
2297 not correspond to human muscles.
2299 This 1D array is defined in an image file for ease of
2300 creation/visualization. Here is an example muscle profile image.
2302 #+caption: A muscle profile image that describes the strengths
2303 #+caption: of each motor neuron in a muscle. White is weakest
2304 #+caption: and dark red is strongest. This particular pattern
2305 #+caption: has weaker motor neurons at the beginning, just
2306 #+caption: like human muscle.
2307 #+name: muscle-recruit
2308 #+ATTR_LaTeX: :width 7cm
2309 [[./images/basic-muscle.png]]
2311 *** Muscle meta-data
2313 #+caption: Program to deal with loading muscle data from a Blender
2314 #+caption: file's metadata.
2315 #+name: motor-pool
2316 #+begin_listing clojure
2317 #+BEGIN_SRC clojure
2318 (defn muscle-profile-image
2319 "Get the muscle-profile image from the node's blender meta-data."
2320 [#^Node muscle]
2321 (if-let [image (meta-data muscle "muscle")]
2322 (load-image image)))
2324 (defn muscle-strength
2325 "Return the strength of this muscle, or 1 if it is not defined."
2326 [#^Node muscle]
2327 (if-let [strength (meta-data muscle "strength")]
2328 strength 1))
2330 (defn motor-pool
2331 "Return a vector where each entry is the strength of the \"motor
2332 neuron\" at that part in the muscle."
2333 [#^Node muscle]
2334 (let [profile (muscle-profile-image muscle)]
2335 (vec
2336 (let [width (.getWidth profile)]
2337 (for [x (range width)]
2338 (- 255
2339 (bit-and
2340 0x0000FF
2341 (.getRGB profile x 0))))))))
2342 #+END_SRC
2343 #+end_listing
2345 Of note here is =motor-pool= which interprets the muscle-profile
2346 image in a way that allows me to use gradients between white and
2347 red, instead of shades of gray as I've been using for all the
2348 other senses. This is purely an aesthetic touch.
2350 *** Creating muscles
2352 #+caption: This is the core movement function in =CORTEX=, which
2353 #+caption: implements muscles that report on their activation.
2354 #+name: muscle-kernel
2355 #+begin_listing clojure
2356 #+BEGIN_SRC clojure
2357 (defn movement-kernel
2358 "Returns a function which when called with a integer value inside a
2359 running simulation will cause movement in the creature according
2360 to the muscle's position and strength profile. Each function
2361 returns the amount of force applied / max force."
2362 [#^Node creature #^Node muscle]
2363 (let [target (closest-node creature muscle)
2364 axis
2365 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2366 strength (muscle-strength muscle)
2368 pool (motor-pool muscle)
2369 pool-integral (reductions + pool)
2370 forces
2371 (vec (map #(float (* strength (/ % (last pool-integral))))
2372 pool-integral))
2373 control (.getControl target RigidBodyControl)]
2374 (fn [n]
2375 (let [pool-index (max 0 (min n (dec (count pool))))
2376 force (forces pool-index)]
2377 (.applyTorque control (.mult axis force))
2378 (float (/ force strength))))))
2380 (defn movement!
2381 "Endow the creature with the power of movement. Returns a sequence
2382 of functions, each of which accept an integer value and will
2383 activate their corresponding muscle."
2384 [#^Node creature]
2385 (for [muscle (muscles creature)]
2386 (movement-kernel creature muscle)))
2387 #+END_SRC
2388 #+end_listing
2391 =movement-kernel= creates a function that controls the movement
2392 of the nearest physical node to the muscle node. The muscle exerts
2393 a rotational force dependent on it's orientation to the object in
2394 the Blender file. The function returned by =movement-kernel= is
2395 also a sense function: it returns the percent of the total muscle
2396 strength that is currently being employed. This is analogous to
2397 muscle tension in humans and completes the sense of proprioception
2398 begun in the last chapter.
2400 ** =CORTEX= brings complex creatures to life!
2402 The ultimate test of =CORTEX= is to create a creature with the full
2403 gamut of senses and put it though its paces.
2405 With all senses enabled, my right hand model looks like an
2406 intricate marionette hand with several strings for each finger:
2408 #+caption: View of the hand model with all sense nodes. You can see
2409 #+caption: the joint, muscle, ear, and eye nodes here.
2410 #+name: hand-nodes-1
2411 #+ATTR_LaTeX: :width 11cm
2412 [[./images/hand-with-all-senses2.png]]
2414 #+caption: An alternate view of the hand.
2415 #+name: hand-nodes-2
2416 #+ATTR_LaTeX: :width 15cm
2417 [[./images/hand-with-all-senses3.png]]
2419 With the hand fully rigged with senses, I can run it though a test
2420 that will test everything.
2422 #+caption: Selected frames from a full test of the hand with all
2423 #+caption: senses. Note especially the interactions the hand has
2424 #+caption: with itself: it feels its own palm and fingers, and when
2425 #+caption: it curls its fingers, it sees them with its eye (which
2426 #+caption: is located in the center of the palm. The red block
2427 #+caption: appears with a pure tone sound. The hand then uses its
2428 #+caption: muscles to launch the cube!
2429 #+name: integration
2430 #+ATTR_LaTeX: :width 15cm
2431 [[./images/integration.png]]
2433 ** =CORTEX= enables many possibilities for further research
2435 Often times, the hardest part of building a system involving
2436 creatures is dealing with physics and graphics. =CORTEX= removes
2437 much of this initial difficulty and leaves researchers free to
2438 directly pursue their ideas. I hope that even novices with a
2439 passing curiosity about simulated touch or creature evolution will
2440 be able to use cortex for experimentation. =CORTEX= is a completely
2441 simulated world, and far from being a disadvantage, its simulated
2442 nature enables you to create senses and creatures that would be
2443 impossible to make in the real world.
2445 While not by any means a complete list, here are some paths
2446 =CORTEX= is well suited to help you explore:
2448 - Empathy :: my empathy program leaves many areas for
2449 improvement, among which are using vision to infer
2450 proprioception and looking up sensory experience with imagined
2451 vision, touch, and sound.
2452 - Evolution :: Karl Sims created a rich environment for simulating
2453 the evolution of creatures on a Connection Machine
2454 (\cite{sims-evolving-creatures}). Today, this can be redone
2455 and expanded with =CORTEX= on an ordinary computer.
2456 - Exotic senses :: Cortex enables many fascinating senses that are
2457 not possible to build in the real world. For example,
2458 telekinesis is an interesting avenue to explore. You can also
2459 make a ``semantic'' sense which looks up metadata tags on
2460 objects in the environment the metadata tags might contain
2461 other sensory information.
2462 - Imagination via subworlds :: this would involve a creature with
2463 an effector which creates an entire new sub-simulation where
2464 the creature has direct control over placement/creation of
2465 objects via simulated telekinesis. The creature observes this
2466 sub-world through its normal senses and uses its observations
2467 to make predictions about its top level world.
2468 - Simulated prescience :: step the simulation forward a few ticks,
2469 gather sensory data, then supply this data for the creature as
2470 one of its actual senses. The cost of prescience is slowing
2471 the simulation down by a factor proportional to however far
2472 you want the entities to see into the future. What happens
2473 when two evolved creatures that can each see into the future
2474 fight each other?
2475 - Swarm creatures :: Program a group of creatures that cooperate
2476 with each other. Because the creatures would be simulated, you
2477 could investigate computationally complex rules of behavior
2478 which still, from the group's point of view, would happen in
2479 real time. Interactions could be as simple as cellular
2480 organisms communicating via flashing lights, or as complex as
2481 humanoids completing social tasks, etc.
2482 - =HACKER= for writing muscle-control programs :: Presented with a
2483 low-level muscle control / sense API, generate higher level
2484 programs for accomplishing various stated goals. Example goals
2485 might be "extend all your fingers" or "move your hand into the
2486 area with blue light" or "decrease the angle of this joint".
2487 It would be like Sussman's HACKER, except it would operate
2488 with much more data in a more realistic world. Start off with
2489 "calisthenics" to develop subroutines over the motor control
2490 API. The low level programming code might be a turning machine
2491 that could develop programs to iterate over a "tape" where
2492 each entry in the tape could control recruitment of the fibers
2493 in a muscle.
2494 - Sense fusion :: There is much work to be done on sense
2495 integration -- building up a coherent picture of the world and
2496 the things in it. With =CORTEX= as a base, you can explore
2497 concepts like self-organizing maps or cross modal clustering
2498 in ways that have never before been tried.
2499 - Inverse kinematics :: experiments in sense guided motor control
2500 are easy given =CORTEX='s support -- you can get right to the
2501 hard control problems without worrying about physics or
2502 senses.
2504 \newpage
2506 * =EMPATH=: action recognition in a simulated worm
2508 Here I develop a computational model of empathy, using =CORTEX= as a
2509 base. Empathy in this context is the ability to observe another
2510 creature and infer what sorts of sensations that creature is
2511 feeling. My empathy algorithm involves multiple phases. First is
2512 free-play, where the creature moves around and gains sensory
2513 experience. From this experience I construct a representation of the
2514 creature's sensory state space, which I call \Phi-space. Using
2515 \Phi-space, I construct an efficient function which takes the
2516 limited data that comes from observing another creature and enriches
2517 it with a full compliment of imagined sensory data. I can then use
2518 the imagined sensory data to recognize what the observed creature is
2519 doing and feeling, using straightforward embodied action predicates.
2520 This is all demonstrated with using a simple worm-like creature, and
2521 recognizing worm-actions based on limited data.
2523 #+caption: Here is the worm with which we will be working. It is
2524 #+caption: composed of 5 segments. Each segment has a pair of
2525 #+caption: extensor and flexor muscles. Each of the worm's four
2526 #+caption: joints is a hinge joint which allows about 30 degrees of
2527 #+caption: rotation to either side. Each segment of the worm is
2528 #+caption: touch-capable and has a uniform distribution of touch
2529 #+caption: sensors on each of its faces. Each joint has a
2530 #+caption: proprioceptive sense to detect relative positions. The
2531 #+caption: worm segments are all the same except for the first one,
2532 #+caption: which has a much higher weight than the others to allow
2533 #+caption: for easy manual motor control.
2534 #+name: basic-worm-view
2535 #+ATTR_LaTeX: :width 10cm
2536 [[./images/basic-worm-view.png]]
2538 #+caption: Program for reading a worm from a Blender file and
2539 #+caption: outfitting it with the senses of proprioception,
2540 #+caption: touch, and the ability to move, as specified in the
2541 #+caption: Blender file.
2542 #+name: get-worm
2543 #+begin_listing clojure
2544 #+begin_src clojure
2545 (defn worm []
2546 (let [model (load-blender-model "Models/worm/worm.blend")]
2547 {:body (doto model (body!))
2548 :touch (touch! model)
2549 :proprioception (proprioception! model)
2550 :muscles (movement! model)}))
2551 #+end_src
2552 #+end_listing
2554 ** Embodiment factors action recognition into manageable parts
2556 Using empathy, I divide the problem of action recognition into a
2557 recognition process expressed in the language of a full compliment
2558 of senses, and an imaginative process that generates full sensory
2559 data from partial sensory data. Splitting the action recognition
2560 problem in this manner greatly reduces the total amount of work to
2561 recognize actions: The imaginative process is mostly just matching
2562 previous experience, and the recognition process gets to use all
2563 the senses to directly describe any action.
2565 ** Action recognition is easy with a full gamut of senses
2567 Embodied representation using multiple senses such as touch,
2568 proprioception, and muscle tension turns out be exceedingly
2569 efficient at describing body-centered actions. It is the right
2570 language for the job. For example, it takes only around 5 lines of
2571 clojure code to describe the action of curling using embodied
2572 primitives. It takes about 10 lines to describe the seemingly
2573 complicated action of wiggling.
2575 The following action predicates each take a stream of sensory
2576 experience, observe however much of it they desire, and decide
2577 whether the worm is doing the action they describe. =curled?=
2578 relies on proprioception, =resting?= relies on touch, =wiggling?=
2579 relies on a Fourier analysis of muscle contraction, and
2580 =grand-circle?= relies on touch and reuses =curled?= in its
2581 definition, showing how embodied predicates can be composed.
2584 #+caption: Program for detecting whether the worm is curled. This is the
2585 #+caption: simplest action predicate, because it only uses the last frame
2586 #+caption: of sensory experience, and only uses proprioceptive data. Even
2587 #+caption: this simple predicate, however, is automatically frame
2588 #+caption: independent and ignores vermopomorphic\protect\footnotemark
2589 #+caption: \space differences such as worm textures and colors.
2590 #+name: curled
2591 #+begin_listing clojure
2592 #+begin_src clojure
2593 (defn curled?
2594 "Is the worm curled up?"
2595 [experiences]
2596 (every?
2597 (fn [[_ _ bend]]
2598 (> (Math/sin bend) 0.64))
2599 (:proprioception (peek experiences))))
2600 #+end_src
2601 #+end_listing
2603 #+BEGIN_LaTeX
2604 \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
2605 #+END_LaTeX
2607 #+caption: Program for summarizing the touch information in a patch
2608 #+caption: of skin.
2609 #+name: touch-summary
2610 #+begin_listing clojure
2611 #+begin_src clojure
2612 (defn contact
2613 "Determine how much contact a particular worm segment has with
2614 other objects. Returns a value between 0 and 1, where 1 is full
2615 contact and 0 is no contact."
2616 [touch-region [coords contact :as touch]]
2617 (-> (zipmap coords contact)
2618 (select-keys touch-region)
2619 (vals)
2620 (#(map first %))
2621 (average)
2622 (* 10)
2623 (- 1)
2624 (Math/abs)))
2625 #+end_src
2626 #+end_listing
2629 #+caption: Program for detecting whether the worm is at rest. This program
2630 #+caption: uses a summary of the tactile information from the underbelly
2631 #+caption: of the worm, and is only true if every segment is touching the
2632 #+caption: floor. Note that this function contains no references to
2633 #+caption: proprioception at all.
2634 #+name: resting
2635 #+begin_listing clojure
2636 #+begin_src clojure
2637 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2639 (defn resting?
2640 "Is the worm resting on the ground?"
2641 [experiences]
2642 (every?
2643 (fn [touch-data]
2644 (< 0.9 (contact worm-segment-bottom touch-data)))
2645 (:touch (peek experiences))))
2646 #+end_src
2647 #+end_listing
2649 #+caption: Program for detecting whether the worm is curled up into a
2650 #+caption: full circle. Here the embodied approach begins to shine, as
2651 #+caption: I am able to both use a previous action predicate (=curled?=)
2652 #+caption: as well as the direct tactile experience of the head and tail.
2653 #+name: grand-circle
2654 #+begin_listing clojure
2655 #+begin_src clojure
2656 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2658 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2660 (defn grand-circle?
2661 "Does the worm form a majestic circle (one end touching the other)?"
2662 [experiences]
2663 (and (curled? experiences)
2664 (let [worm-touch (:touch (peek experiences))
2665 tail-touch (worm-touch 0)
2666 head-touch (worm-touch 4)]
2667 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2668 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2669 #+end_src
2670 #+end_listing
2673 #+caption: Program for detecting whether the worm has been wiggling for
2674 #+caption: the last few frames. It uses a Fourier analysis of the muscle
2675 #+caption: contractions of the worm's tail to determine wiggling. This is
2676 #+caption: significant because there is no particular frame that clearly
2677 #+caption: indicates that the worm is wiggling --- only when multiple frames
2678 #+caption: are analyzed together is the wiggling revealed. Defining
2679 #+caption: wiggling this way also gives the worm an opportunity to learn
2680 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2681 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2682 #+caption: from actual wiggling, but this definition gives it to us for free.
2683 #+name: wiggling
2684 #+begin_listing clojure
2685 #+begin_src clojure
2686 (defn fft [nums]
2687 (map
2688 #(.getReal %)
2689 (.transform
2690 (FastFourierTransformer. DftNormalization/STANDARD)
2691 (double-array nums) TransformType/FORWARD)))
2693 (def indexed (partial map-indexed vector))
2695 (defn max-indexed [s]
2696 (first (sort-by (comp - second) (indexed s))))
2698 (defn wiggling?
2699 "Is the worm wiggling?"
2700 [experiences]
2701 (let [analysis-interval 0x40]
2702 (when (> (count experiences) analysis-interval)
2703 (let [a-flex 3
2704 a-ex 2
2705 muscle-activity
2706 (map :muscle (vector:last-n experiences analysis-interval))
2707 base-activity
2708 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2709 (= 2
2710 (first
2711 (max-indexed
2712 (map #(Math/abs %)
2713 (take 20 (fft base-activity))))))))))
2714 #+end_src
2715 #+end_listing
2717 With these action predicates, I can now recognize the actions of
2718 the worm while it is moving under my control and I have access to
2719 all the worm's senses.
2721 #+caption: Use the action predicates defined earlier to report on
2722 #+caption: what the worm is doing while in simulation.
2723 #+name: report-worm-activity
2724 #+begin_listing clojure
2725 #+begin_src clojure
2726 (defn debug-experience
2727 [experiences text]
2728 (cond
2729 (grand-circle? experiences) (.setText text "Grand Circle")
2730 (curled? experiences) (.setText text "Curled")
2731 (wiggling? experiences) (.setText text "Wiggling")
2732 (resting? experiences) (.setText text "Resting")))
2733 #+end_src
2734 #+end_listing
2736 #+caption: Using =debug-experience=, the body-centered predicates
2737 #+caption: work together to classify the behavior of the worm.
2738 #+caption: the predicates are operating with access to the worm's
2739 #+caption: full sensory data.
2740 #+name: basic-worm-view
2741 #+ATTR_LaTeX: :width 10cm
2742 [[./images/worm-identify-init.png]]
2744 These action predicates satisfy the recognition requirement of an
2745 empathic recognition system. There is power in the simplicity of
2746 the action predicates. They describe their actions without getting
2747 confused in visual details of the worm. Each one is independent of
2748 position and rotation, but more than that, they are each
2749 independent of irrelevant visual details of the worm and the
2750 environment. They will work regardless of whether the worm is a
2751 different color or heavily textured, or if the environment has
2752 strange lighting.
2754 Consider how the human act of jumping might be described with
2755 body-centered action predicates: You might specify that jumping is
2756 mainly the feeling of your knees bending, your thigh muscles
2757 contracting, and your inner ear experiencing a certain sort of back
2758 and forth acceleration. This representation is a very concrete
2759 description of jumping, couched in terms of muscles and senses, but
2760 it also has the ability to describe almost all kinds of jumping, a
2761 generality that you might think could only be achieved by a very
2762 abstract description. The body centered jumping predicate does not
2763 have terms that consider the color of a person's skin or whether
2764 they are male or female, instead it gets right to the meat of what
2765 jumping actually /is/.
2767 Of course, the action predicates are not directly applicable to
2768 video data, which lacks the advanced sensory information which they
2769 require!
2771 The trick now is to make the action predicates work even when the
2772 sensory data on which they depend is absent!
2774 ** \Phi-space describes the worm's experiences
2776 As a first step towards building empathy, I need to gather all of
2777 the worm's experiences during free play. I use a simple vector to
2778 store all the experiences.
2780 Each element of the experience vector exists in the vast space of
2781 all possible worm-experiences. Most of this vast space is actually
2782 unreachable due to physical constraints of the worm's body. For
2783 example, the worm's segments are connected by hinge joints that put
2784 a practical limit on the worm's range of motions without limiting
2785 its degrees of freedom. Some groupings of senses are impossible;
2786 the worm can not be bent into a circle so that its ends are
2787 touching and at the same time not also experience the sensation of
2788 touching itself.
2790 As the worm moves around during free play and its experience vector
2791 grows larger, the vector begins to define a subspace which is all
2792 the sensations the worm can practically experience during normal
2793 operation. I call this subspace \Phi-space, short for
2794 physical-space. The experience vector defines a path through
2795 \Phi-space. This path has interesting properties that all derive
2796 from physical embodiment. The proprioceptive components of the path
2797 vary smoothly, because in order for the worm to move from one
2798 position to another, it must pass through the intermediate
2799 positions. The path invariably forms loops as common actions are
2800 repeated. Finally and most importantly, proprioception alone
2801 actually gives very strong inference about the other senses. For
2802 example, when the worm is proprioceptively flat over several
2803 frames, you can infer that it is touching the ground and that its
2804 muscles are not active, because if the muscles were active, the
2805 worm would be moving and would not remain perfectly flat. In order
2806 to stay flat, the worm has to be touching the ground, or it would
2807 again be moving out of the flat position due to gravity. If the
2808 worm is positioned in such a way that it interacts with itself,
2809 then it is very likely to be feeling the same tactile feelings as
2810 the last time it was in that position, because it has the same body
2811 as then. As you observe multiple frames of proprioceptive data, you
2812 can become increasingly confident about the exact activations of
2813 the worm's muscles, because it generally takes a unique combination
2814 of muscle contractions to transform the worm's body along a
2815 specific path through \Phi-space.
2817 The worm's total life experience is a long looping path through
2818 \Phi-space. I will now introduce simple way of taking that
2819 experience path and building a function that can infer complete
2820 sensory experience given only a stream of proprioceptive data. This
2821 /empathy/ function will provide a bridge to use the body centered
2822 action predicates on video-like streams of information.
2824 ** Empathy is the process of building paths in \Phi-space
2826 Here is the core of a basic empathy algorithm, starting with an
2827 experience vector:
2829 An /experience-index/ is an index into the grand experience vector
2830 that defines the worm's life. It is a time-stamp for each set of
2831 sensations the worm has experienced.
2833 First, I group the experience-indices into bins according to the
2834 similarity of their proprioceptive data. I organize my bins into a
2835 3 level hierarchy. The smallest bins have an approximate size of
2836 0.001 radians in all proprioceptive dimensions. Each higher level
2837 is 10x bigger than the level below it.
2839 The bins serve as a hashing function for proprioceptive data. Given
2840 a single piece of proprioceptive experience, the bins allow me to
2841 rapidly find all other similar experience-indices of past
2842 experience that had a very similar proprioceptive configuration.
2843 When looking up a proprioceptive experience, if the smallest bin
2844 does not match any previous experience, then I use successively
2845 larger bins until a match is found or I reach the largest bin.
2847 Given a sequence of proprioceptive input, I use the bins to
2848 generate a set of similar experiences for each input using the
2849 tiered proprioceptive bins.
2851 Finally, to infer sensory data, I select the longest consecutive
2852 chain of experiences that threads through the sets of similar
2853 experiences, starting with the current moment as a root and going
2854 backwards. Consecutive experience means that the experiences appear
2855 next to each other in the experience vector.
2857 A stream of proprioceptive input might be:
2859 #+BEGIN_EXAMPLE
2860 [ flat, flat, flat, flat, flat, flat, lift-head ]
2861 #+END_EXAMPLE
2863 The worm's previous experience of lying on the ground and lifting
2864 its head generates possible interpretations for each frame (the
2865 numbers are experience-indices):
2867 #+BEGIN_EXAMPLE
2868 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2869 1 1 1 1 1 1 1 4
2870 2 2 2 2 2 2 2
2871 3 3 3 3 3 3 3
2872 6 6 6 6 6 6 6
2873 7 7 7 7 7 7 7
2874 8 8 8 8 8 8 8
2875 9 9 9 9 9 9 9
2876 #+END_EXAMPLE
2878 These interpretations suggest a new path through phi space:
2880 #+BEGIN_EXAMPLE
2881 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2882 6 7 8 9 1 2 3 4
2883 #+END_EXAMPLE
2885 The new path through \Phi-space is synthesized from two actual
2886 paths that the creature has experienced: the "1-2-3-4" chain and
2887 the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
2888 ends with the worm lifting its head. It originated from a short
2889 training session where the worm rested on the floor for a brief
2890 while and then raised its head. The "6-7-8-9" chain is part of a
2891 longer chain of inactivity where the worm simply rested on the
2892 floor without moving. It is preferred over a "1-2-3" chain (which
2893 also describes inactivity) because it is longer. The main ideas
2894 again:
2896 - Imagined \Phi-space paths are synthesized by looping and mixing
2897 previous experiences.
2899 - Longer experience paths (less edits) are preferred.
2901 - The present is more important than the past --- more recent
2902 events take precedence in interpretation.
2904 This algorithm has three advantages:
2906 1. It's simple
2908 3. It's very fast -- retrieving possible interpretations takes
2909 constant time. Tracing through chains of interpretations takes
2910 time proportional to the average number of experiences in a
2911 proprioceptive bin. Redundant experiences in \Phi-space can be
2912 merged to save computation.
2914 2. It protects from wrong interpretations of transient ambiguous
2915 proprioceptive data. For example, if the worm is flat for just
2916 an instant, this flatness will not be interpreted as implying
2917 that the worm has its muscles relaxed, since the flatness is
2918 part of a longer chain which includes a distinct pattern of
2919 muscle activation. Markov chains or other memoryless statistical
2920 models that operate on individual frames may very well make this
2921 mistake.
2923 #+caption: Program to convert an experience vector into a
2924 #+caption: proprioceptively binned lookup function.
2925 #+name: bin
2926 #+begin_listing clojure
2927 #+begin_src clojure
2928 (defn bin [digits]
2929 (fn [angles]
2930 (->> angles
2931 (flatten)
2932 (map (juxt #(Math/sin %) #(Math/cos %)))
2933 (flatten)
2934 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2936 (defn gen-phi-scan
2937 "Nearest-neighbors with binning. Only returns a result if
2938 the proprioceptive data is within 10% of a previously recorded
2939 result in all dimensions."
2940 [phi-space]
2941 (let [bin-keys (map bin [3 2 1])
2942 bin-maps
2943 (map (fn [bin-key]
2944 (group-by
2945 (comp bin-key :proprioception phi-space)
2946 (range (count phi-space)))) bin-keys)
2947 lookups (map (fn [bin-key bin-map]
2948 (fn [proprio] (bin-map (bin-key proprio))))
2949 bin-keys bin-maps)]
2950 (fn lookup [proprio-data]
2951 (set (some #(% proprio-data) lookups)))))
2952 #+end_src
2953 #+end_listing
2955 #+caption: =longest-thread= finds the longest path of consecutive
2956 #+caption: past experiences to explain proprioceptive worm data from
2957 #+caption: previous data. Here, the film strip represents the
2958 #+caption: creature's previous experience. Sort sequences of
2959 #+caption: memories are spliced together to match the
2960 #+caption: proprioceptive data. Their carry the other senses
2961 #+caption: along with them.
2962 #+name: phi-space-history-scan
2963 #+ATTR_LaTeX: :width 10cm
2964 [[./images/film-of-imagination.png]]
2966 =longest-thread= infers sensory data by stitching together pieces
2967 from previous experience. It prefers longer chains of previous
2968 experience to shorter ones. For example, during training the worm
2969 might rest on the ground for one second before it performs its
2970 exercises. If during recognition the worm rests on the ground for
2971 five seconds, =longest-thread= will accommodate this five second
2972 rest period by looping the one second rest chain five times.
2974 =longest-thread= takes time proportional to the average number of
2975 entries in a proprioceptive bin, because for each element in the
2976 starting bin it performs a series of set lookups in the preceding
2977 bins. If the total history is limited, then this takes time
2978 proportional to a only a constant multiple of the number of entries
2979 in the starting bin. This analysis also applies, even if the action
2980 requires multiple longest chains -- it's still the average number
2981 of entries in a proprioceptive bin times the desired chain length.
2982 Because =longest-thread= is so efficient and simple, I can
2983 interpret worm-actions in real time.
2985 #+caption: Program to calculate empathy by tracing though \Phi-space
2986 #+caption: and finding the longest (ie. most coherent) interpretation
2987 #+caption: of the data.
2988 #+name: longest-thread
2989 #+begin_listing clojure
2990 #+begin_src clojure
2991 (defn longest-thread
2992 "Find the longest thread from phi-index-sets. The index sets should
2993 be ordered from most recent to least recent."
2994 [phi-index-sets]
2995 (loop [result '()
2996 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2997 (if (empty? phi-index-sets)
2998 (vec result)
2999 (let [threads
3000 (for [thread-base thread-bases]
3001 (loop [thread (list thread-base)
3002 remaining remaining]
3003 (let [next-index (dec (first thread))]
3004 (cond (empty? remaining) thread
3005 (contains? (first remaining) next-index)
3006 (recur
3007 (cons next-index thread) (rest remaining))
3008 :else thread))))
3009 longest-thread
3010 (reduce (fn [thread-a thread-b]
3011 (if (> (count thread-a) (count thread-b))
3012 thread-a thread-b))
3013 '(nil)
3014 threads)]
3015 (recur (concat longest-thread result)
3016 (drop (count longest-thread) phi-index-sets))))))
3017 #+end_src
3018 #+end_listing
3020 There is one final piece, which is to replace missing sensory data
3021 with a best-guess estimate. While I could fill in missing data by
3022 using a gradient over the closest known sensory data points,
3023 averages can be misleading. It is certainly possible to create an
3024 impossible sensory state by averaging two possible sensory states.
3025 For example, consider moving your hand in an arc over your head. If
3026 for some reason you only have the initial and final positions of
3027 this movement in your \Phi-space, averaging them together will
3028 produce the proprioceptive sensation of having your hand /inside/
3029 your head, which is physically impossible to ever experience
3030 (barring motor adaption illusions). Therefore I simply replicate
3031 the most recent sensory experience to fill in the gaps.
3033 #+caption: Fill in blanks in sensory experience by replicating the most
3034 #+caption: recent experience.
3035 #+name: infer-nils
3036 #+begin_listing clojure
3037 #+begin_src clojure
3038 (defn infer-nils
3039 "Replace nils with the next available non-nil element in the
3040 sequence, or barring that, 0."
3041 [s]
3042 (loop [i (dec (count s))
3043 v (transient s)]
3044 (if (zero? i) (persistent! v)
3045 (if-let [cur (v i)]
3046 (if (get v (dec i) 0)
3047 (recur (dec i) v)
3048 (recur (dec i) (assoc! v (dec i) cur)))
3049 (recur i (assoc! v i 0))))))
3050 #+end_src
3051 #+end_listing
3053 ** =EMPATH= recognizes actions efficiently
3055 To use =EMPATH= with the worm, I first need to gather a set of
3056 experiences from the worm that includes the actions I want to
3057 recognize. The =generate-phi-space= program (listing
3058 \ref{generate-phi-space} runs the worm through a series of
3059 exercises and gathers those experiences into a vector. The
3060 =do-all-the-things= program is a routine expressed in a simple
3061 muscle contraction script language for automated worm control. It
3062 causes the worm to rest, curl, and wiggle over about 700 frames
3063 (approx. 11 seconds).
3065 #+caption: Program to gather the worm's experiences into a vector for
3066 #+caption: further processing. The =motor-control-program= line uses
3067 #+caption: a motor control script that causes the worm to execute a series
3068 #+caption: of ``exercises'' that include all the action predicates.
3069 #+name: generate-phi-space
3070 #+begin_listing clojure
3071 #+begin_src clojure
3072 (def do-all-the-things
3073 (concat
3074 curl-script
3075 [[300 :d-ex 40]
3076 [320 :d-ex 0]]
3077 (shift-script 280 (take 16 wiggle-script))))
3079 (defn generate-phi-space []
3080 (let [experiences (atom [])]
3081 (run-world
3082 (apply-map
3083 worm-world
3084 (merge
3085 (worm-world-defaults)
3086 {:end-frame 700
3087 :motor-control
3088 (motor-control-program worm-muscle-labels do-all-the-things)
3089 :experiences experiences})))
3090 @experiences))
3091 #+end_src
3092 #+end_listing
3094 #+caption: Use =longest-thread= and a \Phi-space generated from a short
3095 #+caption: exercise routine to interpret actions during free play.
3096 #+name: empathy-debug
3097 #+begin_listing clojure
3098 #+begin_src clojure
3099 (defn init []
3100 (def phi-space (generate-phi-space))
3101 (def phi-scan (gen-phi-scan phi-space)))
3103 (defn empathy-demonstration []
3104 (let [proprio (atom ())]
3105 (fn
3106 [experiences text]
3107 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3108 (swap! proprio (partial cons phi-indices))
3109 (let [exp-thread (longest-thread (take 300 @proprio))
3110 empathy (mapv phi-space (infer-nils exp-thread))]
3111 (println-repl (vector:last-n exp-thread 22))
3112 (cond
3113 (grand-circle? empathy) (.setText text "Grand Circle")
3114 (curled? empathy) (.setText text "Curled")
3115 (wiggling? empathy) (.setText text "Wiggling")
3116 (resting? empathy) (.setText text "Resting")
3117 :else (.setText text "Unknown")))))))
3119 (defn empathy-experiment [record]
3120 (.start (worm-world :experience-watch (debug-experience-phi)
3121 :record record :worm worm*)))
3122 #+end_src
3123 #+end_listing
3125 These programs create a test for the empathy system. First, the
3126 worm's \Phi-space is generated from a simple motor script. Then the
3127 worm is re-created in an environment almost exactly identical to
3128 the testing environment for the action-predicates, with one major
3129 difference : the only sensory information available to the system
3130 is proprioception. From just the proprioception data and
3131 \Phi-space, =longest-thread= synthesizes a complete record the last
3132 300 sensory experiences of the worm. These synthesized experiences
3133 are fed directly into the action predicates =grand-circle?=,
3134 =curled?=, =wiggling?=, and =resting?= and their outputs are
3135 printed to the screen at each frame.
3137 The result of running =empathy-experiment= is that the system is
3138 generally able to interpret worm actions using the action-predicates
3139 on simulated sensory data just as well as with actual data. Figure
3140 \ref{empathy-debug-image} was generated using =empathy-experiment=:
3142 #+caption: From only proprioceptive data, =EMPATH= was able to infer
3143 #+caption: the complete sensory experience and classify four poses
3144 #+caption: (The last panel shows a composite image of /wiggling/,
3145 #+caption: a dynamic pose.)
3146 #+name: empathy-debug-image
3147 #+ATTR_LaTeX: :width 10cm :placement [H]
3148 [[./images/empathy-1.png]]
3150 One way to measure the performance of =EMPATH= is to compare the
3151 suitability of the imagined sense experience to trigger the same
3152 action predicates as the real sensory experience.
3154 #+caption: Determine how closely empathy approximates actual
3155 #+caption: sensory data.
3156 #+name: test-empathy-accuracy
3157 #+begin_listing clojure
3158 #+begin_src clojure
3159 (def worm-action-label
3160 (juxt grand-circle? curled? wiggling?))
3162 (defn compare-empathy-with-baseline [matches]
3163 (let [proprio (atom ())]
3164 (fn
3165 [experiences text]
3166 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3167 (swap! proprio (partial cons phi-indices))
3168 (let [exp-thread (longest-thread (take 300 @proprio))
3169 empathy (mapv phi-space (infer-nils exp-thread))
3170 experience-matches-empathy
3171 (= (worm-action-label experiences)
3172 (worm-action-label empathy))]
3173 (println-repl experience-matches-empathy)
3174 (swap! matches #(conj % experience-matches-empathy)))))))
3176 (defn accuracy [v]
3177 (float (/ (count (filter true? v)) (count v))))
3179 (defn test-empathy-accuracy []
3180 (let [res (atom [])]
3181 (run-world
3182 (worm-world :experience-watch
3183 (compare-empathy-with-baseline res)
3184 :worm worm*))
3185 (accuracy @res)))
3186 #+end_src
3187 #+end_listing
3189 Running =test-empathy-accuracy= using the very short exercise
3190 program =do-all-the-things= defined in listing
3191 \ref{generate-phi-space}, and then doing a similar pattern of
3192 activity using manual control of the worm, yields an accuracy of
3193 around 73%. This is based on very limited worm experience, and
3194 almost all errors are due to the worm's \Phi-space being too
3195 incomplete to properly interpret common poses. By manually training
3196 the worm for longer using =init-interactive= defined in listing
3197 \ref{manual-phi-space}, the accuracy dramatically improves:
3199 #+caption: Program to generate \Phi-space using manual training.
3200 #+name: manual-phi-space
3201 #+begin_listing clojure
3202 #+begin_src clojure
3203 (defn init-interactive []
3204 (def phi-space
3205 (let [experiences (atom [])]
3206 (run-world
3207 (apply-map
3208 worm-world
3209 (merge
3210 (worm-world-defaults)
3211 {:experiences experiences})))
3212 @experiences))
3213 (def phi-scan (gen-phi-scan phi-space)))
3214 #+end_src
3215 #+end_listing
3217 =init-interactive= allows me to take direct control of the worm's
3218 muscles and run it through each characteristic movement I care
3219 about. After about 1 minute of manual training, I was able to
3220 achieve 95% accuracy on manual testing of the worm using
3221 =test-empathy-accuracy=. The majority of disagreements are near the
3222 transition boundaries from one type of action to another. During
3223 these transitions the exact label for the action is often unclear,
3224 and disagreement between empathy and experience is practically
3225 irrelevant. Thus, the system's effective identification accuracy is
3226 even higher than 95%. When I watch this system myself, I generally
3227 see no errors in action identification compared to my own judgment
3228 of what the worm is doing.
3230 ** Digression: Learning touch sensor layout through free play
3232 In the previous chapter I showed how to compute actions in terms of
3233 body-centered predicates, but some of those predicates relied on
3234 the average touch activation of pre-defined regions of the worm's
3235 skin. What if, instead of receiving touch pre-grouped into the six
3236 faces of each worm segment, the true partitioning of the worm's
3237 skin was unknown? This is more similar to how a nerve fiber bundle
3238 might be arranged inside an animal. While two fibers that are close
3239 in a nerve bundle /might/ correspond to two touch sensors that are
3240 close together on the skin, the process of taking a complicated
3241 surface and forcing it into essentially a 2D circle requires that
3242 some regions of skin that are close together in the animal end up
3243 far apart in the nerve bundle.
3245 In this chapter I show how to automatically learn the
3246 skin-partitioning of a worm segment by free exploration. As the
3247 worm rolls around on the floor, large sections of its surface get
3248 activated. If the worm has stopped moving, then whatever region of
3249 skin that is touching the floor is probably an important region,
3250 and should be recorded. The code I provide relies on the worm
3251 segment having flat faces, but still demonstrates a primitive kind
3252 of multi-sensory bootstrapping that I find appealing.
3254 #+caption: Program to detect whether the worm is in a resting state
3255 #+caption: with one face touching the floor.
3256 #+name: pure-touch
3257 #+begin_listing clojure
3258 #+begin_src clojure
3259 (def full-contact [(float 0.0) (float 0.1)])
3261 (defn pure-touch?
3262 "This is worm specific code to determine if a large region of touch
3263 sensors is either all on or all off."
3264 [[coords touch :as touch-data]]
3265 (= (set (map first touch)) (set full-contact)))
3266 #+end_src
3267 #+end_listing
3269 After collecting these important regions, there will many nearly
3270 similar touch regions. While for some purposes the subtle
3271 differences between these regions will be important, for my
3272 purposes I collapse them into mostly non-overlapping sets using
3273 =remove-similar= in listing \ref{remove-similar}
3275 #+caption: Program to take a list of sets of points and ``collapse them''
3276 #+caption: so that the remaining sets in the list are significantly
3277 #+caption: different from each other. Prefer smaller sets to larger ones.
3278 #+name: remove-similar
3279 #+begin_listing clojure
3280 #+begin_src clojure
3281 (defn remove-similar
3282 [coll]
3283 (loop [result () coll (sort-by (comp - count) coll)]
3284 (if (empty? coll) result
3285 (let [[x & xs] coll
3286 c (count x)]
3287 (if (some
3288 (fn [other-set]
3289 (let [oc (count other-set)]
3290 (< (- (count (union other-set x)) c) (* oc 0.1))))
3291 xs)
3292 (recur result xs)
3293 (recur (cons x result) xs))))))
3294 #+end_src
3295 #+end_listing
3297 Actually running this simulation is easy given =CORTEX='s facilities.
3299 #+caption: Collect experiences while the worm moves around. Filter the touch
3300 #+caption: sensations by stable ones, collapse similar ones together,
3301 #+caption: and report the regions learned.
3302 #+name: learn-touch
3303 #+begin_listing clojure
3304 #+begin_src clojure
3305 (defn learn-touch-regions []
3306 (let [experiences (atom [])
3307 world (apply-map
3308 worm-world
3309 (assoc (worm-segment-defaults)
3310 :experiences experiences))]
3311 (run-world world)
3312 (->>
3313 @experiences
3314 (drop 175)
3315 ;; access the single segment's touch data
3316 (map (comp first :touch))
3317 ;; only deal with "pure" touch data to determine surfaces
3318 (filter pure-touch?)
3319 ;; associate coordinates with touch values
3320 (map (partial apply zipmap))
3321 ;; select those regions where contact is being made
3322 (map (partial group-by second))
3323 (map #(get % full-contact))
3324 (map (partial map first))
3325 ;; remove redundant/subset regions
3326 (map set)
3327 remove-similar)))
3329 (defn learn-and-view-touch-regions []
3330 (map view-touch-region
3331 (learn-touch-regions)))
3332 #+end_src
3333 #+end_listing
3335 The only thing remaining to define is the particular motion the worm
3336 must take. I accomplish this with a simple motor control program.
3338 #+caption: Motor control program for making the worm roll on the ground.
3339 #+caption: This could also be replaced with random motion.
3340 #+name: worm-roll
3341 #+begin_listing clojure
3342 #+begin_src clojure
3343 (defn touch-kinesthetics []
3344 [[170 :lift-1 40]
3345 [190 :lift-1 19]
3346 [206 :lift-1 0]
3348 [400 :lift-2 40]
3349 [410 :lift-2 0]
3351 [570 :lift-2 40]
3352 [590 :lift-2 21]
3353 [606 :lift-2 0]
3355 [800 :lift-1 30]
3356 [809 :lift-1 0]
3358 [900 :roll-2 40]
3359 [905 :roll-2 20]
3360 [910 :roll-2 0]
3362 [1000 :roll-2 40]
3363 [1005 :roll-2 20]
3364 [1010 :roll-2 0]
3366 [1100 :roll-2 40]
3367 [1105 :roll-2 20]
3368 [1110 :roll-2 0]
3369 ])
3370 #+end_src
3371 #+end_listing
3374 #+caption: The small worm rolls around on the floor, driven
3375 #+caption: by the motor control program in listing \ref{worm-roll}.
3376 #+name: worm-roll
3377 #+ATTR_LaTeX: :width 12cm
3378 [[./images/worm-roll.png]]
3380 #+caption: After completing its adventures, the worm now knows
3381 #+caption: how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were
3382 #+caption: deemed important by
3383 #+caption: =learn-touch-regions=. Each white square in the rectangles
3384 #+caption: above is a cluster of ``related" touch nodes as determined
3385 #+caption: by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces.
3386 #+name: worm-touch-map
3387 #+ATTR_LaTeX: :width 12cm
3388 [[./images/touch-learn.png]]
3390 While simple, =learn-touch-regions= exploits regularities in both
3391 the worm's physiology and the worm's environment to correctly
3392 deduce that the worm has six sides. Note that =learn-touch-regions=
3393 would work just as well even if the worm's touch sense data were
3394 completely scrambled. The cross shape is just for convenience. This
3395 example justifies the use of pre-defined touch regions in =EMPATH=.
3397 ** Recognizing an object using embodied representation
3399 At the beginning of the thesis, I suggested that we might recognize
3400 the chair in Figure \ref{hidden-chair} by imagining ourselves in
3401 the position of the man and realizing that he must be sitting on
3402 something in order to maintain that position. Here, I present a
3403 brief elaboration on how to this might be done.
3405 First, I need the feeling of leaning or resting /on/ some other
3406 object that is not the floor. This feeling is easy to describe
3407 using an embodied representation.
3409 #+caption: Program describing the sense of leaning or resting on something.
3410 #+caption: This involves a relaxed posture, the feeling of touching something,
3411 #+caption: and a period of stability where the worm does not move.
3412 #+name: draped
3413 #+begin_listing clojure
3414 #+begin_src clojure
3415 (defn draped?
3416 "Is the worm:
3417 -- not flat (the floor is not a 'chair')
3418 -- supported (not using its muscles to hold its position)
3419 -- stable (not changing its position)
3420 -- touching something (must register contact)"
3421 [experiences]
3422 (let [b2-hash (bin 2)
3423 touch (:touch (peek experiences))
3424 total-contact
3425 (reduce
3427 (map #(contact all-touch-coordinates %)
3428 (rest touch)))]
3429 (println total-contact)
3430 (and (not (resting? experiences))
3431 (every?
3432 zero?
3433 (-> experiences
3434 (vector:last-n 25)
3435 (#(map :muscle %))
3436 (flatten)))
3437 (-> experiences
3438 (vector:last-n 20)
3439 (#(map (comp b2-hash flatten :proprioception) %))
3440 (set)
3441 (count) (= 1))
3442 (< 0.03 total-contact))))
3443 #+end_src
3444 #+end_listing
3446 #+caption: The =draped?= predicate detects the presence of the cube
3447 #+caption: whenever the worm interacts with it. The details of the
3448 #+caption: cube are irrelevant; only the way it influences the
3449 #+caption: worm's body matters. The ``unknown'' label on the fifth
3450 #+caption: frame is due to the fact that the worm is not
3451 #+caption: stationary. =draped?= will only declare that the worm
3452 #+caption: is draped if it has been still for a while.
3453 #+name: draped-video
3454 #+ATTR_LaTeX: :width 13cm
3455 [[./images/draped.png]]
3457 Though this is a simple example, using the =draped?= predicate to
3458 detect a cube has interesting advantages. The =draped?= predicate
3459 describes the cube not in terms of properties that the cube has,
3460 but instead in terms of how the worm interacts with it physically.
3461 This means that the cube can still be detected even if it is not
3462 visible, as long as its influence on the worm's body is visible.
3464 This system will also see the virtual cube created by a
3465 ``mimeworm", which uses its muscles in a very controlled way to
3466 mimic the appearance of leaning on a cube. The system will
3467 anticipate that there is an actual invisible cube that provides
3468 support!
3470 #+caption: Can you see the thing that this person is leaning on?
3471 #+caption: What properties does it have, other than how it makes
3472 #+caption: the man's elbow and shoulder feel? I wonder if people
3473 #+caption: who can actually maintain this pose easily still see the
3474 #+caption: support?
3475 #+name: mime
3476 #+ATTR_LaTeX: :width 6cm
3477 [[./images/pablo-the-mime.png]]
3479 This makes me wonder about the psychology of actual mimes. Suppose
3480 for a moment that people have something analogous to \Phi-space and
3481 that one of the ways that they find objects in a scene is by their
3482 relation to other people's bodies. Suppose that a person watches a
3483 person miming an invisible wall. For a person with no experience
3484 with miming, their \Phi-space will only have entries that describe
3485 the scene with the sensation of their hands touching a wall. This
3486 sensation of touch will create a strong impression of a wall, even
3487 though the wall would have to be invisible. A person with
3488 experience in miming however, will have entries in their \Phi-space
3489 that describe the wall-miming position without a sense of touch. It
3490 will not seem to such as person that an invisible wall is present,
3491 but merely that the mime is holding out their hands in a special
3492 way. Thus, the theory that humans use something like \Phi-space
3493 weakly predicts that learning how to mime should break the power of
3494 miming illusions. Most optical illusions still work no matter how
3495 much you know about them, so this proposal would be quite
3496 interesting to test, as it predicts a non-standard result!
3499 #+BEGIN_LaTeX
3500 \clearpage
3501 #+END_LaTeX
3503 * Contributions
3505 The big idea behind this thesis is a new way to represent and
3506 recognize physical actions, which I call /empathic representation/.
3507 Actions are represented as predicates which have access to the
3508 totality of a creature's sensory abilities. To recognize the
3509 physical actions of another creature similar to yourself, you
3510 imagine what they would feel by examining the position of their body
3511 and relating it to your own previous experience.
3513 Empathic representation of physical actions is robust and general.
3514 Because the representation is body-centered, it avoids baking in a
3515 particular viewpoint like you might get from learning from example
3516 videos. Because empathic representation relies on all of a
3517 creature's senses, it can describe exactly what an action /feels
3518 like/ without getting caught up in irrelevant details such as visual
3519 appearance. I think it is important that a correct description of
3520 jumping (for example) should not include irrelevant details such as
3521 the color of a person's clothes or skin; empathic representation can
3522 get right to the heart of what jumping is by describing it in terms
3523 of touch, muscle contractions, and a brief feeling of
3524 weightlessness. Empathic representation is very low-level in that it
3525 describes actions using concrete sensory data with little
3526 abstraction, but it has the generality of much more abstract
3527 representations!
3529 Another important contribution of this thesis is the development of
3530 the =CORTEX= system, a complete environment for creating simulated
3531 creatures. You have seen how to implement five senses: touch,
3532 proprioception, hearing, vision, and muscle tension. You have seen
3533 how to create new creatures using Blender, a 3D modeling tool.
3535 As a minor digression, you also saw how I used =CORTEX= to enable a
3536 tiny worm to discover the topology of its skin simply by rolling on
3537 the ground. You also saw how to detect objects using only embodied
3538 predicates.
3540 In conclusion, for this thesis I:
3542 - Developed the idea of embodied representation, which describes
3543 actions that a creature can do in terms of first-person sensory
3544 data.
3546 - Developed a method of empathic action recognition which uses
3547 previous embodied experience and embodied representation of
3548 actions to greatly constrain the possible interpretations of an
3549 action.
3551 - Created =EMPATH=, a program which uses empathic action
3552 recognition to recognize physical actions in a simple model
3553 involving segmented worm-like creatures.
3555 - Created =CORTEX=, a comprehensive platform for embodied AI
3556 experiments. It is the base on which =EMPATH= is built.
3558 #+BEGIN_LaTeX
3559 \clearpage
3560 \appendix
3561 #+END_LaTeX
3563 * Appendix: =CORTEX= User Guide
3565 Those who write a thesis should endeavor to make their code not only
3566 accessible, but actually usable, as a way to pay back the community
3567 that made the thesis possible in the first place. This thesis would
3568 not be possible without Free Software such as jMonkeyEngine3,
3569 Blender, clojure, =emacs=, =ffmpeg=, and many other tools. That is
3570 why I have included this user guide, in the hope that someone else
3571 might find =CORTEX= useful.
3573 ** Obtaining =CORTEX=
3575 You can get cortex from its mercurial repository at
3576 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3577 releases at http://aurellem.org/cortex/releases/. As a condition of
3578 making this thesis, I have also provided Professor Winston the
3579 =CORTEX= source, and he knows how to run the demos and get started.
3580 You may also email me at =cortex@aurellem.org= and I may help where
3581 I can.
3583 ** Running =CORTEX=
3585 =CORTEX= comes with README and INSTALL files that will guide you
3586 through installation and running the test suite. In particular you
3587 should look at test =cortex.test= which contains test suites that
3588 run through all senses and multiple creatures.
3590 ** Creating creatures
3592 Creatures are created using /Blender/, a free 3D modeling program.
3593 You will need Blender version 2.6 when using the =CORTEX= included
3594 in this thesis. You create a =CORTEX= creature in a similar manner
3595 to modeling anything in Blender, except that you also create
3596 several trees of empty nodes which define the creature's senses.
3598 *** Mass
3600 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3601 to the object with the mass in jMonkeyEngine units. Note that
3602 setting the mass to 0 causes the object to be immovable.
3604 *** Joints
3606 Joints are created by creating an empty node named =joints= and
3607 then creating any number of empty child nodes to represent your
3608 creature's joints. The joint will automatically connect the
3609 closest two physical objects. It will help to set the empty node's
3610 display mode to ``Arrows'' so that you can clearly see the
3611 direction of the axes.
3613 Joint nodes should have the following metadata under the ``joint''
3614 label:
3616 #+BEGIN_SRC clojure
3617 ;; ONE of the following, under the label "joint":
3618 {:type :point}
3620 ;; OR
3622 {:type :hinge
3623 :limit [<limit-low> <limit-high>]
3624 :axis (Vector3f. <x> <y> <z>)}
3625 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3627 ;; OR
3629 {:type :cone
3630 :limit-xz <lim-xz>
3631 :limit-xy <lim-xy>
3632 :twist <lim-twist>} ;(use XZY rotation mode in Blender!)
3633 #+END_SRC
3635 *** Eyes
3637 Eyes are created by creating an empty node named =eyes= and then
3638 creating any number of empty child nodes to represent your
3639 creature's eyes.
3641 Eye nodes should have the following metadata under the ``eye''
3642 label:
3644 #+BEGIN_SRC clojure
3645 {:red <red-retina-definition>
3646 :blue <blue-retina-definition>
3647 :green <green-retina-definition>
3648 :all <all-retina-definition>
3649 (<0xrrggbb> <custom-retina-image>)...
3651 #+END_SRC
3653 Any of the color channels may be omitted. You may also include
3654 your own color selectors, and in fact :red is equivalent to
3655 0xFF0000 and so forth. The eye will be placed at the same position
3656 as the empty node and will bind to the neatest physical object.
3657 The eye will point outward from the X-axis of the node, and ``up''
3658 will be in the direction of the X-axis of the node. It will help
3659 to set the empty node's display mode to ``Arrows'' so that you can
3660 clearly see the direction of the axes.
3662 Each retina file should contain white pixels wherever you want to be
3663 sensitive to your chosen color. If you want the entire field of
3664 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3665 white.
3667 Here is a sample retinal map:
3669 #+caption: An example retinal profile image. White pixels are
3670 #+caption: photo-sensitive elements. The distribution of white
3671 #+caption: pixels is denser in the middle and falls off at the
3672 #+caption: edges and is inspired by the human retina.
3673 #+name: retina
3674 #+ATTR_LaTeX: :width 7cm :placement [H]
3675 [[./images/retina-small.png]]
3677 *** Hearing
3679 Ears are created by creating an empty node named =ears= and then
3680 creating any number of empty child nodes to represent your
3681 creature's ears.
3683 Ear nodes do not require any metadata.
3685 The ear will bind to and follow the closest physical node.
3687 *** Touch
3689 Touch is handled similarly to mass. To make a particular object
3690 touch sensitive, add metadata of the following form under the
3691 object's ``touch'' metadata field:
3693 #+BEGIN_EXAMPLE
3694 <touch-UV-map-file-name>
3695 #+END_EXAMPLE
3697 You may also include an optional ``scale'' metadata number to
3698 specify the length of the touch feelers. The default is $0.1$,
3699 and this is generally sufficient.
3701 The touch UV should contain white pixels for each touch sensor.
3703 Here is an example touch-uv map that approximates a human finger,
3704 and its corresponding model.
3706 #+caption: This is the tactile-sensor-profile for the upper segment
3707 #+caption: of a fingertip. It defines regions of high touch sensitivity
3708 #+caption: (where there are many white pixels) and regions of low
3709 #+caption: sensitivity (where white pixels are sparse).
3710 #+name: guide-fingertip-UV
3711 #+ATTR_LaTeX: :width 9cm :placement [H]
3712 [[./images/finger-UV.png]]
3714 #+caption: The fingertip UV-image form above applied to a simple
3715 #+caption: model of a fingertip.
3716 #+name: guide-fingertip
3717 #+ATTR_LaTeX: :width 9cm :placement [H]
3718 [[./images/finger-1.png]]
3720 *** Proprioception
3722 Proprioception is tied to each joint node -- nothing special must
3723 be done in a Blender model to enable proprioception other than
3724 creating joint nodes.
3726 *** Muscles
3728 Muscles are created by creating an empty node named =muscles= and
3729 then creating any number of empty child nodes to represent your
3730 creature's muscles.
3733 Muscle nodes should have the following metadata under the
3734 ``muscle'' label:
3736 #+BEGIN_EXAMPLE
3737 <muscle-profile-file-name>
3738 #+END_EXAMPLE
3740 Muscles should also have a ``strength'' metadata entry describing
3741 the muscle's total strength at full activation.
3743 Muscle profiles are simple images that contain the relative amount
3744 of muscle power in each simulated alpha motor neuron. The width of
3745 the image is the total size of the motor pool, and the redness of
3746 each neuron is the relative power of that motor pool.
3748 While the profile image can have any dimensions, only the first
3749 line of pixels is used to define the muscle. Here is a sample
3750 muscle profile image that defines a human-like muscle.
3752 #+caption: A muscle profile image that describes the strengths
3753 #+caption: of each motor neuron in a muscle. White is weakest
3754 #+caption: and dark red is strongest. This particular pattern
3755 #+caption: has weaker motor neurons at the beginning, just
3756 #+caption: like human muscle.
3757 #+name: muscle-recruit
3758 #+ATTR_LaTeX: :width 7cm :placement [H]
3759 [[./images/basic-muscle.png]]
3761 Muscles twist the nearest physical object about the muscle node's
3762 Z-axis. I recommend using the ``Single Arrow'' display mode for
3763 muscles and using the right hand rule to determine which way the
3764 muscle will twist. To make a segment that can twist in multiple
3765 directions, create multiple, differently aligned muscles.
3767 ** =CORTEX= API
3769 These are the some functions exposed by =CORTEX= for creating
3770 worlds and simulating creatures. These are in addition to
3771 jMonkeyEngine3's extensive library, which is documented elsewhere.
3773 *** Simulation
3774 - =(world root-node key-map setup-fn update-fn)= :: create
3775 a simulation.
3776 - /root-node/ :: a =com.jme3.scene.Node= object which
3777 contains all of the objects that should be in the
3778 simulation.
3780 - /key-map/ :: a map from strings describing keys to
3781 functions that should be executed whenever that key is
3782 pressed. the functions should take a SimpleApplication
3783 object and a boolean value. The SimpleApplication is the
3784 current simulation that is running, and the boolean is true
3785 if the key is being pressed, and false if it is being
3786 released. As an example,
3787 #+BEGIN_SRC clojure
3788 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3789 #+END_SRC
3790 is a valid key-map which will cause the simulation to print
3791 a message whenever the 'j' key on the keyboard is pressed.
3793 - /setup-fn/ :: a function that takes a =SimpleApplication=
3794 object. It is called once when initializing the simulation.
3795 Use it to create things like lights, change the gravity,
3796 initialize debug nodes, etc.
3798 - /update-fn/ :: this function takes a =SimpleApplication=
3799 object and a float and is called every frame of the
3800 simulation. The float tells how many seconds is has been
3801 since the last frame was rendered, according to whatever
3802 clock jme is currently using. The default is to use IsoTimer
3803 which will result in this value always being the same.
3805 - =(position-camera world position rotation)= :: set the position
3806 of the simulation's main camera.
3808 - =(enable-debug world)= :: turn on debug wireframes for each
3809 simulated object.
3811 - =(set-gravity world gravity)= :: set the gravity of a running
3812 simulation.
3814 - =(box length width height & {options})= :: create a box in the
3815 simulation. Options is a hash map specifying texture, mass,
3816 etc. Possible options are =:name=, =:color=, =:mass=,
3817 =:friction=, =:texture=, =:material=, =:position=,
3818 =:rotation=, =:shape=, and =:physical?=.
3820 - =(sphere radius & {options})= :: create a sphere in the simulation.
3821 Options are the same as in =box=.
3823 - =(load-blender-model file-name)= :: create a node structure
3824 representing the model described in a Blender file.
3826 - =(light-up-everything world)= :: distribute a standard compliment
3827 of lights throughout the simulation. Should be adequate for most
3828 purposes.
3830 - =(node-seq node)= :: return a recursive list of the node's
3831 children.
3833 - =(nodify name children)= :: construct a node given a node-name and
3834 desired children.
3836 - =(add-element world element)= :: add an object to a running world
3837 simulation.
3839 - =(set-accuracy world accuracy)= :: change the accuracy of the
3840 world's physics simulator.
3842 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3843 construct that is useful for loading textures and is required
3844 for smooth interaction with jMonkeyEngine library functions.
3846 - =(load-bullet)= :: unpack native libraries and initialize the
3847 bullet physics subsystem. This function is required before
3848 other world building functions are called.
3850 *** Creature Manipulation / Import
3852 - =(body! creature)= :: give the creature a physical body.
3854 - =(vision! creature)= :: give the creature a sense of vision.
3855 Returns a list of functions which will each, when called
3856 during a simulation, return the vision data for the channel of
3857 one of the eyes. The functions are ordered depending on the
3858 alphabetical order of the names of the eye nodes in the
3859 Blender file. The data returned by the functions is a vector
3860 containing the eye's /topology/, a vector of coordinates, and
3861 the eye's /data/, a vector of RGB values filtered by the eye's
3862 sensitivity.
3864 - =(hearing! creature)= :: give the creature a sense of hearing.
3865 Returns a list of functions, one for each ear, that when
3866 called will return a frame's worth of hearing data for that
3867 ear. The functions are ordered depending on the alphabetical
3868 order of the names of the ear nodes in the Blender file. The
3869 data returned by the functions is an array of PCM (pulse code
3870 modulated) wav data.
3872 - =(touch! creature)= :: give the creature a sense of touch. Returns
3873 a single function that must be called with the /root node/ of
3874 the world, and which will return a vector of /touch-data/
3875 one entry for each touch sensitive component, each entry of
3876 which contains a /topology/ that specifies the distribution of
3877 touch sensors, and the /data/, which is a vector of
3878 =[activation, length]= pairs for each touch hair.
3880 - =(proprioception! creature)= :: give the creature the sense of
3881 proprioception. Returns a list of functions, one for each
3882 joint, that when called during a running simulation will
3883 report the =[heading, pitch, roll]= of the joint.
3885 - =(movement! creature)= :: give the creature the power of movement.
3886 Creates a list of functions, one for each muscle, that when
3887 called with an integer, will set the recruitment of that
3888 muscle to that integer, and will report the current power
3889 being exerted by the muscle. Order of muscles is determined by
3890 the alphabetical sort order of the names of the muscle nodes.
3892 *** Visualization/Debug
3894 - =(view-vision)= :: create a function that when called with a list
3895 of visual data returned from the functions made by =vision!=,
3896 will display that visual data on the screen.
3898 - =(view-hearing)= :: same as =view-vision= but for hearing.
3900 - =(view-touch)= :: same as =view-vision= but for touch.
3902 - =(view-proprioception)= :: same as =view-vision= but for
3903 proprioception.
3905 - =(view-movement)= :: same as =view-vision= but for muscles.
3907 - =(view anything)= :: =view= is a polymorphic function that allows
3908 you to inspect almost anything you could reasonably expect to
3909 be able to ``see'' in =CORTEX=.
3911 - =(text anything)= :: =text= is a polymorphic function that allows
3912 you to convert practically anything into a text string.
3914 - =(println-repl anything)= :: print messages to clojure's repl
3915 instead of the simulation's terminal window.
3917 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3918 function will import all jMonkeyEngine3 classes for immediate
3919 use.
3921 - =(display-dilated-time world timer)= :: Shows the time as it is
3922 flowing in the simulation on a HUD display.