view thesis/cortex.org @ 556:531bcd85d153

very minor spell check and formating.
author Robert McIntyre <rlm@mit.edu>
date Fri, 02 May 2014 14:31:39 -0400
parents c8f7d10b1a2a
children 830dc2e426b6
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * Empathy \& Embodiment: problem solving strategies
46 By the time you have read this thesis, you will understand a novel
47 approach to representing and recognizing physical actions using
48 embodiment and empathy. You will also see one way to efficiently
49 implement physical empathy for embodied creatures. Finally, you will
50 become familiar with =CORTEX=, a system for designing and simulating
51 creatures with rich senses, which I have designed as a library that
52 you can use in your own research. Note that I /do not/ process video
53 directly --- I start with knowledge of the positions of a creature's
54 body parts and work from there.
56 This is the core vision of my thesis: That one of the important ways
57 in which we understand others is by imagining ourselves in their
58 position and emphatically feeling experiences relative to our own
59 bodies. By understanding events in terms of our own previous
60 corporeal experience, we greatly constrain the possibilities of what
61 would otherwise be an unwieldy exponential search. This extra
62 constraint can be the difference between easily understanding what
63 is happening in a video and being completely lost in a sea of
64 incomprehensible color and movement.
66 ** The problem: recognizing actions is hard!
68 Examine figure \ref{cat-drink}. What is happening? As you, and
69 indeed very young children, can easily determine, this is an image
70 of drinking.
72 #+caption: A cat drinking some water. Identifying this action is
73 #+caption: beyond the capabilities of existing computer vision systems.
74 #+name: cat-drink
75 #+ATTR_LaTeX: :width 7cm
76 [[./images/cat-drinking.jpg]]
78 Nevertheless, it is beyond the state of the art for a computer
79 vision program to describe what's happening in this image. Part of
80 the problem is that many computer vision systems focus on
81 pixel-level details or comparisons to example images (such as
82 \cite{volume-action-recognition}), but the 3D world is so variable
83 that it is hard to describe the world in terms of possible images.
85 In fact, the contents of a scene may have much less to do with
86 pixel probabilities than with recognizing various affordances:
87 things you can move, objects you can grasp, spaces that can be
88 filled . For example, what processes might enable you to see the
89 chair in figure \ref{hidden-chair}?
91 #+caption: The chair in this image is quite obvious to humans, but
92 #+caption: it can't be found by any modern computer vision program.
93 #+name: hidden-chair
94 #+ATTR_LaTeX: :width 10cm
95 [[./images/fat-person-sitting-at-desk.jpg]]
97 Finally, how is it that you can easily tell the difference between
98 how the girl's /muscles/ are working in figure \ref{girl}?
100 #+caption: The mysterious ``common sense'' appears here as you are able
101 #+caption: to discern the difference in how the girl's arm muscles
102 #+caption: are activated between the two images. When you compare
103 #+caption: these two images, do you feel something in your own arm
104 #+caption: muscles?
105 #+name: girl
106 #+ATTR_LaTeX: :width 7cm
107 [[./images/wall-push.png]]
109 Each of these examples tells us something about what might be going
110 on in our minds as we easily solve these recognition problems:
112 - The hidden chair shows us that we are strongly triggered by cues
113 relating to the position of human bodies, and that we can
114 determine the overall physical configuration of a human body even
115 if much of that body is occluded.
117 - The picture of the girl pushing against the wall tells us that we
118 have common sense knowledge about the kinetics of our own bodies.
119 We know well how our muscles would have to work to maintain us in
120 most positions, and we can easily project this self-knowledge to
121 imagined positions triggered by images of the human body.
123 - The cat tells us that imagination of some kind plays an important
124 role in understanding actions. The question is: Can we be more
125 precise about what sort of imagination is required to understand
126 these actions?
128 ** A step forward: the sensorimotor-centered approach
130 In this thesis, I explore the idea that our knowledge of our own
131 bodies, combined with our own rich senses, enables us to recognize
132 the actions of others.
134 For example, I think humans are able to label the cat video as
135 ``drinking'' because they imagine /themselves/ as the cat, and
136 imagine putting their face up against a stream of water and
137 sticking out their tongue. In that imagined world, they can feel
138 the cool water hitting their tongue, and feel the water entering
139 their body, and are able to recognize that /feeling/ as drinking.
140 So, the label of the action is not really in the pixels of the
141 image, but is found clearly in a simulation / recollection inspired
142 by those pixels. An imaginative system, having been trained on
143 drinking and non-drinking examples and learning that the most
144 important component of drinking is the feeling of water flowing
145 down one's throat, would analyze a video of a cat drinking in the
146 following manner:
148 1. Create a physical model of the video by putting a ``fuzzy''
149 model of its own body in place of the cat. Possibly also create
150 a simulation of the stream of water.
152 2. Play out this simulated scene and generate imagined sensory
153 experience. This will include relevant muscle contractions, a
154 close up view of the stream from the cat's perspective, and most
155 importantly, the imagined feeling of water entering the mouth.
156 The imagined sensory experience can come from a simulation of
157 the event, but can also be pattern-matched from previous,
158 similar embodied experience.
160 3. The action is now easily identified as drinking by the sense of
161 taste alone. The other senses (such as the tongue moving in and
162 out) help to give plausibility to the simulated action. Note that
163 the sense of vision, while critical in creating the simulation,
164 is not critical for identifying the action from the simulation.
166 For the chair examples, the process is even easier:
168 1. Align a model of your body to the person in the image.
170 2. Generate proprioceptive sensory data from this alignment.
172 3. Use the imagined proprioceptive data as a key to lookup related
173 sensory experience associated with that particular proprioceptive
174 feeling.
176 4. Retrieve the feeling of your bottom resting on a surface, your
177 knees bent, and your leg muscles relaxed.
179 5. This sensory information is consistent with your =sitting?=
180 sensory predicate, so you (and the entity in the image) must be
181 sitting.
183 6. There must be a chair-like object since you are sitting.
185 Empathy offers yet another alternative to the age-old AI
186 representation question: ``What is a chair?'' --- A chair is the
187 feeling of sitting!
189 One powerful advantage of empathic problem solving is that it
190 factors the action recognition problem into two easier problems. To
191 use empathy, you need an /aligner/, which takes the video and a
192 model of your body, and aligns the model with the video. Then, you
193 need a /recognizer/, which uses the aligned model to interpret the
194 action. The power in this method lies in the fact that you describe
195 all actions from a body-centered viewpoint. You are less tied to
196 the particulars of any visual representation of the actions. If you
197 teach the system what ``running'' is, and you have a good enough
198 aligner, the system will from then on be able to recognize running
199 from any point of view -- even strange points of view like above or
200 underneath the runner. This is in contrast to action recognition
201 schemes that try to identify actions using a non-embodied approach.
202 If these systems learn about running as viewed from the side, they
203 will not automatically be able to recognize running from any other
204 viewpoint.
206 Another powerful advantage is that using the language of multiple
207 body-centered rich senses to describe body-centered actions offers
208 a massive boost in descriptive capability. Consider how difficult
209 it would be to compose a set of HOG (Histogram of Oriented
210 Gradients) filters to describe the action of a simple worm-creature
211 ``curling'' so that its head touches its tail, and then behold the
212 simplicity of describing thus action in a language designed for the
213 task (listing \ref{grand-circle-intro}):
215 #+caption: Body-centered actions are best expressed in a body-centered
216 #+caption: language. This code detects when the worm has curled into a
217 #+caption: full circle. Imagine how you would replicate this functionality
218 #+caption: using low-level pixel features such as HOG filters!
219 #+name: grand-circle-intro
220 #+begin_listing clojure
221 #+begin_src clojure
222 (defn grand-circle?
223 "Does the worm form a majestic circle (one end touching the other)?"
224 [experiences]
225 (and (curled? experiences)
226 (let [worm-touch (:touch (peek experiences))
227 tail-touch (worm-touch 0)
228 head-touch (worm-touch 4)]
229 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
230 (< 0.2 (contact worm-segment-top-tip head-touch))))))
231 #+end_src
232 #+end_listing
234 ** =EMPATH= recognizes actions using empathy
236 Exploring these ideas further demands a concrete implementation, so
237 first, I built a system for constructing virtual creatures with
238 physiologically plausible sensorimotor systems and detailed
239 environments. The result is =CORTEX=, which I describe in chapter
240 \ref{sec-2}.
242 Next, I wrote routines which enabled a simple worm-like creature to
243 infer the actions of a second worm-like creature, using only its
244 own prior sensorimotor experiences and knowledge of the second
245 worm's joint positions. This program, =EMPATH=, is described in
246 chapter \ref{sec-3}. It's main components are:
248 - Embodied Action Definitions :: Many otherwise complicated actions
249 are easily described in the language of a full suite of
250 body-centered, rich senses and experiences. For example,
251 drinking is the feeling of water flowing down your throat, and
252 cooling your insides. It's often accompanied by bringing your
253 hand close to your face, or bringing your face close to water.
254 Sitting down is the feeling of bending your knees, activating
255 your quadriceps, then feeling a surface with your bottom and
256 relaxing your legs. These body-centered action descriptions
257 can be either learned or hard coded.
259 - Guided Play :: The creature moves around and experiences the
260 world through its unique perspective. As the creature moves,
261 it gathers experiences that satisfy the embodied action
262 definitions.
264 - Posture imitation :: When trying to interpret a video or image,
265 the creature takes a model of itself and aligns it with
266 whatever it sees. This alignment might even cross species, as
267 when humans try to align themselves with things like ponies,
268 dogs, or other humans with a different body type.
270 - Empathy :: The alignment triggers associations with
271 sensory data from prior experiences. For example, the
272 alignment itself easily maps to proprioceptive data. Any
273 sounds or obvious skin contact in the video can to a lesser
274 extent trigger previous experience keyed to hearing or touch.
275 Segments of previous experiences gained from play are stitched
276 together to form a coherent and complete sensory portrait of
277 the scene.
279 - Recognition :: With the scene described in terms of remembered
280 first person sensory events, the creature can now run its
281 action-definition programs (such as the one in listing
282 \ref{grand-circle-intro}) on this synthesized sensory data,
283 just as it would if it were actually experiencing the scene
284 first-hand. If previous experience has been accurately
285 retrieved, and if it is analogous enough to the scene, then
286 the creature will correctly identify the action in the scene.
288 My program, =EMPATH= uses this empathic problem solving technique
289 to interpret the actions of a simple, worm-like creature.
291 #+caption: The worm performs many actions during free play such as
292 #+caption: curling, wiggling, and resting.
293 #+name: worm-intro
294 #+ATTR_LaTeX: :width 15cm
295 [[./images/worm-intro-white.png]]
297 #+caption: =EMPATH= recognized and classified each of these
298 #+caption: poses by inferring the complete sensory experience
299 #+caption: from proprioceptive data.
300 #+name: worm-recognition-intro
301 #+ATTR_LaTeX: :width 15cm
302 [[./images/worm-poses.png]]
304 *** Main Results
306 - After one-shot supervised training, =EMPATH= was able to
307 recognize a wide variety of static poses and dynamic
308 actions---ranging from curling in a circle to wiggling with a
309 particular frequency --- with 95\% accuracy.
311 - These results were completely independent of viewing angle
312 because the underlying body-centered language fundamentally is
313 independent; once an action is learned, it can be recognized
314 equally well from any viewing angle.
316 - =EMPATH= is surprisingly short; the sensorimotor-centered
317 language provided by =CORTEX= resulted in extremely economical
318 recognition routines --- about 500 lines in all --- suggesting
319 that such representations are very powerful, and often
320 indispensable for the types of recognition tasks considered here.
322 - For expediency's sake, I relied on direct knowledge of joint
323 positions in this proof of concept. However, I believe that the
324 structure of =EMPATH= and =CORTEX= will make future work to
325 enable video analysis much easier than it would otherwise be.
327 ** =EMPATH= is built on =CORTEX=, a creature builder.
329 I built =CORTEX= to be a general AI research platform for doing
330 experiments involving multiple rich senses and a wide variety and
331 number of creatures. I intend it to be useful as a library for many
332 more projects than just this thesis. =CORTEX= was necessary to meet
333 a need among AI researchers at CSAIL and beyond, which is that
334 people often will invent wonderful ideas that are best expressed in
335 the language of creatures and senses, but in order to explore those
336 ideas they must first build a platform in which they can create
337 simulated creatures with rich senses! There are many ideas that
338 would be simple to execute (such as =EMPATH= or Larson's
339 self-organizing maps (\cite{larson-symbols})), but attached to them
340 is the multi-month effort to make a good creature simulator. Often,
341 that initial investment of time proves to be too much, and the
342 project must make do with a lesser environment or be abandoned
343 entirely.
345 =CORTEX= is well suited as an environment for embodied AI research
346 for three reasons:
348 - You can design new creatures using Blender (\cite{blender}), a
349 popular, free 3D modeling program. Each sense can be specified
350 using special blender nodes with biologically inspired
351 parameters. You need not write any code to create a creature, and
352 can use a wide library of pre-existing blender models as a base
353 for your own creatures.
355 - =CORTEX= implements a wide variety of senses: touch,
356 proprioception, vision, hearing, and muscle tension. Complicated
357 senses like touch and vision involve multiple sensory elements
358 embedded in a 2D surface. You have complete control over the
359 distribution of these sensor elements through the use of simple
360 image files. =CORTEX= implements more comprehensive hearing than
361 any other creature simulation system available.
363 - =CORTEX= supports any number of creatures and any number of
364 senses. Time in =CORTEX= dilates so that the simulated creatures
365 always perceive a perfectly smooth flow of time, regardless of
366 the actual computational load.
368 =CORTEX= is built on top of =jMonkeyEngine3=
369 (\cite{jmonkeyengine}), which is a video game engine designed to
370 create cross-platform 3D desktop games. =CORTEX= is mainly written
371 in clojure, a dialect of =LISP= that runs on the Java Virtual
372 Machine (JVM). The API for creating and simulating creatures and
373 senses is entirely expressed in clojure, though many senses are
374 implemented at the layer of jMonkeyEngine or below. For example,
375 for the sense of hearing I use a layer of clojure code on top of a
376 layer of java JNI bindings that drive a layer of =C++= code which
377 implements a modified version of =OpenAL= to support multiple
378 listeners. =CORTEX= is the only simulation environment that I know
379 of that can support multiple entities that can each hear the world
380 from their own perspective. Other senses also require a small layer
381 of Java code. =CORTEX= also uses =bullet=, a physics simulator
382 written in =C=.
384 #+caption: Here is the worm from figure \ref{worm-intro} modeled
385 #+caption: in Blender, a free 3D-modeling program. Senses and
386 #+caption: joints are described using special nodes in Blender.
387 #+name: worm-recognition-intro-2
388 #+ATTR_LaTeX: :width 12cm
389 [[./images/blender-worm.png]]
391 Here are some things I anticipate that =CORTEX= might be used for:
393 - exploring new ideas about sensory integration
394 - distributed communication among swarm creatures
395 - self-learning using free exploration,
396 - evolutionary algorithms involving creature construction
397 - exploration of exotic senses and effectors that are not possible
398 in the real world (such as telekinesis or a semantic sense)
399 - imagination using subworlds
401 During one test with =CORTEX=, I created 3,000 creatures each with
402 its own independent senses and ran them all at only 1/80 real time.
403 In another test, I created a detailed model of my own hand,
404 equipped with a realistic distribution of touch (more sensitive at
405 the fingertips), as well as eyes and ears, and it ran at around 1/4
406 real time.
408 #+BEGIN_LaTeX
409 \begin{sidewaysfigure}
410 \includegraphics[width=9.5in]{images/full-hand.png}
411 \caption{
412 I modeled my own right hand in Blender and rigged it with all the
413 senses that {\tt CORTEX} supports. My simulated hand has a
414 biologically inspired distribution of touch sensors. The senses are
415 displayed on the right, and the simulation is displayed on the
416 left. Notice that my hand is curling its fingers, that it can see
417 its own finger from the eye in its palm, and that it can feel its
418 own thumb touching its palm.}
419 \end{sidewaysfigure}
420 #+END_LaTeX
422 * COMMENT Designing =CORTEX=
424 In this chapter, I outline the design decisions that went into
425 making =CORTEX=, along with some details about its implementation.
426 (A practical guide to getting started with =CORTEX=, which skips
427 over the history and implementation details presented here, is
428 provided in an appendix at the end of this thesis.)
430 Throughout this project, I intended for =CORTEX= to be flexible and
431 extensible enough to be useful for other researchers who want to
432 test ideas of their own. To this end, wherever I have had to make
433 architectural choices about =CORTEX=, I have chosen to give as much
434 freedom to the user as possible, so that =CORTEX= may be used for
435 things I have not foreseen.
437 ** Building in simulation versus reality
438 The most important architectural decision of all is the choice to
439 use a computer-simulated environment in the first place! The world
440 is a vast and rich place, and for now simulations are a very poor
441 reflection of its complexity. It may be that there is a significant
442 qualitative difference between dealing with senses in the real
443 world and dealing with pale facsimiles of them in a simulation
444 (\cite{brooks-representation}). What are the advantages and
445 disadvantages of a simulation vs. reality?
447 *** Simulation
449 The advantages of virtual reality are that when everything is a
450 simulation, experiments in that simulation are absolutely
451 reproducible. It's also easier to change the creature and
452 environment to explore new situations and different sensory
453 combinations.
455 If the world is to be simulated on a computer, then not only do
456 you have to worry about whether the creature's senses are rich
457 enough to learn from the world, but whether the world itself is
458 rendered with enough detail and realism to give enough working
459 material to the creature's senses. To name just a few
460 difficulties facing modern physics simulators: destructibility of
461 the environment, simulation of water/other fluids, large areas,
462 nonrigid bodies, lots of objects, smoke. I don't know of any
463 computer simulation that would allow a creature to take a rock
464 and grind it into fine dust, then use that dust to make a clay
465 sculpture, at least not without spending years calculating the
466 interactions of every single small grain of dust. Maybe a
467 simulated world with today's limitations doesn't provide enough
468 richness for real intelligence to evolve.
470 *** Reality
472 The other approach for playing with senses is to hook your
473 software up to real cameras, microphones, robots, etc., and let it
474 loose in the real world. This has the advantage of eliminating
475 concerns about simulating the world at the expense of increasing
476 the complexity of implementing the senses. Instead of just
477 grabbing the current rendered frame for processing, you have to
478 use an actual camera with real lenses and interact with photons to
479 get an image. It is much harder to change the creature, which is
480 now partly a physical robot of some sort, since doing so involves
481 changing things around in the real world instead of modifying
482 lines of code. While the real world is very rich and definitely
483 provides enough stimulation for intelligence to develop (as
484 evidenced by our own existence), it is also uncontrollable in the
485 sense that a particular situation cannot be recreated perfectly or
486 saved for later use. It is harder to conduct Science because it is
487 harder to repeat an experiment. The worst thing about using the
488 real world instead of a simulation is the matter of time. Instead
489 of simulated time you get the constant and unstoppable flow of
490 real time. This severely limits the sorts of software you can use
491 to program an AI, because all sense inputs must be handled in real
492 time. Complicated ideas may have to be implemented in hardware or
493 may simply be impossible given the current speed of our
494 processors. Contrast this with a simulation, in which the flow of
495 time in the simulated world can be slowed down to accommodate the
496 limitations of the creature's programming. In terms of cost, doing
497 everything in software is far cheaper than building custom
498 real-time hardware. All you need is a laptop and some patience.
500 ** Simulated time enables rapid prototyping \& simple programs
502 I envision =CORTEX= being used to support rapid prototyping and
503 iteration of ideas. Even if I could put together a well constructed
504 kit for creating robots, it would still not be enough because of
505 the scourge of real-time processing. Anyone who wants to test their
506 ideas in the real world must always worry about getting their
507 algorithms to run fast enough to process information in real time.
508 The need for real time processing only increases if multiple senses
509 are involved. In the extreme case, even simple algorithms will have
510 to be accelerated by ASIC chips or FPGAs, turning what would
511 otherwise be a few lines of code and a 10x speed penalty into a
512 multi-month ordeal. For this reason, =CORTEX= supports
513 /time-dilation/, which scales back the framerate of the simulation
514 in proportion to the amount of processing each frame. From the
515 perspective of the creatures inside the simulation, time always
516 appears to flow at a constant rate, regardless of how complicated
517 the environment becomes or how many creatures are in the
518 simulation. The cost is that =CORTEX= can sometimes run slower than
519 real time. Time dilation works both ways, however --- simulations
520 of very simple creatures in =CORTEX= generally run at 40x real-time
521 on my machine!
523 ** All sense organs are two-dimensional surfaces
525 If =CORTEX= is to support a wide variety of senses, it would help
526 to have a better understanding of what a sense actually is! While
527 vision, touch, and hearing all seem like they are quite different
528 things, I was surprised to learn during the course of this thesis
529 that they (and all physical senses) can be expressed as exactly the
530 same mathematical object!
532 Human beings are three-dimensional objects, and the nerves that
533 transmit data from our various sense organs to our brain are
534 essentially one-dimensional. This leaves up to two dimensions in
535 which our sensory information may flow. For example, imagine your
536 skin: it is a two-dimensional surface around a three-dimensional
537 object (your body). It has discrete touch sensors embedded at
538 various points, and the density of these sensors corresponds to the
539 sensitivity of that region of skin. Each touch sensor connects to a
540 nerve, all of which eventually are bundled together as they travel
541 up the spinal cord to the brain. Intersect the spinal nerves with a
542 guillotining plane and you will see all of the sensory data of the
543 skin revealed in a roughly circular two-dimensional image which is
544 the cross section of the spinal cord. Points on this image that are
545 close together in this circle represent touch sensors that are
546 /probably/ close together on the skin, although there is of course
547 some cutting and rearrangement that has to be done to transfer the
548 complicated surface of the skin onto a two dimensional image.
550 Most human senses consist of many discrete sensors of various
551 properties distributed along a surface at various densities. For
552 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
553 disks, and Ruffini's endings (\cite{textbook901}), which detect
554 pressure and vibration of various intensities. For ears, it is the
555 stereocilia distributed along the basilar membrane inside the
556 cochlea; each one is sensitive to a slightly different frequency of
557 sound. For eyes, it is rods and cones distributed along the surface
558 of the retina. In each case, we can describe the sense with a
559 surface and a distribution of sensors along that surface.
561 In fact, almost every human sense can be effectively described in
562 terms of a surface containing embedded sensors. If the sense had
563 any more dimensions, then there wouldn't be enough room in the
564 spinal cord to transmit the information!
566 Therefore, =CORTEX= must support the ability to create objects and
567 then be able to ``paint'' points along their surfaces to describe
568 each sense.
570 Fortunately this idea is already a well known computer graphics
571 technique called /UV-mapping/. In UV-mapping, the three-dimensional
572 surface of a model is cut and smooshed until it fits on a
573 two-dimensional image. You paint whatever you want on that image,
574 and when the three-dimensional shape is rendered in a game the
575 smooshing and cutting is reversed and the image appears on the
576 three-dimensional object.
578 To make a sense, interpret the UV-image as describing the
579 distribution of that senses sensors. To get different types of
580 sensors, you can either use a different color for each type of
581 sensor, or use multiple UV-maps, each labeled with that sensor
582 type. I generally use a white pixel to mean the presence of a
583 sensor and a black pixel to mean the absence of a sensor, and use
584 one UV-map for each sensor-type within a given sense.
586 #+CAPTION: The UV-map for an elongated icososphere. The white
587 #+caption: dots each represent a touch sensor. They are dense
588 #+caption: in the regions that describe the tip of the finger,
589 #+caption: and less dense along the dorsal side of the finger
590 #+caption: opposite the tip.
591 #+name: finger-UV
592 #+ATTR_latex: :width 10cm
593 [[./images/finger-UV.png]]
595 #+caption: Ventral side of the UV-mapped finger. Notice the
596 #+caption: density of touch sensors at the tip.
597 #+name: finger-side-view
598 #+ATTR_LaTeX: :width 10cm
599 [[./images/finger-1.png]]
601 ** Video game engines provide ready-made physics and shading
603 I did not need to write my own physics simulation code or shader to
604 build =CORTEX=. Doing so would lead to a system that is impossible
605 for anyone but myself to use anyway. Instead, I use a video game
606 engine as a base and modify it to accommodate the additional needs
607 of =CORTEX=. Video game engines are an ideal starting point to
608 build =CORTEX=, because they are not far from being creature
609 building systems themselves.
611 First off, general purpose video game engines come with a physics
612 engine and lighting / sound system. The physics system provides
613 tools that can be co-opted to serve as touch, proprioception, and
614 muscles. Since some games support split screen views, a good video
615 game engine will allow you to efficiently create multiple cameras
616 in the simulated world that can be used as eyes. Video game systems
617 offer integrated asset management for things like textures and
618 creature models, providing an avenue for defining creatures. They
619 also understand UV-mapping, since this technique is used to apply a
620 texture to a model. Finally, because video game engines support a
621 large number of developers, as long as =CORTEX= doesn't stray too
622 far from the base system, other researchers can turn to this
623 community for help when doing their research.
625 ** =CORTEX= is based on jMonkeyEngine3
627 While preparing to build =CORTEX= I studied several video game
628 engines to see which would best serve as a base. The top contenders
629 were:
631 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID software
632 in 1997. All the source code was released by ID software into
633 the Public Domain several years ago, and as a result it has
634 been ported to many different languages. This engine was
635 famous for its advanced use of realistic shading and it had
636 decent and fast physics simulation. The main advantage of the
637 Quake II engine is its simplicity, but I ultimately rejected
638 it because the engine is too tied to the concept of a
639 first-person shooter game. One of the problems I had was that
640 there does not seem to be any easy way to attach multiple
641 cameras to a single character. There are also several physics
642 clipping issues that are corrected in a way that only applies
643 to the main character and do not apply to arbitrary objects.
645 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
646 and Quake I engines and is used by Valve in the Half-Life
647 series of games. The physics simulation in the Source Engine
648 is quite accurate and probably the best out of all the engines
649 I investigated. There is also an extensive community actively
650 working with the engine. However, applications that use the
651 Source Engine must be written in C++, the code is not open, it
652 only runs on Windows, and the tools that come with the SDK to
653 handle models and textures are complicated and awkward to use.
655 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
656 games in Java. It uses OpenGL to render to the screen and uses
657 screengraphs to avoid drawing things that do not appear on the
658 screen. It has an active community and several games in the
659 pipeline. The engine was not built to serve any particular
660 game but is instead meant to be used for any 3D game.
662 I chose jMonkeyEngine3 because it had the most features out of all
663 the free projects I looked at, and because I could then write my
664 code in clojure, an implementation of =LISP= that runs on the JVM.
666 ** =CORTEX= uses Blender to create creature models
668 For the simple worm-like creatures I will use later on in this
669 thesis, I could define a simple API in =CORTEX= that would allow
670 one to create boxes, spheres, etc., and leave that API as the sole
671 way to create creatures. However, for =CORTEX= to truly be useful
672 for other projects, it needs a way to construct complicated
673 creatures. If possible, it would be nice to leverage work that has
674 already been done by the community of 3D modelers, or at least
675 enable people who are talented at modeling but not programming to
676 design =CORTEX= creatures.
678 Therefore I use Blender, a free 3D modeling program, as the main
679 way to create creatures in =CORTEX=. However, the creatures modeled
680 in Blender must also be simple to simulate in jMonkeyEngine3's game
681 engine, and must also be easy to rig with =CORTEX='s senses. I
682 accomplish this with extensive use of Blender's ``empty nodes.''
684 Empty nodes have no mass, physical presence, or appearance, but
685 they can hold metadata and have names. I use a tree structure of
686 empty nodes to specify senses in the following manner:
688 - Create a single top-level empty node whose name is the name of
689 the sense.
690 - Add empty nodes which each contain meta-data relevant to the
691 sense, including a UV-map describing the number/distribution of
692 sensors if applicable.
693 - Make each empty-node the child of the top-level node.
695 #+caption: An example of annotating a creature model with empty
696 #+caption: nodes to describe the layout of senses. There are
697 #+caption: multiple empty nodes which each describe the position
698 #+caption: of muscles, ears, eyes, or joints.
699 #+name: sense-nodes
700 #+ATTR_LaTeX: :width 10cm
701 [[./images/empty-sense-nodes.png]]
703 ** Bodies are composed of segments connected by joints
705 Blender is a general purpose animation tool, which has been used in
706 the past to create high quality movies such as Sintel
707 (\cite{blender}). Though Blender can model and render even
708 complicated things like water, it is crucial to keep models that
709 are meant to be simulated as creatures simple. =Bullet=, which
710 =CORTEX= uses though jMonkeyEngine3, is a rigid-body physics
711 system. This offers a compromise between the expressiveness of a
712 game level and the speed at which it can be simulated, and it means
713 that creatures should be naturally expressed as rigid components
714 held together by joint constraints.
716 But humans are more like a squishy bag wrapped around some hard
717 bones which define the overall shape. When we move, our skin bends
718 and stretches to accommodate the new positions of our bones.
720 One way to make bodies composed of rigid pieces connected by joints
721 /seem/ more human-like is to use an /armature/, (or /rigging/)
722 system, which defines a overall ``body mesh'' and defines how the
723 mesh deforms as a function of the position of each ``bone'' which
724 is a standard rigid body. This technique is used extensively to
725 model humans and create realistic animations. It is not a good
726 technique for physical simulation because it is a lie -- the skin
727 is not a physical part of the simulation and does not interact with
728 any objects in the world or itself. Objects will pass right though
729 the skin until they come in contact with the underlying bone, which
730 is a physical object. Without simulating the skin, the sense of
731 touch has little meaning, and the creature's own vision will lie to
732 it about the true extent of its body. Simulating the skin as a
733 physical object requires some way to continuously update the
734 physical model of the skin along with the movement of the bones,
735 which is unacceptably slow compared to rigid body simulation.
737 Therefore, instead of using the human-like ``bony meatbag''
738 approach, I decided to base my body plans on multiple solid objects
739 that are connected by joints, inspired by the robot =EVE= from the
740 movie WALL-E.
742 #+caption: =EVE= from the movie WALL-E. This body plan turns
743 #+caption: out to be much better suited to my purposes than a more
744 #+caption: human-like one.
745 #+ATTR_LaTeX: :width 10cm
746 [[./images/Eve.jpg]]
748 =EVE='s body is composed of several rigid components that are held
749 together by invisible joint constraints. This is what I mean by
750 /eve-like/. The main reason that I use eve-like bodies is for
751 simulation efficiency, and so that there will be correspondence
752 between the AI's senses and the physical presence of its body. Each
753 individual section is simulated by a separate rigid body that
754 corresponds exactly with its visual representation and does not
755 change. Sections are connected by invisible joints that are well
756 supported in jMonkeyEngine3. Bullet, the physics backend for
757 jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
758 connected by joints. Just because sections are rigid does not mean
759 they have to stay as one piece forever; they can be dynamically
760 replaced with multiple sections to simulate splitting in two. This
761 could be used to simulate retractable claws or =EVE='s hands, which
762 are able to coalesce into one object in the movie.
764 *** Solidifying/Connecting a body
766 =CORTEX= creates a creature in two steps: first, it traverses the
767 nodes in the blender file and creates physical representations for
768 any of them that have mass defined in their blender meta-data.
770 #+caption: Program for iterating through the nodes in a blender file
771 #+caption: and generating physical jMonkeyEngine3 objects with mass
772 #+caption: and a matching physics shape.
773 #+name: physical
774 #+begin_listing clojure
775 #+begin_src clojure
776 (defn physical!
777 "Iterate through the nodes in creature and make them real physical
778 objects in the simulation."
779 [#^Node creature]
780 (dorun
781 (map
782 (fn [geom]
783 (let [physics-control
784 (RigidBodyControl.
785 (HullCollisionShape.
786 (.getMesh geom))
787 (if-let [mass (meta-data geom "mass")]
788 (float mass) (float 1)))]
789 (.addControl geom physics-control)))
790 (filter #(isa? (class %) Geometry )
791 (node-seq creature)))))
792 #+end_src
793 #+end_listing
795 The next step to making a proper body is to connect those pieces
796 together with joints. jMonkeyEngine has a large array of joints
797 available via =bullet=, such as Point2Point, Cone, Hinge, and a
798 generic Six Degree of Freedom joint, with or without spring
799 restitution.
801 Joints are treated a lot like proper senses, in that there is a
802 top-level empty node named ``joints'' whose children each
803 represent a joint.
805 #+caption: View of the hand model in Blender showing the main ``joints''
806 #+caption: node (highlighted in yellow) and its children which each
807 #+caption: represent a joint in the hand. Each joint node has metadata
808 #+caption: specifying what sort of joint it is.
809 #+name: blender-hand
810 #+ATTR_LaTeX: :width 10cm
811 [[./images/hand-screenshot1.png]]
814 =CORTEX='s procedure for binding the creature together with joints
815 is as follows:
817 - Find the children of the ``joints'' node.
818 - Determine the two spatials the joint is meant to connect.
819 - Create the joint based on the meta-data of the empty node.
821 The higher order function =sense-nodes= from =cortex.sense=
822 simplifies finding the joints based on their parent ``joints''
823 node.
825 #+caption: Retrieving the children empty nodes from a single
826 #+caption: named empty node is a common pattern in =CORTEX=
827 #+caption: further instances of this technique for the senses
828 #+caption: will be omitted
829 #+name: get-empty-nodes
830 #+begin_listing clojure
831 #+begin_src clojure
832 (defn sense-nodes
833 "For some senses there is a special empty blender node whose
834 children are considered markers for an instance of that sense. This
835 function generates functions to find those children, given the name
836 of the special parent node."
837 [parent-name]
838 (fn [#^Node creature]
839 (if-let [sense-node (.getChild creature parent-name)]
840 (seq (.getChildren sense-node)) [])))
842 (def
843 ^{:doc "Return the children of the creature's \"joints\" node."
844 :arglists '([creature])}
845 joints
846 (sense-nodes "joints"))
847 #+end_src
848 #+end_listing
850 To find a joint's targets, =CORTEX= creates a small cube, centered
851 around the empty-node, and grows the cube exponentially until it
852 intersects two physical objects. The objects are ordered according
853 to the joint's rotation, with the first one being the object that
854 has more negative coordinates in the joint's reference frame.
855 Since the objects must be physical, the empty-node itself escapes
856 detection. Because the objects must be physical, =joint-targets=
857 must be called /after/ =physical!= is called.
859 #+caption: Program to find the targets of a joint node by
860 #+caption: exponentially growth of a search cube.
861 #+name: joint-targets
862 #+begin_listing clojure
863 #+begin_src clojure
864 (defn joint-targets
865 "Return the two closest two objects to the joint object, ordered
866 from bottom to top according to the joint's rotation."
867 [#^Node parts #^Node joint]
868 (loop [radius (float 0.01)]
869 (let [results (CollisionResults.)]
870 (.collideWith
871 parts
872 (BoundingBox. (.getWorldTranslation joint)
873 radius radius radius) results)
874 (let [targets
875 (distinct
876 (map #(.getGeometry %) results))]
877 (if (>= (count targets) 2)
878 (sort-by
879 #(let [joint-ref-frame-position
880 (jme-to-blender
881 (.mult
882 (.inverse (.getWorldRotation joint))
883 (.subtract (.getWorldTranslation %)
884 (.getWorldTranslation joint))))]
885 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
886 (take 2 targets))
887 (recur (float (* radius 2))))))))
888 #+end_src
889 #+end_listing
891 Once =CORTEX= finds all joints and targets, it creates them using
892 a dispatch on the metadata of each joint node.
894 #+caption: Program to dispatch on blender metadata and create joints
895 #+caption: suitable for physical simulation.
896 #+name: joint-dispatch
897 #+begin_listing clojure
898 #+begin_src clojure
899 (defmulti joint-dispatch
900 "Translate blender pseudo-joints into real JME joints."
901 (fn [constraints & _]
902 (:type constraints)))
904 (defmethod joint-dispatch :point
905 [constraints control-a control-b pivot-a pivot-b rotation]
906 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
907 (.setLinearLowerLimit Vector3f/ZERO)
908 (.setLinearUpperLimit Vector3f/ZERO)))
910 (defmethod joint-dispatch :hinge
911 [constraints control-a control-b pivot-a pivot-b rotation]
912 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
913 [limit-1 limit-2] (:limit constraints)
914 hinge-axis (.mult rotation (blender-to-jme axis))]
915 (doto (HingeJoint. control-a control-b pivot-a pivot-b
916 hinge-axis hinge-axis)
917 (.setLimit limit-1 limit-2))))
919 (defmethod joint-dispatch :cone
920 [constraints control-a control-b pivot-a pivot-b rotation]
921 (let [limit-xz (:limit-xz constraints)
922 limit-xy (:limit-xy constraints)
923 twist (:twist constraints)]
924 (doto (ConeJoint. control-a control-b pivot-a pivot-b
925 rotation rotation)
926 (.setLimit (float limit-xz) (float limit-xy)
927 (float twist)))))
928 #+end_src
929 #+end_listing
931 All that is left for joints is to combine the above pieces into
932 something that can operate on the collection of nodes that a
933 blender file represents.
935 #+caption: Program to completely create a joint given information
936 #+caption: from a blender file.
937 #+name: connect
938 #+begin_listing clojure
939 #+begin_src clojure
940 (defn connect
941 "Create a joint between 'obj-a and 'obj-b at the location of
942 'joint. The type of joint is determined by the metadata on 'joint.
944 Here are some examples:
945 {:type :point}
946 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
947 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
949 {:type :cone :limit-xz 0]
950 :limit-xy 0]
951 :twist 0]} (use XZY rotation mode in blender!)"
952 [#^Node obj-a #^Node obj-b #^Node joint]
953 (let [control-a (.getControl obj-a RigidBodyControl)
954 control-b (.getControl obj-b RigidBodyControl)
955 joint-center (.getWorldTranslation joint)
956 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
957 pivot-a (world-to-local obj-a joint-center)
958 pivot-b (world-to-local obj-b joint-center)]
959 (if-let
960 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
961 ;; A side-effect of creating a joint registers
962 ;; it with both physics objects which in turn
963 ;; will register the joint with the physics system
964 ;; when the simulation is started.
965 (joint-dispatch constraints
966 control-a control-b
967 pivot-a pivot-b
968 joint-rotation))))
969 #+end_src
970 #+end_listing
972 In general, whenever =CORTEX= exposes a sense (or in this case
973 physicality), it provides a function of the type =sense!=, which
974 takes in a collection of nodes and augments it to support that
975 sense. The function returns any controls necessary to use that
976 sense. In this case =body!= creates a physical body and returns no
977 control functions.
979 #+caption: Program to give joints to a creature.
980 #+name: joints
981 #+begin_listing clojure
982 #+begin_src clojure
983 (defn joints!
984 "Connect the solid parts of the creature with physical joints. The
985 joints are taken from the \"joints\" node in the creature."
986 [#^Node creature]
987 (dorun
988 (map
989 (fn [joint]
990 (let [[obj-a obj-b] (joint-targets creature joint)]
991 (connect obj-a obj-b joint)))
992 (joints creature))))
993 (defn body!
994 "Endow the creature with a physical body connected with joints. The
995 particulars of the joints and the masses of each body part are
996 determined in blender."
997 [#^Node creature]
998 (physical! creature)
999 (joints! creature))
1000 #+end_src
1001 #+end_listing
1003 All of the code you have just seen amounts to only 130 lines, yet
1004 because it builds on top of Blender and jMonkeyEngine3, those few
1005 lines pack quite a punch!
1007 The hand from figure \ref{blender-hand}, which was modeled after
1008 my own right hand, can now be given joints and simulated as a
1009 creature.
1011 #+caption: With the ability to create physical creatures from blender,
1012 #+caption: =CORTEX= gets one step closer to becoming a full creature
1013 #+caption: simulation environment.
1014 #+name: physical-hand
1015 #+ATTR_LaTeX: :width 15cm
1016 [[./images/physical-hand.png]]
1018 ** Sight reuses standard video game components...
1020 Vision is one of the most important senses for humans, so I need to
1021 build a simulated sense of vision for my AI. I will do this with
1022 simulated eyes. Each eye can be independently moved and should see
1023 its own version of the world depending on where it is.
1025 Making these simulated eyes a reality is simple because
1026 jMonkeyEngine already contains extensive support for multiple views
1027 of the same 3D simulated world. The reason jMonkeyEngine has this
1028 support is because the support is necessary to create games with
1029 split-screen views. Multiple views are also used to create
1030 efficient pseudo-reflections by rendering the scene from a certain
1031 perspective and then projecting it back onto a surface in the 3D
1032 world.
1034 #+caption: jMonkeyEngine supports multiple views to enable
1035 #+caption: split-screen games, like GoldenEye, which was one of
1036 #+caption: the first games to use split-screen views.
1037 #+name: goldeneye
1038 #+ATTR_LaTeX: :width 10cm
1039 [[./images/goldeneye-4-player.png]]
1041 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
1043 jMonkeyEngine allows you to create a =ViewPort=, which represents a
1044 view of the simulated world. You can create as many of these as you
1045 want. Every frame, the =RenderManager= iterates through each
1046 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
1047 is a =FrameBuffer= which represents the rendered image in the GPU.
1049 #+caption: =ViewPorts= are cameras in the world. During each frame,
1050 #+caption: the =RenderManager= records a snapshot of what each view
1051 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
1052 #+name: rendermanagers
1053 #+ATTR_LaTeX: :width 10cm
1054 [[./images/diagram_rendermanager2.png]]
1056 Each =ViewPort= can have any number of attached =SceneProcessor=
1057 objects, which are called every time a new frame is rendered. A
1058 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1059 whatever it wants to the data. Often this consists of invoking GPU
1060 specific operations on the rendered image. The =SceneProcessor= can
1061 also copy the GPU image data to RAM and process it with the CPU.
1063 *** Appropriating Views for Vision
1065 Each eye in the simulated creature needs its own =ViewPort= so
1066 that it can see the world from its own perspective. To this
1067 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1068 any arbitrary continuation function for further processing. That
1069 continuation function may perform both CPU and GPU operations on
1070 the data. To make this easy for the continuation function, the
1071 =SceneProcessor= maintains appropriately sized buffers in RAM to
1072 hold the data. It does not do any copying from the GPU to the CPU
1073 itself because it is a slow operation.
1075 #+caption: Function to make the rendered scene in jMonkeyEngine
1076 #+caption: available for further processing.
1077 #+name: pipeline-1
1078 #+begin_listing clojure
1079 #+begin_src clojure
1080 (defn vision-pipeline
1081 "Create a SceneProcessor object which wraps a vision processing
1082 continuation function. The continuation is a function that takes
1083 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1084 each of which has already been appropriately sized."
1085 [continuation]
1086 (let [byte-buffer (atom nil)
1087 renderer (atom nil)
1088 image (atom nil)]
1089 (proxy [SceneProcessor] []
1090 (initialize
1091 [renderManager viewPort]
1092 (let [cam (.getCamera viewPort)
1093 width (.getWidth cam)
1094 height (.getHeight cam)]
1095 (reset! renderer (.getRenderer renderManager))
1096 (reset! byte-buffer
1097 (BufferUtils/createByteBuffer
1098 (* width height 4)))
1099 (reset! image (BufferedImage.
1100 width height
1101 BufferedImage/TYPE_4BYTE_ABGR))))
1102 (isInitialized [] (not (nil? @byte-buffer)))
1103 (reshape [_ _ _])
1104 (preFrame [_])
1105 (postQueue [_])
1106 (postFrame
1107 [#^FrameBuffer fb]
1108 (.clear @byte-buffer)
1109 (continuation @renderer fb @byte-buffer @image))
1110 (cleanup []))))
1111 #+end_src
1112 #+end_listing
1114 The continuation function given to =vision-pipeline= above will be
1115 given a =Renderer= and three containers for image data. The
1116 =FrameBuffer= references the GPU image data, but the pixel data
1117 can not be used directly on the CPU. The =ByteBuffer= and
1118 =BufferedImage= are initially "empty" but are sized to hold the
1119 data in the =FrameBuffer=. I call transferring the GPU image data
1120 to the CPU structures "mixing" the image data.
1122 *** Optical sensor arrays are described with images and referenced with metadata
1124 The vision pipeline described above handles the flow of rendered
1125 images. Now, =CORTEX= needs simulated eyes to serve as the source
1126 of these images.
1128 An eye is described in blender in the same way as a joint. They
1129 are zero dimensional empty objects with no geometry whose local
1130 coordinate system determines the orientation of the resulting eye.
1131 All eyes are children of a parent node named "eyes" just as all
1132 joints have a parent named "joints". An eye binds to the nearest
1133 physical object with =bind-sense=.
1135 #+caption: Here, the camera is created based on metadata on the
1136 #+caption: eye-node and attached to the nearest physical object
1137 #+caption: with =bind-sense=
1138 #+name: add-eye
1139 #+begin_listing clojure
1140 #+begin_src clojure
1141 (defn add-eye!
1142 "Create a Camera centered on the current position of 'eye which
1143 follows the closest physical node in 'creature. The camera will
1144 point in the X direction and use the Z vector as up as determined
1145 by the rotation of these vectors in blender coordinate space. Use
1146 XZY rotation for the node in blender."
1147 [#^Node creature #^Spatial eye]
1148 (let [target (closest-node creature eye)
1149 [cam-width cam-height]
1150 ;;[640 480] ;; graphics card on laptop doesn't support
1151 ;; arbitrary dimensions.
1152 (eye-dimensions eye)
1153 cam (Camera. cam-width cam-height)
1154 rot (.getWorldRotation eye)]
1155 (.setLocation cam (.getWorldTranslation eye))
1156 (.lookAtDirection
1157 cam ; this part is not a mistake and
1158 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1159 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1160 (.setFrustumPerspective
1161 cam (float 45)
1162 (float (/ (.getWidth cam) (.getHeight cam)))
1163 (float 1)
1164 (float 1000))
1165 (bind-sense target cam) cam))
1166 #+end_src
1167 #+end_listing
1169 *** Simulated Retina
1171 An eye is a surface (the retina) which contains many discrete
1172 sensors to detect light. These sensors can have different
1173 light-sensing properties. In humans, each discrete sensor is
1174 sensitive to red, blue, green, or gray. These different types of
1175 sensors can have different spatial distributions along the retina.
1176 In humans, there is a fovea in the center of the retina which has
1177 a very high density of color sensors, and a blind spot which has
1178 no sensors at all. Sensor density decreases in proportion to
1179 distance from the fovea.
1181 I want to be able to model any retinal configuration, so my
1182 eye-nodes in blender contain metadata pointing to images that
1183 describe the precise position of the individual sensors using
1184 white pixels. The meta-data also describes the precise sensitivity
1185 to light that the sensors described in the image have. An eye can
1186 contain any number of these images. For example, the metadata for
1187 an eye might look like this:
1189 #+begin_src clojure
1190 {0xFF0000 "Models/test-creature/retina-small.png"}
1191 #+end_src
1193 #+caption: An example retinal profile image. White pixels are
1194 #+caption: photo-sensitive elements. The distribution of white
1195 #+caption: pixels is denser in the middle and falls off at the
1196 #+caption: edges and is inspired by the human retina.
1197 #+name: retina
1198 #+ATTR_LaTeX: :width 7cm
1199 [[./images/retina-small.png]]
1201 Together, the number 0xFF0000 and the image above describe the
1202 placement of red-sensitive sensory elements.
1204 Meta-data to very crudely approximate a human eye might be
1205 something like this:
1207 #+begin_src clojure
1208 (let [retinal-profile "Models/test-creature/retina-small.png"]
1209 {0xFF0000 retinal-profile
1210 0x00FF00 retinal-profile
1211 0x0000FF retinal-profile
1212 0xFFFFFF retinal-profile})
1213 #+end_src
1215 The numbers that serve as keys in the map determine a sensor's
1216 relative sensitivity to the channels red, green, and blue. These
1217 sensitivity values are packed into an integer in the order
1218 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1219 image are added together with these sensitivities as linear
1220 weights. Therefore, 0xFF0000 means sensitive to red only while
1221 0xFFFFFF means sensitive to all colors equally (gray).
1223 #+caption: This is the core of vision in =CORTEX=. A given eye node
1224 #+caption: is converted into a function that returns visual
1225 #+caption: information from the simulation.
1226 #+name: vision-kernel
1227 #+begin_listing clojure
1228 #+BEGIN_SRC clojure
1229 (defn vision-kernel
1230 "Returns a list of functions, each of which will return a color
1231 channel's worth of visual information when called inside a running
1232 simulation."
1233 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1234 (let [retinal-map (retina-sensor-profile eye)
1235 camera (add-eye! creature eye)
1236 vision-image
1237 (atom
1238 (BufferedImage. (.getWidth camera)
1239 (.getHeight camera)
1240 BufferedImage/TYPE_BYTE_BINARY))
1241 register-eye!
1242 (runonce
1243 (fn [world]
1244 (add-camera!
1245 world camera
1246 (let [counter (atom 0)]
1247 (fn [r fb bb bi]
1248 (if (zero? (rem (swap! counter inc) (inc skip)))
1249 (reset! vision-image
1250 (BufferedImage! r fb bb bi))))))))]
1251 (vec
1252 (map
1253 (fn [[key image]]
1254 (let [whites (white-coordinates image)
1255 topology (vec (collapse whites))
1256 sensitivity (sensitivity-presets key key)]
1257 (attached-viewport.
1258 (fn [world]
1259 (register-eye! world)
1260 (vector
1261 topology
1262 (vec
1263 (for [[x y] whites]
1264 (pixel-sense
1265 sensitivity
1266 (.getRGB @vision-image x y))))))
1267 register-eye!)))
1268 retinal-map))))
1269 #+END_SRC
1270 #+end_listing
1272 Note that since each of the functions generated by =vision-kernel=
1273 shares the same =register-eye!= function, the eye will be
1274 registered only once the first time any of the functions from the
1275 list returned by =vision-kernel= is called. Each of the functions
1276 returned by =vision-kernel= also allows access to the =Viewport=
1277 through which it receives images.
1279 All the hard work has been done; all that remains is to apply
1280 =vision-kernel= to each eye in the creature and gather the results
1281 into one list of functions.
1284 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1285 #+caption: environment for experimenting with different types of
1286 #+caption: eyes.
1287 #+name: vision!
1288 #+begin_listing clojure
1289 #+BEGIN_SRC clojure
1290 (defn vision!
1291 "Returns a list of functions, each of which returns visual sensory
1292 data when called inside a running simulation."
1293 [#^Node creature & {skip :skip :or {skip 0}}]
1294 (reduce
1295 concat
1296 (for [eye (eyes creature)]
1297 (vision-kernel creature eye))))
1298 #+END_SRC
1299 #+end_listing
1301 #+caption: Simulated vision with a test creature and the
1302 #+caption: human-like eye approximation. Notice how each channel
1303 #+caption: of the eye responds differently to the differently
1304 #+caption: colored balls.
1305 #+name: worm-vision-test.
1306 #+ATTR_LaTeX: :width 13cm
1307 [[./images/worm-vision.png]]
1309 The vision code is not much more complicated than the body code,
1310 and enables multiple further paths for simulated vision. For
1311 example, it is quite easy to create bifocal vision -- you just
1312 make two eyes next to each other in blender! It is also possible
1313 to encode vision transforms in the retinal files. For example, the
1314 human like retina file in figure \ref{retina} approximates a
1315 log-polar transform.
1317 This vision code has already been absorbed by the jMonkeyEngine
1318 community and is now (in modified form) part of a system for
1319 capturing in-game video to a file.
1321 ** ...but hearing must be built from scratch
1323 At the end of this chapter I will have simulated ears that work the
1324 same way as the simulated eyes in the last chapter. I will be able to
1325 place any number of ear-nodes in a blender file, and they will bind to
1326 the closest physical object and follow it as it moves around. Each ear
1327 will provide access to the sound data it picks up between every frame.
1329 Hearing is one of the more difficult senses to simulate, because there
1330 is less support for obtaining the actual sound data that is processed
1331 by jMonkeyEngine3. There is no "split-screen" support for rendering
1332 sound from different points of view, and there is no way to directly
1333 access the rendered sound data.
1335 =CORTEX='s hearing is unique because it does not have any
1336 limitations compared to other simulation environments. As far as I
1337 know, there is no other system that supports multiple listeners,
1338 and the sound demo at the end of this chapter is the first time
1339 it's been done in a video game environment.
1341 *** Brief Description of jMonkeyEngine's Sound System
1343 jMonkeyEngine's sound system works as follows:
1345 - jMonkeyEngine uses the =AppSettings= for the particular
1346 application to determine what sort of =AudioRenderer= should be
1347 used.
1348 - Although some support is provided for multiple AudioRenderer
1349 backends, jMonkeyEngine at the time of this writing will either
1350 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1351 - jMonkeyEngine tries to figure out what sort of system you're
1352 running and extracts the appropriate native libraries.
1353 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1354 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1355 - =OpenAL= renders the 3D sound and feeds the rendered sound
1356 directly to any of various sound output devices with which it
1357 knows how to communicate.
1359 A consequence of this is that there's no way to access the actual
1360 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1361 one /listener/ (it renders sound data from only one perspective),
1362 which normally isn't a problem for games, but becomes a problem
1363 when trying to make multiple AI creatures that can each hear the
1364 world from a different perspective.
1366 To make many AI creatures in jMonkeyEngine that can each hear the
1367 world from their own perspective, or to make a single creature with
1368 many ears, it is necessary to go all the way back to =OpenAL= and
1369 implement support for simulated hearing there.
1371 *** Extending =OpenAl=
1373 Extending =OpenAL= to support multiple listeners requires 500
1374 lines of =C= code and is too hairy to mention here. Instead, I
1375 will show a small amount of extension code and go over the high
1376 level strategy. Full source is of course available with the
1377 =CORTEX= distribution if you're interested.
1379 =OpenAL= goes to great lengths to support many different systems,
1380 all with different sound capabilities and interfaces. It
1381 accomplishes this difficult task by providing code for many
1382 different sound backends in pseudo-objects called /Devices/.
1383 There's a device for the Linux Open Sound System and the Advanced
1384 Linux Sound Architecture, there's one for Direct Sound on Windows,
1385 and there's even one for Solaris. =OpenAL= solves the problem of
1386 platform independence by providing all these Devices.
1388 Wrapper libraries such as LWJGL are free to examine the system on
1389 which they are running and then select an appropriate device for
1390 that system.
1392 There are also a few "special" devices that don't interface with
1393 any particular system. These include the Null Device, which
1394 doesn't do anything, and the Wave Device, which writes whatever
1395 sound it receives to a file, if everything has been set up
1396 correctly when configuring =OpenAL=.
1398 Actual mixing (Doppler shift and distance.environment-based
1399 attenuation) of the sound data happens in the Devices, and they
1400 are the only point in the sound rendering process where this data
1401 is available.
1403 Therefore, in order to support multiple listeners, and get the
1404 sound data in a form that the AIs can use, it is necessary to
1405 create a new Device which supports this feature.
1407 Adding a device to OpenAL is rather tricky -- there are five
1408 separate files in the =OpenAL= source tree that must be modified
1409 to do so. I named my device the "Multiple Audio Send" Device, or
1410 =Send= Device for short, since it sends audio data back to the
1411 calling application like an Aux-Send cable on a mixing board.
1413 The main idea behind the Send device is to take advantage of the
1414 fact that LWJGL only manages one /context/ when using OpenAL. A
1415 /context/ is like a container that holds samples and keeps track
1416 of where the listener is. In order to support multiple listeners,
1417 the Send device identifies the LWJGL context as the master
1418 context, and creates any number of slave contexts to represent
1419 additional listeners. Every time the device renders sound, it
1420 synchronizes every source from the master LWJGL context to the
1421 slave contexts. Then, it renders each context separately, using a
1422 different listener for each one. The rendered sound is made
1423 available via JNI to jMonkeyEngine.
1425 Switching between contexts is not the normal operation of a
1426 Device, and one of the problems with doing so is that a Device
1427 normally keeps around a few pieces of state such as the
1428 =ClickRemoval= array above which will become corrupted if the
1429 contexts are not rendered in parallel. The solution is to create a
1430 copy of this normally global device state for each context, and
1431 copy it back and forth into and out of the actual device state
1432 whenever a context is rendered.
1434 The core of the =Send= device is the =syncSources= function, which
1435 does the job of copying all relevant data from one context to
1436 another.
1438 #+caption: Program for extending =OpenAL= to support multiple
1439 #+caption: listeners via context copying/switching.
1440 #+name: sync-openal-sources
1441 #+begin_listing c
1442 #+BEGIN_SRC c
1443 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1444 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1445 ALuint master = masterSource->source;
1446 ALuint slave = slaveSource->source;
1447 ALCcontext *current = alcGetCurrentContext();
1449 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1450 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1451 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1452 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1453 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1454 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1455 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1456 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1457 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1458 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1459 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1460 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1461 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1463 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1464 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1465 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1467 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1468 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1470 alcMakeContextCurrent(masterCtx);
1471 ALint source_type;
1472 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1474 // Only static sources are currently synchronized!
1475 if (AL_STATIC == source_type){
1476 ALint master_buffer;
1477 ALint slave_buffer;
1478 alGetSourcei(master, AL_BUFFER, &master_buffer);
1479 alcMakeContextCurrent(slaveCtx);
1480 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1481 if (master_buffer != slave_buffer){
1482 alSourcei(slave, AL_BUFFER, master_buffer);
1486 // Synchronize the state of the two sources.
1487 alcMakeContextCurrent(masterCtx);
1488 ALint masterState;
1489 ALint slaveState;
1491 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1492 alcMakeContextCurrent(slaveCtx);
1493 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1495 if (masterState != slaveState){
1496 switch (masterState){
1497 case AL_INITIAL : alSourceRewind(slave); break;
1498 case AL_PLAYING : alSourcePlay(slave); break;
1499 case AL_PAUSED : alSourcePause(slave); break;
1500 case AL_STOPPED : alSourceStop(slave); break;
1503 // Restore whatever context was previously active.
1504 alcMakeContextCurrent(current);
1506 #+END_SRC
1507 #+end_listing
1509 With this special context-switching device, and some ugly JNI
1510 bindings that are not worth mentioning, =CORTEX= gains the ability
1511 to access multiple sound streams from =OpenAL=.
1513 #+caption: Program to create an ear from a blender empty node. The ear
1514 #+caption: follows around the nearest physical object and passes
1515 #+caption: all sensory data to a continuation function.
1516 #+name: add-ear
1517 #+begin_listing clojure
1518 #+BEGIN_SRC clojure
1519 (defn add-ear!
1520 "Create a Listener centered on the current position of 'ear
1521 which follows the closest physical node in 'creature and
1522 sends sound data to 'continuation."
1523 [#^Application world #^Node creature #^Spatial ear continuation]
1524 (let [target (closest-node creature ear)
1525 lis (Listener.)
1526 audio-renderer (.getAudioRenderer world)
1527 sp (hearing-pipeline continuation)]
1528 (.setLocation lis (.getWorldTranslation ear))
1529 (.setRotation lis (.getWorldRotation ear))
1530 (bind-sense target lis)
1531 (update-listener-velocity! target lis)
1532 (.addListener audio-renderer lis)
1533 (.registerSoundProcessor audio-renderer lis sp)))
1534 #+END_SRC
1535 #+end_listing
1537 The =Send= device, unlike most of the other devices in =OpenAL=,
1538 does not render sound unless asked. This enables the system to
1539 slow down or speed up depending on the needs of the AIs who are
1540 using it to listen. If the device tried to render samples in
1541 real-time, a complicated AI whose mind takes 100 seconds of
1542 computer time to simulate 1 second of AI-time would miss almost
1543 all of the sound in its environment!
1545 #+caption: Program to enable arbitrary hearing in =CORTEX=
1546 #+name: hearing
1547 #+begin_listing clojure
1548 #+BEGIN_SRC clojure
1549 (defn hearing-kernel
1550 "Returns a function which returns auditory sensory data when called
1551 inside a running simulation."
1552 [#^Node creature #^Spatial ear]
1553 (let [hearing-data (atom [])
1554 register-listener!
1555 (runonce
1556 (fn [#^Application world]
1557 (add-ear!
1558 world creature ear
1559 (comp #(reset! hearing-data %)
1560 byteBuffer->pulse-vector))))]
1561 (fn [#^Application world]
1562 (register-listener! world)
1563 (let [data @hearing-data
1564 topology
1565 (vec (map #(vector % 0) (range 0 (count data))))]
1566 [topology data]))))
1568 (defn hearing!
1569 "Endow the creature in a particular world with the sense of
1570 hearing. Will return a sequence of functions, one for each ear,
1571 which when called will return the auditory data from that ear."
1572 [#^Node creature]
1573 (for [ear (ears creature)]
1574 (hearing-kernel creature ear)))
1575 #+END_SRC
1576 #+end_listing
1578 Armed with these functions, =CORTEX= is able to test possibly the
1579 first ever instance of multiple listeners in a video game engine
1580 based simulation!
1582 #+caption: Here a simple creature responds to sound by changing
1583 #+caption: its color from gray to green when the total volume
1584 #+caption: goes over a threshold.
1585 #+name: sound-test
1586 #+begin_listing java
1587 #+BEGIN_SRC java
1588 /**
1589 * Respond to sound! This is the brain of an AI entity that
1590 * hears its surroundings and reacts to them.
1591 */
1592 public void process(ByteBuffer audioSamples,
1593 int numSamples, AudioFormat format) {
1594 audioSamples.clear();
1595 byte[] data = new byte[numSamples];
1596 float[] out = new float[numSamples];
1597 audioSamples.get(data);
1598 FloatSampleTools.
1599 byte2floatInterleaved
1600 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1602 float max = Float.NEGATIVE_INFINITY;
1603 for (float f : out){if (f > max) max = f;}
1604 audioSamples.clear();
1606 if (max > 0.1){
1607 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1609 else {
1610 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1612 #+END_SRC
1613 #+end_listing
1615 #+caption: First ever simulation of multiple listeners in =CORTEX=.
1616 #+caption: Each cube is a creature which processes sound data with
1617 #+caption: the =process= function from listing \ref{sound-test}.
1618 #+caption: the ball is constantly emitting a pure tone of
1619 #+caption: constant volume. As it approaches the cubes, they each
1620 #+caption: change color in response to the sound.
1621 #+name: sound-cubes.
1622 #+ATTR_LaTeX: :width 10cm
1623 [[./images/java-hearing-test.png]]
1625 This system of hearing has also been co-opted by the
1626 jMonkeyEngine3 community and is used to record audio for demo
1627 videos.
1629 ** Hundreds of hair-like elements provide a sense of touch
1631 Touch is critical to navigation and spatial reasoning and as such I
1632 need a simulated version of it to give to my AI creatures.
1634 Human skin has a wide array of touch sensors, each of which
1635 specialize in detecting different vibrational modes and pressures.
1636 These sensors can integrate a vast expanse of skin (i.e. your
1637 entire palm), or a tiny patch of skin at the tip of your finger.
1638 The hairs of the skin help detect objects before they even come
1639 into contact with the skin proper.
1641 However, touch in my simulated world can not exactly correspond to
1642 human touch because my creatures are made out of completely rigid
1643 segments that don't deform like human skin.
1645 Instead of measuring deformation or vibration, I surround each
1646 rigid part with a plenitude of hair-like objects (/feelers/) which
1647 do not interact with the physical world. Physical objects can pass
1648 through them with no effect. The feelers are able to tell when
1649 other objects pass through them, and they constantly report how
1650 much of their extent is covered. So even though the creature's body
1651 parts do not deform, the feelers create a margin around those body
1652 parts which achieves a sense of touch which is a hybrid between a
1653 human's sense of deformation and sense from hairs.
1655 Implementing touch in jMonkeyEngine follows a different technical
1656 route than vision and hearing. Those two senses piggybacked off
1657 jMonkeyEngine's 3D audio and video rendering subsystems. To
1658 simulate touch, I use jMonkeyEngine's physics system to execute
1659 many small collision detections, one for each feeler. The placement
1660 of the feelers is determined by a UV-mapped image which shows where
1661 each feeler should be on the 3D surface of the body.
1663 *** Defining Touch Meta-Data in Blender
1665 Each geometry can have a single UV map which describes the
1666 position of the feelers which will constitute its sense of touch.
1667 This image path is stored under the ``touch'' key. The image itself
1668 is black and white, with black meaning a feeler length of 0 (no
1669 feeler is present) and white meaning a feeler length of =scale=,
1670 which is a float stored under the key "scale".
1672 #+caption: Touch does not use empty nodes, to store metadata,
1673 #+caption: because the metadata of each solid part of a
1674 #+caption: creature's body is sufficient.
1675 #+name: touch-meta-data
1676 #+begin_listing clojure
1677 #+BEGIN_SRC clojure
1678 (defn tactile-sensor-profile
1679 "Return the touch-sensor distribution image in BufferedImage format,
1680 or nil if it does not exist."
1681 [#^Geometry obj]
1682 (if-let [image-path (meta-data obj "touch")]
1683 (load-image image-path)))
1685 (defn tactile-scale
1686 "Return the length of each feeler. Default scale is 0.01
1687 jMonkeyEngine units."
1688 [#^Geometry obj]
1689 (if-let [scale (meta-data obj "scale")]
1690 scale 0.1))
1691 #+END_SRC
1692 #+end_listing
1694 Here is an example of a UV-map which specifies the position of
1695 touch sensors along the surface of the upper segment of a fingertip.
1697 #+caption: This is the tactile-sensor-profile for the upper segment
1698 #+caption: of a fingertip. It defines regions of high touch sensitivity
1699 #+caption: (where there are many white pixels) and regions of low
1700 #+caption: sensitivity (where white pixels are sparse).
1701 #+name: fingertip-UV
1702 #+ATTR_LaTeX: :width 13cm
1703 [[./images/finger-UV.png]]
1705 *** Implementation Summary
1707 To simulate touch there are three conceptual steps. For each solid
1708 object in the creature, you first have to get UV image and scale
1709 parameter which define the position and length of the feelers.
1710 Then, you use the triangles which comprise the mesh and the UV
1711 data stored in the mesh to determine the world-space position and
1712 orientation of each feeler. Then once every frame, update these
1713 positions and orientations to match the current position and
1714 orientation of the object, and use physics collision detection to
1715 gather tactile data.
1717 Extracting the meta-data has already been described. The third
1718 step, physics collision detection, is handled in =touch-kernel=.
1719 Translating the positions and orientations of the feelers from the
1720 UV-map to world-space is itself a three-step process.
1722 - Find the triangles which make up the mesh in pixel-space and in
1723 world-space. \\(=triangles=, =pixel-triangles=).
1725 - Find the coordinates of each feeler in world-space. These are
1726 the origins of the feelers. (=feeler-origins=).
1728 - Calculate the normals of the triangles in world space, and add
1729 them to each of the origins of the feelers. These are the
1730 normalized coordinates of the tips of the feelers.
1731 (=feeler-tips=).
1733 *** Triangle Math
1735 The rigid objects which make up a creature have an underlying
1736 =Geometry=, which is a =Mesh= plus a =Material= and other
1737 important data involved with displaying the object.
1739 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1740 vertices which have coordinates in world space and UV space.
1742 Here, =triangles= gets all the world-space triangles which
1743 comprise a mesh, while =pixel-triangles= gets those same triangles
1744 expressed in pixel coordinates (which are UV coordinates scaled to
1745 fit the height and width of the UV image).
1747 #+caption: Programs to extract triangles from a geometry and get
1748 #+caption: their vertices in both world and UV-coordinates.
1749 #+name: get-triangles
1750 #+begin_listing clojure
1751 #+BEGIN_SRC clojure
1752 (defn triangle
1753 "Get the triangle specified by triangle-index from the mesh."
1754 [#^Geometry geo triangle-index]
1755 (triangle-seq
1756 (let [scratch (Triangle.)]
1757 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1759 (defn triangles
1760 "Return a sequence of all the Triangles which comprise a given
1761 Geometry."
1762 [#^Geometry geo]
1763 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1765 (defn triangle-vertex-indices
1766 "Get the triangle vertex indices of a given triangle from a given
1767 mesh."
1768 [#^Mesh mesh triangle-index]
1769 (let [indices (int-array 3)]
1770 (.getTriangle mesh triangle-index indices)
1771 (vec indices)))
1773 (defn vertex-UV-coord
1774 "Get the UV-coordinates of the vertex named by vertex-index"
1775 [#^Mesh mesh vertex-index]
1776 (let [UV-buffer
1777 (.getData
1778 (.getBuffer
1779 mesh
1780 VertexBuffer$Type/TexCoord))]
1781 [(.get UV-buffer (* vertex-index 2))
1782 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1784 (defn pixel-triangle [#^Geometry geo image index]
1785 (let [mesh (.getMesh geo)
1786 width (.getWidth image)
1787 height (.getHeight image)]
1788 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1789 (map (partial vertex-UV-coord mesh)
1790 (triangle-vertex-indices mesh index))))))
1792 (defn pixel-triangles
1793 "The pixel-space triangles of the Geometry, in the same order as
1794 (triangles geo)"
1795 [#^Geometry geo image]
1796 (let [height (.getHeight image)
1797 width (.getWidth image)]
1798 (map (partial pixel-triangle geo image)
1799 (range (.getTriangleCount (.getMesh geo))))))
1800 #+END_SRC
1801 #+end_listing
1803 *** The Affine Transform from one Triangle to Another
1805 =pixel-triangles= gives us the mesh triangles expressed in pixel
1806 coordinates and =triangles= gives us the mesh triangles expressed
1807 in world coordinates. The tactile-sensor-profile gives the
1808 position of each feeler in pixel-space. In order to convert
1809 pixel-space coordinates into world-space coordinates we need
1810 something that takes coordinates on the surface of one triangle
1811 and gives the corresponding coordinates on the surface of another
1812 triangle.
1814 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1815 into any other by a combination of translation, scaling, and
1816 rotation. The affine transformation from one triangle to another
1817 is readily computable if the triangle is expressed in terms of a
1818 $4x4$ matrix.
1820 #+BEGIN_LaTeX
1821 $$
1822 \begin{bmatrix}
1823 x_1 & x_2 & x_3 & n_x \\
1824 y_1 & y_2 & y_3 & n_y \\
1825 z_1 & z_2 & z_3 & n_z \\
1826 1 & 1 & 1 & 1
1827 \end{bmatrix}
1828 $$
1829 #+END_LaTeX
1831 Here, the first three columns of the matrix are the vertices of
1832 the triangle. The last column is the right-handed unit normal of
1833 the triangle.
1835 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1836 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1837 is $T_{2}T_{1}^{-1}$.
1839 The clojure code below recapitulates the formulas above, using
1840 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1841 transformation.
1843 #+caption: Program to interpret triangles as affine transforms.
1844 #+name: triangle-affine
1845 #+begin_listing clojure
1846 #+BEGIN_SRC clojure
1847 (defn triangle->matrix4f
1848 "Converts the triangle into a 4x4 matrix: The first three columns
1849 contain the vertices of the triangle; the last contains the unit
1850 normal of the triangle. The bottom row is filled with 1s."
1851 [#^Triangle t]
1852 (let [mat (Matrix4f.)
1853 [vert-1 vert-2 vert-3]
1854 (mapv #(.get t %) (range 3))
1855 unit-normal (do (.calculateNormal t)(.getNormal t))
1856 vertices [vert-1 vert-2 vert-3 unit-normal]]
1857 (dorun
1858 (for [row (range 4) col (range 3)]
1859 (do
1860 (.set mat col row (.get (vertices row) col))
1861 (.set mat 3 row 1)))) mat))
1863 (defn triangles->affine-transform
1864 "Returns the affine transformation that converts each vertex in the
1865 first triangle into the corresponding vertex in the second
1866 triangle."
1867 [#^Triangle tri-1 #^Triangle tri-2]
1868 (.mult
1869 (triangle->matrix4f tri-2)
1870 (.invert (triangle->matrix4f tri-1))))
1871 #+END_SRC
1872 #+end_listing
1874 *** Triangle Boundaries
1876 For efficiency's sake I will divide the tactile-profile image into
1877 small squares which inscribe each pixel-triangle, then extract the
1878 points which lie inside the triangle and map them to 3D-space using
1879 =triangle-transform= above. To do this I need a function,
1880 =convex-bounds= which finds the smallest box which inscribes a 2D
1881 triangle.
1883 =inside-triangle?= determines whether a point is inside a triangle
1884 in 2D pixel-space.
1886 #+caption: Program to efficiently determine point inclusion
1887 #+caption: in a triangle.
1888 #+name: in-triangle
1889 #+begin_listing clojure
1890 #+BEGIN_SRC clojure
1891 (defn convex-bounds
1892 "Returns the smallest square containing the given vertices, as a
1893 vector of integers [left top width height]."
1894 [verts]
1895 (let [xs (map first verts)
1896 ys (map second verts)
1897 x0 (Math/floor (apply min xs))
1898 y0 (Math/floor (apply min ys))
1899 x1 (Math/ceil (apply max xs))
1900 y1 (Math/ceil (apply max ys))]
1901 [x0 y0 (- x1 x0) (- y1 y0)]))
1903 (defn same-side?
1904 "Given the points p1 and p2 and the reference point ref, is point p
1905 on the same side of the line that goes through p1 and p2 as ref is?"
1906 [p1 p2 ref p]
1907 (<=
1909 (.dot
1910 (.cross (.subtract p2 p1) (.subtract p p1))
1911 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1913 (defn inside-triangle?
1914 "Is the point inside the triangle?"
1915 {:author "Dylan Holmes"}
1916 [#^Triangle tri #^Vector3f p]
1917 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1918 (and
1919 (same-side? vert-1 vert-2 vert-3 p)
1920 (same-side? vert-2 vert-3 vert-1 p)
1921 (same-side? vert-3 vert-1 vert-2 p))))
1922 #+END_SRC
1923 #+end_listing
1925 *** Feeler Coordinates
1927 The triangle-related functions above make short work of
1928 calculating the positions and orientations of each feeler in
1929 world-space.
1931 #+caption: Program to get the coordinates of ``feelers '' in
1932 #+caption: both world and UV-coordinates.
1933 #+name: feeler-coordinates
1934 #+begin_listing clojure
1935 #+BEGIN_SRC clojure
1936 (defn feeler-pixel-coords
1937 "Returns the coordinates of the feelers in pixel space in lists, one
1938 list for each triangle, ordered in the same way as (triangles) and
1939 (pixel-triangles)."
1940 [#^Geometry geo image]
1941 (map
1942 (fn [pixel-triangle]
1943 (filter
1944 (fn [coord]
1945 (inside-triangle? (->triangle pixel-triangle)
1946 (->vector3f coord)))
1947 (white-coordinates image (convex-bounds pixel-triangle))))
1948 (pixel-triangles geo image)))
1950 (defn feeler-world-coords
1951 "Returns the coordinates of the feelers in world space in lists, one
1952 list for each triangle, ordered in the same way as (triangles) and
1953 (pixel-triangles)."
1954 [#^Geometry geo image]
1955 (let [transforms
1956 (map #(triangles->affine-transform
1957 (->triangle %1) (->triangle %2))
1958 (pixel-triangles geo image)
1959 (triangles geo))]
1960 (map (fn [transform coords]
1961 (map #(.mult transform (->vector3f %)) coords))
1962 transforms (feeler-pixel-coords geo image))))
1963 #+END_SRC
1964 #+end_listing
1966 #+caption: Program to get the position of the base and tip of
1967 #+caption: each ``feeler''
1968 #+name: feeler-tips
1969 #+begin_listing clojure
1970 #+BEGIN_SRC clojure
1971 (defn feeler-origins
1972 "The world space coordinates of the root of each feeler."
1973 [#^Geometry geo image]
1974 (reduce concat (feeler-world-coords geo image)))
1976 (defn feeler-tips
1977 "The world space coordinates of the tip of each feeler."
1978 [#^Geometry geo image]
1979 (let [world-coords (feeler-world-coords geo image)
1980 normals
1981 (map
1982 (fn [triangle]
1983 (.calculateNormal triangle)
1984 (.clone (.getNormal triangle)))
1985 (map ->triangle (triangles geo)))]
1987 (mapcat (fn [origins normal]
1988 (map #(.add % normal) origins))
1989 world-coords normals)))
1991 (defn touch-topology
1992 [#^Geometry geo image]
1993 (collapse (reduce concat (feeler-pixel-coords geo image))))
1994 #+END_SRC
1995 #+end_listing
1997 *** Simulated Touch
1999 Now that the functions to construct feelers are complete,
2000 =touch-kernel= generates functions to be called from within a
2001 simulation that perform the necessary physics collisions to
2002 collect tactile data, and =touch!= recursively applies it to every
2003 node in the creature.
2005 #+caption: Efficient program to transform a ray from
2006 #+caption: one position to another.
2007 #+name: set-ray
2008 #+begin_listing clojure
2009 #+BEGIN_SRC clojure
2010 (defn set-ray [#^Ray ray #^Matrix4f transform
2011 #^Vector3f origin #^Vector3f tip]
2012 ;; Doing everything locally reduces garbage collection by enough to
2013 ;; be worth it.
2014 (.mult transform origin (.getOrigin ray))
2015 (.mult transform tip (.getDirection ray))
2016 (.subtractLocal (.getDirection ray) (.getOrigin ray))
2017 (.normalizeLocal (.getDirection ray)))
2018 #+END_SRC
2019 #+end_listing
2021 #+caption: This is the core of touch in =CORTEX= each feeler
2022 #+caption: follows the object it is bound to, reporting any
2023 #+caption: collisions that may happen.
2024 #+name: touch-kernel
2025 #+begin_listing clojure
2026 #+BEGIN_SRC clojure
2027 (defn touch-kernel
2028 "Constructs a function which will return tactile sensory data from
2029 'geo when called from inside a running simulation"
2030 [#^Geometry geo]
2031 (if-let
2032 [profile (tactile-sensor-profile geo)]
2033 (let [ray-reference-origins (feeler-origins geo profile)
2034 ray-reference-tips (feeler-tips geo profile)
2035 ray-length (tactile-scale geo)
2036 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
2037 topology (touch-topology geo profile)
2038 correction (float (* ray-length -0.2))]
2039 ;; slight tolerance for very close collisions.
2040 (dorun
2041 (map (fn [origin tip]
2042 (.addLocal origin (.mult (.subtract tip origin)
2043 correction)))
2044 ray-reference-origins ray-reference-tips))
2045 (dorun (map #(.setLimit % ray-length) current-rays))
2046 (fn [node]
2047 (let [transform (.getWorldMatrix geo)]
2048 (dorun
2049 (map (fn [ray ref-origin ref-tip]
2050 (set-ray ray transform ref-origin ref-tip))
2051 current-rays ray-reference-origins
2052 ray-reference-tips))
2053 (vector
2054 topology
2055 (vec
2056 (for [ray current-rays]
2057 (do
2058 (let [results (CollisionResults.)]
2059 (.collideWith node ray results)
2060 (let [touch-objects
2061 (filter #(not (= geo (.getGeometry %)))
2062 results)
2063 limit (.getLimit ray)]
2064 [(if (empty? touch-objects)
2065 limit
2066 (let [response
2067 (apply min (map #(.getDistance %)
2068 touch-objects))]
2069 (FastMath/clamp
2070 (float
2071 (if (> response limit) (float 0.0)
2072 (+ response correction)))
2073 (float 0.0)
2074 limit)))
2075 limit])))))))))))
2076 #+END_SRC
2077 #+end_listing
2079 Armed with the =touch!= function, =CORTEX= becomes capable of
2080 giving creatures a sense of touch. A simple test is to create a
2081 cube that is outfitted with a uniform distribution of touch
2082 sensors. It can feel the ground and any balls that it touches.
2084 #+caption: =CORTEX= interface for creating touch in a simulated
2085 #+caption: creature.
2086 #+name: touch
2087 #+begin_listing clojure
2088 #+BEGIN_SRC clojure
2089 (defn touch!
2090 "Endow the creature with the sense of touch. Returns a sequence of
2091 functions, one for each body part with a tactile-sensor-profile,
2092 each of which when called returns sensory data for that body part."
2093 [#^Node creature]
2094 (filter
2095 (comp not nil?)
2096 (map touch-kernel
2097 (filter #(isa? (class %) Geometry)
2098 (node-seq creature)))))
2099 #+END_SRC
2100 #+end_listing
2102 The tactile-sensor-profile image for the touch cube is a simple
2103 cross with a uniform distribution of touch sensors:
2105 #+caption: The touch profile for the touch-cube. Each pure white
2106 #+caption: pixel defines a touch sensitive feeler.
2107 #+name: touch-cube-uv-map
2108 #+ATTR_LaTeX: :width 7cm
2109 [[./images/touch-profile.png]]
2111 #+caption: The touch cube reacts to cannonballs. The black, red,
2112 #+caption: and white cross on the right is a visual display of
2113 #+caption: the creature's touch. White means that it is feeling
2114 #+caption: something strongly, black is not feeling anything,
2115 #+caption: and gray is in-between. The cube can feel both the
2116 #+caption: floor and the ball. Notice that when the ball causes
2117 #+caption: the cube to tip, that the bottom face can still feel
2118 #+caption: part of the ground.
2119 #+name: touch-cube-uv-map-2
2120 #+ATTR_LaTeX: :width 15cm
2121 [[./images/touch-cube.png]]
2123 ** Proprioception provides knowledge of your own body's position
2125 Close your eyes, and touch your nose with your right index finger.
2126 How did you do it? You could not see your hand, and neither your
2127 hand nor your nose could use the sense of touch to guide the path
2128 of your hand. There are no sound cues, and Taste and Smell
2129 certainly don't provide any help. You know where your hand is
2130 without your other senses because of Proprioception.
2132 Humans can sometimes loose this sense through viral infections or
2133 damage to the spinal cord or brain, and when they do, they loose
2134 the ability to control their own bodies without looking directly at
2135 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2136 Hat]] (\cite{man-wife-hat}), a woman named Christina looses this
2137 sense and has to learn how to move by carefully watching her arms
2138 and legs. She describes proprioception as the "eyes of the body,
2139 the way the body sees itself".
2141 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2142 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2143 positions of each body part by monitoring muscle strain and length.
2145 It's clear that this is a vital sense for fluid, graceful movement.
2146 It's also particularly easy to implement in jMonkeyEngine.
2148 My simulated proprioception calculates the relative angles of each
2149 joint from the rest position defined in the blender file. This
2150 simulates the muscle-spindles and joint capsules. I will deal with
2151 Golgi tendon organs, which calculate muscle strain, in the next
2152 chapter.
2154 *** Helper functions
2156 =absolute-angle= calculates the angle between two vectors,
2157 relative to a third axis vector. This angle is the number of
2158 radians you have to move counterclockwise around the axis vector
2159 to get from the first to the second vector. It is not commutative
2160 like a normal dot-product angle is.
2162 The purpose of these functions is to build a system of angle
2163 measurement that is biologically plausible.
2165 #+caption: Program to measure angles along a vector
2166 #+name: helpers
2167 #+begin_listing clojure
2168 #+BEGIN_SRC clojure
2169 (defn right-handed?
2170 "true iff the three vectors form a right handed coordinate
2171 system. The three vectors do not have to be normalized or
2172 orthogonal."
2173 [vec1 vec2 vec3]
2174 (pos? (.dot (.cross vec1 vec2) vec3)))
2176 (defn absolute-angle
2177 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2178 [0 (* 2 Math/PI)]."
2179 [vec1 vec2 axis]
2180 (let [angle (.angleBetween vec1 vec2)]
2181 (if (right-handed? vec1 vec2 axis)
2182 angle (- (* 2 Math/PI) angle))))
2183 #+END_SRC
2184 #+end_listing
2186 *** Proprioception Kernel
2188 Given a joint, =proprioception-kernel= produces a function that
2189 calculates the Euler angles between the objects the joint
2190 connects. The only tricky part here is making the angles relative
2191 to the joint's initial ``straightness''.
2193 #+caption: Program to return biologically reasonable proprioceptive
2194 #+caption: data for each joint.
2195 #+name: proprioception
2196 #+begin_listing clojure
2197 #+BEGIN_SRC clojure
2198 (defn proprioception-kernel
2199 "Returns a function which returns proprioceptive sensory data when
2200 called inside a running simulation."
2201 [#^Node parts #^Node joint]
2202 (let [[obj-a obj-b] (joint-targets parts joint)
2203 joint-rot (.getWorldRotation joint)
2204 x0 (.mult joint-rot Vector3f/UNIT_X)
2205 y0 (.mult joint-rot Vector3f/UNIT_Y)
2206 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2207 (fn []
2208 (let [rot-a (.clone (.getWorldRotation obj-a))
2209 rot-b (.clone (.getWorldRotation obj-b))
2210 x (.mult rot-a x0)
2211 y (.mult rot-a y0)
2212 z (.mult rot-a z0)
2214 X (.mult rot-b x0)
2215 Y (.mult rot-b y0)
2216 Z (.mult rot-b z0)
2217 heading (Math/atan2 (.dot X z) (.dot X x))
2218 pitch (Math/atan2 (.dot X y) (.dot X x))
2220 ;; rotate x-vector back to origin
2221 reverse
2222 (doto (Quaternion.)
2223 (.fromAngleAxis
2224 (.angleBetween X x)
2225 (let [cross (.normalize (.cross X x))]
2226 (if (= 0 (.length cross)) y cross))))
2227 roll (absolute-angle (.mult reverse Y) y x)]
2228 [heading pitch roll]))))
2230 (defn proprioception!
2231 "Endow the creature with the sense of proprioception. Returns a
2232 sequence of functions, one for each child of the \"joints\" node in
2233 the creature, which each report proprioceptive information about
2234 that joint."
2235 [#^Node creature]
2236 ;; extract the body's joints
2237 (let [senses (map (partial proprioception-kernel creature)
2238 (joints creature))]
2239 (fn []
2240 (map #(%) senses))))
2241 #+END_SRC
2242 #+end_listing
2244 =proprioception!= maps =proprioception-kernel= across all the
2245 joints of the creature. It uses the same list of joints that
2246 =joints= uses. Proprioception is the easiest sense to implement in
2247 =CORTEX=, and it will play a crucial role when efficiently
2248 implementing empathy.
2250 #+caption: In the upper right corner, the three proprioceptive
2251 #+caption: angle measurements are displayed. Red is yaw, Green is
2252 #+caption: pitch, and White is roll.
2253 #+name: proprio
2254 #+ATTR_LaTeX: :width 11cm
2255 [[./images/proprio.png]]
2257 ** Muscles contain both sensors and effectors
2259 Surprisingly enough, terrestrial creatures only move by using
2260 torque applied about their joints. There's not a single straight
2261 line of force in the human body at all! (A straight line of force
2262 would correspond to some sort of jet or rocket propulsion.)
2264 In humans, muscles are composed of muscle fibers which can contract
2265 to exert force. The muscle fibers which compose a muscle are
2266 partitioned into discrete groups which are each controlled by a
2267 single alpha motor neuron. A single alpha motor neuron might
2268 control as little as three or as many as one thousand muscle
2269 fibers. When the alpha motor neuron is engaged by the spinal cord,
2270 it activates all of the muscle fibers to which it is attached. The
2271 spinal cord generally engages the alpha motor neurons which control
2272 few muscle fibers before the motor neurons which control many
2273 muscle fibers. This recruitment strategy allows for precise
2274 movements at low strength. The collection of all motor neurons that
2275 control a muscle is called the motor pool. The brain essentially
2276 says "activate 30% of the motor pool" and the spinal cord recruits
2277 motor neurons until 30% are activated. Since the distribution of
2278 power among motor neurons is unequal and recruitment goes from
2279 weakest to strongest, the first 30% of the motor pool might be 5%
2280 of the strength of the muscle.
2282 My simulated muscles follow a similar design: Each muscle is
2283 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2284 the array represents a motor neuron which controls a number of
2285 muscle fibers equal to the value of the entry. Each muscle has a
2286 scalar strength factor which determines the total force the muscle
2287 can exert when all motor neurons are activated. The effector
2288 function for a muscle takes a number to index into the motor pool,
2289 and then "activates" all the motor neurons whose index is lower or
2290 equal to the number. Each motor-neuron will apply force in
2291 proportion to its value in the array. Lower values cause less
2292 force. The lower values can be put at the "beginning" of the 1-D
2293 array to simulate the layout of actual human muscles, which are
2294 capable of more precise movements when exerting less force. Or, the
2295 motor pool can simulate more exotic recruitment strategies which do
2296 not correspond to human muscles.
2298 This 1D array is defined in an image file for ease of
2299 creation/visualization. Here is an example muscle profile image.
2301 #+caption: A muscle profile image that describes the strengths
2302 #+caption: of each motor neuron in a muscle. White is weakest
2303 #+caption: and dark red is strongest. This particular pattern
2304 #+caption: has weaker motor neurons at the beginning, just
2305 #+caption: like human muscle.
2306 #+name: muscle-recruit
2307 #+ATTR_LaTeX: :width 7cm
2308 [[./images/basic-muscle.png]]
2310 *** Muscle meta-data
2312 #+caption: Program to deal with loading muscle data from a blender
2313 #+caption: file's metadata.
2314 #+name: motor-pool
2315 #+begin_listing clojure
2316 #+BEGIN_SRC clojure
2317 (defn muscle-profile-image
2318 "Get the muscle-profile image from the node's blender meta-data."
2319 [#^Node muscle]
2320 (if-let [image (meta-data muscle "muscle")]
2321 (load-image image)))
2323 (defn muscle-strength
2324 "Return the strength of this muscle, or 1 if it is not defined."
2325 [#^Node muscle]
2326 (if-let [strength (meta-data muscle "strength")]
2327 strength 1))
2329 (defn motor-pool
2330 "Return a vector where each entry is the strength of the \"motor
2331 neuron\" at that part in the muscle."
2332 [#^Node muscle]
2333 (let [profile (muscle-profile-image muscle)]
2334 (vec
2335 (let [width (.getWidth profile)]
2336 (for [x (range width)]
2337 (- 255
2338 (bit-and
2339 0x0000FF
2340 (.getRGB profile x 0))))))))
2341 #+END_SRC
2342 #+end_listing
2344 Of note here is =motor-pool= which interprets the muscle-profile
2345 image in a way that allows me to use gradients between white and
2346 red, instead of shades of gray as I've been using for all the
2347 other senses. This is purely an aesthetic touch.
2349 *** Creating muscles
2351 #+caption: This is the core movement function in =CORTEX=, which
2352 #+caption: implements muscles that report on their activation.
2353 #+name: muscle-kernel
2354 #+begin_listing clojure
2355 #+BEGIN_SRC clojure
2356 (defn movement-kernel
2357 "Returns a function which when called with a integer value inside a
2358 running simulation will cause movement in the creature according
2359 to the muscle's position and strength profile. Each function
2360 returns the amount of force applied / max force."
2361 [#^Node creature #^Node muscle]
2362 (let [target (closest-node creature muscle)
2363 axis
2364 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2365 strength (muscle-strength muscle)
2367 pool (motor-pool muscle)
2368 pool-integral (reductions + pool)
2369 forces
2370 (vec (map #(float (* strength (/ % (last pool-integral))))
2371 pool-integral))
2372 control (.getControl target RigidBodyControl)]
2373 (fn [n]
2374 (let [pool-index (max 0 (min n (dec (count pool))))
2375 force (forces pool-index)]
2376 (.applyTorque control (.mult axis force))
2377 (float (/ force strength))))))
2379 (defn movement!
2380 "Endow the creature with the power of movement. Returns a sequence
2381 of functions, each of which accept an integer value and will
2382 activate their corresponding muscle."
2383 [#^Node creature]
2384 (for [muscle (muscles creature)]
2385 (movement-kernel creature muscle)))
2386 #+END_SRC
2387 #+end_listing
2390 =movement-kernel= creates a function that controls the movement
2391 of the nearest physical node to the muscle node. The muscle exerts
2392 a rotational force dependent on it's orientation to the object in
2393 the blender file. The function returned by =movement-kernel= is
2394 also a sense function: it returns the percent of the total muscle
2395 strength that is currently being employed. This is analogous to
2396 muscle tension in humans and completes the sense of proprioception
2397 begun in the last chapter.
2399 ** =CORTEX= brings complex creatures to life!
2401 The ultimate test of =CORTEX= is to create a creature with the full
2402 gamut of senses and put it though its paces.
2404 With all senses enabled, my right hand model looks like an
2405 intricate marionette hand with several strings for each finger:
2407 #+caption: View of the hand model with all sense nodes. You can see
2408 #+caption: the joint, muscle, ear, and eye nodes here.
2409 #+name: hand-nodes-1
2410 #+ATTR_LaTeX: :width 11cm
2411 [[./images/hand-with-all-senses2.png]]
2413 #+caption: An alternate view of the hand.
2414 #+name: hand-nodes-2
2415 #+ATTR_LaTeX: :width 15cm
2416 [[./images/hand-with-all-senses3.png]]
2418 With the hand fully rigged with senses, I can run it though a test
2419 that will test everything.
2421 #+caption: Selected frames from a full test of the hand with all
2422 #+caption: senses. Note especially the interactions the hand has
2423 #+caption: with itself: it feels its own palm and fingers, and when
2424 #+caption: it curls its fingers, it sees them with its eye (which
2425 #+caption: is located in the center of the palm. The red block
2426 #+caption: appears with a pure tone sound. The hand then uses its
2427 #+caption: muscles to launch the cube!
2428 #+name: integration
2429 #+ATTR_LaTeX: :width 15cm
2430 [[./images/integration.png]]
2432 ** =CORTEX= enables many possibilities for further research
2434 Often times, the hardest part of building a system involving
2435 creatures is dealing with physics and graphics. =CORTEX= removes
2436 much of this initial difficulty and leaves researchers free to
2437 directly pursue their ideas. I hope that even undergrads with a
2438 passing curiosity about simulated touch or creature evolution will
2439 be able to use cortex for experimentation. =CORTEX= is a completely
2440 simulated world, and far from being a disadvantage, its simulated
2441 nature enables you to create senses and creatures that would be
2442 impossible to make in the real world.
2444 While not by any means a complete list, here are some paths
2445 =CORTEX= is well suited to help you explore:
2447 - Empathy :: my empathy program leaves many areas for
2448 improvement, among which are using vision to infer
2449 proprioception and looking up sensory experience with imagined
2450 vision, touch, and sound.
2451 - Evolution :: Karl Sims created a rich environment for simulating
2452 the evolution of creatures on a Connection Machine
2453 (\cite{sims-evolving-creatures}). Today, this can be redone
2454 and expanded with =CORTEX= on an ordinary computer.
2455 - Exotic senses :: Cortex enables many fascinating senses that are
2456 not possible to build in the real world. For example,
2457 telekinesis is an interesting avenue to explore. You can also
2458 make a ``semantic'' sense which looks up metadata tags on
2459 objects in the environment the metadata tags might contain
2460 other sensory information.
2461 - Imagination via subworlds :: this would involve a creature with
2462 an effector which creates an entire new sub-simulation where
2463 the creature has direct control over placement/creation of
2464 objects via simulated telekinesis. The creature observes this
2465 sub-world through its normal senses and uses its observations
2466 to make predictions about its top level world.
2467 - Simulated prescience :: step the simulation forward a few ticks,
2468 gather sensory data, then supply this data for the creature as
2469 one of its actual senses. The cost of prescience is slowing
2470 the simulation down by a factor proportional to however far
2471 you want the entities to see into the future. What happens
2472 when two evolved creatures that can each see into the future
2473 fight each other?
2474 - Swarm creatures :: Program a group of creatures that cooperate
2475 with each other. Because the creatures would be simulated, you
2476 could investigate computationally complex rules of behavior
2477 which still, from the group's point of view, would happen in
2478 real time. Interactions could be as simple as cellular
2479 organisms communicating via flashing lights, or as complex as
2480 humanoids completing social tasks, etc.
2481 - =HACKER= for writing muscle-control programs :: Presented with a
2482 low-level muscle control / sense API, generate higher level
2483 programs for accomplishing various stated goals. Example goals
2484 might be "extend all your fingers" or "move your hand into the
2485 area with blue light" or "decrease the angle of this joint".
2486 It would be like Sussman's HACKER, except it would operate
2487 with much more data in a more realistic world. Start off with
2488 "calisthenics" to develop subroutines over the motor control
2489 API. The low level programming code might be a turning machine
2490 that could develop programs to iterate over a "tape" where
2491 each entry in the tape could control recruitment of the fibers
2492 in a muscle.
2493 - Sense fusion :: There is much work to be done on sense
2494 integration -- building up a coherent picture of the world and
2495 the things in it. With =CORTEX= as a base, you can explore
2496 concepts like self-organizing maps or cross modal clustering
2497 in ways that have never before been tried.
2498 - Inverse kinematics :: experiments in sense guided motor control
2499 are easy given =CORTEX='s support -- you can get right to the
2500 hard control problems without worrying about physics or
2501 senses.
2503 \newpage
2505 * COMMENT =EMPATH=: action recognition in a simulated worm
2507 Here I develop a computational model of empathy, using =CORTEX= as a
2508 base. Empathy in this context is the ability to observe another
2509 creature and infer what sorts of sensations that creature is
2510 feeling. My empathy algorithm involves multiple phases. First is
2511 free-play, where the creature moves around and gains sensory
2512 experience. From this experience I construct a representation of the
2513 creature's sensory state space, which I call \Phi-space. Using
2514 \Phi-space, I construct an efficient function which takes the
2515 limited data that comes from observing another creature and enriches
2516 it with a full compliment of imagined sensory data. I can then use
2517 the imagined sensory data to recognize what the observed creature is
2518 doing and feeling, using straightforward embodied action predicates.
2519 This is all demonstrated with using a simple worm-like creature, and
2520 recognizing worm-actions based on limited data.
2522 #+caption: Here is the worm with which we will be working. It is
2523 #+caption: composed of 5 segments. Each segment has a pair of
2524 #+caption: extensor and flexor muscles. Each of the worm's four
2525 #+caption: joints is a hinge joint which allows about 30 degrees of
2526 #+caption: rotation to either side. Each segment of the worm is
2527 #+caption: touch-capable and has a uniform distribution of touch
2528 #+caption: sensors on each of its faces. Each joint has a
2529 #+caption: proprioceptive sense to detect relative positions. The
2530 #+caption: worm segments are all the same except for the first one,
2531 #+caption: which has a much higher weight than the others to allow
2532 #+caption: for easy manual motor control.
2533 #+name: basic-worm-view
2534 #+ATTR_LaTeX: :width 10cm
2535 [[./images/basic-worm-view.png]]
2537 #+caption: Program for reading a worm from a blender file and
2538 #+caption: outfitting it with the senses of proprioception,
2539 #+caption: touch, and the ability to move, as specified in the
2540 #+caption: blender file.
2541 #+name: get-worm
2542 #+begin_listing clojure
2543 #+begin_src clojure
2544 (defn worm []
2545 (let [model (load-blender-model "Models/worm/worm.blend")]
2546 {:body (doto model (body!))
2547 :touch (touch! model)
2548 :proprioception (proprioception! model)
2549 :muscles (movement! model)}))
2550 #+end_src
2551 #+end_listing
2553 ** Embodiment factors action recognition into manageable parts
2555 Using empathy, I divide the problem of action recognition into a
2556 recognition process expressed in the language of a full compliment
2557 of senses, and an imaginative process that generates full sensory
2558 data from partial sensory data. Splitting the action recognition
2559 problem in this manner greatly reduces the total amount of work to
2560 recognize actions: The imaginative process is mostly just matching
2561 previous experience, and the recognition process gets to use all
2562 the senses to directly describe any action.
2564 ** Action recognition is easy with a full gamut of senses
2566 Embodied representation using multiple senses such as touch,
2567 proprioception, and muscle tension turns out be exceedingly
2568 efficient at describing body-centered actions. It is the right
2569 language for the job. For example, it takes only around 5 lines of
2570 clojure code to describe the action of curling using embodied
2571 primitives. It takes about 10 lines to describe the seemingly
2572 complicated action of wiggling.
2574 The following action predicates each take a stream of sensory
2575 experience, observe however much of it they desire, and decide
2576 whether the worm is doing the action they describe. =curled?=
2577 relies on proprioception, =resting?= relies on touch, =wiggling?=
2578 relies on a Fourier analysis of muscle contraction, and
2579 =grand-circle?= relies on touch and reuses =curled?= in its
2580 definition, showing how embodied predicates can be composed.
2583 #+caption: Program for detecting whether the worm is curled. This is the
2584 #+caption: simplest action predicate, because it only uses the last frame
2585 #+caption: of sensory experience, and only uses proprioceptive data. Even
2586 #+caption: this simple predicate, however, is automatically frame
2587 #+caption: independent and ignores vermopomorphic\protect\footnotemark
2588 #+caption: \space differences such as worm textures and colors.
2589 #+name: curled
2590 #+begin_listing clojure
2591 #+begin_src clojure
2592 (defn curled?
2593 "Is the worm curled up?"
2594 [experiences]
2595 (every?
2596 (fn [[_ _ bend]]
2597 (> (Math/sin bend) 0.64))
2598 (:proprioception (peek experiences))))
2599 #+end_src
2600 #+end_listing
2602 #+BEGIN_LaTeX
2603 \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
2604 #+END_LaTeX
2606 #+caption: Program for summarizing the touch information in a patch
2607 #+caption: of skin.
2608 #+name: touch-summary
2609 #+begin_listing clojure
2610 #+begin_src clojure
2611 (defn contact
2612 "Determine how much contact a particular worm segment has with
2613 other objects. Returns a value between 0 and 1, where 1 is full
2614 contact and 0 is no contact."
2615 [touch-region [coords contact :as touch]]
2616 (-> (zipmap coords contact)
2617 (select-keys touch-region)
2618 (vals)
2619 (#(map first %))
2620 (average)
2621 (* 10)
2622 (- 1)
2623 (Math/abs)))
2624 #+end_src
2625 #+end_listing
2628 #+caption: Program for detecting whether the worm is at rest. This program
2629 #+caption: uses a summary of the tactile information from the underbelly
2630 #+caption: of the worm, and is only true if every segment is touching the
2631 #+caption: floor. Note that this function contains no references to
2632 #+caption: proprioception at all.
2633 #+name: resting
2634 #+begin_listing clojure
2635 #+begin_src clojure
2636 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2638 (defn resting?
2639 "Is the worm resting on the ground?"
2640 [experiences]
2641 (every?
2642 (fn [touch-data]
2643 (< 0.9 (contact worm-segment-bottom touch-data)))
2644 (:touch (peek experiences))))
2645 #+end_src
2646 #+end_listing
2648 #+caption: Program for detecting whether the worm is curled up into a
2649 #+caption: full circle. Here the embodied approach begins to shine, as
2650 #+caption: I am able to both use a previous action predicate (=curled?=)
2651 #+caption: as well as the direct tactile experience of the head and tail.
2652 #+name: grand-circle
2653 #+begin_listing clojure
2654 #+begin_src clojure
2655 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2657 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2659 (defn grand-circle?
2660 "Does the worm form a majestic circle (one end touching the other)?"
2661 [experiences]
2662 (and (curled? experiences)
2663 (let [worm-touch (:touch (peek experiences))
2664 tail-touch (worm-touch 0)
2665 head-touch (worm-touch 4)]
2666 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2667 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2668 #+end_src
2669 #+end_listing
2672 #+caption: Program for detecting whether the worm has been wiggling for
2673 #+caption: the last few frames. It uses a Fourier analysis of the muscle
2674 #+caption: contractions of the worm's tail to determine wiggling. This is
2675 #+caption: significant because there is no particular frame that clearly
2676 #+caption: indicates that the worm is wiggling --- only when multiple frames
2677 #+caption: are analyzed together is the wiggling revealed. Defining
2678 #+caption: wiggling this way also gives the worm an opportunity to learn
2679 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2680 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2681 #+caption: from actual wiggling, but this definition gives it to us for free.
2682 #+name: wiggling
2683 #+begin_listing clojure
2684 #+begin_src clojure
2685 (defn fft [nums]
2686 (map
2687 #(.getReal %)
2688 (.transform
2689 (FastFourierTransformer. DftNormalization/STANDARD)
2690 (double-array nums) TransformType/FORWARD)))
2692 (def indexed (partial map-indexed vector))
2694 (defn max-indexed [s]
2695 (first (sort-by (comp - second) (indexed s))))
2697 (defn wiggling?
2698 "Is the worm wiggling?"
2699 [experiences]
2700 (let [analysis-interval 0x40]
2701 (when (> (count experiences) analysis-interval)
2702 (let [a-flex 3
2703 a-ex 2
2704 muscle-activity
2705 (map :muscle (vector:last-n experiences analysis-interval))
2706 base-activity
2707 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2708 (= 2
2709 (first
2710 (max-indexed
2711 (map #(Math/abs %)
2712 (take 20 (fft base-activity))))))))))
2713 #+end_src
2714 #+end_listing
2716 With these action predicates, I can now recognize the actions of
2717 the worm while it is moving under my control and I have access to
2718 all the worm's senses.
2720 #+caption: Use the action predicates defined earlier to report on
2721 #+caption: what the worm is doing while in simulation.
2722 #+name: report-worm-activity
2723 #+begin_listing clojure
2724 #+begin_src clojure
2725 (defn debug-experience
2726 [experiences text]
2727 (cond
2728 (grand-circle? experiences) (.setText text "Grand Circle")
2729 (curled? experiences) (.setText text "Curled")
2730 (wiggling? experiences) (.setText text "Wiggling")
2731 (resting? experiences) (.setText text "Resting")))
2732 #+end_src
2733 #+end_listing
2735 #+caption: Using =debug-experience=, the body-centered predicates
2736 #+caption: work together to classify the behavior of the worm.
2737 #+caption: the predicates are operating with access to the worm's
2738 #+caption: full sensory data.
2739 #+name: basic-worm-view
2740 #+ATTR_LaTeX: :width 10cm
2741 [[./images/worm-identify-init.png]]
2743 These action predicates satisfy the recognition requirement of an
2744 empathic recognition system. There is power in the simplicity of
2745 the action predicates. They describe their actions without getting
2746 confused in visual details of the worm. Each one is independent of
2747 position and rotation, but more than that, they are each
2748 independent of irrelevant visual details of the worm and the
2749 environment. They will work regardless of whether the worm is a
2750 different color or heavily textured, or if the environment has
2751 strange lighting.
2753 Consider how the human act of jumping might be described with
2754 body-centered action predicates: You might specify that jumping is
2755 mainly the feeling of your knees bending, your thigh muscles
2756 contracting, and your inner ear experiencing a certain sort of back
2757 and forth acceleration. This representation is a very concrete
2758 description of jumping, couched in terms of muscles and senses, but
2759 it also has the ability to describe almost all kinds of jumping, a
2760 generality that you might think could only be achieved by a very
2761 abstract description. The body centered jumping predicate does not
2762 have terms that consider the color of a person's skin or whether
2763 they are male or female, instead it gets right to the meat of what
2764 jumping actually /is/.
2766 Of course, the action predicates are not directly applicable to
2767 video data, which lacks the advanced sensory information which they
2768 require!
2770 The trick now is to make the action predicates work even when the
2771 sensory data on which they depend is absent!
2773 ** \Phi-space describes the worm's experiences
2775 As a first step towards building empathy, I need to gather all of
2776 the worm's experiences during free play. I use a simple vector to
2777 store all the experiences.
2779 Each element of the experience vector exists in the vast space of
2780 all possible worm-experiences. Most of this vast space is actually
2781 unreachable due to physical constraints of the worm's body. For
2782 example, the worm's segments are connected by hinge joints that put
2783 a practical limit on the worm's range of motions without limiting
2784 its degrees of freedom. Some groupings of senses are impossible;
2785 the worm can not be bent into a circle so that its ends are
2786 touching and at the same time not also experience the sensation of
2787 touching itself.
2789 As the worm moves around during free play and its experience vector
2790 grows larger, the vector begins to define a subspace which is all
2791 the sensations the worm can practically experience during normal
2792 operation. I call this subspace \Phi-space, short for
2793 physical-space. The experience vector defines a path through
2794 \Phi-space. This path has interesting properties that all derive
2795 from physical embodiment. The proprioceptive components of the path
2796 vary smoothly, because in order for the worm to move from one
2797 position to another, it must pass through the intermediate
2798 positions. The path invariably forms loops as common actions are
2799 repeated. Finally and most importantly, proprioception alone
2800 actually gives very strong inference about the other senses. For
2801 example, when the worm is proprioceptively flat over several
2802 frames, you can infer that it is touching the ground and that its
2803 muscles are not active, because if the muscles were active, the
2804 worm would be moving and would not remain perfectly flat. In order
2805 to stay flat, the worm has to be touching the ground, or it would
2806 again be moving out of the flat position due to gravity. If the
2807 worm is positioned in such a way that it interacts with itself,
2808 then it is very likely to be feeling the same tactile feelings as
2809 the last time it was in that position, because it has the same body
2810 as then. As you observe multiple frames of proprioceptive data, you
2811 can become increasingly confident about the exact activations of
2812 the worm's muscles, because it generally takes a unique combination
2813 of muscle contractions to transform the worm's body along a
2814 specific path through \Phi-space.
2816 The worm's total life experience is a long looping path through
2817 \Phi-space. I will now introduce simple way of taking that
2818 experience path and building a function that can infer complete
2819 sensory experience given only a stream of proprioceptive data. This
2820 /empathy/ function will provide a bridge to use the body centered
2821 action predicates on video-like streams of information.
2823 ** Empathy is the process of building paths in \Phi-space
2825 Here is the core of a basic empathy algorithm, starting with an
2826 experience vector:
2828 An /experience-index/ is an index into the grand experience vector
2829 that defines the worm's life. It is a time-stamp for each set of
2830 sensations the worm has experienced.
2832 First, group the experience-indices into bins according to the
2833 similarity of their proprioceptive data. I organize my bins into a
2834 3 level hierarchy. The smallest bins have an approximate size of
2835 0.001 radians in all proprioceptive dimensions. Each higher level
2836 is 10x bigger than the level below it.
2838 The bins serve as a hashing function for proprioceptive data. Given
2839 a single piece of proprioceptive experience, the bins allow us to
2840 rapidly find all other similar experience-indices of past
2841 experience that had a very similar proprioceptive configuration.
2842 When looking up a proprioceptive experience, if the smallest bin
2843 does not match any previous experience, then successively larger
2844 bins are used until a match is found or we reach the largest bin.
2846 Given a sequence of proprioceptive input, I use the bins to
2847 generate a set of similar experiences for each input using the
2848 tiered proprioceptive bins.
2850 Finally, to infer sensory data, I select the longest consecutive
2851 chain of experiences that threads through the sets of similar
2852 experiences, starting with the current moment as a root and going
2853 backwards. Consecutive experience means that the experiences appear
2854 next to each other in the experience vector.
2856 A stream of proprioceptive input might be:
2858 #+BEGIN_EXAMPLE
2859 [ flat, flat, flat, flat, flat, flat, lift-head ]
2860 #+END_EXAMPLE
2862 The worm's previous experience of lying on the ground and lifting
2863 its head generates possible interpretations for each frame (the
2864 numbers are experience-indices):
2866 #+BEGIN_EXAMPLE
2867 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2868 1 1 1 1 1 1 1 4
2869 2 2 2 2 2 2 2
2870 3 3 3 3 3 3 3
2871 6 6 6 6 6 6 6
2872 7 7 7 7 7 7 7
2873 8 8 8 8 8 8 8
2874 9 9 9 9 9 9 9
2875 #+END_EXAMPLE
2877 These interpretations suggest a new path through phi space:
2879 #+BEGIN_EXAMPLE
2880 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
2881 6 7 8 9 1 2 3 4
2882 #+END_EXAMPLE
2884 The new path through \Phi-space is synthesized from two actual
2885 paths that the creature has experienced: the "1-2-3-4" chain and
2886 the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
2887 ends with the worm lifting its head. It originated from a short
2888 training session where the worm rested on the floor for a brief
2889 while and then raised its head. The "6-7-8-9" chain is part of a
2890 longer chain of inactivity where the worm simply rested on the
2891 floor without moving. It is preferred over a "1-2-3" chain (which
2892 also describes inactivity) because it is longer. The main ideas
2893 again:
2895 - Imagined \Phi-space paths are synthesized by looping and mixing
2896 previous experiences.
2898 - Longer experience paths (less edits) are preferred.
2900 - The present is more important than the past --- more recent
2901 events take precedence in interpretation.
2903 This algorithm has three advantages:
2905 1. It's simple
2907 3. It's very fast -- retrieving possible interpretations takes
2908 constant time. Tracing through chains of interpretations takes
2909 time proportional to the average number of experiences in a
2910 proprioceptive bin. Redundant experiences in \Phi-space can be
2911 merged to save computation.
2913 2. It protects from wrong interpretations of transient ambiguous
2914 proprioceptive data. For example, if the worm is flat for just
2915 an instant, this flatness will not be interpreted as implying
2916 that the worm has its muscles relaxed, since the flatness is
2917 part of a longer chain which includes a distinct pattern of
2918 muscle activation. Markov chains or other memoryless statistical
2919 models that operate on individual frames may very well make this
2920 mistake.
2922 #+caption: Program to convert an experience vector into a
2923 #+caption: proprioceptively binned lookup function.
2924 #+name: bin
2925 #+begin_listing clojure
2926 #+begin_src clojure
2927 (defn bin [digits]
2928 (fn [angles]
2929 (->> angles
2930 (flatten)
2931 (map (juxt #(Math/sin %) #(Math/cos %)))
2932 (flatten)
2933 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2935 (defn gen-phi-scan
2936 "Nearest-neighbors with binning. Only returns a result if
2937 the proprioceptive data is within 10% of a previously recorded
2938 result in all dimensions."
2939 [phi-space]
2940 (let [bin-keys (map bin [3 2 1])
2941 bin-maps
2942 (map (fn [bin-key]
2943 (group-by
2944 (comp bin-key :proprioception phi-space)
2945 (range (count phi-space)))) bin-keys)
2946 lookups (map (fn [bin-key bin-map]
2947 (fn [proprio] (bin-map (bin-key proprio))))
2948 bin-keys bin-maps)]
2949 (fn lookup [proprio-data]
2950 (set (some #(% proprio-data) lookups)))))
2951 #+end_src
2952 #+end_listing
2954 #+caption: =longest-thread= finds the longest path of consecutive
2955 #+caption: past experiences to explain proprioceptive worm data from
2956 #+caption: previous data. Here, the film strip represents the
2957 #+caption: creature's previous experience. Sort sequences of
2958 #+caption: memories are spliced together to match the
2959 #+caption: proprioceptive data. Their carry the other senses
2960 #+caption: along with them.
2961 #+name: phi-space-history-scan
2962 #+ATTR_LaTeX: :width 10cm
2963 [[./images/film-of-imagination.png]]
2965 =longest-thread= infers sensory data by stitching together pieces
2966 from previous experience. It prefers longer chains of previous
2967 experience to shorter ones. For example, during training the worm
2968 might rest on the ground for one second before it performs its
2969 exercises. If during recognition the worm rests on the ground for
2970 five seconds, =longest-thread= will accommodate this five second
2971 rest period by looping the one second rest chain five times.
2973 =longest-thread= takes time proportional to the average number of
2974 entries in a proprioceptive bin, because for each element in the
2975 starting bin it performs a series of set lookups in the preceding
2976 bins. If the total history is limited, then this takes time
2977 proportional to a only a constant multiple of the number of entries
2978 in the starting bin. This analysis also applies, even if the action
2979 requires multiple longest chains -- it's still the average number
2980 of entries in a proprioceptive bin times the desired chain length.
2981 Because =longest-thread= is so efficient and simple, I can
2982 interpret worm-actions in real time.
2984 #+caption: Program to calculate empathy by tracing though \Phi-space
2985 #+caption: and finding the longest (ie. most coherent) interpretation
2986 #+caption: of the data.
2987 #+name: longest-thread
2988 #+begin_listing clojure
2989 #+begin_src clojure
2990 (defn longest-thread
2991 "Find the longest thread from phi-index-sets. The index sets should
2992 be ordered from most recent to least recent."
2993 [phi-index-sets]
2994 (loop [result '()
2995 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2996 (if (empty? phi-index-sets)
2997 (vec result)
2998 (let [threads
2999 (for [thread-base thread-bases]
3000 (loop [thread (list thread-base)
3001 remaining remaining]
3002 (let [next-index (dec (first thread))]
3003 (cond (empty? remaining) thread
3004 (contains? (first remaining) next-index)
3005 (recur
3006 (cons next-index thread) (rest remaining))
3007 :else thread))))
3008 longest-thread
3009 (reduce (fn [thread-a thread-b]
3010 (if (> (count thread-a) (count thread-b))
3011 thread-a thread-b))
3012 '(nil)
3013 threads)]
3014 (recur (concat longest-thread result)
3015 (drop (count longest-thread) phi-index-sets))))))
3016 #+end_src
3017 #+end_listing
3019 There is one final piece, which is to replace missing sensory data
3020 with a best-guess estimate. While I could fill in missing data by
3021 using a gradient over the closest known sensory data points,
3022 averages can be misleading. It is certainly possible to create an
3023 impossible sensory state by averaging two possible sensory states.
3024 For example, consider moving your hand in an arc over your head. If
3025 for some reason you only have the initial and final positions of
3026 this movement in your \Phi-space, averaging them together will
3027 produce the proprioceptive sensation of having your hand /inside/
3028 your head, which is physically impossible to ever experience
3029 (barring motor adaption illusions). Therefore I simply replicate
3030 the most recent sensory experience to fill in the gaps.
3032 #+caption: Fill in blanks in sensory experience by replicating the most
3033 #+caption: recent experience.
3034 #+name: infer-nils
3035 #+begin_listing clojure
3036 #+begin_src clojure
3037 (defn infer-nils
3038 "Replace nils with the next available non-nil element in the
3039 sequence, or barring that, 0."
3040 [s]
3041 (loop [i (dec (count s))
3042 v (transient s)]
3043 (if (zero? i) (persistent! v)
3044 (if-let [cur (v i)]
3045 (if (get v (dec i) 0)
3046 (recur (dec i) v)
3047 (recur (dec i) (assoc! v (dec i) cur)))
3048 (recur i (assoc! v i 0))))))
3049 #+end_src
3050 #+end_listing
3052 ** =EMPATH= recognizes actions efficiently
3054 To use =EMPATH= with the worm, I first need to gather a set of
3055 experiences from the worm that includes the actions I want to
3056 recognize. The =generate-phi-space= program (listing
3057 \ref{generate-phi-space} runs the worm through a series of
3058 exercises and gathers those experiences into a vector. The
3059 =do-all-the-things= program is a routine expressed in a simple
3060 muscle contraction script language for automated worm control. It
3061 causes the worm to rest, curl, and wiggle over about 700 frames
3062 (approx. 11 seconds).
3064 #+caption: Program to gather the worm's experiences into a vector for
3065 #+caption: further processing. The =motor-control-program= line uses
3066 #+caption: a motor control script that causes the worm to execute a series
3067 #+caption: of ``exercises'' that include all the action predicates.
3068 #+name: generate-phi-space
3069 #+begin_listing clojure
3070 #+begin_src clojure
3071 (def do-all-the-things
3072 (concat
3073 curl-script
3074 [[300 :d-ex 40]
3075 [320 :d-ex 0]]
3076 (shift-script 280 (take 16 wiggle-script))))
3078 (defn generate-phi-space []
3079 (let [experiences (atom [])]
3080 (run-world
3081 (apply-map
3082 worm-world
3083 (merge
3084 (worm-world-defaults)
3085 {:end-frame 700
3086 :motor-control
3087 (motor-control-program worm-muscle-labels do-all-the-things)
3088 :experiences experiences})))
3089 @experiences))
3090 #+end_src
3091 #+end_listing
3093 #+caption: Use =longest-thread= and a \Phi-space generated from a short
3094 #+caption: exercise routine to interpret actions during free play.
3095 #+name: empathy-debug
3096 #+begin_listing clojure
3097 #+begin_src clojure
3098 (defn init []
3099 (def phi-space (generate-phi-space))
3100 (def phi-scan (gen-phi-scan phi-space)))
3102 (defn empathy-demonstration []
3103 (let [proprio (atom ())]
3104 (fn
3105 [experiences text]
3106 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3107 (swap! proprio (partial cons phi-indices))
3108 (let [exp-thread (longest-thread (take 300 @proprio))
3109 empathy (mapv phi-space (infer-nils exp-thread))]
3110 (println-repl (vector:last-n exp-thread 22))
3111 (cond
3112 (grand-circle? empathy) (.setText text "Grand Circle")
3113 (curled? empathy) (.setText text "Curled")
3114 (wiggling? empathy) (.setText text "Wiggling")
3115 (resting? empathy) (.setText text "Resting")
3116 :else (.setText text "Unknown")))))))
3118 (defn empathy-experiment [record]
3119 (.start (worm-world :experience-watch (debug-experience-phi)
3120 :record record :worm worm*)))
3121 #+end_src
3122 #+end_listing
3124 These programs create a test for the empathy system. First, the
3125 worm's \Phi-space is generated from a simple motor script. Then the
3126 worm is re-created in an environment almost exactly identical to
3127 the testing environment for the action-predicates, with one major
3128 difference : the only sensory information available to the system
3129 is proprioception. From just the proprioception data and
3130 \Phi-space, =longest-thread= synthesizes a complete record the last
3131 300 sensory experiences of the worm. These synthesized experiences
3132 are fed directly into the action predicates =grand-circle?=,
3133 =curled?=, =wiggling?=, and =resting?= and their outputs are
3134 printed to the screen at each frame.
3136 The result of running =empathy-experiment= is that the system is
3137 generally able to interpret worm actions using the action-predicates
3138 on simulated sensory data just as well as with actual data. Figure
3139 \ref{empathy-debug-image} was generated using =empathy-experiment=:
3141 #+caption: From only proprioceptive data, =EMPATH= was able to infer
3142 #+caption: the complete sensory experience and classify four poses
3143 #+caption: (The last panel shows a composite image of /wiggling/,
3144 #+caption: a dynamic pose.)
3145 #+name: empathy-debug-image
3146 #+ATTR_LaTeX: :width 10cm :placement [H]
3147 [[./images/empathy-1.png]]
3149 One way to measure the performance of =EMPATH= is to compare the
3150 suitability of the imagined sense experience to trigger the same
3151 action predicates as the real sensory experience.
3153 #+caption: Determine how closely empathy approximates actual
3154 #+caption: sensory data.
3155 #+name: test-empathy-accuracy
3156 #+begin_listing clojure
3157 #+begin_src clojure
3158 (def worm-action-label
3159 (juxt grand-circle? curled? wiggling?))
3161 (defn compare-empathy-with-baseline [matches]
3162 (let [proprio (atom ())]
3163 (fn
3164 [experiences text]
3165 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3166 (swap! proprio (partial cons phi-indices))
3167 (let [exp-thread (longest-thread (take 300 @proprio))
3168 empathy (mapv phi-space (infer-nils exp-thread))
3169 experience-matches-empathy
3170 (= (worm-action-label experiences)
3171 (worm-action-label empathy))]
3172 (println-repl experience-matches-empathy)
3173 (swap! matches #(conj % experience-matches-empathy)))))))
3175 (defn accuracy [v]
3176 (float (/ (count (filter true? v)) (count v))))
3178 (defn test-empathy-accuracy []
3179 (let [res (atom [])]
3180 (run-world
3181 (worm-world :experience-watch
3182 (compare-empathy-with-baseline res)
3183 :worm worm*))
3184 (accuracy @res)))
3185 #+end_src
3186 #+end_listing
3188 Running =test-empathy-accuracy= using the very short exercise
3189 program =do-all-the-things= defined in listing
3190 \ref{generate-phi-space}, and then doing a similar pattern of
3191 activity using manual control of the worm, yields an accuracy of
3192 around 73%. This is based on very limited worm experience, and
3193 almost all errors are due to the worm's \Phi-space being too
3194 incomplete to properly interpret common poses. By manually training
3195 the worm for longer using =init-interactive= defined in listing
3196 \ref{manual-phi-space}, the accuracy dramatically improves:
3198 #+caption: Program to generate \Phi-space using manual training.
3199 #+name: manual-phi-space
3200 #+begin_listing clojure
3201 #+begin_src clojure
3202 (defn init-interactive []
3203 (def phi-space
3204 (let [experiences (atom [])]
3205 (run-world
3206 (apply-map
3207 worm-world
3208 (merge
3209 (worm-world-defaults)
3210 {:experiences experiences})))
3211 @experiences))
3212 (def phi-scan (gen-phi-scan phi-space)))
3213 #+end_src
3214 #+end_listing
3216 =init-interactive= allows me to take direct control of the worm's
3217 muscles and run it through each characteristic movement I care
3218 about. After about 1 minute of manual training, I was able to
3219 achieve 95% accuracy on manual testing of the worm using
3220 =test-empathy-accuracy=. The majority of disagreements are near the
3221 transition boundaries from one type of action to another. During
3222 these transitions the exact label for the action is often unclear,
3223 and disagreement between empathy and experience is practically
3224 irrelevant. Thus, the system's effective identification accuracy is
3225 even higher than 95%. When I watch this system myself, I generally
3226 see no errors in action identification compared to my own judgment
3227 of what the worm is doing.
3229 ** Digression: Learning touch sensor layout through free play
3231 In the previous chapter I showed how to compute actions in terms of
3232 body-centered predicates, but some of those predicates relied on
3233 the average touch activation of pre-defined regions of the worm's
3234 skin. What if, instead of receiving touch pre-grouped into the six
3235 faces of each worm segment, the true partitioning of the worm's
3236 skin was unknown? This is more similar to how a nerve fiber bundle
3237 might be arranged inside an animal. While two fibers that are close
3238 in a nerve bundle /might/ correspond to two touch sensors that are
3239 close together on the skin, the process of taking a complicated
3240 surface and forcing it into essentially a 2D circle requires that
3241 some regions of skin that are close together in the animal end up
3242 far apart in the nerve bundle.
3244 In this chapter I show how to automatically learn the
3245 skin-partitioning of a worm segment by free exploration. As the
3246 worm rolls around on the floor, large sections of its surface get
3247 activated. If the worm has stopped moving, then whatever region of
3248 skin that is touching the floor is probably an important region,
3249 and should be recorded. The code I provide relies on the worm
3250 segment having flat faces, but still demonstrates a primitive kind
3251 of multi-sensory bootstrapping that I find appealing.
3253 #+caption: Program to detect whether the worm is in a resting state
3254 #+caption: with one face touching the floor.
3255 #+name: pure-touch
3256 #+begin_listing clojure
3257 #+begin_src clojure
3258 (def full-contact [(float 0.0) (float 0.1)])
3260 (defn pure-touch?
3261 "This is worm specific code to determine if a large region of touch
3262 sensors is either all on or all off."
3263 [[coords touch :as touch-data]]
3264 (= (set (map first touch)) (set full-contact)))
3265 #+end_src
3266 #+end_listing
3268 After collecting these important regions, there will many nearly
3269 similar touch regions. While for some purposes the subtle
3270 differences between these regions will be important, for my
3271 purposes I collapse them into mostly non-overlapping sets using
3272 =remove-similar= in listing \ref{remove-similar}
3274 #+caption: Program to take a list of sets of points and ``collapse them''
3275 #+caption: so that the remaining sets in the list are significantly
3276 #+caption: different from each other. Prefer smaller sets to larger ones.
3277 #+name: remove-similar
3278 #+begin_listing clojure
3279 #+begin_src clojure
3280 (defn remove-similar
3281 [coll]
3282 (loop [result () coll (sort-by (comp - count) coll)]
3283 (if (empty? coll) result
3284 (let [[x & xs] coll
3285 c (count x)]
3286 (if (some
3287 (fn [other-set]
3288 (let [oc (count other-set)]
3289 (< (- (count (union other-set x)) c) (* oc 0.1))))
3290 xs)
3291 (recur result xs)
3292 (recur (cons x result) xs))))))
3293 #+end_src
3294 #+end_listing
3296 Actually running this simulation is easy given =CORTEX='s facilities.
3298 #+caption: Collect experiences while the worm moves around. Filter the touch
3299 #+caption: sensations by stable ones, collapse similar ones together,
3300 #+caption: and report the regions learned.
3301 #+name: learn-touch
3302 #+begin_listing clojure
3303 #+begin_src clojure
3304 (defn learn-touch-regions []
3305 (let [experiences (atom [])
3306 world (apply-map
3307 worm-world
3308 (assoc (worm-segment-defaults)
3309 :experiences experiences))]
3310 (run-world world)
3311 (->>
3312 @experiences
3313 (drop 175)
3314 ;; access the single segment's touch data
3315 (map (comp first :touch))
3316 ;; only deal with "pure" touch data to determine surfaces
3317 (filter pure-touch?)
3318 ;; associate coordinates with touch values
3319 (map (partial apply zipmap))
3320 ;; select those regions where contact is being made
3321 (map (partial group-by second))
3322 (map #(get % full-contact))
3323 (map (partial map first))
3324 ;; remove redundant/subset regions
3325 (map set)
3326 remove-similar)))
3328 (defn learn-and-view-touch-regions []
3329 (map view-touch-region
3330 (learn-touch-regions)))
3331 #+end_src
3332 #+end_listing
3334 The only thing remaining to define is the particular motion the worm
3335 must take. I accomplish this with a simple motor control program.
3337 #+caption: Motor control program for making the worm roll on the ground.
3338 #+caption: This could also be replaced with random motion.
3339 #+name: worm-roll
3340 #+begin_listing clojure
3341 #+begin_src clojure
3342 (defn touch-kinesthetics []
3343 [[170 :lift-1 40]
3344 [190 :lift-1 19]
3345 [206 :lift-1 0]
3347 [400 :lift-2 40]
3348 [410 :lift-2 0]
3350 [570 :lift-2 40]
3351 [590 :lift-2 21]
3352 [606 :lift-2 0]
3354 [800 :lift-1 30]
3355 [809 :lift-1 0]
3357 [900 :roll-2 40]
3358 [905 :roll-2 20]
3359 [910 :roll-2 0]
3361 [1000 :roll-2 40]
3362 [1005 :roll-2 20]
3363 [1010 :roll-2 0]
3365 [1100 :roll-2 40]
3366 [1105 :roll-2 20]
3367 [1110 :roll-2 0]
3368 ])
3369 #+end_src
3370 #+end_listing
3373 #+caption: The small worm rolls around on the floor, driven
3374 #+caption: by the motor control program in listing \ref{worm-roll}.
3375 #+name: worm-roll
3376 #+ATTR_LaTeX: :width 12cm
3377 [[./images/worm-roll.png]]
3379 #+caption: After completing its adventures, the worm now knows
3380 #+caption: how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were
3381 #+caption: deemed important by
3382 #+caption: =learn-touch-regions=. Each white square in the rectangles
3383 #+caption: above is a cluster of ``related" touch nodes as determined
3384 #+caption: by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces.
3385 #+name: worm-touch-map
3386 #+ATTR_LaTeX: :width 12cm
3387 [[./images/touch-learn.png]]
3389 While simple, =learn-touch-regions= exploits regularities in both
3390 the worm's physiology and the worm's environment to correctly
3391 deduce that the worm has six sides. Note that =learn-touch-regions=
3392 would work just as well even if the worm's touch sense data were
3393 completely scrambled. The cross shape is just for convenience. This
3394 example justifies the use of pre-defined touch regions in =EMPATH=.
3396 ** Recognizing an object using embodied representation
3398 At the beginning of the thesis, I suggested that we might recognize
3399 the chair in Figure \ref{hidden-chair} by imagining ourselves in
3400 the position of the man and realizing that he must be sitting on
3401 something in order to maintain that position. Here, I present a
3402 brief elaboration on how to this might be done.
3404 First, I need the feeling of leaning or resting /on/ some other
3405 object that is not the floor. This feeling is easy to describe
3406 using an embodied representation.
3408 #+caption: Program describing the sense of leaning or resting on something.
3409 #+caption: This involves a relaxed posture, the feeling of touching something,
3410 #+caption: and a period of stability where the worm does not move.
3411 #+name: draped
3412 #+begin_listing clojure
3413 #+begin_src clojure
3414 (defn draped?
3415 "Is the worm:
3416 -- not flat (the floor is not a 'chair')
3417 -- supported (not using its muscles to hold its position)
3418 -- stable (not changing its position)
3419 -- touching something (must register contact)"
3420 [experiences]
3421 (let [b2-hash (bin 2)
3422 touch (:touch (peek experiences))
3423 total-contact
3424 (reduce
3426 (map #(contact all-touch-coordinates %)
3427 (rest touch)))]
3428 (println total-contact)
3429 (and (not (resting? experiences))
3430 (every?
3431 zero?
3432 (-> experiences
3433 (vector:last-n 25)
3434 (#(map :muscle %))
3435 (flatten)))
3436 (-> experiences
3437 (vector:last-n 20)
3438 (#(map (comp b2-hash flatten :proprioception) %))
3439 (set)
3440 (count) (= 1))
3441 (< 0.03 total-contact))))
3442 #+end_src
3443 #+end_listing
3445 #+caption: The =draped?= predicate detects the presence of the cube
3446 #+caption: whenever the worm interacts with it. The details of the
3447 #+caption: cube are irrelevant; only the way it influences the
3448 #+caption: worm's body matters. The ``unknown'' label on the fifth
3449 #+caption: frame is due to the fact that the worm is not
3450 #+caption: stationary. =draped?= will only declare that the worm
3451 #+caption: is draped if it has been still for a while.
3452 #+name: draped-video
3453 #+ATTR_LaTeX: :width 13cm
3454 [[./images/draped.png]]
3456 Though this is a simple example, using the =draped?= predicate to
3457 detect a cube has interesting advantages. The =draped?= predicate
3458 describes the cube not in terms of properties that the cube has,
3459 but instead in terms of how the worm interacts with it physically.
3460 This means that the cube can still be detected even if it is not
3461 visible, as long as its influence on the worm's body is visible.
3463 This system will also see the virtual cube created by a
3464 ``mimeworm", which uses its muscles in a very controlled way to
3465 mimic the appearance of leaning on a cube. The system will
3466 anticipate that there is an actual invisible cube that provides
3467 support!
3469 #+caption: Can you see the thing that this person is leaning on?
3470 #+caption: What properties does it have, other than how it makes
3471 #+caption: the man's elbow and shoulder feel? I wonder if people
3472 #+caption: who can actually maintain this pose easily still see the
3473 #+caption: support?
3474 #+name: mime
3475 #+ATTR_LaTeX: :width 6cm
3476 [[./images/pablo-the-mime.png]]
3478 This makes me wonder about the psychology of actual mimes. Suppose
3479 for a moment that people have something analogous to \Phi-space and
3480 that one of the ways that they find objects in a scene is by their
3481 relation to other people's bodies. Suppose that a person watches a
3482 person miming an invisible wall. For a person with no experience
3483 with miming, their \Phi-space will only have entries that describe
3484 the scene with the sensation of their hands touching a wall. This
3485 sensation of touch will create a strong impression of a wall, even
3486 though the wall would have to be invisible. A person with
3487 experience in miming however, will have entries in their \Phi-space
3488 that describe the wall-miming position without a sense of touch. It
3489 will not seem to such as person that an invisible wall is present,
3490 but merely that the mime is holding out their hands in a special
3491 way. Thus, the theory that humans use something like \Phi-space
3492 weakly predicts that learning how to mime should break the power of
3493 miming illusions. Most optical illusions still work no matter how
3494 much you know about them, so this proposal would be quite
3495 interesting to test, as it predicts a non-standard result!
3498 #+BEGIN_LaTeX
3499 \clearpage
3500 #+END_LaTeX
3502 * COMMENT Contributions
3504 The big idea behind this thesis is a new way to represent and
3505 recognize physical actions, which I call /empathic representation/.
3506 Actions are represented as predicates which have access to the
3507 totality of a creature's sensory abilities. To recognize the
3508 physical actions of another creature similar to yourself, you
3509 imagine what they would feel by examining the position of their body
3510 and relating it to your own previous experience.
3512 Empathic representation of physical actions is robust and general.
3513 Because the representation is body-centered, it avoids baking in a
3514 particular viewpoint like you might get from learning from example
3515 videos. Because empathic representation relies on all of a
3516 creature's senses, it can describe exactly what an action /feels
3517 like/ without getting caught up in irrelevant details such as visual
3518 appearance. I think it is important that a correct description of
3519 jumping (for example) should not include irrelevant details such as
3520 the color of a person's clothes or skin; empathic representation can
3521 get right to the heart of what jumping is by describing it in terms
3522 of touch, muscle contractions, and a brief feeling of
3523 weightlessness. Empathic representation is very low-level in that it
3524 describes actions using concrete sensory data with little
3525 abstraction, but it has the generality of much more abstract
3526 representations!
3528 Another important contribution of this thesis is the development of
3529 the =CORTEX= system, a complete environment for creating simulated
3530 creatures. You have seen how to implement five senses: touch,
3531 proprioception, hearing, vision, and muscle tension. You have seen
3532 how to create new creatures using blender, a 3D modeling tool.
3534 As a minor digression, you also saw how I used =CORTEX= to enable a
3535 tiny worm to discover the topology of its skin simply by rolling on
3536 the ground. You also saw how to detect objects using only embodied
3537 predicates.
3539 In conclusion, for this thesis I:
3541 - Developed the idea of embodied representation, which describes
3542 actions that a creature can do in terms of first-person sensory
3543 data.
3545 - Developed a method of empathic action recognition which uses
3546 previous embodied experience and embodied representation of
3547 actions to greatly constrain the possible interpretations of an
3548 action.
3550 - Created =EMPATH=, a program which uses empathic action
3551 recognition to recognize physical actions in a simple model
3552 involving segmented worm-like creatures.
3554 - Created =CORTEX=, a comprehensive platform for embodied AI
3555 experiments. It is the base on which =EMPATH= is built.
3557 #+BEGIN_LaTeX
3558 \clearpage
3559 \appendix
3560 #+END_LaTeX
3562 * COMMENT Appendix: =CORTEX= User Guide
3564 Those who write a thesis should endeavor to make their code not only
3565 accessible, but actually usable, as a way to pay back the community
3566 that made the thesis possible in the first place. This thesis would
3567 not be possible without Free Software such as jMonkeyEngine3,
3568 Blender, clojure, =emacs=, =ffmpeg=, and many other tools. That is
3569 why I have included this user guide, in the hope that someone else
3570 might find =CORTEX= useful.
3572 ** Obtaining =CORTEX=
3574 You can get cortex from its mercurial repository at
3575 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3576 releases at http://aurellem.org/cortex/releases/. As a condition of
3577 making this thesis, I have also provided Professor Winston the
3578 =CORTEX= source, and he knows how to run the demos and get started.
3579 You may also email me at =cortex@aurellem.org= and I may help where
3580 I can.
3582 ** Running =CORTEX=
3584 =CORTEX= comes with README and INSTALL files that will guide you
3585 through installation and running the test suite. In particular you
3586 should look at test =cortex.test= which contains test suites that
3587 run through all senses and multiple creatures.
3589 ** Creating creatures
3591 Creatures are created using /Blender/, a free 3D modeling program.
3592 You will need Blender version 2.6 when using the =CORTEX= included
3593 in this thesis. You create a =CORTEX= creature in a similar manner
3594 to modeling anything in Blender, except that you also create
3595 several trees of empty nodes which define the creature's senses.
3597 *** Mass
3599 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3600 to the object with the mass in jMonkeyEngine units. Note that
3601 setting the mass to 0 causes the object to be immovable.
3603 *** Joints
3605 Joints are created by creating an empty node named =joints= and
3606 then creating any number of empty child nodes to represent your
3607 creature's joints. The joint will automatically connect the
3608 closest two physical objects. It will help to set the empty node's
3609 display mode to ``Arrows'' so that you can clearly see the
3610 direction of the axes.
3612 Joint nodes should have the following metadata under the ``joint''
3613 label:
3615 #+BEGIN_SRC clojure
3616 ;; ONE of the following, under the label "joint":
3617 {:type :point}
3619 ;; OR
3621 {:type :hinge
3622 :limit [<limit-low> <limit-high>]
3623 :axis (Vector3f. <x> <y> <z>)}
3624 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3626 ;; OR
3628 {:type :cone
3629 :limit-xz <lim-xz>
3630 :limit-xy <lim-xy>
3631 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
3632 #+END_SRC
3634 *** Eyes
3636 Eyes are created by creating an empty node named =eyes= and then
3637 creating any number of empty child nodes to represent your
3638 creature's eyes.
3640 Eye nodes should have the following metadata under the ``eye''
3641 label:
3643 #+BEGIN_SRC clojure
3644 {:red <red-retina-definition>
3645 :blue <blue-retina-definition>
3646 :green <green-retina-definition>
3647 :all <all-retina-definition>
3648 (<0xrrggbb> <custom-retina-image>)...
3650 #+END_SRC
3652 Any of the color channels may be omitted. You may also include
3653 your own color selectors, and in fact :red is equivalent to
3654 0xFF0000 and so forth. The eye will be placed at the same position
3655 as the empty node and will bind to the neatest physical object.
3656 The eye will point outward from the X-axis of the node, and ``up''
3657 will be in the direction of the X-axis of the node. It will help
3658 to set the empty node's display mode to ``Arrows'' so that you can
3659 clearly see the direction of the axes.
3661 Each retina file should contain white pixels wherever you want to be
3662 sensitive to your chosen color. If you want the entire field of
3663 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3664 white.
3666 Here is a sample retinal map:
3668 #+caption: An example retinal profile image. White pixels are
3669 #+caption: photo-sensitive elements. The distribution of white
3670 #+caption: pixels is denser in the middle and falls off at the
3671 #+caption: edges and is inspired by the human retina.
3672 #+name: retina
3673 #+ATTR_LaTeX: :width 7cm :placement [H]
3674 [[./images/retina-small.png]]
3676 *** Hearing
3678 Ears are created by creating an empty node named =ears= and then
3679 creating any number of empty child nodes to represent your
3680 creature's ears.
3682 Ear nodes do not require any metadata.
3684 The ear will bind to and follow the closest physical node.
3686 *** Touch
3688 Touch is handled similarly to mass. To make a particular object
3689 touch sensitive, add metadata of the following form under the
3690 object's ``touch'' metadata field:
3692 #+BEGIN_EXAMPLE
3693 <touch-UV-map-file-name>
3694 #+END_EXAMPLE
3696 You may also include an optional ``scale'' metadata number to
3697 specify the length of the touch feelers. The default is $0.1$,
3698 and this is generally sufficient.
3700 The touch UV should contain white pixels for each touch sensor.
3702 Here is an example touch-uv map that approximates a human finger,
3703 and its corresponding model.
3705 #+caption: This is the tactile-sensor-profile for the upper segment
3706 #+caption: of a fingertip. It defines regions of high touch sensitivity
3707 #+caption: (where there are many white pixels) and regions of low
3708 #+caption: sensitivity (where white pixels are sparse).
3709 #+name: guide-fingertip-UV
3710 #+ATTR_LaTeX: :width 9cm :placement [H]
3711 [[./images/finger-UV.png]]
3713 #+caption: The fingertip UV-image form above applied to a simple
3714 #+caption: model of a fingertip.
3715 #+name: guide-fingertip
3716 #+ATTR_LaTeX: :width 9cm :placement [H]
3717 [[./images/finger-1.png]]
3719 *** Proprioception
3721 Proprioception is tied to each joint node -- nothing special must
3722 be done in a blender model to enable proprioception other than
3723 creating joint nodes.
3725 *** Muscles
3727 Muscles are created by creating an empty node named =muscles= and
3728 then creating any number of empty child nodes to represent your
3729 creature's muscles.
3732 Muscle nodes should have the following metadata under the
3733 ``muscle'' label:
3735 #+BEGIN_EXAMPLE
3736 <muscle-profile-file-name>
3737 #+END_EXAMPLE
3739 Muscles should also have a ``strength'' metadata entry describing
3740 the muscle's total strength at full activation.
3742 Muscle profiles are simple images that contain the relative amount
3743 of muscle power in each simulated alpha motor neuron. The width of
3744 the image is the total size of the motor pool, and the redness of
3745 each neuron is the relative power of that motor pool.
3747 While the profile image can have any dimensions, only the first
3748 line of pixels is used to define the muscle. Here is a sample
3749 muscle profile image that defines a human-like muscle.
3751 #+caption: A muscle profile image that describes the strengths
3752 #+caption: of each motor neuron in a muscle. White is weakest
3753 #+caption: and dark red is strongest. This particular pattern
3754 #+caption: has weaker motor neurons at the beginning, just
3755 #+caption: like human muscle.
3756 #+name: muscle-recruit
3757 #+ATTR_LaTeX: :width 7cm :placement [H]
3758 [[./images/basic-muscle.png]]
3760 Muscles twist the nearest physical object about the muscle node's
3761 Z-axis. I recommend using the ``Single Arrow'' display mode for
3762 muscles and using the right hand rule to determine which way the
3763 muscle will twist. To make a segment that can twist in multiple
3764 directions, create multiple, differently aligned muscles.
3766 ** =CORTEX= API
3768 These are the some functions exposed by =CORTEX= for creating
3769 worlds and simulating creatures. These are in addition to
3770 jMonkeyEngine3's extensive library, which is documented elsewhere.
3772 *** Simulation
3773 - =(world root-node key-map setup-fn update-fn)= :: create
3774 a simulation.
3775 - /root-node/ :: a =com.jme3.scene.Node= object which
3776 contains all of the objects that should be in the
3777 simulation.
3779 - /key-map/ :: a map from strings describing keys to
3780 functions that should be executed whenever that key is
3781 pressed. the functions should take a SimpleApplication
3782 object and a boolean value. The SimpleApplication is the
3783 current simulation that is running, and the boolean is true
3784 if the key is being pressed, and false if it is being
3785 released. As an example,
3786 #+BEGIN_SRC clojure
3787 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3788 #+END_SRC
3789 is a valid key-map which will cause the simulation to print
3790 a message whenever the 'j' key on the keyboard is pressed.
3792 - /setup-fn/ :: a function that takes a =SimpleApplication=
3793 object. It is called once when initializing the simulation.
3794 Use it to create things like lights, change the gravity,
3795 initialize debug nodes, etc.
3797 - /update-fn/ :: this function takes a =SimpleApplication=
3798 object and a float and is called every frame of the
3799 simulation. The float tells how many seconds is has been
3800 since the last frame was rendered, according to whatever
3801 clock jme is currently using. The default is to use IsoTimer
3802 which will result in this value always being the same.
3804 - =(position-camera world position rotation)= :: set the position
3805 of the simulation's main camera.
3807 - =(enable-debug world)= :: turn on debug wireframes for each
3808 simulated object.
3810 - =(set-gravity world gravity)= :: set the gravity of a running
3811 simulation.
3813 - =(box length width height & {options})= :: create a box in the
3814 simulation. Options is a hash map specifying texture, mass,
3815 etc. Possible options are =:name=, =:color=, =:mass=,
3816 =:friction=, =:texture=, =:material=, =:position=,
3817 =:rotation=, =:shape=, and =:physical?=.
3819 - =(sphere radius & {options})= :: create a sphere in the simulation.
3820 Options are the same as in =box=.
3822 - =(load-blender-model file-name)= :: create a node structure
3823 representing the model described in a blender file.
3825 - =(light-up-everything world)= :: distribute a standard compliment
3826 of lights throughout the simulation. Should be adequate for most
3827 purposes.
3829 - =(node-seq node)= :: return a recursive list of the node's
3830 children.
3832 - =(nodify name children)= :: construct a node given a node-name and
3833 desired children.
3835 - =(add-element world element)= :: add an object to a running world
3836 simulation.
3838 - =(set-accuracy world accuracy)= :: change the accuracy of the
3839 world's physics simulator.
3841 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3842 construct that is useful for loading textures and is required
3843 for smooth interaction with jMonkeyEngine library functions.
3845 - =(load-bullet)= :: unpack native libraries and initialize the
3846 bullet physics subsystem. This function is required before
3847 other world building functions are called.
3849 *** Creature Manipulation / Import
3851 - =(body! creature)= :: give the creature a physical body.
3853 - =(vision! creature)= :: give the creature a sense of vision.
3854 Returns a list of functions which will each, when called
3855 during a simulation, return the vision data for the channel of
3856 one of the eyes. The functions are ordered depending on the
3857 alphabetical order of the names of the eye nodes in the
3858 blender file. The data returned by the functions is a vector
3859 containing the eye's /topology/, a vector of coordinates, and
3860 the eye's /data/, a vector of RGB values filtered by the eye's
3861 sensitivity.
3863 - =(hearing! creature)= :: give the creature a sense of hearing.
3864 Returns a list of functions, one for each ear, that when
3865 called will return a frame's worth of hearing data for that
3866 ear. The functions are ordered depending on the alphabetical
3867 order of the names of the ear nodes in the blender file. The
3868 data returned by the functions is an array of PCM (pulse code
3869 modulated) wav data.
3871 - =(touch! creature)= :: give the creature a sense of touch. Returns
3872 a single function that must be called with the /root node/ of
3873 the world, and which will return a vector of /touch-data/
3874 one entry for each touch sensitive component, each entry of
3875 which contains a /topology/ that specifies the distribution of
3876 touch sensors, and the /data/, which is a vector of
3877 =[activation, length]= pairs for each touch hair.
3879 - =(proprioception! creature)= :: give the creature the sense of
3880 proprioception. Returns a list of functions, one for each
3881 joint, that when called during a running simulation will
3882 report the =[heading, pitch, roll]= of the joint.
3884 - =(movement! creature)= :: give the creature the power of movement.
3885 Creates a list of functions, one for each muscle, that when
3886 called with an integer, will set the recruitment of that
3887 muscle to that integer, and will report the current power
3888 being exerted by the muscle. Order of muscles is determined by
3889 the alphabetical sort order of the names of the muscle nodes.
3891 *** Visualization/Debug
3893 - =(view-vision)= :: create a function that when called with a list
3894 of visual data returned from the functions made by =vision!=,
3895 will display that visual data on the screen.
3897 - =(view-hearing)= :: same as =view-vision= but for hearing.
3899 - =(view-touch)= :: same as =view-vision= but for touch.
3901 - =(view-proprioception)= :: same as =view-vision= but for
3902 proprioception.
3904 - =(view-movement)= :: same as =view-vision= but for muscles.
3906 - =(view anything)= :: =view= is a polymorphic function that allows
3907 you to inspect almost anything you could reasonably expect to
3908 be able to ``see'' in =CORTEX=.
3910 - =(text anything)= :: =text= is a polymorphic function that allows
3911 you to convert practically anything into a text string.
3913 - =(println-repl anything)= :: print messages to clojure's repl
3914 instead of the simulation's terminal window.
3916 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3917 function will import all jMonkeyEngine3 classes for immediate
3918 use.
3920 - =(display-dilated-time world timer)= :: Shows the time as it is
3921 flowing in the simulation on a HUD display.