view thesis/cortex.org @ 514:447c3c8405a2

accept/reject changes
author Robert McIntyre <rlm@mit.edu>
date Sun, 30 Mar 2014 10:50:05 -0400
parents 07c3feb32df3
children 58fa1ffd481e
line wrap: on
line source
1 #+title: =CORTEX=
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment
6 #+LaTeX_CLASS_OPTIONS: [nofloat]
8 * COMMENT templates
9 #+caption:
10 #+caption:
11 #+caption:
12 #+caption:
13 #+name: name
14 #+begin_listing clojure
15 #+BEGIN_SRC clojure
16 #+END_SRC
17 #+end_listing
19 #+caption:
20 #+caption:
21 #+caption:
22 #+name: name
23 #+ATTR_LaTeX: :width 10cm
24 [[./images/aurellem-gray.png]]
26 #+caption:
27 #+caption:
28 #+caption:
29 #+caption:
30 #+name: name
31 #+begin_listing clojure
32 #+BEGIN_SRC clojure
33 #+END_SRC
34 #+end_listing
36 #+caption:
37 #+caption:
38 #+caption:
39 #+name: name
40 #+ATTR_LaTeX: :width 10cm
41 [[./images/aurellem-gray.png]]
44 * Empathy \& Embodiment: problem solving strategies
46 ** The problem: recognizing actions in video is extremely difficult
47 # developing / requires useful representations
49 Examine the following collection of images. As you, and indeed very
50 young children, can easily determine, each one is a picture of
51 someone drinking.
53 # dxh: cat, cup, drinking fountain, rain, straw, coconut
54 #+caption: A cat drinking some water. Identifying this action is
55 #+caption: beyond the capabilities of existing computer vision systems.
56 #+ATTR_LaTeX: :width 7cm
57 [[./images/cat-drinking.jpg]]
59 Nevertheless, it is beyond the state of the art for a computer
60 vision program to describe what's happening in each of these
61 images, or what's common to them. Part of the problem is that many
62 computer vision systems focus on pixel-level details or probability
63 distributions of pixels, with little focus on [...]
66 In fact, the contents of scene may have much less to do with pixel
67 probabilities than with recognizing various affordances: things you
68 can move, objects you can grasp, spaces that can be filled
69 (Gibson). For example, what processes might enable you to see the
70 chair in figure \ref{hidden-chair}?
71 # Or suppose that you are building a program that recognizes chairs.
72 # How could you ``see'' the chair ?
74 # dxh: blur chair
75 #+caption: The chair in this image is quite obvious to humans, but I
76 #+caption: doubt that any modern computer vision program can find it.
77 #+name: hidden-chair
78 #+ATTR_LaTeX: :width 10cm
79 [[./images/fat-person-sitting-at-desk.jpg]]
85 Finally, how is it that you can easily tell the difference between
86 how the girls /muscles/ are working in figure \ref{girl}?
88 #+caption: The mysterious ``common sense'' appears here as you are able
89 #+caption: to discern the difference in how the girl's arm muscles
90 #+caption: are activated between the two images.
91 #+name: girl
92 #+ATTR_LaTeX: :width 7cm
93 [[./images/wall-push.png]]
98 Each of these examples tells us something about what might be going
99 on in our minds as we easily solve these recognition problems.
101 The hidden chair shows us that we are strongly triggered by cues
102 relating to the position of human bodies, and that we can determine
103 the overall physical configuration of a human body even if much of
104 that body is occluded.
106 The picture of the girl pushing against the wall tells us that we
107 have common sense knowledge about the kinetics of our own bodies.
108 We know well how our muscles would have to work to maintain us in
109 most positions, and we can easily project this self-knowledge to
110 imagined positions triggered by images of the human body.
112 ** A step forward: the sensorimotor-centered approach
113 # ** =EMPATH= recognizes what creatures are doing
114 # neatly solves recognition problems
115 In this thesis, I explore the idea that our knowledge of our own
116 bodies enables us to recognize the actions of others.
118 First, I built a system for constructing virtual creatures with
119 physiologically plausible sensorimotor systems and detailed
120 environments. The result is =CORTEX=, which is described in section
121 \ref{sec-2}. (=CORTEX= was built to be flexible and useful to other
122 AI researchers; it is provided in full with detailed instructions
123 on the web [here].)
125 Next, I wrote routines which enabled a simple worm-like creature to
126 infer the actions of a second worm-like creature, using only its
127 own prior sensorimotor experiences and knowledge of the second
128 worm's joint positions. This program, =EMPATH=, is described in
129 section \ref{sec-3}, and the key results of this experiment are
130 summarized below.
132 #+caption: From only \emph{proprioceptive} data, =EMPATH= was able to infer
133 #+caption: the complete sensory experience and classify these four poses.
134 #+caption: The last image is a composite, depicting the intermediate stages of \emph{wriggling}.
135 #+name: worm-recognition-intro-2
136 #+ATTR_LaTeX: :width 15cm
137 [[./images/empathy-1.png]]
139 # =CORTEX= provides a language for describing the sensorimotor
140 # experiences of various creatures.
142 # Next, I developed an experiment to test the power of =CORTEX='s
143 # sensorimotor-centered language for solving recognition problems. As
144 # a proof of concept, I wrote routines which enabled a simple
145 # worm-like creature to infer the actions of a second worm-like
146 # creature, using only its own previous sensorimotor experiences and
147 # knowledge of the second worm's joints (figure
148 # \ref{worm-recognition-intro-2}). The result of this proof of
149 # concept was the program =EMPATH=, described in section
150 # \ref{sec-3}. The key results of this
152 # Using only first-person sensorimotor experiences and third-person
153 # proprioceptive data,
155 *** Key results
156 - After one-shot supervised training, =EMPATH= was able recognize a
157 wide variety of static poses and dynamic actions---ranging from
158 curling in a circle to wriggling with a particular frequency ---
159 with 95\% accuracy.
160 - These results were completely independent of viewing angle
161 because the underlying body-centered language fundamentally is
162 independent; once an action is learned, it can be recognized
163 equally well from any viewing angle.
164 - =EMPATH= is surprisingly short; the sensorimotor-centered
165 language provided by =CORTEX= resulted in extremely economical
166 recognition routines --- about 0000 lines in all --- suggesting
167 that such representations are very powerful, and often
168 indispensible for the types of recognition tasks considered here.
169 - Although for expediency's sake, I relied on direct knowledge of
170 joint positions in this proof of concept, it would be
171 straightforward to extend =EMPATH= so that it (more
172 realistically) infers joint positions from its visual data.
174 # because the underlying language is fundamentally orientation-independent
176 # recognize the actions of a worm with 95\% accuracy. The
177 # recognition tasks
182 [Talk about these results and what you find promising about them]
184 ** Roadmap
185 [I'm going to explain how =CORTEX= works, then break down how
186 =EMPATH= does its thing. Because the details reveal such-and-such
187 about the approach.]
189 # The success of this simple proof-of-concept offers a tantalizing
192 # explore the idea
193 # The key contribution of this thesis is the idea that body-centered
194 # representations (which express
197 # the
198 # body-centered approach --- in which I try to determine what's
199 # happening in a scene by bringing it into registration with my own
200 # bodily experiences --- are indispensible for recognizing what
201 # creatures are doing in a scene.
203 * COMMENT
204 # body-centered language
206 In this thesis, I'll describe =EMPATH=, which solves a certain
207 class of recognition problems
209 The key idea is to use self-centered (or first-person) language.
211 I have built a system that can express the types of recognition
212 problems in a form amenable to computation. It is split into
213 four parts:
215 - Free/Guided Play :: The creature moves around and experiences the
216 world through its unique perspective. Many otherwise
217 complicated actions are easily described in the language of a
218 full suite of body-centered, rich senses. For example,
219 drinking is the feeling of water sliding down your throat, and
220 cooling your insides. It's often accompanied by bringing your
221 hand close to your face, or bringing your face close to water.
222 Sitting down is the feeling of bending your knees, activating
223 your quadriceps, then feeling a surface with your bottom and
224 relaxing your legs. These body-centered action descriptions
225 can be either learned or hard coded.
226 - Posture Imitation :: When trying to interpret a video or image,
227 the creature takes a model of itself and aligns it with
228 whatever it sees. This alignment can even cross species, as
229 when humans try to align themselves with things like ponies,
230 dogs, or other humans with a different body type.
231 - Empathy :: The alignment triggers associations with
232 sensory data from prior experiences. For example, the
233 alignment itself easily maps to proprioceptive data. Any
234 sounds or obvious skin contact in the video can to a lesser
235 extent trigger previous experience. Segments of previous
236 experiences are stitched together to form a coherent and
237 complete sensory portrait of the scene.
238 - Recognition :: With the scene described in terms of first
239 person sensory events, the creature can now run its
240 action-identification programs on this synthesized sensory
241 data, just as it would if it were actually experiencing the
242 scene first-hand. If previous experience has been accurately
243 retrieved, and if it is analogous enough to the scene, then
244 the creature will correctly identify the action in the scene.
246 For example, I think humans are able to label the cat video as
247 ``drinking'' because they imagine /themselves/ as the cat, and
248 imagine putting their face up against a stream of water and
249 sticking out their tongue. In that imagined world, they can feel
250 the cool water hitting their tongue, and feel the water entering
251 their body, and are able to recognize that /feeling/ as drinking.
252 So, the label of the action is not really in the pixels of the
253 image, but is found clearly in a simulation inspired by those
254 pixels. An imaginative system, having been trained on drinking and
255 non-drinking examples and learning that the most important
256 component of drinking is the feeling of water sliding down one's
257 throat, would analyze a video of a cat drinking in the following
258 manner:
260 1. Create a physical model of the video by putting a ``fuzzy''
261 model of its own body in place of the cat. Possibly also create
262 a simulation of the stream of water.
264 2. Play out this simulated scene and generate imagined sensory
265 experience. This will include relevant muscle contractions, a
266 close up view of the stream from the cat's perspective, and most
267 importantly, the imagined feeling of water entering the
268 mouth. The imagined sensory experience can come from a
269 simulation of the event, but can also be pattern-matched from
270 previous, similar embodied experience.
272 3. The action is now easily identified as drinking by the sense of
273 taste alone. The other senses (such as the tongue moving in and
274 out) help to give plausibility to the simulated action. Note that
275 the sense of vision, while critical in creating the simulation,
276 is not critical for identifying the action from the simulation.
278 For the chair examples, the process is even easier:
280 1. Align a model of your body to the person in the image.
282 2. Generate proprioceptive sensory data from this alignment.
284 3. Use the imagined proprioceptive data as a key to lookup related
285 sensory experience associated with that particular proproceptive
286 feeling.
288 4. Retrieve the feeling of your bottom resting on a surface, your
289 knees bent, and your leg muscles relaxed.
291 5. This sensory information is consistent with the =sitting?=
292 sensory predicate, so you (and the entity in the image) must be
293 sitting.
295 6. There must be a chair-like object since you are sitting.
297 Empathy offers yet another alternative to the age-old AI
298 representation question: ``What is a chair?'' --- A chair is the
299 feeling of sitting.
301 My program, =EMPATH= uses this empathic problem solving technique
302 to interpret the actions of a simple, worm-like creature.
304 #+caption: The worm performs many actions during free play such as
305 #+caption: curling, wiggling, and resting.
306 #+name: worm-intro
307 #+ATTR_LaTeX: :width 15cm
308 [[./images/worm-intro-white.png]]
310 #+caption: =EMPATH= recognized and classified each of these
311 #+caption: poses by inferring the complete sensory experience
312 #+caption: from proprioceptive data.
313 #+name: worm-recognition-intro
314 #+ATTR_LaTeX: :width 15cm
315 [[./images/worm-poses.png]]
317 One powerful advantage of empathic problem solving is that it
318 factors the action recognition problem into two easier problems. To
319 use empathy, you need an /aligner/, which takes the video and a
320 model of your body, and aligns the model with the video. Then, you
321 need a /recognizer/, which uses the aligned model to interpret the
322 action. The power in this method lies in the fact that you describe
323 all actions form a body-centered viewpoint. You are less tied to
324 the particulars of any visual representation of the actions. If you
325 teach the system what ``running'' is, and you have a good enough
326 aligner, the system will from then on be able to recognize running
327 from any point of view, even strange points of view like above or
328 underneath the runner. This is in contrast to action recognition
329 schemes that try to identify actions using a non-embodied approach.
330 If these systems learn about running as viewed from the side, they
331 will not automatically be able to recognize running from any other
332 viewpoint.
334 Another powerful advantage is that using the language of multiple
335 body-centered rich senses to describe body-centerd actions offers a
336 massive boost in descriptive capability. Consider how difficult it
337 would be to compose a set of HOG filters to describe the action of
338 a simple worm-creature ``curling'' so that its head touches its
339 tail, and then behold the simplicity of describing thus action in a
340 language designed for the task (listing \ref{grand-circle-intro}):
342 #+caption: Body-centerd actions are best expressed in a body-centered
343 #+caption: language. This code detects when the worm has curled into a
344 #+caption: full circle. Imagine how you would replicate this functionality
345 #+caption: using low-level pixel features such as HOG filters!
346 #+name: grand-circle-intro
347 #+begin_listing clojure
348 #+begin_src clojure
349 (defn grand-circle?
350 "Does the worm form a majestic circle (one end touching the other)?"
351 [experiences]
352 (and (curled? experiences)
353 (let [worm-touch (:touch (peek experiences))
354 tail-touch (worm-touch 0)
355 head-touch (worm-touch 4)]
356 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
357 (< 0.2 (contact worm-segment-top-tip head-touch))))))
358 #+end_src
359 #+end_listing
361 ** =CORTEX= is a toolkit for building sensate creatures
363 I built =CORTEX= to be a general AI research platform for doing
364 experiments involving multiple rich senses and a wide variety and
365 number of creatures. I intend it to be useful as a library for many
366 more projects than just this thesis. =CORTEX= was necessary to meet
367 a need among AI researchers at CSAIL and beyond, which is that
368 people often will invent neat ideas that are best expressed in the
369 language of creatures and senses, but in order to explore those
370 ideas they must first build a platform in which they can create
371 simulated creatures with rich senses! There are many ideas that
372 would be simple to execute (such as =EMPATH=), but attached to them
373 is the multi-month effort to make a good creature simulator. Often,
374 that initial investment of time proves to be too much, and the
375 project must make do with a lesser environment.
377 =CORTEX= is well suited as an environment for embodied AI research
378 for three reasons:
380 - You can create new creatures using Blender, a popular 3D modeling
381 program. Each sense can be specified using special blender nodes
382 with biologically inspired paramaters. You need not write any
383 code to create a creature, and can use a wide library of
384 pre-existing blender models as a base for your own creatures.
386 - =CORTEX= implements a wide variety of senses: touch,
387 proprioception, vision, hearing, and muscle tension. Complicated
388 senses like touch, and vision involve multiple sensory elements
389 embedded in a 2D surface. You have complete control over the
390 distribution of these sensor elements through the use of simple
391 png image files. In particular, =CORTEX= implements more
392 comprehensive hearing than any other creature simulation system
393 available.
395 - =CORTEX= supports any number of creatures and any number of
396 senses. Time in =CORTEX= dialates so that the simulated creatures
397 always precieve a perfectly smooth flow of time, regardless of
398 the actual computational load.
400 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
401 engine designed to create cross-platform 3D desktop games. =CORTEX=
402 is mainly written in clojure, a dialect of =LISP= that runs on the
403 java virtual machine (JVM). The API for creating and simulating
404 creatures and senses is entirely expressed in clojure, though many
405 senses are implemented at the layer of jMonkeyEngine or below. For
406 example, for the sense of hearing I use a layer of clojure code on
407 top of a layer of java JNI bindings that drive a layer of =C++=
408 code which implements a modified version of =OpenAL= to support
409 multiple listeners. =CORTEX= is the only simulation environment
410 that I know of that can support multiple entities that can each
411 hear the world from their own perspective. Other senses also
412 require a small layer of Java code. =CORTEX= also uses =bullet=, a
413 physics simulator written in =C=.
415 #+caption: Here is the worm from above modeled in Blender, a free
416 #+caption: 3D-modeling program. Senses and joints are described
417 #+caption: using special nodes in Blender.
418 #+name: worm-recognition-intro
419 #+ATTR_LaTeX: :width 12cm
420 [[./images/blender-worm.png]]
422 Here are some thing I anticipate that =CORTEX= might be used for:
424 - exploring new ideas about sensory integration
425 - distributed communication among swarm creatures
426 - self-learning using free exploration,
427 - evolutionary algorithms involving creature construction
428 - exploration of exoitic senses and effectors that are not possible
429 in the real world (such as telekenisis or a semantic sense)
430 - imagination using subworlds
432 During one test with =CORTEX=, I created 3,000 creatures each with
433 their own independent senses and ran them all at only 1/80 real
434 time. In another test, I created a detailed model of my own hand,
435 equipped with a realistic distribution of touch (more sensitive at
436 the fingertips), as well as eyes and ears, and it ran at around 1/4
437 real time.
439 #+BEGIN_LaTeX
440 \begin{sidewaysfigure}
441 \includegraphics[width=9.5in]{images/full-hand.png}
442 \caption{
443 I modeled my own right hand in Blender and rigged it with all the
444 senses that {\tt CORTEX} supports. My simulated hand has a
445 biologically inspired distribution of touch sensors. The senses are
446 displayed on the right, and the simulation is displayed on the
447 left. Notice that my hand is curling its fingers, that it can see
448 its own finger from the eye in its palm, and that it can feel its
449 own thumb touching its palm.}
450 \end{sidewaysfigure}
451 #+END_LaTeX
453 ** Road map
455 By the end of this thesis, you will have seen a novel approach to
456 interpreting video using embodiment and empathy. You will have also
457 seen one way to efficiently implement empathy for embodied
458 creatures. Finally, you will become familiar with =CORTEX=, a system
459 for designing and simulating creatures with rich senses, which you
460 may choose to use in your own research.
462 This is the core vision of my thesis: That one of the important ways
463 in which we understand others is by imagining ourselves in their
464 position and emphatically feeling experiences relative to our own
465 bodies. By understanding events in terms of our own previous
466 corporeal experience, we greatly constrain the possibilities of what
467 would otherwise be an unwieldy exponential search. This extra
468 constraint can be the difference between easily understanding what
469 is happening in a video and being completely lost in a sea of
470 incomprehensible color and movement.
472 - I built =CORTEX=, a comprehensive platform for embodied AI
473 experiments. =CORTEX= supports many features lacking in other
474 systems, such proper simulation of hearing. It is easy to create
475 new =CORTEX= creatures using Blender, a free 3D modeling program.
477 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
478 a worm-like creature using a computational model of empathy.
481 * Designing =CORTEX=
482 In this section, I outline the design decisions that went into
483 making =CORTEX=, along with some details about its
484 implementation. (A practical guide to getting started with =CORTEX=,
485 which skips over the history and implementation details presented
486 here, is provided in an appendix \ref{} at the end of this paper.)
488 Throughout this project, I intended for =CORTEX= to be flexible and
489 extensible enough to be useful for other researchers who want to
490 test out ideas of their own. To this end, wherver I have had to make
491 archetictural choices about =CORTEX=, I have chosen to give as much
492 freedom to the user as possible, so that =CORTEX= may be used for
493 things I have not forseen.
495 ** Building in simulation versus reality
496 The most important archetictural decision of all is the choice to
497 use a computer-simulated environemnt in the first place! The world
498 is a vast and rich place, and for now simulations are a very poor
499 reflection of its complexity. It may be that there is a significant
500 qualatative difference between dealing with senses in the real
501 world and dealing with pale facilimilies of them in a simulation
502 \cite{brooks-representation}. What are the advantages and
503 disadvantages of a simulation vs. reality?
505 *** Simulation
507 The advantages of virtual reality are that when everything is a
508 simulation, experiments in that simulation are absolutely
509 reproducible. It's also easier to change the character and world
510 to explore new situations and different sensory combinations.
512 If the world is to be simulated on a computer, then not only do
513 you have to worry about whether the character's senses are rich
514 enough to learn from the world, but whether the world itself is
515 rendered with enough detail and realism to give enough working
516 material to the character's senses. To name just a few
517 difficulties facing modern physics simulators: destructibility of
518 the environment, simulation of water/other fluids, large areas,
519 nonrigid bodies, lots of objects, smoke. I don't know of any
520 computer simulation that would allow a character to take a rock
521 and grind it into fine dust, then use that dust to make a clay
522 sculpture, at least not without spending years calculating the
523 interactions of every single small grain of dust. Maybe a
524 simulated world with today's limitations doesn't provide enough
525 richness for real intelligence to evolve.
527 *** Reality
529 The other approach for playing with senses is to hook your
530 software up to real cameras, microphones, robots, etc., and let it
531 loose in the real world. This has the advantage of eliminating
532 concerns about simulating the world at the expense of increasing
533 the complexity of implementing the senses. Instead of just
534 grabbing the current rendered frame for processing, you have to
535 use an actual camera with real lenses and interact with photons to
536 get an image. It is much harder to change the character, which is
537 now partly a physical robot of some sort, since doing so involves
538 changing things around in the real world instead of modifying
539 lines of code. While the real world is very rich and definitely
540 provides enough stimulation for intelligence to develop as
541 evidenced by our own existence, it is also uncontrollable in the
542 sense that a particular situation cannot be recreated perfectly or
543 saved for later use. It is harder to conduct science because it is
544 harder to repeat an experiment. The worst thing about using the
545 real world instead of a simulation is the matter of time. Instead
546 of simulated time you get the constant and unstoppable flow of
547 real time. This severely limits the sorts of software you can use
548 to program the AI because all sense inputs must be handled in real
549 time. Complicated ideas may have to be implemented in hardware or
550 may simply be impossible given the current speed of our
551 processors. Contrast this with a simulation, in which the flow of
552 time in the simulated world can be slowed down to accommodate the
553 limitations of the character's programming. In terms of cost,
554 doing everything in software is far cheaper than building custom
555 real-time hardware. All you need is a laptop and some patience.
557 ** Simulated time enables rapid prototyping and complex scenes
559 I envision =CORTEX= being used to support rapid prototyping and
560 iteration of ideas. Even if I could put together a well constructed
561 kit for creating robots, it would still not be enough because of
562 the scourge of real-time processing. Anyone who wants to test their
563 ideas in the real world must always worry about getting their
564 algorithms to run fast enough to process information in real time.
565 The need for real time processing only increases if multiple senses
566 are involved. In the extreme case, even simple algorithms will have
567 to be accelerated by ASIC chips or FPGAs, turning what would
568 otherwise be a few lines of code and a 10x speed penality into a
569 multi-month ordeal. For this reason, =CORTEX= supports
570 /time-dialiation/, which scales back the framerate of the
571 simulation in proportion to the amount of processing each frame.
572 From the perspective of the creatures inside the simulation, time
573 always appears to flow at a constant rate, regardless of how
574 complicated the envorimnent becomes or how many creatures are in
575 the simulation. The cost is that =CORTEX= can sometimes run slower
576 than real time. This can also be an advantage, however ---
577 simulations of very simple creatures in =CORTEX= generally run at
578 40x on my machine!
580 ** All sense organs are two-dimensional surfaces
582 If =CORTEX= is to support a wide variety of senses, it would help
583 to have a better understanding of what a ``sense'' actually is!
584 While vision, touch, and hearing all seem like they are quite
585 different things, I was supprised to learn during the course of
586 this thesis that they (and all physical senses) can be expressed as
587 exactly the same mathematical object due to a dimensional argument!
589 Human beings are three-dimensional objects, and the nerves that
590 transmit data from our various sense organs to our brain are
591 essentially one-dimensional. This leaves up to two dimensions in
592 which our sensory information may flow. For example, imagine your
593 skin: it is a two-dimensional surface around a three-dimensional
594 object (your body). It has discrete touch sensors embedded at
595 various points, and the density of these sensors corresponds to the
596 sensitivity of that region of skin. Each touch sensor connects to a
597 nerve, all of which eventually are bundled together as they travel
598 up the spinal cord to the brain. Intersect the spinal nerves with a
599 guillotining plane and you will see all of the sensory data of the
600 skin revealed in a roughly circular two-dimensional image which is
601 the cross section of the spinal cord. Points on this image that are
602 close together in this circle represent touch sensors that are
603 /probably/ close together on the skin, although there is of course
604 some cutting and rearrangement that has to be done to transfer the
605 complicated surface of the skin onto a two dimensional image.
607 Most human senses consist of many discrete sensors of various
608 properties distributed along a surface at various densities. For
609 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
610 disks, and Ruffini's endings, which detect pressure and vibration
611 of various intensities. For ears, it is the stereocilia distributed
612 along the basilar membrane inside the cochlea; each one is
613 sensitive to a slightly different frequency of sound. For eyes, it
614 is rods and cones distributed along the surface of the retina. In
615 each case, we can describe the sense with a surface and a
616 distribution of sensors along that surface.
618 The neat idea is that every human sense can be effectively
619 described in terms of a surface containing embedded sensors. If the
620 sense had any more dimensions, then there wouldn't be enough room
621 in the spinal chord to transmit the information!
623 Therefore, =CORTEX= must support the ability to create objects and
624 then be able to ``paint'' points along their surfaces to describe
625 each sense.
627 Fortunately this idea is already a well known computer graphics
628 technique called called /UV-mapping/. The three-dimensional surface
629 of a model is cut and smooshed until it fits on a two-dimensional
630 image. You paint whatever you want on that image, and when the
631 three-dimensional shape is rendered in a game the smooshing and
632 cutting is reversed and the image appears on the three-dimensional
633 object.
635 To make a sense, interpret the UV-image as describing the
636 distribution of that senses sensors. To get different types of
637 sensors, you can either use a different color for each type of
638 sensor, or use multiple UV-maps, each labeled with that sensor
639 type. I generally use a white pixel to mean the presence of a
640 sensor and a black pixel to mean the absence of a sensor, and use
641 one UV-map for each sensor-type within a given sense.
643 #+CAPTION: The UV-map for an elongated icososphere. The white
644 #+caption: dots each represent a touch sensor. They are dense
645 #+caption: in the regions that describe the tip of the finger,
646 #+caption: and less dense along the dorsal side of the finger
647 #+caption: opposite the tip.
648 #+name: finger-UV
649 #+ATTR_latex: :width 10cm
650 [[./images/finger-UV.png]]
652 #+caption: Ventral side of the UV-mapped finger. Notice the
653 #+caption: density of touch sensors at the tip.
654 #+name: finger-side-view
655 #+ATTR_LaTeX: :width 10cm
656 [[./images/finger-1.png]]
658 ** Video game engines provide ready-made physics and shading
660 I did not need to write my own physics simulation code or shader to
661 build =CORTEX=. Doing so would lead to a system that is impossible
662 for anyone but myself to use anyway. Instead, I use a video game
663 engine as a base and modify it to accomodate the additional needs
664 of =CORTEX=. Video game engines are an ideal starting point to
665 build =CORTEX=, because they are not far from being creature
666 building systems themselves.
668 First off, general purpose video game engines come with a physics
669 engine and lighting / sound system. The physics system provides
670 tools that can be co-opted to serve as touch, proprioception, and
671 muscles. Since some games support split screen views, a good video
672 game engine will allow you to efficiently create multiple cameras
673 in the simulated world that can be used as eyes. Video game systems
674 offer integrated asset management for things like textures and
675 creatures models, providing an avenue for defining creatures. They
676 also understand UV-mapping, since this technique is used to apply a
677 texture to a model. Finally, because video game engines support a
678 large number of users, as long as =CORTEX= doesn't stray too far
679 from the base system, other researchers can turn to this community
680 for help when doing their research.
682 ** =CORTEX= is based on jMonkeyEngine3
684 While preparing to build =CORTEX= I studied several video game
685 engines to see which would best serve as a base. The top contenders
686 were:
688 - [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID
689 software in 1997. All the source code was released by ID
690 software into the Public Domain several years ago, and as a
691 result it has been ported to many different languages. This
692 engine was famous for its advanced use of realistic shading
693 and had decent and fast physics simulation. The main advantage
694 of the Quake II engine is its simplicity, but I ultimately
695 rejected it because the engine is too tied to the concept of a
696 first-person shooter game. One of the problems I had was that
697 there does not seem to be any easy way to attach multiple
698 cameras to a single character. There are also several physics
699 clipping issues that are corrected in a way that only applies
700 to the main character and do not apply to arbitrary objects.
702 - [[http://source.valvesoftware.com/][Source Engine]] :: The Source Engine evolved from the Quake II
703 and Quake I engines and is used by Valve in the Half-Life
704 series of games. The physics simulation in the Source Engine
705 is quite accurate and probably the best out of all the engines
706 I investigated. There is also an extensive community actively
707 working with the engine. However, applications that use the
708 Source Engine must be written in C++, the code is not open, it
709 only runs on Windows, and the tools that come with the SDK to
710 handle models and textures are complicated and awkward to use.
712 - [[http://jmonkeyengine.com/][jMonkeyEngine3]] :: jMonkeyEngine3 is a new library for creating
713 games in Java. It uses OpenGL to render to the screen and uses
714 screengraphs to avoid drawing things that do not appear on the
715 screen. It has an active community and several games in the
716 pipeline. The engine was not built to serve any particular
717 game but is instead meant to be used for any 3D game.
719 I chose jMonkeyEngine3 because it because it had the most features
720 out of all the free projects I looked at, and because I could then
721 write my code in clojure, an implementation of =LISP= that runs on
722 the JVM.
724 ** =CORTEX= uses Blender to create creature models
726 For the simple worm-like creatures I will use later on in this
727 thesis, I could define a simple API in =CORTEX= that would allow
728 one to create boxes, spheres, etc., and leave that API as the sole
729 way to create creatures. However, for =CORTEX= to truly be useful
730 for other projects, it needs a way to construct complicated
731 creatures. If possible, it would be nice to leverage work that has
732 already been done by the community of 3D modelers, or at least
733 enable people who are talented at moedling but not programming to
734 design =CORTEX= creatures.
736 Therefore, I use Blender, a free 3D modeling program, as the main
737 way to create creatures in =CORTEX=. However, the creatures modeled
738 in Blender must also be simple to simulate in jMonkeyEngine3's game
739 engine, and must also be easy to rig with =CORTEX='s senses. I
740 accomplish this with extensive use of Blender's ``empty nodes.''
742 Empty nodes have no mass, physical presence, or appearance, but
743 they can hold metadata and have names. I use a tree structure of
744 empty nodes to specify senses in the following manner:
746 - Create a single top-level empty node whose name is the name of
747 the sense.
748 - Add empty nodes which each contain meta-data relevant to the
749 sense, including a UV-map describing the number/distribution of
750 sensors if applicable.
751 - Make each empty-node the child of the top-level node.
753 #+caption: An example of annoting a creature model with empty
754 #+caption: nodes to describe the layout of senses. There are
755 #+caption: multiple empty nodes which each describe the position
756 #+caption: of muscles, ears, eyes, or joints.
757 #+name: sense-nodes
758 #+ATTR_LaTeX: :width 10cm
759 [[./images/empty-sense-nodes.png]]
761 ** Bodies are composed of segments connected by joints
763 Blender is a general purpose animation tool, which has been used in
764 the past to create high quality movies such as Sintel
765 \cite{blender}. Though Blender can model and render even complicated
766 things like water, it is crucual to keep models that are meant to
767 be simulated as creatures simple. =Bullet=, which =CORTEX= uses
768 though jMonkeyEngine3, is a rigid-body physics system. This offers
769 a compromise between the expressiveness of a game level and the
770 speed at which it can be simulated, and it means that creatures
771 should be naturally expressed as rigid components held together by
772 joint constraints.
774 But humans are more like a squishy bag with wrapped around some
775 hard bones which define the overall shape. When we move, our skin
776 bends and stretches to accomodate the new positions of our bones.
778 One way to make bodies composed of rigid pieces connected by joints
779 /seem/ more human-like is to use an /armature/, (or /rigging/)
780 system, which defines a overall ``body mesh'' and defines how the
781 mesh deforms as a function of the position of each ``bone'' which
782 is a standard rigid body. This technique is used extensively to
783 model humans and create realistic animations. It is not a good
784 technique for physical simulation, however because it creates a lie
785 -- the skin is not a physical part of the simulation and does not
786 interact with any objects in the world or itself. Objects will pass
787 right though the skin until they come in contact with the
788 underlying bone, which is a physical object. Whithout simulating
789 the skin, the sense of touch has little meaning, and the creature's
790 own vision will lie to it about the true extent of its body.
791 Simulating the skin as a physical object requires some way to
792 continuously update the physical model of the skin along with the
793 movement of the bones, which is unacceptably slow compared to rigid
794 body simulation.
796 Therefore, instead of using the human-like ``deformable bag of
797 bones'' approach, I decided to base my body plans on multiple solid
798 objects that are connected by joints, inspired by the robot =EVE=
799 from the movie WALL-E.
801 #+caption: =EVE= from the movie WALL-E. This body plan turns
802 #+caption: out to be much better suited to my purposes than a more
803 #+caption: human-like one.
804 #+ATTR_LaTeX: :width 10cm
805 [[./images/Eve.jpg]]
807 =EVE='s body is composed of several rigid components that are held
808 together by invisible joint constraints. This is what I mean by
809 ``eve-like''. The main reason that I use eve-style bodies is for
810 efficiency, and so that there will be correspondence between the
811 AI's semses and the physical presence of its body. Each individual
812 section is simulated by a separate rigid body that corresponds
813 exactly with its visual representation and does not change.
814 Sections are connected by invisible joints that are well supported
815 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
816 can efficiently simulate hundreds of rigid bodies connected by
817 joints. Just because sections are rigid does not mean they have to
818 stay as one piece forever; they can be dynamically replaced with
819 multiple sections to simulate splitting in two. This could be used
820 to simulate retractable claws or =EVE='s hands, which are able to
821 coalesce into one object in the movie.
823 *** Solidifying/Connecting a body
825 =CORTEX= creates a creature in two steps: first, it traverses the
826 nodes in the blender file and creates physical representations for
827 any of them that have mass defined in their blender meta-data.
829 #+caption: Program for iterating through the nodes in a blender file
830 #+caption: and generating physical jMonkeyEngine3 objects with mass
831 #+caption: and a matching physics shape.
832 #+name: name
833 #+begin_listing clojure
834 #+begin_src clojure
835 (defn physical!
836 "Iterate through the nodes in creature and make them real physical
837 objects in the simulation."
838 [#^Node creature]
839 (dorun
840 (map
841 (fn [geom]
842 (let [physics-control
843 (RigidBodyControl.
844 (HullCollisionShape.
845 (.getMesh geom))
846 (if-let [mass (meta-data geom "mass")]
847 (float mass) (float 1)))]
848 (.addControl geom physics-control)))
849 (filter #(isa? (class %) Geometry )
850 (node-seq creature)))))
851 #+end_src
852 #+end_listing
854 The next step to making a proper body is to connect those pieces
855 together with joints. jMonkeyEngine has a large array of joints
856 available via =bullet=, such as Point2Point, Cone, Hinge, and a
857 generic Six Degree of Freedom joint, with or without spring
858 restitution.
860 Joints are treated a lot like proper senses, in that there is a
861 top-level empty node named ``joints'' whose children each
862 represent a joint.
864 #+caption: View of the hand model in Blender showing the main ``joints''
865 #+caption: node (highlighted in yellow) and its children which each
866 #+caption: represent a joint in the hand. Each joint node has metadata
867 #+caption: specifying what sort of joint it is.
868 #+name: blender-hand
869 #+ATTR_LaTeX: :width 10cm
870 [[./images/hand-screenshot1.png]]
873 =CORTEX='s procedure for binding the creature together with joints
874 is as follows:
876 - Find the children of the ``joints'' node.
877 - Determine the two spatials the joint is meant to connect.
878 - Create the joint based on the meta-data of the empty node.
880 The higher order function =sense-nodes= from =cortex.sense=
881 simplifies finding the joints based on their parent ``joints''
882 node.
884 #+caption: Retrieving the children empty nodes from a single
885 #+caption: named empty node is a common pattern in =CORTEX=
886 #+caption: further instances of this technique for the senses
887 #+caption: will be omitted
888 #+name: get-empty-nodes
889 #+begin_listing clojure
890 #+begin_src clojure
891 (defn sense-nodes
892 "For some senses there is a special empty blender node whose
893 children are considered markers for an instance of that sense. This
894 function generates functions to find those children, given the name
895 of the special parent node."
896 [parent-name]
897 (fn [#^Node creature]
898 (if-let [sense-node (.getChild creature parent-name)]
899 (seq (.getChildren sense-node)) [])))
901 (def
902 ^{:doc "Return the children of the creature's \"joints\" node."
903 :arglists '([creature])}
904 joints
905 (sense-nodes "joints"))
906 #+end_src
907 #+end_listing
909 To find a joint's targets, =CORTEX= creates a small cube, centered
910 around the empty-node, and grows the cube exponentially until it
911 intersects two physical objects. The objects are ordered according
912 to the joint's rotation, with the first one being the object that
913 has more negative coordinates in the joint's reference frame.
914 Since the objects must be physical, the empty-node itself escapes
915 detection. Because the objects must be physical, =joint-targets=
916 must be called /after/ =physical!= is called.
918 #+caption: Program to find the targets of a joint node by
919 #+caption: exponentiallly growth of a search cube.
920 #+name: joint-targets
921 #+begin_listing clojure
922 #+begin_src clojure
923 (defn joint-targets
924 "Return the two closest two objects to the joint object, ordered
925 from bottom to top according to the joint's rotation."
926 [#^Node parts #^Node joint]
927 (loop [radius (float 0.01)]
928 (let [results (CollisionResults.)]
929 (.collideWith
930 parts
931 (BoundingBox. (.getWorldTranslation joint)
932 radius radius radius) results)
933 (let [targets
934 (distinct
935 (map #(.getGeometry %) results))]
936 (if (>= (count targets) 2)
937 (sort-by
938 #(let [joint-ref-frame-position
939 (jme-to-blender
940 (.mult
941 (.inverse (.getWorldRotation joint))
942 (.subtract (.getWorldTranslation %)
943 (.getWorldTranslation joint))))]
944 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
945 (take 2 targets))
946 (recur (float (* radius 2))))))))
947 #+end_src
948 #+end_listing
950 Once =CORTEX= finds all joints and targets, it creates them using
951 a dispatch on the metadata of each joint node.
953 #+caption: Program to dispatch on blender metadata and create joints
954 #+caption: sutiable for physical simulation.
955 #+name: joint-dispatch
956 #+begin_listing clojure
957 #+begin_src clojure
958 (defmulti joint-dispatch
959 "Translate blender pseudo-joints into real JME joints."
960 (fn [constraints & _]
961 (:type constraints)))
963 (defmethod joint-dispatch :point
964 [constraints control-a control-b pivot-a pivot-b rotation]
965 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
966 (.setLinearLowerLimit Vector3f/ZERO)
967 (.setLinearUpperLimit Vector3f/ZERO)))
969 (defmethod joint-dispatch :hinge
970 [constraints control-a control-b pivot-a pivot-b rotation]
971 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
972 [limit-1 limit-2] (:limit constraints)
973 hinge-axis (.mult rotation (blender-to-jme axis))]
974 (doto (HingeJoint. control-a control-b pivot-a pivot-b
975 hinge-axis hinge-axis)
976 (.setLimit limit-1 limit-2))))
978 (defmethod joint-dispatch :cone
979 [constraints control-a control-b pivot-a pivot-b rotation]
980 (let [limit-xz (:limit-xz constraints)
981 limit-xy (:limit-xy constraints)
982 twist (:twist constraints)]
983 (doto (ConeJoint. control-a control-b pivot-a pivot-b
984 rotation rotation)
985 (.setLimit (float limit-xz) (float limit-xy)
986 (float twist)))))
987 #+end_src
988 #+end_listing
990 All that is left for joints it to combine the above pieces into a
991 something that can operate on the collection of nodes that a
992 blender file represents.
994 #+caption: Program to completely create a joint given information
995 #+caption: from a blender file.
996 #+name: connect
997 #+begin_listing clojure
998 #+begin_src clojure
999 (defn connect
1000 "Create a joint between 'obj-a and 'obj-b at the location of
1001 'joint. The type of joint is determined by the metadata on 'joint.
1003 Here are some examples:
1004 {:type :point}
1005 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
1006 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
1008 {:type :cone :limit-xz 0]
1009 :limit-xy 0]
1010 :twist 0]} (use XZY rotation mode in blender!)"
1011 [#^Node obj-a #^Node obj-b #^Node joint]
1012 (let [control-a (.getControl obj-a RigidBodyControl)
1013 control-b (.getControl obj-b RigidBodyControl)
1014 joint-center (.getWorldTranslation joint)
1015 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
1016 pivot-a (world-to-local obj-a joint-center)
1017 pivot-b (world-to-local obj-b joint-center)]
1018 (if-let
1019 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
1020 ;; A side-effect of creating a joint registers
1021 ;; it with both physics objects which in turn
1022 ;; will register the joint with the physics system
1023 ;; when the simulation is started.
1024 (joint-dispatch constraints
1025 control-a control-b
1026 pivot-a pivot-b
1027 joint-rotation))))
1028 #+end_src
1029 #+end_listing
1031 In general, whenever =CORTEX= exposes a sense (or in this case
1032 physicality), it provides a function of the type =sense!=, which
1033 takes in a collection of nodes and augments it to support that
1034 sense. The function returns any controlls necessary to use that
1035 sense. In this case =body!= cerates a physical body and returns no
1036 control functions.
1038 #+caption: Program to give joints to a creature.
1039 #+name: name
1040 #+begin_listing clojure
1041 #+begin_src clojure
1042 (defn joints!
1043 "Connect the solid parts of the creature with physical joints. The
1044 joints are taken from the \"joints\" node in the creature."
1045 [#^Node creature]
1046 (dorun
1047 (map
1048 (fn [joint]
1049 (let [[obj-a obj-b] (joint-targets creature joint)]
1050 (connect obj-a obj-b joint)))
1051 (joints creature))))
1052 (defn body!
1053 "Endow the creature with a physical body connected with joints. The
1054 particulars of the joints and the masses of each body part are
1055 determined in blender."
1056 [#^Node creature]
1057 (physical! creature)
1058 (joints! creature))
1059 #+end_src
1060 #+end_listing
1062 All of the code you have just seen amounts to only 130 lines, yet
1063 because it builds on top of Blender and jMonkeyEngine3, those few
1064 lines pack quite a punch!
1066 The hand from figure \ref{blender-hand}, which was modeled after
1067 my own right hand, can now be given joints and simulated as a
1068 creature.
1070 #+caption: With the ability to create physical creatures from blender,
1071 #+caption: =CORTEX= gets one step closer to becomming a full creature
1072 #+caption: simulation environment.
1073 #+name: name
1074 #+ATTR_LaTeX: :width 15cm
1075 [[./images/physical-hand.png]]
1077 ** Sight reuses standard video game components...
1079 Vision is one of the most important senses for humans, so I need to
1080 build a simulated sense of vision for my AI. I will do this with
1081 simulated eyes. Each eye can be independently moved and should see
1082 its own version of the world depending on where it is.
1084 Making these simulated eyes a reality is simple because
1085 jMonkeyEngine already contains extensive support for multiple views
1086 of the same 3D simulated world. The reason jMonkeyEngine has this
1087 support is because the support is necessary to create games with
1088 split-screen views. Multiple views are also used to create
1089 efficient pseudo-reflections by rendering the scene from a certain
1090 perspective and then projecting it back onto a surface in the 3D
1091 world.
1093 #+caption: jMonkeyEngine supports multiple views to enable
1094 #+caption: split-screen games, like GoldenEye, which was one of
1095 #+caption: the first games to use split-screen views.
1096 #+name: name
1097 #+ATTR_LaTeX: :width 10cm
1098 [[./images/goldeneye-4-player.png]]
1100 *** A Brief Description of jMonkeyEngine's Rendering Pipeline
1102 jMonkeyEngine allows you to create a =ViewPort=, which represents a
1103 view of the simulated world. You can create as many of these as you
1104 want. Every frame, the =RenderManager= iterates through each
1105 =ViewPort=, rendering the scene in the GPU. For each =ViewPort= there
1106 is a =FrameBuffer= which represents the rendered image in the GPU.
1108 #+caption: =ViewPorts= are cameras in the world. During each frame,
1109 #+caption: the =RenderManager= records a snapshot of what each view
1110 #+caption: is currently seeing; these snapshots are =FrameBuffer= objects.
1111 #+name: rendermanagers
1112 #+ATTR_LaTeX: :width 10cm
1113 [[./images/diagram_rendermanager2.png]]
1115 Each =ViewPort= can have any number of attached =SceneProcessor=
1116 objects, which are called every time a new frame is rendered. A
1117 =SceneProcessor= receives its =ViewPort's= =FrameBuffer= and can do
1118 whatever it wants to the data. Often this consists of invoking GPU
1119 specific operations on the rendered image. The =SceneProcessor= can
1120 also copy the GPU image data to RAM and process it with the CPU.
1122 *** Appropriating Views for Vision
1124 Each eye in the simulated creature needs its own =ViewPort= so
1125 that it can see the world from its own perspective. To this
1126 =ViewPort=, I add a =SceneProcessor= that feeds the visual data to
1127 any arbitrary continuation function for further processing. That
1128 continuation function may perform both CPU and GPU operations on
1129 the data. To make this easy for the continuation function, the
1130 =SceneProcessor= maintains appropriately sized buffers in RAM to
1131 hold the data. It does not do any copying from the GPU to the CPU
1132 itself because it is a slow operation.
1134 #+caption: Function to make the rendered secne in jMonkeyEngine
1135 #+caption: available for further processing.
1136 #+name: pipeline-1
1137 #+begin_listing clojure
1138 #+begin_src clojure
1139 (defn vision-pipeline
1140 "Create a SceneProcessor object which wraps a vision processing
1141 continuation function. The continuation is a function that takes
1142 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
1143 each of which has already been appropriately sized."
1144 [continuation]
1145 (let [byte-buffer (atom nil)
1146 renderer (atom nil)
1147 image (atom nil)]
1148 (proxy [SceneProcessor] []
1149 (initialize
1150 [renderManager viewPort]
1151 (let [cam (.getCamera viewPort)
1152 width (.getWidth cam)
1153 height (.getHeight cam)]
1154 (reset! renderer (.getRenderer renderManager))
1155 (reset! byte-buffer
1156 (BufferUtils/createByteBuffer
1157 (* width height 4)))
1158 (reset! image (BufferedImage.
1159 width height
1160 BufferedImage/TYPE_4BYTE_ABGR))))
1161 (isInitialized [] (not (nil? @byte-buffer)))
1162 (reshape [_ _ _])
1163 (preFrame [_])
1164 (postQueue [_])
1165 (postFrame
1166 [#^FrameBuffer fb]
1167 (.clear @byte-buffer)
1168 (continuation @renderer fb @byte-buffer @image))
1169 (cleanup []))))
1170 #+end_src
1171 #+end_listing
1173 The continuation function given to =vision-pipeline= above will be
1174 given a =Renderer= and three containers for image data. The
1175 =FrameBuffer= references the GPU image data, but the pixel data
1176 can not be used directly on the CPU. The =ByteBuffer= and
1177 =BufferedImage= are initially "empty" but are sized to hold the
1178 data in the =FrameBuffer=. I call transferring the GPU image data
1179 to the CPU structures "mixing" the image data.
1181 *** Optical sensor arrays are described with images and referenced with metadata
1183 The vision pipeline described above handles the flow of rendered
1184 images. Now, =CORTEX= needs simulated eyes to serve as the source
1185 of these images.
1187 An eye is described in blender in the same way as a joint. They
1188 are zero dimensional empty objects with no geometry whose local
1189 coordinate system determines the orientation of the resulting eye.
1190 All eyes are children of a parent node named "eyes" just as all
1191 joints have a parent named "joints". An eye binds to the nearest
1192 physical object with =bind-sense=.
1194 #+caption: Here, the camera is created based on metadata on the
1195 #+caption: eye-node and attached to the nearest physical object
1196 #+caption: with =bind-sense=
1197 #+name: add-eye
1198 #+begin_listing clojure
1199 (defn add-eye!
1200 "Create a Camera centered on the current position of 'eye which
1201 follows the closest physical node in 'creature. The camera will
1202 point in the X direction and use the Z vector as up as determined
1203 by the rotation of these vectors in blender coordinate space. Use
1204 XZY rotation for the node in blender."
1205 [#^Node creature #^Spatial eye]
1206 (let [target (closest-node creature eye)
1207 [cam-width cam-height]
1208 ;;[640 480] ;; graphics card on laptop doesn't support
1209 ;; arbitray dimensions.
1210 (eye-dimensions eye)
1211 cam (Camera. cam-width cam-height)
1212 rot (.getWorldRotation eye)]
1213 (.setLocation cam (.getWorldTranslation eye))
1214 (.lookAtDirection
1215 cam ; this part is not a mistake and
1216 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
1217 (.mult rot Vector3f/UNIT_Y)) ; blender as the UP vector.
1218 (.setFrustumPerspective
1219 cam (float 45)
1220 (float (/ (.getWidth cam) (.getHeight cam)))
1221 (float 1)
1222 (float 1000))
1223 (bind-sense target cam) cam))
1224 #+end_listing
1226 *** Simulated Retina
1228 An eye is a surface (the retina) which contains many discrete
1229 sensors to detect light. These sensors can have different
1230 light-sensing properties. In humans, each discrete sensor is
1231 sensitive to red, blue, green, or gray. These different types of
1232 sensors can have different spatial distributions along the retina.
1233 In humans, there is a fovea in the center of the retina which has
1234 a very high density of color sensors, and a blind spot which has
1235 no sensors at all. Sensor density decreases in proportion to
1236 distance from the fovea.
1238 I want to be able to model any retinal configuration, so my
1239 eye-nodes in blender contain metadata pointing to images that
1240 describe the precise position of the individual sensors using
1241 white pixels. The meta-data also describes the precise sensitivity
1242 to light that the sensors described in the image have. An eye can
1243 contain any number of these images. For example, the metadata for
1244 an eye might look like this:
1246 #+begin_src clojure
1247 {0xFF0000 "Models/test-creature/retina-small.png"}
1248 #+end_src
1250 #+caption: An example retinal profile image. White pixels are
1251 #+caption: photo-sensitive elements. The distribution of white
1252 #+caption: pixels is denser in the middle and falls off at the
1253 #+caption: edges and is inspired by the human retina.
1254 #+name: retina
1255 #+ATTR_LaTeX: :width 7cm
1256 [[./images/retina-small.png]]
1258 Together, the number 0xFF0000 and the image image above describe
1259 the placement of red-sensitive sensory elements.
1261 Meta-data to very crudely approximate a human eye might be
1262 something like this:
1264 #+begin_src clojure
1265 (let [retinal-profile "Models/test-creature/retina-small.png"]
1266 {0xFF0000 retinal-profile
1267 0x00FF00 retinal-profile
1268 0x0000FF retinal-profile
1269 0xFFFFFF retinal-profile})
1270 #+end_src
1272 The numbers that serve as keys in the map determine a sensor's
1273 relative sensitivity to the channels red, green, and blue. These
1274 sensitivity values are packed into an integer in the order
1275 =|_|R|G|B|= in 8-bit fields. The RGB values of a pixel in the
1276 image are added together with these sensitivities as linear
1277 weights. Therefore, 0xFF0000 means sensitive to red only while
1278 0xFFFFFF means sensitive to all colors equally (gray).
1280 #+caption: This is the core of vision in =CORTEX=. A given eye node
1281 #+caption: is converted into a function that returns visual
1282 #+caption: information from the simulation.
1283 #+name: vision-kernel
1284 #+begin_listing clojure
1285 #+BEGIN_SRC clojure
1286 (defn vision-kernel
1287 "Returns a list of functions, each of which will return a color
1288 channel's worth of visual information when called inside a running
1289 simulation."
1290 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
1291 (let [retinal-map (retina-sensor-profile eye)
1292 camera (add-eye! creature eye)
1293 vision-image
1294 (atom
1295 (BufferedImage. (.getWidth camera)
1296 (.getHeight camera)
1297 BufferedImage/TYPE_BYTE_BINARY))
1298 register-eye!
1299 (runonce
1300 (fn [world]
1301 (add-camera!
1302 world camera
1303 (let [counter (atom 0)]
1304 (fn [r fb bb bi]
1305 (if (zero? (rem (swap! counter inc) (inc skip)))
1306 (reset! vision-image
1307 (BufferedImage! r fb bb bi))))))))]
1308 (vec
1309 (map
1310 (fn [[key image]]
1311 (let [whites (white-coordinates image)
1312 topology (vec (collapse whites))
1313 sensitivity (sensitivity-presets key key)]
1314 (attached-viewport.
1315 (fn [world]
1316 (register-eye! world)
1317 (vector
1318 topology
1319 (vec
1320 (for [[x y] whites]
1321 (pixel-sense
1322 sensitivity
1323 (.getRGB @vision-image x y))))))
1324 register-eye!)))
1325 retinal-map))))
1326 #+END_SRC
1327 #+end_listing
1329 Note that since each of the functions generated by =vision-kernel=
1330 shares the same =register-eye!= function, the eye will be
1331 registered only once the first time any of the functions from the
1332 list returned by =vision-kernel= is called. Each of the functions
1333 returned by =vision-kernel= also allows access to the =Viewport=
1334 through which it receives images.
1336 All the hard work has been done; all that remains is to apply
1337 =vision-kernel= to each eye in the creature and gather the results
1338 into one list of functions.
1341 #+caption: With =vision!=, =CORTEX= is already a fine simulation
1342 #+caption: environment for experimenting with different types of
1343 #+caption: eyes.
1344 #+name: vision!
1345 #+begin_listing clojure
1346 #+BEGIN_SRC clojure
1347 (defn vision!
1348 "Returns a list of functions, each of which returns visual sensory
1349 data when called inside a running simulation."
1350 [#^Node creature & {skip :skip :or {skip 0}}]
1351 (reduce
1352 concat
1353 (for [eye (eyes creature)]
1354 (vision-kernel creature eye))))
1355 #+END_SRC
1356 #+end_listing
1358 #+caption: Simulated vision with a test creature and the
1359 #+caption: human-like eye approximation. Notice how each channel
1360 #+caption: of the eye responds differently to the differently
1361 #+caption: colored balls.
1362 #+name: worm-vision-test.
1363 #+ATTR_LaTeX: :width 13cm
1364 [[./images/worm-vision.png]]
1366 The vision code is not much more complicated than the body code,
1367 and enables multiple further paths for simulated vision. For
1368 example, it is quite easy to create bifocal vision -- you just
1369 make two eyes next to each other in blender! It is also possible
1370 to encode vision transforms in the retinal files. For example, the
1371 human like retina file in figure \ref{retina} approximates a
1372 log-polar transform.
1374 This vision code has already been absorbed by the jMonkeyEngine
1375 community and is now (in modified form) part of a system for
1376 capturing in-game video to a file.
1378 ** ...but hearing must be built from scratch
1380 At the end of this section I will have simulated ears that work the
1381 same way as the simulated eyes in the last section. I will be able to
1382 place any number of ear-nodes in a blender file, and they will bind to
1383 the closest physical object and follow it as it moves around. Each ear
1384 will provide access to the sound data it picks up between every frame.
1386 Hearing is one of the more difficult senses to simulate, because there
1387 is less support for obtaining the actual sound data that is processed
1388 by jMonkeyEngine3. There is no "split-screen" support for rendering
1389 sound from different points of view, and there is no way to directly
1390 access the rendered sound data.
1392 =CORTEX='s hearing is unique because it does not have any
1393 limitations compared to other simulation environments. As far as I
1394 know, there is no other system that supports multiple listerers,
1395 and the sound demo at the end of this section is the first time
1396 it's been done in a video game environment.
1398 *** Brief Description of jMonkeyEngine's Sound System
1400 jMonkeyEngine's sound system works as follows:
1402 - jMonkeyEngine uses the =AppSettings= for the particular
1403 application to determine what sort of =AudioRenderer= should be
1404 used.
1405 - Although some support is provided for multiple AudioRendering
1406 backends, jMonkeyEngine at the time of this writing will either
1407 pick no =AudioRenderer= at all, or the =LwjglAudioRenderer=.
1408 - jMonkeyEngine tries to figure out what sort of system you're
1409 running and extracts the appropriate native libraries.
1410 - The =LwjglAudioRenderer= uses the [[http://lwjgl.org/][=LWJGL=]] (LightWeight Java Game
1411 Library) bindings to interface with a C library called [[http://kcat.strangesoft.net/openal.html][=OpenAL=]]
1412 - =OpenAL= renders the 3D sound and feeds the rendered sound
1413 directly to any of various sound output devices with which it
1414 knows how to communicate.
1416 A consequence of this is that there's no way to access the actual
1417 sound data produced by =OpenAL=. Even worse, =OpenAL= only supports
1418 one /listener/ (it renders sound data from only one perspective),
1419 which normally isn't a problem for games, but becomes a problem
1420 when trying to make multiple AI creatures that can each hear the
1421 world from a different perspective.
1423 To make many AI creatures in jMonkeyEngine that can each hear the
1424 world from their own perspective, or to make a single creature with
1425 many ears, it is necessary to go all the way back to =OpenAL= and
1426 implement support for simulated hearing there.
1428 *** Extending =OpenAl=
1430 Extending =OpenAL= to support multiple listeners requires 500
1431 lines of =C= code and is too hairy to mention here. Instead, I
1432 will show a small amount of extension code and go over the high
1433 level stragety. Full source is of course available with the
1434 =CORTEX= distribution if you're interested.
1436 =OpenAL= goes to great lengths to support many different systems,
1437 all with different sound capabilities and interfaces. It
1438 accomplishes this difficult task by providing code for many
1439 different sound backends in pseudo-objects called /Devices/.
1440 There's a device for the Linux Open Sound System and the Advanced
1441 Linux Sound Architecture, there's one for Direct Sound on Windows,
1442 and there's even one for Solaris. =OpenAL= solves the problem of
1443 platform independence by providing all these Devices.
1445 Wrapper libraries such as LWJGL are free to examine the system on
1446 which they are running and then select an appropriate device for
1447 that system.
1449 There are also a few "special" devices that don't interface with
1450 any particular system. These include the Null Device, which
1451 doesn't do anything, and the Wave Device, which writes whatever
1452 sound it receives to a file, if everything has been set up
1453 correctly when configuring =OpenAL=.
1455 Actual mixing (doppler shift and distance.environment-based
1456 attenuation) of the sound data happens in the Devices, and they
1457 are the only point in the sound rendering process where this data
1458 is available.
1460 Therefore, in order to support multiple listeners, and get the
1461 sound data in a form that the AIs can use, it is necessary to
1462 create a new Device which supports this feature.
1464 Adding a device to OpenAL is rather tricky -- there are five
1465 separate files in the =OpenAL= source tree that must be modified
1466 to do so. I named my device the "Multiple Audio Send" Device, or
1467 =Send= Device for short, since it sends audio data back to the
1468 calling application like an Aux-Send cable on a mixing board.
1470 The main idea behind the Send device is to take advantage of the
1471 fact that LWJGL only manages one /context/ when using OpenAL. A
1472 /context/ is like a container that holds samples and keeps track
1473 of where the listener is. In order to support multiple listeners,
1474 the Send device identifies the LWJGL context as the master
1475 context, and creates any number of slave contexts to represent
1476 additional listeners. Every time the device renders sound, it
1477 synchronizes every source from the master LWJGL context to the
1478 slave contexts. Then, it renders each context separately, using a
1479 different listener for each one. The rendered sound is made
1480 available via JNI to jMonkeyEngine.
1482 Switching between contexts is not the normal operation of a
1483 Device, and one of the problems with doing so is that a Device
1484 normally keeps around a few pieces of state such as the
1485 =ClickRemoval= array above which will become corrupted if the
1486 contexts are not rendered in parallel. The solution is to create a
1487 copy of this normally global device state for each context, and
1488 copy it back and forth into and out of the actual device state
1489 whenever a context is rendered.
1491 The core of the =Send= device is the =syncSources= function, which
1492 does the job of copying all relevant data from one context to
1493 another.
1495 #+caption: Program for extending =OpenAL= to support multiple
1496 #+caption: listeners via context copying/switching.
1497 #+name: sync-openal-sources
1498 #+begin_listing c
1499 #+BEGIN_SRC c
1500 void syncSources(ALsource *masterSource, ALsource *slaveSource,
1501 ALCcontext *masterCtx, ALCcontext *slaveCtx){
1502 ALuint master = masterSource->source;
1503 ALuint slave = slaveSource->source;
1504 ALCcontext *current = alcGetCurrentContext();
1506 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
1507 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
1508 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
1509 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
1510 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
1511 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
1512 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
1513 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
1514 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
1515 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
1516 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
1517 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
1518 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
1520 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
1521 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
1522 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
1524 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
1525 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
1527 alcMakeContextCurrent(masterCtx);
1528 ALint source_type;
1529 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
1531 // Only static sources are currently synchronized!
1532 if (AL_STATIC == source_type){
1533 ALint master_buffer;
1534 ALint slave_buffer;
1535 alGetSourcei(master, AL_BUFFER, &master_buffer);
1536 alcMakeContextCurrent(slaveCtx);
1537 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
1538 if (master_buffer != slave_buffer){
1539 alSourcei(slave, AL_BUFFER, master_buffer);
1543 // Synchronize the state of the two sources.
1544 alcMakeContextCurrent(masterCtx);
1545 ALint masterState;
1546 ALint slaveState;
1548 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
1549 alcMakeContextCurrent(slaveCtx);
1550 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
1552 if (masterState != slaveState){
1553 switch (masterState){
1554 case AL_INITIAL : alSourceRewind(slave); break;
1555 case AL_PLAYING : alSourcePlay(slave); break;
1556 case AL_PAUSED : alSourcePause(slave); break;
1557 case AL_STOPPED : alSourceStop(slave); break;
1560 // Restore whatever context was previously active.
1561 alcMakeContextCurrent(current);
1563 #+END_SRC
1564 #+end_listing
1566 With this special context-switching device, and some ugly JNI
1567 bindings that are not worth mentioning, =CORTEX= gains the ability
1568 to access multiple sound streams from =OpenAL=.
1570 #+caption: Program to create an ear from a blender empty node. The ear
1571 #+caption: follows around the nearest physical object and passes
1572 #+caption: all sensory data to a continuation function.
1573 #+name: add-ear
1574 #+begin_listing clojure
1575 #+BEGIN_SRC clojure
1576 (defn add-ear!
1577 "Create a Listener centered on the current position of 'ear
1578 which follows the closest physical node in 'creature and
1579 sends sound data to 'continuation."
1580 [#^Application world #^Node creature #^Spatial ear continuation]
1581 (let [target (closest-node creature ear)
1582 lis (Listener.)
1583 audio-renderer (.getAudioRenderer world)
1584 sp (hearing-pipeline continuation)]
1585 (.setLocation lis (.getWorldTranslation ear))
1586 (.setRotation lis (.getWorldRotation ear))
1587 (bind-sense target lis)
1588 (update-listener-velocity! target lis)
1589 (.addListener audio-renderer lis)
1590 (.registerSoundProcessor audio-renderer lis sp)))
1591 #+END_SRC
1592 #+end_listing
1594 The =Send= device, unlike most of the other devices in =OpenAL=,
1595 does not render sound unless asked. This enables the system to
1596 slow down or speed up depending on the needs of the AIs who are
1597 using it to listen. If the device tried to render samples in
1598 real-time, a complicated AI whose mind takes 100 seconds of
1599 computer time to simulate 1 second of AI-time would miss almost
1600 all of the sound in its environment!
1602 #+caption: Program to enable arbitrary hearing in =CORTEX=
1603 #+name: hearing
1604 #+begin_listing clojure
1605 #+BEGIN_SRC clojure
1606 (defn hearing-kernel
1607 "Returns a function which returns auditory sensory data when called
1608 inside a running simulation."
1609 [#^Node creature #^Spatial ear]
1610 (let [hearing-data (atom [])
1611 register-listener!
1612 (runonce
1613 (fn [#^Application world]
1614 (add-ear!
1615 world creature ear
1616 (comp #(reset! hearing-data %)
1617 byteBuffer->pulse-vector))))]
1618 (fn [#^Application world]
1619 (register-listener! world)
1620 (let [data @hearing-data
1621 topology
1622 (vec (map #(vector % 0) (range 0 (count data))))]
1623 [topology data]))))
1625 (defn hearing!
1626 "Endow the creature in a particular world with the sense of
1627 hearing. Will return a sequence of functions, one for each ear,
1628 which when called will return the auditory data from that ear."
1629 [#^Node creature]
1630 (for [ear (ears creature)]
1631 (hearing-kernel creature ear)))
1632 #+END_SRC
1633 #+end_listing
1635 Armed with these functions, =CORTEX= is able to test possibly the
1636 first ever instance of multiple listeners in a video game engine
1637 based simulation!
1639 #+caption: Here a simple creature responds to sound by changing
1640 #+caption: its color from gray to green when the total volume
1641 #+caption: goes over a threshold.
1642 #+name: sound-test
1643 #+begin_listing java
1644 #+BEGIN_SRC java
1645 /**
1646 * Respond to sound! This is the brain of an AI entity that
1647 * hears its surroundings and reacts to them.
1648 */
1649 public void process(ByteBuffer audioSamples,
1650 int numSamples, AudioFormat format) {
1651 audioSamples.clear();
1652 byte[] data = new byte[numSamples];
1653 float[] out = new float[numSamples];
1654 audioSamples.get(data);
1655 FloatSampleTools.
1656 byte2floatInterleaved
1657 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
1659 float max = Float.NEGATIVE_INFINITY;
1660 for (float f : out){if (f > max) max = f;}
1661 audioSamples.clear();
1663 if (max > 0.1){
1664 entity.getMaterial().setColor("Color", ColorRGBA.Green);
1666 else {
1667 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
1669 #+END_SRC
1670 #+end_listing
1672 #+caption: First ever simulation of multiple listerners in =CORTEX=.
1673 #+caption: Each cube is a creature which processes sound data with
1674 #+caption: the =process= function from listing \ref{sound-test}.
1675 #+caption: the ball is constantally emiting a pure tone of
1676 #+caption: constant volume. As it approaches the cubes, they each
1677 #+caption: change color in response to the sound.
1678 #+name: sound-cubes.
1679 #+ATTR_LaTeX: :width 10cm
1680 [[./images/java-hearing-test.png]]
1682 This system of hearing has also been co-opted by the
1683 jMonkeyEngine3 community and is used to record audio for demo
1684 videos.
1686 ** Hundreds of hair-like elements provide a sense of touch
1688 Touch is critical to navigation and spatial reasoning and as such I
1689 need a simulated version of it to give to my AI creatures.
1691 Human skin has a wide array of touch sensors, each of which
1692 specialize in detecting different vibrational modes and pressures.
1693 These sensors can integrate a vast expanse of skin (i.e. your
1694 entire palm), or a tiny patch of skin at the tip of your finger.
1695 The hairs of the skin help detect objects before they even come
1696 into contact with the skin proper.
1698 However, touch in my simulated world can not exactly correspond to
1699 human touch because my creatures are made out of completely rigid
1700 segments that don't deform like human skin.
1702 Instead of measuring deformation or vibration, I surround each
1703 rigid part with a plenitude of hair-like objects (/feelers/) which
1704 do not interact with the physical world. Physical objects can pass
1705 through them with no effect. The feelers are able to tell when
1706 other objects pass through them, and they constantly report how
1707 much of their extent is covered. So even though the creature's body
1708 parts do not deform, the feelers create a margin around those body
1709 parts which achieves a sense of touch which is a hybrid between a
1710 human's sense of deformation and sense from hairs.
1712 Implementing touch in jMonkeyEngine follows a different technical
1713 route than vision and hearing. Those two senses piggybacked off
1714 jMonkeyEngine's 3D audio and video rendering subsystems. To
1715 simulate touch, I use jMonkeyEngine's physics system to execute
1716 many small collision detections, one for each feeler. The placement
1717 of the feelers is determined by a UV-mapped image which shows where
1718 each feeler should be on the 3D surface of the body.
1720 *** Defining Touch Meta-Data in Blender
1722 Each geometry can have a single UV map which describes the
1723 position of the feelers which will constitute its sense of touch.
1724 This image path is stored under the ``touch'' key. The image itself
1725 is black and white, with black meaning a feeler length of 0 (no
1726 feeler is present) and white meaning a feeler length of =scale=,
1727 which is a float stored under the key "scale".
1729 #+caption: Touch does not use empty nodes, to store metadata,
1730 #+caption: because the metadata of each solid part of a
1731 #+caption: creature's body is sufficient.
1732 #+name: touch-meta-data
1733 #+begin_listing clojure
1734 #+BEGIN_SRC clojure
1735 (defn tactile-sensor-profile
1736 "Return the touch-sensor distribution image in BufferedImage format,
1737 or nil if it does not exist."
1738 [#^Geometry obj]
1739 (if-let [image-path (meta-data obj "touch")]
1740 (load-image image-path)))
1742 (defn tactile-scale
1743 "Return the length of each feeler. Default scale is 0.01
1744 jMonkeyEngine units."
1745 [#^Geometry obj]
1746 (if-let [scale (meta-data obj "scale")]
1747 scale 0.1))
1748 #+END_SRC
1749 #+end_listing
1751 Here is an example of a UV-map which specifies the position of
1752 touch sensors along the surface of the upper segment of a fingertip.
1754 #+caption: This is the tactile-sensor-profile for the upper segment
1755 #+caption: of a fingertip. It defines regions of high touch sensitivity
1756 #+caption: (where there are many white pixels) and regions of low
1757 #+caption: sensitivity (where white pixels are sparse).
1758 #+name: fingertip-UV
1759 #+ATTR_LaTeX: :width 13cm
1760 [[./images/finger-UV.png]]
1762 *** Implementation Summary
1764 To simulate touch there are three conceptual steps. For each solid
1765 object in the creature, you first have to get UV image and scale
1766 parameter which define the position and length of the feelers.
1767 Then, you use the triangles which comprise the mesh and the UV
1768 data stored in the mesh to determine the world-space position and
1769 orientation of each feeler. Then once every frame, update these
1770 positions and orientations to match the current position and
1771 orientation of the object, and use physics collision detection to
1772 gather tactile data.
1774 Extracting the meta-data has already been described. The third
1775 step, physics collision detection, is handled in =touch-kernel=.
1776 Translating the positions and orientations of the feelers from the
1777 UV-map to world-space is itself a three-step process.
1779 - Find the triangles which make up the mesh in pixel-space and in
1780 world-space. \\(=triangles=, =pixel-triangles=).
1782 - Find the coordinates of each feeler in world-space. These are
1783 the origins of the feelers. (=feeler-origins=).
1785 - Calculate the normals of the triangles in world space, and add
1786 them to each of the origins of the feelers. These are the
1787 normalized coordinates of the tips of the feelers.
1788 (=feeler-tips=).
1790 *** Triangle Math
1792 The rigid objects which make up a creature have an underlying
1793 =Geometry=, which is a =Mesh= plus a =Material= and other
1794 important data involved with displaying the object.
1796 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
1797 vertices which have coordinates in world space and UV space.
1799 Here, =triangles= gets all the world-space triangles which
1800 comprise a mesh, while =pixel-triangles= gets those same triangles
1801 expressed in pixel coordinates (which are UV coordinates scaled to
1802 fit the height and width of the UV image).
1804 #+caption: Programs to extract triangles from a geometry and get
1805 #+caption: their verticies in both world and UV-coordinates.
1806 #+name: get-triangles
1807 #+begin_listing clojure
1808 #+BEGIN_SRC clojure
1809 (defn triangle
1810 "Get the triangle specified by triangle-index from the mesh."
1811 [#^Geometry geo triangle-index]
1812 (triangle-seq
1813 (let [scratch (Triangle.)]
1814 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
1816 (defn triangles
1817 "Return a sequence of all the Triangles which comprise a given
1818 Geometry."
1819 [#^Geometry geo]
1820 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
1822 (defn triangle-vertex-indices
1823 "Get the triangle vertex indices of a given triangle from a given
1824 mesh."
1825 [#^Mesh mesh triangle-index]
1826 (let [indices (int-array 3)]
1827 (.getTriangle mesh triangle-index indices)
1828 (vec indices)))
1830 (defn vertex-UV-coord
1831 "Get the UV-coordinates of the vertex named by vertex-index"
1832 [#^Mesh mesh vertex-index]
1833 (let [UV-buffer
1834 (.getData
1835 (.getBuffer
1836 mesh
1837 VertexBuffer$Type/TexCoord))]
1838 [(.get UV-buffer (* vertex-index 2))
1839 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
1841 (defn pixel-triangle [#^Geometry geo image index]
1842 (let [mesh (.getMesh geo)
1843 width (.getWidth image)
1844 height (.getHeight image)]
1845 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
1846 (map (partial vertex-UV-coord mesh)
1847 (triangle-vertex-indices mesh index))))))
1849 (defn pixel-triangles
1850 "The pixel-space triangles of the Geometry, in the same order as
1851 (triangles geo)"
1852 [#^Geometry geo image]
1853 (let [height (.getHeight image)
1854 width (.getWidth image)]
1855 (map (partial pixel-triangle geo image)
1856 (range (.getTriangleCount (.getMesh geo))))))
1857 #+END_SRC
1858 #+end_listing
1860 *** The Affine Transform from one Triangle to Another
1862 =pixel-triangles= gives us the mesh triangles expressed in pixel
1863 coordinates and =triangles= gives us the mesh triangles expressed
1864 in world coordinates. The tactile-sensor-profile gives the
1865 position of each feeler in pixel-space. In order to convert
1866 pixel-space coordinates into world-space coordinates we need
1867 something that takes coordinates on the surface of one triangle
1868 and gives the corresponding coordinates on the surface of another
1869 triangle.
1871 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed
1872 into any other by a combination of translation, scaling, and
1873 rotation. The affine transformation from one triangle to another
1874 is readily computable if the triangle is expressed in terms of a
1875 $4x4$ matrix.
1877 #+BEGIN_LaTeX
1878 $$
1879 \begin{bmatrix}
1880 x_1 & x_2 & x_3 & n_x \\
1881 y_1 & y_2 & y_3 & n_y \\
1882 z_1 & z_2 & z_3 & n_z \\
1883 1 & 1 & 1 & 1
1884 \end{bmatrix}
1885 $$
1886 #+END_LaTeX
1888 Here, the first three columns of the matrix are the vertices of
1889 the triangle. The last column is the right-handed unit normal of
1890 the triangle.
1892 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
1893 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
1894 is $T_{2}T_{1}^{-1}$.
1896 The clojure code below recapitulates the formulas above, using
1897 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
1898 transformation.
1900 #+caption: Program to interpert triangles as affine transforms.
1901 #+name: triangle-affine
1902 #+begin_listing clojure
1903 #+BEGIN_SRC clojure
1904 (defn triangle->matrix4f
1905 "Converts the triangle into a 4x4 matrix: The first three columns
1906 contain the vertices of the triangle; the last contains the unit
1907 normal of the triangle. The bottom row is filled with 1s."
1908 [#^Triangle t]
1909 (let [mat (Matrix4f.)
1910 [vert-1 vert-2 vert-3]
1911 (mapv #(.get t %) (range 3))
1912 unit-normal (do (.calculateNormal t)(.getNormal t))
1913 vertices [vert-1 vert-2 vert-3 unit-normal]]
1914 (dorun
1915 (for [row (range 4) col (range 3)]
1916 (do
1917 (.set mat col row (.get (vertices row) col))
1918 (.set mat 3 row 1)))) mat))
1920 (defn triangles->affine-transform
1921 "Returns the affine transformation that converts each vertex in the
1922 first triangle into the corresponding vertex in the second
1923 triangle."
1924 [#^Triangle tri-1 #^Triangle tri-2]
1925 (.mult
1926 (triangle->matrix4f tri-2)
1927 (.invert (triangle->matrix4f tri-1))))
1928 #+END_SRC
1929 #+end_listing
1931 *** Triangle Boundaries
1933 For efficiency's sake I will divide the tactile-profile image into
1934 small squares which inscribe each pixel-triangle, then extract the
1935 points which lie inside the triangle and map them to 3D-space using
1936 =triangle-transform= above. To do this I need a function,
1937 =convex-bounds= which finds the smallest box which inscribes a 2D
1938 triangle.
1940 =inside-triangle?= determines whether a point is inside a triangle
1941 in 2D pixel-space.
1943 #+caption: Program to efficiently determine point includion
1944 #+caption: in a triangle.
1945 #+name: in-triangle
1946 #+begin_listing clojure
1947 #+BEGIN_SRC clojure
1948 (defn convex-bounds
1949 "Returns the smallest square containing the given vertices, as a
1950 vector of integers [left top width height]."
1951 [verts]
1952 (let [xs (map first verts)
1953 ys (map second verts)
1954 x0 (Math/floor (apply min xs))
1955 y0 (Math/floor (apply min ys))
1956 x1 (Math/ceil (apply max xs))
1957 y1 (Math/ceil (apply max ys))]
1958 [x0 y0 (- x1 x0) (- y1 y0)]))
1960 (defn same-side?
1961 "Given the points p1 and p2 and the reference point ref, is point p
1962 on the same side of the line that goes through p1 and p2 as ref is?"
1963 [p1 p2 ref p]
1964 (<=
1966 (.dot
1967 (.cross (.subtract p2 p1) (.subtract p p1))
1968 (.cross (.subtract p2 p1) (.subtract ref p1)))))
1970 (defn inside-triangle?
1971 "Is the point inside the triangle?"
1972 {:author "Dylan Holmes"}
1973 [#^Triangle tri #^Vector3f p]
1974 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
1975 (and
1976 (same-side? vert-1 vert-2 vert-3 p)
1977 (same-side? vert-2 vert-3 vert-1 p)
1978 (same-side? vert-3 vert-1 vert-2 p))))
1979 #+END_SRC
1980 #+end_listing
1982 *** Feeler Coordinates
1984 The triangle-related functions above make short work of
1985 calculating the positions and orientations of each feeler in
1986 world-space.
1988 #+caption: Program to get the coordinates of ``feelers '' in
1989 #+caption: both world and UV-coordinates.
1990 #+name: feeler-coordinates
1991 #+begin_listing clojure
1992 #+BEGIN_SRC clojure
1993 (defn feeler-pixel-coords
1994 "Returns the coordinates of the feelers in pixel space in lists, one
1995 list for each triangle, ordered in the same way as (triangles) and
1996 (pixel-triangles)."
1997 [#^Geometry geo image]
1998 (map
1999 (fn [pixel-triangle]
2000 (filter
2001 (fn [coord]
2002 (inside-triangle? (->triangle pixel-triangle)
2003 (->vector3f coord)))
2004 (white-coordinates image (convex-bounds pixel-triangle))))
2005 (pixel-triangles geo image)))
2007 (defn feeler-world-coords
2008 "Returns the coordinates of the feelers in world space in lists, one
2009 list for each triangle, ordered in the same way as (triangles) and
2010 (pixel-triangles)."
2011 [#^Geometry geo image]
2012 (let [transforms
2013 (map #(triangles->affine-transform
2014 (->triangle %1) (->triangle %2))
2015 (pixel-triangles geo image)
2016 (triangles geo))]
2017 (map (fn [transform coords]
2018 (map #(.mult transform (->vector3f %)) coords))
2019 transforms (feeler-pixel-coords geo image))))
2020 #+END_SRC
2021 #+end_listing
2023 #+caption: Program to get the position of the base and tip of
2024 #+caption: each ``feeler''
2025 #+name: feeler-tips
2026 #+begin_listing clojure
2027 #+BEGIN_SRC clojure
2028 (defn feeler-origins
2029 "The world space coordinates of the root of each feeler."
2030 [#^Geometry geo image]
2031 (reduce concat (feeler-world-coords geo image)))
2033 (defn feeler-tips
2034 "The world space coordinates of the tip of each feeler."
2035 [#^Geometry geo image]
2036 (let [world-coords (feeler-world-coords geo image)
2037 normals
2038 (map
2039 (fn [triangle]
2040 (.calculateNormal triangle)
2041 (.clone (.getNormal triangle)))
2042 (map ->triangle (triangles geo)))]
2044 (mapcat (fn [origins normal]
2045 (map #(.add % normal) origins))
2046 world-coords normals)))
2048 (defn touch-topology
2049 [#^Geometry geo image]
2050 (collapse (reduce concat (feeler-pixel-coords geo image))))
2051 #+END_SRC
2052 #+end_listing
2054 *** Simulated Touch
2056 Now that the functions to construct feelers are complete,
2057 =touch-kernel= generates functions to be called from within a
2058 simulation that perform the necessary physics collisions to
2059 collect tactile data, and =touch!= recursively applies it to every
2060 node in the creature.
2062 #+caption: Efficient program to transform a ray from
2063 #+caption: one position to another.
2064 #+name: set-ray
2065 #+begin_listing clojure
2066 #+BEGIN_SRC clojure
2067 (defn set-ray [#^Ray ray #^Matrix4f transform
2068 #^Vector3f origin #^Vector3f tip]
2069 ;; Doing everything locally reduces garbage collection by enough to
2070 ;; be worth it.
2071 (.mult transform origin (.getOrigin ray))
2072 (.mult transform tip (.getDirection ray))
2073 (.subtractLocal (.getDirection ray) (.getOrigin ray))
2074 (.normalizeLocal (.getDirection ray)))
2075 #+END_SRC
2076 #+end_listing
2078 #+caption: This is the core of touch in =CORTEX= each feeler
2079 #+caption: follows the object it is bound to, reporting any
2080 #+caption: collisions that may happen.
2081 #+name: touch-kernel
2082 #+begin_listing clojure
2083 #+BEGIN_SRC clojure
2084 (defn touch-kernel
2085 "Constructs a function which will return tactile sensory data from
2086 'geo when called from inside a running simulation"
2087 [#^Geometry geo]
2088 (if-let
2089 [profile (tactile-sensor-profile geo)]
2090 (let [ray-reference-origins (feeler-origins geo profile)
2091 ray-reference-tips (feeler-tips geo profile)
2092 ray-length (tactile-scale geo)
2093 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
2094 topology (touch-topology geo profile)
2095 correction (float (* ray-length -0.2))]
2096 ;; slight tolerance for very close collisions.
2097 (dorun
2098 (map (fn [origin tip]
2099 (.addLocal origin (.mult (.subtract tip origin)
2100 correction)))
2101 ray-reference-origins ray-reference-tips))
2102 (dorun (map #(.setLimit % ray-length) current-rays))
2103 (fn [node]
2104 (let [transform (.getWorldMatrix geo)]
2105 (dorun
2106 (map (fn [ray ref-origin ref-tip]
2107 (set-ray ray transform ref-origin ref-tip))
2108 current-rays ray-reference-origins
2109 ray-reference-tips))
2110 (vector
2111 topology
2112 (vec
2113 (for [ray current-rays]
2114 (do
2115 (let [results (CollisionResults.)]
2116 (.collideWith node ray results)
2117 (let [touch-objects
2118 (filter #(not (= geo (.getGeometry %)))
2119 results)
2120 limit (.getLimit ray)]
2121 [(if (empty? touch-objects)
2122 limit
2123 (let [response
2124 (apply min (map #(.getDistance %)
2125 touch-objects))]
2126 (FastMath/clamp
2127 (float
2128 (if (> response limit) (float 0.0)
2129 (+ response correction)))
2130 (float 0.0)
2131 limit)))
2132 limit])))))))))))
2133 #+END_SRC
2134 #+end_listing
2136 Armed with the =touch!= function, =CORTEX= becomes capable of
2137 giving creatures a sense of touch. A simple test is to create a
2138 cube that is outfitted with a uniform distrubition of touch
2139 sensors. It can feel the ground and any balls that it touches.
2141 #+caption: =CORTEX= interface for creating touch in a simulated
2142 #+caption: creature.
2143 #+name: touch
2144 #+begin_listing clojure
2145 #+BEGIN_SRC clojure
2146 (defn touch!
2147 "Endow the creature with the sense of touch. Returns a sequence of
2148 functions, one for each body part with a tactile-sensor-profile,
2149 each of which when called returns sensory data for that body part."
2150 [#^Node creature]
2151 (filter
2152 (comp not nil?)
2153 (map touch-kernel
2154 (filter #(isa? (class %) Geometry)
2155 (node-seq creature)))))
2156 #+END_SRC
2157 #+end_listing
2159 The tactile-sensor-profile image for the touch cube is a simple
2160 cross with a unifom distribution of touch sensors:
2162 #+caption: The touch profile for the touch-cube. Each pure white
2163 #+caption: pixel defines a touch sensitive feeler.
2164 #+name: touch-cube-uv-map
2165 #+ATTR_LaTeX: :width 7cm
2166 [[./images/touch-profile.png]]
2168 #+caption: The touch cube reacts to canonballs. The black, red,
2169 #+caption: and white cross on the right is a visual display of
2170 #+caption: the creature's touch. White means that it is feeling
2171 #+caption: something strongly, black is not feeling anything,
2172 #+caption: and gray is in-between. The cube can feel both the
2173 #+caption: floor and the ball. Notice that when the ball causes
2174 #+caption: the cube to tip, that the bottom face can still feel
2175 #+caption: part of the ground.
2176 #+name: touch-cube-uv-map
2177 #+ATTR_LaTeX: :width 15cm
2178 [[./images/touch-cube.png]]
2180 ** Proprioception provides knowledge of your own body's position
2182 Close your eyes, and touch your nose with your right index finger.
2183 How did you do it? You could not see your hand, and neither your
2184 hand nor your nose could use the sense of touch to guide the path
2185 of your hand. There are no sound cues, and Taste and Smell
2186 certainly don't provide any help. You know where your hand is
2187 without your other senses because of Proprioception.
2189 Humans can sometimes loose this sense through viral infections or
2190 damage to the spinal cord or brain, and when they do, they loose
2191 the ability to control their own bodies without looking directly at
2192 the parts they want to move. In [[http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat][The Man Who Mistook His Wife for a
2193 Hat]], a woman named Christina looses this sense and has to learn how
2194 to move by carefully watching her arms and legs. She describes
2195 proprioception as the "eyes of the body, the way the body sees
2196 itself".
2198 Proprioception in humans is mediated by [[http://en.wikipedia.org/wiki/Articular_capsule][joint capsules]], [[http://en.wikipedia.org/wiki/Muscle_spindle][muscle
2199 spindles]], and the [[http://en.wikipedia.org/wiki/Golgi_tendon_organ][Golgi tendon organs]]. These measure the relative
2200 positions of each body part by monitoring muscle strain and length.
2202 It's clear that this is a vital sense for fluid, graceful movement.
2203 It's also particularly easy to implement in jMonkeyEngine.
2205 My simulated proprioception calculates the relative angles of each
2206 joint from the rest position defined in the blender file. This
2207 simulates the muscle-spindles and joint capsules. I will deal with
2208 Golgi tendon organs, which calculate muscle strain, in the next
2209 section.
2211 *** Helper functions
2213 =absolute-angle= calculates the angle between two vectors,
2214 relative to a third axis vector. This angle is the number of
2215 radians you have to move counterclockwise around the axis vector
2216 to get from the first to the second vector. It is not commutative
2217 like a normal dot-product angle is.
2219 The purpose of these functions is to build a system of angle
2220 measurement that is biologically plausable.
2222 #+caption: Program to measure angles along a vector
2223 #+name: helpers
2224 #+begin_listing clojure
2225 #+BEGIN_SRC clojure
2226 (defn right-handed?
2227 "true iff the three vectors form a right handed coordinate
2228 system. The three vectors do not have to be normalized or
2229 orthogonal."
2230 [vec1 vec2 vec3]
2231 (pos? (.dot (.cross vec1 vec2) vec3)))
2233 (defn absolute-angle
2234 "The angle between 'vec1 and 'vec2 around 'axis. In the range
2235 [0 (* 2 Math/PI)]."
2236 [vec1 vec2 axis]
2237 (let [angle (.angleBetween vec1 vec2)]
2238 (if (right-handed? vec1 vec2 axis)
2239 angle (- (* 2 Math/PI) angle))))
2240 #+END_SRC
2241 #+end_listing
2243 *** Proprioception Kernel
2245 Given a joint, =proprioception-kernel= produces a function that
2246 calculates the Euler angles between the the objects the joint
2247 connects. The only tricky part here is making the angles relative
2248 to the joint's initial ``straightness''.
2250 #+caption: Program to return biologially reasonable proprioceptive
2251 #+caption: data for each joint.
2252 #+name: proprioception
2253 #+begin_listing clojure
2254 #+BEGIN_SRC clojure
2255 (defn proprioception-kernel
2256 "Returns a function which returns proprioceptive sensory data when
2257 called inside a running simulation."
2258 [#^Node parts #^Node joint]
2259 (let [[obj-a obj-b] (joint-targets parts joint)
2260 joint-rot (.getWorldRotation joint)
2261 x0 (.mult joint-rot Vector3f/UNIT_X)
2262 y0 (.mult joint-rot Vector3f/UNIT_Y)
2263 z0 (.mult joint-rot Vector3f/UNIT_Z)]
2264 (fn []
2265 (let [rot-a (.clone (.getWorldRotation obj-a))
2266 rot-b (.clone (.getWorldRotation obj-b))
2267 x (.mult rot-a x0)
2268 y (.mult rot-a y0)
2269 z (.mult rot-a z0)
2271 X (.mult rot-b x0)
2272 Y (.mult rot-b y0)
2273 Z (.mult rot-b z0)
2274 heading (Math/atan2 (.dot X z) (.dot X x))
2275 pitch (Math/atan2 (.dot X y) (.dot X x))
2277 ;; rotate x-vector back to origin
2278 reverse
2279 (doto (Quaternion.)
2280 (.fromAngleAxis
2281 (.angleBetween X x)
2282 (let [cross (.normalize (.cross X x))]
2283 (if (= 0 (.length cross)) y cross))))
2284 roll (absolute-angle (.mult reverse Y) y x)]
2285 [heading pitch roll]))))
2287 (defn proprioception!
2288 "Endow the creature with the sense of proprioception. Returns a
2289 sequence of functions, one for each child of the \"joints\" node in
2290 the creature, which each report proprioceptive information about
2291 that joint."
2292 [#^Node creature]
2293 ;; extract the body's joints
2294 (let [senses (map (partial proprioception-kernel creature)
2295 (joints creature))]
2296 (fn []
2297 (map #(%) senses))))
2298 #+END_SRC
2299 #+end_listing
2301 =proprioception!= maps =proprioception-kernel= across all the
2302 joints of the creature. It uses the same list of joints that
2303 =joints= uses. Proprioception is the easiest sense to implement in
2304 =CORTEX=, and it will play a crucial role when efficiently
2305 implementing empathy.
2307 #+caption: In the upper right corner, the three proprioceptive
2308 #+caption: angle measurements are displayed. Red is yaw, Green is
2309 #+caption: pitch, and White is roll.
2310 #+name: proprio
2311 #+ATTR_LaTeX: :width 11cm
2312 [[./images/proprio.png]]
2314 ** Muscles contain both sensors and effectors
2316 Surprisingly enough, terrestrial creatures only move by using
2317 torque applied about their joints. There's not a single straight
2318 line of force in the human body at all! (A straight line of force
2319 would correspond to some sort of jet or rocket propulsion.)
2321 In humans, muscles are composed of muscle fibers which can contract
2322 to exert force. The muscle fibers which compose a muscle are
2323 partitioned into discrete groups which are each controlled by a
2324 single alpha motor neuron. A single alpha motor neuron might
2325 control as little as three or as many as one thousand muscle
2326 fibers. When the alpha motor neuron is engaged by the spinal cord,
2327 it activates all of the muscle fibers to which it is attached. The
2328 spinal cord generally engages the alpha motor neurons which control
2329 few muscle fibers before the motor neurons which control many
2330 muscle fibers. This recruitment strategy allows for precise
2331 movements at low strength. The collection of all motor neurons that
2332 control a muscle is called the motor pool. The brain essentially
2333 says "activate 30% of the motor pool" and the spinal cord recruits
2334 motor neurons until 30% are activated. Since the distribution of
2335 power among motor neurons is unequal and recruitment goes from
2336 weakest to strongest, the first 30% of the motor pool might be 5%
2337 of the strength of the muscle.
2339 My simulated muscles follow a similar design: Each muscle is
2340 defined by a 1-D array of numbers (the "motor pool"). Each entry in
2341 the array represents a motor neuron which controls a number of
2342 muscle fibers equal to the value of the entry. Each muscle has a
2343 scalar strength factor which determines the total force the muscle
2344 can exert when all motor neurons are activated. The effector
2345 function for a muscle takes a number to index into the motor pool,
2346 and then "activates" all the motor neurons whose index is lower or
2347 equal to the number. Each motor-neuron will apply force in
2348 proportion to its value in the array. Lower values cause less
2349 force. The lower values can be put at the "beginning" of the 1-D
2350 array to simulate the layout of actual human muscles, which are
2351 capable of more precise movements when exerting less force. Or, the
2352 motor pool can simulate more exotic recruitment strategies which do
2353 not correspond to human muscles.
2355 This 1D array is defined in an image file for ease of
2356 creation/visualization. Here is an example muscle profile image.
2358 #+caption: A muscle profile image that describes the strengths
2359 #+caption: of each motor neuron in a muscle. White is weakest
2360 #+caption: and dark red is strongest. This particular pattern
2361 #+caption: has weaker motor neurons at the beginning, just
2362 #+caption: like human muscle.
2363 #+name: muscle-recruit
2364 #+ATTR_LaTeX: :width 7cm
2365 [[./images/basic-muscle.png]]
2367 *** Muscle meta-data
2369 #+caption: Program to deal with loading muscle data from a blender
2370 #+caption: file's metadata.
2371 #+name: motor-pool
2372 #+begin_listing clojure
2373 #+BEGIN_SRC clojure
2374 (defn muscle-profile-image
2375 "Get the muscle-profile image from the node's blender meta-data."
2376 [#^Node muscle]
2377 (if-let [image (meta-data muscle "muscle")]
2378 (load-image image)))
2380 (defn muscle-strength
2381 "Return the strength of this muscle, or 1 if it is not defined."
2382 [#^Node muscle]
2383 (if-let [strength (meta-data muscle "strength")]
2384 strength 1))
2386 (defn motor-pool
2387 "Return a vector where each entry is the strength of the \"motor
2388 neuron\" at that part in the muscle."
2389 [#^Node muscle]
2390 (let [profile (muscle-profile-image muscle)]
2391 (vec
2392 (let [width (.getWidth profile)]
2393 (for [x (range width)]
2394 (- 255
2395 (bit-and
2396 0x0000FF
2397 (.getRGB profile x 0))))))))
2398 #+END_SRC
2399 #+end_listing
2401 Of note here is =motor-pool= which interprets the muscle-profile
2402 image in a way that allows me to use gradients between white and
2403 red, instead of shades of gray as I've been using for all the
2404 other senses. This is purely an aesthetic touch.
2406 *** Creating muscles
2408 #+caption: This is the core movement functoion in =CORTEX=, which
2409 #+caption: implements muscles that report on their activation.
2410 #+name: muscle-kernel
2411 #+begin_listing clojure
2412 #+BEGIN_SRC clojure
2413 (defn movement-kernel
2414 "Returns a function which when called with a integer value inside a
2415 running simulation will cause movement in the creature according
2416 to the muscle's position and strength profile. Each function
2417 returns the amount of force applied / max force."
2418 [#^Node creature #^Node muscle]
2419 (let [target (closest-node creature muscle)
2420 axis
2421 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
2422 strength (muscle-strength muscle)
2424 pool (motor-pool muscle)
2425 pool-integral (reductions + pool)
2426 forces
2427 (vec (map #(float (* strength (/ % (last pool-integral))))
2428 pool-integral))
2429 control (.getControl target RigidBodyControl)]
2430 ;;(println-repl (.getName target) axis)
2431 (fn [n]
2432 (let [pool-index (max 0 (min n (dec (count pool))))
2433 force (forces pool-index)]
2434 (.applyTorque control (.mult axis force))
2435 (float (/ force strength))))))
2437 (defn movement!
2438 "Endow the creature with the power of movement. Returns a sequence
2439 of functions, each of which accept an integer value and will
2440 activate their corresponding muscle."
2441 [#^Node creature]
2442 (for [muscle (muscles creature)]
2443 (movement-kernel creature muscle)))
2444 #+END_SRC
2445 #+end_listing
2448 =movement-kernel= creates a function that will move the nearest
2449 physical object to the muscle node. The muscle exerts a rotational
2450 force dependent on it's orientation to the object in the blender
2451 file. The function returned by =movement-kernel= is also a sense
2452 function: it returns the percent of the total muscle strength that
2453 is currently being employed. This is analogous to muscle tension
2454 in humans and completes the sense of proprioception begun in the
2455 last section.
2457 ** =CORTEX= brings complex creatures to life!
2459 The ultimate test of =CORTEX= is to create a creature with the full
2460 gamut of senses and put it though its paces.
2462 With all senses enabled, my right hand model looks like an
2463 intricate marionette hand with several strings for each finger:
2465 #+caption: View of the hand model with all sense nodes. You can see
2466 #+caption: the joint, muscle, ear, and eye nodess here.
2467 #+name: hand-nodes-1
2468 #+ATTR_LaTeX: :width 11cm
2469 [[./images/hand-with-all-senses2.png]]
2471 #+caption: An alternate view of the hand.
2472 #+name: hand-nodes-2
2473 #+ATTR_LaTeX: :width 15cm
2474 [[./images/hand-with-all-senses3.png]]
2476 With the hand fully rigged with senses, I can run it though a test
2477 that will test everything.
2479 #+caption: A full test of the hand with all senses. Note expecially
2480 #+caption: the interactions the hand has with itself: it feels
2481 #+caption: its own palm and fingers, and when it curls its fingers,
2482 #+caption: it sees them with its eye (which is located in the center
2483 #+caption: of the palm. The red block appears with a pure tone sound.
2484 #+caption: The hand then uses its muscles to launch the cube!
2485 #+name: integration
2486 #+ATTR_LaTeX: :width 16cm
2487 [[./images/integration.png]]
2489 ** =CORTEX= enables many possiblities for further research
2491 Often times, the hardest part of building a system involving
2492 creatures is dealing with physics and graphics. =CORTEX= removes
2493 much of this initial difficulty and leaves researchers free to
2494 directly pursue their ideas. I hope that even undergrads with a
2495 passing curiosity about simulated touch or creature evolution will
2496 be able to use cortex for experimentation. =CORTEX= is a completely
2497 simulated world, and far from being a disadvantage, its simulated
2498 nature enables you to create senses and creatures that would be
2499 impossible to make in the real world.
2501 While not by any means a complete list, here are some paths
2502 =CORTEX= is well suited to help you explore:
2504 - Empathy :: my empathy program leaves many areas for
2505 improvement, among which are using vision to infer
2506 proprioception and looking up sensory experience with imagined
2507 vision, touch, and sound.
2508 - Evolution :: Karl Sims created a rich environment for
2509 simulating the evolution of creatures on a connection
2510 machine. Today, this can be redone and expanded with =CORTEX=
2511 on an ordinary computer.
2512 - Exotic senses :: Cortex enables many fascinating senses that are
2513 not possible to build in the real world. For example,
2514 telekinesis is an interesting avenue to explore. You can also
2515 make a ``semantic'' sense which looks up metadata tags on
2516 objects in the environment the metadata tags might contain
2517 other sensory information.
2518 - Imagination via subworlds :: this would involve a creature with
2519 an effector which creates an entire new sub-simulation where
2520 the creature has direct control over placement/creation of
2521 objects via simulated telekinesis. The creature observes this
2522 sub-world through it's normal senses and uses its observations
2523 to make predictions about its top level world.
2524 - Simulated prescience :: step the simulation forward a few ticks,
2525 gather sensory data, then supply this data for the creature as
2526 one of its actual senses. The cost of prescience is slowing
2527 the simulation down by a factor proportional to however far
2528 you want the entities to see into the future. What happens
2529 when two evolved creatures that can each see into the future
2530 fight each other?
2531 - Swarm creatures :: Program a group of creatures that cooperate
2532 with each other. Because the creatures would be simulated, you
2533 could investigate computationally complex rules of behavior
2534 which still, from the group's point of view, would happen in
2535 ``real time''. Interactions could be as simple as cellular
2536 organisms communicating via flashing lights, or as complex as
2537 humanoids completing social tasks, etc.
2538 - =HACKER= for writing muscle-control programs :: Presented with
2539 low-level muscle control/ sense API, generate higher level
2540 programs for accomplishing various stated goals. Example goals
2541 might be "extend all your fingers" or "move your hand into the
2542 area with blue light" or "decrease the angle of this joint".
2543 It would be like Sussman's HACKER, except it would operate
2544 with much more data in a more realistic world. Start off with
2545 "calisthenics" to develop subroutines over the motor control
2546 API. This would be the "spinal chord" of a more intelligent
2547 creature. The low level programming code might be a turning
2548 machine that could develop programs to iterate over a "tape"
2549 where each entry in the tape could control recruitment of the
2550 fibers in a muscle.
2551 - Sense fusion :: There is much work to be done on sense
2552 integration -- building up a coherent picture of the world and
2553 the things in it with =CORTEX= as a base, you can explore
2554 concepts like self-organizing maps or cross modal clustering
2555 in ways that have never before been tried.
2556 - Inverse kinematics :: experiments in sense guided motor control
2557 are easy given =CORTEX='s support -- you can get right to the
2558 hard control problems without worrying about physics or
2559 senses.
2561 * =EMPATH=: the simulated worm experiment
2562 # Empathy in a simulated worm
2564 Here I develop a computational model of empathy, using =CORTEX= as a
2565 base. Empathy in this context is the ability to observe another
2566 creature and infer what sorts of sensations that creature is
2567 feeling. My empathy algorithm involves multiple phases. First is
2568 free-play, where the creature moves around and gains sensory
2569 experience. From this experience I construct a representation of the
2570 creature's sensory state space, which I call \Phi-space. Using
2571 \Phi-space, I construct an efficient function which takes the
2572 limited data that comes from observing another creature and enriches
2573 it full compliment of imagined sensory data. I can then use the
2574 imagined sensory data to recognize what the observed creature is
2575 doing and feeling, using straightforward embodied action predicates.
2576 This is all demonstrated with using a simple worm-like creature, and
2577 recognizing worm-actions based on limited data.
2579 #+caption: Here is the worm with which we will be working.
2580 #+caption: It is composed of 5 segments. Each segment has a
2581 #+caption: pair of extensor and flexor muscles. Each of the
2582 #+caption: worm's four joints is a hinge joint which allows
2583 #+caption: about 30 degrees of rotation to either side. Each segment
2584 #+caption: of the worm is touch-capable and has a uniform
2585 #+caption: distribution of touch sensors on each of its faces.
2586 #+caption: Each joint has a proprioceptive sense to detect
2587 #+caption: relative positions. The worm segments are all the
2588 #+caption: same except for the first one, which has a much
2589 #+caption: higher weight than the others to allow for easy
2590 #+caption: manual motor control.
2591 #+name: basic-worm-view
2592 #+ATTR_LaTeX: :width 10cm
2593 [[./images/basic-worm-view.png]]
2595 #+caption: Program for reading a worm from a blender file and
2596 #+caption: outfitting it with the senses of proprioception,
2597 #+caption: touch, and the ability to move, as specified in the
2598 #+caption: blender file.
2599 #+name: get-worm
2600 #+begin_listing clojure
2601 #+begin_src clojure
2602 (defn worm []
2603 (let [model (load-blender-model "Models/worm/worm.blend")]
2604 {:body (doto model (body!))
2605 :touch (touch! model)
2606 :proprioception (proprioception! model)
2607 :muscles (movement! model)}))
2608 #+end_src
2609 #+end_listing
2611 ** Embodiment factors action recognition into managable parts
2613 Using empathy, I divide the problem of action recognition into a
2614 recognition process expressed in the language of a full compliment
2615 of senses, and an imaganitive process that generates full sensory
2616 data from partial sensory data. Splitting the action recognition
2617 problem in this manner greatly reduces the total amount of work to
2618 recognize actions: The imaganitive process is mostly just matching
2619 previous experience, and the recognition process gets to use all
2620 the senses to directly describe any action.
2622 ** Action recognition is easy with a full gamut of senses
2624 Embodied representations using multiple senses such as touch,
2625 proprioception, and muscle tension turns out be be exceedingly
2626 efficient at describing body-centered actions. It is the ``right
2627 language for the job''. For example, it takes only around 5 lines
2628 of LISP code to describe the action of ``curling'' using embodied
2629 primitives. It takes about 10 lines to describe the seemingly
2630 complicated action of wiggling.
2632 The following action predicates each take a stream of sensory
2633 experience, observe however much of it they desire, and decide
2634 whether the worm is doing the action they describe. =curled?=
2635 relies on proprioception, =resting?= relies on touch, =wiggling?=
2636 relies on a fourier analysis of muscle contraction, and
2637 =grand-circle?= relies on touch and reuses =curled?= as a gaurd.
2639 #+caption: Program for detecting whether the worm is curled. This is the
2640 #+caption: simplest action predicate, because it only uses the last frame
2641 #+caption: of sensory experience, and only uses proprioceptive data. Even
2642 #+caption: this simple predicate, however, is automatically frame
2643 #+caption: independent and ignores vermopomorphic differences such as
2644 #+caption: worm textures and colors.
2645 #+name: curled
2646 #+begin_listing clojure
2647 #+begin_src clojure
2648 (defn curled?
2649 "Is the worm curled up?"
2650 [experiences]
2651 (every?
2652 (fn [[_ _ bend]]
2653 (> (Math/sin bend) 0.64))
2654 (:proprioception (peek experiences))))
2655 #+end_src
2656 #+end_listing
2658 #+caption: Program for summarizing the touch information in a patch
2659 #+caption: of skin.
2660 #+name: touch-summary
2661 #+begin_listing clojure
2662 #+begin_src clojure
2663 (defn contact
2664 "Determine how much contact a particular worm segment has with
2665 other objects. Returns a value between 0 and 1, where 1 is full
2666 contact and 0 is no contact."
2667 [touch-region [coords contact :as touch]]
2668 (-> (zipmap coords contact)
2669 (select-keys touch-region)
2670 (vals)
2671 (#(map first %))
2672 (average)
2673 (* 10)
2674 (- 1)
2675 (Math/abs)))
2676 #+end_src
2677 #+end_listing
2680 #+caption: Program for detecting whether the worm is at rest. This program
2681 #+caption: uses a summary of the tactile information from the underbelly
2682 #+caption: of the worm, and is only true if every segment is touching the
2683 #+caption: floor. Note that this function contains no references to
2684 #+caption: proprioction at all.
2685 #+name: resting
2686 #+begin_listing clojure
2687 #+begin_src clojure
2688 (def worm-segment-bottom (rect-region [8 15] [14 22]))
2690 (defn resting?
2691 "Is the worm resting on the ground?"
2692 [experiences]
2693 (every?
2694 (fn [touch-data]
2695 (< 0.9 (contact worm-segment-bottom touch-data)))
2696 (:touch (peek experiences))))
2697 #+end_src
2698 #+end_listing
2700 #+caption: Program for detecting whether the worm is curled up into a
2701 #+caption: full circle. Here the embodied approach begins to shine, as
2702 #+caption: I am able to both use a previous action predicate (=curled?=)
2703 #+caption: as well as the direct tactile experience of the head and tail.
2704 #+name: grand-circle
2705 #+begin_listing clojure
2706 #+begin_src clojure
2707 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
2709 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
2711 (defn grand-circle?
2712 "Does the worm form a majestic circle (one end touching the other)?"
2713 [experiences]
2714 (and (curled? experiences)
2715 (let [worm-touch (:touch (peek experiences))
2716 tail-touch (worm-touch 0)
2717 head-touch (worm-touch 4)]
2718 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
2719 (< 0.55 (contact worm-segment-top-tip head-touch))))))
2720 #+end_src
2721 #+end_listing
2724 #+caption: Program for detecting whether the worm has been wiggling for
2725 #+caption: the last few frames. It uses a fourier analysis of the muscle
2726 #+caption: contractions of the worm's tail to determine wiggling. This is
2727 #+caption: signigicant because there is no particular frame that clearly
2728 #+caption: indicates that the worm is wiggling --- only when multiple frames
2729 #+caption: are analyzed together is the wiggling revealed. Defining
2730 #+caption: wiggling this way also gives the worm an opportunity to learn
2731 #+caption: and recognize ``frustrated wiggling'', where the worm tries to
2732 #+caption: wiggle but can't. Frustrated wiggling is very visually different
2733 #+caption: from actual wiggling, but this definition gives it to us for free.
2734 #+name: wiggling
2735 #+begin_listing clojure
2736 #+begin_src clojure
2737 (defn fft [nums]
2738 (map
2739 #(.getReal %)
2740 (.transform
2741 (FastFourierTransformer. DftNormalization/STANDARD)
2742 (double-array nums) TransformType/FORWARD)))
2744 (def indexed (partial map-indexed vector))
2746 (defn max-indexed [s]
2747 (first (sort-by (comp - second) (indexed s))))
2749 (defn wiggling?
2750 "Is the worm wiggling?"
2751 [experiences]
2752 (let [analysis-interval 0x40]
2753 (when (> (count experiences) analysis-interval)
2754 (let [a-flex 3
2755 a-ex 2
2756 muscle-activity
2757 (map :muscle (vector:last-n experiences analysis-interval))
2758 base-activity
2759 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
2760 (= 2
2761 (first
2762 (max-indexed
2763 (map #(Math/abs %)
2764 (take 20 (fft base-activity))))))))))
2765 #+end_src
2766 #+end_listing
2768 With these action predicates, I can now recognize the actions of
2769 the worm while it is moving under my control and I have access to
2770 all the worm's senses.
2772 #+caption: Use the action predicates defined earlier to report on
2773 #+caption: what the worm is doing while in simulation.
2774 #+name: report-worm-activity
2775 #+begin_listing clojure
2776 #+begin_src clojure
2777 (defn debug-experience
2778 [experiences text]
2779 (cond
2780 (grand-circle? experiences) (.setText text "Grand Circle")
2781 (curled? experiences) (.setText text "Curled")
2782 (wiggling? experiences) (.setText text "Wiggling")
2783 (resting? experiences) (.setText text "Resting")))
2784 #+end_src
2785 #+end_listing
2787 #+caption: Using =debug-experience=, the body-centered predicates
2788 #+caption: work together to classify the behaviour of the worm.
2789 #+caption: the predicates are operating with access to the worm's
2790 #+caption: full sensory data.
2791 #+name: basic-worm-view
2792 #+ATTR_LaTeX: :width 10cm
2793 [[./images/worm-identify-init.png]]
2795 These action predicates satisfy the recognition requirement of an
2796 empathic recognition system. There is power in the simplicity of
2797 the action predicates. They describe their actions without getting
2798 confused in visual details of the worm. Each one is frame
2799 independent, but more than that, they are each indepent of
2800 irrelevant visual details of the worm and the environment. They
2801 will work regardless of whether the worm is a different color or
2802 hevaily textured, or if the environment has strange lighting.
2804 The trick now is to make the action predicates work even when the
2805 sensory data on which they depend is absent. If I can do that, then
2806 I will have gained much,
2808 ** \Phi-space describes the worm's experiences
2810 As a first step towards building empathy, I need to gather all of
2811 the worm's experiences during free play. I use a simple vector to
2812 store all the experiences.
2814 Each element of the experience vector exists in the vast space of
2815 all possible worm-experiences. Most of this vast space is actually
2816 unreachable due to physical constraints of the worm's body. For
2817 example, the worm's segments are connected by hinge joints that put
2818 a practical limit on the worm's range of motions without limiting
2819 its degrees of freedom. Some groupings of senses are impossible;
2820 the worm can not be bent into a circle so that its ends are
2821 touching and at the same time not also experience the sensation of
2822 touching itself.
2824 As the worm moves around during free play and its experience vector
2825 grows larger, the vector begins to define a subspace which is all
2826 the sensations the worm can practicaly experience during normal
2827 operation. I call this subspace \Phi-space, short for
2828 physical-space. The experience vector defines a path through
2829 \Phi-space. This path has interesting properties that all derive
2830 from physical embodiment. The proprioceptive components are
2831 completely smooth, because in order for the worm to move from one
2832 position to another, it must pass through the intermediate
2833 positions. The path invariably forms loops as actions are repeated.
2834 Finally and most importantly, proprioception actually gives very
2835 strong inference about the other senses. For example, when the worm
2836 is flat, you can infer that it is touching the ground and that its
2837 muscles are not active, because if the muscles were active, the
2838 worm would be moving and would not be perfectly flat. In order to
2839 stay flat, the worm has to be touching the ground, or it would
2840 again be moving out of the flat position due to gravity. If the
2841 worm is positioned in such a way that it interacts with itself,
2842 then it is very likely to be feeling the same tactile feelings as
2843 the last time it was in that position, because it has the same body
2844 as then. If you observe multiple frames of proprioceptive data,
2845 then you can become increasingly confident about the exact
2846 activations of the worm's muscles, because it generally takes a
2847 unique combination of muscle contractions to transform the worm's
2848 body along a specific path through \Phi-space.
2850 There is a simple way of taking \Phi-space and the total ordering
2851 provided by an experience vector and reliably infering the rest of
2852 the senses.
2854 ** ``Empathy'' requires retracing steps though \Phi-space
2856 Here is the core of a basic empathy algorithm, starting with an
2857 experience vector:
2859 First, group the experiences into tiered proprioceptive bins. I use
2860 powers of 10 and 3 bins, and the smallest bin has an approximate
2861 size of 0.001 radians in all proprioceptive dimensions.
2863 Then, given a sequence of proprioceptive input, generate a set of
2864 matching experience records for each input, using the tiered
2865 proprioceptive bins.
2867 Finally, to infer sensory data, select the longest consective chain
2868 of experiences. Conecutive experience means that the experiences
2869 appear next to each other in the experience vector.
2871 This algorithm has three advantages:
2873 1. It's simple
2875 3. It's very fast -- retrieving possible interpretations takes
2876 constant time. Tracing through chains of interpretations takes
2877 time proportional to the average number of experiences in a
2878 proprioceptive bin. Redundant experiences in \Phi-space can be
2879 merged to save computation.
2881 2. It protects from wrong interpretations of transient ambiguous
2882 proprioceptive data. For example, if the worm is flat for just
2883 an instant, this flattness will not be interpreted as implying
2884 that the worm has its muscles relaxed, since the flattness is
2885 part of a longer chain which includes a distinct pattern of
2886 muscle activation. Markov chains or other memoryless statistical
2887 models that operate on individual frames may very well make this
2888 mistake.
2890 #+caption: Program to convert an experience vector into a
2891 #+caption: proprioceptively binned lookup function.
2892 #+name: bin
2893 #+begin_listing clojure
2894 #+begin_src clojure
2895 (defn bin [digits]
2896 (fn [angles]
2897 (->> angles
2898 (flatten)
2899 (map (juxt #(Math/sin %) #(Math/cos %)))
2900 (flatten)
2901 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
2903 (defn gen-phi-scan
2904 "Nearest-neighbors with binning. Only returns a result if
2905 the propriceptive data is within 10% of a previously recorded
2906 result in all dimensions."
2907 [phi-space]
2908 (let [bin-keys (map bin [3 2 1])
2909 bin-maps
2910 (map (fn [bin-key]
2911 (group-by
2912 (comp bin-key :proprioception phi-space)
2913 (range (count phi-space)))) bin-keys)
2914 lookups (map (fn [bin-key bin-map]
2915 (fn [proprio] (bin-map (bin-key proprio))))
2916 bin-keys bin-maps)]
2917 (fn lookup [proprio-data]
2918 (set (some #(% proprio-data) lookups)))))
2919 #+end_src
2920 #+end_listing
2922 #+caption: =longest-thread= finds the longest path of consecutive
2923 #+caption: experiences to explain proprioceptive worm data.
2924 #+name: phi-space-history-scan
2925 #+ATTR_LaTeX: :width 10cm
2926 [[./images/aurellem-gray.png]]
2928 =longest-thread= infers sensory data by stitching together pieces
2929 from previous experience. It prefers longer chains of previous
2930 experience to shorter ones. For example, during training the worm
2931 might rest on the ground for one second before it performs its
2932 excercises. If during recognition the worm rests on the ground for
2933 five seconds, =longest-thread= will accomodate this five second
2934 rest period by looping the one second rest chain five times.
2936 =longest-thread= takes time proportinal to the average number of
2937 entries in a proprioceptive bin, because for each element in the
2938 starting bin it performes a series of set lookups in the preceeding
2939 bins. If the total history is limited, then this is only a constant
2940 multiple times the number of entries in the starting bin. This
2941 analysis also applies even if the action requires multiple longest
2942 chains -- it's still the average number of entries in a
2943 proprioceptive bin times the desired chain length. Because
2944 =longest-thread= is so efficient and simple, I can interpret
2945 worm-actions in real time.
2947 #+caption: Program to calculate empathy by tracing though \Phi-space
2948 #+caption: and finding the longest (ie. most coherent) interpretation
2949 #+caption: of the data.
2950 #+name: longest-thread
2951 #+begin_listing clojure
2952 #+begin_src clojure
2953 (defn longest-thread
2954 "Find the longest thread from phi-index-sets. The index sets should
2955 be ordered from most recent to least recent."
2956 [phi-index-sets]
2957 (loop [result '()
2958 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
2959 (if (empty? phi-index-sets)
2960 (vec result)
2961 (let [threads
2962 (for [thread-base thread-bases]
2963 (loop [thread (list thread-base)
2964 remaining remaining]
2965 (let [next-index (dec (first thread))]
2966 (cond (empty? remaining) thread
2967 (contains? (first remaining) next-index)
2968 (recur
2969 (cons next-index thread) (rest remaining))
2970 :else thread))))
2971 longest-thread
2972 (reduce (fn [thread-a thread-b]
2973 (if (> (count thread-a) (count thread-b))
2974 thread-a thread-b))
2975 '(nil)
2976 threads)]
2977 (recur (concat longest-thread result)
2978 (drop (count longest-thread) phi-index-sets))))))
2979 #+end_src
2980 #+end_listing
2982 There is one final piece, which is to replace missing sensory data
2983 with a best-guess estimate. While I could fill in missing data by
2984 using a gradient over the closest known sensory data points,
2985 averages can be misleading. It is certainly possible to create an
2986 impossible sensory state by averaging two possible sensory states.
2987 Therefore, I simply replicate the most recent sensory experience to
2988 fill in the gaps.
2990 #+caption: Fill in blanks in sensory experience by replicating the most
2991 #+caption: recent experience.
2992 #+name: infer-nils
2993 #+begin_listing clojure
2994 #+begin_src clojure
2995 (defn infer-nils
2996 "Replace nils with the next available non-nil element in the
2997 sequence, or barring that, 0."
2998 [s]
2999 (loop [i (dec (count s))
3000 v (transient s)]
3001 (if (zero? i) (persistent! v)
3002 (if-let [cur (v i)]
3003 (if (get v (dec i) 0)
3004 (recur (dec i) v)
3005 (recur (dec i) (assoc! v (dec i) cur)))
3006 (recur i (assoc! v i 0))))))
3007 #+end_src
3008 #+end_listing
3010 ** =EMPATH= recognizes actions efficiently
3012 To use =EMPATH= with the worm, I first need to gather a set of
3013 experiences from the worm that includes the actions I want to
3014 recognize. The =generate-phi-space= program (listing
3015 \ref{generate-phi-space} runs the worm through a series of
3016 exercices and gatheres those experiences into a vector. The
3017 =do-all-the-things= program is a routine expressed in a simple
3018 muscle contraction script language for automated worm control. It
3019 causes the worm to rest, curl, and wiggle over about 700 frames
3020 (approx. 11 seconds).
3022 #+caption: Program to gather the worm's experiences into a vector for
3023 #+caption: further processing. The =motor-control-program= line uses
3024 #+caption: a motor control script that causes the worm to execute a series
3025 #+caption: of ``exercices'' that include all the action predicates.
3026 #+name: generate-phi-space
3027 #+begin_listing clojure
3028 #+begin_src clojure
3029 (def do-all-the-things
3030 (concat
3031 curl-script
3032 [[300 :d-ex 40]
3033 [320 :d-ex 0]]
3034 (shift-script 280 (take 16 wiggle-script))))
3036 (defn generate-phi-space []
3037 (let [experiences (atom [])]
3038 (run-world
3039 (apply-map
3040 worm-world
3041 (merge
3042 (worm-world-defaults)
3043 {:end-frame 700
3044 :motor-control
3045 (motor-control-program worm-muscle-labels do-all-the-things)
3046 :experiences experiences})))
3047 @experiences))
3048 #+end_src
3049 #+end_listing
3051 #+caption: Use longest thread and a phi-space generated from a short
3052 #+caption: exercise routine to interpret actions during free play.
3053 #+name: empathy-debug
3054 #+begin_listing clojure
3055 #+begin_src clojure
3056 (defn init []
3057 (def phi-space (generate-phi-space))
3058 (def phi-scan (gen-phi-scan phi-space)))
3060 (defn empathy-demonstration []
3061 (let [proprio (atom ())]
3062 (fn
3063 [experiences text]
3064 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3065 (swap! proprio (partial cons phi-indices))
3066 (let [exp-thread (longest-thread (take 300 @proprio))
3067 empathy (mapv phi-space (infer-nils exp-thread))]
3068 (println-repl (vector:last-n exp-thread 22))
3069 (cond
3070 (grand-circle? empathy) (.setText text "Grand Circle")
3071 (curled? empathy) (.setText text "Curled")
3072 (wiggling? empathy) (.setText text "Wiggling")
3073 (resting? empathy) (.setText text "Resting")
3074 :else (.setText text "Unknown")))))))
3076 (defn empathy-experiment [record]
3077 (.start (worm-world :experience-watch (debug-experience-phi)
3078 :record record :worm worm*)))
3079 #+end_src
3080 #+end_listing
3082 The result of running =empathy-experiment= is that the system is
3083 generally able to interpret worm actions using the action-predicates
3084 on simulated sensory data just as well as with actual data. Figure
3085 \ref{empathy-debug-image} was generated using =empathy-experiment=:
3087 #+caption: From only proprioceptive data, =EMPATH= was able to infer
3088 #+caption: the complete sensory experience and classify four poses
3089 #+caption: (The last panel shows a composite image of \emph{wriggling},
3090 #+caption: a dynamic pose.)
3091 #+name: empathy-debug-image
3092 #+ATTR_LaTeX: :width 10cm :placement [H]
3093 [[./images/empathy-1.png]]
3095 One way to measure the performance of =EMPATH= is to compare the
3096 sutiability of the imagined sense experience to trigger the same
3097 action predicates as the real sensory experience.
3099 #+caption: Determine how closely empathy approximates actual
3100 #+caption: sensory data.
3101 #+name: test-empathy-accuracy
3102 #+begin_listing clojure
3103 #+begin_src clojure
3104 (def worm-action-label
3105 (juxt grand-circle? curled? wiggling?))
3107 (defn compare-empathy-with-baseline [matches]
3108 (let [proprio (atom ())]
3109 (fn
3110 [experiences text]
3111 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
3112 (swap! proprio (partial cons phi-indices))
3113 (let [exp-thread (longest-thread (take 300 @proprio))
3114 empathy (mapv phi-space (infer-nils exp-thread))
3115 experience-matches-empathy
3116 (= (worm-action-label experiences)
3117 (worm-action-label empathy))]
3118 (println-repl experience-matches-empathy)
3119 (swap! matches #(conj % experience-matches-empathy)))))))
3121 (defn accuracy [v]
3122 (float (/ (count (filter true? v)) (count v))))
3124 (defn test-empathy-accuracy []
3125 (let [res (atom [])]
3126 (run-world
3127 (worm-world :experience-watch
3128 (compare-empathy-with-baseline res)
3129 :worm worm*))
3130 (accuracy @res)))
3131 #+end_src
3132 #+end_listing
3134 Running =test-empathy-accuracy= using the very short exercise
3135 program defined in listing \ref{generate-phi-space}, and then doing
3136 a similar pattern of activity manually yeilds an accuracy of around
3137 73%. This is based on very limited worm experience. By training the
3138 worm for longer, the accuracy dramatically improves.
3140 #+caption: Program to generate \Phi-space using manual training.
3141 #+name: manual-phi-space
3142 #+begin_listing clojure
3143 #+begin_src clojure
3144 (defn init-interactive []
3145 (def phi-space
3146 (let [experiences (atom [])]
3147 (run-world
3148 (apply-map
3149 worm-world
3150 (merge
3151 (worm-world-defaults)
3152 {:experiences experiences})))
3153 @experiences))
3154 (def phi-scan (gen-phi-scan phi-space)))
3155 #+end_src
3156 #+end_listing
3158 After about 1 minute of manual training, I was able to achieve 95%
3159 accuracy on manual testing of the worm using =init-interactive= and
3160 =test-empathy-accuracy=. The majority of errors are near the
3161 boundaries of transitioning from one type of action to another.
3162 During these transitions the exact label for the action is more open
3163 to interpretation, and dissaggrement between empathy and experience
3164 is more excusable.
3166 ** Digression: Learn touch sensor layout through free play
3168 In the previous section I showed how to compute actions in terms of
3169 body-centered predicates which relied averate touch activation of
3170 pre-defined regions of the worm's skin. What if, instead of
3171 recieving touch pre-grouped into the six faces of each worm
3172 segment, the true topology of the worm's skin was unknown? This is
3173 more similiar to how a nerve fiber bundle might be arranged. While
3174 two fibers that are close in a nerve bundle /might/ correspond to
3175 two touch sensors that are close together on the skin, the process
3176 of taking a complicated surface and forcing it into essentially a
3177 circle requires some cuts and rerragenments.
3179 In this section I show how to automatically learn the skin-topology of
3180 a worm segment by free exploration. As the worm rolls around on the
3181 floor, large sections of its surface get activated. If the worm has
3182 stopped moving, then whatever region of skin that is touching the
3183 floor is probably an important region, and should be recorded.
3185 #+caption: Program to detect whether the worm is in a resting state
3186 #+caption: with one face touching the floor.
3187 #+name: pure-touch
3188 #+begin_listing clojure
3189 #+begin_src clojure
3190 (def full-contact [(float 0.0) (float 0.1)])
3192 (defn pure-touch?
3193 "This is worm specific code to determine if a large region of touch
3194 sensors is either all on or all off."
3195 [[coords touch :as touch-data]]
3196 (= (set (map first touch)) (set full-contact)))
3197 #+end_src
3198 #+end_listing
3200 After collecting these important regions, there will many nearly
3201 similiar touch regions. While for some purposes the subtle
3202 differences between these regions will be important, for my
3203 purposes I colapse them into mostly non-overlapping sets using
3204 =remove-similiar= in listing \ref{remove-similiar}
3206 #+caption: Program to take a lits of set of points and ``collapse them''
3207 #+caption: so that the remaining sets in the list are siginificantly
3208 #+caption: different from each other. Prefer smaller sets to larger ones.
3209 #+name: remove-similiar
3210 #+begin_listing clojure
3211 #+begin_src clojure
3212 (defn remove-similar
3213 [coll]
3214 (loop [result () coll (sort-by (comp - count) coll)]
3215 (if (empty? coll) result
3216 (let [[x & xs] coll
3217 c (count x)]
3218 (if (some
3219 (fn [other-set]
3220 (let [oc (count other-set)]
3221 (< (- (count (union other-set x)) c) (* oc 0.1))))
3222 xs)
3223 (recur result xs)
3224 (recur (cons x result) xs))))))
3225 #+end_src
3226 #+end_listing
3228 Actually running this simulation is easy given =CORTEX='s facilities.
3230 #+caption: Collect experiences while the worm moves around. Filter the touch
3231 #+caption: sensations by stable ones, collapse similiar ones together,
3232 #+caption: and report the regions learned.
3233 #+name: learn-touch
3234 #+begin_listing clojure
3235 #+begin_src clojure
3236 (defn learn-touch-regions []
3237 (let [experiences (atom [])
3238 world (apply-map
3239 worm-world
3240 (assoc (worm-segment-defaults)
3241 :experiences experiences))]
3242 (run-world world)
3243 (->>
3244 @experiences
3245 (drop 175)
3246 ;; access the single segment's touch data
3247 (map (comp first :touch))
3248 ;; only deal with "pure" touch data to determine surfaces
3249 (filter pure-touch?)
3250 ;; associate coordinates with touch values
3251 (map (partial apply zipmap))
3252 ;; select those regions where contact is being made
3253 (map (partial group-by second))
3254 (map #(get % full-contact))
3255 (map (partial map first))
3256 ;; remove redundant/subset regions
3257 (map set)
3258 remove-similar)))
3260 (defn learn-and-view-touch-regions []
3261 (map view-touch-region
3262 (learn-touch-regions)))
3263 #+end_src
3264 #+end_listing
3266 The only thing remining to define is the particular motion the worm
3267 must take. I accomplish this with a simple motor control program.
3269 #+caption: Motor control program for making the worm roll on the ground.
3270 #+caption: This could also be replaced with random motion.
3271 #+name: worm-roll
3272 #+begin_listing clojure
3273 #+begin_src clojure
3274 (defn touch-kinesthetics []
3275 [[170 :lift-1 40]
3276 [190 :lift-1 19]
3277 [206 :lift-1 0]
3279 [400 :lift-2 40]
3280 [410 :lift-2 0]
3282 [570 :lift-2 40]
3283 [590 :lift-2 21]
3284 [606 :lift-2 0]
3286 [800 :lift-1 30]
3287 [809 :lift-1 0]
3289 [900 :roll-2 40]
3290 [905 :roll-2 20]
3291 [910 :roll-2 0]
3293 [1000 :roll-2 40]
3294 [1005 :roll-2 20]
3295 [1010 :roll-2 0]
3297 [1100 :roll-2 40]
3298 [1105 :roll-2 20]
3299 [1110 :roll-2 0]
3300 ])
3301 #+end_src
3302 #+end_listing
3305 #+caption: The small worm rolls around on the floor, driven
3306 #+caption: by the motor control program in listing \ref{worm-roll}.
3307 #+name: worm-roll
3308 #+ATTR_LaTeX: :width 12cm
3309 [[./images/worm-roll.png]]
3312 #+caption: After completing its adventures, the worm now knows
3313 #+caption: how its touch sensors are arranged along its skin. These
3314 #+caption: are the regions that were deemed important by
3315 #+caption: =learn-touch-regions=. Note that the worm has discovered
3316 #+caption: that it has six sides.
3317 #+name: worm-touch-map
3318 #+ATTR_LaTeX: :width 12cm
3319 [[./images/touch-learn.png]]
3321 While simple, =learn-touch-regions= exploits regularities in both
3322 the worm's physiology and the worm's environment to correctly
3323 deduce that the worm has six sides. Note that =learn-touch-regions=
3324 would work just as well even if the worm's touch sense data were
3325 completely scrambled. The cross shape is just for convienence. This
3326 example justifies the use of pre-defined touch regions in =EMPATH=.
3328 * Contributions
3330 In this thesis you have seen the =CORTEX= system, a complete
3331 environment for creating simulated creatures. You have seen how to
3332 implement five senses: touch, proprioception, hearing, vision, and
3333 muscle tension. You have seen how to create new creatues using
3334 blender, a 3D modeling tool. I hope that =CORTEX= will be useful in
3335 further research projects. To this end I have included the full
3336 source to =CORTEX= along with a large suite of tests and examples. I
3337 have also created a user guide for =CORTEX= which is inculded in an
3338 appendix to this thesis \ref{}.
3339 # dxh: todo reference appendix
3341 You have also seen how I used =CORTEX= as a platform to attach the
3342 /action recognition/ problem, which is the problem of recognizing
3343 actions in video. You saw a simple system called =EMPATH= which
3344 ientifies actions by first describing actions in a body-centerd,
3345 rich sense language, then infering a full range of sensory
3346 experience from limited data using previous experience gained from
3347 free play.
3349 As a minor digression, you also saw how I used =CORTEX= to enable a
3350 tiny worm to discover the topology of its skin simply by rolling on
3351 the ground.
3353 In conclusion, the main contributions of this thesis are:
3355 - =CORTEX=, a system for creating simulated creatures with rich
3356 senses.
3357 - =EMPATH=, a program for recognizing actions by imagining sensory
3358 experience.
3360 # An anatomical joke:
3361 # - Training
3362 # - Skeletal imitation
3363 # - Sensory fleshing-out
3364 # - Classification
3365 #+BEGIN_LaTeX
3366 \appendix
3367 #+END_LaTeX
3368 * Appendix: =CORTEX= User Guide
3370 Those who write a thesis should endeavor to make their code not only
3371 accessable, but actually useable, as a way to pay back the community
3372 that made the thesis possible in the first place. This thesis would
3373 not be possible without Free Software such as jMonkeyEngine3,
3374 Blender, clojure, emacs, ffmpeg, and many other tools. That is why I
3375 have included this user guide, in the hope that someone else might
3376 find =CORTEX= useful.
3378 ** Obtaining =CORTEX=
3380 You can get cortex from its mercurial repository at
3381 http://hg.bortreb.com/cortex. You may also download =CORTEX=
3382 releases at http://aurellem.org/cortex/releases/. As a condition of
3383 making this thesis, I have also provided Professor Winston the
3384 =CORTEX= source, and he knows how to run the demos and get started.
3385 You may also email me at =cortex@aurellem.org= and I may help where
3386 I can.
3388 ** Running =CORTEX=
3390 =CORTEX= comes with README and INSTALL files that will guide you
3391 through installation and running the test suite. In particular you
3392 should look at test =cortex.test= which contains test suites that
3393 run through all senses and multiple creatures.
3395 ** Creating creatures
3397 Creatures are created using /Blender/, a free 3D modeling program.
3398 You will need Blender version 2.6 when using the =CORTEX= included
3399 in this thesis. You create a =CORTEX= creature in a similiar manner
3400 to modeling anything in Blender, except that you also create
3401 several trees of empty nodes which define the creature's senses.
3403 *** Mass
3405 To give an object mass in =CORTEX=, add a ``mass'' metadata label
3406 to the object with the mass in jMonkeyEngine units. Note that
3407 setting the mass to 0 causes the object to be immovable.
3409 *** Joints
3411 Joints are created by creating an empty node named =joints= and
3412 then creating any number of empty child nodes to represent your
3413 creature's joints. The joint will automatically connect the
3414 closest two physical objects. It will help to set the empty node's
3415 display mode to ``Arrows'' so that you can clearly see the
3416 direction of the axes.
3418 Joint nodes should have the following metadata under the ``joint''
3419 label:
3421 #+BEGIN_SRC clojure
3422 ;; ONE OF the following, under the label "joint":
3423 {:type :point}
3425 ;; OR
3427 {:type :hinge
3428 :limit [<limit-low> <limit-high>]
3429 :axis (Vector3f. <x> <y> <z>)}
3430 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
3432 ;; OR
3434 {:type :cone
3435 :limit-xz <lim-xz>
3436 :limit-xy <lim-xy>
3437 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
3438 #+END_SRC
3440 *** Eyes
3442 Eyes are created by creating an empty node named =eyes= and then
3443 creating any number of empty child nodes to represent your
3444 creature's eyes.
3446 Eye nodes should have the following metadata under the ``eye''
3447 label:
3449 #+BEGIN_SRC clojure
3450 {:red <red-retina-definition>
3451 :blue <blue-retina-definition>
3452 :green <green-retina-definition>
3453 :all <all-retina-definition>
3454 (<0xrrggbb> <custom-retina-image>)...
3456 #+END_SRC
3458 Any of the color channels may be omitted. You may also include
3459 your own color selectors, and in fact :red is equivalent to
3460 0xFF0000 and so forth. The eye will be placed at the same position
3461 as the empty node and will bind to the neatest physical object.
3462 The eye will point outward from the X-axis of the node, and ``up''
3463 will be in the direction of the X-axis of the node. It will help
3464 to set the empty node's display mode to ``Arrows'' so that you can
3465 clearly see the direction of the axes.
3467 Each retina file should contain white pixels whever you want to be
3468 sensitive to your chosen color. If you want the entire field of
3469 view, specify :all of 0xFFFFFF and a retinal map that is entirely
3470 white.
3472 Here is a sample retinal map:
3474 #+caption: An example retinal profile image. White pixels are
3475 #+caption: photo-sensitive elements. The distribution of white
3476 #+caption: pixels is denser in the middle and falls off at the
3477 #+caption: edges and is inspired by the human retina.
3478 #+name: retina
3479 #+ATTR_LaTeX: :width 7cm :placement [H]
3480 [[./images/retina-small.png]]
3482 *** Hearing
3484 Ears are created by creating an empty node named =ears= and then
3485 creating any number of empty child nodes to represent your
3486 creature's ears.
3488 Ear nodes do not require any metadata.
3490 The ear will bind to and follow the closest physical node.
3492 *** Touch
3494 Touch is handled similarly to mass. To make a particular object
3495 touch sensitive, add metadata of the following form under the
3496 object's ``touch'' metadata field:
3498 #+BEGIN_EXAMPLE
3499 <touch-UV-map-file-name>
3500 #+END_EXAMPLE
3502 You may also include an optional ``scale'' metadata number to
3503 specifiy the length of the touch feelers. The default is $0.1$,
3504 and this is generally sufficient.
3506 The touch UV should contain white pixels for each touch sensor.
3508 Here is an example touch-uv map that approximates a human finger,
3509 and its corresponding model.
3511 #+caption: This is the tactile-sensor-profile for the upper segment
3512 #+caption: of a fingertip. It defines regions of high touch sensitivity
3513 #+caption: (where there are many white pixels) and regions of low
3514 #+caption: sensitivity (where white pixels are sparse).
3515 #+name: guide-fingertip-UV
3516 #+ATTR_LaTeX: :width 9cm :placement [H]
3517 [[./images/finger-UV.png]]
3519 #+caption: The fingertip UV-image form above applied to a simple
3520 #+caption: model of a fingertip.
3521 #+name: guide-fingertip
3522 #+ATTR_LaTeX: :width 9cm :placement [H]
3523 [[./images/finger-2.png]]
3525 *** Propriocepotion
3527 Proprioception is tied to each joint node -- nothing special must
3528 be done in a blender model to enable proprioception other than
3529 creating joint nodes.
3531 *** Muscles
3533 Muscles are created by creating an empty node named =muscles= and
3534 then creating any number of empty child nodes to represent your
3535 creature's muscles.
3538 Muscle nodes should have the following metadata under the
3539 ``muscle'' label:
3541 #+BEGIN_EXAMPLE
3542 <muscle-profile-file-name>
3543 #+END_EXAMPLE
3545 Muscles should also have a ``strength'' metadata entry describing
3546 the muscle's total strength at full activation.
3548 Muscle profiles are simple images that contain the relative amount
3549 of muscle power in each simulated alpha motor neuron. The width of
3550 the image is the total size of the motor pool, and the redness of
3551 each neuron is the relative power of that motor pool.
3553 While the profile image can have any dimensions, only the first
3554 line of pixels is used to define the muscle. Here is a sample
3555 muscle profile image that defines a human-like muscle.
3557 #+caption: A muscle profile image that describes the strengths
3558 #+caption: of each motor neuron in a muscle. White is weakest
3559 #+caption: and dark red is strongest. This particular pattern
3560 #+caption: has weaker motor neurons at the beginning, just
3561 #+caption: like human muscle.
3562 #+name: muscle-recruit
3563 #+ATTR_LaTeX: :width 7cm :placement [H]
3564 [[./images/basic-muscle.png]]
3566 Muscles twist the nearest physical object about the muscle node's
3567 Z-axis. I recommend using the ``Single Arrow'' display mode for
3568 muscles and using the right hand rule to determine which way the
3569 muscle will twist. To make a segment that can twist in multiple
3570 directions, create multiple, differently aligned muscles.
3572 ** =CORTEX= API
3574 These are the some functions exposed by =CORTEX= for creating
3575 worlds and simulating creatures. These are in addition to
3576 jMonkeyEngine3's extensive library, which is documented elsewhere.
3578 *** Simulation
3579 - =(world root-node key-map setup-fn update-fn)= :: create
3580 a simulation.
3581 - /root-node/ :: a =com.jme3.scene.Node= object which
3582 contains all of the objects that should be in the
3583 simulation.
3585 - /key-map/ :: a map from strings describing keys to
3586 functions that should be executed whenever that key is
3587 pressed. the functions should take a SimpleApplication
3588 object and a boolean value. The SimpleApplication is the
3589 current simulation that is running, and the boolean is true
3590 if the key is being pressed, and false if it is being
3591 released. As an example,
3592 #+BEGIN_SRC clojure
3593 {"key-j" (fn [game value] (if value (println "key j pressed")))}
3594 #+END_SRC
3595 is a valid key-map which will cause the simulation to print
3596 a message whenever the 'j' key on the keyboard is pressed.
3598 - /setup-fn/ :: a function that takes a =SimpleApplication=
3599 object. It is called once when initializing the simulation.
3600 Use it to create things like lights, change the gravity,
3601 initialize debug nodes, etc.
3603 - /update-fn/ :: this function takes a =SimpleApplication=
3604 object and a float and is called every frame of the
3605 simulation. The float tells how many seconds is has been
3606 since the last frame was rendered, according to whatever
3607 clock jme is currently using. The default is to use IsoTimer
3608 which will result in this value always being the same.
3610 - =(position-camera world position rotation)= :: set the position
3611 of the simulation's main camera.
3613 - =(enable-debug world)= :: turn on debug wireframes for each
3614 simulated object.
3616 - =(set-gravity world gravity)= :: set the gravity of a running
3617 simulation.
3619 - =(box length width height & {options})= :: create a box in the
3620 simulation. Options is a hash map specifying texture, mass,
3621 etc. Possible options are =:name=, =:color=, =:mass=,
3622 =:friction=, =:texture=, =:material=, =:position=,
3623 =:rotation=, =:shape=, and =:physical?=.
3625 - =(sphere radius & {options})= :: create a sphere in the simulation.
3626 Options are the same as in =box=.
3628 - =(load-blender-model file-name)= :: create a node structure
3629 representing that described in a blender file.
3631 - =(light-up-everything world)= :: distribute a standard compliment
3632 of lights throught the simulation. Should be adequate for most
3633 purposes.
3635 - =(node-seq node)= :: return a recursuve list of the node's
3636 children.
3638 - =(nodify name children)= :: construct a node given a node-name and
3639 desired children.
3641 - =(add-element world element)= :: add an object to a running world
3642 simulation.
3644 - =(set-accuracy world accuracy)= :: change the accuracy of the
3645 world's physics simulator.
3647 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
3648 construct that is useful for loading textures and is required
3649 for smooth interaction with jMonkeyEngine library functions.
3651 - =(load-bullet)= :: unpack native libraries and initialize
3652 blender. This function is required before other world building
3653 functions are called.
3655 *** Creature Manipulation / Import
3657 - =(body! creature)= :: give the creature a physical body.
3659 - =(vision! creature)= :: give the creature a sense of vision.
3660 Returns a list of functions which will each, when called
3661 during a simulation, return the vision data for the channel of
3662 one of the eyes. The functions are ordered depending on the
3663 alphabetical order of the names of the eye nodes in the
3664 blender file. The data returned by the functions is a vector
3665 containing the eye's /topology/, a vector of coordinates, and
3666 the eye's /data/, a vector of RGB values filtered by the eye's
3667 sensitivity.
3669 - =(hearing! creature)= :: give the creature a sense of hearing.
3670 Returns a list of functions, one for each ear, that when
3671 called will return a frame's worth of hearing data for that
3672 ear. The functions are ordered depending on the alphabetical
3673 order of the names of the ear nodes in the blender file. The
3674 data returned by the functions is an array PCM encoded wav
3675 data.
3677 - =(touch! creature)= :: give the creature a sense of touch. Returns
3678 a single function that must be called with the /root node/ of
3679 the world, and which will return a vector of /touch-data/
3680 one entry for each touch sensitive component, each entry of
3681 which contains a /topology/ that specifies the distribution of
3682 touch sensors, and the /data/, which is a vector of
3683 =[activation, length]= pairs for each touch hair.
3685 - =(proprioception! creature)= :: give the creature the sense of
3686 proprioception. Returns a list of functions, one for each
3687 joint, that when called during a running simulation will
3688 report the =[headnig, pitch, roll]= of the joint.
3690 - =(movement! creature)= :: give the creature the power of movement.
3691 Creates a list of functions, one for each muscle, that when
3692 called with an integer, will set the recruitment of that
3693 muscle to that integer, and will report the current power
3694 being exerted by the muscle. Order of muscles is determined by
3695 the alphabetical sort order of the names of the muscle nodes.
3697 *** Visualization/Debug
3699 - =(view-vision)= :: create a function that when called with a list
3700 of visual data returned from the functions made by =vision!=,
3701 will display that visual data on the screen.
3703 - =(view-hearing)= :: same as =view-vision= but for hearing.
3705 - =(view-touch)= :: same as =view-vision= but for touch.
3707 - =(view-proprioception)= :: same as =view-vision= but for
3708 proprioception.
3710 - =(view-movement)= :: same as =view-vision= but for
3711 proprioception.
3713 - =(view anything)= :: =view= is a polymorphic function that allows
3714 you to inspect almost anything you could reasonably expect to
3715 be able to ``see'' in =CORTEX=.
3717 - =(text anything)= :: =text= is a polymorphic function that allows
3718 you to convert practically anything into a text string.
3720 - =(println-repl anything)= :: print messages to clojure's repl
3721 instead of the simulation's terminal window.
3723 - =(mega-import-jme3)= :: for experimenting at the REPL. This
3724 function will import all jMonkeyEngine3 classes for immediate
3725 use.
3727 - =(display-dialated-time world timer)= :: Shows the time as it is
3728 flowing in the simulation on a HUD display.