cortex: thesis/cortex.tex annotate

annotate thesis/cortex.tex @ 572:202c6d19acad

add index page as part of aurellem redesign.

author	Robert McIntyre <rlm@mit.edu>
date	Sun, 08 Mar 2015 22:08:17 -0700
parents
children

rev	line source
rlm@572	1
rlm@572	2 \section{Empathy $\backslash$ Embodiment: problem solving strategies}
rlm@572	3 \label{sec-1}
rlm@572	4
rlm@572	5 By the time you have read this thesis, you will understand a novel
rlm@572	6 approach to representing and recognizing physical actions using
rlm@572	7 embodiment and empathy. You will also see one way to efficiently
rlm@572	8 implement physical empathy for embodied creatures. Finally, you will
rlm@572	9 become familiar with \texttt{CORTEX}, a system for designing and simulating
rlm@572	10 creatures with rich senses, which I have designed as a library that
rlm@572	11 you can use in your own research. Note that I \emph{do not} process video
rlm@572	12 directly --- I start with knowledge of the positions of a creature's
rlm@572	13 body parts and work from there.
rlm@572	14
rlm@572	15 This is the core vision of my thesis: That one of the important ways
rlm@572	16 in which we understand others is by imagining ourselves in their
rlm@572	17 position and empathically feeling experiences relative to our own
rlm@572	18 bodies. By understanding events in terms of our own previous
rlm@572	19 corporeal experience, we greatly constrain the possibilities of what
rlm@572	20 would otherwise be an unwieldy exponential search. This extra
rlm@572	21 constraint can be the difference between easily understanding what
rlm@572	22 is happening in a video and being completely lost in a sea of
rlm@572	23 incomprehensible color and movement.
rlm@572	24
rlm@572	25 \subsection{The problem: recognizing actions is hard!}
rlm@572	26 \label{sec-1-1}
rlm@572	27
rlm@572	28 Examine figure \ref{cat-drink}. What is happening? As you, and
rlm@572	29 indeed very young children, can easily determine, this is an image
rlm@572	30 of drinking.
rlm@572	31
rlm@572	32 \begin{figure}[htb]
rlm@572	33 \centering
rlm@572	34 \includegraphics[width=7cm]{./images/cat-drinking.jpg}
rlm@572	35 \caption{\label{cat-drink}A cat drinking some water. Identifying this action is beyond the capabilities of existing computer vision systems.}
rlm@572	36 \end{figure}
rlm@572	37
rlm@572	38 Nevertheless, it is beyond the state of the art for a computer
rlm@572	39 vision program to describe what's happening in this image. Part of
rlm@572	40 the problem is that many computer vision systems focus on
rlm@572	41 pixel-level details or comparisons to example images (such as
rlm@572	42 \cite{volume-action-recognition}), but the 3D world is so variable
rlm@572	43 that it is hard to describe the world in terms of possible images.
rlm@572	44
rlm@572	45 In fact, the contents of a scene may have much less to do with
rlm@572	46 pixel probabilities than with recognizing various affordances:
rlm@572	47 things you can move, objects you can grasp, spaces that can be
rlm@572	48 filled . For example, what processes might enable you to see the
rlm@572	49 chair in figure \ref{hidden-chair}?
rlm@572	50
rlm@572	51 \begin{figure}[htb]
rlm@572	52 \centering
rlm@572	53 \includegraphics[width=10cm]{./images/fat-person-sitting-at-desk.jpg}
rlm@572	54 \caption{\label{hidden-chair}The chair in this image is quite obvious to humans, but it can't be found by any modern computer vision program.}
rlm@572	55 \end{figure}
rlm@572	56
rlm@572	57 Finally, how is it that you can easily tell the difference between
rlm@572	58 how the girl's \emph{muscles} are working in figure \ref{girl}?
rlm@572	59
rlm@572	60 \begin{figure}[htb]
rlm@572	61 \centering
rlm@572	62 \includegraphics[width=7cm]{./images/wall-push.png}
rlm@572	63 \caption{\label{girl}The mysterious ``common sense'' appears here as you are able to discern the difference in how the girl's arm muscles are activated between the two images. When you compare these two images, do you feel something in your own arm muscles?}
rlm@572	64 \end{figure}
rlm@572	65
rlm@572	66 Each of these examples tells us something about what might be going
rlm@572	67 on in our minds as we easily solve these recognition problems:
rlm@572	68
rlm@572	69 \begin{itemize}
rlm@572	70 \item The hidden chair shows us that we are strongly triggered by cues
rlm@572	71 relating to the position of human bodies, and that we can
rlm@572	72 determine the overall physical configuration of a human body even
rlm@572	73 if much of that body is occluded.
rlm@572	74
rlm@572	75 \item The picture of the girl pushing against the wall tells us that we
rlm@572	76 have common sense knowledge about the kinetics of our own bodies.
rlm@572	77 We know well how our muscles would have to work to maintain us in
rlm@572	78 most positions, and we can easily project this self-knowledge to
rlm@572	79 imagined positions triggered by images of the human body.
rlm@572	80
rlm@572	81 \item The cat tells us that imagination of some kind plays an important
rlm@572	82 role in understanding actions. The question is: Can we be more
rlm@572	83 precise about what sort of imagination is required to understand
rlm@572	84 these actions?
rlm@572	85 \end{itemize}
rlm@572	86
rlm@572	87 \subsection{A step forward: the sensorimotor-centered approach}
rlm@572	88 \label{sec-1-2}
rlm@572	89
rlm@572	90 In this thesis, I explore the idea that our knowledge of our own
rlm@572	91 bodies, combined with our own rich senses, enables us to recognize
rlm@572	92 the actions of others.
rlm@572	93
rlm@572	94 For example, I think humans are able to label the cat video as
rlm@572	95 ``drinking'' because they imagine \emph{themselves} as the cat, and
rlm@572	96 imagine putting their face up against a stream of water and
rlm@572	97 sticking out their tongue. In that imagined world, they can feel
rlm@572	98 the cool water hitting their tongue, and feel the water entering
rlm@572	99 their body, and are able to recognize that \emph{feeling} as drinking.
rlm@572	100 So, the label of the action is not really in the pixels of the
rlm@572	101 image, but is found clearly in a simulation / recollection inspired
rlm@572	102 by those pixels. An imaginative system, having been trained on
rlm@572	103 drinking and non-drinking examples and learning that the most
rlm@572	104 important component of drinking is the feeling of water flowing
rlm@572	105 down one's throat, would analyze a video of a cat drinking in the
rlm@572	106 following manner:
rlm@572	107
rlm@572	108 \begin{enumerate}
rlm@572	109 \item Create a physical model of the video by putting a ``fuzzy''
rlm@572	110 model of its own body in place of the cat. Possibly also create
rlm@572	111 a simulation of the stream of water.
rlm@572	112
rlm@572	113 \item Play out this simulated scene and generate imagined sensory
rlm@572	114 experience. This will include relevant muscle contractions, a
rlm@572	115 close up view of the stream from the cat's perspective, and most
rlm@572	116 importantly, the imagined feeling of water entering the mouth.
rlm@572	117 The imagined sensory experience can come from a simulation of
rlm@572	118 the event, but can also be pattern-matched from previous,
rlm@572	119 similar embodied experience.
rlm@572	120
rlm@572	121 \item The action is now easily identified as drinking by the sense of
rlm@572	122 taste alone. The other senses (such as the tongue moving in and
rlm@572	123 out) help to give plausibility to the simulated action. Note that
rlm@572	124 the sense of vision, while critical in creating the simulation,
rlm@572	125 is not critical for identifying the action from the simulation.
rlm@572	126 \end{enumerate}
rlm@572	127
rlm@572	128 For the chair examples, the process is even easier:
rlm@572	129
rlm@572	130 \begin{enumerate}
rlm@572	131 \item Align a model of your body to the person in the image.
rlm@572	132
rlm@572	133 \item Generate proprioceptive sensory data from this alignment.
rlm@572	134
rlm@572	135 \item Use the imagined proprioceptive data as a key to lookup related
rlm@572	136 sensory experience associated with that particular proprioceptive
rlm@572	137 feeling.
rlm@572	138
rlm@572	139 \item Retrieve the feeling of your bottom resting on a surface, your
rlm@572	140 knees bent, and your leg muscles relaxed.
rlm@572	141
rlm@572	142 \item This sensory information is consistent with your \texttt{sitting?}
rlm@572	143 sensory predicate, so you (and the entity in the image) must be
rlm@572	144 sitting.
rlm@572	145
rlm@572	146 \item There must be a chair-like object since you are sitting.
rlm@572	147 \end{enumerate}
rlm@572	148
rlm@572	149 Empathy offers yet another alternative to the age-old AI
rlm@572	150 representation question: ``What is a chair?'' --- A chair is the
rlm@572	151 feeling of sitting!
rlm@572	152
rlm@572	153 One powerful advantage of empathic problem solving is that it
rlm@572	154 factors the action recognition problem into two easier problems. To
rlm@572	155 use empathy, you need an \emph{aligner}, which takes the video and a
rlm@572	156 model of your body, and aligns the model with the video. Then, you
rlm@572	157 need a \emph{recognizer}, which uses the aligned model to interpret the
rlm@572	158 action. The power in this method lies in the fact that you describe
rlm@572	159 all actions from a body-centered viewpoint. You are less tied to
rlm@572	160 the particulars of any visual representation of the actions. If you
rlm@572	161 teach the system what ``running'' is, and you have a good enough
rlm@572	162 aligner, the system will from then on be able to recognize running
rlm@572	163 from any point of view -- even strange points of view like above or
rlm@572	164 underneath the runner. This is in contrast to action recognition
rlm@572	165 schemes that try to identify actions using a non-embodied approach.
rlm@572	166 If these systems learn about running as viewed from the side, they
rlm@572	167 will not automatically be able to recognize running from any other
rlm@572	168 viewpoint.
rlm@572	169
rlm@572	170 Another powerful advantage is that using the language of multiple
rlm@572	171 body-centered rich senses to describe body-centered actions offers
rlm@572	172 a massive boost in descriptive capability. Consider how difficult
rlm@572	173 it would be to compose a set of HOG (Histogram of Oriented
rlm@572	174 Gradients) filters to describe the action of a simple worm-creature
rlm@572	175 ``curling'' so that its head touches its tail, and then behold the
rlm@572	176 simplicity of describing thus action in a language designed for the
rlm@572	177 task (listing \ref{grand-circle-intro}):
rlm@572	178
rlm@572	179 \begin{listing}
rlm@572	180 \begin{verbatim}
rlm@572	181 (defn grand-circle?
rlm@572	182 "Does the worm form a majestic circle (one end touching the other)?"
rlm@572	183 [experiences]
rlm@572	184 (and (curled? experiences)
rlm@572	185 (let [worm-touch (:touch (peek experiences))
rlm@572	186 tail-touch (worm-touch 0)
rlm@572	187 head-touch (worm-touch 4)]
rlm@572	188 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
rlm@572	189 (< 0.2 (contact worm-segment-top-tip head-touch))))))
rlm@572	190 \end{verbatim}
rlm@572	191 \caption{\label{grand-circle-intro}Body-centered actions are best expressed in a body-centered language. This code detects when the worm has curled into a full circle. Imagine how you would replicate this functionality using low-level pixel features such as HOG filters!}
rlm@572	192 \end{listing}
rlm@572	193
rlm@572	194 \subsection{\texttt{EMPATH} recognizes actions using empathy}
rlm@572	195 \label{sec-1-3}
rlm@572	196
rlm@572	197 Exploring these ideas further demands a concrete implementation, so
rlm@572	198 first, I built a system for constructing virtual creatures with
rlm@572	199 physiologically plausible sensorimotor systems and detailed
rlm@572	200 environments. The result is \texttt{CORTEX}, which I describe in chapter
rlm@572	201 \ref{sec-2}.
rlm@572	202
rlm@572	203 Next, I wrote routines which enabled a simple worm-like creature to
rlm@572	204 infer the actions of a second worm-like creature, using only its
rlm@572	205 own prior sensorimotor experiences and knowledge of the second
rlm@572	206 worm's joint positions. This program, \texttt{EMPATH}, is described in
rlm@572	207 chapter \ref{sec-3}. It's main components are:
rlm@572	208
rlm@572	209 \begin{description}
rlm@572	210 \item[{Embodied Action Definitions}] Many otherwise complicated actions
rlm@572	211 are easily described in the language of a full suite of
rlm@572	212 body-centered, rich senses and experiences. For example,
rlm@572	213 drinking is the feeling of water flowing down your throat, and
rlm@572	214 cooling your insides. It's often accompanied by bringing your
rlm@572	215 hand close to your face, or bringing your face close to water.
rlm@572	216 Sitting down is the feeling of bending your knees, activating
rlm@572	217 your quadriceps, then feeling a surface with your bottom and
rlm@572	218 relaxing your legs. These body-centered action descriptions
rlm@572	219 can be either learned or hard coded.
rlm@572	220
rlm@572	221 \item[{Guided Play }] The creature moves around and experiences the
rlm@572	222 world through its unique perspective. As the creature moves,
rlm@572	223 it gathers experiences that satisfy the embodied action
rlm@572	224 definitions.
rlm@572	225
rlm@572	226 \item[{Posture Imitation}] When trying to interpret a video or image,
rlm@572	227 the creature takes a model of itself and aligns it with
rlm@572	228 whatever it sees. This alignment might even cross species, as
rlm@572	229 when humans try to align themselves with things like ponies,
rlm@572	230 dogs, or other humans with a different body type.
rlm@572	231
rlm@572	232 \item[{Empathy }] The alignment triggers associations with
rlm@572	233 sensory data from prior experiences. For example, the
rlm@572	234 alignment itself easily maps to proprioceptive data. Any
rlm@572	235 sounds or obvious skin contact in the video can to a lesser
rlm@572	236 extent trigger previous experience keyed to hearing or touch.
rlm@572	237 Segments of previous experiences gained from play are stitched
rlm@572	238 together to form a coherent and complete sensory portrait of
rlm@572	239 the scene.
rlm@572	240
rlm@572	241 \item[{Recognition}] With the scene described in terms of remembered
rlm@572	242 first person sensory events, the creature can now run its
rlm@572	243 action-definition programs (such as the one in listing
rlm@572	244 \ref{grand-circle-intro}) on this synthesized sensory data,
rlm@572	245 just as it would if it were actually experiencing the scene
rlm@572	246 first-hand. If previous experience has been accurately
rlm@572	247 retrieved, and if it is analogous enough to the scene, then
rlm@572	248 the creature will correctly identify the action in the scene.
rlm@572	249 \end{description}
rlm@572	250
rlm@572	251 My program \texttt{EMPATH} uses this empathic problem solving technique
rlm@572	252 to interpret the actions of a simple, worm-like creature.
rlm@572	253
rlm@572	254 \begin{figure}[htb]
rlm@572	255 \centering
rlm@572	256 \includegraphics[width=15cm]{./images/worm-intro-white.png}
rlm@572	257 \caption{\label{worm-intro}The worm performs many actions during free play such as curling, wiggling, and resting.}
rlm@572	258 \end{figure}
rlm@572	259
rlm@572	260 \begin{figure}[htb]
rlm@572	261 \centering
rlm@572	262 \includegraphics[width=15cm]{./images/worm-poses.png}
rlm@572	263 \caption{\label{worm-recognition-intro}\texttt{EMPATH} recognized and classified each of these poses by inferring the complete sensory experience from proprioceptive data.}
rlm@572	264 \end{figure}
rlm@572	265
rlm@572	266 \subsubsection{Main Results}
rlm@572	267 \label{sec-1-3-1}
rlm@572	268
rlm@572	269 \begin{itemize}
rlm@572	270 \item After one-shot supervised training, \texttt{EMPATH} was able to
rlm@572	271 recognize a wide variety of static poses and dynamic
rlm@572	272 actions---ranging from curling in a circle to wiggling with a
rlm@572	273 particular frequency --- with 95$\backslash$ accuracy.
rlm@572	274
rlm@572	275 \item These results were completely independent of viewing angle
rlm@572	276 because the underlying body-centered language fundamentally is
rlm@572	277 independent; once an action is learned, it can be recognized
rlm@572	278 equally well from any viewing angle.
rlm@572	279
rlm@572	280 \item \texttt{EMPATH} is surprisingly short; the sensorimotor-centered
rlm@572	281 language provided by \texttt{CORTEX} resulted in extremely economical
rlm@572	282 recognition routines --- about 500 lines in all --- suggesting
rlm@572	283 that such representations are very powerful, and often
rlm@572	284 indispensable for the types of recognition tasks considered here.
rlm@572	285
rlm@572	286 \item For expediency's sake, I relied on direct knowledge of joint
rlm@572	287 positions in this proof of concept. However, I believe that the
rlm@572	288 structure of \texttt{EMPATH} and \texttt{CORTEX} will make future work to
rlm@572	289 enable video analysis much easier than it would otherwise be.
rlm@572	290 \end{itemize}
rlm@572	291
rlm@572	292 \subsection{\texttt{EMPATH} is built on \texttt{CORTEX}, a creature builder.}
rlm@572	293 \label{sec-1-4}
rlm@572	294
rlm@572	295 I built \texttt{CORTEX} to be a general AI research platform for doing
rlm@572	296 experiments involving multiple rich senses and a wide variety and
rlm@572	297 number of creatures. I intend it to be useful as a library for many
rlm@572	298 more projects than just this thesis. \texttt{CORTEX} was necessary to meet
rlm@572	299 a need among AI researchers at CSAIL and beyond, which is that
rlm@572	300 people often will invent wonderful ideas that are best expressed in
rlm@572	301 the language of creatures and senses, but in order to explore those
rlm@572	302 ideas they must first build a platform in which they can create
rlm@572	303 simulated creatures with rich senses! There are many ideas that
rlm@572	304 would be simple to execute (such as \texttt{EMPATH} or Larson's
rlm@572	305 self-organizing maps (\cite{larson-symbols})), but attached to them
rlm@572	306 is the multi-month effort to make a good creature simulator. Often,
rlm@572	307 that initial investment of time proves to be too much, and the
rlm@572	308 project must make do with a lesser environment or be abandoned
rlm@572	309 entirely.
rlm@572	310
rlm@572	311 \texttt{CORTEX} is well suited as an environment for embodied AI research
rlm@572	312 for three reasons:
rlm@572	313
rlm@572	314 \begin{itemize}
rlm@572	315 \item You can design new creatures using Blender (\cite{blender}), a
rlm@572	316 popular, free 3D modeling program. Each sense can be specified
rlm@572	317 using special Blender nodes with biologically inspired
rlm@572	318 parameters. You need not write any code to create a creature, and
rlm@572	319 can use a wide library of pre-existing Blender models as a base
rlm@572	320 for your own creatures.
rlm@572	321
rlm@572	322 \item \texttt{CORTEX} implements a wide variety of senses: touch,
rlm@572	323 proprioception, vision, hearing, and muscle tension. Complicated
rlm@572	324 senses like touch and vision involve multiple sensory elements
rlm@572	325 embedded in a 2D surface. You have complete control over the
rlm@572	326 distribution of these sensor elements through the use of simple
rlm@572	327 image files. \texttt{CORTEX} implements more comprehensive hearing than
rlm@572	328 any other creature simulation system available.
rlm@572	329
rlm@572	330 \item \texttt{CORTEX} supports any number of creatures and any number of
rlm@572	331 senses. Time in \texttt{CORTEX} dilates so that the simulated creatures
rlm@572	332 always perceive a perfectly smooth flow of time, regardless of
rlm@572	333 the actual computational load.
rlm@572	334 \end{itemize}
rlm@572	335
rlm@572	336 \texttt{CORTEX} is built on top of \texttt{jMonkeyEngine3}
rlm@572	337 (\cite{jmonkeyengine}), which is a video game engine designed to
rlm@572	338 create cross-platform 3D desktop games. \texttt{CORTEX} is mainly written
rlm@572	339 in clojure, a dialect of \texttt{LISP} that runs on the Java Virtual
rlm@572	340 Machine (JVM). The API for creating and simulating creatures and
rlm@572	341 senses is entirely expressed in clojure, though many senses are
rlm@572	342 implemented at the layer of jMonkeyEngine or below. For example,
rlm@572	343 for the sense of hearing I use a layer of clojure code on top of a
rlm@572	344 layer of java JNI bindings that drive a layer of \texttt{C++} code which
rlm@572	345 implements a modified version of \texttt{OpenAL} to support multiple
rlm@572	346 listeners. \texttt{CORTEX} is the only simulation environment that I know
rlm@572	347 of that can support multiple entities that can each hear the world
rlm@572	348 from their own perspective. Other senses also require a small layer
rlm@572	349 of Java code. \texttt{CORTEX} also uses \texttt{bullet}, a physics simulator
rlm@572	350 written in \texttt{C}.
rlm@572	351
rlm@572	352 \begin{figure}[htb]
rlm@572	353 \centering
rlm@572	354 \includegraphics[width=12cm]{./images/blender-worm.png}
rlm@572	355 \caption{\label{worm-recognition-intro-2}Here is the worm from figure \ref{worm-intro} modeled in Blender, a free 3D-modeling program. Senses and joints are described using special nodes in Blender.}
rlm@572	356 \end{figure}
rlm@572	357
rlm@572	358 Here are some things I anticipate that \texttt{CORTEX} might be used for:
rlm@572	359
rlm@572	360 \begin{itemize}
rlm@572	361 \item exploring new ideas about sensory integration
rlm@572	362 \item distributed communication among swarm creatures
rlm@572	363 \item self-learning using free exploration,
rlm@572	364 \item evolutionary algorithms involving creature construction
rlm@572	365 \item exploration of exotic senses and effectors that are not possible
rlm@572	366 in the real world (such as telekinesis or a semantic sense)
rlm@572	367 \item imagination using subworlds
rlm@572	368 \end{itemize}
rlm@572	369
rlm@572	370 During one test with \texttt{CORTEX}, I created 3,000 creatures each with
rlm@572	371 its own independent senses and ran them all at only 1/80 real time.
rlm@572	372 In another test, I created a detailed model of my own hand,
rlm@572	373 equipped with a realistic distribution of touch (more sensitive at
rlm@572	374 the fingertips), as well as eyes and ears, and it ran at around 1/4
rlm@572	375 real time.
rlm@572	376
rlm@572	377 \begin{sidewaysfigure}
rlm@572	378 \includegraphics[width=8.5in]{images/full-hand.png}
rlm@572	379 \caption{
rlm@572	380 I modeled my own right hand in Blender and rigged it with all the
rlm@572	381 senses that {\tt CORTEX} supports. My simulated hand has a
rlm@572	382 biologically inspired distribution of touch sensors. The senses are
rlm@572	383 displayed on the right (the red/black squares are raw sensory output),
rlm@572	384 and the simulation is displayed on the
rlm@572	385 left. Notice that my hand is curling its fingers, that it can see
rlm@572	386 its own finger from the eye in its palm, and that it can feel its
rlm@572	387 own thumb touching its palm.}
rlm@572	388 \end{sidewaysfigure}
rlm@572	389
rlm@572	390 \section{Designing \texttt{CORTEX}}
rlm@572	391 \label{sec-2}
rlm@572	392
rlm@572	393 In this chapter, I outline the design decisions that went into
rlm@572	394 making \texttt{CORTEX}, along with some details about its implementation.
rlm@572	395 (A practical guide to getting started with \texttt{CORTEX}, which skips
rlm@572	396 over the history and implementation details presented here, is
rlm@572	397 provided in an appendix at the end of this thesis.)
rlm@572	398
rlm@572	399 Throughout this project, I intended for \texttt{CORTEX} to be flexible and
rlm@572	400 extensible enough to be useful for other researchers who want to
rlm@572	401 test ideas of their own. To this end, wherever I have had to make
rlm@572	402 architectural choices about \texttt{CORTEX}, I have chosen to give as much
rlm@572	403 freedom to the user as possible, so that \texttt{CORTEX} may be used for
rlm@572	404 things I have not foreseen.
rlm@572	405
rlm@572	406 \subsection{Building in simulation versus reality}
rlm@572	407 \label{sec-2-1}
rlm@572	408 The most important architectural decision of all is the choice to
rlm@572	409 use a computer-simulated environment in the first place! The world
rlm@572	410 is a vast and rich place, and for now simulations are a very poor
rlm@572	411 reflection of its complexity. It may be that there is a significant
rlm@572	412 qualitative difference between dealing with senses in the real
rlm@572	413 world and dealing with pale facsimiles of them in a simulation
rlm@572	414 (\cite{brooks-representation}). What are the advantages and
rlm@572	415 disadvantages of a simulation vs. reality?
rlm@572	416
rlm@572	417 \subsubsection{Simulation}
rlm@572	418 \label{sec-2-1-1}
rlm@572	419
rlm@572	420 The advantages of virtual reality are that when everything is a
rlm@572	421 simulation, experiments in that simulation are absolutely
rlm@572	422 reproducible. It's also easier to change the creature and
rlm@572	423 environment to explore new situations and different sensory
rlm@572	424 combinations.
rlm@572	425
rlm@572	426 If the world is to be simulated on a computer, then not only do
rlm@572	427 you have to worry about whether the creature's senses are rich
rlm@572	428 enough to learn from the world, but whether the world itself is
rlm@572	429 rendered with enough detail and realism to give enough working
rlm@572	430 material to the creature's senses. To name just a few
rlm@572	431 difficulties facing modern physics simulators: destructibility of
rlm@572	432 the environment, simulation of water/other fluids, large areas,
rlm@572	433 nonrigid bodies, lots of objects, smoke. I don't know of any
rlm@572	434 computer simulation that would allow a creature to take a rock
rlm@572	435 and grind it into fine dust, then use that dust to make a clay
rlm@572	436 sculpture, at least not without spending years calculating the
rlm@572	437 interactions of every single small grain of dust. Maybe a
rlm@572	438 simulated world with today's limitations doesn't provide enough
rlm@572	439 richness for real intelligence to evolve.
rlm@572	440
rlm@572	441 \subsubsection{Reality}
rlm@572	442 \label{sec-2-1-2}
rlm@572	443
rlm@572	444 The other approach for playing with senses is to hook your
rlm@572	445 software up to real cameras, microphones, robots, etc., and let it
rlm@572	446 loose in the real world. This has the advantage of eliminating
rlm@572	447 concerns about simulating the world at the expense of increasing
rlm@572	448 the complexity of implementing the senses. Instead of just
rlm@572	449 grabbing the current rendered frame for processing, you have to
rlm@572	450 use an actual camera with real lenses and interact with photons to
rlm@572	451 get an image. It is much harder to change the creature, which is
rlm@572	452 now partly a physical robot of some sort, since doing so involves
rlm@572	453 changing things around in the real world instead of modifying
rlm@572	454 lines of code. While the real world is very rich and definitely
rlm@572	455 provides enough stimulation for intelligence to develop (as
rlm@572	456 evidenced by our own existence), it is also uncontrollable in the
rlm@572	457 sense that a particular situation cannot be recreated perfectly or
rlm@572	458 saved for later use. It is harder to conduct Science because it is
rlm@572	459 harder to repeat an experiment. The worst thing about using the
rlm@572	460 real world instead of a simulation is the matter of time. Instead
rlm@572	461 of simulated time you get the constant and unstoppable flow of
rlm@572	462 real time. This severely limits the sorts of software you can use
rlm@572	463 to program an AI, because all sense inputs must be handled in real
rlm@572	464 time. Complicated ideas may have to be implemented in hardware or
rlm@572	465 may simply be impossible given the current speed of our
rlm@572	466 processors. Contrast this with a simulation, in which the flow of
rlm@572	467 time in the simulated world can be slowed down to accommodate the
rlm@572	468 limitations of the creature's programming. In terms of cost, doing
rlm@572	469 everything in software is far cheaper than building custom
rlm@572	470 real-time hardware. All you need is a laptop and some patience.
rlm@572	471
rlm@572	472 \subsection{Simulated time enables rapid prototyping $\backslash$ simple programs}
rlm@572	473 \label{sec-2-2}
rlm@572	474
rlm@572	475 I envision \texttt{CORTEX} being used to support rapid prototyping and
rlm@572	476 iteration of ideas. Even if I could put together a well constructed
rlm@572	477 kit for creating robots, it would still not be enough because of
rlm@572	478 the scourge of real-time processing. Anyone who wants to test their
rlm@572	479 ideas in the real world must always worry about getting their
rlm@572	480 algorithms to run fast enough to process information in real time.
rlm@572	481 The need for real time processing only increases if multiple senses
rlm@572	482 are involved. In the extreme case, even simple algorithms will have
rlm@572	483 to be accelerated by ASIC chips or FPGAs, turning what would
rlm@572	484 otherwise be a few lines of code and a 10x speed penalty into a
rlm@572	485 multi-month ordeal. For this reason, \texttt{CORTEX} supports
rlm@572	486 \emph{time-dilation}, which scales back the framerate of the simulation
rlm@572	487 in proportion to the amount of processing each frame. From the
rlm@572	488 perspective of the creatures inside the simulation, time always
rlm@572	489 appears to flow at a constant rate, regardless of how complicated
rlm@572	490 the environment becomes or how many creatures are in the
rlm@572	491 simulation. The cost is that \texttt{CORTEX} can sometimes run slower than
rlm@572	492 real time. Time dilation works both ways, however --- simulations
rlm@572	493 of very simple creatures in \texttt{CORTEX} generally run at 40x real-time
rlm@572	494 on my machine!
rlm@572	495
rlm@572	496 \subsection{All sense organs are two-dimensional surfaces}
rlm@572	497 \label{sec-2-3}
rlm@572	498
rlm@572	499 If \texttt{CORTEX} is to support a wide variety of senses, it would help
rlm@572	500 to have a better understanding of what a sense actually is! While
rlm@572	501 vision, touch, and hearing all seem like they are quite different
rlm@572	502 things, I was surprised to learn during the course of this thesis
rlm@572	503 that they (and all physical senses) can be expressed as exactly the
rlm@572	504 same mathematical object!
rlm@572	505
rlm@572	506 Human beings are three-dimensional objects, and the nerves that
rlm@572	507 transmit data from our various sense organs to our brain are
rlm@572	508 essentially one-dimensional. This leaves up to two dimensions in
rlm@572	509 which our sensory information may flow. For example, imagine your
rlm@572	510 skin: it is a two-dimensional surface around a three-dimensional
rlm@572	511 object (your body). It has discrete touch sensors embedded at
rlm@572	512 various points, and the density of these sensors corresponds to the
rlm@572	513 sensitivity of that region of skin. Each touch sensor connects to a
rlm@572	514 nerve, all of which eventually are bundled together as they travel
rlm@572	515 up the spinal cord to the brain. Intersect the spinal nerves with a
rlm@572	516 guillotining plane and you will see all of the sensory data of the
rlm@572	517 skin revealed in a roughly circular two-dimensional image which is
rlm@572	518 the cross section of the spinal cord. Points on this image that are
rlm@572	519 close together in this circle represent touch sensors that are
rlm@572	520 \emph{probably} close together on the skin, although there is of course
rlm@572	521 some cutting and rearrangement that has to be done to transfer the
rlm@572	522 complicated surface of the skin onto a two dimensional image.
rlm@572	523
rlm@572	524 Most human senses consist of many discrete sensors of various
rlm@572	525 properties distributed along a surface at various densities. For
rlm@572	526 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
rlm@572	527 disks, and Ruffini's endings (\cite{textbook901}), which detect
rlm@572	528 pressure and vibration of various intensities. For ears, it is the
rlm@572	529 stereocilia distributed along the basilar membrane inside the
rlm@572	530 cochlea; each one is sensitive to a slightly different frequency of
rlm@572	531 sound. For eyes, it is rods and cones distributed along the surface
rlm@572	532 of the retina. In each case, we can describe the sense with a
rlm@572	533 surface and a distribution of sensors along that surface.
rlm@572	534
rlm@572	535 In fact, almost every human sense can be effectively described in
rlm@572	536 terms of a surface containing embedded sensors. If the sense had
rlm@572	537 any more dimensions, then there wouldn't be enough room in the
rlm@572	538 spinal cord to transmit the information!
rlm@572	539
rlm@572	540 Therefore, \texttt{CORTEX} must support the ability to create objects and
rlm@572	541 then be able to ``paint'' points along their surfaces to describe
rlm@572	542 each sense.
rlm@572	543
rlm@572	544 Fortunately this idea is already a well known computer graphics
rlm@572	545 technique called \emph{UV-mapping}. In UV-mapping, the three-dimensional
rlm@572	546 surface of a model is cut and smooshed until it fits on a
rlm@572	547 two-dimensional image. You paint whatever you want on that image,
rlm@572	548 and when the three-dimensional shape is rendered in a game the
rlm@572	549 smooshing and cutting is reversed and the image appears on the
rlm@572	550 three-dimensional object.
rlm@572	551
rlm@572	552 To make a sense, interpret the UV-image as describing the
rlm@572	553 distribution of that senses' sensors. To get different types of
rlm@572	554 sensors, you can either use a different color for each type of
rlm@572	555 sensor, or use multiple UV-maps, each labeled with that sensor
rlm@572	556 type. I generally use a white pixel to mean the presence of a
rlm@572	557 sensor and a black pixel to mean the absence of a sensor, and use
rlm@572	558 one UV-map for each sensor-type within a given sense.
rlm@572	559
rlm@572	560 \begin{figure}[htb]
rlm@572	561 \centering
rlm@572	562 \includegraphics[width=10cm]{./images/finger-UV.png}
rlm@572	563 \caption{\label{finger-UV}The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.}
rlm@572	564 \end{figure}
rlm@572	565
rlm@572	566 \begin{figure}[htb]
rlm@572	567 \centering
rlm@572	568 \includegraphics[width=10cm]{./images/finger-1.png}
rlm@572	569 \caption{\label{finger-side-view}Ventral side of the UV-mapped finger. Note the density of touch sensors at the tip.}
rlm@572	570 \end{figure}
rlm@572	571
rlm@572	572 \subsection{Video game engines provide ready-made physics and shading}
rlm@572	573 \label{sec-2-4}
rlm@572	574
rlm@572	575 I did not need to write my own physics simulation code or shader to
rlm@572	576 build \texttt{CORTEX}. Doing so would lead to a system that is impossible
rlm@572	577 for anyone but myself to use anyway. Instead, I use a video game
rlm@572	578 engine as a base and modify it to accommodate the additional needs
rlm@572	579 of \texttt{CORTEX}. Video game engines are an ideal starting point to
rlm@572	580 build \texttt{CORTEX}, because they are not far from being creature
rlm@572	581 building systems themselves.
rlm@572	582
rlm@572	583 First off, general purpose video game engines come with a physics
rlm@572	584 engine and lighting / sound system. The physics system provides
rlm@572	585 tools that can be co-opted to serve as touch, proprioception, and
rlm@572	586 muscles. Because some games support split screen views, a good
rlm@572	587 video game engine will allow you to efficiently create multiple
rlm@572	588 cameras in the simulated world that can be used as eyes. Video game
rlm@572	589 systems offer integrated asset management for things like textures
rlm@572	590 and creature models, providing an avenue for defining creatures.
rlm@572	591 They also understand UV-mapping, because this technique is used to
rlm@572	592 apply a texture to a model. Finally, because video game engines
rlm@572	593 support a large number of developers, as long as \texttt{CORTEX} doesn't
rlm@572	594 stray too far from the base system, other researchers can turn to
rlm@572	595 this community for help when doing their research.
rlm@572	596
rlm@572	597 \subsection{\texttt{CORTEX} is based on jMonkeyEngine3}
rlm@572	598 \label{sec-2-5}
rlm@572	599
rlm@572	600 While preparing to build \texttt{CORTEX} I studied several video game
rlm@572	601 engines to see which would best serve as a base. The top contenders
rlm@572	602 were:
rlm@572	603
rlm@572	604 \begin{description}
rlm@572	605 \item[{\href{http://www.idsoftware.com}{Quake II}/\href{http://www.bytonic.de/html/jake2.html}{Jake2}}] The Quake II engine was designed by ID software
rlm@572	606 in 1997. All the source code was released by ID software into
rlm@572	607 the Public Domain several years ago, and as a result it has
rlm@572	608 been ported to many different languages. This engine was
rlm@572	609 famous for its advanced use of realistic shading and it had
rlm@572	610 decent and fast physics simulation. The main advantage of the
rlm@572	611 Quake II engine is its simplicity, but I ultimately rejected
rlm@572	612 it because the engine is too tied to the concept of a
rlm@572	613 first-person shooter game. One of the problems I had was that
rlm@572	614 there does not seem to be any easy way to attach multiple
rlm@572	615 cameras to a single character. There are also several physics
rlm@572	616 clipping issues that are corrected in a way that only applies
rlm@572	617 to the main character and do not apply to arbitrary objects.
rlm@572	618
rlm@572	619 \item[{\href{http://source.valvesoftware.com/}{Source Engine} }] The Source Engine evolved from the Quake II
rlm@572	620 and Quake I engines and is used by Valve in the Half-Life
rlm@572	621 series of games. The physics simulation in the Source Engine
rlm@572	622 is quite accurate and probably the best out of all the engines
rlm@572	623 I investigated. There is also an extensive community actively
rlm@572	624 working with the engine. However, applications that use the
rlm@572	625 Source Engine must be written in C++, the code is not open, it
rlm@572	626 only runs on Windows, and the tools that come with the SDK to
rlm@572	627 handle models and textures are complicated and awkward to use.
rlm@572	628
rlm@572	629 \item[{\href{http://jmonkeyengine.com/}{jMonkeyEngine3}}] jMonkeyEngine3 is a new library for creating
rlm@572	630 games in Java. It uses OpenGL to render to the screen and uses
rlm@572	631 screengraphs to avoid drawing things that do not appear on the
rlm@572	632 screen. It has an active community and several games in the
rlm@572	633 pipeline. The engine was not built to serve any particular
rlm@572	634 game but is instead meant to be used for any 3D game.
rlm@572	635 \end{description}
rlm@572	636
rlm@572	637 I chose jMonkeyEngine3 because it had the most features out of all
rlm@572	638 the free projects I looked at, and because I could then write my
rlm@572	639 code in clojure, an implementation of \texttt{LISP} that runs on the JVM.
rlm@572	640
rlm@572	641 \subsection{\texttt{CORTEX} uses Blender to create creature models}
rlm@572	642 \label{sec-2-6}
rlm@572	643
rlm@572	644 For the simple worm-like creatures I will use later on in this
rlm@572	645 thesis, I could define a simple API in \texttt{CORTEX} that would allow
rlm@572	646 one to create boxes, spheres, etc., and leave that API as the sole
rlm@572	647 way to create creatures. However, for \texttt{CORTEX} to truly be useful
rlm@572	648 for other projects, it needs a way to construct complicated
rlm@572	649 creatures. If possible, it would be nice to leverage work that has
rlm@572	650 already been done by the community of 3D modelers, or at least
rlm@572	651 enable people who are talented at modeling but not programming to
rlm@572	652 design \texttt{CORTEX} creatures.
rlm@572	653
rlm@572	654 Therefore I use Blender, a free 3D modeling program, as the main
rlm@572	655 way to create creatures in \texttt{CORTEX}. However, the creatures modeled
rlm@572	656 in Blender must also be simple to simulate in jMonkeyEngine3's game
rlm@572	657 engine, and must also be easy to rig with \texttt{CORTEX}'s senses. I
rlm@572	658 accomplish this with extensive use of Blender's ``empty nodes.''
rlm@572	659
rlm@572	660 Empty nodes have no mass, physical presence, or appearance, but
rlm@572	661 they can hold metadata and have names. I use a tree structure of
rlm@572	662 empty nodes to specify senses in the following manner:
rlm@572	663
rlm@572	664 \begin{itemize}
rlm@572	665 \item Create a single top-level empty node whose name is the name of
rlm@572	666 the sense.
rlm@572	667 \item Add empty nodes which each contain meta-data relevant to the
rlm@572	668 sense, including a UV-map describing the number/distribution of
rlm@572	669 sensors if applicable.
rlm@572	670 \item Make each empty-node the child of the top-level node.
rlm@572	671 \end{itemize}
rlm@572	672
rlm@572	673 \begin{figure}[htb]
rlm@572	674 \centering
rlm@572	675 \includegraphics[width=10cm]{./images/empty-sense-nodes.png}
rlm@572	676 \caption{\label{sense-nodes}An example of annotating a creature model with empty nodes to describe the layout of senses. There are multiple empty nodes which each describe the position of muscles, ears, eyes, or joints.}
rlm@572	677 \end{figure}
rlm@572	678
rlm@572	679 \subsection{Bodies are composed of segments connected by joints}
rlm@572	680 \label{sec-2-7}
rlm@572	681
rlm@572	682 Blender is a general purpose animation tool, which has been used in
rlm@572	683 the past to create high quality movies such as Sintel
rlm@572	684 (\cite{blender}). Though Blender can model and render even
rlm@572	685 complicated things like water, it is crucial to keep models that
rlm@572	686 are meant to be simulated as creatures simple. \texttt{Bullet}, which
rlm@572	687 \texttt{CORTEX} uses though jMonkeyEngine3, is a rigid-body physics
rlm@572	688 system. This offers a compromise between the expressiveness of a
rlm@572	689 game level and the speed at which it can be simulated, and it means
rlm@572	690 that creatures should be naturally expressed as rigid components
rlm@572	691 held together by joint constraints.
rlm@572	692
rlm@572	693 But humans are more like a squishy bag wrapped around some hard
rlm@572	694 bones which define the overall shape. When we move, our skin bends
rlm@572	695 and stretches to accommodate the new positions of our bones.
rlm@572	696
rlm@572	697 One way to make bodies composed of rigid pieces connected by joints
rlm@572	698 \emph{seem} more human-like is to use an \emph{armature}, (or \emph{rigging})
rlm@572	699 system, which defines a overall ``body mesh'' and defines how the
rlm@572	700 mesh deforms as a function of the position of each ``bone'' which
rlm@572	701 is a standard rigid body. This technique is used extensively to
rlm@572	702 model humans and create realistic animations. It is not a good
rlm@572	703 technique for physical simulation because it is a lie -- the skin
rlm@572	704 is not a physical part of the simulation and does not interact with
rlm@572	705 any objects in the world or itself. Objects will pass right though
rlm@572	706 the skin until they come in contact with the underlying bone, which
rlm@572	707 is a physical object. Without simulating the skin, the sense of
rlm@572	708 touch has little meaning, and the creature's own vision will lie to
rlm@572	709 it about the true extent of its body. Simulating the skin as a
rlm@572	710 physical object requires some way to continuously update the
rlm@572	711 physical model of the skin along with the movement of the bones,
rlm@572	712 which is unacceptably slow compared to rigid body simulation.
rlm@572	713
rlm@572	714 Therefore, instead of using the human-like ``bony meatbag''
rlm@572	715 approach, I decided to base my body plans on multiple solid objects
rlm@572	716 that are connected by joints, inspired by the robot \texttt{EVE} from the
rlm@572	717 movie WALL-E.
rlm@572	718
rlm@572	719 \begin{figure}[htb]
rlm@572	720 \centering
rlm@572	721 \includegraphics[width=10cm]{./images/Eve.jpg}
rlm@572	722 \caption{\texttt{EVE} from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one.}
rlm@572	723 \end{figure}
rlm@572	724
rlm@572	725 \texttt{EVE}'s body is composed of several rigid components that are held
rlm@572	726 together by invisible joint constraints. This is what I mean by
rlm@572	727 \emph{eve-like}. The main reason that I use eve-like bodies is for
rlm@572	728 simulation efficiency, and so that there will be correspondence
rlm@572	729 between the AI's senses and the physical presence of its body. Each
rlm@572	730 individual section is simulated by a separate rigid body that
rlm@572	731 corresponds exactly with its visual representation and does not
rlm@572	732 change. Sections are connected by invisible joints that are well
rlm@572	733 supported in jMonkeyEngine3. Bullet, the physics backend for
rlm@572	734 jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
rlm@572	735 connected by joints. Just because sections are rigid does not mean
rlm@572	736 they have to stay as one piece forever; they can be dynamically
rlm@572	737 replaced with multiple sections to simulate splitting in two. This
rlm@572	738 could be used to simulate retractable claws or \texttt{EVE}'s hands, which
rlm@572	739 are able to coalesce into one object in the movie.
rlm@572	740
rlm@572	741 \subsubsection{Solidifying/Connecting a body}
rlm@572	742 \label{sec-2-7-1}
rlm@572	743
rlm@572	744 \texttt{CORTEX} creates a creature in two steps: first, it traverses the
rlm@572	745 nodes in the Blender file and creates physical representations for
rlm@572	746 any of them that have mass defined in their Blender meta-data.
rlm@572	747
rlm@572	748 \begin{listing}
rlm@572	749 \begin{verbatim}
rlm@572	750 (defn physical!
rlm@572	751 "Iterate through the nodes in creature and make them real physical
rlm@572	752 objects in the simulation."
rlm@572	753 [#^Node creature]
rlm@572	754 (dorun
rlm@572	755 (map
rlm@572	756 (fn [geom]
rlm@572	757 (let [physics-control
rlm@572	758 (RigidBodyControl.
rlm@572	759 (HullCollisionShape.
rlm@572	760 (.getMesh geom))
rlm@572	761 (if-let [mass (meta-data geom "mass")]
rlm@572	762 (float mass) (float 1)))]
rlm@572	763 (.addControl geom physics-control)))
rlm@572	764 (filter #(isa? (class %) Geometry )
rlm@572	765 (node-seq creature)))))
rlm@572	766 \end{verbatim}
rlm@572	767 \caption{\label{physical}Program for iterating through the nodes in a Blender file and generating physical jMonkeyEngine3 objects with mass and a matching physics shape.}
rlm@572	768 \end{listing}
rlm@572	769
rlm@572	770 The next step to making a proper body is to connect those pieces
rlm@572	771 together with joints. jMonkeyEngine has a large array of joints
rlm@572	772 available via \texttt{bullet}, such as Point2Point, Cone, Hinge, and a
rlm@572	773 generic Six Degree of Freedom joint, with or without spring
rlm@572	774 restitution.
rlm@572	775
rlm@572	776 Joints are treated a lot like proper senses, in that there is a
rlm@572	777 top-level empty node named ``joints'' whose children each
rlm@572	778 represent a joint.
rlm@572	779
rlm@572	780 \begin{figure}[htb]
rlm@572	781 \centering
rlm@572	782 \includegraphics[width=10cm]{./images/hand-screenshot1.png}
rlm@572	783 \caption{\label{blender-hand}View of the hand model in Blender showing the main ``joints'' node (highlighted in yellow) and its children which each represent a joint in the hand. Each joint node has metadata specifying what sort of joint it is.}
rlm@572	784 \end{figure}
rlm@572	785
rlm@572	786
rlm@572	787 \texttt{CORTEX}'s procedure for binding the creature together with joints
rlm@572	788 is as follows:
rlm@572	789
rlm@572	790 \begin{itemize}
rlm@572	791 \item Find the children of the ``joints'' node.
rlm@572	792 \item Determine the two spatials the joint is meant to connect.
rlm@572	793 \item Create the joint based on the meta-data of the empty node.
rlm@572	794 \end{itemize}
rlm@572	795
rlm@572	796 The higher order function \texttt{sense-nodes} from \texttt{cortex.sense}
rlm@572	797 simplifies finding the joints based on their parent ``joints''
rlm@572	798 node.
rlm@572	799
rlm@572	800 \begin{listing}
rlm@572	801 \begin{verbatim}
rlm@572	802 (defn sense-nodes
rlm@572	803 "For some senses there is a special empty Blender node whose
rlm@572	804 children are considered markers for an instance of that sense. This
rlm@572	805 function generates functions to find those children, given the name
rlm@572	806 of the special parent node."
rlm@572	807 [parent-name]
rlm@572	808 (fn [#^Node creature]
rlm@572	809 (if-let [sense-node (.getChild creature parent-name)]
rlm@572	810 (seq (.getChildren sense-node)) [])))
rlm@572	811
rlm@572	812 (def
rlm@572	813 ^{:doc "Return the children of the creature's \"joints\" node."
rlm@572	814 :arglists '([creature])}
rlm@572	815 joints
rlm@572	816 (sense-nodes "joints"))
rlm@572	817 \end{verbatim}
rlm@572	818 \caption{\label{get-empty-nodes}Retrieving the children empty nodes from a single named empty node is a common pattern in \texttt{CORTEX}. Further instances of this technique for the senses will be omitted}
rlm@572	819 \end{listing}
rlm@572	820
rlm@572	821 To find a joint's targets, \texttt{CORTEX} creates a small cube, centered
rlm@572	822 around the empty-node, and grows the cube exponentially until it
rlm@572	823 intersects two physical objects. The objects are ordered according
rlm@572	824 to the joint's rotation, with the first one being the object that
rlm@572	825 has more negative coordinates in the joint's reference frame.
rlm@572	826 Because the objects must be physical, the empty-node itself
rlm@572	827 escapes detection. Because the objects must be physical,
rlm@572	828 \texttt{joint-targets} must be called \emph{after} \texttt{physical!} is called.
rlm@572	829
rlm@572	830 \begin{listing}
rlm@572	831 \begin{verbatim}
rlm@572	832 (defn joint-targets
rlm@572	833 "Return the two closest two objects to the joint object, ordered
rlm@572	834 from bottom to top according to the joint's rotation."
rlm@572	835 [#^Node parts #^Node joint]
rlm@572	836 (loop [radius (float 0.01)]
rlm@572	837 (let [results (CollisionResults.)]
rlm@572	838 (.collideWith
rlm@572	839 parts
rlm@572	840 (BoundingBox. (.getWorldTranslation joint)
rlm@572	841 radius radius radius) results)
rlm@572	842 (let [targets
rlm@572	843 (distinct
rlm@572	844 (map #(.getGeometry %) results))]
rlm@572	845 (if (>= (count targets) 2)
rlm@572	846 (sort-by
rlm@572	847 #(let [joint-ref-frame-position
rlm@572	848 (jme-to-blender
rlm@572	849 (.mult
rlm@572	850 (.inverse (.getWorldRotation joint))
rlm@572	851 (.subtract (.getWorldTranslation %)
rlm@572	852 (.getWorldTranslation joint))))]
rlm@572	853 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
rlm@572	854 (take 2 targets))
rlm@572	855 (recur (float (* radius 2))))))))
rlm@572	856 \end{verbatim}
rlm@572	857 \caption{\label{joint-targets}Program to find the targets of a joint node by exponentially growth of a search cube.}
rlm@572	858 \end{listing}
rlm@572	859
rlm@572	860 Once \texttt{CORTEX} finds all joints and targets, it creates them using
rlm@572	861 a dispatch on the metadata of each joint node.
rlm@572	862
rlm@572	863 \begin{listing}
rlm@572	864 \begin{verbatim}
rlm@572	865 (defmulti joint-dispatch
rlm@572	866 "Translate Blender pseudo-joints into real JME joints."
rlm@572	867 (fn [constraints & _]
rlm@572	868 (:type constraints)))
rlm@572	869
rlm@572	870 (defmethod joint-dispatch :point
rlm@572	871 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572	872 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
rlm@572	873 (.setLinearLowerLimit Vector3f/ZERO)
rlm@572	874 (.setLinearUpperLimit Vector3f/ZERO)))
rlm@572	875
rlm@572	876 (defmethod joint-dispatch :hinge
rlm@572	877 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572	878 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
rlm@572	879 [limit-1 limit-2] (:limit constraints)
rlm@572	880 hinge-axis (.mult rotation (blender-to-jme axis))]
rlm@572	881 (doto (HingeJoint. control-a control-b pivot-a pivot-b
rlm@572	882 hinge-axis hinge-axis)
rlm@572	883 (.setLimit limit-1 limit-2))))
rlm@572	884
rlm@572	885 (defmethod joint-dispatch :cone
rlm@572	886 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572	887 (let [limit-xz (:limit-xz constraints)
rlm@572	888 limit-xy (:limit-xy constraints)
rlm@572	889 twist (:twist constraints)]
rlm@572	890 (doto (ConeJoint. control-a control-b pivot-a pivot-b
rlm@572	891 rotation rotation)
rlm@572	892 (.setLimit (float limit-xz) (float limit-xy)
rlm@572	893 (float twist)))))
rlm@572	894 \end{verbatim}
rlm@572	895 \caption{\label{joint-dispatch}Program to dispatch on Blender metadata and create joints suitable for physical simulation.}
rlm@572	896 \end{listing}
rlm@572	897
rlm@572	898 All that is left for joints is to combine the above pieces into
rlm@572	899 something that can operate on the collection of nodes that a
rlm@572	900 Blender file represents.
rlm@572	901
rlm@572	902 \begin{listing}
rlm@572	903 \begin{verbatim}
rlm@572	904 (defn connect
rlm@572	905 "Create a joint between 'obj-a and 'obj-b at the location of
rlm@572	906 'joint. The type of joint is determined by the metadata on 'joint.
rlm@572	907
rlm@572	908 Here are some examples:
rlm@572	909 {:type :point}
rlm@572	910 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
rlm@572	911 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572	912
rlm@572	913 {:type :cone :limit-xz 0]
rlm@572	914 :limit-xy 0]
rlm@572	915 :twist 0]} (use XZY rotation mode in Blender!)"
rlm@572	916 [#^Node obj-a #^Node obj-b #^Node joint]
rlm@572	917 (let [control-a (.getControl obj-a RigidBodyControl)
rlm@572	918 control-b (.getControl obj-b RigidBodyControl)
rlm@572	919 joint-center (.getWorldTranslation joint)
rlm@572	920 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
rlm@572	921 pivot-a (world-to-local obj-a joint-center)
rlm@572	922 pivot-b (world-to-local obj-b joint-center)]
rlm@572	923 (if-let
rlm@572	924 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
rlm@572	925 ;; A side-effect of creating a joint registers
rlm@572	926 ;; it with both physics objects which in turn
rlm@572	927 ;; will register the joint with the physics system
rlm@572	928 ;; when the simulation is started.
rlm@572	929 (joint-dispatch constraints
rlm@572	930 control-a control-b
rlm@572	931 pivot-a pivot-b
rlm@572	932 joint-rotation))))
rlm@572	933 \end{verbatim}
rlm@572	934 \caption{\label{connect}Program to completely create a joint given information from a Blender file.}
rlm@572	935 \end{listing}
rlm@572	936
rlm@572	937 In general, whenever \texttt{CORTEX} exposes a sense (or in this case
rlm@572	938 physicality), it provides a function of the type \texttt{sense!}, which
rlm@572	939 takes in a collection of nodes and augments it to support that
rlm@572	940 sense. The function returns any controls necessary to use that
rlm@572	941 sense. In this case \texttt{body!} creates a physical body and returns no
rlm@572	942 control functions.
rlm@572	943
rlm@572	944 \begin{listing}
rlm@572	945 \begin{verbatim}
rlm@572	946 (defn joints!
rlm@572	947 "Connect the solid parts of the creature with physical joints. The
rlm@572	948 joints are taken from the \"joints\" node in the creature."
rlm@572	949 [#^Node creature]
rlm@572	950 (dorun
rlm@572	951 (map
rlm@572	952 (fn [joint]
rlm@572	953 (let [[obj-a obj-b] (joint-targets creature joint)]
rlm@572	954 (connect obj-a obj-b joint)))
rlm@572	955 (joints creature))))
rlm@572	956 (defn body!
rlm@572	957 "Endow the creature with a physical body connected with joints. The
rlm@572	958 particulars of the joints and the masses of each body part are
rlm@572	959 determined in Blender."
rlm@572	960 [#^Node creature]
rlm@572	961 (physical! creature)
rlm@572	962 (joints! creature))
rlm@572	963 \end{verbatim}
rlm@572	964 \caption{\label{joints}Program to give joints to a creature.}
rlm@572	965 \end{listing}
rlm@572	966
rlm@572	967 All of the code you have just seen amounts to only 130 lines, yet
rlm@572	968 because it builds on top of Blender and jMonkeyEngine3, those few
rlm@572	969 lines pack quite a punch!
rlm@572	970
rlm@572	971 The hand from figure \ref{blender-hand}, which was modeled after
rlm@572	972 my own right hand, can now be given joints and simulated as a
rlm@572	973 creature.
rlm@572	974
rlm@572	975 \begin{figure}[htb]
rlm@572	976 \centering
rlm@572	977 \includegraphics[width=15cm]{./images/physical-hand.png}
rlm@572	978 \caption{\label{physical-hand}With the ability to create physical creatures from Blender, \texttt{CORTEX} gets one step closer to becoming a full creature simulation environment.}
rlm@572	979 \end{figure}
rlm@572	980
rlm@572	981 \subsection{Sight reuses standard video game components\ldots{}}
rlm@572	982 \label{sec-2-8}
rlm@572	983
rlm@572	984 Vision is one of the most important senses for humans, so I need to
rlm@572	985 build a simulated sense of vision for my AI. I will do this with
rlm@572	986 simulated eyes. Each eye can be independently moved and should see
rlm@572	987 its own version of the world depending on where it is.
rlm@572	988
rlm@572	989 Making these simulated eyes a reality is simple because
rlm@572	990 jMonkeyEngine already contains extensive support for multiple views
rlm@572	991 of the same 3D simulated world. The reason jMonkeyEngine has this
rlm@572	992 support is because the support is necessary to create games with
rlm@572	993 split-screen views. Multiple views are also used to create
rlm@572	994 efficient pseudo-reflections by rendering the scene from a certain
rlm@572	995 perspective and then projecting it back onto a surface in the 3D
rlm@572	996 world.
rlm@572	997
rlm@572	998 \begin{figure}[htb]
rlm@572	999 \centering
rlm@572	1000 \includegraphics[width=10cm]{./images/goldeneye-4-player.png}
rlm@572	1001 \caption{\label{goldeneye}jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.}
rlm@572	1002 \end{figure}
rlm@572	1003
rlm@572	1004 \subsubsection{A Brief Description of jMonkeyEngine's Rendering Pipeline}
rlm@572	1005 \label{sec-2-8-1}
rlm@572	1006
rlm@572	1007 jMonkeyEngine allows you to create a \texttt{ViewPort}, which represents a
rlm@572	1008 view of the simulated world. You can create as many of these as you
rlm@572	1009 want. Every frame, the \texttt{RenderManager} iterates through each
rlm@572	1010 \texttt{ViewPort}, rendering the scene in the GPU. For each \texttt{ViewPort} there
rlm@572	1011 is a \texttt{FrameBuffer} which represents the rendered image in the GPU.
rlm@572	1012
rlm@572	1013 \begin{figure}[htb]
rlm@572	1014 \centering
rlm@572	1015 \includegraphics[width=10cm]{./images/diagram_rendermanager2.png}
rlm@572	1016 \caption{\label{rendermanagers}\texttt{ViewPorts} are cameras in the world. During each frame, the \texttt{RenderManager} records a snapshot of what each view is currently seeing; these snapshots are \texttt{FrameBuffer} objects.}
rlm@572	1017 \end{figure}
rlm@572	1018
rlm@572	1019 Each \texttt{ViewPort} can have any number of attached \texttt{SceneProcessor}
rlm@572	1020 objects, which are called every time a new frame is rendered. A
rlm@572	1021 \texttt{SceneProcessor} receives its \texttt{ViewPort's} \texttt{FrameBuffer} and can do
rlm@572	1022 whatever it wants to the data. Often this consists of invoking GPU
rlm@572	1023 specific operations on the rendered image. The \texttt{SceneProcessor} can
rlm@572	1024 also copy the GPU image data to RAM and process it with the CPU.
rlm@572	1025
rlm@572	1026 \subsubsection{Appropriating Views for Vision}
rlm@572	1027 \label{sec-2-8-2}
rlm@572	1028
rlm@572	1029 Each eye in the simulated creature needs its own \texttt{ViewPort} so
rlm@572	1030 that it can see the world from its own perspective. To this
rlm@572	1031 \texttt{ViewPort}, I add a \texttt{SceneProcessor} that feeds the visual data to
rlm@572	1032 any arbitrary continuation function for further processing. That
rlm@572	1033 continuation function may perform both CPU and GPU operations on
rlm@572	1034 the data. To make this easy for the continuation function, the
rlm@572	1035 \texttt{SceneProcessor} maintains appropriately sized buffers in RAM to
rlm@572	1036 hold the data. It does not do any copying from the GPU to the CPU
rlm@572	1037 itself because it is a slow operation.
rlm@572	1038
rlm@572	1039 \begin{listing}
rlm@572	1040 \begin{verbatim}
rlm@572	1041 (defn vision-pipeline
rlm@572	1042 "Create a SceneProcessor object which wraps a vision processing
rlm@572	1043 continuation function. The continuation is a function that takes
rlm@572	1044 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@572	1045 each of which has already been appropriately sized."
rlm@572	1046 [continuation]
rlm@572	1047 (let [byte-buffer (atom nil)
rlm@572	1048 renderer (atom nil)
rlm@572	1049 image (atom nil)]
rlm@572	1050 (proxy [SceneProcessor] []
rlm@572	1051 (initialize
rlm@572	1052 [renderManager viewPort]
rlm@572	1053 (let [cam (.getCamera viewPort)
rlm@572	1054 width (.getWidth cam)
rlm@572	1055 height (.getHeight cam)]
rlm@572	1056 (reset! renderer (.getRenderer renderManager))
rlm@572	1057 (reset! byte-buffer
rlm@572	1058 (BufferUtils/createByteBuffer
rlm@572	1059 (* width height 4)))
rlm@572	1060 (reset! image (BufferedImage.
rlm@572	1061 width height
rlm@572	1062 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@572	1063 (isInitialized [] (not (nil? @byte-buffer)))
rlm@572	1064 (reshape [_ _ _])
rlm@572	1065 (preFrame [_])
rlm@572	1066 (postQueue [_])
rlm@572	1067 (postFrame
rlm@572	1068 [#^FrameBuffer fb]
rlm@572	1069 (.clear @byte-buffer)
rlm@572	1070 (continuation @renderer fb @byte-buffer @image))
rlm@572	1071 (cleanup []))))
rlm@572	1072 \end{verbatim}
rlm@572	1073 \caption{\label{pipeline-1}Function to make the rendered scene in jMonkeyEngine available for further processing.}
rlm@572	1074 \end{listing}
rlm@572	1075
rlm@572	1076 The continuation function given to \texttt{vision-pipeline} above will be
rlm@572	1077 given a \texttt{Renderer} and three containers for image data. The
rlm@572	1078 \texttt{FrameBuffer} references the GPU image data, but the pixel data
rlm@572	1079 can not be used directly on the CPU. The \texttt{ByteBuffer} and
rlm@572	1080 \texttt{BufferedImage} are initially "empty" but are sized to hold the
rlm@572	1081 data in the \texttt{FrameBuffer}. I call transferring the GPU image data
rlm@572	1082 to the CPU structures "mixing" the image data.
rlm@572	1083
rlm@572	1084 \subsubsection{Optical sensor arrays are described with images and referenced with metadata}
rlm@572	1085 \label{sec-2-8-3}
rlm@572	1086
rlm@572	1087 The vision pipeline described above handles the flow of rendered
rlm@572	1088 images. Now, \texttt{CORTEX} needs simulated eyes to serve as the source
rlm@572	1089 of these images.
rlm@572	1090
rlm@572	1091 An eye is described in Blender in the same way as a joint. They
rlm@572	1092 are zero dimensional empty objects with no geometry whose local
rlm@572	1093 coordinate system determines the orientation of the resulting eye.
rlm@572	1094 All eyes are children of a parent node named "eyes" just as all
rlm@572	1095 joints have a parent named "joints". An eye binds to the nearest
rlm@572	1096 physical object with \texttt{bind-sense}.
rlm@572	1097
rlm@572	1098 \begin{listing}
rlm@572	1099 \begin{verbatim}
rlm@572	1100 (defn add-eye!
rlm@572	1101 "Create a Camera centered on the current position of 'eye which
rlm@572	1102 follows the closest physical node in 'creature. The camera will
rlm@572	1103 point in the X direction and use the Z vector as up as determined
rlm@572	1104 by the rotation of these vectors in Blender coordinate space. Use
rlm@572	1105 XZY rotation for the node in Blender."
rlm@572	1106 [#^Node creature #^Spatial eye]
rlm@572	1107 (let [target (closest-node creature eye)
rlm@572	1108 [cam-width cam-height]
rlm@572	1109 ;;[640 480] ;; graphics card on laptop doesn't support
rlm@572	1110 ;; arbitrary dimensions.
rlm@572	1111 (eye-dimensions eye)
rlm@572	1112 cam (Camera. cam-width cam-height)
rlm@572	1113 rot (.getWorldRotation eye)]
rlm@572	1114 (.setLocation cam (.getWorldTranslation eye))
rlm@572	1115 (.lookAtDirection
rlm@572	1116 cam ; this part is not a mistake and
rlm@572	1117 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
rlm@572	1118 (.mult rot Vector3f/UNIT_Y)) ; Blender as the UP vector.
rlm@572	1119 (.setFrustumPerspective
rlm@572	1120 cam (float 45)
rlm@572	1121 (float (/ (.getWidth cam) (.getHeight cam)))
rlm@572	1122 (float 1)
rlm@572	1123 (float 1000))
rlm@572	1124 (bind-sense target cam) cam))
rlm@572	1125 \end{verbatim}
rlm@572	1126 \caption{\label{add-eye}Here, the camera is created based on metadata on the eye-node and attached to the nearest physical object with \texttt{bind-sense}}
rlm@572	1127 \end{listing}
rlm@572	1128
rlm@572	1129 \subsubsection{Simulated Retina}
rlm@572	1130 \label{sec-2-8-4}
rlm@572	1131
rlm@572	1132 An eye is a surface (the retina) which contains many discrete
rlm@572	1133 sensors to detect light. These sensors can have different
rlm@572	1134 light-sensing properties. In humans, each discrete sensor is
rlm@572	1135 sensitive to red, blue, green, or gray. These different types of
rlm@572	1136 sensors can have different spatial distributions along the retina.
rlm@572	1137 In humans, there is a fovea in the center of the retina which has
rlm@572	1138 a very high density of color sensors, and a blind spot which has
rlm@572	1139 no sensors at all. Sensor density decreases in proportion to
rlm@572	1140 distance from the fovea.
rlm@572	1141
rlm@572	1142 I want to be able to model any retinal configuration, so my
rlm@572	1143 eye-nodes in Blender contain metadata pointing to images that
rlm@572	1144 describe the precise position of the individual sensors using
rlm@572	1145 white pixels. The meta-data also describes the precise sensitivity
rlm@572	1146 to light that the sensors described in the image have. An eye can
rlm@572	1147 contain any number of these images. For example, the metadata for
rlm@572	1148 an eye might look like this:
rlm@572	1149
rlm@572	1150 \begin{verbatim}
rlm@572	1151 {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@572	1152 \end{verbatim}
rlm@572	1153
rlm@572	1154 \begin{figure}[htb]
rlm@572	1155 \centering
rlm@572	1156 \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572	1157 \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572	1158 \end{figure}
rlm@572	1159
rlm@572	1160 Together, the number 0xFF0000 and the image above describe the
rlm@572	1161 placement of red-sensitive sensory elements.
rlm@572	1162
rlm@572	1163 Meta-data to very crudely approximate a human eye might be
rlm@572	1164 something like this:
rlm@572	1165
rlm@572	1166 \begin{verbatim}
rlm@572	1167 (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@572	1168 {0xFF0000 retinal-profile
rlm@572	1169 0x00FF00 retinal-profile
rlm@572	1170 0x0000FF retinal-profile
rlm@572	1171 0xFFFFFF retinal-profile})
rlm@572	1172 \end{verbatim}
rlm@572	1173
rlm@572	1174 The numbers that serve as keys in the map determine a sensor's
rlm@572	1175 relative sensitivity to the channels red, green, and blue. These
rlm@572	1176 sensitivity values are packed into an integer in the order
rlm@572	1177 \texttt{\|\_\|R\|G\|B\|} in 8-bit fields. The RGB values of a pixel in the
rlm@572	1178 image are added together with these sensitivities as linear
rlm@572	1179 weights. Therefore, 0xFF0000 means sensitive to red only while
rlm@572	1180 0xFFFFFF means sensitive to all colors equally (gray).
rlm@572	1181
rlm@572	1182 \begin{listing}
rlm@572	1183 \begin{verbatim}
rlm@572	1184 (defn vision-kernel
rlm@572	1185 "Returns a list of functions, each of which will return a color
rlm@572	1186 channel's worth of visual information when called inside a running
rlm@572	1187 simulation."
rlm@572	1188 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@572	1189 (let [retinal-map (retina-sensor-profile eye)
rlm@572	1190 camera (add-eye! creature eye)
rlm@572	1191 vision-image
rlm@572	1192 (atom
rlm@572	1193 (BufferedImage. (.getWidth camera)
rlm@572	1194 (.getHeight camera)
rlm@572	1195 BufferedImage/TYPE_BYTE_BINARY))
rlm@572	1196 register-eye!
rlm@572	1197 (runonce
rlm@572	1198 (fn [world]
rlm@572	1199 (add-camera!
rlm@572	1200 world camera
rlm@572	1201 (let [counter (atom 0)]
rlm@572	1202 (fn [r fb bb bi]
rlm@572	1203 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@572	1204 (reset! vision-image
rlm@572	1205 (BufferedImage! r fb bb bi))))))))]
rlm@572	1206 (vec
rlm@572	1207 (map
rlm@572	1208 (fn [[key image]]
rlm@572	1209 (let [whites (white-coordinates image)
rlm@572	1210 topology (vec (collapse whites))
rlm@572	1211 sensitivity (sensitivity-presets key key)]
rlm@572	1212 (attached-viewport.
rlm@572	1213 (fn [world]
rlm@572	1214 (register-eye! world)
rlm@572	1215 (vector
rlm@572	1216 topology
rlm@572	1217 (vec
rlm@572	1218 (for [[x y] whites]
rlm@572	1219 (pixel-sense
rlm@572	1220 sensitivity
rlm@572	1221 (.getRGB @vision-image x y))))))
rlm@572	1222 register-eye!)))
rlm@572	1223 retinal-map))))
rlm@572	1224 \end{verbatim}
rlm@572	1225 \caption{\label{vision-kernel}This is the core of vision in \texttt{CORTEX}. A given eye node is converted into a function that returns visual information from the simulation.}
rlm@572	1226 \end{listing}
rlm@572	1227
rlm@572	1228 Note that because each of the functions generated by
rlm@572	1229 \texttt{vision-kernel} shares the same \texttt{register-eye!} function, the eye
rlm@572	1230 will be registered only once the first time any of the functions
rlm@572	1231 from the list returned by \texttt{vision-kernel} is called. Each of the
rlm@572	1232 functions returned by \texttt{vision-kernel} also allows access to the
rlm@572	1233 \texttt{Viewport} through which it receives images.
rlm@572	1234
rlm@572	1235 All the hard work has been done; all that remains is to apply
rlm@572	1236 \texttt{vision-kernel} to each eye in the creature and gather the results
rlm@572	1237 into one list of functions.
rlm@572	1238
rlm@572	1239
rlm@572	1240 \begin{listing}
rlm@572	1241 \begin{verbatim}
rlm@572	1242 (defn vision!
rlm@572	1243 "Returns a list of functions, each of which returns visual sensory
rlm@572	1244 data when called inside a running simulation."
rlm@572	1245 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@572	1246 (reduce
rlm@572	1247 concat
rlm@572	1248 (for [eye (eyes creature)]
rlm@572	1249 (vision-kernel creature eye))))
rlm@572	1250 \end{verbatim}
rlm@572	1251 \caption{\label{vision}With \texttt{vision!}, \texttt{CORTEX} is already a fine simulation environment for experimenting with different types of eyes.}
rlm@572	1252 \end{listing}
rlm@572	1253
rlm@572	1254 \begin{figure}[htb]
rlm@572	1255 \centering
rlm@572	1256 \includegraphics[width=13cm]{./images/worm-vision.png}
rlm@572	1257 \caption{\label{worm-vision-test.}Simulated vision with a test creature and the human-like eye approximation. Notice how each channel of the eye responds differently to the differently colored balls.}
rlm@572	1258 \end{figure}
rlm@572	1259
rlm@572	1260 The vision code is not much more complicated than the body code,
rlm@572	1261 and enables multiple further paths for simulated vision. For
rlm@572	1262 example, it is quite easy to create bifocal vision -- you just
rlm@572	1263 make two eyes next to each other in Blender! It is also possible
rlm@572	1264 to encode vision transforms in the retinal files. For example, the
rlm@572	1265 human like retina file in figure \ref{retina} approximates a
rlm@572	1266 log-polar transform.
rlm@572	1267
rlm@572	1268 This vision code has already been absorbed by the jMonkeyEngine
rlm@572	1269 community and is now (in modified form) part of a system for
rlm@572	1270 capturing in-game video to a file.
rlm@572	1271
rlm@572	1272 \subsection{\ldots{}but hearing must be built from scratch}
rlm@572	1273 \label{sec-2-9}
rlm@572	1274
rlm@572	1275 At the end of this chapter I will have simulated ears that work the
rlm@572	1276 same way as the simulated eyes in the last chapter. I will be able to
rlm@572	1277 place any number of ear-nodes in a Blender file, and they will bind to
rlm@572	1278 the closest physical object and follow it as it moves around. Each ear
rlm@572	1279 will provide access to the sound data it picks up between every frame.
rlm@572	1280
rlm@572	1281 Hearing is one of the more difficult senses to simulate, because there
rlm@572	1282 is less support for obtaining the actual sound data that is processed
rlm@572	1283 by jMonkeyEngine3. There is no "split-screen" support for rendering
rlm@572	1284 sound from different points of view, and there is no way to directly
rlm@572	1285 access the rendered sound data.
rlm@572	1286
rlm@572	1287 \texttt{CORTEX}'s hearing is unique because it does not have any
rlm@572	1288 limitations compared to other simulation environments. As far as I
rlm@572	1289 know, there is no other system that supports multiple listeners,
rlm@572	1290 and the sound demo at the end of this chapter is the first time
rlm@572	1291 it's been done in a video game environment.
rlm@572	1292
rlm@572	1293 \subsubsection{Brief Description of jMonkeyEngine's Sound System}
rlm@572	1294 \label{sec-2-9-1}
rlm@572	1295
rlm@572	1296 jMonkeyEngine's sound system works as follows:
rlm@572	1297
rlm@572	1298 \begin{itemize}
rlm@572	1299 \item jMonkeyEngine uses the \texttt{AppSettings} for the particular
rlm@572	1300 application to determine what sort of \texttt{AudioRenderer} should be
rlm@572	1301 used.
rlm@572	1302 \item Although some support is provided for multiple AudioRenderer
rlm@572	1303 backends, jMonkeyEngine at the time of this writing will either
rlm@572	1304 pick no \texttt{AudioRenderer} at all, or the \texttt{LwjglAudioRenderer}.
rlm@572	1305 \item jMonkeyEngine tries to figure out what sort of system you're
rlm@572	1306 running and extracts the appropriate native libraries.
rlm@572	1307 \item The \texttt{LwjglAudioRenderer} uses the \href{http://lwjgl.org/}{\texttt{LWJGL}} (LightWeight Java Game
rlm@572	1308 Library) bindings to interface with a C library called \href{http://kcat.strangesoft.net/openal.html}{\texttt{OpenAL}}
rlm@572	1309 \item \texttt{OpenAL} renders the 3D sound and feeds the rendered sound
rlm@572	1310 directly to any of various sound output devices with which it
rlm@572	1311 knows how to communicate.
rlm@572	1312 \end{itemize}
rlm@572	1313
rlm@572	1314 A consequence of this is that there's no way to access the actual
rlm@572	1315 sound data produced by \texttt{OpenAL}. Even worse, \texttt{OpenAL} only supports
rlm@572	1316 one \emph{listener} (it renders sound data from only one perspective),
rlm@572	1317 which normally isn't a problem for games, but becomes a problem
rlm@572	1318 when trying to make multiple AI creatures that can each hear the
rlm@572	1319 world from a different perspective.
rlm@572	1320
rlm@572	1321 To make many AI creatures in jMonkeyEngine that can each hear the
rlm@572	1322 world from their own perspective, or to make a single creature with
rlm@572	1323 many ears, it is necessary to go all the way back to \texttt{OpenAL} and
rlm@572	1324 implement support for simulated hearing there.
rlm@572	1325
rlm@572	1326 \subsubsection{Extending \texttt{OpenAl}}
rlm@572	1327 \label{sec-2-9-2}
rlm@572	1328
rlm@572	1329 Extending \texttt{OpenAL} to support multiple listeners requires 500
rlm@572	1330 lines of \texttt{C} code and is too complicated to mention here. Instead,
rlm@572	1331 I will show a small amount of extension code and go over the high
rlm@572	1332 level strategy. Full source is of course available with the
rlm@572	1333 \texttt{CORTEX} distribution if you're interested.
rlm@572	1334
rlm@572	1335 \texttt{OpenAL} goes to great lengths to support many different systems,
rlm@572	1336 all with different sound capabilities and interfaces. It
rlm@572	1337 accomplishes this difficult task by providing code for many
rlm@572	1338 different sound backends in pseudo-objects called \emph{Devices}.
rlm@572	1339 There's a device for the Linux Open Sound System and the Advanced
rlm@572	1340 Linux Sound Architecture, there's one for Direct Sound on Windows,
rlm@572	1341 and there's even one for Solaris. \texttt{OpenAL} solves the problem of
rlm@572	1342 platform independence by providing all these Devices.
rlm@572	1343
rlm@572	1344 Wrapper libraries such as LWJGL are free to examine the system on
rlm@572	1345 which they are running and then select an appropriate device for
rlm@572	1346 that system.
rlm@572	1347
rlm@572	1348 There are also a few "special" devices that don't interface with
rlm@572	1349 any particular system. These include the Null Device, which
rlm@572	1350 doesn't do anything, and the Wave Device, which writes whatever
rlm@572	1351 sound it receives to a file, if everything has been set up
rlm@572	1352 correctly when configuring \texttt{OpenAL}.
rlm@572	1353
rlm@572	1354 Actual mixing (Doppler shift and distance.environment-based
rlm@572	1355 attenuation) of the sound data happens in the Devices, and they
rlm@572	1356 are the only point in the sound rendering process where this data
rlm@572	1357 is available.
rlm@572	1358
rlm@572	1359 Therefore, in order to support multiple listeners, and get the
rlm@572	1360 sound data in a form that the AIs can use, it is necessary to
rlm@572	1361 create a new Device which supports this feature.
rlm@572	1362
rlm@572	1363 Adding a device to OpenAL is rather tricky -- there are five
rlm@572	1364 separate files in the \texttt{OpenAL} source tree that must be modified
rlm@572	1365 to do so. I named my device the "Multiple Audio Send" Device, or
rlm@572	1366 \texttt{Send} Device for short, since it sends audio data back to the
rlm@572	1367 calling application like an Aux-Send cable on a mixing board.
rlm@572	1368
rlm@572	1369 The main idea behind the Send device is to take advantage of the
rlm@572	1370 fact that LWJGL only manages one \emph{context} when using OpenAL. A
rlm@572	1371 \emph{context} is like a container that holds samples and keeps track
rlm@572	1372 of where the listener is. In order to support multiple listeners,
rlm@572	1373 the Send device identifies the LWJGL context as the master
rlm@572	1374 context, and creates any number of slave contexts to represent
rlm@572	1375 additional listeners. Every time the device renders sound, it
rlm@572	1376 synchronizes every source from the master LWJGL context to the
rlm@572	1377 slave contexts. Then, it renders each context separately, using a
rlm@572	1378 different listener for each one. The rendered sound is made
rlm@572	1379 available via JNI to jMonkeyEngine.
rlm@572	1380
rlm@572	1381 Switching between contexts is not the normal operation of a
rlm@572	1382 Device, and one of the problems with doing so is that a Device
rlm@572	1383 normally keeps around a few pieces of state such as the
rlm@572	1384 \texttt{ClickRemoval} array above which will become corrupted if the
rlm@572	1385 contexts are not rendered in parallel. The solution is to create a
rlm@572	1386 copy of this normally global device state for each context, and
rlm@572	1387 copy it back and forth into and out of the actual device state
rlm@572	1388 whenever a context is rendered.
rlm@572	1389
rlm@572	1390 The core of the \texttt{Send} device is the \texttt{syncSources} function, which
rlm@572	1391 does the job of copying all relevant data from one context to
rlm@572	1392 another.
rlm@572	1393
rlm@572	1394 \begin{listing}
rlm@572	1395 \begin{verbatim}
rlm@572	1396 void syncSources(ALsource masterSource, ALsource slaveSource,
rlm@572	1397 ALCcontext masterCtx, ALCcontext slaveCtx){
rlm@572	1398 ALuint master = masterSource->source;
rlm@572	1399 ALuint slave = slaveSource->source;
rlm@572	1400 ALCcontext *current = alcGetCurrentContext();
rlm@572	1401
rlm@572	1402 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
rlm@572	1403 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
rlm@572	1404 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
rlm@572	1405 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
rlm@572	1406 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
rlm@572	1407 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
rlm@572	1408 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
rlm@572	1409 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
rlm@572	1410 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
rlm@572	1411 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
rlm@572	1412 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
rlm@572	1413 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
rlm@572	1414 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
rlm@572	1415
rlm@572	1416 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
rlm@572	1417 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
rlm@572	1418 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
rlm@572	1419
rlm@572	1420 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
rlm@572	1421 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
rlm@572	1422
rlm@572	1423 alcMakeContextCurrent(masterCtx);
rlm@572	1424 ALint source_type;
rlm@572	1425 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
rlm@572	1426
rlm@572	1427 // Only static sources are currently synchronized!
rlm@572	1428 if (AL_STATIC == source_type){
rlm@572	1429 ALint master_buffer;
rlm@572	1430 ALint slave_buffer;
rlm@572	1431 alGetSourcei(master, AL_BUFFER, &master_buffer);
rlm@572	1432 alcMakeContextCurrent(slaveCtx);
rlm@572	1433 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
rlm@572	1434 if (master_buffer != slave_buffer){
rlm@572	1435 alSourcei(slave, AL_BUFFER, master_buffer);
rlm@572	1436 }
rlm@572	1437 }
rlm@572	1438
rlm@572	1439 // Synchronize the state of the two sources.
rlm@572	1440 alcMakeContextCurrent(masterCtx);
rlm@572	1441 ALint masterState;
rlm@572	1442 ALint slaveState;
rlm@572	1443
rlm@572	1444 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
rlm@572	1445 alcMakeContextCurrent(slaveCtx);
rlm@572	1446 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
rlm@572	1447
rlm@572	1448 if (masterState != slaveState){
rlm@572	1449 switch (masterState){
rlm@572	1450 case AL_INITIAL : alSourceRewind(slave); break;
rlm@572	1451 case AL_PLAYING : alSourcePlay(slave); break;
rlm@572	1452 case AL_PAUSED : alSourcePause(slave); break;
rlm@572	1453 case AL_STOPPED : alSourceStop(slave); break;
rlm@572	1454 }
rlm@572	1455 }
rlm@572	1456 // Restore whatever context was previously active.
rlm@572	1457 alcMakeContextCurrent(current);
rlm@572	1458 }
rlm@572	1459 \end{verbatim}
rlm@572	1460 \caption{\label{sync-openal-sources}Program for extending \texttt{OpenAL} to support multiple listeners via context copying/switching.}
rlm@572	1461 \end{listing}
rlm@572	1462
rlm@572	1463 With this special context-switching device, and some ugly JNI
rlm@572	1464 bindings that are not worth mentioning, \texttt{CORTEX} gains the ability
rlm@572	1465 to access multiple sound streams from \texttt{OpenAL}.
rlm@572	1466
rlm@572	1467 \begin{listing}
rlm@572	1468 \begin{verbatim}
rlm@572	1469 (defn add-ear!
rlm@572	1470 "Create a Listener centered on the current position of 'ear
rlm@572	1471 which follows the closest physical node in 'creature and
rlm@572	1472 sends sound data to 'continuation."
rlm@572	1473 [#^Application world #^Node creature #^Spatial ear continuation]
rlm@572	1474 (let [target (closest-node creature ear)
rlm@572	1475 lis (Listener.)
rlm@572	1476 audio-renderer (.getAudioRenderer world)
rlm@572	1477 sp (hearing-pipeline continuation)]
rlm@572	1478 (.setLocation lis (.getWorldTranslation ear))
rlm@572	1479 (.setRotation lis (.getWorldRotation ear))
rlm@572	1480 (bind-sense target lis)
rlm@572	1481 (update-listener-velocity! target lis)
rlm@572	1482 (.addListener audio-renderer lis)
rlm@572	1483 (.registerSoundProcessor audio-renderer lis sp)))
rlm@572	1484 \end{verbatim}
rlm@572	1485 \caption{\label{add-ear}Program to create an ear from a Blender empty node. The ear follows around the nearest physical object and passes all sensory data to a continuation function.}
rlm@572	1486 \end{listing}
rlm@572	1487
rlm@572	1488 The \texttt{Send} device, unlike most of the other devices in \texttt{OpenAL},
rlm@572	1489 does not render sound unless asked. This enables the system to
rlm@572	1490 slow down or speed up depending on the needs of the AIs who are
rlm@572	1491 using it to listen. If the device tried to render samples in
rlm@572	1492 real-time, a complicated AI whose mind takes 100 seconds of
rlm@572	1493 computer time to simulate 1 second of AI-time would miss almost
rlm@572	1494 all of the sound in its environment!
rlm@572	1495
rlm@572	1496 \begin{listing}
rlm@572	1497 \begin{verbatim}
rlm@572	1498 (defn hearing-kernel
rlm@572	1499 "Returns a function which returns auditory sensory data when called
rlm@572	1500 inside a running simulation."
rlm@572	1501 [#^Node creature #^Spatial ear]
rlm@572	1502 (let [hearing-data (atom [])
rlm@572	1503 register-listener!
rlm@572	1504 (runonce
rlm@572	1505 (fn [#^Application world]
rlm@572	1506 (add-ear!
rlm@572	1507 world creature ear
rlm@572	1508 (comp #(reset! hearing-data %)
rlm@572	1509 byteBuffer->pulse-vector))))]
rlm@572	1510 (fn [#^Application world]
rlm@572	1511 (register-listener! world)
rlm@572	1512 (let [data @hearing-data
rlm@572	1513 topology
rlm@572	1514 (vec (map #(vector % 0) (range 0 (count data))))]
rlm@572	1515 [topology data]))))
rlm@572	1516
rlm@572	1517 (defn hearing!
rlm@572	1518 "Endow the creature in a particular world with the sense of
rlm@572	1519 hearing. Will return a sequence of functions, one for each ear,
rlm@572	1520 which when called will return the auditory data from that ear."
rlm@572	1521 [#^Node creature]
rlm@572	1522 (for [ear (ears creature)]
rlm@572	1523 (hearing-kernel creature ear)))
rlm@572	1524 \end{verbatim}
rlm@572	1525 \caption{\label{hearing}Program to enable arbitrary hearing in \texttt{CORTEX}}
rlm@572	1526 \end{listing}
rlm@572	1527
rlm@572	1528 Armed with these functions, \texttt{CORTEX} is able to test possibly the
rlm@572	1529 first ever instance of multiple listeners in a video game engine
rlm@572	1530 based simulation!
rlm@572	1531
rlm@572	1532 \begin{listing}
rlm@572	1533 \begin{verbatim}
rlm@572	1534 /**
rlm@572	1535 * Respond to sound! This is the brain of an AI entity that
rlm@572	1536 * hears its surroundings and reacts to them.
rlm@572	1537 */
rlm@572	1538 public void process(ByteBuffer audioSamples,
rlm@572	1539 int numSamples, AudioFormat format) {
rlm@572	1540 audioSamples.clear();
rlm@572	1541 byte[] data = new byte[numSamples];
rlm@572	1542 float[] out = new float[numSamples];
rlm@572	1543 audioSamples.get(data);
rlm@572	1544 FloatSampleTools.
rlm@572	1545 byte2floatInterleaved
rlm@572	1546 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
rlm@572	1547
rlm@572	1548 float max = Float.NEGATIVE_INFINITY;
rlm@572	1549 for (float f : out){if (f > max) max = f;}
rlm@572	1550 audioSamples.clear();
rlm@572	1551
rlm@572	1552 if (max > 0.1){
rlm@572	1553 entity.getMaterial().setColor("Color", ColorRGBA.Green);
rlm@572	1554 }
rlm@572	1555 else {
rlm@572	1556 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
rlm@572	1557 }
rlm@572	1558 \end{verbatim}
rlm@572	1559 \caption{\label{sound-test}Here a simple creature responds to sound by changing its color from gray to green when the total volume goes over a threshold.}
rlm@572	1560 \end{listing}
rlm@572	1561
rlm@572	1562 \begin{figure}[htb]
rlm@572	1563 \centering
rlm@572	1564 \includegraphics[width=10cm]{./images/java-hearing-test.png}
rlm@572	1565 \caption{\label{sound-cubes.}First ever simulation of multiple listeners in \texttt{CORTEX}. Each cube is a creature which processes sound data with the \texttt{process} function from listing \ref{sound-test}. the ball is constantly emitting a pure tone of constant volume. As it approaches the cubes, they each change color in response to the sound.}
rlm@572	1566 \end{figure}
rlm@572	1567
rlm@572	1568 This system of hearing has also been co-opted by the
rlm@572	1569 jMonkeyEngine3 community and is used to record audio for demo
rlm@572	1570 videos.
rlm@572	1571
rlm@572	1572 \subsection{Hundreds of hair-like elements provide a sense of touch}
rlm@572	1573 \label{sec-2-10}
rlm@572	1574
rlm@572	1575 Touch is critical to navigation and spatial reasoning and as such I
rlm@572	1576 need a simulated version of it to give to my AI creatures.
rlm@572	1577
rlm@572	1578 Human skin has a wide array of touch sensors, each of which
rlm@572	1579 specialize in detecting different vibrational modes and pressures.
rlm@572	1580 These sensors can integrate a vast expanse of skin (i.e. your
rlm@572	1581 entire palm), or a tiny patch of skin at the tip of your finger.
rlm@572	1582 The hairs of the skin help detect objects before they even come
rlm@572	1583 into contact with the skin proper.
rlm@572	1584
rlm@572	1585 However, touch in my simulated world can not exactly correspond to
rlm@572	1586 human touch because my creatures are made out of completely rigid
rlm@572	1587 segments that don't deform like human skin.
rlm@572	1588
rlm@572	1589 Instead of measuring deformation or vibration, I surround each
rlm@572	1590 rigid part with a plenitude of hair-like objects (\emph{feelers}) which
rlm@572	1591 do not interact with the physical world. Physical objects can pass
rlm@572	1592 through them with no effect. The feelers are able to tell when
rlm@572	1593 other objects pass through them, and they constantly report how
rlm@572	1594 much of their extent is covered. So even though the creature's body
rlm@572	1595 parts do not deform, the feelers create a margin around those body
rlm@572	1596 parts which achieves a sense of touch which is a hybrid between a
rlm@572	1597 human's sense of deformation and sense from hairs.
rlm@572	1598
rlm@572	1599 Implementing touch in jMonkeyEngine follows a different technical
rlm@572	1600 route than vision and hearing. Those two senses piggybacked off
rlm@572	1601 jMonkeyEngine's 3D audio and video rendering subsystems. To
rlm@572	1602 simulate touch, I use jMonkeyEngine's physics system to execute
rlm@572	1603 many small collision detections, one for each feeler. The placement
rlm@572	1604 of the feelers is determined by a UV-mapped image which shows where
rlm@572	1605 each feeler should be on the 3D surface of the body.
rlm@572	1606
rlm@572	1607 \subsubsection{Defining Touch Meta-Data in Blender}
rlm@572	1608 \label{sec-2-10-1}
rlm@572	1609
rlm@572	1610 Each geometry can have a single UV map which describes the
rlm@572	1611 position of the feelers which will constitute its sense of touch.
rlm@572	1612 This image path is stored under the ``touch'' key. The image itself
rlm@572	1613 is black and white, with black meaning a feeler length of 0 (no
rlm@572	1614 feeler is present) and white meaning a feeler length of \texttt{scale},
rlm@572	1615 which is a float stored under the key "scale".
rlm@572	1616
rlm@572	1617 \begin{listing}
rlm@572	1618 \begin{verbatim}
rlm@572	1619 (defn tactile-sensor-profile
rlm@572	1620 "Return the touch-sensor distribution image in BufferedImage format,
rlm@572	1621 or nil if it does not exist."
rlm@572	1622 [#^Geometry obj]
rlm@572	1623 (if-let [image-path (meta-data obj "touch")]
rlm@572	1624 (load-image image-path)))
rlm@572	1625
rlm@572	1626 (defn tactile-scale
rlm@572	1627 "Return the length of each feeler. Default scale is 0.01
rlm@572	1628 jMonkeyEngine units."
rlm@572	1629 [#^Geometry obj]
rlm@572	1630 (if-let [scale (meta-data obj "scale")]
rlm@572	1631 scale 0.1))
rlm@572	1632 \end{verbatim}
rlm@572	1633 \caption{\label{touch-meta-data}Touch does not use empty nodes, to store metadata, because the metadata of each solid part of a creature's body is sufficient.}
rlm@572	1634 \end{listing}
rlm@572	1635
rlm@572	1636 Here is an example of a UV-map which specifies the position of
rlm@572	1637 touch sensors along the surface of the upper segment of a fingertip.
rlm@572	1638
rlm@572	1639 \begin{figure}[htb]
rlm@572	1640 \centering
rlm@572	1641 \includegraphics[width=13cm]{./images/finger-UV.png}
rlm@572	1642 \caption{\label{fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572	1643 \end{figure}
rlm@572	1644
rlm@572	1645 \subsubsection{Implementation Summary}
rlm@572	1646 \label{sec-2-10-2}
rlm@572	1647
rlm@572	1648 To simulate touch there are three conceptual steps. For each solid
rlm@572	1649 object in the creature, you first have to get UV image and scale
rlm@572	1650 parameter which define the position and length of the feelers.
rlm@572	1651 Then, you use the triangles which comprise the mesh and the UV
rlm@572	1652 data stored in the mesh to determine the world-space position and
rlm@572	1653 orientation of each feeler. Then once every frame, update these
rlm@572	1654 positions and orientations to match the current position and
rlm@572	1655 orientation of the object, and use physics collision detection to
rlm@572	1656 gather tactile data.
rlm@572	1657
rlm@572	1658 Extracting the meta-data has already been described. The third
rlm@572	1659 step, physics collision detection, is handled in \texttt{touch-kernel}.
rlm@572	1660 Translating the positions and orientations of the feelers from the
rlm@572	1661 UV-map to world-space is itself a three-step process.
rlm@572	1662
rlm@572	1663 \begin{itemize}
rlm@572	1664 \item Find the triangles which make up the mesh in pixel-space and in
rlm@572	1665 world-space. $\backslash$(\texttt{triangles}, \texttt{pixel-triangles}).
rlm@572	1666
rlm@572	1667 \item Find the coordinates of each feeler in world-space. These are
rlm@572	1668 the origins of the feelers. (\texttt{feeler-origins}).
rlm@572	1669
rlm@572	1670 \item Calculate the normals of the triangles in world space, and add
rlm@572	1671 them to each of the origins of the feelers. These are the
rlm@572	1672 normalized coordinates of the tips of the feelers.
rlm@572	1673 (\texttt{feeler-tips}).
rlm@572	1674 \end{itemize}
rlm@572	1675
rlm@572	1676 \subsubsection{Triangle Math}
rlm@572	1677 \label{sec-2-10-3}
rlm@572	1678
rlm@572	1679 The rigid objects which make up a creature have an underlying
rlm@572	1680 \texttt{Geometry}, which is a \texttt{Mesh} plus a \texttt{Material} and other
rlm@572	1681 important data involved with displaying the object.
rlm@572	1682
rlm@572	1683 A \texttt{Mesh} is composed of \texttt{Triangles}, and each \texttt{Triangle} has three
rlm@572	1684 vertices which have coordinates in world space and UV space.
rlm@572	1685
rlm@572	1686 Here, \texttt{triangles} gets all the world-space triangles which
rlm@572	1687 comprise a mesh, while \texttt{pixel-triangles} gets those same triangles
rlm@572	1688 expressed in pixel coordinates (which are UV coordinates scaled to
rlm@572	1689 fit the height and width of the UV image).
rlm@572	1690
rlm@572	1691 \begin{listing}
rlm@572	1692 \begin{verbatim}
rlm@572	1693 (defn triangle
rlm@572	1694 "Get the triangle specified by triangle-index from the mesh."
rlm@572	1695 [#^Geometry geo triangle-index]
rlm@572	1696 (triangle-seq
rlm@572	1697 (let [scratch (Triangle.)]
rlm@572	1698 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
rlm@572	1699
rlm@572	1700 (defn triangles
rlm@572	1701 "Return a sequence of all the Triangles which comprise a given
rlm@572	1702 Geometry."
rlm@572	1703 [#^Geometry geo]
rlm@572	1704 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
rlm@572	1705
rlm@572	1706 (defn triangle-vertex-indices
rlm@572	1707 "Get the triangle vertex indices of a given triangle from a given
rlm@572	1708 mesh."
rlm@572	1709 [#^Mesh mesh triangle-index]
rlm@572	1710 (let [indices (int-array 3)]
rlm@572	1711 (.getTriangle mesh triangle-index indices)
rlm@572	1712 (vec indices)))
rlm@572	1713
rlm@572	1714 (defn vertex-UV-coord
rlm@572	1715 "Get the UV-coordinates of the vertex named by vertex-index"
rlm@572	1716 [#^Mesh mesh vertex-index]
rlm@572	1717 (let [UV-buffer
rlm@572	1718 (.getData
rlm@572	1719 (.getBuffer
rlm@572	1720 mesh
rlm@572	1721 VertexBuffer$Type/TexCoord))]
rlm@572	1722 [(.get UV-buffer (* vertex-index 2))
rlm@572	1723 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
rlm@572	1724
rlm@572	1725 (defn pixel-triangle [#^Geometry geo image index]
rlm@572	1726 (let [mesh (.getMesh geo)
rlm@572	1727 width (.getWidth image)
rlm@572	1728 height (.getHeight image)]
rlm@572	1729 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
rlm@572	1730 (map (partial vertex-UV-coord mesh)
rlm@572	1731 (triangle-vertex-indices mesh index))))))
rlm@572	1732
rlm@572	1733 (defn pixel-triangles
rlm@572	1734 "The pixel-space triangles of the Geometry, in the same order as
rlm@572	1735 (triangles geo)"
rlm@572	1736 [#^Geometry geo image]
rlm@572	1737 (let [height (.getHeight image)
rlm@572	1738 width (.getWidth image)]
rlm@572	1739 (map (partial pixel-triangle geo image)
rlm@572	1740 (range (.getTriangleCount (.getMesh geo))))))
rlm@572	1741 \end{verbatim}
rlm@572	1742 \caption{\label{get-triangles}Programs to extract triangles from a geometry and get their vertices in both world and UV-coordinates.}
rlm@572	1743 \end{listing}
rlm@572	1744
rlm@572	1745 \subsubsection{The Affine Transform from one Triangle to Another}
rlm@572	1746 \label{sec-2-10-4}
rlm@572	1747
rlm@572	1748 \texttt{pixel-triangles} gives us the mesh triangles expressed in pixel
rlm@572	1749 coordinates and \texttt{triangles} gives us the mesh triangles expressed
rlm@572	1750 in world coordinates. The tactile-sensor-profile gives the
rlm@572	1751 position of each feeler in pixel-space. In order to convert
rlm@572	1752 pixel-space coordinates into world-space coordinates we need
rlm@572	1753 something that takes coordinates on the surface of one triangle
rlm@572	1754 and gives the corresponding coordinates on the surface of another
rlm@572	1755 triangle.
rlm@572	1756
rlm@572	1757 Triangles are \href{http://mathworld.wolfram.com/AffineTransformation.html }{affine}, which means any triangle can be transformed
rlm@572	1758 into any other by a combination of translation, scaling, and
rlm@572	1759 rotation. The affine transformation from one triangle to another
rlm@572	1760 is readily computable if the triangle is expressed in terms of a
rlm@572	1761 $4x4$ matrix.
rlm@572	1762
rlm@572	1763 $$
rlm@572	1764 \begin{bmatrix}
rlm@572	1765 x_1 & x_2 & x_3 & n_x \\
rlm@572	1766 y_1 & y_2 & y_3 & n_y \\
rlm@572	1767 z_1 & z_2 & z_3 & n_z \\
rlm@572	1768 1 & 1 & 1 & 1
rlm@572	1769 \end{bmatrix}
rlm@572	1770 $$
rlm@572	1771
rlm@572	1772 Here, the first three columns of the matrix are the vertices of
rlm@572	1773 the triangle. The last column is the right-handed unit normal of
rlm@572	1774 the triangle.
rlm@572	1775
rlm@572	1776 With two triangles $T_{1}$ and $T_{2}$ each expressed as a
rlm@572	1777 matrix like above, the affine transform from $T_{1}$ to $T_{2}$
rlm@572	1778 is $T_{2}T_{1}^{-1}$.
rlm@572	1779
rlm@572	1780 The clojure code below recapitulates the formulas above, using
rlm@572	1781 jMonkeyEngine's \texttt{Matrix4f} objects, which can describe any affine
rlm@572	1782 transformation.
rlm@572	1783
rlm@572	1784 \begin{listing}
rlm@572	1785 \begin{verbatim}
rlm@572	1786 (defn triangle->matrix4f
rlm@572	1787 "Converts the triangle into a 4x4 matrix: The first three columns
rlm@572	1788 contain the vertices of the triangle; the last contains the unit
rlm@572	1789 normal of the triangle. The bottom row is filled with 1s."
rlm@572	1790 [#^Triangle t]
rlm@572	1791 (let [mat (Matrix4f.)
rlm@572	1792 [vert-1 vert-2 vert-3]
rlm@572	1793 (mapv #(.get t %) (range 3))
rlm@572	1794 unit-normal (do (.calculateNormal t)(.getNormal t))
rlm@572	1795 vertices [vert-1 vert-2 vert-3 unit-normal]]
rlm@572	1796 (dorun
rlm@572	1797 (for [row (range 4) col (range 3)]
rlm@572	1798 (do
rlm@572	1799 (.set mat col row (.get (vertices row) col))
rlm@572	1800 (.set mat 3 row 1)))) mat))
rlm@572	1801
rlm@572	1802 (defn triangles->affine-transform
rlm@572	1803 "Returns the affine transformation that converts each vertex in the
rlm@572	1804 first triangle into the corresponding vertex in the second
rlm@572	1805 triangle."
rlm@572	1806 [#^Triangle tri-1 #^Triangle tri-2]
rlm@572	1807 (.mult
rlm@572	1808 (triangle->matrix4f tri-2)
rlm@572	1809 (.invert (triangle->matrix4f tri-1))))
rlm@572	1810 \end{verbatim}
rlm@572	1811 \caption{\label{triangle-affine}Program to interpret triangles as affine transforms.}
rlm@572	1812 \end{listing}
rlm@572	1813
rlm@572	1814 \subsubsection{Triangle Boundaries}
rlm@572	1815 \label{sec-2-10-5}
rlm@572	1816
rlm@572	1817 For efficiency's sake I will divide the tactile-profile image into
rlm@572	1818 small squares which inscribe each pixel-triangle, then extract the
rlm@572	1819 points which lie inside the triangle and map them to 3D-space using
rlm@572	1820 \texttt{triangle-transform} above. To do this I need a function,
rlm@572	1821 \texttt{convex-bounds} which finds the smallest box which inscribes a 2D
rlm@572	1822 triangle.
rlm@572	1823
rlm@572	1824 \texttt{inside-triangle?} determines whether a point is inside a triangle
rlm@572	1825 in 2D pixel-space.
rlm@572	1826
rlm@572	1827 \begin{listing}
rlm@572	1828 \begin{verbatim}
rlm@572	1829 (defn convex-bounds
rlm@572	1830 "Returns the smallest square containing the given vertices, as a
rlm@572	1831 vector of integers [left top width height]."
rlm@572	1832 [verts]
rlm@572	1833 (let [xs (map first verts)
rlm@572	1834 ys (map second verts)
rlm@572	1835 x0 (Math/floor (apply min xs))
rlm@572	1836 y0 (Math/floor (apply min ys))
rlm@572	1837 x1 (Math/ceil (apply max xs))
rlm@572	1838 y1 (Math/ceil (apply max ys))]
rlm@572	1839 [x0 y0 (- x1 x0) (- y1 y0)]))
rlm@572	1840
rlm@572	1841 (defn same-side?
rlm@572	1842 "Given the points p1 and p2 and the reference point ref, is point p
rlm@572	1843 on the same side of the line that goes through p1 and p2 as ref is?"
rlm@572	1844 [p1 p2 ref p]
rlm@572	1845 (<=
rlm@572	1846 0
rlm@572	1847 (.dot
rlm@572	1848 (.cross (.subtract p2 p1) (.subtract p p1))
rlm@572	1849 (.cross (.subtract p2 p1) (.subtract ref p1)))))
rlm@572	1850
rlm@572	1851 (defn inside-triangle?
rlm@572	1852 "Is the point inside the triangle?"
rlm@572	1853 {:author "Dylan Holmes"}
rlm@572	1854 [#^Triangle tri #^Vector3f p]
rlm@572	1855 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
rlm@572	1856 (and
rlm@572	1857 (same-side? vert-1 vert-2 vert-3 p)
rlm@572	1858 (same-side? vert-2 vert-3 vert-1 p)
rlm@572	1859 (same-side? vert-3 vert-1 vert-2 p))))
rlm@572	1860 \end{verbatim}
rlm@572	1861 \caption{\label{in-triangle}Program to efficiently determine point inclusion in a triangle.}
rlm@572	1862 \end{listing}
rlm@572	1863
rlm@572	1864 \subsubsection{Feeler Coordinates}
rlm@572	1865 \label{sec-2-10-6}
rlm@572	1866
rlm@572	1867 The triangle-related functions above make short work of
rlm@572	1868 calculating the positions and orientations of each feeler in
rlm@572	1869 world-space.
rlm@572	1870
rlm@572	1871 \begin{listing}
rlm@572	1872 \begin{verbatim}
rlm@572	1873 (defn feeler-pixel-coords
rlm@572	1874 "Returns the coordinates of the feelers in pixel space in lists, one
rlm@572	1875 list for each triangle, ordered in the same way as (triangles) and
rlm@572	1876 (pixel-triangles)."
rlm@572	1877 [#^Geometry geo image]
rlm@572	1878 (map
rlm@572	1879 (fn [pixel-triangle]
rlm@572	1880 (filter
rlm@572	1881 (fn [coord]
rlm@572	1882 (inside-triangle? (->triangle pixel-triangle)
rlm@572	1883 (->vector3f coord)))
rlm@572	1884 (white-coordinates image (convex-bounds pixel-triangle))))
rlm@572	1885 (pixel-triangles geo image)))
rlm@572	1886
rlm@572	1887 (defn feeler-world-coords
rlm@572	1888 "Returns the coordinates of the feelers in world space in lists, one
rlm@572	1889 list for each triangle, ordered in the same way as (triangles) and
rlm@572	1890 (pixel-triangles)."
rlm@572	1891 [#^Geometry geo image]
rlm@572	1892 (let [transforms
rlm@572	1893 (map #(triangles->affine-transform
rlm@572	1894 (->triangle %1) (->triangle %2))
rlm@572	1895 (pixel-triangles geo image)
rlm@572	1896 (triangles geo))]
rlm@572	1897 (map (fn [transform coords]
rlm@572	1898 (map #(.mult transform (->vector3f %)) coords))
rlm@572	1899 transforms (feeler-pixel-coords geo image))))
rlm@572	1900 \end{verbatim}
rlm@572	1901 \caption{\label{feeler-coordinates}Program to get the coordinates of ``feelers '' in both world and UV-coordinates.}
rlm@572	1902 \end{listing}
rlm@572	1903
rlm@572	1904 \begin{listing}
rlm@572	1905 \begin{verbatim}
rlm@572	1906 (defn feeler-origins
rlm@572	1907 "The world space coordinates of the root of each feeler."
rlm@572	1908 [#^Geometry geo image]
rlm@572	1909 (reduce concat (feeler-world-coords geo image)))
rlm@572	1910
rlm@572	1911 (defn feeler-tips
rlm@572	1912 "The world space coordinates of the tip of each feeler."
rlm@572	1913 [#^Geometry geo image]
rlm@572	1914 (let [world-coords (feeler-world-coords geo image)
rlm@572	1915 normals
rlm@572	1916 (map
rlm@572	1917 (fn [triangle]
rlm@572	1918 (.calculateNormal triangle)
rlm@572	1919 (.clone (.getNormal triangle)))
rlm@572	1920 (map ->triangle (triangles geo)))]
rlm@572	1921
rlm@572	1922 (mapcat (fn [origins normal]
rlm@572	1923 (map #(.add % normal) origins))
rlm@572	1924 world-coords normals)))
rlm@572	1925
rlm@572	1926 (defn touch-topology
rlm@572	1927 [#^Geometry geo image]
rlm@572	1928 (collapse (reduce concat (feeler-pixel-coords geo image))))
rlm@572	1929 \end{verbatim}
rlm@572	1930 \caption{\label{feeler-tips}Program to get the position of the base and tip of each ``feeler''}
rlm@572	1931 \end{listing}
rlm@572	1932
rlm@572	1933 \subsubsection{Simulated Touch}
rlm@572	1934 \label{sec-2-10-7}
rlm@572	1935
rlm@572	1936 Now that the functions to construct feelers are complete,
rlm@572	1937 \texttt{touch-kernel} generates functions to be called from within a
rlm@572	1938 simulation that perform the necessary physics collisions to
rlm@572	1939 collect tactile data, and \texttt{touch!} recursively applies it to every
rlm@572	1940 node in the creature.
rlm@572	1941
rlm@572	1942 \begin{listing}
rlm@572	1943 \begin{verbatim}
rlm@572	1944 (defn set-ray [#^Ray ray #^Matrix4f transform
rlm@572	1945 #^Vector3f origin #^Vector3f tip]
rlm@572	1946 ;; Doing everything locally reduces garbage collection by enough to
rlm@572	1947 ;; be worth it.
rlm@572	1948 (.mult transform origin (.getOrigin ray))
rlm@572	1949 (.mult transform tip (.getDirection ray))
rlm@572	1950 (.subtractLocal (.getDirection ray) (.getOrigin ray))
rlm@572	1951 (.normalizeLocal (.getDirection ray)))
rlm@572	1952 \end{verbatim}
rlm@572	1953 \caption{\label{set-ray}Efficient program to transform a ray from one position to another.}
rlm@572	1954 \end{listing}
rlm@572	1955
rlm@572	1956 \begin{listing}
rlm@572	1957 \begin{verbatim}
rlm@572	1958 (defn touch-kernel
rlm@572	1959 "Constructs a function which will return tactile sensory data from
rlm@572	1960 'geo when called from inside a running simulation"
rlm@572	1961 [#^Geometry geo]
rlm@572	1962 (if-let
rlm@572	1963 [profile (tactile-sensor-profile geo)]
rlm@572	1964 (let [ray-reference-origins (feeler-origins geo profile)
rlm@572	1965 ray-reference-tips (feeler-tips geo profile)
rlm@572	1966 ray-length (tactile-scale geo)
rlm@572	1967 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
rlm@572	1968 topology (touch-topology geo profile)
rlm@572	1969 correction (float (* ray-length -0.2))]
rlm@572	1970 ;; slight tolerance for very close collisions.
rlm@572	1971 (dorun
rlm@572	1972 (map (fn [origin tip]
rlm@572	1973 (.addLocal origin (.mult (.subtract tip origin)
rlm@572	1974 correction)))
rlm@572	1975 ray-reference-origins ray-reference-tips))
rlm@572	1976 (dorun (map #(.setLimit % ray-length) current-rays))
rlm@572	1977 (fn [node]
rlm@572	1978 (let [transform (.getWorldMatrix geo)]
rlm@572	1979 (dorun
rlm@572	1980 (map (fn [ray ref-origin ref-tip]
rlm@572	1981 (set-ray ray transform ref-origin ref-tip))
rlm@572	1982 current-rays ray-reference-origins
rlm@572	1983 ray-reference-tips))
rlm@572	1984 (vector
rlm@572	1985 topology
rlm@572	1986 (vec
rlm@572	1987 (for [ray current-rays]
rlm@572	1988 (do
rlm@572	1989 (let [results (CollisionResults.)]
rlm@572	1990 (.collideWith node ray results)
rlm@572	1991 (let [touch-objects
rlm@572	1992 (filter #(not (= geo (.getGeometry %)))
rlm@572	1993 results)
rlm@572	1994 limit (.getLimit ray)]
rlm@572	1995 [(if (empty? touch-objects)
rlm@572	1996 limit
rlm@572	1997 (let [response
rlm@572	1998 (apply min (map #(.getDistance %)
rlm@572	1999 touch-objects))]
rlm@572	2000 (FastMath/clamp
rlm@572	2001 (float
rlm@572	2002 (if (> response limit) (float 0.0)
rlm@572	2003 (+ response correction)))
rlm@572	2004 (float 0.0)
rlm@572	2005 limit)))
rlm@572	2006 limit])))))))))))
rlm@572	2007 \end{verbatim}
rlm@572	2008 \caption{\label{touch-kernel}This is the core of touch in \texttt{CORTEX} each feeler follows the object it is bound to, reporting any collisions that may happen.}
rlm@572	2009 \end{listing}
rlm@572	2010
rlm@572	2011 Armed with the \texttt{touch!} function, \texttt{CORTEX} becomes capable of
rlm@572	2012 giving creatures a sense of touch. A simple test is to create a
rlm@572	2013 cube that is outfitted with a uniform distribution of touch
rlm@572	2014 sensors. It can feel the ground and any balls that it touches.
rlm@572	2015
rlm@572	2016 \begin{listing}
rlm@572	2017 \begin{verbatim}
rlm@572	2018 (defn touch!
rlm@572	2019 "Endow the creature with the sense of touch. Returns a sequence of
rlm@572	2020 functions, one for each body part with a tactile-sensor-profile,
rlm@572	2021 each of which when called returns sensory data for that body part."
rlm@572	2022 [#^Node creature]
rlm@572	2023 (filter
rlm@572	2024 (comp not nil?)
rlm@572	2025 (map touch-kernel
rlm@572	2026 (filter #(isa? (class %) Geometry)
rlm@572	2027 (node-seq creature)))))
rlm@572	2028 \end{verbatim}
rlm@572	2029 \caption{\label{touch}\texttt{CORTEX} interface for creating touch in a simulated creature.}
rlm@572	2030 \end{listing}
rlm@572	2031
rlm@572	2032 The tactile-sensor-profile image for the touch cube is a simple
rlm@572	2033 cross with a uniform distribution of touch sensors:
rlm@572	2034
rlm@572	2035 \begin{figure}[htb]
rlm@572	2036 \centering
rlm@572	2037 \includegraphics[width=7cm]{./images/touch-profile.png}
rlm@572	2038 \caption{\label{touch-cube-uv-map}The touch profile for the touch-cube. Each pure white pixel defines a touch sensitive feeler.}
rlm@572	2039 \end{figure}
rlm@572	2040
rlm@572	2041 \begin{figure}[htb]
rlm@572	2042 \centering
rlm@572	2043 \includegraphics[width=15cm]{./images/touch-cube.png}
rlm@572	2044 \caption{\label{touch-cube-uv-map-2}The touch cube reacts to cannonballs. The black, red, and white cross on the right is a visual display of the creature's touch. White means that it is feeling something strongly, black is not feeling anything, and gray is in-between. The cube can feel both the floor and the ball. Notice that when the ball causes the cube to tip, that the bottom face can still feel part of the ground.}
rlm@572	2045 \end{figure}
rlm@572	2046
rlm@572	2047 \subsection{Proprioception provides knowledge of your own body's position}
rlm@572	2048 \label{sec-2-11}
rlm@572	2049
rlm@572	2050 Close your eyes, and touch your nose with your right index finger.
rlm@572	2051 How did you do it? You could not see your hand, and neither your
rlm@572	2052 hand nor your nose could use the sense of touch to guide the path
rlm@572	2053 of your hand. There are no sound cues, and Taste and Smell
rlm@572	2054 certainly don't provide any help. You know where your hand is
rlm@572	2055 without your other senses because of Proprioception.
rlm@572	2056
rlm@572	2057 Humans can sometimes loose this sense through viral infections or
rlm@572	2058 damage to the spinal cord or brain, and when they do, they loose
rlm@572	2059 the ability to control their own bodies without looking directly at
rlm@572	2060 the parts they want to move. In \href{http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat}{The Man Who Mistook His Wife for a
rlm@572	2061 Hat} (\cite{man-wife-hat}), a woman named Christina looses this
rlm@572	2062 sense and has to learn how to move by carefully watching her arms
rlm@572	2063 and legs. She describes proprioception as the "eyes of the body,
rlm@572	2064 the way the body sees itself".
rlm@572	2065
rlm@572	2066 Proprioception in humans is mediated by \href{http://en.wikipedia.org/wiki/Articular_capsule}{joint capsules}, \href{http://en.wikipedia.org/wiki/Muscle_spindle}{muscle
rlm@572	2067 spindles}, and the \href{http://en.wikipedia.org/wiki/Golgi_tendon_organ}{Golgi tendon organs}. These measure the relative
rlm@572	2068 positions of each body part by monitoring muscle strain and length.
rlm@572	2069
rlm@572	2070 It's clear that this is a vital sense for fluid, graceful movement.
rlm@572	2071 It's also particularly easy to implement in jMonkeyEngine.
rlm@572	2072
rlm@572	2073 My simulated proprioception calculates the relative angles of each
rlm@572	2074 joint from the rest position defined in the Blender file. This
rlm@572	2075 simulates the muscle-spindles and joint capsules. I will deal with
rlm@572	2076 Golgi tendon organs, which calculate muscle strain, in the next
rlm@572	2077 section (2.12).
rlm@572	2078
rlm@572	2079 \subsubsection{Helper functions}
rlm@572	2080 \label{sec-2-11-1}
rlm@572	2081
rlm@572	2082 \texttt{absolute-angle} calculates the angle between two vectors,
rlm@572	2083 relative to a third axis vector. This angle is the number of
rlm@572	2084 radians you have to move counterclockwise around the axis vector
rlm@572	2085 to get from the first to the second vector. It is not commutative
rlm@572	2086 like a normal dot-product angle is.
rlm@572	2087
rlm@572	2088 The purpose of these functions is to build a system of angle
rlm@572	2089 measurement that is biologically plausible.
rlm@572	2090
rlm@572	2091 \begin{listing}
rlm@572	2092 \begin{verbatim}
rlm@572	2093 (defn right-handed?
rlm@572	2094 "true iff the three vectors form a right handed coordinate
rlm@572	2095 system. The three vectors do not have to be normalized or
rlm@572	2096 orthogonal."
rlm@572	2097 [vec1 vec2 vec3]
rlm@572	2098 (pos? (.dot (.cross vec1 vec2) vec3)))
rlm@572	2099
rlm@572	2100 (defn absolute-angle
rlm@572	2101 "The angle between 'vec1 and 'vec2 around 'axis. In the range
rlm@572	2102 [0 (* 2 Math/PI)]."
rlm@572	2103 [vec1 vec2 axis]
rlm@572	2104 (let [angle (.angleBetween vec1 vec2)]
rlm@572	2105 (if (right-handed? vec1 vec2 axis)
rlm@572	2106 angle (- (* 2 Math/PI) angle))))
rlm@572	2107 \end{verbatim}
rlm@572	2108 \caption{\label{helpers}Program to measure angles along a vector}
rlm@572	2109 \end{listing}
rlm@572	2110
rlm@572	2111 \subsubsection{Proprioception Kernel}
rlm@572	2112 \label{sec-2-11-2}
rlm@572	2113
rlm@572	2114 Given a joint, \texttt{proprioception-kernel} produces a function that
rlm@572	2115 calculates the Euler angles between the objects the joint
rlm@572	2116 connects. The only tricky part here is making the angles relative
rlm@572	2117 to the joint's initial ``straightness''.
rlm@572	2118
rlm@572	2119 \begin{listing}
rlm@572	2120 \begin{verbatim}
rlm@572	2121 (defn proprioception-kernel
rlm@572	2122 "Returns a function which returns proprioceptive sensory data when
rlm@572	2123 called inside a running simulation."
rlm@572	2124 [#^Node parts #^Node joint]
rlm@572	2125 (let [[obj-a obj-b] (joint-targets parts joint)
rlm@572	2126 joint-rot (.getWorldRotation joint)
rlm@572	2127 x0 (.mult joint-rot Vector3f/UNIT_X)
rlm@572	2128 y0 (.mult joint-rot Vector3f/UNIT_Y)
rlm@572	2129 z0 (.mult joint-rot Vector3f/UNIT_Z)]
rlm@572	2130 (fn []
rlm@572	2131 (let [rot-a (.clone (.getWorldRotation obj-a))
rlm@572	2132 rot-b (.clone (.getWorldRotation obj-b))
rlm@572	2133 x (.mult rot-a x0)
rlm@572	2134 y (.mult rot-a y0)
rlm@572	2135 z (.mult rot-a z0)
rlm@572	2136
rlm@572	2137 X (.mult rot-b x0)
rlm@572	2138 Y (.mult rot-b y0)
rlm@572	2139 Z (.mult rot-b z0)
rlm@572	2140 heading (Math/atan2 (.dot X z) (.dot X x))
rlm@572	2141 pitch (Math/atan2 (.dot X y) (.dot X x))
rlm@572	2142
rlm@572	2143 ;; rotate x-vector back to origin
rlm@572	2144 reverse
rlm@572	2145 (doto (Quaternion.)
rlm@572	2146 (.fromAngleAxis
rlm@572	2147 (.angleBetween X x)
rlm@572	2148 (let [cross (.normalize (.cross X x))]
rlm@572	2149 (if (= 0 (.length cross)) y cross))))
rlm@572	2150 roll (absolute-angle (.mult reverse Y) y x)]
rlm@572	2151 [heading pitch roll]))))
rlm@572	2152
rlm@572	2153 (defn proprioception!
rlm@572	2154 "Endow the creature with the sense of proprioception. Returns a
rlm@572	2155 sequence of functions, one for each child of the \"joints\" node in
rlm@572	2156 the creature, which each report proprioceptive information about
rlm@572	2157 that joint."
rlm@572	2158 [#^Node creature]
rlm@572	2159 ;; extract the body's joints
rlm@572	2160 (let [senses (map (partial proprioception-kernel creature)
rlm@572	2161 (joints creature))]
rlm@572	2162 (fn []
rlm@572	2163 (map #(%) senses))))
rlm@572	2164 \end{verbatim}
rlm@572	2165 \caption{\label{proprioception}Program to return biologically reasonable proprioceptive data for each joint.}
rlm@572	2166 \end{listing}
rlm@572	2167
rlm@572	2168 \texttt{proprioception!} maps \texttt{proprioception-kernel} across all the
rlm@572	2169 joints of the creature. It uses the same list of joints that
rlm@572	2170 \texttt{joints} uses. Proprioception is the easiest sense to implement in
rlm@572	2171 \texttt{CORTEX}, and it will play a crucial role when efficiently
rlm@572	2172 implementing empathy.
rlm@572	2173
rlm@572	2174 \begin{figure}[htb]
rlm@572	2175 \centering
rlm@572	2176 \includegraphics[width=11cm]{./images/proprio.png}
rlm@572	2177 \caption{\label{proprio}In the upper right corner, the three proprioceptive angle measurements are displayed. Red is yaw, Green is pitch, and White is roll.}
rlm@572	2178 \end{figure}
rlm@572	2179
rlm@572	2180 \subsection{Muscles contain both sensors and effectors}
rlm@572	2181 \label{sec-2-12}
rlm@572	2182
rlm@572	2183 Surprisingly enough, terrestrial creatures only move by using
rlm@572	2184 torque applied about their joints. There's not a single straight
rlm@572	2185 line of force in the human body at all! (A straight line of force
rlm@572	2186 would correspond to some sort of jet or rocket propulsion.)
rlm@572	2187
rlm@572	2188 In humans, muscles are composed of muscle fibers which can contract
rlm@572	2189 to exert force. The muscle fibers which compose a muscle are
rlm@572	2190 partitioned into discrete groups which are each controlled by a
rlm@572	2191 single alpha motor neuron. A single alpha motor neuron might
rlm@572	2192 control as little as three or as many as one thousand muscle
rlm@572	2193 fibers. When the alpha motor neuron is engaged by the spinal cord,
rlm@572	2194 it activates all of the muscle fibers to which it is attached. The
rlm@572	2195 spinal cord generally engages the alpha motor neurons which control
rlm@572	2196 few muscle fibers before the motor neurons which control many
rlm@572	2197 muscle fibers. This recruitment strategy allows for precise
rlm@572	2198 movements at low strength. The collection of all motor neurons that
rlm@572	2199 control a muscle is called the motor pool. The brain essentially
rlm@572	2200 says "activate 30\% of the motor pool" and the spinal cord recruits
rlm@572	2201 motor neurons until 30\% are activated. Since the distribution of
rlm@572	2202 power among motor neurons is unequal and recruitment goes from
rlm@572	2203 weakest to strongest, the first 30\% of the motor pool might be 5\%
rlm@572	2204 of the strength of the muscle.
rlm@572	2205
rlm@572	2206 My simulated muscles follow a similar design: Each muscle is
rlm@572	2207 defined by a 1-D array of numbers (the "motor pool"). Each entry in
rlm@572	2208 the array represents a motor neuron which controls a number of
rlm@572	2209 muscle fibers equal to the value of the entry. Each muscle has a
rlm@572	2210 scalar strength factor which determines the total force the muscle
rlm@572	2211 can exert when all motor neurons are activated. The effector
rlm@572	2212 function for a muscle takes a number to index into the motor pool,
rlm@572	2213 and then "activates" all the motor neurons whose index is lower or
rlm@572	2214 equal to the number. Each motor-neuron will apply force in
rlm@572	2215 proportion to its value in the array. Lower values cause less
rlm@572	2216 force. The lower values can be put at the "beginning" of the 1-D
rlm@572	2217 array to simulate the layout of actual human muscles, which are
rlm@572	2218 capable of more precise movements when exerting less force. Or, the
rlm@572	2219 motor pool can simulate more exotic recruitment strategies which do
rlm@572	2220 not correspond to human muscles.
rlm@572	2221
rlm@572	2222 This 1D array is defined in an image file for ease of
rlm@572	2223 creation/visualization. Here is an example muscle profile image.
rlm@572	2224
rlm@572	2225 \begin{figure}[htb]
rlm@572	2226 \centering
rlm@572	2227 \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572	2228 \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572	2229 \end{figure}
rlm@572	2230
rlm@572	2231 \subsubsection{Muscle meta-data}
rlm@572	2232 \label{sec-2-12-1}
rlm@572	2233
rlm@572	2234 \begin{listing}
rlm@572	2235 \begin{verbatim}
rlm@572	2236 (defn muscle-profile-image
rlm@572	2237 "Get the muscle-profile image from the node's Blender meta-data."
rlm@572	2238 [#^Node muscle]
rlm@572	2239 (if-let [image (meta-data muscle "muscle")]
rlm@572	2240 (load-image image)))
rlm@572	2241
rlm@572	2242 (defn muscle-strength
rlm@572	2243 "Return the strength of this muscle, or 1 if it is not defined."
rlm@572	2244 [#^Node muscle]
rlm@572	2245 (if-let [strength (meta-data muscle "strength")]
rlm@572	2246 strength 1))
rlm@572	2247
rlm@572	2248 (defn motor-pool
rlm@572	2249 "Return a vector where each entry is the strength of the \"motor
rlm@572	2250 neuron\" at that part in the muscle."
rlm@572	2251 [#^Node muscle]
rlm@572	2252 (let [profile (muscle-profile-image muscle)]
rlm@572	2253 (vec
rlm@572	2254 (let [width (.getWidth profile)]
rlm@572	2255 (for [x (range width)]
rlm@572	2256 (- 255
rlm@572	2257 (bit-and
rlm@572	2258 0x0000FF
rlm@572	2259 (.getRGB profile x 0))))))))
rlm@572	2260 \end{verbatim}
rlm@572	2261 \caption{\label{motor-pool}Program to deal with loading muscle data from a Blender file's metadata.}
rlm@572	2262 \end{listing}
rlm@572	2263
rlm@572	2264 Of note here is \texttt{motor-pool} which interprets the muscle-profile
rlm@572	2265 image in a way that allows me to use gradients between white and
rlm@572	2266 red, instead of shades of gray as I've been using for all the
rlm@572	2267 other senses. This is purely an aesthetic touch.
rlm@572	2268
rlm@572	2269 \subsubsection{Creating muscles}
rlm@572	2270 \label{sec-2-12-2}
rlm@572	2271
rlm@572	2272 \begin{listing}
rlm@572	2273 \begin{verbatim}
rlm@572	2274 (defn movement-kernel
rlm@572	2275 "Returns a function which when called with a integer value inside a
rlm@572	2276 running simulation will cause movement in the creature according
rlm@572	2277 to the muscle's position and strength profile. Each function
rlm@572	2278 returns the amount of force applied / max force."
rlm@572	2279 [#^Node creature #^Node muscle]
rlm@572	2280 (let [target (closest-node creature muscle)
rlm@572	2281 axis
rlm@572	2282 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
rlm@572	2283 strength (muscle-strength muscle)
rlm@572	2284
rlm@572	2285 pool (motor-pool muscle)
rlm@572	2286 pool-integral (reductions + pool)
rlm@572	2287 forces
rlm@572	2288 (vec (map #(float (* strength (/ % (last pool-integral))))
rlm@572	2289 pool-integral))
rlm@572	2290 control (.getControl target RigidBodyControl)]
rlm@572	2291 (fn [n]
rlm@572	2292 (let [pool-index (max 0 (min n (dec (count pool))))
rlm@572	2293 force (forces pool-index)]
rlm@572	2294 (.applyTorque control (.mult axis force))
rlm@572	2295 (float (/ force strength))))))
rlm@572	2296
rlm@572	2297 (defn movement!
rlm@572	2298 "Endow the creature with the power of movement. Returns a sequence
rlm@572	2299 of functions, each of which accept an integer value and will
rlm@572	2300 activate their corresponding muscle."
rlm@572	2301 [#^Node creature]
rlm@572	2302 (for [muscle (muscles creature)]
rlm@572	2303 (movement-kernel creature muscle)))
rlm@572	2304 \end{verbatim}
rlm@572	2305 \caption{\label{muscle-kernel}This is the core movement function in \texttt{CORTEX}, which implements muscles that report on their activation.}
rlm@572	2306 \end{listing}
rlm@572	2307
rlm@572	2308
rlm@572	2309 \texttt{movement-kernel} creates a function that controls the movement
rlm@572	2310 of the nearest physical node to the muscle node. The muscle exerts
rlm@572	2311 a rotational force dependent on it's orientation to the object in
rlm@572	2312 the Blender file. The function returned by \texttt{movement-kernel} is
rlm@572	2313 also a sense function: it returns the percent of the total muscle
rlm@572	2314 strength that is currently being employed. This is analogous to
rlm@572	2315 muscle tension in humans and completes the sense of proprioception
rlm@572	2316 begun in the last chapter.
rlm@572	2317
rlm@572	2318 \subsection{\texttt{CORTEX} brings complex creatures to life!}
rlm@572	2319 \label{sec-2-13}
rlm@572	2320
rlm@572	2321 The ultimate test of \texttt{CORTEX} is to create a creature with the full
rlm@572	2322 gamut of senses and put it though its paces.
rlm@572	2323
rlm@572	2324 With all senses enabled, my right hand model looks like an
rlm@572	2325 intricate marionette hand with several strings for each finger:
rlm@572	2326
rlm@572	2327 \begin{figure}[htb]
rlm@572	2328 \centering
rlm@572	2329 \includegraphics[width=11cm]{./images/hand-with-all-senses2.png}
rlm@572	2330 \caption{\label{hand-nodes-1}View of the hand model with all sense nodes. You can see the joint, muscle, ear, and eye nodes here.}
rlm@572	2331 \end{figure}
rlm@572	2332
rlm@572	2333 \begin{figure}[htb]
rlm@572	2334 \centering
rlm@572	2335 \includegraphics[width=15cm]{./images/hand-with-all-senses3.png}
rlm@572	2336 \caption{\label{hand-nodes-2}An alternate view of the hand.}
rlm@572	2337 \end{figure}
rlm@572	2338
rlm@572	2339 With the hand fully rigged with senses, I can run it though a test
rlm@572	2340 that will test everything.
rlm@572	2341
rlm@572	2342 \begin{figure}[htb]
rlm@572	2343 \centering
rlm@572	2344 \includegraphics[width=15cm]{./images/integration.png}
rlm@572	2345 \caption{\label{integration}Selected frames from a full test of the hand with all senses. Note especially the interactions the hand has with itself: it feels its own palm and fingers, and when it curls its fingers, it sees them with its eye (which is located in the center of the palm. The red block appears with a pure tone sound. The hand then uses its muscles to launch the cube!}
rlm@572	2346 \end{figure}
rlm@572	2347
rlm@572	2348 \subsection{\texttt{CORTEX} enables many possibilities for further research}
rlm@572	2349 \label{sec-2-14}
rlm@572	2350
rlm@572	2351 Often times, the hardest part of building a system involving
rlm@572	2352 creatures is dealing with physics and graphics. \texttt{CORTEX} removes
rlm@572	2353 much of this initial difficulty and leaves researchers free to
rlm@572	2354 directly pursue their ideas. I hope that even novices with a
rlm@572	2355 passing curiosity about simulated touch or creature evolution will
rlm@572	2356 be able to use cortex for experimentation. \texttt{CORTEX} is a completely
rlm@572	2357 simulated world, and far from being a disadvantage, its simulated
rlm@572	2358 nature enables you to create senses and creatures that would be
rlm@572	2359 impossible to make in the real world.
rlm@572	2360
rlm@572	2361 While not by any means a complete list, here are some paths
rlm@572	2362 \texttt{CORTEX} is well suited to help you explore:
rlm@572	2363
rlm@572	2364 \begin{description}
rlm@572	2365 \item[{Empathy }] my empathy program leaves many areas for
rlm@572	2366 improvement, among which are using vision to infer
rlm@572	2367 proprioception and looking up sensory experience with imagined
rlm@572	2368 vision, touch, and sound.
rlm@572	2369 \item[{Evolution}] Karl Sims created a rich environment for simulating
rlm@572	2370 the evolution of creatures on a Connection Machine
rlm@572	2371 (\cite{sims-evolving-creatures}). Today, this can be redone
rlm@572	2372 and expanded with \texttt{CORTEX} on an ordinary computer.
rlm@572	2373 \item[{Exotic senses }] Cortex enables many fascinating senses that are
rlm@572	2374 not possible to build in the real world. For example,
rlm@572	2375 telekinesis is an interesting avenue to explore. You can also
rlm@572	2376 make a ``semantic'' sense which looks up metadata tags on
rlm@572	2377 objects in the environment the metadata tags might contain
rlm@572	2378 other sensory information.
rlm@572	2379 \item[{Imagination via subworlds}] this would involve a creature with
rlm@572	2380 an effector which creates an entire new sub-simulation where
rlm@572	2381 the creature has direct control over placement/creation of
rlm@572	2382 objects via simulated telekinesis. The creature observes this
rlm@572	2383 sub-world through its normal senses and uses its observations
rlm@572	2384 to make predictions about its top level world.
rlm@572	2385 \item[{Simulated prescience}] step the simulation forward a few ticks,
rlm@572	2386 gather sensory data, then supply this data for the creature as
rlm@572	2387 one of its actual senses. The cost of prescience is slowing
rlm@572	2388 the simulation down by a factor proportional to however far
rlm@572	2389 you want the entities to see into the future. What happens
rlm@572	2390 when two evolved creatures that can each see into the future
rlm@572	2391 fight each other?
rlm@572	2392 \item[{Swarm creatures}] Program a group of creatures that cooperate
rlm@572	2393 with each other. Because the creatures would be simulated, you
rlm@572	2394 could investigate computationally complex rules of behavior
rlm@572	2395 which still, from the group's point of view, would happen in
rlm@572	2396 real time. Interactions could be as simple as cellular
rlm@572	2397 organisms communicating via flashing lights, or as complex as
rlm@572	2398 humanoids completing social tasks, etc.
rlm@572	2399 \item[{\texttt{HACKER} for writing muscle-control programs}] Presented with a
rlm@572	2400 low-level muscle control / sense API, generate higher level
rlm@572	2401 programs for accomplishing various stated goals. Example goals
rlm@572	2402 might be "extend all your fingers" or "move your hand into the
rlm@572	2403 area with blue light" or "decrease the angle of this joint".
rlm@572	2404 It would be like Sussman's HACKER, except it would operate
rlm@572	2405 with much more data in a more realistic world. Start off with
rlm@572	2406 "calisthenics" to develop subroutines over the motor control
rlm@572	2407 API. The low level programming code might be a turning machine
rlm@572	2408 that could develop programs to iterate over a "tape" where
rlm@572	2409 each entry in the tape could control recruitment of the fibers
rlm@572	2410 in a muscle.
rlm@572	2411 \item[{Sense fusion}] There is much work to be done on sense
rlm@572	2412 integration -- building up a coherent picture of the world and
rlm@572	2413 the things in it. With \texttt{CORTEX} as a base, you can explore
rlm@572	2414 concepts like self-organizing maps or cross modal clustering
rlm@572	2415 in ways that have never before been tried.
rlm@572	2416 \item[{Inverse kinematics}] experiments in sense guided motor control
rlm@572	2417 are easy given \texttt{CORTEX}'s support -- you can get right to the
rlm@572	2418 hard control problems without worrying about physics or
rlm@572	2419 senses.
rlm@572	2420 \end{description}
rlm@572	2421
rlm@572	2422 \newpage
rlm@572	2423
rlm@572	2424 \section{\texttt{EMPATH}: action recognition in a simulated worm}
rlm@572	2425 \label{sec-3}
rlm@572	2426
rlm@572	2427 Here I develop a computational model of empathy, using \texttt{CORTEX} as a
rlm@572	2428 base. Empathy in this context is the ability to observe another
rlm@572	2429 creature and infer what sorts of sensations that creature is
rlm@572	2430 feeling. My empathy algorithm involves multiple phases. First is
rlm@572	2431 free-play, where the creature moves around and gains sensory
rlm@572	2432 experience. From this experience I construct a representation of the
rlm@572	2433 creature's sensory state space, which I call $\Phi$-space. Using
rlm@572	2434 $\Phi$-space, I construct an efficient function which takes the
rlm@572	2435 limited data that comes from observing another creature and enriches
rlm@572	2436 it with a full compliment of imagined sensory data. I can then use
rlm@572	2437 the imagined sensory data to recognize what the observed creature is
rlm@572	2438 doing and feeling, using straightforward embodied action predicates.
rlm@572	2439 This is all demonstrated with using a simple worm-like creature, and
rlm@572	2440 recognizing worm-actions based on limited data.
rlm@572	2441
rlm@572	2442 \begin{figure}[htb]
rlm@572	2443 \centering
rlm@572	2444 \includegraphics[width=10cm]{./images/basic-worm-view.png}
rlm@572	2445 \caption{\label{basic-worm-view}Here is the worm with which we will be working. It is composed of 5 segments. Each segment has a pair of extensor and flexor muscles. Each of the worm's four joints is a hinge joint which allows about 30 degrees of rotation to either side. Each segment of the worm is touch-capable and has a uniform distribution of touch sensors on each of its faces. Each joint has a proprioceptive sense to detect relative positions. The worm segments are all the same except for the first one, which has a much higher weight than the others to allow for easy manual motor control.}
rlm@572	2446 \end{figure}
rlm@572	2447
rlm@572	2448 \begin{listing}
rlm@572	2449 \begin{verbatim}
rlm@572	2450 (defn worm []
rlm@572	2451 (let [model (load-blender-model "Models/worm/worm.blend")]
rlm@572	2452 {:body (doto model (body!))
rlm@572	2453 :touch (touch! model)
rlm@572	2454 :proprioception (proprioception! model)
rlm@572	2455 :muscles (movement! model)}))
rlm@572	2456 \end{verbatim}
rlm@572	2457 \caption{\label{get-worm}Program for reading a worm from a Blender file and outfitting it with the senses of proprioception, touch, and the ability to move, as specified in the Blender file.}
rlm@572	2458 \end{listing}
rlm@572	2459
rlm@572	2460 \subsection{Embodiment factors action recognition into manageable parts}
rlm@572	2461 \label{sec-3-1}
rlm@572	2462
rlm@572	2463 Using empathy, I divide the problem of action recognition into a
rlm@572	2464 recognition process expressed in the language of a full compliment
rlm@572	2465 of senses, and an imaginative process that generates full sensory
rlm@572	2466 data from partial sensory data. Splitting the action recognition
rlm@572	2467 problem in this manner greatly reduces the total amount of work to
rlm@572	2468 recognize actions: The imaginative process is mostly just matching
rlm@572	2469 previous experience, and the recognition process gets to use all
rlm@572	2470 the senses to directly describe any action.
rlm@572	2471
rlm@572	2472 \subsection{Action recognition is easy with a full gamut of senses}
rlm@572	2473 \label{sec-3-2}
rlm@572	2474
rlm@572	2475 Embodied representation using multiple senses such as touch,
rlm@572	2476 proprioception, and muscle tension turns out be exceedingly
rlm@572	2477 efficient at describing body-centered actions. It is the right
rlm@572	2478 language for the job. For example, it takes only around 5 lines of
rlm@572	2479 clojure code to describe the action of curling using embodied
rlm@572	2480 primitives. It takes about 10 lines to describe the seemingly
rlm@572	2481 complicated action of wiggling.
rlm@572	2482
rlm@572	2483 The following action predicates each take a stream of sensory
rlm@572	2484 experience, observe however much of it they desire, and decide
rlm@572	2485 whether the worm is doing the action they describe. \texttt{curled?}
rlm@572	2486 relies on proprioception, \texttt{resting?} relies on touch, \texttt{wiggling?}
rlm@572	2487 relies on a Fourier analysis of muscle contraction, and
rlm@572	2488 \texttt{grand-circle?} relies on touch and reuses \texttt{curled?} in its
rlm@572	2489 definition, showing how embodied predicates can be composed.
rlm@572	2490
rlm@572	2491
rlm@572	2492 \begin{listing}
rlm@572	2493 \begin{verbatim}
rlm@572	2494 (defn curled?
rlm@572	2495 "Is the worm curled up?"
rlm@572	2496 [experiences]
rlm@572	2497 (every?
rlm@572	2498 (fn [[_ _ bend]]
rlm@572	2499 (> (Math/sin bend) 0.64))
rlm@572	2500 (:proprioception (peek experiences))))
rlm@572	2501 \end{verbatim}
rlm@572	2502 \caption{\label{curled}Program for detecting whether the worm is curled. This is the simplest action predicate, because it only uses the last frame of sensory experience, and only uses proprioceptive data. Even this simple predicate, however, is automatically frame independent and ignores vermopomorphic\protect\footnotemark \space differences such as worm textures and colors.}
rlm@572	2503 \end{listing}
rlm@572	2504
rlm@572	2505 \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
rlm@572	2506
rlm@572	2507 \begin{listing}
rlm@572	2508 \begin{verbatim}
rlm@572	2509 (defn contact
rlm@572	2510 "Determine how much contact a particular worm segment has with
rlm@572	2511 other objects. Returns a value between 0 and 1, where 1 is full
rlm@572	2512 contact and 0 is no contact."
rlm@572	2513 [touch-region [coords contact :as touch]]
rlm@572	2514 (-> (zipmap coords contact)
rlm@572	2515 (select-keys touch-region)
rlm@572	2516 (vals)
rlm@572	2517 (#(map first %))
rlm@572	2518 (average)
rlm@572	2519 (* 10)
rlm@572	2520 (- 1)
rlm@572	2521 (Math/abs)))
rlm@572	2522 \end{verbatim}
rlm@572	2523 \caption{\label{touch-summary}Program for summarizing the touch information in a patch of skin.}
rlm@572	2524 \end{listing}
rlm@572	2525
rlm@572	2526
rlm@572	2527 \begin{listing}
rlm@572	2528 \begin{verbatim}
rlm@572	2529 (def worm-segment-bottom (rect-region [8 15] [14 22]))
rlm@572	2530
rlm@572	2531 (defn resting?
rlm@572	2532 "Is the worm resting on the ground?"
rlm@572	2533 [experiences]
rlm@572	2534 (every?
rlm@572	2535 (fn [touch-data]
rlm@572	2536 (< 0.9 (contact worm-segment-bottom touch-data)))
rlm@572	2537 (:touch (peek experiences))))
rlm@572	2538 \end{verbatim}
rlm@572	2539 \caption{\label{resting}Program for detecting whether the worm is at rest. This program uses a summary of the tactile information from the underbelly of the worm, and is only true if every segment is touching the floor. Note that this function contains no references to proprioception at all.}
rlm@572	2540 \end{listing}
rlm@572	2541
rlm@572	2542 \begin{listing}
rlm@572	2543 \begin{verbatim}
rlm@572	2544 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
rlm@572	2545
rlm@572	2546 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
rlm@572	2547
rlm@572	2548 (defn grand-circle?
rlm@572	2549 "Does the worm form a majestic circle (one end touching the other)?"
rlm@572	2550 [experiences]
rlm@572	2551 (and (curled? experiences)
rlm@572	2552 (let [worm-touch (:touch (peek experiences))
rlm@572	2553 tail-touch (worm-touch 0)
rlm@572	2554 head-touch (worm-touch 4)]
rlm@572	2555 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
rlm@572	2556 (< 0.55 (contact worm-segment-top-tip head-touch))))))
rlm@572	2557 \end{verbatim}
rlm@572	2558 \caption{\label{grand-circle}Program for detecting whether the worm is curled up into a full circle. Here the embodied approach begins to shine, as I am able to both use a previous action predicate (\texttt{curled?}) as well as the direct tactile experience of the head and tail.}
rlm@572	2559 \end{listing}
rlm@572	2560
rlm@572	2561
rlm@572	2562 \begin{listing}
rlm@572	2563 \begin{verbatim}
rlm@572	2564 (defn fft [nums]
rlm@572	2565 (map
rlm@572	2566 #(.getReal %)
rlm@572	2567 (.transform
rlm@572	2568 (FastFourierTransformer. DftNormalization/STANDARD)
rlm@572	2569 (double-array nums) TransformType/FORWARD)))
rlm@572	2570
rlm@572	2571 (def indexed (partial map-indexed vector))
rlm@572	2572
rlm@572	2573 (defn max-indexed [s]
rlm@572	2574 (first (sort-by (comp - second) (indexed s))))
rlm@572	2575
rlm@572	2576 (defn wiggling?
rlm@572	2577 "Is the worm wiggling?"
rlm@572	2578 [experiences]
rlm@572	2579 (let [analysis-interval 0x40]
rlm@572	2580 (when (> (count experiences) analysis-interval)
rlm@572	2581 (let [a-flex 3
rlm@572	2582 a-ex 2
rlm@572	2583 muscle-activity
rlm@572	2584 (map :muscle (vector:last-n experiences analysis-interval))
rlm@572	2585 base-activity
rlm@572	2586 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
rlm@572	2587 (= 2
rlm@572	2588 (first
rlm@572	2589 (max-indexed
rlm@572	2590 (map #(Math/abs %)
rlm@572	2591 (take 20 (fft base-activity))))))))))
rlm@572	2592 \end{verbatim}
rlm@572	2593 \caption{\label{wiggling}Program for detecting whether the worm has been wiggling for the last few frames. It uses a Fourier analysis of the muscle contractions of the worm's tail to determine wiggling. This is significant because there is no particular frame that clearly indicates that the worm is wiggling --- only when multiple frames are analyzed together is the wiggling revealed. Defining wiggling this way also gives the worm an opportunity to learn and recognize ``frustrated wiggling'', where the worm tries to wiggle but can't. Frustrated wiggling is very visually different from actual wiggling, but this definition gives it to us for free.}
rlm@572	2594 \end{listing}
rlm@572	2595
rlm@572	2596 With these action predicates, I can now recognize the actions of
rlm@572	2597 the worm while it is moving under my control and I have access to
rlm@572	2598 all the worm's senses.
rlm@572	2599
rlm@572	2600 \begin{listing}
rlm@572	2601 \begin{verbatim}
rlm@572	2602 (defn debug-experience
rlm@572	2603 [experiences text]
rlm@572	2604 (cond
rlm@572	2605 (grand-circle? experiences) (.setText text "Grand Circle")
rlm@572	2606 (curled? experiences) (.setText text "Curled")
rlm@572	2607 (wiggling? experiences) (.setText text "Wiggling")
rlm@572	2608 (resting? experiences) (.setText text "Resting")))
rlm@572	2609 \end{verbatim}
rlm@572	2610 \caption{\label{report-worm-activity}Use the action predicates defined earlier to report on what the worm is doing while in simulation.}
rlm@572	2611 \end{listing}
rlm@572	2612
rlm@572	2613 \begin{figure}[htb]
rlm@572	2614 \centering
rlm@572	2615 \includegraphics[width=10cm]{./images/worm-identify-init.png}
rlm@572	2616 \caption{\label{basic-worm-view}Using \texttt{debug-experience}, the body-centered predicates work together to classify the behavior of the worm. the predicates are operating with access to the worm's full sensory data.}
rlm@572	2617 \end{figure}
rlm@572	2618
rlm@572	2619 These action predicates satisfy the recognition requirement of an
rlm@572	2620 empathic recognition system. There is power in the simplicity of
rlm@572	2621 the action predicates. They describe their actions without getting
rlm@572	2622 confused in visual details of the worm. Each one is independent of
rlm@572	2623 position and rotation, but more than that, they are each
rlm@572	2624 independent of irrelevant visual details of the worm and the
rlm@572	2625 environment. They will work regardless of whether the worm is a
rlm@572	2626 different color or heavily textured, or if the environment has
rlm@572	2627 strange lighting.
rlm@572	2628
rlm@572	2629 Consider how the human act of jumping might be described with
rlm@572	2630 body-centered action predicates: You might specify that jumping is
rlm@572	2631 mainly the feeling of your knees bending, your thigh muscles
rlm@572	2632 contracting, and your inner ear experiencing a certain sort of back
rlm@572	2633 and forth acceleration. This representation is a very concrete
rlm@572	2634 description of jumping, couched in terms of muscles and senses, but
rlm@572	2635 it also has the ability to describe almost all kinds of jumping, a
rlm@572	2636 generality that you might think could only be achieved by a very
rlm@572	2637 abstract description. The body centered jumping predicate does not
rlm@572	2638 have terms that consider the color of a person's skin or whether
rlm@572	2639 they are male or female, instead it gets right to the meat of what
rlm@572	2640 jumping actually \emph{is}.
rlm@572	2641
rlm@572	2642 Of course, the action predicates are not directly applicable to
rlm@572	2643 video data, which lacks the advanced sensory information which they
rlm@572	2644 require!
rlm@572	2645
rlm@572	2646 The trick now is to make the action predicates work even when the
rlm@572	2647 sensory data on which they depend is absent!
rlm@572	2648
rlm@572	2649 \subsection{$\Phi$-space describes the worm's experiences}
rlm@572	2650 \label{sec-3-3}
rlm@572	2651
rlm@572	2652 As a first step towards building empathy, I need to gather all of
rlm@572	2653 the worm's experiences during free play. I use a simple vector to
rlm@572	2654 store all the experiences.
rlm@572	2655
rlm@572	2656 Each element of the experience vector exists in the vast space of
rlm@572	2657 all possible worm-experiences. Most of this vast space is actually
rlm@572	2658 unreachable due to physical constraints of the worm's body. For
rlm@572	2659 example, the worm's segments are connected by hinge joints that put
rlm@572	2660 a practical limit on the worm's range of motions without limiting
rlm@572	2661 its degrees of freedom. Some groupings of senses are impossible;
rlm@572	2662 the worm can not be bent into a circle so that its ends are
rlm@572	2663 touching and at the same time not also experience the sensation of
rlm@572	2664 touching itself.
rlm@572	2665
rlm@572	2666 As the worm moves around during free play and its experience vector
rlm@572	2667 grows larger, the vector begins to define a subspace which is all
rlm@572	2668 the sensations the worm can practically experience during normal
rlm@572	2669 operation. I call this subspace $\Phi$-space, short for
rlm@572	2670 physical-space. The experience vector defines a path through
rlm@572	2671 $\Phi$-space. This path has interesting properties that all derive
rlm@572	2672 from physical embodiment. The proprioceptive components of the path
rlm@572	2673 vary smoothly, because in order for the worm to move from one
rlm@572	2674 position to another, it must pass through the intermediate
rlm@572	2675 positions. The path invariably forms loops as common actions are
rlm@572	2676 repeated. Finally and most importantly, proprioception alone
rlm@572	2677 actually gives very strong inference about the other senses. For
rlm@572	2678 example, when the worm is proprioceptively flat over several
rlm@572	2679 frames, you can infer that it is touching the ground and that its
rlm@572	2680 muscles are not active, because if the muscles were active, the
rlm@572	2681 worm would be moving and would not remain perfectly flat. In order
rlm@572	2682 to stay flat, the worm has to be touching the ground, or it would
rlm@572	2683 again be moving out of the flat position due to gravity. If the
rlm@572	2684 worm is positioned in such a way that it interacts with itself,
rlm@572	2685 then it is very likely to be feeling the same tactile feelings as
rlm@572	2686 the last time it was in that position, because it has the same body
rlm@572	2687 as then. As you observe multiple frames of proprioceptive data, you
rlm@572	2688 can become increasingly confident about the exact activations of
rlm@572	2689 the worm's muscles, because it generally takes a unique combination
rlm@572	2690 of muscle contractions to transform the worm's body along a
rlm@572	2691 specific path through $\Phi$-space.
rlm@572	2692
rlm@572	2693 The worm's total life experience is a long looping path through
rlm@572	2694 $\Phi$-space. I will now introduce simple way of taking that
rlm@572	2695 experience path and building a function that can infer complete
rlm@572	2696 sensory experience given only a stream of proprioceptive data. This
rlm@572	2697 \emph{empathy} function will provide a bridge to use the body centered
rlm@572	2698 action predicates on video-like streams of information.
rlm@572	2699
rlm@572	2700 \subsection{Empathy is the process of building paths in $\Phi$-space}
rlm@572	2701 \label{sec-3-4}
rlm@572	2702
rlm@572	2703 Here is the core of a basic empathy algorithm, starting with an
rlm@572	2704 experience vector:
rlm@572	2705
rlm@572	2706 An \emph{experience-index} is an index into the grand experience vector
rlm@572	2707 that defines the worm's life. It is a time-stamp for each set of
rlm@572	2708 sensations the worm has experienced.
rlm@572	2709
rlm@572	2710 First, I group the experience-indices into bins according to the
rlm@572	2711 similarity of their proprioceptive data. I organize my bins into a
rlm@572	2712 3 level hierarchy. The smallest bins have an approximate size of
rlm@572	2713 0.001 radians in all proprioceptive dimensions. Each higher level
rlm@572	2714 is 10x bigger than the level below it.
rlm@572	2715
rlm@572	2716 The bins serve as a hashing function for proprioceptive data. Given
rlm@572	2717 a single piece of proprioceptive experience, the bins allow me to
rlm@572	2718 rapidly find all other similar experience-indices of past
rlm@572	2719 experience that had a very similar proprioceptive configuration.
rlm@572	2720 When looking up a proprioceptive experience, if the smallest bin
rlm@572	2721 does not match any previous experience, then I use successively
rlm@572	2722 larger bins until a match is found or I reach the largest bin.
rlm@572	2723
rlm@572	2724 Given a sequence of proprioceptive input, I use the bins to
rlm@572	2725 generate a set of similar experiences for each input using the
rlm@572	2726 tiered proprioceptive bins.
rlm@572	2727
rlm@572	2728 Finally, to infer sensory data, I select the longest consecutive
rlm@572	2729 chain of experiences that threads through the sets of similar
rlm@572	2730 experiences, starting with the current moment as a root and going
rlm@572	2731 backwards. Consecutive experience means that the experiences appear
rlm@572	2732 next to each other in the experience vector.
rlm@572	2733
rlm@572	2734 A stream of proprioceptive input might be:
rlm@572	2735
rlm@572	2736 \begin{verbatim}
rlm@572	2737 [ flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572	2738 \end{verbatim}
rlm@572	2739
rlm@572	2740 The worm's previous experience of lying on the ground and lifting
rlm@572	2741 its head generates possible interpretations for each frame (the
rlm@572	2742 numbers are experience-indices):
rlm@572	2743
rlm@572	2744 \clearpage
rlm@572	2745
rlm@572	2746 \begin{verbatim}
rlm@572	2747 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572	2748 1 1 1 1 1 1 1 4
rlm@572	2749 2 2 2 2 2 2 2
rlm@572	2750 3 3 3 3 3 3 3
rlm@572	2751 6 6 6 6 6 6 6
rlm@572	2752 7 7 7 7 7 7 7
rlm@572	2753 8 8 8 8 8 8 8
rlm@572	2754 9 9 9 9 9 9 9
rlm@572	2755 \end{verbatim}
rlm@572	2756
rlm@572	2757 These interpretations suggest a new path through phi space:
rlm@572	2758
rlm@572	2759 \begin{verbatim}
rlm@572	2760 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572	2761 6 7 8 9 1 2 3 4
rlm@572	2762 \end{verbatim}
rlm@572	2763
rlm@572	2764 The new path through $\Phi$-space is synthesized from two actual
rlm@572	2765 paths that the creature has experienced: the "1-2-3-4" chain and
rlm@572	2766 the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
rlm@572	2767 ends with the worm lifting its head. It originated from a short
rlm@572	2768 training session where the worm rested on the floor for a brief
rlm@572	2769 while and then raised its head. The "6-7-8-9" chain is part of a
rlm@572	2770 longer chain of inactivity where the worm simply rested on the
rlm@572	2771 floor without moving. It is preferred over a "1-2-3" chain (which
rlm@572	2772 also describes inactivity) because it is longer. The main ideas
rlm@572	2773 again:
rlm@572	2774
rlm@572	2775 \begin{itemize}
rlm@572	2776 \item Imagined $\Phi$-space paths are synthesized by looping and mixing
rlm@572	2777 previous experiences.
rlm@572	2778
rlm@572	2779 \item Longer experience paths (less edits) are preferred.
rlm@572	2780
rlm@572	2781 \item The present is more important than the past --- more recent
rlm@572	2782 events take precedence in interpretation.
rlm@572	2783 \end{itemize}
rlm@572	2784
rlm@572	2785 This algorithm has three advantages:
rlm@572	2786
rlm@572	2787 \begin{enumerate}
rlm@572	2788 \item It's simple
rlm@572	2789
rlm@572	2790 \item It's very fast -- retrieving possible interpretations takes
rlm@572	2791 constant time. Tracing through chains of interpretations takes
rlm@572	2792 time proportional to the average number of experiences in a
rlm@572	2793 proprioceptive bin. Redundant experiences in $\Phi$-space can be
rlm@572	2794 merged to save computation.
rlm@572	2795
rlm@572	2796 \item It protects from wrong interpretations of transient ambiguous
rlm@572	2797 proprioceptive data. For example, if the worm is flat for just
rlm@572	2798 an instant, this flatness will not be interpreted as implying
rlm@572	2799 that the worm has its muscles relaxed, since the flatness is
rlm@572	2800 part of a longer chain which includes a distinct pattern of
rlm@572	2801 muscle activation. Markov chains or other memoryless statistical
rlm@572	2802 models that operate on individual frames may very well make this
rlm@572	2803 mistake.
rlm@572	2804 \end{enumerate}
rlm@572	2805
rlm@572	2806 \begin{listing}
rlm@572	2807 \begin{verbatim}
rlm@572	2808 (defn bin [digits]
rlm@572	2809 (fn [angles]
rlm@572	2810 (->> angles
rlm@572	2811 (flatten)
rlm@572	2812 (map (juxt #(Math/sin %) #(Math/cos %)))
rlm@572	2813 (flatten)
rlm@572	2814 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
rlm@572	2815
rlm@572	2816 (defn gen-phi-scan
rlm@572	2817 "Nearest-neighbors with binning. Only returns a result if
rlm@572	2818 the proprioceptive data is within 10% of a previously recorded
rlm@572	2819 result in all dimensions."
rlm@572	2820 [phi-space]
rlm@572	2821 (let [bin-keys (map bin [3 2 1])
rlm@572	2822 bin-maps
rlm@572	2823 (map (fn [bin-key]
rlm@572	2824 (group-by
rlm@572	2825 (comp bin-key :proprioception phi-space)
rlm@572	2826 (range (count phi-space)))) bin-keys)
rlm@572	2827 lookups (map (fn [bin-key bin-map]
rlm@572	2828 (fn [proprio] (bin-map (bin-key proprio))))
rlm@572	2829 bin-keys bin-maps)]
rlm@572	2830 (fn lookup [proprio-data]
rlm@572	2831 (set (some #(% proprio-data) lookups)))))
rlm@572	2832 \end{verbatim}
rlm@572	2833 \caption{\label{bin}Program to convert an experience vector into a proprioceptively binned lookup function.}
rlm@572	2834 \end{listing}
rlm@572	2835
rlm@572	2836 \begin{figure}[htb]
rlm@572	2837 \centering
rlm@572	2838 \includegraphics[width=10cm]{./images/film-of-imagination.png}
rlm@572	2839 \caption{\label{phi-space-history-scan}\texttt{longest-thread} finds the longest path of consecutive past experiences to explain proprioceptive worm data from previous data. Here, the film strip represents the creature's previous experience. Sort sequences of memories are spliced together to match the proprioceptive data. Their carry the other senses along with them.}
rlm@572	2840 \end{figure}
rlm@572	2841
rlm@572	2842 \texttt{longest-thread} infers sensory data by stitching together pieces
rlm@572	2843 from previous experience. It prefers longer chains of previous
rlm@572	2844 experience to shorter ones. For example, during training the worm
rlm@572	2845 might rest on the ground for one second before it performs its
rlm@572	2846 exercises. If during recognition the worm rests on the ground for
rlm@572	2847 five seconds, \texttt{longest-thread} will accommodate this five second
rlm@572	2848 rest period by looping the one second rest chain five times.
rlm@572	2849
rlm@572	2850 \texttt{longest-thread} takes time proportional to the average number of
rlm@572	2851 entries in a proprioceptive bin, because for each element in the
rlm@572	2852 starting bin it performs a series of set lookups in the preceding
rlm@572	2853 bins. If the total history is limited, then this takes time
rlm@572	2854 proportional to a only a constant multiple of the number of entries
rlm@572	2855 in the starting bin. This analysis also applies, even if the action
rlm@572	2856 requires multiple longest chains -- it's still the average number
rlm@572	2857 of entries in a proprioceptive bin times the desired chain length.
rlm@572	2858 Because \texttt{longest-thread} is so efficient and simple, I can
rlm@572	2859 interpret worm-actions in real time.
rlm@572	2860
rlm@572	2861 \begin{listing}
rlm@572	2862 \begin{verbatim}
rlm@572	2863 (defn longest-thread
rlm@572	2864 "Find the longest thread from phi-index-sets. The index sets should
rlm@572	2865 be ordered from most recent to least recent."
rlm@572	2866 [phi-index-sets]
rlm@572	2867 (loop [result '()
rlm@572	2868 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
rlm@572	2869 (if (empty? phi-index-sets)
rlm@572	2870 (vec result)
rlm@572	2871 (let [threads
rlm@572	2872 (for [thread-base thread-bases]
rlm@572	2873 (loop [thread (list thread-base)
rlm@572	2874 remaining remaining]
rlm@572	2875 (let [next-index (dec (first thread))]
rlm@572	2876 (cond (empty? remaining) thread
rlm@572	2877 (contains? (first remaining) next-index)
rlm@572	2878 (recur
rlm@572	2879 (cons next-index thread) (rest remaining))
rlm@572	2880 :else thread))))
rlm@572	2881 longest-thread
rlm@572	2882 (reduce (fn [thread-a thread-b]
rlm@572	2883 (if (> (count thread-a) (count thread-b))
rlm@572	2884 thread-a thread-b))
rlm@572	2885 '(nil)
rlm@572	2886 threads)]
rlm@572	2887 (recur (concat longest-thread result)
rlm@572	2888 (drop (count longest-thread) phi-index-sets))))))
rlm@572	2889 \end{verbatim}
rlm@572	2890 \caption{\label{longest-thread}Program to calculate empathy by tracing though $\Phi$-space and finding the longest (ie. most coherent) interpretation of the data.}
rlm@572	2891 \end{listing}
rlm@572	2892
rlm@572	2893 There is one final piece, which is to replace missing sensory data
rlm@572	2894 with a best-guess estimate. While I could fill in missing data by
rlm@572	2895 using a gradient over the closest known sensory data points,
rlm@572	2896 averages can be misleading. It is certainly possible to create an
rlm@572	2897 impossible sensory state by averaging two possible sensory states.
rlm@572	2898 For example, consider moving your hand in an arc over your head. If
rlm@572	2899 for some reason you only have the initial and final positions of
rlm@572	2900 this movement in your $\Phi$-space, averaging them together will
rlm@572	2901 produce the proprioceptive sensation of having your hand \emph{inside}
rlm@572	2902 your head, which is physically impossible to ever experience
rlm@572	2903 (barring motor adaption illusions). Therefore I simply replicate
rlm@572	2904 the most recent sensory experience to fill in the gaps.
rlm@572	2905
rlm@572	2906 \begin{listing}
rlm@572	2907 \begin{verbatim}
rlm@572	2908 (defn infer-nils
rlm@572	2909 "Replace nils with the next available non-nil element in the
rlm@572	2910 sequence, or barring that, 0."
rlm@572	2911 [s]
rlm@572	2912 (loop [i (dec (count s))
rlm@572	2913 v (transient s)]
rlm@572	2914 (if (zero? i) (persistent! v)
rlm@572	2915 (if-let [cur (v i)]
rlm@572	2916 (if (get v (dec i) 0)
rlm@572	2917 (recur (dec i) v)
rlm@572	2918 (recur (dec i) (assoc! v (dec i) cur)))
rlm@572	2919 (recur i (assoc! v i 0))))))
rlm@572	2920 \end{verbatim}
rlm@572	2921 \caption{\label{infer-nils}Fill in blanks in sensory experience by replicating the most recent experience.}
rlm@572	2922 \end{listing}
rlm@572	2923
rlm@572	2924 \subsection{\texttt{EMPATH} recognizes actions efficiently}
rlm@572	2925 \label{sec-3-5}
rlm@572	2926
rlm@572	2927 To use \texttt{EMPATH} with the worm, I first need to gather a set of
rlm@572	2928 experiences from the worm that includes the actions I want to
rlm@572	2929 recognize. The \texttt{generate-phi-space} program (listing
rlm@572	2930 \ref{generate-phi-space} runs the worm through a series of
rlm@572	2931 exercises and gathers those experiences into a vector. The
rlm@572	2932 \texttt{do-all-the-things} program is a routine expressed in a simple
rlm@572	2933 muscle contraction script language for automated worm control. It
rlm@572	2934 causes the worm to rest, curl, and wiggle over about 700 frames
rlm@572	2935 (approx. 11 seconds).
rlm@572	2936
rlm@572	2937 \begin{listing}
rlm@572	2938 \begin{verbatim}
rlm@572	2939 (def do-all-the-things
rlm@572	2940 (concat
rlm@572	2941 curl-script
rlm@572	2942 [[300 :d-ex 40]
rlm@572	2943 [320 :d-ex 0]]
rlm@572	2944 (shift-script 280 (take 16 wiggle-script))))
rlm@572	2945
rlm@572	2946 (defn generate-phi-space []
rlm@572	2947 (let [experiences (atom [])]
rlm@572	2948 (run-world
rlm@572	2949 (apply-map
rlm@572	2950 worm-world
rlm@572	2951 (merge
rlm@572	2952 (worm-world-defaults)
rlm@572	2953 {:end-frame 700
rlm@572	2954 :motor-control
rlm@572	2955 (motor-control-program worm-muscle-labels do-all-the-things)
rlm@572	2956 :experiences experiences})))
rlm@572	2957 @experiences))
rlm@572	2958 \end{verbatim}
rlm@572	2959 \caption{\label{generate-phi-space}Program to gather the worm's experiences into a vector for further processing. The \texttt{motor-control-program} line uses a motor control script that causes the worm to execute a series of ``exercises'' that include all the action predicates.}
rlm@572	2960 \end{listing}
rlm@572	2961
rlm@572	2962 \begin{listing}
rlm@572	2963 \begin{verbatim}
rlm@572	2964 (defn init []
rlm@572	2965 (def phi-space (generate-phi-space))
rlm@572	2966 (def phi-scan (gen-phi-scan phi-space)))
rlm@572	2967
rlm@572	2968 (defn empathy-demonstration []
rlm@572	2969 (let [proprio (atom ())]
rlm@572	2970 (fn
rlm@572	2971 [experiences text]
rlm@572	2972 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572	2973 (swap! proprio (partial cons phi-indices))
rlm@572	2974 (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572	2975 empathy (mapv phi-space (infer-nils exp-thread))]
rlm@572	2976 (println-repl (vector:last-n exp-thread 22))
rlm@572	2977 (cond
rlm@572	2978 (grand-circle? empathy) (.setText text "Grand Circle")
rlm@572	2979 (curled? empathy) (.setText text "Curled")
rlm@572	2980 (wiggling? empathy) (.setText text "Wiggling")
rlm@572	2981 (resting? empathy) (.setText text "Resting")
rlm@572	2982 :else (.setText text "Unknown")))))))
rlm@572	2983
rlm@572	2984 (defn empathy-experiment [record]
rlm@572	2985 (.start (worm-world :experience-watch (debug-experience-phi)
rlm@572	2986 :record record :worm worm*)))
rlm@572	2987 \end{verbatim}
rlm@572	2988 \caption{\label{empathy-debug}Use \texttt{longest-thread} and a $\Phi$-space generated from a short exercise routine to interpret actions during free play.}
rlm@572	2989 \end{listing}
rlm@572	2990
rlm@572	2991 These programs create a test for the empathy system. First, the
rlm@572	2992 worm's $\Phi$-space is generated from a simple motor script. Then the
rlm@572	2993 worm is re-created in an environment almost exactly identical to
rlm@572	2994 the testing environment for the action-predicates, with one major
rlm@572	2995 difference : the only sensory information available to the system
rlm@572	2996 is proprioception. From just the proprioception data and
rlm@572	2997 $\Phi$-space, \texttt{longest-thread} synthesizes a complete record the last
rlm@572	2998 300 sensory experiences of the worm. These synthesized experiences
rlm@572	2999 are fed directly into the action predicates \texttt{grand-circle?},
rlm@572	3000 \texttt{curled?}, \texttt{wiggling?}, and \texttt{resting?} and their outputs are
rlm@572	3001 printed to the screen at each frame.
rlm@572	3002
rlm@572	3003 The result of running \texttt{empathy-experiment} is that the system is
rlm@572	3004 generally able to interpret worm actions using the action-predicates
rlm@572	3005 on simulated sensory data just as well as with actual data. Figure
rlm@572	3006 \ref{empathy-debug-image} was generated using \texttt{empathy-experiment}:
rlm@572	3007
rlm@572	3008 \begin{figure}[htb]
rlm@572	3009 \centering
rlm@572	3010 \includegraphics[width=10cm]{./images/empathy-1.png}
rlm@572	3011 \caption{\label{empathy-debug-image}From only proprioceptive data, \texttt{EMPATH} was able to infer the complete sensory experience and classify four poses (The last panel shows a composite image of \emph{wiggling}, a dynamic pose.)}
rlm@572	3012 \end{figure}
rlm@572	3013
rlm@572	3014 One way to measure the performance of \texttt{EMPATH} is to compare the
rlm@572	3015 suitability of the imagined sense experience to trigger the same
rlm@572	3016 action predicates as the real sensory experience.
rlm@572	3017
rlm@572	3018 \begin{listing}
rlm@572	3019 \begin{verbatim}
rlm@572	3020 (def worm-action-label
rlm@572	3021 (juxt grand-circle? curled? wiggling?))
rlm@572	3022
rlm@572	3023 (defn compare-empathy-with-baseline [matches]
rlm@572	3024 (let [proprio (atom ())]
rlm@572	3025 (fn
rlm@572	3026 [experiences text]
rlm@572	3027 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572	3028 (swap! proprio (partial cons phi-indices))
rlm@572	3029 (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572	3030 empathy (mapv phi-space (infer-nils exp-thread))
rlm@572	3031 experience-matches-empathy
rlm@572	3032 (= (worm-action-label experiences)
rlm@572	3033 (worm-action-label empathy))]
rlm@572	3034 (println-repl experience-matches-empathy)
rlm@572	3035 (swap! matches #(conj % experience-matches-empathy)))))))
rlm@572	3036
rlm@572	3037 (defn accuracy [v]
rlm@572	3038 (float (/ (count (filter true? v)) (count v))))
rlm@572	3039
rlm@572	3040 (defn test-empathy-accuracy []
rlm@572	3041 (let [res (atom [])]
rlm@572	3042 (run-world
rlm@572	3043 (worm-world :experience-watch
rlm@572	3044 (compare-empathy-with-baseline res)
rlm@572	3045 :worm worm*))
rlm@572	3046 (accuracy @res)))
rlm@572	3047 \end{verbatim}
rlm@572	3048 \caption{\label{test-empathy-accuracy}Determine how closely empathy approximates actual sensory data.}
rlm@572	3049 \end{listing}
rlm@572	3050
rlm@572	3051 Running \texttt{test-empathy-accuracy} using the very short exercise
rlm@572	3052 program \texttt{do-all-the-things} defined in listing
rlm@572	3053 \ref{generate-phi-space}, and then doing a similar pattern of
rlm@572	3054 activity using manual control of the worm, yields an accuracy of
rlm@572	3055 around 73\%. This is based on very limited worm experience, and
rlm@572	3056 almost all errors are due to the worm's $\Phi$-space being too
rlm@572	3057 incomplete to properly interpret common poses. By manually training
rlm@572	3058 the worm for longer using \texttt{init-interactive} defined in listing
rlm@572	3059 \ref{manual-phi-space}, the accuracy dramatically improves:
rlm@572	3060
rlm@572	3061 \begin{listing}
rlm@572	3062 \begin{verbatim}
rlm@572	3063 (defn init-interactive []
rlm@572	3064 (def phi-space
rlm@572	3065 (let [experiences (atom [])]
rlm@572	3066 (run-world
rlm@572	3067 (apply-map
rlm@572	3068 worm-world
rlm@572	3069 (merge
rlm@572	3070 (worm-world-defaults)
rlm@572	3071 {:experiences experiences})))
rlm@572	3072 @experiences))
rlm@572	3073 (def phi-scan (gen-phi-scan phi-space)))
rlm@572	3074 \end{verbatim}
rlm@572	3075 \caption{\label{manual-phi-space}Program to generate $\Phi$-space using manual training.}
rlm@572	3076 \end{listing}
rlm@572	3077
rlm@572	3078 \texttt{init-interactive} allows me to take direct control of the worm's
rlm@572	3079 muscles and run it through each characteristic movement I care
rlm@572	3080 about. After about 1 minute of manual training, I was able to
rlm@572	3081 achieve 95\% accuracy on manual testing of the worm using
rlm@572	3082 \texttt{test-empathy-accuracy}. The majority of disagreements are near the
rlm@572	3083 transition boundaries from one type of action to another. During
rlm@572	3084 these transitions the exact label for the action is often unclear,
rlm@572	3085 and disagreement between empathy and experience is practically
rlm@572	3086 irrelevant. Thus, the system's effective identification accuracy is
rlm@572	3087 even higher than 95\%. When I watch this system myself, I generally
rlm@572	3088 see no errors in action identification compared to my own judgment
rlm@572	3089 of what the worm is doing.
rlm@572	3090
rlm@572	3091 \subsection{Digression: Learning touch sensor layout through free play}
rlm@572	3092 \label{sec-3-6}
rlm@572	3093
rlm@572	3094 In the previous chapter I showed how to compute actions in terms of
rlm@572	3095 body-centered predicates, but some of those predicates relied on
rlm@572	3096 the average touch activation of pre-defined regions of the worm's
rlm@572	3097 skin. What if, instead of receiving touch pre-grouped into the six
rlm@572	3098 faces of each worm segment, the true partitioning of the worm's
rlm@572	3099 skin was unknown? This is more similar to how a nerve fiber bundle
rlm@572	3100 might be arranged inside an animal. While two fibers that are close
rlm@572	3101 in a nerve bundle \emph{might} correspond to two touch sensors that are
rlm@572	3102 close together on the skin, the process of taking a complicated
rlm@572	3103 surface and forcing it into essentially a 2D circle requires that
rlm@572	3104 some regions of skin that are close together in the animal end up
rlm@572	3105 far apart in the nerve bundle.
rlm@572	3106
rlm@572	3107 In this chapter I show how to automatically learn the
rlm@572	3108 skin-partitioning of a worm segment by free exploration. As the
rlm@572	3109 worm rolls around on the floor, large sections of its surface get
rlm@572	3110 activated. If the worm has stopped moving, then whatever region of
rlm@572	3111 skin that is touching the floor is probably an important region,
rlm@572	3112 and should be recorded. The code I provide relies on the worm
rlm@572	3113 segment having flat faces, but still demonstrates a primitive kind
rlm@572	3114 of multi-sensory bootstrapping that I find appealing.
rlm@572	3115
rlm@572	3116 \begin{listing}
rlm@572	3117 \begin{verbatim}
rlm@572	3118 (def full-contact [(float 0.0) (float 0.1)])
rlm@572	3119
rlm@572	3120 (defn pure-touch?
rlm@572	3121 "This is worm specific code to determine if a large region of touch
rlm@572	3122 sensors is either all on or all off."
rlm@572	3123 [[coords touch :as touch-data]]
rlm@572	3124 (= (set (map first touch)) (set full-contact)))
rlm@572	3125 \end{verbatim}
rlm@572	3126 \caption{\label{pure-touch}Program to detect whether the worm is in a resting state with one face touching the floor.}
rlm@572	3127 \end{listing}
rlm@572	3128
rlm@572	3129 After collecting these important regions, there will many nearly
rlm@572	3130 similar touch regions. While for some purposes the subtle
rlm@572	3131 differences between these regions will be important, for my
rlm@572	3132 purposes I collapse them into mostly non-overlapping sets using
rlm@572	3133 \texttt{remove-similar} in listing \ref{remove-similar}
rlm@572	3134
rlm@572	3135 \begin{listing}
rlm@572	3136 \begin{verbatim}
rlm@572	3137 (defn remove-similar
rlm@572	3138 [coll]
rlm@572	3139 (loop [result () coll (sort-by (comp - count) coll)]
rlm@572	3140 (if (empty? coll) result
rlm@572	3141 (let [[x & xs] coll
rlm@572	3142 c (count x)]
rlm@572	3143 (if (some
rlm@572	3144 (fn [other-set]
rlm@572	3145 (let [oc (count other-set)]
rlm@572	3146 (< (- (count (union other-set x)) c) (* oc 0.1))))
rlm@572	3147 xs)
rlm@572	3148 (recur result xs)
rlm@572	3149 (recur (cons x result) xs))))))
rlm@572	3150 \end{verbatim}
rlm@572	3151 \caption{\label{remove-similar}Program to take a list of sets of points and ``collapse them'' so that the remaining sets in the list are significantly different from each other. Prefer smaller sets to larger ones.}
rlm@572	3152 \end{listing}
rlm@572	3153
rlm@572	3154 Actually running this simulation is easy given \texttt{CORTEX}'s facilities.
rlm@572	3155
rlm@572	3156 \begin{listing}
rlm@572	3157 \begin{verbatim}
rlm@572	3158 (defn learn-touch-regions []
rlm@572	3159 (let [experiences (atom [])
rlm@572	3160 world (apply-map
rlm@572	3161 worm-world
rlm@572	3162 (assoc (worm-segment-defaults)
rlm@572	3163 :experiences experiences))]
rlm@572	3164 (run-world world)
rlm@572	3165 (->>
rlm@572	3166 @experiences
rlm@572	3167 (drop 175)
rlm@572	3168 ;; access the single segment's touch data
rlm@572	3169 (map (comp first :touch))
rlm@572	3170 ;; only deal with "pure" touch data to determine surfaces
rlm@572	3171 (filter pure-touch?)
rlm@572	3172 ;; associate coordinates with touch values
rlm@572	3173 (map (partial apply zipmap))
rlm@572	3174 ;; select those regions where contact is being made
rlm@572	3175 (map (partial group-by second))
rlm@572	3176 (map #(get % full-contact))
rlm@572	3177 (map (partial map first))
rlm@572	3178 ;; remove redundant/subset regions
rlm@572	3179 (map set)
rlm@572	3180 remove-similar)))
rlm@572	3181
rlm@572	3182 (defn learn-and-view-touch-regions []
rlm@572	3183 (map view-touch-region
rlm@572	3184 (learn-touch-regions)))
rlm@572	3185 \end{verbatim}
rlm@572	3186 \caption{\label{learn-touch}Collect experiences while the worm moves around. Filter the touch sensations by stable ones, collapse similar ones together, and report the regions learned.}
rlm@572	3187 \end{listing}
rlm@572	3188
rlm@572	3189 The only thing remaining to define is the particular motion the worm
rlm@572	3190 must take. I accomplish this with a simple motor control program.
rlm@572	3191
rlm@572	3192 \begin{listing}
rlm@572	3193 \begin{verbatim}
rlm@572	3194 (defn touch-kinesthetics []
rlm@572	3195 [[170 :lift-1 40]
rlm@572	3196 [190 :lift-1 19]
rlm@572	3197 [206 :lift-1 0]
rlm@572	3198
rlm@572	3199 [400 :lift-2 40]
rlm@572	3200 [410 :lift-2 0]
rlm@572	3201
rlm@572	3202 [570 :lift-2 40]
rlm@572	3203 [590 :lift-2 21]
rlm@572	3204 [606 :lift-2 0]
rlm@572	3205
rlm@572	3206 [800 :lift-1 30]
rlm@572	3207 [809 :lift-1 0]
rlm@572	3208
rlm@572	3209 [900 :roll-2 40]
rlm@572	3210 [905 :roll-2 20]
rlm@572	3211 [910 :roll-2 0]
rlm@572	3212
rlm@572	3213 [1000 :roll-2 40]
rlm@572	3214 [1005 :roll-2 20]
rlm@572	3215 [1010 :roll-2 0]
rlm@572	3216
rlm@572	3217 [1100 :roll-2 40]
rlm@572	3218 [1105 :roll-2 20]
rlm@572	3219 [1110 :roll-2 0]
rlm@572	3220 ])
rlm@572	3221 \end{verbatim}
rlm@572	3222 \caption{\label{worm-roll}Motor control program for making the worm roll on the ground. This could also be replaced with random motion.}
rlm@572	3223 \end{listing}
rlm@572	3224
rlm@572	3225
rlm@572	3226 \begin{figure}[htb]
rlm@572	3227 \centering
rlm@572	3228 \includegraphics[width=12cm]{./images/worm-roll.png}
rlm@572	3229 \caption{\label{worm-roll}The small worm rolls around on the floor, driven by the motor control program in listing \ref{worm-roll}.}
rlm@572	3230 \end{figure}
rlm@572	3231
rlm@572	3232 \begin{figure}[htb]
rlm@572	3233 \centering
rlm@572	3234 \includegraphics[width=12cm]{./images/touch-learn.png}
rlm@572	3235 \caption{\label{worm-touch-map}After completing its adventures, the worm now knows how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were deemed important by \texttt{learn-touch-regions}. Each white square in the rectangles above is a cluster of ``related" touch nodes as determined by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces.}
rlm@572	3236 \end{figure}
rlm@572	3237
rlm@572	3238 While simple, \texttt{learn-touch-regions} exploits regularities in both
rlm@572	3239 the worm's physiology and the worm's environment to correctly
rlm@572	3240 deduce that the worm has six sides. Note that \texttt{learn-touch-regions}
rlm@572	3241 would work just as well even if the worm's touch sense data were
rlm@572	3242 completely scrambled. The cross shape is just for convenience. This
rlm@572	3243 example justifies the use of pre-defined touch regions in \texttt{EMPATH}.
rlm@572	3244
rlm@572	3245 \subsection{Recognizing an object using embodied representation}
rlm@572	3246 \label{sec-3-7}
rlm@572	3247
rlm@572	3248 At the beginning of the thesis, I suggested that we might recognize
rlm@572	3249 the chair in Figure \ref{hidden-chair} by imagining ourselves in
rlm@572	3250 the position of the man and realizing that he must be sitting on
rlm@572	3251 something in order to maintain that position. Here, I present a
rlm@572	3252 brief elaboration on how to this might be done.
rlm@572	3253
rlm@572	3254 First, I need the feeling of leaning or resting \emph{on} some other
rlm@572	3255 object that is not the floor. This feeling is easy to describe
rlm@572	3256 using an embodied representation.
rlm@572	3257
rlm@572	3258 \begin{listing}
rlm@572	3259 \begin{verbatim}
rlm@572	3260 (defn draped?
rlm@572	3261 "Is the worm:
rlm@572	3262 -- not flat (the floor is not a 'chair')
rlm@572	3263 -- supported (not using its muscles to hold its position)
rlm@572	3264 -- stable (not changing its position)
rlm@572	3265 -- touching something (must register contact)"
rlm@572	3266 [experiences]
rlm@572	3267 (let [b2-hash (bin 2)
rlm@572	3268 touch (:touch (peek experiences))
rlm@572	3269 total-contact
rlm@572	3270 (reduce
rlm@572	3271 +
rlm@572	3272 (map #(contact all-touch-coordinates %)
rlm@572	3273 (rest touch)))]
rlm@572	3274 (println total-contact)
rlm@572	3275 (and (not (resting? experiences))
rlm@572	3276 (every?
rlm@572	3277 zero?
rlm@572	3278 (-> experiences
rlm@572	3279 (vector:last-n 25)
rlm@572	3280 (#(map :muscle %))
rlm@572	3281 (flatten)))
rlm@572	3282 (-> experiences
rlm@572	3283 (vector:last-n 20)
rlm@572	3284 (#(map (comp b2-hash flatten :proprioception) %))
rlm@572	3285 (set)
rlm@572	3286 (count) (= 1))
rlm@572	3287 (< 0.03 total-contact))))
rlm@572	3288 \end{verbatim}
rlm@572	3289 \caption{\label{draped}Program describing the sense of leaning or resting on something. This involves a relaxed posture, the feeling of touching something, and a period of stability where the worm does not move.}
rlm@572	3290 \end{listing}
rlm@572	3291
rlm@572	3292 \begin{figure}[htb]
rlm@572	3293 \centering
rlm@572	3294 \includegraphics[width=13cm]{./images/draped.png}
rlm@572	3295 \caption{\label{draped-video}The \texttt{draped?} predicate detects the presence of the cube whenever the worm interacts with it. The details of the cube are irrelevant; only the way it influences the worm's body matters. The ``unknown'' label on the fifth frame is due to the fact that the worm is not stationary. \texttt{draped?} will only declare that the worm is draped if it has been still for a while.}
rlm@572	3296 \end{figure}
rlm@572	3297
rlm@572	3298 Though this is a simple example, using the \texttt{draped?} predicate to
rlm@572	3299 detect a cube has interesting advantages. The \texttt{draped?} predicate
rlm@572	3300 describes the cube not in terms of properties that the cube has,
rlm@572	3301 but instead in terms of how the worm interacts with it physically.
rlm@572	3302 This means that the cube can still be detected even if it is not
rlm@572	3303 visible, as long as its influence on the worm's body is visible.
rlm@572	3304
rlm@572	3305 This system will also see the virtual cube created by a
rlm@572	3306 ``mimeworm", which uses its muscles in a very controlled way to
rlm@572	3307 mimic the appearance of leaning on a cube. The system will
rlm@572	3308 anticipate that there is an actual invisible cube that provides
rlm@572	3309 support!
rlm@572	3310
rlm@572	3311 \begin{figure}[htb]
rlm@572	3312 \centering
rlm@572	3313 \includegraphics[width=6cm]{./images/pablo-the-mime.png}
rlm@572	3314 \caption{\label{mime}Can you see the thing that this person is leaning on? What properties does it have, other than how it makes the man's elbow and shoulder feel? I wonder if people who can actually maintain this pose easily still see the support?}
rlm@572	3315 \end{figure}
rlm@572	3316
rlm@572	3317 This makes me wonder about the psychology of actual mimes. Suppose
rlm@572	3318 for a moment that people have something analogous to $\Phi$-space and
rlm@572	3319 that one of the ways that they find objects in a scene is by their
rlm@572	3320 relation to other people's bodies. Suppose that a person watches a
rlm@572	3321 person miming an invisible wall. For a person with no experience
rlm@572	3322 with miming, their $\Phi$-space will only have entries that describe
rlm@572	3323 the scene with the sensation of their hands touching a wall. This
rlm@572	3324 sensation of touch will create a strong impression of a wall, even
rlm@572	3325 though the wall would have to be invisible. A person with
rlm@572	3326 experience in miming however, will have entries in their $\Phi$-space
rlm@572	3327 that describe the wall-miming position without a sense of touch. It
rlm@572	3328 will not seem to such as person that an invisible wall is present,
rlm@572	3329 but merely that the mime is holding out their hands in a special
rlm@572	3330 way. Thus, the theory that humans use something like $\Phi$-space
rlm@572	3331 weakly predicts that learning how to mime should break the power of
rlm@572	3332 miming illusions. Most optical illusions still work no matter how
rlm@572	3333 much you know about them, so this proposal would be quite
rlm@572	3334 interesting to test, as it predicts a non-standard result!
rlm@572	3335
rlm@572	3336
rlm@572	3337 \clearpage
rlm@572	3338
rlm@572	3339 \section{Contributions}
rlm@572	3340 \label{sec-4}
rlm@572	3341
rlm@572	3342 The big idea behind this thesis is a new way to represent and
rlm@572	3343 recognize physical actions, which I call \emph{empathic representation}.
rlm@572	3344 Actions are represented as predicates which have access to the
rlm@572	3345 totality of a creature's sensory abilities. To recognize the
rlm@572	3346 physical actions of another creature similar to yourself, you
rlm@572	3347 imagine what they would feel by examining the position of their body
rlm@572	3348 and relating it to your own previous experience.
rlm@572	3349
rlm@572	3350 Empathic representation of physical actions is robust and general.
rlm@572	3351 Because the representation is body-centered, it avoids baking in a
rlm@572	3352 particular viewpoint like you might get from learning from example
rlm@572	3353 videos. Because empathic representation relies on all of a
rlm@572	3354 creature's senses, it can describe exactly what an action \emph{feels
rlm@572	3355 like} without getting caught up in irrelevant details such as visual
rlm@572	3356 appearance. I think it is important that a correct description of
rlm@572	3357 jumping (for example) should not include irrelevant details such as
rlm@572	3358 the color of a person's clothes or skin; empathic representation can
rlm@572	3359 get right to the heart of what jumping is by describing it in terms
rlm@572	3360 of touch, muscle contractions, and a brief feeling of
rlm@572	3361 weightlessness. Empathic representation is very low-level in that it
rlm@572	3362 describes actions using concrete sensory data with little
rlm@572	3363 abstraction, but it has the generality of much more abstract
rlm@572	3364 representations!
rlm@572	3365
rlm@572	3366 Another important contribution of this thesis is the development of
rlm@572	3367 the \texttt{CORTEX} system, a complete environment for creating simulated
rlm@572	3368 creatures. You have seen how to implement five senses: touch,
rlm@572	3369 proprioception, hearing, vision, and muscle tension. You have seen
rlm@572	3370 how to create new creatures using Blender, a 3D modeling tool.
rlm@572	3371
rlm@572	3372 As a minor digression, you also saw how I used \texttt{CORTEX} to enable a
rlm@572	3373 tiny worm to discover the topology of its skin simply by rolling on
rlm@572	3374 the ground. You also saw how to detect objects using only embodied
rlm@572	3375 predicates.
rlm@572	3376
rlm@572	3377 In conclusion, for this thesis I:
rlm@572	3378
rlm@572	3379 \begin{itemize}
rlm@572	3380 \item Developed the idea of embodied representation, which describes
rlm@572	3381 actions that a creature can do in terms of first-person sensory
rlm@572	3382 data.
rlm@572	3383
rlm@572	3384 \item Developed a method of empathic action recognition which uses
rlm@572	3385 previous embodied experience and embodied representation of
rlm@572	3386 actions to greatly constrain the possible interpretations of an
rlm@572	3387 action.
rlm@572	3388
rlm@572	3389 \item Created \texttt{EMPATH}, a program which uses empathic action
rlm@572	3390 recognition to recognize physical actions in a simple model
rlm@572	3391 involving segmented worm-like creatures.
rlm@572	3392
rlm@572	3393 \item Created \texttt{CORTEX}, a comprehensive platform for embodied AI
rlm@572	3394 experiments. It is the base on which \texttt{EMPATH} is built.
rlm@572	3395 \end{itemize}
rlm@572	3396
rlm@572	3397 \clearpage
rlm@572	3398 \appendix
rlm@572	3399
rlm@572	3400 \section{Appendix: \texttt{CORTEX} User Guide}
rlm@572	3401 \label{sec-5}
rlm@572	3402
rlm@572	3403 Those who write a thesis should endeavor to make their code not only
rlm@572	3404 accessible, but actually usable, as a way to pay back the community
rlm@572	3405 that made the thesis possible in the first place. This thesis would
rlm@572	3406 not be possible without Free Software such as jMonkeyEngine3,
rlm@572	3407 Blender, clojure, \texttt{emacs}, \texttt{ffmpeg}, and many other tools. That is
rlm@572	3408 why I have included this user guide, in the hope that someone else
rlm@572	3409 might find \texttt{CORTEX} useful.
rlm@572	3410
rlm@572	3411 \subsection{Obtaining \texttt{CORTEX}}
rlm@572	3412 \label{sec-5-1}
rlm@572	3413
rlm@572	3414 You can get cortex from its mercurial repository at
rlm@572	3415 \url{http://hg.bortreb.com/cortex}. You may also download \texttt{CORTEX}
rlm@572	3416 releases at \url{http://aurellem.org/cortex/releases/}. As a condition of
rlm@572	3417 making this thesis, I have also provided Professor Winston the
rlm@572	3418 \texttt{CORTEX} source, and he knows how to run the demos and get started.
rlm@572	3419 You may also email me at \texttt{cortex@aurellem.org} and I may help where
rlm@572	3420 I can.
rlm@572	3421
rlm@572	3422 \subsection{Running \texttt{CORTEX}}
rlm@572	3423 \label{sec-5-2}
rlm@572	3424
rlm@572	3425 \texttt{CORTEX} comes with README and INSTALL files that will guide you
rlm@572	3426 through installation and running the test suite. In particular you
rlm@572	3427 should look at test \texttt{cortex.test} which contains test suites that
rlm@572	3428 run through all senses and multiple creatures.
rlm@572	3429
rlm@572	3430 \subsection{Creating creatures}
rlm@572	3431 \label{sec-5-3}
rlm@572	3432
rlm@572	3433 Creatures are created using \emph{Blender}, a free 3D modeling program.
rlm@572	3434 You will need Blender version 2.6 when using the \texttt{CORTEX} included
rlm@572	3435 in this thesis. You create a \texttt{CORTEX} creature in a similar manner
rlm@572	3436 to modeling anything in Blender, except that you also create
rlm@572	3437 several trees of empty nodes which define the creature's senses.
rlm@572	3438
rlm@572	3439 \subsubsection{Mass}
rlm@572	3440 \label{sec-5-3-1}
rlm@572	3441
rlm@572	3442 To give an object mass in \texttt{CORTEX}, add a ``mass'' metadata label
rlm@572	3443 to the object with the mass in jMonkeyEngine units. Note that
rlm@572	3444 setting the mass to 0 causes the object to be immovable.
rlm@572	3445
rlm@572	3446 \subsubsection{Joints}
rlm@572	3447 \label{sec-5-3-2}
rlm@572	3448
rlm@572	3449 Joints are created by creating an empty node named \texttt{joints} and
rlm@572	3450 then creating any number of empty child nodes to represent your
rlm@572	3451 creature's joints. The joint will automatically connect the
rlm@572	3452 closest two physical objects. It will help to set the empty node's
rlm@572	3453 display mode to ``Arrows'' so that you can clearly see the
rlm@572	3454 direction of the axes.
rlm@572	3455
rlm@572	3456 Joint nodes should have the following metadata under the ``joint''
rlm@572	3457 label:
rlm@572	3458
rlm@572	3459 \begin{verbatim}
rlm@572	3460 ;; ONE of the following, under the label "joint":
rlm@572	3461 {:type :point}
rlm@572	3462
rlm@572	3463 ;; OR
rlm@572	3464
rlm@572	3465 {:type :hinge
rlm@572	3466 :limit [<limit-low> <limit-high>]
rlm@572	3467 :axis (Vector3f. <x> <y> <z>)}
rlm@572	3468 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572	3469
rlm@572	3470 ;; OR
rlm@572	3471
rlm@572	3472 {:type :cone
rlm@572	3473 :limit-xz <lim-xz>
rlm@572	3474 :limit-xy <lim-xy>
rlm@572	3475 :twist <lim-twist>} ;(use XZY rotation mode in Blender!)
rlm@572	3476 \end{verbatim}
rlm@572	3477
rlm@572	3478 \subsubsection{Eyes}
rlm@572	3479 \label{sec-5-3-3}
rlm@572	3480
rlm@572	3481 Eyes are created by creating an empty node named \texttt{eyes} and then
rlm@572	3482 creating any number of empty child nodes to represent your
rlm@572	3483 creature's eyes.
rlm@572	3484
rlm@572	3485 Eye nodes should have the following metadata under the ``eye''
rlm@572	3486 label:
rlm@572	3487
rlm@572	3488 \begin{verbatim}
rlm@572	3489 {:red <red-retina-definition>
rlm@572	3490 :blue <blue-retina-definition>
rlm@572	3491 :green <green-retina-definition>
rlm@572	3492 :all <all-retina-definition>
rlm@572	3493 (<0xrrggbb> <custom-retina-image>)...
rlm@572	3494 }
rlm@572	3495 \end{verbatim}
rlm@572	3496
rlm@572	3497 Any of the color channels may be omitted. You may also include
rlm@572	3498 your own color selectors, and in fact :red is equivalent to
rlm@572	3499 0xFF0000 and so forth. The eye will be placed at the same position
rlm@572	3500 as the empty node and will bind to the neatest physical object.
rlm@572	3501 The eye will point outward from the X-axis of the node, and ``up''
rlm@572	3502 will be in the direction of the X-axis of the node. It will help
rlm@572	3503 to set the empty node's display mode to ``Arrows'' so that you can
rlm@572	3504 clearly see the direction of the axes.
rlm@572	3505
rlm@572	3506 Each retina file should contain white pixels wherever you want to be
rlm@572	3507 sensitive to your chosen color. If you want the entire field of
rlm@572	3508 view, specify :all of 0xFFFFFF and a retinal map that is entirely
rlm@572	3509 white.
rlm@572	3510
rlm@572	3511 Here is a sample retinal map:
rlm@572	3512
rlm@572	3513 \begin{figure}[H]
rlm@572	3514 \centering
rlm@572	3515 \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572	3516 \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572	3517 \end{figure}
rlm@572	3518
rlm@572	3519 \subsubsection{Hearing}
rlm@572	3520 \label{sec-5-3-4}
rlm@572	3521
rlm@572	3522 Ears are created by creating an empty node named \texttt{ears} and then
rlm@572	3523 creating any number of empty child nodes to represent your
rlm@572	3524 creature's ears.
rlm@572	3525
rlm@572	3526 Ear nodes do not require any metadata.
rlm@572	3527
rlm@572	3528 The ear will bind to and follow the closest physical node.
rlm@572	3529
rlm@572	3530 \subsubsection{Touch}
rlm@572	3531 \label{sec-5-3-5}
rlm@572	3532
rlm@572	3533 Touch is handled similarly to mass. To make a particular object
rlm@572	3534 touch sensitive, add metadata of the following form under the
rlm@572	3535 object's ``touch'' metadata field:
rlm@572	3536
rlm@572	3537 \begin{verbatim}
rlm@572	3538 <touch-UV-map-file-name>
rlm@572	3539 \end{verbatim}
rlm@572	3540
rlm@572	3541 You may also include an optional ``scale'' metadata number to
rlm@572	3542 specify the length of the touch feelers. The default is $0.1$,
rlm@572	3543 and this is generally sufficient.
rlm@572	3544
rlm@572	3545 The touch UV should contain white pixels for each touch sensor.
rlm@572	3546
rlm@572	3547 Here is an example touch-uv map that approximates a human finger,
rlm@572	3548 and its corresponding model.
rlm@572	3549
rlm@572	3550 \begin{figure}[htb]
rlm@572	3551 \centering
rlm@572	3552 \includegraphics[width=9cm]{./images/finger-UV.png}
rlm@572	3553 \caption{\label{guide-fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572	3554 \end{figure}
rlm@572	3555
rlm@572	3556 \begin{figure}[htb]
rlm@572	3557 \centering
rlm@572	3558 \includegraphics[width=9cm]{./images/finger-1.png}
rlm@572	3559 \caption{\label{guide-fingertip}The fingertip UV-image form above applied to a simple model of a fingertip.}
rlm@572	3560 \end{figure}
rlm@572	3561
rlm@572	3562 \subsubsection{Proprioception}
rlm@572	3563 \label{sec-5-3-6}
rlm@572	3564
rlm@572	3565 Proprioception is tied to each joint node -- nothing special must
rlm@572	3566 be done in a Blender model to enable proprioception other than
rlm@572	3567 creating joint nodes.
rlm@572	3568
rlm@572	3569 \subsubsection{Muscles}
rlm@572	3570 \label{sec-5-3-7}
rlm@572	3571
rlm@572	3572 Muscles are created by creating an empty node named \texttt{muscles} and
rlm@572	3573 then creating any number of empty child nodes to represent your
rlm@572	3574 creature's muscles.
rlm@572	3575
rlm@572	3576
rlm@572	3577 Muscle nodes should have the following metadata under the
rlm@572	3578 ``muscle'' label:
rlm@572	3579
rlm@572	3580 \begin{verbatim}
rlm@572	3581 <muscle-profile-file-name>
rlm@572	3582 \end{verbatim}
rlm@572	3583
rlm@572	3584 Muscles should also have a ``strength'' metadata entry describing
rlm@572	3585 the muscle's total strength at full activation.
rlm@572	3586
rlm@572	3587 Muscle profiles are simple images that contain the relative amount
rlm@572	3588 of muscle power in each simulated alpha motor neuron. The width of
rlm@572	3589 the image is the total size of the motor pool, and the redness of
rlm@572	3590 each neuron is the relative power of that motor pool.
rlm@572	3591
rlm@572	3592 While the profile image can have any dimensions, only the first
rlm@572	3593 line of pixels is used to define the muscle. Here is a sample
rlm@572	3594 muscle profile image that defines a human-like muscle.
rlm@572	3595
rlm@572	3596 \begin{figure}[htb]
rlm@572	3597 \centering
rlm@572	3598 \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572	3599 \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572	3600 \end{figure}
rlm@572	3601
rlm@572	3602 Muscles twist the nearest physical object about the muscle node's
rlm@572	3603 Z-axis. I recommend using the ``Single Arrow'' display mode for
rlm@572	3604 muscles and using the right hand rule to determine which way the
rlm@572	3605 muscle will twist. To make a segment that can twist in multiple
rlm@572	3606 directions, create multiple, differently aligned muscles.
rlm@572	3607
rlm@572	3608 \subsection{\texttt{CORTEX} API}
rlm@572	3609 \label{sec-5-4}
rlm@572	3610
rlm@572	3611 These are the some functions exposed by \texttt{CORTEX} for creating
rlm@572	3612 worlds and simulating creatures. These are in addition to
rlm@572	3613 jMonkeyEngine3's extensive library, which is documented elsewhere.
rlm@572	3614
rlm@572	3615 \subsubsection{Simulation}
rlm@572	3616 \label{sec-5-4-1}
rlm@572	3617 \begin{description}
rlm@572	3618 \item[{\texttt{(world root-node key-map setup-fn update-fn)}}] create
rlm@572	3619 a simulation.
rlm@572	3620 \begin{description}
rlm@572	3621 \item[{\emph{root-node} }] a \texttt{com.jme3.scene.Node} object which
rlm@572	3622 contains all of the objects that should be in the
rlm@572	3623 simulation.
rlm@572	3624
rlm@572	3625 \item[{\emph{key-map} }] a map from strings describing keys to
rlm@572	3626 functions that should be executed whenever that key is
rlm@572	3627 pressed. the functions should take a SimpleApplication
rlm@572	3628 object and a boolean value. The SimpleApplication is the
rlm@572	3629 current simulation that is running, and the boolean is true
rlm@572	3630 if the key is being pressed, and false if it is being
rlm@572	3631 released. As an example,
rlm@572	3632 \begin{verbatim}
rlm@572	3633 {"key-j" (fn [game value] (if value (println "key j pressed")))}
rlm@572	3634 \end{verbatim}
rlm@572	3635 is a valid key-map which will cause the simulation to print
rlm@572	3636 a message whenever the 'j' key on the keyboard is pressed.
rlm@572	3637
rlm@572	3638 \item[{\emph{setup-fn} }] a function that takes a \texttt{SimpleApplication}
rlm@572	3639 object. It is called once when initializing the simulation.
rlm@572	3640 Use it to create things like lights, change the gravity,
rlm@572	3641 initialize debug nodes, etc.
rlm@572	3642
rlm@572	3643 \item[{\emph{update-fn} }] this function takes a \texttt{SimpleApplication}
rlm@572	3644 object and a float and is called every frame of the
rlm@572	3645 simulation. The float tells how many seconds is has been
rlm@572	3646 since the last frame was rendered, according to whatever
rlm@572	3647 clock jme is currently using. The default is to use IsoTimer
rlm@572	3648 which will result in this value always being the same.
rlm@572	3649 \end{description}
rlm@572	3650
rlm@572	3651 \item[{\texttt{(position-camera world position rotation)}}] set the position
rlm@572	3652 of the simulation's main camera.
rlm@572	3653
rlm@572	3654 \item[{\texttt{(enable-debug world)}}] turn on debug wireframes for each
rlm@572	3655 simulated object.
rlm@572	3656
rlm@572	3657 \item[{\texttt{(set-gravity world gravity)}}] set the gravity of a running
rlm@572	3658 simulation.
rlm@572	3659
rlm@572	3660 \item[{\texttt{(box length width height \& \{options\})}}] create a box in the
rlm@572	3661 simulation. Options is a hash map specifying texture, mass,
rlm@572	3662 etc. Possible options are \texttt{:name}, \texttt{:color}, \texttt{:mass},
rlm@572	3663 \texttt{:friction}, \texttt{:texture}, \texttt{:material}, \texttt{:position},
rlm@572	3664 \texttt{:rotation}, \texttt{:shape}, and \texttt{:physical?}.
rlm@572	3665
rlm@572	3666 \item[{\texttt{(sphere radius \& \{options\})}}] create a sphere in the simulation.
rlm@572	3667 Options are the same as in \texttt{box}.
rlm@572	3668
rlm@572	3669 \item[{\texttt{(load-blender-model file-name)}}] create a node structure
rlm@572	3670 representing the model described in a Blender file.
rlm@572	3671
rlm@572	3672 \item[{\texttt{(light-up-everything world)}}] distribute a standard compliment
rlm@572	3673 of lights throughout the simulation. Should be adequate for most
rlm@572	3674 purposes.
rlm@572	3675
rlm@572	3676 \item[{\texttt{(node-seq node)}}] return a recursive list of the node's
rlm@572	3677 children.
rlm@572	3678
rlm@572	3679 \item[{\texttt{(nodify name children)}}] construct a node given a node-name and
rlm@572	3680 desired children.
rlm@572	3681
rlm@572	3682 \item[{\texttt{(add-element world element)}}] add an object to a running world
rlm@572	3683 simulation.
rlm@572	3684
rlm@572	3685 \item[{\texttt{(set-accuracy world accuracy)}}] change the accuracy of the
rlm@572	3686 world's physics simulator.
rlm@572	3687
rlm@572	3688 \item[{\texttt{(asset-manager)}}] get an \emph{AssetManager}, a jMonkeyEngine
rlm@572	3689 construct that is useful for loading textures and is required
rlm@572	3690 for smooth interaction with jMonkeyEngine library functions.
rlm@572	3691
rlm@572	3692 \item[{\texttt{(load-bullet)} }] unpack native libraries and initialize the
rlm@572	3693 bullet physics subsystem. This function is required before
rlm@572	3694 other world building functions are called.
rlm@572	3695 \end{description}
rlm@572	3696
rlm@572	3697 \subsubsection{Creature Manipulation / Import}
rlm@572	3698 \label{sec-5-4-2}
rlm@572	3699
rlm@572	3700 \begin{description}
rlm@572	3701 \item[{\texttt{(body! creature)}}] give the creature a physical body.
rlm@572	3702
rlm@572	3703 \item[{\texttt{(vision! creature)}}] give the creature a sense of vision.
rlm@572	3704 Returns a list of functions which will each, when called
rlm@572	3705 during a simulation, return the vision data for the channel of
rlm@572	3706 one of the eyes. The functions are ordered depending on the
rlm@572	3707 alphabetical order of the names of the eye nodes in the
rlm@572	3708 Blender file. The data returned by the functions is a vector
rlm@572	3709 containing the eye's \emph{topology}, a vector of coordinates, and
rlm@572	3710 the eye's \emph{data}, a vector of RGB values filtered by the eye's
rlm@572	3711 sensitivity.
rlm@572	3712
rlm@572	3713 \item[{\texttt{(hearing! creature)}}] give the creature a sense of hearing.
rlm@572	3714 Returns a list of functions, one for each ear, that when
rlm@572	3715 called will return a frame's worth of hearing data for that
rlm@572	3716 ear. The functions are ordered depending on the alphabetical
rlm@572	3717 order of the names of the ear nodes in the Blender file. The
rlm@572	3718 data returned by the functions is an array of PCM (pulse code
rlm@572	3719 modulated) wav data.
rlm@572	3720
rlm@572	3721 \item[{\texttt{(touch! creature)}}] give the creature a sense of touch. Returns
rlm@572	3722 a single function that must be called with the \emph{root node} of
rlm@572	3723 the world, and which will return a vector of \emph{touch-data}
rlm@572	3724 one entry for each touch sensitive component, each entry of
rlm@572	3725 which contains a \emph{topology} that specifies the distribution of
rlm@572	3726 touch sensors, and the \emph{data}, which is a vector of
rlm@572	3727 \texttt{[activation, length]} pairs for each touch hair.
rlm@572	3728
rlm@572	3729 \item[{\texttt{(proprioception! creature)}}] give the creature the sense of
rlm@572	3730 proprioception. Returns a list of functions, one for each
rlm@572	3731 joint, that when called during a running simulation will
rlm@572	3732 report the \texttt{[heading, pitch, roll]} of the joint.
rlm@572	3733
rlm@572	3734 \item[{\texttt{(movement! creature)}}] give the creature the power of movement.
rlm@572	3735 Creates a list of functions, one for each muscle, that when
rlm@572	3736 called with an integer, will set the recruitment of that
rlm@572	3737 muscle to that integer, and will report the current power
rlm@572	3738 being exerted by the muscle. Order of muscles is determined by
rlm@572	3739 the alphabetical sort order of the names of the muscle nodes.
rlm@572	3740 \end{description}
rlm@572	3741
rlm@572	3742 \subsubsection{Visualization/Debug}
rlm@572	3743 \label{sec-5-4-3}
rlm@572	3744
rlm@572	3745 \begin{description}
rlm@572	3746 \item[{\texttt{(view-vision)}}] create a function that when called with a list
rlm@572	3747 of visual data returned from the functions made by \texttt{vision!},
rlm@572	3748 will display that visual data on the screen.
rlm@572	3749
rlm@572	3750 \item[{\texttt{(view-hearing)}}] same as \texttt{view-vision} but for hearing.
rlm@572	3751
rlm@572	3752 \item[{\texttt{(view-touch)}}] same as \texttt{view-vision} but for touch.
rlm@572	3753
rlm@572	3754 \item[{\texttt{(view-proprioception)}}] same as \texttt{view-vision} but for
rlm@572	3755 proprioception.
rlm@572	3756
rlm@572	3757 \item[{\texttt{(view-movement)}}] same as \texttt{view-vision} but for muscles.
rlm@572	3758
rlm@572	3759 \item[{\texttt{(view anything)}}] \texttt{view} is a polymorphic function that allows
rlm@572	3760 you to inspect almost anything you could reasonably expect to
rlm@572	3761 be able to ``see'' in \texttt{CORTEX}.
rlm@572	3762
rlm@572	3763 \item[{\texttt{(text anything)}}] \texttt{text} is a polymorphic function that allows
rlm@572	3764 you to convert practically anything into a text string.
rlm@572	3765
rlm@572	3766 \item[{\texttt{(println-repl anything)}}] print messages to clojure's repl
rlm@572	3767 instead of the simulation's terminal window.
rlm@572	3768
rlm@572	3769 \item[{\texttt{(mega-import-jme3)}}] for experimenting at the REPL. This
rlm@572	3770 function will import all jMonkeyEngine3 classes for immediate
rlm@572	3771 use.
rlm@572	3772
rlm@572	3773 \item[{\texttt{(display-dilated-time world timer)}}] Shows the time as it is
rlm@572	3774 flowing in the simulation on a HUD display.
rlm@572	3775 \end{description}

Mercurial > cortex

annotate thesis/cortex.tex @ 572:202c6d19acad