annotate thesis/cortex.tex @ 573:ebdedb039cbb tip

add release.
author Robert McIntyre <rlm@mit.edu>
date Sun, 19 Apr 2015 04:01:53 -0700
parents 202c6d19acad
children
rev   line source
rlm@572 1
rlm@572 2 \section{Empathy $\backslash$ Embodiment: problem solving strategies}
rlm@572 3 \label{sec-1}
rlm@572 4
rlm@572 5 By the time you have read this thesis, you will understand a novel
rlm@572 6 approach to representing and recognizing physical actions using
rlm@572 7 embodiment and empathy. You will also see one way to efficiently
rlm@572 8 implement physical empathy for embodied creatures. Finally, you will
rlm@572 9 become familiar with \texttt{CORTEX}, a system for designing and simulating
rlm@572 10 creatures with rich senses, which I have designed as a library that
rlm@572 11 you can use in your own research. Note that I \emph{do not} process video
rlm@572 12 directly --- I start with knowledge of the positions of a creature's
rlm@572 13 body parts and work from there.
rlm@572 14
rlm@572 15 This is the core vision of my thesis: That one of the important ways
rlm@572 16 in which we understand others is by imagining ourselves in their
rlm@572 17 position and empathically feeling experiences relative to our own
rlm@572 18 bodies. By understanding events in terms of our own previous
rlm@572 19 corporeal experience, we greatly constrain the possibilities of what
rlm@572 20 would otherwise be an unwieldy exponential search. This extra
rlm@572 21 constraint can be the difference between easily understanding what
rlm@572 22 is happening in a video and being completely lost in a sea of
rlm@572 23 incomprehensible color and movement.
rlm@572 24
rlm@572 25 \subsection{The problem: recognizing actions is hard!}
rlm@572 26 \label{sec-1-1}
rlm@572 27
rlm@572 28 Examine figure \ref{cat-drink}. What is happening? As you, and
rlm@572 29 indeed very young children, can easily determine, this is an image
rlm@572 30 of drinking.
rlm@572 31
rlm@572 32 \begin{figure}[htb]
rlm@572 33 \centering
rlm@572 34 \includegraphics[width=7cm]{./images/cat-drinking.jpg}
rlm@572 35 \caption{\label{cat-drink}A cat drinking some water. Identifying this action is beyond the capabilities of existing computer vision systems.}
rlm@572 36 \end{figure}
rlm@572 37
rlm@572 38 Nevertheless, it is beyond the state of the art for a computer
rlm@572 39 vision program to describe what's happening in this image. Part of
rlm@572 40 the problem is that many computer vision systems focus on
rlm@572 41 pixel-level details or comparisons to example images (such as
rlm@572 42 \cite{volume-action-recognition}), but the 3D world is so variable
rlm@572 43 that it is hard to describe the world in terms of possible images.
rlm@572 44
rlm@572 45 In fact, the contents of a scene may have much less to do with
rlm@572 46 pixel probabilities than with recognizing various affordances:
rlm@572 47 things you can move, objects you can grasp, spaces that can be
rlm@572 48 filled . For example, what processes might enable you to see the
rlm@572 49 chair in figure \ref{hidden-chair}?
rlm@572 50
rlm@572 51 \begin{figure}[htb]
rlm@572 52 \centering
rlm@572 53 \includegraphics[width=10cm]{./images/fat-person-sitting-at-desk.jpg}
rlm@572 54 \caption{\label{hidden-chair}The chair in this image is quite obvious to humans, but it can't be found by any modern computer vision program.}
rlm@572 55 \end{figure}
rlm@572 56
rlm@572 57 Finally, how is it that you can easily tell the difference between
rlm@572 58 how the girl's \emph{muscles} are working in figure \ref{girl}?
rlm@572 59
rlm@572 60 \begin{figure}[htb]
rlm@572 61 \centering
rlm@572 62 \includegraphics[width=7cm]{./images/wall-push.png}
rlm@572 63 \caption{\label{girl}The mysterious ``common sense'' appears here as you are able to discern the difference in how the girl's arm muscles are activated between the two images. When you compare these two images, do you feel something in your own arm muscles?}
rlm@572 64 \end{figure}
rlm@572 65
rlm@572 66 Each of these examples tells us something about what might be going
rlm@572 67 on in our minds as we easily solve these recognition problems:
rlm@572 68
rlm@572 69 \begin{itemize}
rlm@572 70 \item The hidden chair shows us that we are strongly triggered by cues
rlm@572 71 relating to the position of human bodies, and that we can
rlm@572 72 determine the overall physical configuration of a human body even
rlm@572 73 if much of that body is occluded.
rlm@572 74
rlm@572 75 \item The picture of the girl pushing against the wall tells us that we
rlm@572 76 have common sense knowledge about the kinetics of our own bodies.
rlm@572 77 We know well how our muscles would have to work to maintain us in
rlm@572 78 most positions, and we can easily project this self-knowledge to
rlm@572 79 imagined positions triggered by images of the human body.
rlm@572 80
rlm@572 81 \item The cat tells us that imagination of some kind plays an important
rlm@572 82 role in understanding actions. The question is: Can we be more
rlm@572 83 precise about what sort of imagination is required to understand
rlm@572 84 these actions?
rlm@572 85 \end{itemize}
rlm@572 86
rlm@572 87 \subsection{A step forward: the sensorimotor-centered approach}
rlm@572 88 \label{sec-1-2}
rlm@572 89
rlm@572 90 In this thesis, I explore the idea that our knowledge of our own
rlm@572 91 bodies, combined with our own rich senses, enables us to recognize
rlm@572 92 the actions of others.
rlm@572 93
rlm@572 94 For example, I think humans are able to label the cat video as
rlm@572 95 ``drinking'' because they imagine \emph{themselves} as the cat, and
rlm@572 96 imagine putting their face up against a stream of water and
rlm@572 97 sticking out their tongue. In that imagined world, they can feel
rlm@572 98 the cool water hitting their tongue, and feel the water entering
rlm@572 99 their body, and are able to recognize that \emph{feeling} as drinking.
rlm@572 100 So, the label of the action is not really in the pixels of the
rlm@572 101 image, but is found clearly in a simulation / recollection inspired
rlm@572 102 by those pixels. An imaginative system, having been trained on
rlm@572 103 drinking and non-drinking examples and learning that the most
rlm@572 104 important component of drinking is the feeling of water flowing
rlm@572 105 down one's throat, would analyze a video of a cat drinking in the
rlm@572 106 following manner:
rlm@572 107
rlm@572 108 \begin{enumerate}
rlm@572 109 \item Create a physical model of the video by putting a ``fuzzy''
rlm@572 110 model of its own body in place of the cat. Possibly also create
rlm@572 111 a simulation of the stream of water.
rlm@572 112
rlm@572 113 \item Play out this simulated scene and generate imagined sensory
rlm@572 114 experience. This will include relevant muscle contractions, a
rlm@572 115 close up view of the stream from the cat's perspective, and most
rlm@572 116 importantly, the imagined feeling of water entering the mouth.
rlm@572 117 The imagined sensory experience can come from a simulation of
rlm@572 118 the event, but can also be pattern-matched from previous,
rlm@572 119 similar embodied experience.
rlm@572 120
rlm@572 121 \item The action is now easily identified as drinking by the sense of
rlm@572 122 taste alone. The other senses (such as the tongue moving in and
rlm@572 123 out) help to give plausibility to the simulated action. Note that
rlm@572 124 the sense of vision, while critical in creating the simulation,
rlm@572 125 is not critical for identifying the action from the simulation.
rlm@572 126 \end{enumerate}
rlm@572 127
rlm@572 128 For the chair examples, the process is even easier:
rlm@572 129
rlm@572 130 \begin{enumerate}
rlm@572 131 \item Align a model of your body to the person in the image.
rlm@572 132
rlm@572 133 \item Generate proprioceptive sensory data from this alignment.
rlm@572 134
rlm@572 135 \item Use the imagined proprioceptive data as a key to lookup related
rlm@572 136 sensory experience associated with that particular proprioceptive
rlm@572 137 feeling.
rlm@572 138
rlm@572 139 \item Retrieve the feeling of your bottom resting on a surface, your
rlm@572 140 knees bent, and your leg muscles relaxed.
rlm@572 141
rlm@572 142 \item This sensory information is consistent with your \texttt{sitting?}
rlm@572 143 sensory predicate, so you (and the entity in the image) must be
rlm@572 144 sitting.
rlm@572 145
rlm@572 146 \item There must be a chair-like object since you are sitting.
rlm@572 147 \end{enumerate}
rlm@572 148
rlm@572 149 Empathy offers yet another alternative to the age-old AI
rlm@572 150 representation question: ``What is a chair?'' --- A chair is the
rlm@572 151 feeling of sitting!
rlm@572 152
rlm@572 153 One powerful advantage of empathic problem solving is that it
rlm@572 154 factors the action recognition problem into two easier problems. To
rlm@572 155 use empathy, you need an \emph{aligner}, which takes the video and a
rlm@572 156 model of your body, and aligns the model with the video. Then, you
rlm@572 157 need a \emph{recognizer}, which uses the aligned model to interpret the
rlm@572 158 action. The power in this method lies in the fact that you describe
rlm@572 159 all actions from a body-centered viewpoint. You are less tied to
rlm@572 160 the particulars of any visual representation of the actions. If you
rlm@572 161 teach the system what ``running'' is, and you have a good enough
rlm@572 162 aligner, the system will from then on be able to recognize running
rlm@572 163 from any point of view -- even strange points of view like above or
rlm@572 164 underneath the runner. This is in contrast to action recognition
rlm@572 165 schemes that try to identify actions using a non-embodied approach.
rlm@572 166 If these systems learn about running as viewed from the side, they
rlm@572 167 will not automatically be able to recognize running from any other
rlm@572 168 viewpoint.
rlm@572 169
rlm@572 170 Another powerful advantage is that using the language of multiple
rlm@572 171 body-centered rich senses to describe body-centered actions offers
rlm@572 172 a massive boost in descriptive capability. Consider how difficult
rlm@572 173 it would be to compose a set of HOG (Histogram of Oriented
rlm@572 174 Gradients) filters to describe the action of a simple worm-creature
rlm@572 175 ``curling'' so that its head touches its tail, and then behold the
rlm@572 176 simplicity of describing thus action in a language designed for the
rlm@572 177 task (listing \ref{grand-circle-intro}):
rlm@572 178
rlm@572 179 \begin{listing}
rlm@572 180 \begin{verbatim}
rlm@572 181 (defn grand-circle?
rlm@572 182 "Does the worm form a majestic circle (one end touching the other)?"
rlm@572 183 [experiences]
rlm@572 184 (and (curled? experiences)
rlm@572 185 (let [worm-touch (:touch (peek experiences))
rlm@572 186 tail-touch (worm-touch 0)
rlm@572 187 head-touch (worm-touch 4)]
rlm@572 188 (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
rlm@572 189 (< 0.2 (contact worm-segment-top-tip head-touch))))))
rlm@572 190 \end{verbatim}
rlm@572 191 \caption{\label{grand-circle-intro}Body-centered actions are best expressed in a body-centered language. This code detects when the worm has curled into a full circle. Imagine how you would replicate this functionality using low-level pixel features such as HOG filters!}
rlm@572 192 \end{listing}
rlm@572 193
rlm@572 194 \subsection{\texttt{EMPATH} recognizes actions using empathy}
rlm@572 195 \label{sec-1-3}
rlm@572 196
rlm@572 197 Exploring these ideas further demands a concrete implementation, so
rlm@572 198 first, I built a system for constructing virtual creatures with
rlm@572 199 physiologically plausible sensorimotor systems and detailed
rlm@572 200 environments. The result is \texttt{CORTEX}, which I describe in chapter
rlm@572 201 \ref{sec-2}.
rlm@572 202
rlm@572 203 Next, I wrote routines which enabled a simple worm-like creature to
rlm@572 204 infer the actions of a second worm-like creature, using only its
rlm@572 205 own prior sensorimotor experiences and knowledge of the second
rlm@572 206 worm's joint positions. This program, \texttt{EMPATH}, is described in
rlm@572 207 chapter \ref{sec-3}. It's main components are:
rlm@572 208
rlm@572 209 \begin{description}
rlm@572 210 \item[{Embodied Action Definitions}] Many otherwise complicated actions
rlm@572 211 are easily described in the language of a full suite of
rlm@572 212 body-centered, rich senses and experiences. For example,
rlm@572 213 drinking is the feeling of water flowing down your throat, and
rlm@572 214 cooling your insides. It's often accompanied by bringing your
rlm@572 215 hand close to your face, or bringing your face close to water.
rlm@572 216 Sitting down is the feeling of bending your knees, activating
rlm@572 217 your quadriceps, then feeling a surface with your bottom and
rlm@572 218 relaxing your legs. These body-centered action descriptions
rlm@572 219 can be either learned or hard coded.
rlm@572 220
rlm@572 221 \item[{Guided Play }] The creature moves around and experiences the
rlm@572 222 world through its unique perspective. As the creature moves,
rlm@572 223 it gathers experiences that satisfy the embodied action
rlm@572 224 definitions.
rlm@572 225
rlm@572 226 \item[{Posture Imitation}] When trying to interpret a video or image,
rlm@572 227 the creature takes a model of itself and aligns it with
rlm@572 228 whatever it sees. This alignment might even cross species, as
rlm@572 229 when humans try to align themselves with things like ponies,
rlm@572 230 dogs, or other humans with a different body type.
rlm@572 231
rlm@572 232 \item[{Empathy }] The alignment triggers associations with
rlm@572 233 sensory data from prior experiences. For example, the
rlm@572 234 alignment itself easily maps to proprioceptive data. Any
rlm@572 235 sounds or obvious skin contact in the video can to a lesser
rlm@572 236 extent trigger previous experience keyed to hearing or touch.
rlm@572 237 Segments of previous experiences gained from play are stitched
rlm@572 238 together to form a coherent and complete sensory portrait of
rlm@572 239 the scene.
rlm@572 240
rlm@572 241 \item[{Recognition}] With the scene described in terms of remembered
rlm@572 242 first person sensory events, the creature can now run its
rlm@572 243 action-definition programs (such as the one in listing
rlm@572 244 \ref{grand-circle-intro}) on this synthesized sensory data,
rlm@572 245 just as it would if it were actually experiencing the scene
rlm@572 246 first-hand. If previous experience has been accurately
rlm@572 247 retrieved, and if it is analogous enough to the scene, then
rlm@572 248 the creature will correctly identify the action in the scene.
rlm@572 249 \end{description}
rlm@572 250
rlm@572 251 My program \texttt{EMPATH} uses this empathic problem solving technique
rlm@572 252 to interpret the actions of a simple, worm-like creature.
rlm@572 253
rlm@572 254 \begin{figure}[htb]
rlm@572 255 \centering
rlm@572 256 \includegraphics[width=15cm]{./images/worm-intro-white.png}
rlm@572 257 \caption{\label{worm-intro}The worm performs many actions during free play such as curling, wiggling, and resting.}
rlm@572 258 \end{figure}
rlm@572 259
rlm@572 260 \begin{figure}[htb]
rlm@572 261 \centering
rlm@572 262 \includegraphics[width=15cm]{./images/worm-poses.png}
rlm@572 263 \caption{\label{worm-recognition-intro}\texttt{EMPATH} recognized and classified each of these poses by inferring the complete sensory experience from proprioceptive data.}
rlm@572 264 \end{figure}
rlm@572 265
rlm@572 266 \subsubsection{Main Results}
rlm@572 267 \label{sec-1-3-1}
rlm@572 268
rlm@572 269 \begin{itemize}
rlm@572 270 \item After one-shot supervised training, \texttt{EMPATH} was able to
rlm@572 271 recognize a wide variety of static poses and dynamic
rlm@572 272 actions---ranging from curling in a circle to wiggling with a
rlm@572 273 particular frequency --- with 95$\backslash$ accuracy.
rlm@572 274
rlm@572 275 \item These results were completely independent of viewing angle
rlm@572 276 because the underlying body-centered language fundamentally is
rlm@572 277 independent; once an action is learned, it can be recognized
rlm@572 278 equally well from any viewing angle.
rlm@572 279
rlm@572 280 \item \texttt{EMPATH} is surprisingly short; the sensorimotor-centered
rlm@572 281 language provided by \texttt{CORTEX} resulted in extremely economical
rlm@572 282 recognition routines --- about 500 lines in all --- suggesting
rlm@572 283 that such representations are very powerful, and often
rlm@572 284 indispensable for the types of recognition tasks considered here.
rlm@572 285
rlm@572 286 \item For expediency's sake, I relied on direct knowledge of joint
rlm@572 287 positions in this proof of concept. However, I believe that the
rlm@572 288 structure of \texttt{EMPATH} and \texttt{CORTEX} will make future work to
rlm@572 289 enable video analysis much easier than it would otherwise be.
rlm@572 290 \end{itemize}
rlm@572 291
rlm@572 292 \subsection{\texttt{EMPATH} is built on \texttt{CORTEX}, a creature builder.}
rlm@572 293 \label{sec-1-4}
rlm@572 294
rlm@572 295 I built \texttt{CORTEX} to be a general AI research platform for doing
rlm@572 296 experiments involving multiple rich senses and a wide variety and
rlm@572 297 number of creatures. I intend it to be useful as a library for many
rlm@572 298 more projects than just this thesis. \texttt{CORTEX} was necessary to meet
rlm@572 299 a need among AI researchers at CSAIL and beyond, which is that
rlm@572 300 people often will invent wonderful ideas that are best expressed in
rlm@572 301 the language of creatures and senses, but in order to explore those
rlm@572 302 ideas they must first build a platform in which they can create
rlm@572 303 simulated creatures with rich senses! There are many ideas that
rlm@572 304 would be simple to execute (such as \texttt{EMPATH} or Larson's
rlm@572 305 self-organizing maps (\cite{larson-symbols})), but attached to them
rlm@572 306 is the multi-month effort to make a good creature simulator. Often,
rlm@572 307 that initial investment of time proves to be too much, and the
rlm@572 308 project must make do with a lesser environment or be abandoned
rlm@572 309 entirely.
rlm@572 310
rlm@572 311 \texttt{CORTEX} is well suited as an environment for embodied AI research
rlm@572 312 for three reasons:
rlm@572 313
rlm@572 314 \begin{itemize}
rlm@572 315 \item You can design new creatures using Blender (\cite{blender}), a
rlm@572 316 popular, free 3D modeling program. Each sense can be specified
rlm@572 317 using special Blender nodes with biologically inspired
rlm@572 318 parameters. You need not write any code to create a creature, and
rlm@572 319 can use a wide library of pre-existing Blender models as a base
rlm@572 320 for your own creatures.
rlm@572 321
rlm@572 322 \item \texttt{CORTEX} implements a wide variety of senses: touch,
rlm@572 323 proprioception, vision, hearing, and muscle tension. Complicated
rlm@572 324 senses like touch and vision involve multiple sensory elements
rlm@572 325 embedded in a 2D surface. You have complete control over the
rlm@572 326 distribution of these sensor elements through the use of simple
rlm@572 327 image files. \texttt{CORTEX} implements more comprehensive hearing than
rlm@572 328 any other creature simulation system available.
rlm@572 329
rlm@572 330 \item \texttt{CORTEX} supports any number of creatures and any number of
rlm@572 331 senses. Time in \texttt{CORTEX} dilates so that the simulated creatures
rlm@572 332 always perceive a perfectly smooth flow of time, regardless of
rlm@572 333 the actual computational load.
rlm@572 334 \end{itemize}
rlm@572 335
rlm@572 336 \texttt{CORTEX} is built on top of \texttt{jMonkeyEngine3}
rlm@572 337 (\cite{jmonkeyengine}), which is a video game engine designed to
rlm@572 338 create cross-platform 3D desktop games. \texttt{CORTEX} is mainly written
rlm@572 339 in clojure, a dialect of \texttt{LISP} that runs on the Java Virtual
rlm@572 340 Machine (JVM). The API for creating and simulating creatures and
rlm@572 341 senses is entirely expressed in clojure, though many senses are
rlm@572 342 implemented at the layer of jMonkeyEngine or below. For example,
rlm@572 343 for the sense of hearing I use a layer of clojure code on top of a
rlm@572 344 layer of java JNI bindings that drive a layer of \texttt{C++} code which
rlm@572 345 implements a modified version of \texttt{OpenAL} to support multiple
rlm@572 346 listeners. \texttt{CORTEX} is the only simulation environment that I know
rlm@572 347 of that can support multiple entities that can each hear the world
rlm@572 348 from their own perspective. Other senses also require a small layer
rlm@572 349 of Java code. \texttt{CORTEX} also uses \texttt{bullet}, a physics simulator
rlm@572 350 written in \texttt{C}.
rlm@572 351
rlm@572 352 \begin{figure}[htb]
rlm@572 353 \centering
rlm@572 354 \includegraphics[width=12cm]{./images/blender-worm.png}
rlm@572 355 \caption{\label{worm-recognition-intro-2}Here is the worm from figure \ref{worm-intro} modeled in Blender, a free 3D-modeling program. Senses and joints are described using special nodes in Blender.}
rlm@572 356 \end{figure}
rlm@572 357
rlm@572 358 Here are some things I anticipate that \texttt{CORTEX} might be used for:
rlm@572 359
rlm@572 360 \begin{itemize}
rlm@572 361 \item exploring new ideas about sensory integration
rlm@572 362 \item distributed communication among swarm creatures
rlm@572 363 \item self-learning using free exploration,
rlm@572 364 \item evolutionary algorithms involving creature construction
rlm@572 365 \item exploration of exotic senses and effectors that are not possible
rlm@572 366 in the real world (such as telekinesis or a semantic sense)
rlm@572 367 \item imagination using subworlds
rlm@572 368 \end{itemize}
rlm@572 369
rlm@572 370 During one test with \texttt{CORTEX}, I created 3,000 creatures each with
rlm@572 371 its own independent senses and ran them all at only 1/80 real time.
rlm@572 372 In another test, I created a detailed model of my own hand,
rlm@572 373 equipped with a realistic distribution of touch (more sensitive at
rlm@572 374 the fingertips), as well as eyes and ears, and it ran at around 1/4
rlm@572 375 real time.
rlm@572 376
rlm@572 377 \begin{sidewaysfigure}
rlm@572 378 \includegraphics[width=8.5in]{images/full-hand.png}
rlm@572 379 \caption{
rlm@572 380 I modeled my own right hand in Blender and rigged it with all the
rlm@572 381 senses that {\tt CORTEX} supports. My simulated hand has a
rlm@572 382 biologically inspired distribution of touch sensors. The senses are
rlm@572 383 displayed on the right (the red/black squares are raw sensory output),
rlm@572 384 and the simulation is displayed on the
rlm@572 385 left. Notice that my hand is curling its fingers, that it can see
rlm@572 386 its own finger from the eye in its palm, and that it can feel its
rlm@572 387 own thumb touching its palm.}
rlm@572 388 \end{sidewaysfigure}
rlm@572 389
rlm@572 390 \section{Designing \texttt{CORTEX}}
rlm@572 391 \label{sec-2}
rlm@572 392
rlm@572 393 In this chapter, I outline the design decisions that went into
rlm@572 394 making \texttt{CORTEX}, along with some details about its implementation.
rlm@572 395 (A practical guide to getting started with \texttt{CORTEX}, which skips
rlm@572 396 over the history and implementation details presented here, is
rlm@572 397 provided in an appendix at the end of this thesis.)
rlm@572 398
rlm@572 399 Throughout this project, I intended for \texttt{CORTEX} to be flexible and
rlm@572 400 extensible enough to be useful for other researchers who want to
rlm@572 401 test ideas of their own. To this end, wherever I have had to make
rlm@572 402 architectural choices about \texttt{CORTEX}, I have chosen to give as much
rlm@572 403 freedom to the user as possible, so that \texttt{CORTEX} may be used for
rlm@572 404 things I have not foreseen.
rlm@572 405
rlm@572 406 \subsection{Building in simulation versus reality}
rlm@572 407 \label{sec-2-1}
rlm@572 408 The most important architectural decision of all is the choice to
rlm@572 409 use a computer-simulated environment in the first place! The world
rlm@572 410 is a vast and rich place, and for now simulations are a very poor
rlm@572 411 reflection of its complexity. It may be that there is a significant
rlm@572 412 qualitative difference between dealing with senses in the real
rlm@572 413 world and dealing with pale facsimiles of them in a simulation
rlm@572 414 (\cite{brooks-representation}). What are the advantages and
rlm@572 415 disadvantages of a simulation vs. reality?
rlm@572 416
rlm@572 417 \subsubsection{Simulation}
rlm@572 418 \label{sec-2-1-1}
rlm@572 419
rlm@572 420 The advantages of virtual reality are that when everything is a
rlm@572 421 simulation, experiments in that simulation are absolutely
rlm@572 422 reproducible. It's also easier to change the creature and
rlm@572 423 environment to explore new situations and different sensory
rlm@572 424 combinations.
rlm@572 425
rlm@572 426 If the world is to be simulated on a computer, then not only do
rlm@572 427 you have to worry about whether the creature's senses are rich
rlm@572 428 enough to learn from the world, but whether the world itself is
rlm@572 429 rendered with enough detail and realism to give enough working
rlm@572 430 material to the creature's senses. To name just a few
rlm@572 431 difficulties facing modern physics simulators: destructibility of
rlm@572 432 the environment, simulation of water/other fluids, large areas,
rlm@572 433 nonrigid bodies, lots of objects, smoke. I don't know of any
rlm@572 434 computer simulation that would allow a creature to take a rock
rlm@572 435 and grind it into fine dust, then use that dust to make a clay
rlm@572 436 sculpture, at least not without spending years calculating the
rlm@572 437 interactions of every single small grain of dust. Maybe a
rlm@572 438 simulated world with today's limitations doesn't provide enough
rlm@572 439 richness for real intelligence to evolve.
rlm@572 440
rlm@572 441 \subsubsection{Reality}
rlm@572 442 \label{sec-2-1-2}
rlm@572 443
rlm@572 444 The other approach for playing with senses is to hook your
rlm@572 445 software up to real cameras, microphones, robots, etc., and let it
rlm@572 446 loose in the real world. This has the advantage of eliminating
rlm@572 447 concerns about simulating the world at the expense of increasing
rlm@572 448 the complexity of implementing the senses. Instead of just
rlm@572 449 grabbing the current rendered frame for processing, you have to
rlm@572 450 use an actual camera with real lenses and interact with photons to
rlm@572 451 get an image. It is much harder to change the creature, which is
rlm@572 452 now partly a physical robot of some sort, since doing so involves
rlm@572 453 changing things around in the real world instead of modifying
rlm@572 454 lines of code. While the real world is very rich and definitely
rlm@572 455 provides enough stimulation for intelligence to develop (as
rlm@572 456 evidenced by our own existence), it is also uncontrollable in the
rlm@572 457 sense that a particular situation cannot be recreated perfectly or
rlm@572 458 saved for later use. It is harder to conduct Science because it is
rlm@572 459 harder to repeat an experiment. The worst thing about using the
rlm@572 460 real world instead of a simulation is the matter of time. Instead
rlm@572 461 of simulated time you get the constant and unstoppable flow of
rlm@572 462 real time. This severely limits the sorts of software you can use
rlm@572 463 to program an AI, because all sense inputs must be handled in real
rlm@572 464 time. Complicated ideas may have to be implemented in hardware or
rlm@572 465 may simply be impossible given the current speed of our
rlm@572 466 processors. Contrast this with a simulation, in which the flow of
rlm@572 467 time in the simulated world can be slowed down to accommodate the
rlm@572 468 limitations of the creature's programming. In terms of cost, doing
rlm@572 469 everything in software is far cheaper than building custom
rlm@572 470 real-time hardware. All you need is a laptop and some patience.
rlm@572 471
rlm@572 472 \subsection{Simulated time enables rapid prototyping $\backslash$ simple programs}
rlm@572 473 \label{sec-2-2}
rlm@572 474
rlm@572 475 I envision \texttt{CORTEX} being used to support rapid prototyping and
rlm@572 476 iteration of ideas. Even if I could put together a well constructed
rlm@572 477 kit for creating robots, it would still not be enough because of
rlm@572 478 the scourge of real-time processing. Anyone who wants to test their
rlm@572 479 ideas in the real world must always worry about getting their
rlm@572 480 algorithms to run fast enough to process information in real time.
rlm@572 481 The need for real time processing only increases if multiple senses
rlm@572 482 are involved. In the extreme case, even simple algorithms will have
rlm@572 483 to be accelerated by ASIC chips or FPGAs, turning what would
rlm@572 484 otherwise be a few lines of code and a 10x speed penalty into a
rlm@572 485 multi-month ordeal. For this reason, \texttt{CORTEX} supports
rlm@572 486 \emph{time-dilation}, which scales back the framerate of the simulation
rlm@572 487 in proportion to the amount of processing each frame. From the
rlm@572 488 perspective of the creatures inside the simulation, time always
rlm@572 489 appears to flow at a constant rate, regardless of how complicated
rlm@572 490 the environment becomes or how many creatures are in the
rlm@572 491 simulation. The cost is that \texttt{CORTEX} can sometimes run slower than
rlm@572 492 real time. Time dilation works both ways, however --- simulations
rlm@572 493 of very simple creatures in \texttt{CORTEX} generally run at 40x real-time
rlm@572 494 on my machine!
rlm@572 495
rlm@572 496 \subsection{All sense organs are two-dimensional surfaces}
rlm@572 497 \label{sec-2-3}
rlm@572 498
rlm@572 499 If \texttt{CORTEX} is to support a wide variety of senses, it would help
rlm@572 500 to have a better understanding of what a sense actually is! While
rlm@572 501 vision, touch, and hearing all seem like they are quite different
rlm@572 502 things, I was surprised to learn during the course of this thesis
rlm@572 503 that they (and all physical senses) can be expressed as exactly the
rlm@572 504 same mathematical object!
rlm@572 505
rlm@572 506 Human beings are three-dimensional objects, and the nerves that
rlm@572 507 transmit data from our various sense organs to our brain are
rlm@572 508 essentially one-dimensional. This leaves up to two dimensions in
rlm@572 509 which our sensory information may flow. For example, imagine your
rlm@572 510 skin: it is a two-dimensional surface around a three-dimensional
rlm@572 511 object (your body). It has discrete touch sensors embedded at
rlm@572 512 various points, and the density of these sensors corresponds to the
rlm@572 513 sensitivity of that region of skin. Each touch sensor connects to a
rlm@572 514 nerve, all of which eventually are bundled together as they travel
rlm@572 515 up the spinal cord to the brain. Intersect the spinal nerves with a
rlm@572 516 guillotining plane and you will see all of the sensory data of the
rlm@572 517 skin revealed in a roughly circular two-dimensional image which is
rlm@572 518 the cross section of the spinal cord. Points on this image that are
rlm@572 519 close together in this circle represent touch sensors that are
rlm@572 520 \emph{probably} close together on the skin, although there is of course
rlm@572 521 some cutting and rearrangement that has to be done to transfer the
rlm@572 522 complicated surface of the skin onto a two dimensional image.
rlm@572 523
rlm@572 524 Most human senses consist of many discrete sensors of various
rlm@572 525 properties distributed along a surface at various densities. For
rlm@572 526 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
rlm@572 527 disks, and Ruffini's endings (\cite{textbook901}), which detect
rlm@572 528 pressure and vibration of various intensities. For ears, it is the
rlm@572 529 stereocilia distributed along the basilar membrane inside the
rlm@572 530 cochlea; each one is sensitive to a slightly different frequency of
rlm@572 531 sound. For eyes, it is rods and cones distributed along the surface
rlm@572 532 of the retina. In each case, we can describe the sense with a
rlm@572 533 surface and a distribution of sensors along that surface.
rlm@572 534
rlm@572 535 In fact, almost every human sense can be effectively described in
rlm@572 536 terms of a surface containing embedded sensors. If the sense had
rlm@572 537 any more dimensions, then there wouldn't be enough room in the
rlm@572 538 spinal cord to transmit the information!
rlm@572 539
rlm@572 540 Therefore, \texttt{CORTEX} must support the ability to create objects and
rlm@572 541 then be able to ``paint'' points along their surfaces to describe
rlm@572 542 each sense.
rlm@572 543
rlm@572 544 Fortunately this idea is already a well known computer graphics
rlm@572 545 technique called \emph{UV-mapping}. In UV-mapping, the three-dimensional
rlm@572 546 surface of a model is cut and smooshed until it fits on a
rlm@572 547 two-dimensional image. You paint whatever you want on that image,
rlm@572 548 and when the three-dimensional shape is rendered in a game the
rlm@572 549 smooshing and cutting is reversed and the image appears on the
rlm@572 550 three-dimensional object.
rlm@572 551
rlm@572 552 To make a sense, interpret the UV-image as describing the
rlm@572 553 distribution of that senses' sensors. To get different types of
rlm@572 554 sensors, you can either use a different color for each type of
rlm@572 555 sensor, or use multiple UV-maps, each labeled with that sensor
rlm@572 556 type. I generally use a white pixel to mean the presence of a
rlm@572 557 sensor and a black pixel to mean the absence of a sensor, and use
rlm@572 558 one UV-map for each sensor-type within a given sense.
rlm@572 559
rlm@572 560 \begin{figure}[htb]
rlm@572 561 \centering
rlm@572 562 \includegraphics[width=10cm]{./images/finger-UV.png}
rlm@572 563 \caption{\label{finger-UV}The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.}
rlm@572 564 \end{figure}
rlm@572 565
rlm@572 566 \begin{figure}[htb]
rlm@572 567 \centering
rlm@572 568 \includegraphics[width=10cm]{./images/finger-1.png}
rlm@572 569 \caption{\label{finger-side-view}Ventral side of the UV-mapped finger. Note the density of touch sensors at the tip.}
rlm@572 570 \end{figure}
rlm@572 571
rlm@572 572 \subsection{Video game engines provide ready-made physics and shading}
rlm@572 573 \label{sec-2-4}
rlm@572 574
rlm@572 575 I did not need to write my own physics simulation code or shader to
rlm@572 576 build \texttt{CORTEX}. Doing so would lead to a system that is impossible
rlm@572 577 for anyone but myself to use anyway. Instead, I use a video game
rlm@572 578 engine as a base and modify it to accommodate the additional needs
rlm@572 579 of \texttt{CORTEX}. Video game engines are an ideal starting point to
rlm@572 580 build \texttt{CORTEX}, because they are not far from being creature
rlm@572 581 building systems themselves.
rlm@572 582
rlm@572 583 First off, general purpose video game engines come with a physics
rlm@572 584 engine and lighting / sound system. The physics system provides
rlm@572 585 tools that can be co-opted to serve as touch, proprioception, and
rlm@572 586 muscles. Because some games support split screen views, a good
rlm@572 587 video game engine will allow you to efficiently create multiple
rlm@572 588 cameras in the simulated world that can be used as eyes. Video game
rlm@572 589 systems offer integrated asset management for things like textures
rlm@572 590 and creature models, providing an avenue for defining creatures.
rlm@572 591 They also understand UV-mapping, because this technique is used to
rlm@572 592 apply a texture to a model. Finally, because video game engines
rlm@572 593 support a large number of developers, as long as \texttt{CORTEX} doesn't
rlm@572 594 stray too far from the base system, other researchers can turn to
rlm@572 595 this community for help when doing their research.
rlm@572 596
rlm@572 597 \subsection{\texttt{CORTEX} is based on jMonkeyEngine3}
rlm@572 598 \label{sec-2-5}
rlm@572 599
rlm@572 600 While preparing to build \texttt{CORTEX} I studied several video game
rlm@572 601 engines to see which would best serve as a base. The top contenders
rlm@572 602 were:
rlm@572 603
rlm@572 604 \begin{description}
rlm@572 605 \item[{\href{http://www.idsoftware.com}{Quake II}/\href{http://www.bytonic.de/html/jake2.html}{Jake2}}] The Quake II engine was designed by ID software
rlm@572 606 in 1997. All the source code was released by ID software into
rlm@572 607 the Public Domain several years ago, and as a result it has
rlm@572 608 been ported to many different languages. This engine was
rlm@572 609 famous for its advanced use of realistic shading and it had
rlm@572 610 decent and fast physics simulation. The main advantage of the
rlm@572 611 Quake II engine is its simplicity, but I ultimately rejected
rlm@572 612 it because the engine is too tied to the concept of a
rlm@572 613 first-person shooter game. One of the problems I had was that
rlm@572 614 there does not seem to be any easy way to attach multiple
rlm@572 615 cameras to a single character. There are also several physics
rlm@572 616 clipping issues that are corrected in a way that only applies
rlm@572 617 to the main character and do not apply to arbitrary objects.
rlm@572 618
rlm@572 619 \item[{\href{http://source.valvesoftware.com/}{Source Engine} }] The Source Engine evolved from the Quake II
rlm@572 620 and Quake I engines and is used by Valve in the Half-Life
rlm@572 621 series of games. The physics simulation in the Source Engine
rlm@572 622 is quite accurate and probably the best out of all the engines
rlm@572 623 I investigated. There is also an extensive community actively
rlm@572 624 working with the engine. However, applications that use the
rlm@572 625 Source Engine must be written in C++, the code is not open, it
rlm@572 626 only runs on Windows, and the tools that come with the SDK to
rlm@572 627 handle models and textures are complicated and awkward to use.
rlm@572 628
rlm@572 629 \item[{\href{http://jmonkeyengine.com/}{jMonkeyEngine3}}] jMonkeyEngine3 is a new library for creating
rlm@572 630 games in Java. It uses OpenGL to render to the screen and uses
rlm@572 631 screengraphs to avoid drawing things that do not appear on the
rlm@572 632 screen. It has an active community and several games in the
rlm@572 633 pipeline. The engine was not built to serve any particular
rlm@572 634 game but is instead meant to be used for any 3D game.
rlm@572 635 \end{description}
rlm@572 636
rlm@572 637 I chose jMonkeyEngine3 because it had the most features out of all
rlm@572 638 the free projects I looked at, and because I could then write my
rlm@572 639 code in clojure, an implementation of \texttt{LISP} that runs on the JVM.
rlm@572 640
rlm@572 641 \subsection{\texttt{CORTEX} uses Blender to create creature models}
rlm@572 642 \label{sec-2-6}
rlm@572 643
rlm@572 644 For the simple worm-like creatures I will use later on in this
rlm@572 645 thesis, I could define a simple API in \texttt{CORTEX} that would allow
rlm@572 646 one to create boxes, spheres, etc., and leave that API as the sole
rlm@572 647 way to create creatures. However, for \texttt{CORTEX} to truly be useful
rlm@572 648 for other projects, it needs a way to construct complicated
rlm@572 649 creatures. If possible, it would be nice to leverage work that has
rlm@572 650 already been done by the community of 3D modelers, or at least
rlm@572 651 enable people who are talented at modeling but not programming to
rlm@572 652 design \texttt{CORTEX} creatures.
rlm@572 653
rlm@572 654 Therefore I use Blender, a free 3D modeling program, as the main
rlm@572 655 way to create creatures in \texttt{CORTEX}. However, the creatures modeled
rlm@572 656 in Blender must also be simple to simulate in jMonkeyEngine3's game
rlm@572 657 engine, and must also be easy to rig with \texttt{CORTEX}'s senses. I
rlm@572 658 accomplish this with extensive use of Blender's ``empty nodes.''
rlm@572 659
rlm@572 660 Empty nodes have no mass, physical presence, or appearance, but
rlm@572 661 they can hold metadata and have names. I use a tree structure of
rlm@572 662 empty nodes to specify senses in the following manner:
rlm@572 663
rlm@572 664 \begin{itemize}
rlm@572 665 \item Create a single top-level empty node whose name is the name of
rlm@572 666 the sense.
rlm@572 667 \item Add empty nodes which each contain meta-data relevant to the
rlm@572 668 sense, including a UV-map describing the number/distribution of
rlm@572 669 sensors if applicable.
rlm@572 670 \item Make each empty-node the child of the top-level node.
rlm@572 671 \end{itemize}
rlm@572 672
rlm@572 673 \begin{figure}[htb]
rlm@572 674 \centering
rlm@572 675 \includegraphics[width=10cm]{./images/empty-sense-nodes.png}
rlm@572 676 \caption{\label{sense-nodes}An example of annotating a creature model with empty nodes to describe the layout of senses. There are multiple empty nodes which each describe the position of muscles, ears, eyes, or joints.}
rlm@572 677 \end{figure}
rlm@572 678
rlm@572 679 \subsection{Bodies are composed of segments connected by joints}
rlm@572 680 \label{sec-2-7}
rlm@572 681
rlm@572 682 Blender is a general purpose animation tool, which has been used in
rlm@572 683 the past to create high quality movies such as Sintel
rlm@572 684 (\cite{blender}). Though Blender can model and render even
rlm@572 685 complicated things like water, it is crucial to keep models that
rlm@572 686 are meant to be simulated as creatures simple. \texttt{Bullet}, which
rlm@572 687 \texttt{CORTEX} uses though jMonkeyEngine3, is a rigid-body physics
rlm@572 688 system. This offers a compromise between the expressiveness of a
rlm@572 689 game level and the speed at which it can be simulated, and it means
rlm@572 690 that creatures should be naturally expressed as rigid components
rlm@572 691 held together by joint constraints.
rlm@572 692
rlm@572 693 But humans are more like a squishy bag wrapped around some hard
rlm@572 694 bones which define the overall shape. When we move, our skin bends
rlm@572 695 and stretches to accommodate the new positions of our bones.
rlm@572 696
rlm@572 697 One way to make bodies composed of rigid pieces connected by joints
rlm@572 698 \emph{seem} more human-like is to use an \emph{armature}, (or \emph{rigging})
rlm@572 699 system, which defines a overall ``body mesh'' and defines how the
rlm@572 700 mesh deforms as a function of the position of each ``bone'' which
rlm@572 701 is a standard rigid body. This technique is used extensively to
rlm@572 702 model humans and create realistic animations. It is not a good
rlm@572 703 technique for physical simulation because it is a lie -- the skin
rlm@572 704 is not a physical part of the simulation and does not interact with
rlm@572 705 any objects in the world or itself. Objects will pass right though
rlm@572 706 the skin until they come in contact with the underlying bone, which
rlm@572 707 is a physical object. Without simulating the skin, the sense of
rlm@572 708 touch has little meaning, and the creature's own vision will lie to
rlm@572 709 it about the true extent of its body. Simulating the skin as a
rlm@572 710 physical object requires some way to continuously update the
rlm@572 711 physical model of the skin along with the movement of the bones,
rlm@572 712 which is unacceptably slow compared to rigid body simulation.
rlm@572 713
rlm@572 714 Therefore, instead of using the human-like ``bony meatbag''
rlm@572 715 approach, I decided to base my body plans on multiple solid objects
rlm@572 716 that are connected by joints, inspired by the robot \texttt{EVE} from the
rlm@572 717 movie WALL-E.
rlm@572 718
rlm@572 719 \begin{figure}[htb]
rlm@572 720 \centering
rlm@572 721 \includegraphics[width=10cm]{./images/Eve.jpg}
rlm@572 722 \caption{\texttt{EVE} from the movie WALL-E. This body plan turns out to be much better suited to my purposes than a more human-like one.}
rlm@572 723 \end{figure}
rlm@572 724
rlm@572 725 \texttt{EVE}'s body is composed of several rigid components that are held
rlm@572 726 together by invisible joint constraints. This is what I mean by
rlm@572 727 \emph{eve-like}. The main reason that I use eve-like bodies is for
rlm@572 728 simulation efficiency, and so that there will be correspondence
rlm@572 729 between the AI's senses and the physical presence of its body. Each
rlm@572 730 individual section is simulated by a separate rigid body that
rlm@572 731 corresponds exactly with its visual representation and does not
rlm@572 732 change. Sections are connected by invisible joints that are well
rlm@572 733 supported in jMonkeyEngine3. Bullet, the physics backend for
rlm@572 734 jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
rlm@572 735 connected by joints. Just because sections are rigid does not mean
rlm@572 736 they have to stay as one piece forever; they can be dynamically
rlm@572 737 replaced with multiple sections to simulate splitting in two. This
rlm@572 738 could be used to simulate retractable claws or \texttt{EVE}'s hands, which
rlm@572 739 are able to coalesce into one object in the movie.
rlm@572 740
rlm@572 741 \subsubsection{Solidifying/Connecting a body}
rlm@572 742 \label{sec-2-7-1}
rlm@572 743
rlm@572 744 \texttt{CORTEX} creates a creature in two steps: first, it traverses the
rlm@572 745 nodes in the Blender file and creates physical representations for
rlm@572 746 any of them that have mass defined in their Blender meta-data.
rlm@572 747
rlm@572 748 \begin{listing}
rlm@572 749 \begin{verbatim}
rlm@572 750 (defn physical!
rlm@572 751 "Iterate through the nodes in creature and make them real physical
rlm@572 752 objects in the simulation."
rlm@572 753 [#^Node creature]
rlm@572 754 (dorun
rlm@572 755 (map
rlm@572 756 (fn [geom]
rlm@572 757 (let [physics-control
rlm@572 758 (RigidBodyControl.
rlm@572 759 (HullCollisionShape.
rlm@572 760 (.getMesh geom))
rlm@572 761 (if-let [mass (meta-data geom "mass")]
rlm@572 762 (float mass) (float 1)))]
rlm@572 763 (.addControl geom physics-control)))
rlm@572 764 (filter #(isa? (class %) Geometry )
rlm@572 765 (node-seq creature)))))
rlm@572 766 \end{verbatim}
rlm@572 767 \caption{\label{physical}Program for iterating through the nodes in a Blender file and generating physical jMonkeyEngine3 objects with mass and a matching physics shape.}
rlm@572 768 \end{listing}
rlm@572 769
rlm@572 770 The next step to making a proper body is to connect those pieces
rlm@572 771 together with joints. jMonkeyEngine has a large array of joints
rlm@572 772 available via \texttt{bullet}, such as Point2Point, Cone, Hinge, and a
rlm@572 773 generic Six Degree of Freedom joint, with or without spring
rlm@572 774 restitution.
rlm@572 775
rlm@572 776 Joints are treated a lot like proper senses, in that there is a
rlm@572 777 top-level empty node named ``joints'' whose children each
rlm@572 778 represent a joint.
rlm@572 779
rlm@572 780 \begin{figure}[htb]
rlm@572 781 \centering
rlm@572 782 \includegraphics[width=10cm]{./images/hand-screenshot1.png}
rlm@572 783 \caption{\label{blender-hand}View of the hand model in Blender showing the main ``joints'' node (highlighted in yellow) and its children which each represent a joint in the hand. Each joint node has metadata specifying what sort of joint it is.}
rlm@572 784 \end{figure}
rlm@572 785
rlm@572 786
rlm@572 787 \texttt{CORTEX}'s procedure for binding the creature together with joints
rlm@572 788 is as follows:
rlm@572 789
rlm@572 790 \begin{itemize}
rlm@572 791 \item Find the children of the ``joints'' node.
rlm@572 792 \item Determine the two spatials the joint is meant to connect.
rlm@572 793 \item Create the joint based on the meta-data of the empty node.
rlm@572 794 \end{itemize}
rlm@572 795
rlm@572 796 The higher order function \texttt{sense-nodes} from \texttt{cortex.sense}
rlm@572 797 simplifies finding the joints based on their parent ``joints''
rlm@572 798 node.
rlm@572 799
rlm@572 800 \begin{listing}
rlm@572 801 \begin{verbatim}
rlm@572 802 (defn sense-nodes
rlm@572 803 "For some senses there is a special empty Blender node whose
rlm@572 804 children are considered markers for an instance of that sense. This
rlm@572 805 function generates functions to find those children, given the name
rlm@572 806 of the special parent node."
rlm@572 807 [parent-name]
rlm@572 808 (fn [#^Node creature]
rlm@572 809 (if-let [sense-node (.getChild creature parent-name)]
rlm@572 810 (seq (.getChildren sense-node)) [])))
rlm@572 811
rlm@572 812 (def
rlm@572 813 ^{:doc "Return the children of the creature's \"joints\" node."
rlm@572 814 :arglists '([creature])}
rlm@572 815 joints
rlm@572 816 (sense-nodes "joints"))
rlm@572 817 \end{verbatim}
rlm@572 818 \caption{\label{get-empty-nodes}Retrieving the children empty nodes from a single named empty node is a common pattern in \texttt{CORTEX}. Further instances of this technique for the senses will be omitted}
rlm@572 819 \end{listing}
rlm@572 820
rlm@572 821 To find a joint's targets, \texttt{CORTEX} creates a small cube, centered
rlm@572 822 around the empty-node, and grows the cube exponentially until it
rlm@572 823 intersects two physical objects. The objects are ordered according
rlm@572 824 to the joint's rotation, with the first one being the object that
rlm@572 825 has more negative coordinates in the joint's reference frame.
rlm@572 826 Because the objects must be physical, the empty-node itself
rlm@572 827 escapes detection. Because the objects must be physical,
rlm@572 828 \texttt{joint-targets} must be called \emph{after} \texttt{physical!} is called.
rlm@572 829
rlm@572 830 \begin{listing}
rlm@572 831 \begin{verbatim}
rlm@572 832 (defn joint-targets
rlm@572 833 "Return the two closest two objects to the joint object, ordered
rlm@572 834 from bottom to top according to the joint's rotation."
rlm@572 835 [#^Node parts #^Node joint]
rlm@572 836 (loop [radius (float 0.01)]
rlm@572 837 (let [results (CollisionResults.)]
rlm@572 838 (.collideWith
rlm@572 839 parts
rlm@572 840 (BoundingBox. (.getWorldTranslation joint)
rlm@572 841 radius radius radius) results)
rlm@572 842 (let [targets
rlm@572 843 (distinct
rlm@572 844 (map #(.getGeometry %) results))]
rlm@572 845 (if (>= (count targets) 2)
rlm@572 846 (sort-by
rlm@572 847 #(let [joint-ref-frame-position
rlm@572 848 (jme-to-blender
rlm@572 849 (.mult
rlm@572 850 (.inverse (.getWorldRotation joint))
rlm@572 851 (.subtract (.getWorldTranslation %)
rlm@572 852 (.getWorldTranslation joint))))]
rlm@572 853 (.dot (Vector3f. 1 1 1) joint-ref-frame-position))
rlm@572 854 (take 2 targets))
rlm@572 855 (recur (float (* radius 2))))))))
rlm@572 856 \end{verbatim}
rlm@572 857 \caption{\label{joint-targets}Program to find the targets of a joint node by exponentially growth of a search cube.}
rlm@572 858 \end{listing}
rlm@572 859
rlm@572 860 Once \texttt{CORTEX} finds all joints and targets, it creates them using
rlm@572 861 a dispatch on the metadata of each joint node.
rlm@572 862
rlm@572 863 \begin{listing}
rlm@572 864 \begin{verbatim}
rlm@572 865 (defmulti joint-dispatch
rlm@572 866 "Translate Blender pseudo-joints into real JME joints."
rlm@572 867 (fn [constraints & _]
rlm@572 868 (:type constraints)))
rlm@572 869
rlm@572 870 (defmethod joint-dispatch :point
rlm@572 871 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572 872 (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
rlm@572 873 (.setLinearLowerLimit Vector3f/ZERO)
rlm@572 874 (.setLinearUpperLimit Vector3f/ZERO)))
rlm@572 875
rlm@572 876 (defmethod joint-dispatch :hinge
rlm@572 877 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572 878 (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
rlm@572 879 [limit-1 limit-2] (:limit constraints)
rlm@572 880 hinge-axis (.mult rotation (blender-to-jme axis))]
rlm@572 881 (doto (HingeJoint. control-a control-b pivot-a pivot-b
rlm@572 882 hinge-axis hinge-axis)
rlm@572 883 (.setLimit limit-1 limit-2))))
rlm@572 884
rlm@572 885 (defmethod joint-dispatch :cone
rlm@572 886 [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572 887 (let [limit-xz (:limit-xz constraints)
rlm@572 888 limit-xy (:limit-xy constraints)
rlm@572 889 twist (:twist constraints)]
rlm@572 890 (doto (ConeJoint. control-a control-b pivot-a pivot-b
rlm@572 891 rotation rotation)
rlm@572 892 (.setLimit (float limit-xz) (float limit-xy)
rlm@572 893 (float twist)))))
rlm@572 894 \end{verbatim}
rlm@572 895 \caption{\label{joint-dispatch}Program to dispatch on Blender metadata and create joints suitable for physical simulation.}
rlm@572 896 \end{listing}
rlm@572 897
rlm@572 898 All that is left for joints is to combine the above pieces into
rlm@572 899 something that can operate on the collection of nodes that a
rlm@572 900 Blender file represents.
rlm@572 901
rlm@572 902 \begin{listing}
rlm@572 903 \begin{verbatim}
rlm@572 904 (defn connect
rlm@572 905 "Create a joint between 'obj-a and 'obj-b at the location of
rlm@572 906 'joint. The type of joint is determined by the metadata on 'joint.
rlm@572 907
rlm@572 908 Here are some examples:
rlm@572 909 {:type :point}
rlm@572 910 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
rlm@572 911 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572 912
rlm@572 913 {:type :cone :limit-xz 0]
rlm@572 914 :limit-xy 0]
rlm@572 915 :twist 0]} (use XZY rotation mode in Blender!)"
rlm@572 916 [#^Node obj-a #^Node obj-b #^Node joint]
rlm@572 917 (let [control-a (.getControl obj-a RigidBodyControl)
rlm@572 918 control-b (.getControl obj-b RigidBodyControl)
rlm@572 919 joint-center (.getWorldTranslation joint)
rlm@572 920 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
rlm@572 921 pivot-a (world-to-local obj-a joint-center)
rlm@572 922 pivot-b (world-to-local obj-b joint-center)]
rlm@572 923 (if-let
rlm@572 924 [constraints (map-vals eval (read-string (meta-data joint "joint")))]
rlm@572 925 ;; A side-effect of creating a joint registers
rlm@572 926 ;; it with both physics objects which in turn
rlm@572 927 ;; will register the joint with the physics system
rlm@572 928 ;; when the simulation is started.
rlm@572 929 (joint-dispatch constraints
rlm@572 930 control-a control-b
rlm@572 931 pivot-a pivot-b
rlm@572 932 joint-rotation))))
rlm@572 933 \end{verbatim}
rlm@572 934 \caption{\label{connect}Program to completely create a joint given information from a Blender file.}
rlm@572 935 \end{listing}
rlm@572 936
rlm@572 937 In general, whenever \texttt{CORTEX} exposes a sense (or in this case
rlm@572 938 physicality), it provides a function of the type \texttt{sense!}, which
rlm@572 939 takes in a collection of nodes and augments it to support that
rlm@572 940 sense. The function returns any controls necessary to use that
rlm@572 941 sense. In this case \texttt{body!} creates a physical body and returns no
rlm@572 942 control functions.
rlm@572 943
rlm@572 944 \begin{listing}
rlm@572 945 \begin{verbatim}
rlm@572 946 (defn joints!
rlm@572 947 "Connect the solid parts of the creature with physical joints. The
rlm@572 948 joints are taken from the \"joints\" node in the creature."
rlm@572 949 [#^Node creature]
rlm@572 950 (dorun
rlm@572 951 (map
rlm@572 952 (fn [joint]
rlm@572 953 (let [[obj-a obj-b] (joint-targets creature joint)]
rlm@572 954 (connect obj-a obj-b joint)))
rlm@572 955 (joints creature))))
rlm@572 956 (defn body!
rlm@572 957 "Endow the creature with a physical body connected with joints. The
rlm@572 958 particulars of the joints and the masses of each body part are
rlm@572 959 determined in Blender."
rlm@572 960 [#^Node creature]
rlm@572 961 (physical! creature)
rlm@572 962 (joints! creature))
rlm@572 963 \end{verbatim}
rlm@572 964 \caption{\label{joints}Program to give joints to a creature.}
rlm@572 965 \end{listing}
rlm@572 966
rlm@572 967 All of the code you have just seen amounts to only 130 lines, yet
rlm@572 968 because it builds on top of Blender and jMonkeyEngine3, those few
rlm@572 969 lines pack quite a punch!
rlm@572 970
rlm@572 971 The hand from figure \ref{blender-hand}, which was modeled after
rlm@572 972 my own right hand, can now be given joints and simulated as a
rlm@572 973 creature.
rlm@572 974
rlm@572 975 \begin{figure}[htb]
rlm@572 976 \centering
rlm@572 977 \includegraphics[width=15cm]{./images/physical-hand.png}
rlm@572 978 \caption{\label{physical-hand}With the ability to create physical creatures from Blender, \texttt{CORTEX} gets one step closer to becoming a full creature simulation environment.}
rlm@572 979 \end{figure}
rlm@572 980
rlm@572 981 \subsection{Sight reuses standard video game components\ldots{}}
rlm@572 982 \label{sec-2-8}
rlm@572 983
rlm@572 984 Vision is one of the most important senses for humans, so I need to
rlm@572 985 build a simulated sense of vision for my AI. I will do this with
rlm@572 986 simulated eyes. Each eye can be independently moved and should see
rlm@572 987 its own version of the world depending on where it is.
rlm@572 988
rlm@572 989 Making these simulated eyes a reality is simple because
rlm@572 990 jMonkeyEngine already contains extensive support for multiple views
rlm@572 991 of the same 3D simulated world. The reason jMonkeyEngine has this
rlm@572 992 support is because the support is necessary to create games with
rlm@572 993 split-screen views. Multiple views are also used to create
rlm@572 994 efficient pseudo-reflections by rendering the scene from a certain
rlm@572 995 perspective and then projecting it back onto a surface in the 3D
rlm@572 996 world.
rlm@572 997
rlm@572 998 \begin{figure}[htb]
rlm@572 999 \centering
rlm@572 1000 \includegraphics[width=10cm]{./images/goldeneye-4-player.png}
rlm@572 1001 \caption{\label{goldeneye}jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.}
rlm@572 1002 \end{figure}
rlm@572 1003
rlm@572 1004 \subsubsection{A Brief Description of jMonkeyEngine's Rendering Pipeline}
rlm@572 1005 \label{sec-2-8-1}
rlm@572 1006
rlm@572 1007 jMonkeyEngine allows you to create a \texttt{ViewPort}, which represents a
rlm@572 1008 view of the simulated world. You can create as many of these as you
rlm@572 1009 want. Every frame, the \texttt{RenderManager} iterates through each
rlm@572 1010 \texttt{ViewPort}, rendering the scene in the GPU. For each \texttt{ViewPort} there
rlm@572 1011 is a \texttt{FrameBuffer} which represents the rendered image in the GPU.
rlm@572 1012
rlm@572 1013 \begin{figure}[htb]
rlm@572 1014 \centering
rlm@572 1015 \includegraphics[width=10cm]{./images/diagram_rendermanager2.png}
rlm@572 1016 \caption{\label{rendermanagers}\texttt{ViewPorts} are cameras in the world. During each frame, the \texttt{RenderManager} records a snapshot of what each view is currently seeing; these snapshots are \texttt{FrameBuffer} objects.}
rlm@572 1017 \end{figure}
rlm@572 1018
rlm@572 1019 Each \texttt{ViewPort} can have any number of attached \texttt{SceneProcessor}
rlm@572 1020 objects, which are called every time a new frame is rendered. A
rlm@572 1021 \texttt{SceneProcessor} receives its \texttt{ViewPort's} \texttt{FrameBuffer} and can do
rlm@572 1022 whatever it wants to the data. Often this consists of invoking GPU
rlm@572 1023 specific operations on the rendered image. The \texttt{SceneProcessor} can
rlm@572 1024 also copy the GPU image data to RAM and process it with the CPU.
rlm@572 1025
rlm@572 1026 \subsubsection{Appropriating Views for Vision}
rlm@572 1027 \label{sec-2-8-2}
rlm@572 1028
rlm@572 1029 Each eye in the simulated creature needs its own \texttt{ViewPort} so
rlm@572 1030 that it can see the world from its own perspective. To this
rlm@572 1031 \texttt{ViewPort}, I add a \texttt{SceneProcessor} that feeds the visual data to
rlm@572 1032 any arbitrary continuation function for further processing. That
rlm@572 1033 continuation function may perform both CPU and GPU operations on
rlm@572 1034 the data. To make this easy for the continuation function, the
rlm@572 1035 \texttt{SceneProcessor} maintains appropriately sized buffers in RAM to
rlm@572 1036 hold the data. It does not do any copying from the GPU to the CPU
rlm@572 1037 itself because it is a slow operation.
rlm@572 1038
rlm@572 1039 \begin{listing}
rlm@572 1040 \begin{verbatim}
rlm@572 1041 (defn vision-pipeline
rlm@572 1042 "Create a SceneProcessor object which wraps a vision processing
rlm@572 1043 continuation function. The continuation is a function that takes
rlm@572 1044 [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@572 1045 each of which has already been appropriately sized."
rlm@572 1046 [continuation]
rlm@572 1047 (let [byte-buffer (atom nil)
rlm@572 1048 renderer (atom nil)
rlm@572 1049 image (atom nil)]
rlm@572 1050 (proxy [SceneProcessor] []
rlm@572 1051 (initialize
rlm@572 1052 [renderManager viewPort]
rlm@572 1053 (let [cam (.getCamera viewPort)
rlm@572 1054 width (.getWidth cam)
rlm@572 1055 height (.getHeight cam)]
rlm@572 1056 (reset! renderer (.getRenderer renderManager))
rlm@572 1057 (reset! byte-buffer
rlm@572 1058 (BufferUtils/createByteBuffer
rlm@572 1059 (* width height 4)))
rlm@572 1060 (reset! image (BufferedImage.
rlm@572 1061 width height
rlm@572 1062 BufferedImage/TYPE_4BYTE_ABGR))))
rlm@572 1063 (isInitialized [] (not (nil? @byte-buffer)))
rlm@572 1064 (reshape [_ _ _])
rlm@572 1065 (preFrame [_])
rlm@572 1066 (postQueue [_])
rlm@572 1067 (postFrame
rlm@572 1068 [#^FrameBuffer fb]
rlm@572 1069 (.clear @byte-buffer)
rlm@572 1070 (continuation @renderer fb @byte-buffer @image))
rlm@572 1071 (cleanup []))))
rlm@572 1072 \end{verbatim}
rlm@572 1073 \caption{\label{pipeline-1}Function to make the rendered scene in jMonkeyEngine available for further processing.}
rlm@572 1074 \end{listing}
rlm@572 1075
rlm@572 1076 The continuation function given to \texttt{vision-pipeline} above will be
rlm@572 1077 given a \texttt{Renderer} and three containers for image data. The
rlm@572 1078 \texttt{FrameBuffer} references the GPU image data, but the pixel data
rlm@572 1079 can not be used directly on the CPU. The \texttt{ByteBuffer} and
rlm@572 1080 \texttt{BufferedImage} are initially "empty" but are sized to hold the
rlm@572 1081 data in the \texttt{FrameBuffer}. I call transferring the GPU image data
rlm@572 1082 to the CPU structures "mixing" the image data.
rlm@572 1083
rlm@572 1084 \subsubsection{Optical sensor arrays are described with images and referenced with metadata}
rlm@572 1085 \label{sec-2-8-3}
rlm@572 1086
rlm@572 1087 The vision pipeline described above handles the flow of rendered
rlm@572 1088 images. Now, \texttt{CORTEX} needs simulated eyes to serve as the source
rlm@572 1089 of these images.
rlm@572 1090
rlm@572 1091 An eye is described in Blender in the same way as a joint. They
rlm@572 1092 are zero dimensional empty objects with no geometry whose local
rlm@572 1093 coordinate system determines the orientation of the resulting eye.
rlm@572 1094 All eyes are children of a parent node named "eyes" just as all
rlm@572 1095 joints have a parent named "joints". An eye binds to the nearest
rlm@572 1096 physical object with \texttt{bind-sense}.
rlm@572 1097
rlm@572 1098 \begin{listing}
rlm@572 1099 \begin{verbatim}
rlm@572 1100 (defn add-eye!
rlm@572 1101 "Create a Camera centered on the current position of 'eye which
rlm@572 1102 follows the closest physical node in 'creature. The camera will
rlm@572 1103 point in the X direction and use the Z vector as up as determined
rlm@572 1104 by the rotation of these vectors in Blender coordinate space. Use
rlm@572 1105 XZY rotation for the node in Blender."
rlm@572 1106 [#^Node creature #^Spatial eye]
rlm@572 1107 (let [target (closest-node creature eye)
rlm@572 1108 [cam-width cam-height]
rlm@572 1109 ;;[640 480] ;; graphics card on laptop doesn't support
rlm@572 1110 ;; arbitrary dimensions.
rlm@572 1111 (eye-dimensions eye)
rlm@572 1112 cam (Camera. cam-width cam-height)
rlm@572 1113 rot (.getWorldRotation eye)]
rlm@572 1114 (.setLocation cam (.getWorldTranslation eye))
rlm@572 1115 (.lookAtDirection
rlm@572 1116 cam ; this part is not a mistake and
rlm@572 1117 (.mult rot Vector3f/UNIT_X) ; is consistent with using Z in
rlm@572 1118 (.mult rot Vector3f/UNIT_Y)) ; Blender as the UP vector.
rlm@572 1119 (.setFrustumPerspective
rlm@572 1120 cam (float 45)
rlm@572 1121 (float (/ (.getWidth cam) (.getHeight cam)))
rlm@572 1122 (float 1)
rlm@572 1123 (float 1000))
rlm@572 1124 (bind-sense target cam) cam))
rlm@572 1125 \end{verbatim}
rlm@572 1126 \caption{\label{add-eye}Here, the camera is created based on metadata on the eye-node and attached to the nearest physical object with \texttt{bind-sense}}
rlm@572 1127 \end{listing}
rlm@572 1128
rlm@572 1129 \subsubsection{Simulated Retina}
rlm@572 1130 \label{sec-2-8-4}
rlm@572 1131
rlm@572 1132 An eye is a surface (the retina) which contains many discrete
rlm@572 1133 sensors to detect light. These sensors can have different
rlm@572 1134 light-sensing properties. In humans, each discrete sensor is
rlm@572 1135 sensitive to red, blue, green, or gray. These different types of
rlm@572 1136 sensors can have different spatial distributions along the retina.
rlm@572 1137 In humans, there is a fovea in the center of the retina which has
rlm@572 1138 a very high density of color sensors, and a blind spot which has
rlm@572 1139 no sensors at all. Sensor density decreases in proportion to
rlm@572 1140 distance from the fovea.
rlm@572 1141
rlm@572 1142 I want to be able to model any retinal configuration, so my
rlm@572 1143 eye-nodes in Blender contain metadata pointing to images that
rlm@572 1144 describe the precise position of the individual sensors using
rlm@572 1145 white pixels. The meta-data also describes the precise sensitivity
rlm@572 1146 to light that the sensors described in the image have. An eye can
rlm@572 1147 contain any number of these images. For example, the metadata for
rlm@572 1148 an eye might look like this:
rlm@572 1149
rlm@572 1150 \begin{verbatim}
rlm@572 1151 {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@572 1152 \end{verbatim}
rlm@572 1153
rlm@572 1154 \begin{figure}[htb]
rlm@572 1155 \centering
rlm@572 1156 \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572 1157 \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572 1158 \end{figure}
rlm@572 1159
rlm@572 1160 Together, the number 0xFF0000 and the image above describe the
rlm@572 1161 placement of red-sensitive sensory elements.
rlm@572 1162
rlm@572 1163 Meta-data to very crudely approximate a human eye might be
rlm@572 1164 something like this:
rlm@572 1165
rlm@572 1166 \begin{verbatim}
rlm@572 1167 (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@572 1168 {0xFF0000 retinal-profile
rlm@572 1169 0x00FF00 retinal-profile
rlm@572 1170 0x0000FF retinal-profile
rlm@572 1171 0xFFFFFF retinal-profile})
rlm@572 1172 \end{verbatim}
rlm@572 1173
rlm@572 1174 The numbers that serve as keys in the map determine a sensor's
rlm@572 1175 relative sensitivity to the channels red, green, and blue. These
rlm@572 1176 sensitivity values are packed into an integer in the order
rlm@572 1177 \texttt{|\_|R|G|B|} in 8-bit fields. The RGB values of a pixel in the
rlm@572 1178 image are added together with these sensitivities as linear
rlm@572 1179 weights. Therefore, 0xFF0000 means sensitive to red only while
rlm@572 1180 0xFFFFFF means sensitive to all colors equally (gray).
rlm@572 1181
rlm@572 1182 \begin{listing}
rlm@572 1183 \begin{verbatim}
rlm@572 1184 (defn vision-kernel
rlm@572 1185 "Returns a list of functions, each of which will return a color
rlm@572 1186 channel's worth of visual information when called inside a running
rlm@572 1187 simulation."
rlm@572 1188 [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@572 1189 (let [retinal-map (retina-sensor-profile eye)
rlm@572 1190 camera (add-eye! creature eye)
rlm@572 1191 vision-image
rlm@572 1192 (atom
rlm@572 1193 (BufferedImage. (.getWidth camera)
rlm@572 1194 (.getHeight camera)
rlm@572 1195 BufferedImage/TYPE_BYTE_BINARY))
rlm@572 1196 register-eye!
rlm@572 1197 (runonce
rlm@572 1198 (fn [world]
rlm@572 1199 (add-camera!
rlm@572 1200 world camera
rlm@572 1201 (let [counter (atom 0)]
rlm@572 1202 (fn [r fb bb bi]
rlm@572 1203 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@572 1204 (reset! vision-image
rlm@572 1205 (BufferedImage! r fb bb bi))))))))]
rlm@572 1206 (vec
rlm@572 1207 (map
rlm@572 1208 (fn [[key image]]
rlm@572 1209 (let [whites (white-coordinates image)
rlm@572 1210 topology (vec (collapse whites))
rlm@572 1211 sensitivity (sensitivity-presets key key)]
rlm@572 1212 (attached-viewport.
rlm@572 1213 (fn [world]
rlm@572 1214 (register-eye! world)
rlm@572 1215 (vector
rlm@572 1216 topology
rlm@572 1217 (vec
rlm@572 1218 (for [[x y] whites]
rlm@572 1219 (pixel-sense
rlm@572 1220 sensitivity
rlm@572 1221 (.getRGB @vision-image x y))))))
rlm@572 1222 register-eye!)))
rlm@572 1223 retinal-map))))
rlm@572 1224 \end{verbatim}
rlm@572 1225 \caption{\label{vision-kernel}This is the core of vision in \texttt{CORTEX}. A given eye node is converted into a function that returns visual information from the simulation.}
rlm@572 1226 \end{listing}
rlm@572 1227
rlm@572 1228 Note that because each of the functions generated by
rlm@572 1229 \texttt{vision-kernel} shares the same \texttt{register-eye!} function, the eye
rlm@572 1230 will be registered only once the first time any of the functions
rlm@572 1231 from the list returned by \texttt{vision-kernel} is called. Each of the
rlm@572 1232 functions returned by \texttt{vision-kernel} also allows access to the
rlm@572 1233 \texttt{Viewport} through which it receives images.
rlm@572 1234
rlm@572 1235 All the hard work has been done; all that remains is to apply
rlm@572 1236 \texttt{vision-kernel} to each eye in the creature and gather the results
rlm@572 1237 into one list of functions.
rlm@572 1238
rlm@572 1239
rlm@572 1240 \begin{listing}
rlm@572 1241 \begin{verbatim}
rlm@572 1242 (defn vision!
rlm@572 1243 "Returns a list of functions, each of which returns visual sensory
rlm@572 1244 data when called inside a running simulation."
rlm@572 1245 [#^Node creature & {skip :skip :or {skip 0}}]
rlm@572 1246 (reduce
rlm@572 1247 concat
rlm@572 1248 (for [eye (eyes creature)]
rlm@572 1249 (vision-kernel creature eye))))
rlm@572 1250 \end{verbatim}
rlm@572 1251 \caption{\label{vision}With \texttt{vision!}, \texttt{CORTEX} is already a fine simulation environment for experimenting with different types of eyes.}
rlm@572 1252 \end{listing}
rlm@572 1253
rlm@572 1254 \begin{figure}[htb]
rlm@572 1255 \centering
rlm@572 1256 \includegraphics[width=13cm]{./images/worm-vision.png}
rlm@572 1257 \caption{\label{worm-vision-test.}Simulated vision with a test creature and the human-like eye approximation. Notice how each channel of the eye responds differently to the differently colored balls.}
rlm@572 1258 \end{figure}
rlm@572 1259
rlm@572 1260 The vision code is not much more complicated than the body code,
rlm@572 1261 and enables multiple further paths for simulated vision. For
rlm@572 1262 example, it is quite easy to create bifocal vision -- you just
rlm@572 1263 make two eyes next to each other in Blender! It is also possible
rlm@572 1264 to encode vision transforms in the retinal files. For example, the
rlm@572 1265 human like retina file in figure \ref{retina} approximates a
rlm@572 1266 log-polar transform.
rlm@572 1267
rlm@572 1268 This vision code has already been absorbed by the jMonkeyEngine
rlm@572 1269 community and is now (in modified form) part of a system for
rlm@572 1270 capturing in-game video to a file.
rlm@572 1271
rlm@572 1272 \subsection{\ldots{}but hearing must be built from scratch}
rlm@572 1273 \label{sec-2-9}
rlm@572 1274
rlm@572 1275 At the end of this chapter I will have simulated ears that work the
rlm@572 1276 same way as the simulated eyes in the last chapter. I will be able to
rlm@572 1277 place any number of ear-nodes in a Blender file, and they will bind to
rlm@572 1278 the closest physical object and follow it as it moves around. Each ear
rlm@572 1279 will provide access to the sound data it picks up between every frame.
rlm@572 1280
rlm@572 1281 Hearing is one of the more difficult senses to simulate, because there
rlm@572 1282 is less support for obtaining the actual sound data that is processed
rlm@572 1283 by jMonkeyEngine3. There is no "split-screen" support for rendering
rlm@572 1284 sound from different points of view, and there is no way to directly
rlm@572 1285 access the rendered sound data.
rlm@572 1286
rlm@572 1287 \texttt{CORTEX}'s hearing is unique because it does not have any
rlm@572 1288 limitations compared to other simulation environments. As far as I
rlm@572 1289 know, there is no other system that supports multiple listeners,
rlm@572 1290 and the sound demo at the end of this chapter is the first time
rlm@572 1291 it's been done in a video game environment.
rlm@572 1292
rlm@572 1293 \subsubsection{Brief Description of jMonkeyEngine's Sound System}
rlm@572 1294 \label{sec-2-9-1}
rlm@572 1295
rlm@572 1296 jMonkeyEngine's sound system works as follows:
rlm@572 1297
rlm@572 1298 \begin{itemize}
rlm@572 1299 \item jMonkeyEngine uses the \texttt{AppSettings} for the particular
rlm@572 1300 application to determine what sort of \texttt{AudioRenderer} should be
rlm@572 1301 used.
rlm@572 1302 \item Although some support is provided for multiple AudioRenderer
rlm@572 1303 backends, jMonkeyEngine at the time of this writing will either
rlm@572 1304 pick no \texttt{AudioRenderer} at all, or the \texttt{LwjglAudioRenderer}.
rlm@572 1305 \item jMonkeyEngine tries to figure out what sort of system you're
rlm@572 1306 running and extracts the appropriate native libraries.
rlm@572 1307 \item The \texttt{LwjglAudioRenderer} uses the \href{http://lwjgl.org/}{\texttt{LWJGL}} (LightWeight Java Game
rlm@572 1308 Library) bindings to interface with a C library called \href{http://kcat.strangesoft.net/openal.html}{\texttt{OpenAL}}
rlm@572 1309 \item \texttt{OpenAL} renders the 3D sound and feeds the rendered sound
rlm@572 1310 directly to any of various sound output devices with which it
rlm@572 1311 knows how to communicate.
rlm@572 1312 \end{itemize}
rlm@572 1313
rlm@572 1314 A consequence of this is that there's no way to access the actual
rlm@572 1315 sound data produced by \texttt{OpenAL}. Even worse, \texttt{OpenAL} only supports
rlm@572 1316 one \emph{listener} (it renders sound data from only one perspective),
rlm@572 1317 which normally isn't a problem for games, but becomes a problem
rlm@572 1318 when trying to make multiple AI creatures that can each hear the
rlm@572 1319 world from a different perspective.
rlm@572 1320
rlm@572 1321 To make many AI creatures in jMonkeyEngine that can each hear the
rlm@572 1322 world from their own perspective, or to make a single creature with
rlm@572 1323 many ears, it is necessary to go all the way back to \texttt{OpenAL} and
rlm@572 1324 implement support for simulated hearing there.
rlm@572 1325
rlm@572 1326 \subsubsection{Extending \texttt{OpenAl}}
rlm@572 1327 \label{sec-2-9-2}
rlm@572 1328
rlm@572 1329 Extending \texttt{OpenAL} to support multiple listeners requires 500
rlm@572 1330 lines of \texttt{C} code and is too complicated to mention here. Instead,
rlm@572 1331 I will show a small amount of extension code and go over the high
rlm@572 1332 level strategy. Full source is of course available with the
rlm@572 1333 \texttt{CORTEX} distribution if you're interested.
rlm@572 1334
rlm@572 1335 \texttt{OpenAL} goes to great lengths to support many different systems,
rlm@572 1336 all with different sound capabilities and interfaces. It
rlm@572 1337 accomplishes this difficult task by providing code for many
rlm@572 1338 different sound backends in pseudo-objects called \emph{Devices}.
rlm@572 1339 There's a device for the Linux Open Sound System and the Advanced
rlm@572 1340 Linux Sound Architecture, there's one for Direct Sound on Windows,
rlm@572 1341 and there's even one for Solaris. \texttt{OpenAL} solves the problem of
rlm@572 1342 platform independence by providing all these Devices.
rlm@572 1343
rlm@572 1344 Wrapper libraries such as LWJGL are free to examine the system on
rlm@572 1345 which they are running and then select an appropriate device for
rlm@572 1346 that system.
rlm@572 1347
rlm@572 1348 There are also a few "special" devices that don't interface with
rlm@572 1349 any particular system. These include the Null Device, which
rlm@572 1350 doesn't do anything, and the Wave Device, which writes whatever
rlm@572 1351 sound it receives to a file, if everything has been set up
rlm@572 1352 correctly when configuring \texttt{OpenAL}.
rlm@572 1353
rlm@572 1354 Actual mixing (Doppler shift and distance.environment-based
rlm@572 1355 attenuation) of the sound data happens in the Devices, and they
rlm@572 1356 are the only point in the sound rendering process where this data
rlm@572 1357 is available.
rlm@572 1358
rlm@572 1359 Therefore, in order to support multiple listeners, and get the
rlm@572 1360 sound data in a form that the AIs can use, it is necessary to
rlm@572 1361 create a new Device which supports this feature.
rlm@572 1362
rlm@572 1363 Adding a device to OpenAL is rather tricky -- there are five
rlm@572 1364 separate files in the \texttt{OpenAL} source tree that must be modified
rlm@572 1365 to do so. I named my device the "Multiple Audio Send" Device, or
rlm@572 1366 \texttt{Send} Device for short, since it sends audio data back to the
rlm@572 1367 calling application like an Aux-Send cable on a mixing board.
rlm@572 1368
rlm@572 1369 The main idea behind the Send device is to take advantage of the
rlm@572 1370 fact that LWJGL only manages one \emph{context} when using OpenAL. A
rlm@572 1371 \emph{context} is like a container that holds samples and keeps track
rlm@572 1372 of where the listener is. In order to support multiple listeners,
rlm@572 1373 the Send device identifies the LWJGL context as the master
rlm@572 1374 context, and creates any number of slave contexts to represent
rlm@572 1375 additional listeners. Every time the device renders sound, it
rlm@572 1376 synchronizes every source from the master LWJGL context to the
rlm@572 1377 slave contexts. Then, it renders each context separately, using a
rlm@572 1378 different listener for each one. The rendered sound is made
rlm@572 1379 available via JNI to jMonkeyEngine.
rlm@572 1380
rlm@572 1381 Switching between contexts is not the normal operation of a
rlm@572 1382 Device, and one of the problems with doing so is that a Device
rlm@572 1383 normally keeps around a few pieces of state such as the
rlm@572 1384 \texttt{ClickRemoval} array above which will become corrupted if the
rlm@572 1385 contexts are not rendered in parallel. The solution is to create a
rlm@572 1386 copy of this normally global device state for each context, and
rlm@572 1387 copy it back and forth into and out of the actual device state
rlm@572 1388 whenever a context is rendered.
rlm@572 1389
rlm@572 1390 The core of the \texttt{Send} device is the \texttt{syncSources} function, which
rlm@572 1391 does the job of copying all relevant data from one context to
rlm@572 1392 another.
rlm@572 1393
rlm@572 1394 \begin{listing}
rlm@572 1395 \begin{verbatim}
rlm@572 1396 void syncSources(ALsource *masterSource, ALsource *slaveSource,
rlm@572 1397 ALCcontext *masterCtx, ALCcontext *slaveCtx){
rlm@572 1398 ALuint master = masterSource->source;
rlm@572 1399 ALuint slave = slaveSource->source;
rlm@572 1400 ALCcontext *current = alcGetCurrentContext();
rlm@572 1401
rlm@572 1402 syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
rlm@572 1403 syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
rlm@572 1404 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
rlm@572 1405 syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
rlm@572 1406 syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
rlm@572 1407 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
rlm@572 1408 syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
rlm@572 1409 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
rlm@572 1410 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
rlm@572 1411 syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
rlm@572 1412 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
rlm@572 1413 syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
rlm@572 1414 syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
rlm@572 1415
rlm@572 1416 syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
rlm@572 1417 syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
rlm@572 1418 syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
rlm@572 1419
rlm@572 1420 syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
rlm@572 1421 syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
rlm@572 1422
rlm@572 1423 alcMakeContextCurrent(masterCtx);
rlm@572 1424 ALint source_type;
rlm@572 1425 alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
rlm@572 1426
rlm@572 1427 // Only static sources are currently synchronized!
rlm@572 1428 if (AL_STATIC == source_type){
rlm@572 1429 ALint master_buffer;
rlm@572 1430 ALint slave_buffer;
rlm@572 1431 alGetSourcei(master, AL_BUFFER, &master_buffer);
rlm@572 1432 alcMakeContextCurrent(slaveCtx);
rlm@572 1433 alGetSourcei(slave, AL_BUFFER, &slave_buffer);
rlm@572 1434 if (master_buffer != slave_buffer){
rlm@572 1435 alSourcei(slave, AL_BUFFER, master_buffer);
rlm@572 1436 }
rlm@572 1437 }
rlm@572 1438
rlm@572 1439 // Synchronize the state of the two sources.
rlm@572 1440 alcMakeContextCurrent(masterCtx);
rlm@572 1441 ALint masterState;
rlm@572 1442 ALint slaveState;
rlm@572 1443
rlm@572 1444 alGetSourcei(master, AL_SOURCE_STATE, &masterState);
rlm@572 1445 alcMakeContextCurrent(slaveCtx);
rlm@572 1446 alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
rlm@572 1447
rlm@572 1448 if (masterState != slaveState){
rlm@572 1449 switch (masterState){
rlm@572 1450 case AL_INITIAL : alSourceRewind(slave); break;
rlm@572 1451 case AL_PLAYING : alSourcePlay(slave); break;
rlm@572 1452 case AL_PAUSED : alSourcePause(slave); break;
rlm@572 1453 case AL_STOPPED : alSourceStop(slave); break;
rlm@572 1454 }
rlm@572 1455 }
rlm@572 1456 // Restore whatever context was previously active.
rlm@572 1457 alcMakeContextCurrent(current);
rlm@572 1458 }
rlm@572 1459 \end{verbatim}
rlm@572 1460 \caption{\label{sync-openal-sources}Program for extending \texttt{OpenAL} to support multiple listeners via context copying/switching.}
rlm@572 1461 \end{listing}
rlm@572 1462
rlm@572 1463 With this special context-switching device, and some ugly JNI
rlm@572 1464 bindings that are not worth mentioning, \texttt{CORTEX} gains the ability
rlm@572 1465 to access multiple sound streams from \texttt{OpenAL}.
rlm@572 1466
rlm@572 1467 \begin{listing}
rlm@572 1468 \begin{verbatim}
rlm@572 1469 (defn add-ear!
rlm@572 1470 "Create a Listener centered on the current position of 'ear
rlm@572 1471 which follows the closest physical node in 'creature and
rlm@572 1472 sends sound data to 'continuation."
rlm@572 1473 [#^Application world #^Node creature #^Spatial ear continuation]
rlm@572 1474 (let [target (closest-node creature ear)
rlm@572 1475 lis (Listener.)
rlm@572 1476 audio-renderer (.getAudioRenderer world)
rlm@572 1477 sp (hearing-pipeline continuation)]
rlm@572 1478 (.setLocation lis (.getWorldTranslation ear))
rlm@572 1479 (.setRotation lis (.getWorldRotation ear))
rlm@572 1480 (bind-sense target lis)
rlm@572 1481 (update-listener-velocity! target lis)
rlm@572 1482 (.addListener audio-renderer lis)
rlm@572 1483 (.registerSoundProcessor audio-renderer lis sp)))
rlm@572 1484 \end{verbatim}
rlm@572 1485 \caption{\label{add-ear}Program to create an ear from a Blender empty node. The ear follows around the nearest physical object and passes all sensory data to a continuation function.}
rlm@572 1486 \end{listing}
rlm@572 1487
rlm@572 1488 The \texttt{Send} device, unlike most of the other devices in \texttt{OpenAL},
rlm@572 1489 does not render sound unless asked. This enables the system to
rlm@572 1490 slow down or speed up depending on the needs of the AIs who are
rlm@572 1491 using it to listen. If the device tried to render samples in
rlm@572 1492 real-time, a complicated AI whose mind takes 100 seconds of
rlm@572 1493 computer time to simulate 1 second of AI-time would miss almost
rlm@572 1494 all of the sound in its environment!
rlm@572 1495
rlm@572 1496 \begin{listing}
rlm@572 1497 \begin{verbatim}
rlm@572 1498 (defn hearing-kernel
rlm@572 1499 "Returns a function which returns auditory sensory data when called
rlm@572 1500 inside a running simulation."
rlm@572 1501 [#^Node creature #^Spatial ear]
rlm@572 1502 (let [hearing-data (atom [])
rlm@572 1503 register-listener!
rlm@572 1504 (runonce
rlm@572 1505 (fn [#^Application world]
rlm@572 1506 (add-ear!
rlm@572 1507 world creature ear
rlm@572 1508 (comp #(reset! hearing-data %)
rlm@572 1509 byteBuffer->pulse-vector))))]
rlm@572 1510 (fn [#^Application world]
rlm@572 1511 (register-listener! world)
rlm@572 1512 (let [data @hearing-data
rlm@572 1513 topology
rlm@572 1514 (vec (map #(vector % 0) (range 0 (count data))))]
rlm@572 1515 [topology data]))))
rlm@572 1516
rlm@572 1517 (defn hearing!
rlm@572 1518 "Endow the creature in a particular world with the sense of
rlm@572 1519 hearing. Will return a sequence of functions, one for each ear,
rlm@572 1520 which when called will return the auditory data from that ear."
rlm@572 1521 [#^Node creature]
rlm@572 1522 (for [ear (ears creature)]
rlm@572 1523 (hearing-kernel creature ear)))
rlm@572 1524 \end{verbatim}
rlm@572 1525 \caption{\label{hearing}Program to enable arbitrary hearing in \texttt{CORTEX}}
rlm@572 1526 \end{listing}
rlm@572 1527
rlm@572 1528 Armed with these functions, \texttt{CORTEX} is able to test possibly the
rlm@572 1529 first ever instance of multiple listeners in a video game engine
rlm@572 1530 based simulation!
rlm@572 1531
rlm@572 1532 \begin{listing}
rlm@572 1533 \begin{verbatim}
rlm@572 1534 /**
rlm@572 1535 * Respond to sound! This is the brain of an AI entity that
rlm@572 1536 * hears its surroundings and reacts to them.
rlm@572 1537 */
rlm@572 1538 public void process(ByteBuffer audioSamples,
rlm@572 1539 int numSamples, AudioFormat format) {
rlm@572 1540 audioSamples.clear();
rlm@572 1541 byte[] data = new byte[numSamples];
rlm@572 1542 float[] out = new float[numSamples];
rlm@572 1543 audioSamples.get(data);
rlm@572 1544 FloatSampleTools.
rlm@572 1545 byte2floatInterleaved
rlm@572 1546 (data, 0, out, 0, numSamples/format.getFrameSize(), format);
rlm@572 1547
rlm@572 1548 float max = Float.NEGATIVE_INFINITY;
rlm@572 1549 for (float f : out){if (f > max) max = f;}
rlm@572 1550 audioSamples.clear();
rlm@572 1551
rlm@572 1552 if (max > 0.1){
rlm@572 1553 entity.getMaterial().setColor("Color", ColorRGBA.Green);
rlm@572 1554 }
rlm@572 1555 else {
rlm@572 1556 entity.getMaterial().setColor("Color", ColorRGBA.Gray);
rlm@572 1557 }
rlm@572 1558 \end{verbatim}
rlm@572 1559 \caption{\label{sound-test}Here a simple creature responds to sound by changing its color from gray to green when the total volume goes over a threshold.}
rlm@572 1560 \end{listing}
rlm@572 1561
rlm@572 1562 \begin{figure}[htb]
rlm@572 1563 \centering
rlm@572 1564 \includegraphics[width=10cm]{./images/java-hearing-test.png}
rlm@572 1565 \caption{\label{sound-cubes.}First ever simulation of multiple listeners in \texttt{CORTEX}. Each cube is a creature which processes sound data with the \texttt{process} function from listing \ref{sound-test}. the ball is constantly emitting a pure tone of constant volume. As it approaches the cubes, they each change color in response to the sound.}
rlm@572 1566 \end{figure}
rlm@572 1567
rlm@572 1568 This system of hearing has also been co-opted by the
rlm@572 1569 jMonkeyEngine3 community and is used to record audio for demo
rlm@572 1570 videos.
rlm@572 1571
rlm@572 1572 \subsection{Hundreds of hair-like elements provide a sense of touch}
rlm@572 1573 \label{sec-2-10}
rlm@572 1574
rlm@572 1575 Touch is critical to navigation and spatial reasoning and as such I
rlm@572 1576 need a simulated version of it to give to my AI creatures.
rlm@572 1577
rlm@572 1578 Human skin has a wide array of touch sensors, each of which
rlm@572 1579 specialize in detecting different vibrational modes and pressures.
rlm@572 1580 These sensors can integrate a vast expanse of skin (i.e. your
rlm@572 1581 entire palm), or a tiny patch of skin at the tip of your finger.
rlm@572 1582 The hairs of the skin help detect objects before they even come
rlm@572 1583 into contact with the skin proper.
rlm@572 1584
rlm@572 1585 However, touch in my simulated world can not exactly correspond to
rlm@572 1586 human touch because my creatures are made out of completely rigid
rlm@572 1587 segments that don't deform like human skin.
rlm@572 1588
rlm@572 1589 Instead of measuring deformation or vibration, I surround each
rlm@572 1590 rigid part with a plenitude of hair-like objects (\emph{feelers}) which
rlm@572 1591 do not interact with the physical world. Physical objects can pass
rlm@572 1592 through them with no effect. The feelers are able to tell when
rlm@572 1593 other objects pass through them, and they constantly report how
rlm@572 1594 much of their extent is covered. So even though the creature's body
rlm@572 1595 parts do not deform, the feelers create a margin around those body
rlm@572 1596 parts which achieves a sense of touch which is a hybrid between a
rlm@572 1597 human's sense of deformation and sense from hairs.
rlm@572 1598
rlm@572 1599 Implementing touch in jMonkeyEngine follows a different technical
rlm@572 1600 route than vision and hearing. Those two senses piggybacked off
rlm@572 1601 jMonkeyEngine's 3D audio and video rendering subsystems. To
rlm@572 1602 simulate touch, I use jMonkeyEngine's physics system to execute
rlm@572 1603 many small collision detections, one for each feeler. The placement
rlm@572 1604 of the feelers is determined by a UV-mapped image which shows where
rlm@572 1605 each feeler should be on the 3D surface of the body.
rlm@572 1606
rlm@572 1607 \subsubsection{Defining Touch Meta-Data in Blender}
rlm@572 1608 \label{sec-2-10-1}
rlm@572 1609
rlm@572 1610 Each geometry can have a single UV map which describes the
rlm@572 1611 position of the feelers which will constitute its sense of touch.
rlm@572 1612 This image path is stored under the ``touch'' key. The image itself
rlm@572 1613 is black and white, with black meaning a feeler length of 0 (no
rlm@572 1614 feeler is present) and white meaning a feeler length of \texttt{scale},
rlm@572 1615 which is a float stored under the key "scale".
rlm@572 1616
rlm@572 1617 \begin{listing}
rlm@572 1618 \begin{verbatim}
rlm@572 1619 (defn tactile-sensor-profile
rlm@572 1620 "Return the touch-sensor distribution image in BufferedImage format,
rlm@572 1621 or nil if it does not exist."
rlm@572 1622 [#^Geometry obj]
rlm@572 1623 (if-let [image-path (meta-data obj "touch")]
rlm@572 1624 (load-image image-path)))
rlm@572 1625
rlm@572 1626 (defn tactile-scale
rlm@572 1627 "Return the length of each feeler. Default scale is 0.01
rlm@572 1628 jMonkeyEngine units."
rlm@572 1629 [#^Geometry obj]
rlm@572 1630 (if-let [scale (meta-data obj "scale")]
rlm@572 1631 scale 0.1))
rlm@572 1632 \end{verbatim}
rlm@572 1633 \caption{\label{touch-meta-data}Touch does not use empty nodes, to store metadata, because the metadata of each solid part of a creature's body is sufficient.}
rlm@572 1634 \end{listing}
rlm@572 1635
rlm@572 1636 Here is an example of a UV-map which specifies the position of
rlm@572 1637 touch sensors along the surface of the upper segment of a fingertip.
rlm@572 1638
rlm@572 1639 \begin{figure}[htb]
rlm@572 1640 \centering
rlm@572 1641 \includegraphics[width=13cm]{./images/finger-UV.png}
rlm@572 1642 \caption{\label{fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572 1643 \end{figure}
rlm@572 1644
rlm@572 1645 \subsubsection{Implementation Summary}
rlm@572 1646 \label{sec-2-10-2}
rlm@572 1647
rlm@572 1648 To simulate touch there are three conceptual steps. For each solid
rlm@572 1649 object in the creature, you first have to get UV image and scale
rlm@572 1650 parameter which define the position and length of the feelers.
rlm@572 1651 Then, you use the triangles which comprise the mesh and the UV
rlm@572 1652 data stored in the mesh to determine the world-space position and
rlm@572 1653 orientation of each feeler. Then once every frame, update these
rlm@572 1654 positions and orientations to match the current position and
rlm@572 1655 orientation of the object, and use physics collision detection to
rlm@572 1656 gather tactile data.
rlm@572 1657
rlm@572 1658 Extracting the meta-data has already been described. The third
rlm@572 1659 step, physics collision detection, is handled in \texttt{touch-kernel}.
rlm@572 1660 Translating the positions and orientations of the feelers from the
rlm@572 1661 UV-map to world-space is itself a three-step process.
rlm@572 1662
rlm@572 1663 \begin{itemize}
rlm@572 1664 \item Find the triangles which make up the mesh in pixel-space and in
rlm@572 1665 world-space. $\backslash$(\texttt{triangles}, \texttt{pixel-triangles}).
rlm@572 1666
rlm@572 1667 \item Find the coordinates of each feeler in world-space. These are
rlm@572 1668 the origins of the feelers. (\texttt{feeler-origins}).
rlm@572 1669
rlm@572 1670 \item Calculate the normals of the triangles in world space, and add
rlm@572 1671 them to each of the origins of the feelers. These are the
rlm@572 1672 normalized coordinates of the tips of the feelers.
rlm@572 1673 (\texttt{feeler-tips}).
rlm@572 1674 \end{itemize}
rlm@572 1675
rlm@572 1676 \subsubsection{Triangle Math}
rlm@572 1677 \label{sec-2-10-3}
rlm@572 1678
rlm@572 1679 The rigid objects which make up a creature have an underlying
rlm@572 1680 \texttt{Geometry}, which is a \texttt{Mesh} plus a \texttt{Material} and other
rlm@572 1681 important data involved with displaying the object.
rlm@572 1682
rlm@572 1683 A \texttt{Mesh} is composed of \texttt{Triangles}, and each \texttt{Triangle} has three
rlm@572 1684 vertices which have coordinates in world space and UV space.
rlm@572 1685
rlm@572 1686 Here, \texttt{triangles} gets all the world-space triangles which
rlm@572 1687 comprise a mesh, while \texttt{pixel-triangles} gets those same triangles
rlm@572 1688 expressed in pixel coordinates (which are UV coordinates scaled to
rlm@572 1689 fit the height and width of the UV image).
rlm@572 1690
rlm@572 1691 \begin{listing}
rlm@572 1692 \begin{verbatim}
rlm@572 1693 (defn triangle
rlm@572 1694 "Get the triangle specified by triangle-index from the mesh."
rlm@572 1695 [#^Geometry geo triangle-index]
rlm@572 1696 (triangle-seq
rlm@572 1697 (let [scratch (Triangle.)]
rlm@572 1698 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
rlm@572 1699
rlm@572 1700 (defn triangles
rlm@572 1701 "Return a sequence of all the Triangles which comprise a given
rlm@572 1702 Geometry."
rlm@572 1703 [#^Geometry geo]
rlm@572 1704 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
rlm@572 1705
rlm@572 1706 (defn triangle-vertex-indices
rlm@572 1707 "Get the triangle vertex indices of a given triangle from a given
rlm@572 1708 mesh."
rlm@572 1709 [#^Mesh mesh triangle-index]
rlm@572 1710 (let [indices (int-array 3)]
rlm@572 1711 (.getTriangle mesh triangle-index indices)
rlm@572 1712 (vec indices)))
rlm@572 1713
rlm@572 1714 (defn vertex-UV-coord
rlm@572 1715 "Get the UV-coordinates of the vertex named by vertex-index"
rlm@572 1716 [#^Mesh mesh vertex-index]
rlm@572 1717 (let [UV-buffer
rlm@572 1718 (.getData
rlm@572 1719 (.getBuffer
rlm@572 1720 mesh
rlm@572 1721 VertexBuffer$Type/TexCoord))]
rlm@572 1722 [(.get UV-buffer (* vertex-index 2))
rlm@572 1723 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
rlm@572 1724
rlm@572 1725 (defn pixel-triangle [#^Geometry geo image index]
rlm@572 1726 (let [mesh (.getMesh geo)
rlm@572 1727 width (.getWidth image)
rlm@572 1728 height (.getHeight image)]
rlm@572 1729 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
rlm@572 1730 (map (partial vertex-UV-coord mesh)
rlm@572 1731 (triangle-vertex-indices mesh index))))))
rlm@572 1732
rlm@572 1733 (defn pixel-triangles
rlm@572 1734 "The pixel-space triangles of the Geometry, in the same order as
rlm@572 1735 (triangles geo)"
rlm@572 1736 [#^Geometry geo image]
rlm@572 1737 (let [height (.getHeight image)
rlm@572 1738 width (.getWidth image)]
rlm@572 1739 (map (partial pixel-triangle geo image)
rlm@572 1740 (range (.getTriangleCount (.getMesh geo))))))
rlm@572 1741 \end{verbatim}
rlm@572 1742 \caption{\label{get-triangles}Programs to extract triangles from a geometry and get their vertices in both world and UV-coordinates.}
rlm@572 1743 \end{listing}
rlm@572 1744
rlm@572 1745 \subsubsection{The Affine Transform from one Triangle to Another}
rlm@572 1746 \label{sec-2-10-4}
rlm@572 1747
rlm@572 1748 \texttt{pixel-triangles} gives us the mesh triangles expressed in pixel
rlm@572 1749 coordinates and \texttt{triangles} gives us the mesh triangles expressed
rlm@572 1750 in world coordinates. The tactile-sensor-profile gives the
rlm@572 1751 position of each feeler in pixel-space. In order to convert
rlm@572 1752 pixel-space coordinates into world-space coordinates we need
rlm@572 1753 something that takes coordinates on the surface of one triangle
rlm@572 1754 and gives the corresponding coordinates on the surface of another
rlm@572 1755 triangle.
rlm@572 1756
rlm@572 1757 Triangles are \href{http://mathworld.wolfram.com/AffineTransformation.html }{affine}, which means any triangle can be transformed
rlm@572 1758 into any other by a combination of translation, scaling, and
rlm@572 1759 rotation. The affine transformation from one triangle to another
rlm@572 1760 is readily computable if the triangle is expressed in terms of a
rlm@572 1761 \(4x4\) matrix.
rlm@572 1762
rlm@572 1763 $$
rlm@572 1764 \begin{bmatrix}
rlm@572 1765 x_1 & x_2 & x_3 & n_x \\
rlm@572 1766 y_1 & y_2 & y_3 & n_y \\
rlm@572 1767 z_1 & z_2 & z_3 & n_z \\
rlm@572 1768 1 & 1 & 1 & 1
rlm@572 1769 \end{bmatrix}
rlm@572 1770 $$
rlm@572 1771
rlm@572 1772 Here, the first three columns of the matrix are the vertices of
rlm@572 1773 the triangle. The last column is the right-handed unit normal of
rlm@572 1774 the triangle.
rlm@572 1775
rlm@572 1776 With two triangles \(T_{1}\) and \(T_{2}\) each expressed as a
rlm@572 1777 matrix like above, the affine transform from \(T_{1}\) to \(T_{2}\)
rlm@572 1778 is \(T_{2}T_{1}^{-1}\).
rlm@572 1779
rlm@572 1780 The clojure code below recapitulates the formulas above, using
rlm@572 1781 jMonkeyEngine's \texttt{Matrix4f} objects, which can describe any affine
rlm@572 1782 transformation.
rlm@572 1783
rlm@572 1784 \begin{listing}
rlm@572 1785 \begin{verbatim}
rlm@572 1786 (defn triangle->matrix4f
rlm@572 1787 "Converts the triangle into a 4x4 matrix: The first three columns
rlm@572 1788 contain the vertices of the triangle; the last contains the unit
rlm@572 1789 normal of the triangle. The bottom row is filled with 1s."
rlm@572 1790 [#^Triangle t]
rlm@572 1791 (let [mat (Matrix4f.)
rlm@572 1792 [vert-1 vert-2 vert-3]
rlm@572 1793 (mapv #(.get t %) (range 3))
rlm@572 1794 unit-normal (do (.calculateNormal t)(.getNormal t))
rlm@572 1795 vertices [vert-1 vert-2 vert-3 unit-normal]]
rlm@572 1796 (dorun
rlm@572 1797 (for [row (range 4) col (range 3)]
rlm@572 1798 (do
rlm@572 1799 (.set mat col row (.get (vertices row) col))
rlm@572 1800 (.set mat 3 row 1)))) mat))
rlm@572 1801
rlm@572 1802 (defn triangles->affine-transform
rlm@572 1803 "Returns the affine transformation that converts each vertex in the
rlm@572 1804 first triangle into the corresponding vertex in the second
rlm@572 1805 triangle."
rlm@572 1806 [#^Triangle tri-1 #^Triangle tri-2]
rlm@572 1807 (.mult
rlm@572 1808 (triangle->matrix4f tri-2)
rlm@572 1809 (.invert (triangle->matrix4f tri-1))))
rlm@572 1810 \end{verbatim}
rlm@572 1811 \caption{\label{triangle-affine}Program to interpret triangles as affine transforms.}
rlm@572 1812 \end{listing}
rlm@572 1813
rlm@572 1814 \subsubsection{Triangle Boundaries}
rlm@572 1815 \label{sec-2-10-5}
rlm@572 1816
rlm@572 1817 For efficiency's sake I will divide the tactile-profile image into
rlm@572 1818 small squares which inscribe each pixel-triangle, then extract the
rlm@572 1819 points which lie inside the triangle and map them to 3D-space using
rlm@572 1820 \texttt{triangle-transform} above. To do this I need a function,
rlm@572 1821 \texttt{convex-bounds} which finds the smallest box which inscribes a 2D
rlm@572 1822 triangle.
rlm@572 1823
rlm@572 1824 \texttt{inside-triangle?} determines whether a point is inside a triangle
rlm@572 1825 in 2D pixel-space.
rlm@572 1826
rlm@572 1827 \begin{listing}
rlm@572 1828 \begin{verbatim}
rlm@572 1829 (defn convex-bounds
rlm@572 1830 "Returns the smallest square containing the given vertices, as a
rlm@572 1831 vector of integers [left top width height]."
rlm@572 1832 [verts]
rlm@572 1833 (let [xs (map first verts)
rlm@572 1834 ys (map second verts)
rlm@572 1835 x0 (Math/floor (apply min xs))
rlm@572 1836 y0 (Math/floor (apply min ys))
rlm@572 1837 x1 (Math/ceil (apply max xs))
rlm@572 1838 y1 (Math/ceil (apply max ys))]
rlm@572 1839 [x0 y0 (- x1 x0) (- y1 y0)]))
rlm@572 1840
rlm@572 1841 (defn same-side?
rlm@572 1842 "Given the points p1 and p2 and the reference point ref, is point p
rlm@572 1843 on the same side of the line that goes through p1 and p2 as ref is?"
rlm@572 1844 [p1 p2 ref p]
rlm@572 1845 (<=
rlm@572 1846 0
rlm@572 1847 (.dot
rlm@572 1848 (.cross (.subtract p2 p1) (.subtract p p1))
rlm@572 1849 (.cross (.subtract p2 p1) (.subtract ref p1)))))
rlm@572 1850
rlm@572 1851 (defn inside-triangle?
rlm@572 1852 "Is the point inside the triangle?"
rlm@572 1853 {:author "Dylan Holmes"}
rlm@572 1854 [#^Triangle tri #^Vector3f p]
rlm@572 1855 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
rlm@572 1856 (and
rlm@572 1857 (same-side? vert-1 vert-2 vert-3 p)
rlm@572 1858 (same-side? vert-2 vert-3 vert-1 p)
rlm@572 1859 (same-side? vert-3 vert-1 vert-2 p))))
rlm@572 1860 \end{verbatim}
rlm@572 1861 \caption{\label{in-triangle}Program to efficiently determine point inclusion in a triangle.}
rlm@572 1862 \end{listing}
rlm@572 1863
rlm@572 1864 \subsubsection{Feeler Coordinates}
rlm@572 1865 \label{sec-2-10-6}
rlm@572 1866
rlm@572 1867 The triangle-related functions above make short work of
rlm@572 1868 calculating the positions and orientations of each feeler in
rlm@572 1869 world-space.
rlm@572 1870
rlm@572 1871 \begin{listing}
rlm@572 1872 \begin{verbatim}
rlm@572 1873 (defn feeler-pixel-coords
rlm@572 1874 "Returns the coordinates of the feelers in pixel space in lists, one
rlm@572 1875 list for each triangle, ordered in the same way as (triangles) and
rlm@572 1876 (pixel-triangles)."
rlm@572 1877 [#^Geometry geo image]
rlm@572 1878 (map
rlm@572 1879 (fn [pixel-triangle]
rlm@572 1880 (filter
rlm@572 1881 (fn [coord]
rlm@572 1882 (inside-triangle? (->triangle pixel-triangle)
rlm@572 1883 (->vector3f coord)))
rlm@572 1884 (white-coordinates image (convex-bounds pixel-triangle))))
rlm@572 1885 (pixel-triangles geo image)))
rlm@572 1886
rlm@572 1887 (defn feeler-world-coords
rlm@572 1888 "Returns the coordinates of the feelers in world space in lists, one
rlm@572 1889 list for each triangle, ordered in the same way as (triangles) and
rlm@572 1890 (pixel-triangles)."
rlm@572 1891 [#^Geometry geo image]
rlm@572 1892 (let [transforms
rlm@572 1893 (map #(triangles->affine-transform
rlm@572 1894 (->triangle %1) (->triangle %2))
rlm@572 1895 (pixel-triangles geo image)
rlm@572 1896 (triangles geo))]
rlm@572 1897 (map (fn [transform coords]
rlm@572 1898 (map #(.mult transform (->vector3f %)) coords))
rlm@572 1899 transforms (feeler-pixel-coords geo image))))
rlm@572 1900 \end{verbatim}
rlm@572 1901 \caption{\label{feeler-coordinates}Program to get the coordinates of ``feelers '' in both world and UV-coordinates.}
rlm@572 1902 \end{listing}
rlm@572 1903
rlm@572 1904 \begin{listing}
rlm@572 1905 \begin{verbatim}
rlm@572 1906 (defn feeler-origins
rlm@572 1907 "The world space coordinates of the root of each feeler."
rlm@572 1908 [#^Geometry geo image]
rlm@572 1909 (reduce concat (feeler-world-coords geo image)))
rlm@572 1910
rlm@572 1911 (defn feeler-tips
rlm@572 1912 "The world space coordinates of the tip of each feeler."
rlm@572 1913 [#^Geometry geo image]
rlm@572 1914 (let [world-coords (feeler-world-coords geo image)
rlm@572 1915 normals
rlm@572 1916 (map
rlm@572 1917 (fn [triangle]
rlm@572 1918 (.calculateNormal triangle)
rlm@572 1919 (.clone (.getNormal triangle)))
rlm@572 1920 (map ->triangle (triangles geo)))]
rlm@572 1921
rlm@572 1922 (mapcat (fn [origins normal]
rlm@572 1923 (map #(.add % normal) origins))
rlm@572 1924 world-coords normals)))
rlm@572 1925
rlm@572 1926 (defn touch-topology
rlm@572 1927 [#^Geometry geo image]
rlm@572 1928 (collapse (reduce concat (feeler-pixel-coords geo image))))
rlm@572 1929 \end{verbatim}
rlm@572 1930 \caption{\label{feeler-tips}Program to get the position of the base and tip of each ``feeler''}
rlm@572 1931 \end{listing}
rlm@572 1932
rlm@572 1933 \subsubsection{Simulated Touch}
rlm@572 1934 \label{sec-2-10-7}
rlm@572 1935
rlm@572 1936 Now that the functions to construct feelers are complete,
rlm@572 1937 \texttt{touch-kernel} generates functions to be called from within a
rlm@572 1938 simulation that perform the necessary physics collisions to
rlm@572 1939 collect tactile data, and \texttt{touch!} recursively applies it to every
rlm@572 1940 node in the creature.
rlm@572 1941
rlm@572 1942 \begin{listing}
rlm@572 1943 \begin{verbatim}
rlm@572 1944 (defn set-ray [#^Ray ray #^Matrix4f transform
rlm@572 1945 #^Vector3f origin #^Vector3f tip]
rlm@572 1946 ;; Doing everything locally reduces garbage collection by enough to
rlm@572 1947 ;; be worth it.
rlm@572 1948 (.mult transform origin (.getOrigin ray))
rlm@572 1949 (.mult transform tip (.getDirection ray))
rlm@572 1950 (.subtractLocal (.getDirection ray) (.getOrigin ray))
rlm@572 1951 (.normalizeLocal (.getDirection ray)))
rlm@572 1952 \end{verbatim}
rlm@572 1953 \caption{\label{set-ray}Efficient program to transform a ray from one position to another.}
rlm@572 1954 \end{listing}
rlm@572 1955
rlm@572 1956 \begin{listing}
rlm@572 1957 \begin{verbatim}
rlm@572 1958 (defn touch-kernel
rlm@572 1959 "Constructs a function which will return tactile sensory data from
rlm@572 1960 'geo when called from inside a running simulation"
rlm@572 1961 [#^Geometry geo]
rlm@572 1962 (if-let
rlm@572 1963 [profile (tactile-sensor-profile geo)]
rlm@572 1964 (let [ray-reference-origins (feeler-origins geo profile)
rlm@572 1965 ray-reference-tips (feeler-tips geo profile)
rlm@572 1966 ray-length (tactile-scale geo)
rlm@572 1967 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
rlm@572 1968 topology (touch-topology geo profile)
rlm@572 1969 correction (float (* ray-length -0.2))]
rlm@572 1970 ;; slight tolerance for very close collisions.
rlm@572 1971 (dorun
rlm@572 1972 (map (fn [origin tip]
rlm@572 1973 (.addLocal origin (.mult (.subtract tip origin)
rlm@572 1974 correction)))
rlm@572 1975 ray-reference-origins ray-reference-tips))
rlm@572 1976 (dorun (map #(.setLimit % ray-length) current-rays))
rlm@572 1977 (fn [node]
rlm@572 1978 (let [transform (.getWorldMatrix geo)]
rlm@572 1979 (dorun
rlm@572 1980 (map (fn [ray ref-origin ref-tip]
rlm@572 1981 (set-ray ray transform ref-origin ref-tip))
rlm@572 1982 current-rays ray-reference-origins
rlm@572 1983 ray-reference-tips))
rlm@572 1984 (vector
rlm@572 1985 topology
rlm@572 1986 (vec
rlm@572 1987 (for [ray current-rays]
rlm@572 1988 (do
rlm@572 1989 (let [results (CollisionResults.)]
rlm@572 1990 (.collideWith node ray results)
rlm@572 1991 (let [touch-objects
rlm@572 1992 (filter #(not (= geo (.getGeometry %)))
rlm@572 1993 results)
rlm@572 1994 limit (.getLimit ray)]
rlm@572 1995 [(if (empty? touch-objects)
rlm@572 1996 limit
rlm@572 1997 (let [response
rlm@572 1998 (apply min (map #(.getDistance %)
rlm@572 1999 touch-objects))]
rlm@572 2000 (FastMath/clamp
rlm@572 2001 (float
rlm@572 2002 (if (> response limit) (float 0.0)
rlm@572 2003 (+ response correction)))
rlm@572 2004 (float 0.0)
rlm@572 2005 limit)))
rlm@572 2006 limit])))))))))))
rlm@572 2007 \end{verbatim}
rlm@572 2008 \caption{\label{touch-kernel}This is the core of touch in \texttt{CORTEX} each feeler follows the object it is bound to, reporting any collisions that may happen.}
rlm@572 2009 \end{listing}
rlm@572 2010
rlm@572 2011 Armed with the \texttt{touch!} function, \texttt{CORTEX} becomes capable of
rlm@572 2012 giving creatures a sense of touch. A simple test is to create a
rlm@572 2013 cube that is outfitted with a uniform distribution of touch
rlm@572 2014 sensors. It can feel the ground and any balls that it touches.
rlm@572 2015
rlm@572 2016 \begin{listing}
rlm@572 2017 \begin{verbatim}
rlm@572 2018 (defn touch!
rlm@572 2019 "Endow the creature with the sense of touch. Returns a sequence of
rlm@572 2020 functions, one for each body part with a tactile-sensor-profile,
rlm@572 2021 each of which when called returns sensory data for that body part."
rlm@572 2022 [#^Node creature]
rlm@572 2023 (filter
rlm@572 2024 (comp not nil?)
rlm@572 2025 (map touch-kernel
rlm@572 2026 (filter #(isa? (class %) Geometry)
rlm@572 2027 (node-seq creature)))))
rlm@572 2028 \end{verbatim}
rlm@572 2029 \caption{\label{touch}\texttt{CORTEX} interface for creating touch in a simulated creature.}
rlm@572 2030 \end{listing}
rlm@572 2031
rlm@572 2032 The tactile-sensor-profile image for the touch cube is a simple
rlm@572 2033 cross with a uniform distribution of touch sensors:
rlm@572 2034
rlm@572 2035 \begin{figure}[htb]
rlm@572 2036 \centering
rlm@572 2037 \includegraphics[width=7cm]{./images/touch-profile.png}
rlm@572 2038 \caption{\label{touch-cube-uv-map}The touch profile for the touch-cube. Each pure white pixel defines a touch sensitive feeler.}
rlm@572 2039 \end{figure}
rlm@572 2040
rlm@572 2041 \begin{figure}[htb]
rlm@572 2042 \centering
rlm@572 2043 \includegraphics[width=15cm]{./images/touch-cube.png}
rlm@572 2044 \caption{\label{touch-cube-uv-map-2}The touch cube reacts to cannonballs. The black, red, and white cross on the right is a visual display of the creature's touch. White means that it is feeling something strongly, black is not feeling anything, and gray is in-between. The cube can feel both the floor and the ball. Notice that when the ball causes the cube to tip, that the bottom face can still feel part of the ground.}
rlm@572 2045 \end{figure}
rlm@572 2046
rlm@572 2047 \subsection{Proprioception provides knowledge of your own body's position}
rlm@572 2048 \label{sec-2-11}
rlm@572 2049
rlm@572 2050 Close your eyes, and touch your nose with your right index finger.
rlm@572 2051 How did you do it? You could not see your hand, and neither your
rlm@572 2052 hand nor your nose could use the sense of touch to guide the path
rlm@572 2053 of your hand. There are no sound cues, and Taste and Smell
rlm@572 2054 certainly don't provide any help. You know where your hand is
rlm@572 2055 without your other senses because of Proprioception.
rlm@572 2056
rlm@572 2057 Humans can sometimes loose this sense through viral infections or
rlm@572 2058 damage to the spinal cord or brain, and when they do, they loose
rlm@572 2059 the ability to control their own bodies without looking directly at
rlm@572 2060 the parts they want to move. In \href{http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat}{The Man Who Mistook His Wife for a
rlm@572 2061 Hat} (\cite{man-wife-hat}), a woman named Christina looses this
rlm@572 2062 sense and has to learn how to move by carefully watching her arms
rlm@572 2063 and legs. She describes proprioception as the "eyes of the body,
rlm@572 2064 the way the body sees itself".
rlm@572 2065
rlm@572 2066 Proprioception in humans is mediated by \href{http://en.wikipedia.org/wiki/Articular_capsule}{joint capsules}, \href{http://en.wikipedia.org/wiki/Muscle_spindle}{muscle
rlm@572 2067 spindles}, and the \href{http://en.wikipedia.org/wiki/Golgi_tendon_organ}{Golgi tendon organs}. These measure the relative
rlm@572 2068 positions of each body part by monitoring muscle strain and length.
rlm@572 2069
rlm@572 2070 It's clear that this is a vital sense for fluid, graceful movement.
rlm@572 2071 It's also particularly easy to implement in jMonkeyEngine.
rlm@572 2072
rlm@572 2073 My simulated proprioception calculates the relative angles of each
rlm@572 2074 joint from the rest position defined in the Blender file. This
rlm@572 2075 simulates the muscle-spindles and joint capsules. I will deal with
rlm@572 2076 Golgi tendon organs, which calculate muscle strain, in the next
rlm@572 2077 section (2.12).
rlm@572 2078
rlm@572 2079 \subsubsection{Helper functions}
rlm@572 2080 \label{sec-2-11-1}
rlm@572 2081
rlm@572 2082 \texttt{absolute-angle} calculates the angle between two vectors,
rlm@572 2083 relative to a third axis vector. This angle is the number of
rlm@572 2084 radians you have to move counterclockwise around the axis vector
rlm@572 2085 to get from the first to the second vector. It is not commutative
rlm@572 2086 like a normal dot-product angle is.
rlm@572 2087
rlm@572 2088 The purpose of these functions is to build a system of angle
rlm@572 2089 measurement that is biologically plausible.
rlm@572 2090
rlm@572 2091 \begin{listing}
rlm@572 2092 \begin{verbatim}
rlm@572 2093 (defn right-handed?
rlm@572 2094 "true iff the three vectors form a right handed coordinate
rlm@572 2095 system. The three vectors do not have to be normalized or
rlm@572 2096 orthogonal."
rlm@572 2097 [vec1 vec2 vec3]
rlm@572 2098 (pos? (.dot (.cross vec1 vec2) vec3)))
rlm@572 2099
rlm@572 2100 (defn absolute-angle
rlm@572 2101 "The angle between 'vec1 and 'vec2 around 'axis. In the range
rlm@572 2102 [0 (* 2 Math/PI)]."
rlm@572 2103 [vec1 vec2 axis]
rlm@572 2104 (let [angle (.angleBetween vec1 vec2)]
rlm@572 2105 (if (right-handed? vec1 vec2 axis)
rlm@572 2106 angle (- (* 2 Math/PI) angle))))
rlm@572 2107 \end{verbatim}
rlm@572 2108 \caption{\label{helpers}Program to measure angles along a vector}
rlm@572 2109 \end{listing}
rlm@572 2110
rlm@572 2111 \subsubsection{Proprioception Kernel}
rlm@572 2112 \label{sec-2-11-2}
rlm@572 2113
rlm@572 2114 Given a joint, \texttt{proprioception-kernel} produces a function that
rlm@572 2115 calculates the Euler angles between the objects the joint
rlm@572 2116 connects. The only tricky part here is making the angles relative
rlm@572 2117 to the joint's initial ``straightness''.
rlm@572 2118
rlm@572 2119 \begin{listing}
rlm@572 2120 \begin{verbatim}
rlm@572 2121 (defn proprioception-kernel
rlm@572 2122 "Returns a function which returns proprioceptive sensory data when
rlm@572 2123 called inside a running simulation."
rlm@572 2124 [#^Node parts #^Node joint]
rlm@572 2125 (let [[obj-a obj-b] (joint-targets parts joint)
rlm@572 2126 joint-rot (.getWorldRotation joint)
rlm@572 2127 x0 (.mult joint-rot Vector3f/UNIT_X)
rlm@572 2128 y0 (.mult joint-rot Vector3f/UNIT_Y)
rlm@572 2129 z0 (.mult joint-rot Vector3f/UNIT_Z)]
rlm@572 2130 (fn []
rlm@572 2131 (let [rot-a (.clone (.getWorldRotation obj-a))
rlm@572 2132 rot-b (.clone (.getWorldRotation obj-b))
rlm@572 2133 x (.mult rot-a x0)
rlm@572 2134 y (.mult rot-a y0)
rlm@572 2135 z (.mult rot-a z0)
rlm@572 2136
rlm@572 2137 X (.mult rot-b x0)
rlm@572 2138 Y (.mult rot-b y0)
rlm@572 2139 Z (.mult rot-b z0)
rlm@572 2140 heading (Math/atan2 (.dot X z) (.dot X x))
rlm@572 2141 pitch (Math/atan2 (.dot X y) (.dot X x))
rlm@572 2142
rlm@572 2143 ;; rotate x-vector back to origin
rlm@572 2144 reverse
rlm@572 2145 (doto (Quaternion.)
rlm@572 2146 (.fromAngleAxis
rlm@572 2147 (.angleBetween X x)
rlm@572 2148 (let [cross (.normalize (.cross X x))]
rlm@572 2149 (if (= 0 (.length cross)) y cross))))
rlm@572 2150 roll (absolute-angle (.mult reverse Y) y x)]
rlm@572 2151 [heading pitch roll]))))
rlm@572 2152
rlm@572 2153 (defn proprioception!
rlm@572 2154 "Endow the creature with the sense of proprioception. Returns a
rlm@572 2155 sequence of functions, one for each child of the \"joints\" node in
rlm@572 2156 the creature, which each report proprioceptive information about
rlm@572 2157 that joint."
rlm@572 2158 [#^Node creature]
rlm@572 2159 ;; extract the body's joints
rlm@572 2160 (let [senses (map (partial proprioception-kernel creature)
rlm@572 2161 (joints creature))]
rlm@572 2162 (fn []
rlm@572 2163 (map #(%) senses))))
rlm@572 2164 \end{verbatim}
rlm@572 2165 \caption{\label{proprioception}Program to return biologically reasonable proprioceptive data for each joint.}
rlm@572 2166 \end{listing}
rlm@572 2167
rlm@572 2168 \texttt{proprioception!} maps \texttt{proprioception-kernel} across all the
rlm@572 2169 joints of the creature. It uses the same list of joints that
rlm@572 2170 \texttt{joints} uses. Proprioception is the easiest sense to implement in
rlm@572 2171 \texttt{CORTEX}, and it will play a crucial role when efficiently
rlm@572 2172 implementing empathy.
rlm@572 2173
rlm@572 2174 \begin{figure}[htb]
rlm@572 2175 \centering
rlm@572 2176 \includegraphics[width=11cm]{./images/proprio.png}
rlm@572 2177 \caption{\label{proprio}In the upper right corner, the three proprioceptive angle measurements are displayed. Red is yaw, Green is pitch, and White is roll.}
rlm@572 2178 \end{figure}
rlm@572 2179
rlm@572 2180 \subsection{Muscles contain both sensors and effectors}
rlm@572 2181 \label{sec-2-12}
rlm@572 2182
rlm@572 2183 Surprisingly enough, terrestrial creatures only move by using
rlm@572 2184 torque applied about their joints. There's not a single straight
rlm@572 2185 line of force in the human body at all! (A straight line of force
rlm@572 2186 would correspond to some sort of jet or rocket propulsion.)
rlm@572 2187
rlm@572 2188 In humans, muscles are composed of muscle fibers which can contract
rlm@572 2189 to exert force. The muscle fibers which compose a muscle are
rlm@572 2190 partitioned into discrete groups which are each controlled by a
rlm@572 2191 single alpha motor neuron. A single alpha motor neuron might
rlm@572 2192 control as little as three or as many as one thousand muscle
rlm@572 2193 fibers. When the alpha motor neuron is engaged by the spinal cord,
rlm@572 2194 it activates all of the muscle fibers to which it is attached. The
rlm@572 2195 spinal cord generally engages the alpha motor neurons which control
rlm@572 2196 few muscle fibers before the motor neurons which control many
rlm@572 2197 muscle fibers. This recruitment strategy allows for precise
rlm@572 2198 movements at low strength. The collection of all motor neurons that
rlm@572 2199 control a muscle is called the motor pool. The brain essentially
rlm@572 2200 says "activate 30\% of the motor pool" and the spinal cord recruits
rlm@572 2201 motor neurons until 30\% are activated. Since the distribution of
rlm@572 2202 power among motor neurons is unequal and recruitment goes from
rlm@572 2203 weakest to strongest, the first 30\% of the motor pool might be 5\%
rlm@572 2204 of the strength of the muscle.
rlm@572 2205
rlm@572 2206 My simulated muscles follow a similar design: Each muscle is
rlm@572 2207 defined by a 1-D array of numbers (the "motor pool"). Each entry in
rlm@572 2208 the array represents a motor neuron which controls a number of
rlm@572 2209 muscle fibers equal to the value of the entry. Each muscle has a
rlm@572 2210 scalar strength factor which determines the total force the muscle
rlm@572 2211 can exert when all motor neurons are activated. The effector
rlm@572 2212 function for a muscle takes a number to index into the motor pool,
rlm@572 2213 and then "activates" all the motor neurons whose index is lower or
rlm@572 2214 equal to the number. Each motor-neuron will apply force in
rlm@572 2215 proportion to its value in the array. Lower values cause less
rlm@572 2216 force. The lower values can be put at the "beginning" of the 1-D
rlm@572 2217 array to simulate the layout of actual human muscles, which are
rlm@572 2218 capable of more precise movements when exerting less force. Or, the
rlm@572 2219 motor pool can simulate more exotic recruitment strategies which do
rlm@572 2220 not correspond to human muscles.
rlm@572 2221
rlm@572 2222 This 1D array is defined in an image file for ease of
rlm@572 2223 creation/visualization. Here is an example muscle profile image.
rlm@572 2224
rlm@572 2225 \begin{figure}[htb]
rlm@572 2226 \centering
rlm@572 2227 \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572 2228 \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572 2229 \end{figure}
rlm@572 2230
rlm@572 2231 \subsubsection{Muscle meta-data}
rlm@572 2232 \label{sec-2-12-1}
rlm@572 2233
rlm@572 2234 \begin{listing}
rlm@572 2235 \begin{verbatim}
rlm@572 2236 (defn muscle-profile-image
rlm@572 2237 "Get the muscle-profile image from the node's Blender meta-data."
rlm@572 2238 [#^Node muscle]
rlm@572 2239 (if-let [image (meta-data muscle "muscle")]
rlm@572 2240 (load-image image)))
rlm@572 2241
rlm@572 2242 (defn muscle-strength
rlm@572 2243 "Return the strength of this muscle, or 1 if it is not defined."
rlm@572 2244 [#^Node muscle]
rlm@572 2245 (if-let [strength (meta-data muscle "strength")]
rlm@572 2246 strength 1))
rlm@572 2247
rlm@572 2248 (defn motor-pool
rlm@572 2249 "Return a vector where each entry is the strength of the \"motor
rlm@572 2250 neuron\" at that part in the muscle."
rlm@572 2251 [#^Node muscle]
rlm@572 2252 (let [profile (muscle-profile-image muscle)]
rlm@572 2253 (vec
rlm@572 2254 (let [width (.getWidth profile)]
rlm@572 2255 (for [x (range width)]
rlm@572 2256 (- 255
rlm@572 2257 (bit-and
rlm@572 2258 0x0000FF
rlm@572 2259 (.getRGB profile x 0))))))))
rlm@572 2260 \end{verbatim}
rlm@572 2261 \caption{\label{motor-pool}Program to deal with loading muscle data from a Blender file's metadata.}
rlm@572 2262 \end{listing}
rlm@572 2263
rlm@572 2264 Of note here is \texttt{motor-pool} which interprets the muscle-profile
rlm@572 2265 image in a way that allows me to use gradients between white and
rlm@572 2266 red, instead of shades of gray as I've been using for all the
rlm@572 2267 other senses. This is purely an aesthetic touch.
rlm@572 2268
rlm@572 2269 \subsubsection{Creating muscles}
rlm@572 2270 \label{sec-2-12-2}
rlm@572 2271
rlm@572 2272 \begin{listing}
rlm@572 2273 \begin{verbatim}
rlm@572 2274 (defn movement-kernel
rlm@572 2275 "Returns a function which when called with a integer value inside a
rlm@572 2276 running simulation will cause movement in the creature according
rlm@572 2277 to the muscle's position and strength profile. Each function
rlm@572 2278 returns the amount of force applied / max force."
rlm@572 2279 [#^Node creature #^Node muscle]
rlm@572 2280 (let [target (closest-node creature muscle)
rlm@572 2281 axis
rlm@572 2282 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
rlm@572 2283 strength (muscle-strength muscle)
rlm@572 2284
rlm@572 2285 pool (motor-pool muscle)
rlm@572 2286 pool-integral (reductions + pool)
rlm@572 2287 forces
rlm@572 2288 (vec (map #(float (* strength (/ % (last pool-integral))))
rlm@572 2289 pool-integral))
rlm@572 2290 control (.getControl target RigidBodyControl)]
rlm@572 2291 (fn [n]
rlm@572 2292 (let [pool-index (max 0 (min n (dec (count pool))))
rlm@572 2293 force (forces pool-index)]
rlm@572 2294 (.applyTorque control (.mult axis force))
rlm@572 2295 (float (/ force strength))))))
rlm@572 2296
rlm@572 2297 (defn movement!
rlm@572 2298 "Endow the creature with the power of movement. Returns a sequence
rlm@572 2299 of functions, each of which accept an integer value and will
rlm@572 2300 activate their corresponding muscle."
rlm@572 2301 [#^Node creature]
rlm@572 2302 (for [muscle (muscles creature)]
rlm@572 2303 (movement-kernel creature muscle)))
rlm@572 2304 \end{verbatim}
rlm@572 2305 \caption{\label{muscle-kernel}This is the core movement function in \texttt{CORTEX}, which implements muscles that report on their activation.}
rlm@572 2306 \end{listing}
rlm@572 2307
rlm@572 2308
rlm@572 2309 \texttt{movement-kernel} creates a function that controls the movement
rlm@572 2310 of the nearest physical node to the muscle node. The muscle exerts
rlm@572 2311 a rotational force dependent on it's orientation to the object in
rlm@572 2312 the Blender file. The function returned by \texttt{movement-kernel} is
rlm@572 2313 also a sense function: it returns the percent of the total muscle
rlm@572 2314 strength that is currently being employed. This is analogous to
rlm@572 2315 muscle tension in humans and completes the sense of proprioception
rlm@572 2316 begun in the last chapter.
rlm@572 2317
rlm@572 2318 \subsection{\texttt{CORTEX} brings complex creatures to life!}
rlm@572 2319 \label{sec-2-13}
rlm@572 2320
rlm@572 2321 The ultimate test of \texttt{CORTEX} is to create a creature with the full
rlm@572 2322 gamut of senses and put it though its paces.
rlm@572 2323
rlm@572 2324 With all senses enabled, my right hand model looks like an
rlm@572 2325 intricate marionette hand with several strings for each finger:
rlm@572 2326
rlm@572 2327 \begin{figure}[htb]
rlm@572 2328 \centering
rlm@572 2329 \includegraphics[width=11cm]{./images/hand-with-all-senses2.png}
rlm@572 2330 \caption{\label{hand-nodes-1}View of the hand model with all sense nodes. You can see the joint, muscle, ear, and eye nodes here.}
rlm@572 2331 \end{figure}
rlm@572 2332
rlm@572 2333 \begin{figure}[htb]
rlm@572 2334 \centering
rlm@572 2335 \includegraphics[width=15cm]{./images/hand-with-all-senses3.png}
rlm@572 2336 \caption{\label{hand-nodes-2}An alternate view of the hand.}
rlm@572 2337 \end{figure}
rlm@572 2338
rlm@572 2339 With the hand fully rigged with senses, I can run it though a test
rlm@572 2340 that will test everything.
rlm@572 2341
rlm@572 2342 \begin{figure}[htb]
rlm@572 2343 \centering
rlm@572 2344 \includegraphics[width=15cm]{./images/integration.png}
rlm@572 2345 \caption{\label{integration}Selected frames from a full test of the hand with all senses. Note especially the interactions the hand has with itself: it feels its own palm and fingers, and when it curls its fingers, it sees them with its eye (which is located in the center of the palm. The red block appears with a pure tone sound. The hand then uses its muscles to launch the cube!}
rlm@572 2346 \end{figure}
rlm@572 2347
rlm@572 2348 \subsection{\texttt{CORTEX} enables many possibilities for further research}
rlm@572 2349 \label{sec-2-14}
rlm@572 2350
rlm@572 2351 Often times, the hardest part of building a system involving
rlm@572 2352 creatures is dealing with physics and graphics. \texttt{CORTEX} removes
rlm@572 2353 much of this initial difficulty and leaves researchers free to
rlm@572 2354 directly pursue their ideas. I hope that even novices with a
rlm@572 2355 passing curiosity about simulated touch or creature evolution will
rlm@572 2356 be able to use cortex for experimentation. \texttt{CORTEX} is a completely
rlm@572 2357 simulated world, and far from being a disadvantage, its simulated
rlm@572 2358 nature enables you to create senses and creatures that would be
rlm@572 2359 impossible to make in the real world.
rlm@572 2360
rlm@572 2361 While not by any means a complete list, here are some paths
rlm@572 2362 \texttt{CORTEX} is well suited to help you explore:
rlm@572 2363
rlm@572 2364 \begin{description}
rlm@572 2365 \item[{Empathy }] my empathy program leaves many areas for
rlm@572 2366 improvement, among which are using vision to infer
rlm@572 2367 proprioception and looking up sensory experience with imagined
rlm@572 2368 vision, touch, and sound.
rlm@572 2369 \item[{Evolution}] Karl Sims created a rich environment for simulating
rlm@572 2370 the evolution of creatures on a Connection Machine
rlm@572 2371 (\cite{sims-evolving-creatures}). Today, this can be redone
rlm@572 2372 and expanded with \texttt{CORTEX} on an ordinary computer.
rlm@572 2373 \item[{Exotic senses }] Cortex enables many fascinating senses that are
rlm@572 2374 not possible to build in the real world. For example,
rlm@572 2375 telekinesis is an interesting avenue to explore. You can also
rlm@572 2376 make a ``semantic'' sense which looks up metadata tags on
rlm@572 2377 objects in the environment the metadata tags might contain
rlm@572 2378 other sensory information.
rlm@572 2379 \item[{Imagination via subworlds}] this would involve a creature with
rlm@572 2380 an effector which creates an entire new sub-simulation where
rlm@572 2381 the creature has direct control over placement/creation of
rlm@572 2382 objects via simulated telekinesis. The creature observes this
rlm@572 2383 sub-world through its normal senses and uses its observations
rlm@572 2384 to make predictions about its top level world.
rlm@572 2385 \item[{Simulated prescience}] step the simulation forward a few ticks,
rlm@572 2386 gather sensory data, then supply this data for the creature as
rlm@572 2387 one of its actual senses. The cost of prescience is slowing
rlm@572 2388 the simulation down by a factor proportional to however far
rlm@572 2389 you want the entities to see into the future. What happens
rlm@572 2390 when two evolved creatures that can each see into the future
rlm@572 2391 fight each other?
rlm@572 2392 \item[{Swarm creatures}] Program a group of creatures that cooperate
rlm@572 2393 with each other. Because the creatures would be simulated, you
rlm@572 2394 could investigate computationally complex rules of behavior
rlm@572 2395 which still, from the group's point of view, would happen in
rlm@572 2396 real time. Interactions could be as simple as cellular
rlm@572 2397 organisms communicating via flashing lights, or as complex as
rlm@572 2398 humanoids completing social tasks, etc.
rlm@572 2399 \item[{\texttt{HACKER} for writing muscle-control programs}] Presented with a
rlm@572 2400 low-level muscle control / sense API, generate higher level
rlm@572 2401 programs for accomplishing various stated goals. Example goals
rlm@572 2402 might be "extend all your fingers" or "move your hand into the
rlm@572 2403 area with blue light" or "decrease the angle of this joint".
rlm@572 2404 It would be like Sussman's HACKER, except it would operate
rlm@572 2405 with much more data in a more realistic world. Start off with
rlm@572 2406 "calisthenics" to develop subroutines over the motor control
rlm@572 2407 API. The low level programming code might be a turning machine
rlm@572 2408 that could develop programs to iterate over a "tape" where
rlm@572 2409 each entry in the tape could control recruitment of the fibers
rlm@572 2410 in a muscle.
rlm@572 2411 \item[{Sense fusion}] There is much work to be done on sense
rlm@572 2412 integration -- building up a coherent picture of the world and
rlm@572 2413 the things in it. With \texttt{CORTEX} as a base, you can explore
rlm@572 2414 concepts like self-organizing maps or cross modal clustering
rlm@572 2415 in ways that have never before been tried.
rlm@572 2416 \item[{Inverse kinematics}] experiments in sense guided motor control
rlm@572 2417 are easy given \texttt{CORTEX}'s support -- you can get right to the
rlm@572 2418 hard control problems without worrying about physics or
rlm@572 2419 senses.
rlm@572 2420 \end{description}
rlm@572 2421
rlm@572 2422 \newpage
rlm@572 2423
rlm@572 2424 \section{\texttt{EMPATH}: action recognition in a simulated worm}
rlm@572 2425 \label{sec-3}
rlm@572 2426
rlm@572 2427 Here I develop a computational model of empathy, using \texttt{CORTEX} as a
rlm@572 2428 base. Empathy in this context is the ability to observe another
rlm@572 2429 creature and infer what sorts of sensations that creature is
rlm@572 2430 feeling. My empathy algorithm involves multiple phases. First is
rlm@572 2431 free-play, where the creature moves around and gains sensory
rlm@572 2432 experience. From this experience I construct a representation of the
rlm@572 2433 creature's sensory state space, which I call \(\Phi\)-space. Using
rlm@572 2434 \(\Phi\)-space, I construct an efficient function which takes the
rlm@572 2435 limited data that comes from observing another creature and enriches
rlm@572 2436 it with a full compliment of imagined sensory data. I can then use
rlm@572 2437 the imagined sensory data to recognize what the observed creature is
rlm@572 2438 doing and feeling, using straightforward embodied action predicates.
rlm@572 2439 This is all demonstrated with using a simple worm-like creature, and
rlm@572 2440 recognizing worm-actions based on limited data.
rlm@572 2441
rlm@572 2442 \begin{figure}[htb]
rlm@572 2443 \centering
rlm@572 2444 \includegraphics[width=10cm]{./images/basic-worm-view.png}
rlm@572 2445 \caption{\label{basic-worm-view}Here is the worm with which we will be working. It is composed of 5 segments. Each segment has a pair of extensor and flexor muscles. Each of the worm's four joints is a hinge joint which allows about 30 degrees of rotation to either side. Each segment of the worm is touch-capable and has a uniform distribution of touch sensors on each of its faces. Each joint has a proprioceptive sense to detect relative positions. The worm segments are all the same except for the first one, which has a much higher weight than the others to allow for easy manual motor control.}
rlm@572 2446 \end{figure}
rlm@572 2447
rlm@572 2448 \begin{listing}
rlm@572 2449 \begin{verbatim}
rlm@572 2450 (defn worm []
rlm@572 2451 (let [model (load-blender-model "Models/worm/worm.blend")]
rlm@572 2452 {:body (doto model (body!))
rlm@572 2453 :touch (touch! model)
rlm@572 2454 :proprioception (proprioception! model)
rlm@572 2455 :muscles (movement! model)}))
rlm@572 2456 \end{verbatim}
rlm@572 2457 \caption{\label{get-worm}Program for reading a worm from a Blender file and outfitting it with the senses of proprioception, touch, and the ability to move, as specified in the Blender file.}
rlm@572 2458 \end{listing}
rlm@572 2459
rlm@572 2460 \subsection{Embodiment factors action recognition into manageable parts}
rlm@572 2461 \label{sec-3-1}
rlm@572 2462
rlm@572 2463 Using empathy, I divide the problem of action recognition into a
rlm@572 2464 recognition process expressed in the language of a full compliment
rlm@572 2465 of senses, and an imaginative process that generates full sensory
rlm@572 2466 data from partial sensory data. Splitting the action recognition
rlm@572 2467 problem in this manner greatly reduces the total amount of work to
rlm@572 2468 recognize actions: The imaginative process is mostly just matching
rlm@572 2469 previous experience, and the recognition process gets to use all
rlm@572 2470 the senses to directly describe any action.
rlm@572 2471
rlm@572 2472 \subsection{Action recognition is easy with a full gamut of senses}
rlm@572 2473 \label{sec-3-2}
rlm@572 2474
rlm@572 2475 Embodied representation using multiple senses such as touch,
rlm@572 2476 proprioception, and muscle tension turns out be exceedingly
rlm@572 2477 efficient at describing body-centered actions. It is the right
rlm@572 2478 language for the job. For example, it takes only around 5 lines of
rlm@572 2479 clojure code to describe the action of curling using embodied
rlm@572 2480 primitives. It takes about 10 lines to describe the seemingly
rlm@572 2481 complicated action of wiggling.
rlm@572 2482
rlm@572 2483 The following action predicates each take a stream of sensory
rlm@572 2484 experience, observe however much of it they desire, and decide
rlm@572 2485 whether the worm is doing the action they describe. \texttt{curled?}
rlm@572 2486 relies on proprioception, \texttt{resting?} relies on touch, \texttt{wiggling?}
rlm@572 2487 relies on a Fourier analysis of muscle contraction, and
rlm@572 2488 \texttt{grand-circle?} relies on touch and reuses \texttt{curled?} in its
rlm@572 2489 definition, showing how embodied predicates can be composed.
rlm@572 2490
rlm@572 2491
rlm@572 2492 \begin{listing}
rlm@572 2493 \begin{verbatim}
rlm@572 2494 (defn curled?
rlm@572 2495 "Is the worm curled up?"
rlm@572 2496 [experiences]
rlm@572 2497 (every?
rlm@572 2498 (fn [[_ _ bend]]
rlm@572 2499 (> (Math/sin bend) 0.64))
rlm@572 2500 (:proprioception (peek experiences))))
rlm@572 2501 \end{verbatim}
rlm@572 2502 \caption{\label{curled}Program for detecting whether the worm is curled. This is the simplest action predicate, because it only uses the last frame of sensory experience, and only uses proprioceptive data. Even this simple predicate, however, is automatically frame independent and ignores vermopomorphic\protect\footnotemark \space differences such as worm textures and colors.}
rlm@572 2503 \end{listing}
rlm@572 2504
rlm@572 2505 \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
rlm@572 2506
rlm@572 2507 \begin{listing}
rlm@572 2508 \begin{verbatim}
rlm@572 2509 (defn contact
rlm@572 2510 "Determine how much contact a particular worm segment has with
rlm@572 2511 other objects. Returns a value between 0 and 1, where 1 is full
rlm@572 2512 contact and 0 is no contact."
rlm@572 2513 [touch-region [coords contact :as touch]]
rlm@572 2514 (-> (zipmap coords contact)
rlm@572 2515 (select-keys touch-region)
rlm@572 2516 (vals)
rlm@572 2517 (#(map first %))
rlm@572 2518 (average)
rlm@572 2519 (* 10)
rlm@572 2520 (- 1)
rlm@572 2521 (Math/abs)))
rlm@572 2522 \end{verbatim}
rlm@572 2523 \caption{\label{touch-summary}Program for summarizing the touch information in a patch of skin.}
rlm@572 2524 \end{listing}
rlm@572 2525
rlm@572 2526
rlm@572 2527 \begin{listing}
rlm@572 2528 \begin{verbatim}
rlm@572 2529 (def worm-segment-bottom (rect-region [8 15] [14 22]))
rlm@572 2530
rlm@572 2531 (defn resting?
rlm@572 2532 "Is the worm resting on the ground?"
rlm@572 2533 [experiences]
rlm@572 2534 (every?
rlm@572 2535 (fn [touch-data]
rlm@572 2536 (< 0.9 (contact worm-segment-bottom touch-data)))
rlm@572 2537 (:touch (peek experiences))))
rlm@572 2538 \end{verbatim}
rlm@572 2539 \caption{\label{resting}Program for detecting whether the worm is at rest. This program uses a summary of the tactile information from the underbelly of the worm, and is only true if every segment is touching the floor. Note that this function contains no references to proprioception at all.}
rlm@572 2540 \end{listing}
rlm@572 2541
rlm@572 2542 \begin{listing}
rlm@572 2543 \begin{verbatim}
rlm@572 2544 (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
rlm@572 2545
rlm@572 2546 (def worm-segment-top-tip (rect-region [0 15] [7 22]))
rlm@572 2547
rlm@572 2548 (defn grand-circle?
rlm@572 2549 "Does the worm form a majestic circle (one end touching the other)?"
rlm@572 2550 [experiences]
rlm@572 2551 (and (curled? experiences)
rlm@572 2552 (let [worm-touch (:touch (peek experiences))
rlm@572 2553 tail-touch (worm-touch 0)
rlm@572 2554 head-touch (worm-touch 4)]
rlm@572 2555 (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
rlm@572 2556 (< 0.55 (contact worm-segment-top-tip head-touch))))))
rlm@572 2557 \end{verbatim}
rlm@572 2558 \caption{\label{grand-circle}Program for detecting whether the worm is curled up into a full circle. Here the embodied approach begins to shine, as I am able to both use a previous action predicate (\texttt{curled?}) as well as the direct tactile experience of the head and tail.}
rlm@572 2559 \end{listing}
rlm@572 2560
rlm@572 2561
rlm@572 2562 \begin{listing}
rlm@572 2563 \begin{verbatim}
rlm@572 2564 (defn fft [nums]
rlm@572 2565 (map
rlm@572 2566 #(.getReal %)
rlm@572 2567 (.transform
rlm@572 2568 (FastFourierTransformer. DftNormalization/STANDARD)
rlm@572 2569 (double-array nums) TransformType/FORWARD)))
rlm@572 2570
rlm@572 2571 (def indexed (partial map-indexed vector))
rlm@572 2572
rlm@572 2573 (defn max-indexed [s]
rlm@572 2574 (first (sort-by (comp - second) (indexed s))))
rlm@572 2575
rlm@572 2576 (defn wiggling?
rlm@572 2577 "Is the worm wiggling?"
rlm@572 2578 [experiences]
rlm@572 2579 (let [analysis-interval 0x40]
rlm@572 2580 (when (> (count experiences) analysis-interval)
rlm@572 2581 (let [a-flex 3
rlm@572 2582 a-ex 2
rlm@572 2583 muscle-activity
rlm@572 2584 (map :muscle (vector:last-n experiences analysis-interval))
rlm@572 2585 base-activity
rlm@572 2586 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
rlm@572 2587 (= 2
rlm@572 2588 (first
rlm@572 2589 (max-indexed
rlm@572 2590 (map #(Math/abs %)
rlm@572 2591 (take 20 (fft base-activity))))))))))
rlm@572 2592 \end{verbatim}
rlm@572 2593 \caption{\label{wiggling}Program for detecting whether the worm has been wiggling for the last few frames. It uses a Fourier analysis of the muscle contractions of the worm's tail to determine wiggling. This is significant because there is no particular frame that clearly indicates that the worm is wiggling --- only when multiple frames are analyzed together is the wiggling revealed. Defining wiggling this way also gives the worm an opportunity to learn and recognize ``frustrated wiggling'', where the worm tries to wiggle but can't. Frustrated wiggling is very visually different from actual wiggling, but this definition gives it to us for free.}
rlm@572 2594 \end{listing}
rlm@572 2595
rlm@572 2596 With these action predicates, I can now recognize the actions of
rlm@572 2597 the worm while it is moving under my control and I have access to
rlm@572 2598 all the worm's senses.
rlm@572 2599
rlm@572 2600 \begin{listing}
rlm@572 2601 \begin{verbatim}
rlm@572 2602 (defn debug-experience
rlm@572 2603 [experiences text]
rlm@572 2604 (cond
rlm@572 2605 (grand-circle? experiences) (.setText text "Grand Circle")
rlm@572 2606 (curled? experiences) (.setText text "Curled")
rlm@572 2607 (wiggling? experiences) (.setText text "Wiggling")
rlm@572 2608 (resting? experiences) (.setText text "Resting")))
rlm@572 2609 \end{verbatim}
rlm@572 2610 \caption{\label{report-worm-activity}Use the action predicates defined earlier to report on what the worm is doing while in simulation.}
rlm@572 2611 \end{listing}
rlm@572 2612
rlm@572 2613 \begin{figure}[htb]
rlm@572 2614 \centering
rlm@572 2615 \includegraphics[width=10cm]{./images/worm-identify-init.png}
rlm@572 2616 \caption{\label{basic-worm-view}Using \texttt{debug-experience}, the body-centered predicates work together to classify the behavior of the worm. the predicates are operating with access to the worm's full sensory data.}
rlm@572 2617 \end{figure}
rlm@572 2618
rlm@572 2619 These action predicates satisfy the recognition requirement of an
rlm@572 2620 empathic recognition system. There is power in the simplicity of
rlm@572 2621 the action predicates. They describe their actions without getting
rlm@572 2622 confused in visual details of the worm. Each one is independent of
rlm@572 2623 position and rotation, but more than that, they are each
rlm@572 2624 independent of irrelevant visual details of the worm and the
rlm@572 2625 environment. They will work regardless of whether the worm is a
rlm@572 2626 different color or heavily textured, or if the environment has
rlm@572 2627 strange lighting.
rlm@572 2628
rlm@572 2629 Consider how the human act of jumping might be described with
rlm@572 2630 body-centered action predicates: You might specify that jumping is
rlm@572 2631 mainly the feeling of your knees bending, your thigh muscles
rlm@572 2632 contracting, and your inner ear experiencing a certain sort of back
rlm@572 2633 and forth acceleration. This representation is a very concrete
rlm@572 2634 description of jumping, couched in terms of muscles and senses, but
rlm@572 2635 it also has the ability to describe almost all kinds of jumping, a
rlm@572 2636 generality that you might think could only be achieved by a very
rlm@572 2637 abstract description. The body centered jumping predicate does not
rlm@572 2638 have terms that consider the color of a person's skin or whether
rlm@572 2639 they are male or female, instead it gets right to the meat of what
rlm@572 2640 jumping actually \emph{is}.
rlm@572 2641
rlm@572 2642 Of course, the action predicates are not directly applicable to
rlm@572 2643 video data, which lacks the advanced sensory information which they
rlm@572 2644 require!
rlm@572 2645
rlm@572 2646 The trick now is to make the action predicates work even when the
rlm@572 2647 sensory data on which they depend is absent!
rlm@572 2648
rlm@572 2649 \subsection{\(\Phi\)-space describes the worm's experiences}
rlm@572 2650 \label{sec-3-3}
rlm@572 2651
rlm@572 2652 As a first step towards building empathy, I need to gather all of
rlm@572 2653 the worm's experiences during free play. I use a simple vector to
rlm@572 2654 store all the experiences.
rlm@572 2655
rlm@572 2656 Each element of the experience vector exists in the vast space of
rlm@572 2657 all possible worm-experiences. Most of this vast space is actually
rlm@572 2658 unreachable due to physical constraints of the worm's body. For
rlm@572 2659 example, the worm's segments are connected by hinge joints that put
rlm@572 2660 a practical limit on the worm's range of motions without limiting
rlm@572 2661 its degrees of freedom. Some groupings of senses are impossible;
rlm@572 2662 the worm can not be bent into a circle so that its ends are
rlm@572 2663 touching and at the same time not also experience the sensation of
rlm@572 2664 touching itself.
rlm@572 2665
rlm@572 2666 As the worm moves around during free play and its experience vector
rlm@572 2667 grows larger, the vector begins to define a subspace which is all
rlm@572 2668 the sensations the worm can practically experience during normal
rlm@572 2669 operation. I call this subspace \(\Phi\)-space, short for
rlm@572 2670 physical-space. The experience vector defines a path through
rlm@572 2671 \(\Phi\)-space. This path has interesting properties that all derive
rlm@572 2672 from physical embodiment. The proprioceptive components of the path
rlm@572 2673 vary smoothly, because in order for the worm to move from one
rlm@572 2674 position to another, it must pass through the intermediate
rlm@572 2675 positions. The path invariably forms loops as common actions are
rlm@572 2676 repeated. Finally and most importantly, proprioception alone
rlm@572 2677 actually gives very strong inference about the other senses. For
rlm@572 2678 example, when the worm is proprioceptively flat over several
rlm@572 2679 frames, you can infer that it is touching the ground and that its
rlm@572 2680 muscles are not active, because if the muscles were active, the
rlm@572 2681 worm would be moving and would not remain perfectly flat. In order
rlm@572 2682 to stay flat, the worm has to be touching the ground, or it would
rlm@572 2683 again be moving out of the flat position due to gravity. If the
rlm@572 2684 worm is positioned in such a way that it interacts with itself,
rlm@572 2685 then it is very likely to be feeling the same tactile feelings as
rlm@572 2686 the last time it was in that position, because it has the same body
rlm@572 2687 as then. As you observe multiple frames of proprioceptive data, you
rlm@572 2688 can become increasingly confident about the exact activations of
rlm@572 2689 the worm's muscles, because it generally takes a unique combination
rlm@572 2690 of muscle contractions to transform the worm's body along a
rlm@572 2691 specific path through \(\Phi\)-space.
rlm@572 2692
rlm@572 2693 The worm's total life experience is a long looping path through
rlm@572 2694 \(\Phi\)-space. I will now introduce simple way of taking that
rlm@572 2695 experience path and building a function that can infer complete
rlm@572 2696 sensory experience given only a stream of proprioceptive data. This
rlm@572 2697 \emph{empathy} function will provide a bridge to use the body centered
rlm@572 2698 action predicates on video-like streams of information.
rlm@572 2699
rlm@572 2700 \subsection{Empathy is the process of building paths in \(\Phi\)-space}
rlm@572 2701 \label{sec-3-4}
rlm@572 2702
rlm@572 2703 Here is the core of a basic empathy algorithm, starting with an
rlm@572 2704 experience vector:
rlm@572 2705
rlm@572 2706 An \emph{experience-index} is an index into the grand experience vector
rlm@572 2707 that defines the worm's life. It is a time-stamp for each set of
rlm@572 2708 sensations the worm has experienced.
rlm@572 2709
rlm@572 2710 First, I group the experience-indices into bins according to the
rlm@572 2711 similarity of their proprioceptive data. I organize my bins into a
rlm@572 2712 3 level hierarchy. The smallest bins have an approximate size of
rlm@572 2713 0.001 radians in all proprioceptive dimensions. Each higher level
rlm@572 2714 is 10x bigger than the level below it.
rlm@572 2715
rlm@572 2716 The bins serve as a hashing function for proprioceptive data. Given
rlm@572 2717 a single piece of proprioceptive experience, the bins allow me to
rlm@572 2718 rapidly find all other similar experience-indices of past
rlm@572 2719 experience that had a very similar proprioceptive configuration.
rlm@572 2720 When looking up a proprioceptive experience, if the smallest bin
rlm@572 2721 does not match any previous experience, then I use successively
rlm@572 2722 larger bins until a match is found or I reach the largest bin.
rlm@572 2723
rlm@572 2724 Given a sequence of proprioceptive input, I use the bins to
rlm@572 2725 generate a set of similar experiences for each input using the
rlm@572 2726 tiered proprioceptive bins.
rlm@572 2727
rlm@572 2728 Finally, to infer sensory data, I select the longest consecutive
rlm@572 2729 chain of experiences that threads through the sets of similar
rlm@572 2730 experiences, starting with the current moment as a root and going
rlm@572 2731 backwards. Consecutive experience means that the experiences appear
rlm@572 2732 next to each other in the experience vector.
rlm@572 2733
rlm@572 2734 A stream of proprioceptive input might be:
rlm@572 2735
rlm@572 2736 \begin{verbatim}
rlm@572 2737 [ flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572 2738 \end{verbatim}
rlm@572 2739
rlm@572 2740 The worm's previous experience of lying on the ground and lifting
rlm@572 2741 its head generates possible interpretations for each frame (the
rlm@572 2742 numbers are experience-indices):
rlm@572 2743
rlm@572 2744 \clearpage
rlm@572 2745
rlm@572 2746 \begin{verbatim}
rlm@572 2747 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572 2748 1 1 1 1 1 1 1 4
rlm@572 2749 2 2 2 2 2 2 2
rlm@572 2750 3 3 3 3 3 3 3
rlm@572 2751 6 6 6 6 6 6 6
rlm@572 2752 7 7 7 7 7 7 7
rlm@572 2753 8 8 8 8 8 8 8
rlm@572 2754 9 9 9 9 9 9 9
rlm@572 2755 \end{verbatim}
rlm@572 2756
rlm@572 2757 These interpretations suggest a new path through phi space:
rlm@572 2758
rlm@572 2759 \begin{verbatim}
rlm@572 2760 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572 2761 6 7 8 9 1 2 3 4
rlm@572 2762 \end{verbatim}
rlm@572 2763
rlm@572 2764 The new path through \(\Phi\)-space is synthesized from two actual
rlm@572 2765 paths that the creature has experienced: the "1-2-3-4" chain and
rlm@572 2766 the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
rlm@572 2767 ends with the worm lifting its head. It originated from a short
rlm@572 2768 training session where the worm rested on the floor for a brief
rlm@572 2769 while and then raised its head. The "6-7-8-9" chain is part of a
rlm@572 2770 longer chain of inactivity where the worm simply rested on the
rlm@572 2771 floor without moving. It is preferred over a "1-2-3" chain (which
rlm@572 2772 also describes inactivity) because it is longer. The main ideas
rlm@572 2773 again:
rlm@572 2774
rlm@572 2775 \begin{itemize}
rlm@572 2776 \item Imagined \(\Phi\)-space paths are synthesized by looping and mixing
rlm@572 2777 previous experiences.
rlm@572 2778
rlm@572 2779 \item Longer experience paths (less edits) are preferred.
rlm@572 2780
rlm@572 2781 \item The present is more important than the past --- more recent
rlm@572 2782 events take precedence in interpretation.
rlm@572 2783 \end{itemize}
rlm@572 2784
rlm@572 2785 This algorithm has three advantages:
rlm@572 2786
rlm@572 2787 \begin{enumerate}
rlm@572 2788 \item It's simple
rlm@572 2789
rlm@572 2790 \item It's very fast -- retrieving possible interpretations takes
rlm@572 2791 constant time. Tracing through chains of interpretations takes
rlm@572 2792 time proportional to the average number of experiences in a
rlm@572 2793 proprioceptive bin. Redundant experiences in \(\Phi\)-space can be
rlm@572 2794 merged to save computation.
rlm@572 2795
rlm@572 2796 \item It protects from wrong interpretations of transient ambiguous
rlm@572 2797 proprioceptive data. For example, if the worm is flat for just
rlm@572 2798 an instant, this flatness will not be interpreted as implying
rlm@572 2799 that the worm has its muscles relaxed, since the flatness is
rlm@572 2800 part of a longer chain which includes a distinct pattern of
rlm@572 2801 muscle activation. Markov chains or other memoryless statistical
rlm@572 2802 models that operate on individual frames may very well make this
rlm@572 2803 mistake.
rlm@572 2804 \end{enumerate}
rlm@572 2805
rlm@572 2806 \begin{listing}
rlm@572 2807 \begin{verbatim}
rlm@572 2808 (defn bin [digits]
rlm@572 2809 (fn [angles]
rlm@572 2810 (->> angles
rlm@572 2811 (flatten)
rlm@572 2812 (map (juxt #(Math/sin %) #(Math/cos %)))
rlm@572 2813 (flatten)
rlm@572 2814 (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
rlm@572 2815
rlm@572 2816 (defn gen-phi-scan
rlm@572 2817 "Nearest-neighbors with binning. Only returns a result if
rlm@572 2818 the proprioceptive data is within 10% of a previously recorded
rlm@572 2819 result in all dimensions."
rlm@572 2820 [phi-space]
rlm@572 2821 (let [bin-keys (map bin [3 2 1])
rlm@572 2822 bin-maps
rlm@572 2823 (map (fn [bin-key]
rlm@572 2824 (group-by
rlm@572 2825 (comp bin-key :proprioception phi-space)
rlm@572 2826 (range (count phi-space)))) bin-keys)
rlm@572 2827 lookups (map (fn [bin-key bin-map]
rlm@572 2828 (fn [proprio] (bin-map (bin-key proprio))))
rlm@572 2829 bin-keys bin-maps)]
rlm@572 2830 (fn lookup [proprio-data]
rlm@572 2831 (set (some #(% proprio-data) lookups)))))
rlm@572 2832 \end{verbatim}
rlm@572 2833 \caption{\label{bin}Program to convert an experience vector into a proprioceptively binned lookup function.}
rlm@572 2834 \end{listing}
rlm@572 2835
rlm@572 2836 \begin{figure}[htb]
rlm@572 2837 \centering
rlm@572 2838 \includegraphics[width=10cm]{./images/film-of-imagination.png}
rlm@572 2839 \caption{\label{phi-space-history-scan}\texttt{longest-thread} finds the longest path of consecutive past experiences to explain proprioceptive worm data from previous data. Here, the film strip represents the creature's previous experience. Sort sequences of memories are spliced together to match the proprioceptive data. Their carry the other senses along with them.}
rlm@572 2840 \end{figure}
rlm@572 2841
rlm@572 2842 \texttt{longest-thread} infers sensory data by stitching together pieces
rlm@572 2843 from previous experience. It prefers longer chains of previous
rlm@572 2844 experience to shorter ones. For example, during training the worm
rlm@572 2845 might rest on the ground for one second before it performs its
rlm@572 2846 exercises. If during recognition the worm rests on the ground for
rlm@572 2847 five seconds, \texttt{longest-thread} will accommodate this five second
rlm@572 2848 rest period by looping the one second rest chain five times.
rlm@572 2849
rlm@572 2850 \texttt{longest-thread} takes time proportional to the average number of
rlm@572 2851 entries in a proprioceptive bin, because for each element in the
rlm@572 2852 starting bin it performs a series of set lookups in the preceding
rlm@572 2853 bins. If the total history is limited, then this takes time
rlm@572 2854 proportional to a only a constant multiple of the number of entries
rlm@572 2855 in the starting bin. This analysis also applies, even if the action
rlm@572 2856 requires multiple longest chains -- it's still the average number
rlm@572 2857 of entries in a proprioceptive bin times the desired chain length.
rlm@572 2858 Because \texttt{longest-thread} is so efficient and simple, I can
rlm@572 2859 interpret worm-actions in real time.
rlm@572 2860
rlm@572 2861 \begin{listing}
rlm@572 2862 \begin{verbatim}
rlm@572 2863 (defn longest-thread
rlm@572 2864 "Find the longest thread from phi-index-sets. The index sets should
rlm@572 2865 be ordered from most recent to least recent."
rlm@572 2866 [phi-index-sets]
rlm@572 2867 (loop [result '()
rlm@572 2868 [thread-bases & remaining :as phi-index-sets] phi-index-sets]
rlm@572 2869 (if (empty? phi-index-sets)
rlm@572 2870 (vec result)
rlm@572 2871 (let [threads
rlm@572 2872 (for [thread-base thread-bases]
rlm@572 2873 (loop [thread (list thread-base)
rlm@572 2874 remaining remaining]
rlm@572 2875 (let [next-index (dec (first thread))]
rlm@572 2876 (cond (empty? remaining) thread
rlm@572 2877 (contains? (first remaining) next-index)
rlm@572 2878 (recur
rlm@572 2879 (cons next-index thread) (rest remaining))
rlm@572 2880 :else thread))))
rlm@572 2881 longest-thread
rlm@572 2882 (reduce (fn [thread-a thread-b]
rlm@572 2883 (if (> (count thread-a) (count thread-b))
rlm@572 2884 thread-a thread-b))
rlm@572 2885 '(nil)
rlm@572 2886 threads)]
rlm@572 2887 (recur (concat longest-thread result)
rlm@572 2888 (drop (count longest-thread) phi-index-sets))))))
rlm@572 2889 \end{verbatim}
rlm@572 2890 \caption{\label{longest-thread}Program to calculate empathy by tracing though \(\Phi\)-space and finding the longest (ie. most coherent) interpretation of the data.}
rlm@572 2891 \end{listing}
rlm@572 2892
rlm@572 2893 There is one final piece, which is to replace missing sensory data
rlm@572 2894 with a best-guess estimate. While I could fill in missing data by
rlm@572 2895 using a gradient over the closest known sensory data points,
rlm@572 2896 averages can be misleading. It is certainly possible to create an
rlm@572 2897 impossible sensory state by averaging two possible sensory states.
rlm@572 2898 For example, consider moving your hand in an arc over your head. If
rlm@572 2899 for some reason you only have the initial and final positions of
rlm@572 2900 this movement in your \(\Phi\)-space, averaging them together will
rlm@572 2901 produce the proprioceptive sensation of having your hand \emph{inside}
rlm@572 2902 your head, which is physically impossible to ever experience
rlm@572 2903 (barring motor adaption illusions). Therefore I simply replicate
rlm@572 2904 the most recent sensory experience to fill in the gaps.
rlm@572 2905
rlm@572 2906 \begin{listing}
rlm@572 2907 \begin{verbatim}
rlm@572 2908 (defn infer-nils
rlm@572 2909 "Replace nils with the next available non-nil element in the
rlm@572 2910 sequence, or barring that, 0."
rlm@572 2911 [s]
rlm@572 2912 (loop [i (dec (count s))
rlm@572 2913 v (transient s)]
rlm@572 2914 (if (zero? i) (persistent! v)
rlm@572 2915 (if-let [cur (v i)]
rlm@572 2916 (if (get v (dec i) 0)
rlm@572 2917 (recur (dec i) v)
rlm@572 2918 (recur (dec i) (assoc! v (dec i) cur)))
rlm@572 2919 (recur i (assoc! v i 0))))))
rlm@572 2920 \end{verbatim}
rlm@572 2921 \caption{\label{infer-nils}Fill in blanks in sensory experience by replicating the most recent experience.}
rlm@572 2922 \end{listing}
rlm@572 2923
rlm@572 2924 \subsection{\texttt{EMPATH} recognizes actions efficiently}
rlm@572 2925 \label{sec-3-5}
rlm@572 2926
rlm@572 2927 To use \texttt{EMPATH} with the worm, I first need to gather a set of
rlm@572 2928 experiences from the worm that includes the actions I want to
rlm@572 2929 recognize. The \texttt{generate-phi-space} program (listing
rlm@572 2930 \ref{generate-phi-space} runs the worm through a series of
rlm@572 2931 exercises and gathers those experiences into a vector. The
rlm@572 2932 \texttt{do-all-the-things} program is a routine expressed in a simple
rlm@572 2933 muscle contraction script language for automated worm control. It
rlm@572 2934 causes the worm to rest, curl, and wiggle over about 700 frames
rlm@572 2935 (approx. 11 seconds).
rlm@572 2936
rlm@572 2937 \begin{listing}
rlm@572 2938 \begin{verbatim}
rlm@572 2939 (def do-all-the-things
rlm@572 2940 (concat
rlm@572 2941 curl-script
rlm@572 2942 [[300 :d-ex 40]
rlm@572 2943 [320 :d-ex 0]]
rlm@572 2944 (shift-script 280 (take 16 wiggle-script))))
rlm@572 2945
rlm@572 2946 (defn generate-phi-space []
rlm@572 2947 (let [experiences (atom [])]
rlm@572 2948 (run-world
rlm@572 2949 (apply-map
rlm@572 2950 worm-world
rlm@572 2951 (merge
rlm@572 2952 (worm-world-defaults)
rlm@572 2953 {:end-frame 700
rlm@572 2954 :motor-control
rlm@572 2955 (motor-control-program worm-muscle-labels do-all-the-things)
rlm@572 2956 :experiences experiences})))
rlm@572 2957 @experiences))
rlm@572 2958 \end{verbatim}
rlm@572 2959 \caption{\label{generate-phi-space}Program to gather the worm's experiences into a vector for further processing. The \texttt{motor-control-program} line uses a motor control script that causes the worm to execute a series of ``exercises'' that include all the action predicates.}
rlm@572 2960 \end{listing}
rlm@572 2961
rlm@572 2962 \begin{listing}
rlm@572 2963 \begin{verbatim}
rlm@572 2964 (defn init []
rlm@572 2965 (def phi-space (generate-phi-space))
rlm@572 2966 (def phi-scan (gen-phi-scan phi-space)))
rlm@572 2967
rlm@572 2968 (defn empathy-demonstration []
rlm@572 2969 (let [proprio (atom ())]
rlm@572 2970 (fn
rlm@572 2971 [experiences text]
rlm@572 2972 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572 2973 (swap! proprio (partial cons phi-indices))
rlm@572 2974 (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572 2975 empathy (mapv phi-space (infer-nils exp-thread))]
rlm@572 2976 (println-repl (vector:last-n exp-thread 22))
rlm@572 2977 (cond
rlm@572 2978 (grand-circle? empathy) (.setText text "Grand Circle")
rlm@572 2979 (curled? empathy) (.setText text "Curled")
rlm@572 2980 (wiggling? empathy) (.setText text "Wiggling")
rlm@572 2981 (resting? empathy) (.setText text "Resting")
rlm@572 2982 :else (.setText text "Unknown")))))))
rlm@572 2983
rlm@572 2984 (defn empathy-experiment [record]
rlm@572 2985 (.start (worm-world :experience-watch (debug-experience-phi)
rlm@572 2986 :record record :worm worm*)))
rlm@572 2987 \end{verbatim}
rlm@572 2988 \caption{\label{empathy-debug}Use \texttt{longest-thread} and a \(\Phi\)-space generated from a short exercise routine to interpret actions during free play.}
rlm@572 2989 \end{listing}
rlm@572 2990
rlm@572 2991 These programs create a test for the empathy system. First, the
rlm@572 2992 worm's \(\Phi\)-space is generated from a simple motor script. Then the
rlm@572 2993 worm is re-created in an environment almost exactly identical to
rlm@572 2994 the testing environment for the action-predicates, with one major
rlm@572 2995 difference : the only sensory information available to the system
rlm@572 2996 is proprioception. From just the proprioception data and
rlm@572 2997 \(\Phi\)-space, \texttt{longest-thread} synthesizes a complete record the last
rlm@572 2998 300 sensory experiences of the worm. These synthesized experiences
rlm@572 2999 are fed directly into the action predicates \texttt{grand-circle?},
rlm@572 3000 \texttt{curled?}, \texttt{wiggling?}, and \texttt{resting?} and their outputs are
rlm@572 3001 printed to the screen at each frame.
rlm@572 3002
rlm@572 3003 The result of running \texttt{empathy-experiment} is that the system is
rlm@572 3004 generally able to interpret worm actions using the action-predicates
rlm@572 3005 on simulated sensory data just as well as with actual data. Figure
rlm@572 3006 \ref{empathy-debug-image} was generated using \texttt{empathy-experiment}:
rlm@572 3007
rlm@572 3008 \begin{figure}[htb]
rlm@572 3009 \centering
rlm@572 3010 \includegraphics[width=10cm]{./images/empathy-1.png}
rlm@572 3011 \caption{\label{empathy-debug-image}From only proprioceptive data, \texttt{EMPATH} was able to infer the complete sensory experience and classify four poses (The last panel shows a composite image of \emph{wiggling}, a dynamic pose.)}
rlm@572 3012 \end{figure}
rlm@572 3013
rlm@572 3014 One way to measure the performance of \texttt{EMPATH} is to compare the
rlm@572 3015 suitability of the imagined sense experience to trigger the same
rlm@572 3016 action predicates as the real sensory experience.
rlm@572 3017
rlm@572 3018 \begin{listing}
rlm@572 3019 \begin{verbatim}
rlm@572 3020 (def worm-action-label
rlm@572 3021 (juxt grand-circle? curled? wiggling?))
rlm@572 3022
rlm@572 3023 (defn compare-empathy-with-baseline [matches]
rlm@572 3024 (let [proprio (atom ())]
rlm@572 3025 (fn
rlm@572 3026 [experiences text]
rlm@572 3027 (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572 3028 (swap! proprio (partial cons phi-indices))
rlm@572 3029 (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572 3030 empathy (mapv phi-space (infer-nils exp-thread))
rlm@572 3031 experience-matches-empathy
rlm@572 3032 (= (worm-action-label experiences)
rlm@572 3033 (worm-action-label empathy))]
rlm@572 3034 (println-repl experience-matches-empathy)
rlm@572 3035 (swap! matches #(conj % experience-matches-empathy)))))))
rlm@572 3036
rlm@572 3037 (defn accuracy [v]
rlm@572 3038 (float (/ (count (filter true? v)) (count v))))
rlm@572 3039
rlm@572 3040 (defn test-empathy-accuracy []
rlm@572 3041 (let [res (atom [])]
rlm@572 3042 (run-world
rlm@572 3043 (worm-world :experience-watch
rlm@572 3044 (compare-empathy-with-baseline res)
rlm@572 3045 :worm worm*))
rlm@572 3046 (accuracy @res)))
rlm@572 3047 \end{verbatim}
rlm@572 3048 \caption{\label{test-empathy-accuracy}Determine how closely empathy approximates actual sensory data.}
rlm@572 3049 \end{listing}
rlm@572 3050
rlm@572 3051 Running \texttt{test-empathy-accuracy} using the very short exercise
rlm@572 3052 program \texttt{do-all-the-things} defined in listing
rlm@572 3053 \ref{generate-phi-space}, and then doing a similar pattern of
rlm@572 3054 activity using manual control of the worm, yields an accuracy of
rlm@572 3055 around 73\%. This is based on very limited worm experience, and
rlm@572 3056 almost all errors are due to the worm's \(\Phi\)-space being too
rlm@572 3057 incomplete to properly interpret common poses. By manually training
rlm@572 3058 the worm for longer using \texttt{init-interactive} defined in listing
rlm@572 3059 \ref{manual-phi-space}, the accuracy dramatically improves:
rlm@572 3060
rlm@572 3061 \begin{listing}
rlm@572 3062 \begin{verbatim}
rlm@572 3063 (defn init-interactive []
rlm@572 3064 (def phi-space
rlm@572 3065 (let [experiences (atom [])]
rlm@572 3066 (run-world
rlm@572 3067 (apply-map
rlm@572 3068 worm-world
rlm@572 3069 (merge
rlm@572 3070 (worm-world-defaults)
rlm@572 3071 {:experiences experiences})))
rlm@572 3072 @experiences))
rlm@572 3073 (def phi-scan (gen-phi-scan phi-space)))
rlm@572 3074 \end{verbatim}
rlm@572 3075 \caption{\label{manual-phi-space}Program to generate \(\Phi\)-space using manual training.}
rlm@572 3076 \end{listing}
rlm@572 3077
rlm@572 3078 \texttt{init-interactive} allows me to take direct control of the worm's
rlm@572 3079 muscles and run it through each characteristic movement I care
rlm@572 3080 about. After about 1 minute of manual training, I was able to
rlm@572 3081 achieve 95\% accuracy on manual testing of the worm using
rlm@572 3082 \texttt{test-empathy-accuracy}. The majority of disagreements are near the
rlm@572 3083 transition boundaries from one type of action to another. During
rlm@572 3084 these transitions the exact label for the action is often unclear,
rlm@572 3085 and disagreement between empathy and experience is practically
rlm@572 3086 irrelevant. Thus, the system's effective identification accuracy is
rlm@572 3087 even higher than 95\%. When I watch this system myself, I generally
rlm@572 3088 see no errors in action identification compared to my own judgment
rlm@572 3089 of what the worm is doing.
rlm@572 3090
rlm@572 3091 \subsection{Digression: Learning touch sensor layout through free play}
rlm@572 3092 \label{sec-3-6}
rlm@572 3093
rlm@572 3094 In the previous chapter I showed how to compute actions in terms of
rlm@572 3095 body-centered predicates, but some of those predicates relied on
rlm@572 3096 the average touch activation of pre-defined regions of the worm's
rlm@572 3097 skin. What if, instead of receiving touch pre-grouped into the six
rlm@572 3098 faces of each worm segment, the true partitioning of the worm's
rlm@572 3099 skin was unknown? This is more similar to how a nerve fiber bundle
rlm@572 3100 might be arranged inside an animal. While two fibers that are close
rlm@572 3101 in a nerve bundle \emph{might} correspond to two touch sensors that are
rlm@572 3102 close together on the skin, the process of taking a complicated
rlm@572 3103 surface and forcing it into essentially a 2D circle requires that
rlm@572 3104 some regions of skin that are close together in the animal end up
rlm@572 3105 far apart in the nerve bundle.
rlm@572 3106
rlm@572 3107 In this chapter I show how to automatically learn the
rlm@572 3108 skin-partitioning of a worm segment by free exploration. As the
rlm@572 3109 worm rolls around on the floor, large sections of its surface get
rlm@572 3110 activated. If the worm has stopped moving, then whatever region of
rlm@572 3111 skin that is touching the floor is probably an important region,
rlm@572 3112 and should be recorded. The code I provide relies on the worm
rlm@572 3113 segment having flat faces, but still demonstrates a primitive kind
rlm@572 3114 of multi-sensory bootstrapping that I find appealing.
rlm@572 3115
rlm@572 3116 \begin{listing}
rlm@572 3117 \begin{verbatim}
rlm@572 3118 (def full-contact [(float 0.0) (float 0.1)])
rlm@572 3119
rlm@572 3120 (defn pure-touch?
rlm@572 3121 "This is worm specific code to determine if a large region of touch
rlm@572 3122 sensors is either all on or all off."
rlm@572 3123 [[coords touch :as touch-data]]
rlm@572 3124 (= (set (map first touch)) (set full-contact)))
rlm@572 3125 \end{verbatim}
rlm@572 3126 \caption{\label{pure-touch}Program to detect whether the worm is in a resting state with one face touching the floor.}
rlm@572 3127 \end{listing}
rlm@572 3128
rlm@572 3129 After collecting these important regions, there will many nearly
rlm@572 3130 similar touch regions. While for some purposes the subtle
rlm@572 3131 differences between these regions will be important, for my
rlm@572 3132 purposes I collapse them into mostly non-overlapping sets using
rlm@572 3133 \texttt{remove-similar} in listing \ref{remove-similar}
rlm@572 3134
rlm@572 3135 \begin{listing}
rlm@572 3136 \begin{verbatim}
rlm@572 3137 (defn remove-similar
rlm@572 3138 [coll]
rlm@572 3139 (loop [result () coll (sort-by (comp - count) coll)]
rlm@572 3140 (if (empty? coll) result
rlm@572 3141 (let [[x & xs] coll
rlm@572 3142 c (count x)]
rlm@572 3143 (if (some
rlm@572 3144 (fn [other-set]
rlm@572 3145 (let [oc (count other-set)]
rlm@572 3146 (< (- (count (union other-set x)) c) (* oc 0.1))))
rlm@572 3147 xs)
rlm@572 3148 (recur result xs)
rlm@572 3149 (recur (cons x result) xs))))))
rlm@572 3150 \end{verbatim}
rlm@572 3151 \caption{\label{remove-similar}Program to take a list of sets of points and ``collapse them'' so that the remaining sets in the list are significantly different from each other. Prefer smaller sets to larger ones.}
rlm@572 3152 \end{listing}
rlm@572 3153
rlm@572 3154 Actually running this simulation is easy given \texttt{CORTEX}'s facilities.
rlm@572 3155
rlm@572 3156 \begin{listing}
rlm@572 3157 \begin{verbatim}
rlm@572 3158 (defn learn-touch-regions []
rlm@572 3159 (let [experiences (atom [])
rlm@572 3160 world (apply-map
rlm@572 3161 worm-world
rlm@572 3162 (assoc (worm-segment-defaults)
rlm@572 3163 :experiences experiences))]
rlm@572 3164 (run-world world)
rlm@572 3165 (->>
rlm@572 3166 @experiences
rlm@572 3167 (drop 175)
rlm@572 3168 ;; access the single segment's touch data
rlm@572 3169 (map (comp first :touch))
rlm@572 3170 ;; only deal with "pure" touch data to determine surfaces
rlm@572 3171 (filter pure-touch?)
rlm@572 3172 ;; associate coordinates with touch values
rlm@572 3173 (map (partial apply zipmap))
rlm@572 3174 ;; select those regions where contact is being made
rlm@572 3175 (map (partial group-by second))
rlm@572 3176 (map #(get % full-contact))
rlm@572 3177 (map (partial map first))
rlm@572 3178 ;; remove redundant/subset regions
rlm@572 3179 (map set)
rlm@572 3180 remove-similar)))
rlm@572 3181
rlm@572 3182 (defn learn-and-view-touch-regions []
rlm@572 3183 (map view-touch-region
rlm@572 3184 (learn-touch-regions)))
rlm@572 3185 \end{verbatim}
rlm@572 3186 \caption{\label{learn-touch}Collect experiences while the worm moves around. Filter the touch sensations by stable ones, collapse similar ones together, and report the regions learned.}
rlm@572 3187 \end{listing}
rlm@572 3188
rlm@572 3189 The only thing remaining to define is the particular motion the worm
rlm@572 3190 must take. I accomplish this with a simple motor control program.
rlm@572 3191
rlm@572 3192 \begin{listing}
rlm@572 3193 \begin{verbatim}
rlm@572 3194 (defn touch-kinesthetics []
rlm@572 3195 [[170 :lift-1 40]
rlm@572 3196 [190 :lift-1 19]
rlm@572 3197 [206 :lift-1 0]
rlm@572 3198
rlm@572 3199 [400 :lift-2 40]
rlm@572 3200 [410 :lift-2 0]
rlm@572 3201
rlm@572 3202 [570 :lift-2 40]
rlm@572 3203 [590 :lift-2 21]
rlm@572 3204 [606 :lift-2 0]
rlm@572 3205
rlm@572 3206 [800 :lift-1 30]
rlm@572 3207 [809 :lift-1 0]
rlm@572 3208
rlm@572 3209 [900 :roll-2 40]
rlm@572 3210 [905 :roll-2 20]
rlm@572 3211 [910 :roll-2 0]
rlm@572 3212
rlm@572 3213 [1000 :roll-2 40]
rlm@572 3214 [1005 :roll-2 20]
rlm@572 3215 [1010 :roll-2 0]
rlm@572 3216
rlm@572 3217 [1100 :roll-2 40]
rlm@572 3218 [1105 :roll-2 20]
rlm@572 3219 [1110 :roll-2 0]
rlm@572 3220 ])
rlm@572 3221 \end{verbatim}
rlm@572 3222 \caption{\label{worm-roll}Motor control program for making the worm roll on the ground. This could also be replaced with random motion.}
rlm@572 3223 \end{listing}
rlm@572 3224
rlm@572 3225
rlm@572 3226 \begin{figure}[htb]
rlm@572 3227 \centering
rlm@572 3228 \includegraphics[width=12cm]{./images/worm-roll.png}
rlm@572 3229 \caption{\label{worm-roll}The small worm rolls around on the floor, driven by the motor control program in listing \ref{worm-roll}.}
rlm@572 3230 \end{figure}
rlm@572 3231
rlm@572 3232 \begin{figure}[htb]
rlm@572 3233 \centering
rlm@572 3234 \includegraphics[width=12cm]{./images/touch-learn.png}
rlm@572 3235 \caption{\label{worm-touch-map}After completing its adventures, the worm now knows how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were deemed important by \texttt{learn-touch-regions}. Each white square in the rectangles above is a cluster of ``related" touch nodes as determined by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces.}
rlm@572 3236 \end{figure}
rlm@572 3237
rlm@572 3238 While simple, \texttt{learn-touch-regions} exploits regularities in both
rlm@572 3239 the worm's physiology and the worm's environment to correctly
rlm@572 3240 deduce that the worm has six sides. Note that \texttt{learn-touch-regions}
rlm@572 3241 would work just as well even if the worm's touch sense data were
rlm@572 3242 completely scrambled. The cross shape is just for convenience. This
rlm@572 3243 example justifies the use of pre-defined touch regions in \texttt{EMPATH}.
rlm@572 3244
rlm@572 3245 \subsection{Recognizing an object using embodied representation}
rlm@572 3246 \label{sec-3-7}
rlm@572 3247
rlm@572 3248 At the beginning of the thesis, I suggested that we might recognize
rlm@572 3249 the chair in Figure \ref{hidden-chair} by imagining ourselves in
rlm@572 3250 the position of the man and realizing that he must be sitting on
rlm@572 3251 something in order to maintain that position. Here, I present a
rlm@572 3252 brief elaboration on how to this might be done.
rlm@572 3253
rlm@572 3254 First, I need the feeling of leaning or resting \emph{on} some other
rlm@572 3255 object that is not the floor. This feeling is easy to describe
rlm@572 3256 using an embodied representation.
rlm@572 3257
rlm@572 3258 \begin{listing}
rlm@572 3259 \begin{verbatim}
rlm@572 3260 (defn draped?
rlm@572 3261 "Is the worm:
rlm@572 3262 -- not flat (the floor is not a 'chair')
rlm@572 3263 -- supported (not using its muscles to hold its position)
rlm@572 3264 -- stable (not changing its position)
rlm@572 3265 -- touching something (must register contact)"
rlm@572 3266 [experiences]
rlm@572 3267 (let [b2-hash (bin 2)
rlm@572 3268 touch (:touch (peek experiences))
rlm@572 3269 total-contact
rlm@572 3270 (reduce
rlm@572 3271 +
rlm@572 3272 (map #(contact all-touch-coordinates %)
rlm@572 3273 (rest touch)))]
rlm@572 3274 (println total-contact)
rlm@572 3275 (and (not (resting? experiences))
rlm@572 3276 (every?
rlm@572 3277 zero?
rlm@572 3278 (-> experiences
rlm@572 3279 (vector:last-n 25)
rlm@572 3280 (#(map :muscle %))
rlm@572 3281 (flatten)))
rlm@572 3282 (-> experiences
rlm@572 3283 (vector:last-n 20)
rlm@572 3284 (#(map (comp b2-hash flatten :proprioception) %))
rlm@572 3285 (set)
rlm@572 3286 (count) (= 1))
rlm@572 3287 (< 0.03 total-contact))))
rlm@572 3288 \end{verbatim}
rlm@572 3289 \caption{\label{draped}Program describing the sense of leaning or resting on something. This involves a relaxed posture, the feeling of touching something, and a period of stability where the worm does not move.}
rlm@572 3290 \end{listing}
rlm@572 3291
rlm@572 3292 \begin{figure}[htb]
rlm@572 3293 \centering
rlm@572 3294 \includegraphics[width=13cm]{./images/draped.png}
rlm@572 3295 \caption{\label{draped-video}The \texttt{draped?} predicate detects the presence of the cube whenever the worm interacts with it. The details of the cube are irrelevant; only the way it influences the worm's body matters. The ``unknown'' label on the fifth frame is due to the fact that the worm is not stationary. \texttt{draped?} will only declare that the worm is draped if it has been still for a while.}
rlm@572 3296 \end{figure}
rlm@572 3297
rlm@572 3298 Though this is a simple example, using the \texttt{draped?} predicate to
rlm@572 3299 detect a cube has interesting advantages. The \texttt{draped?} predicate
rlm@572 3300 describes the cube not in terms of properties that the cube has,
rlm@572 3301 but instead in terms of how the worm interacts with it physically.
rlm@572 3302 This means that the cube can still be detected even if it is not
rlm@572 3303 visible, as long as its influence on the worm's body is visible.
rlm@572 3304
rlm@572 3305 This system will also see the virtual cube created by a
rlm@572 3306 ``mimeworm", which uses its muscles in a very controlled way to
rlm@572 3307 mimic the appearance of leaning on a cube. The system will
rlm@572 3308 anticipate that there is an actual invisible cube that provides
rlm@572 3309 support!
rlm@572 3310
rlm@572 3311 \begin{figure}[htb]
rlm@572 3312 \centering
rlm@572 3313 \includegraphics[width=6cm]{./images/pablo-the-mime.png}
rlm@572 3314 \caption{\label{mime}Can you see the thing that this person is leaning on? What properties does it have, other than how it makes the man's elbow and shoulder feel? I wonder if people who can actually maintain this pose easily still see the support?}
rlm@572 3315 \end{figure}
rlm@572 3316
rlm@572 3317 This makes me wonder about the psychology of actual mimes. Suppose
rlm@572 3318 for a moment that people have something analogous to \(\Phi\)-space and
rlm@572 3319 that one of the ways that they find objects in a scene is by their
rlm@572 3320 relation to other people's bodies. Suppose that a person watches a
rlm@572 3321 person miming an invisible wall. For a person with no experience
rlm@572 3322 with miming, their \(\Phi\)-space will only have entries that describe
rlm@572 3323 the scene with the sensation of their hands touching a wall. This
rlm@572 3324 sensation of touch will create a strong impression of a wall, even
rlm@572 3325 though the wall would have to be invisible. A person with
rlm@572 3326 experience in miming however, will have entries in their \(\Phi\)-space
rlm@572 3327 that describe the wall-miming position without a sense of touch. It
rlm@572 3328 will not seem to such as person that an invisible wall is present,
rlm@572 3329 but merely that the mime is holding out their hands in a special
rlm@572 3330 way. Thus, the theory that humans use something like \(\Phi\)-space
rlm@572 3331 weakly predicts that learning how to mime should break the power of
rlm@572 3332 miming illusions. Most optical illusions still work no matter how
rlm@572 3333 much you know about them, so this proposal would be quite
rlm@572 3334 interesting to test, as it predicts a non-standard result!
rlm@572 3335
rlm@572 3336
rlm@572 3337 \clearpage
rlm@572 3338
rlm@572 3339 \section{Contributions}
rlm@572 3340 \label{sec-4}
rlm@572 3341
rlm@572 3342 The big idea behind this thesis is a new way to represent and
rlm@572 3343 recognize physical actions, which I call \emph{empathic representation}.
rlm@572 3344 Actions are represented as predicates which have access to the
rlm@572 3345 totality of a creature's sensory abilities. To recognize the
rlm@572 3346 physical actions of another creature similar to yourself, you
rlm@572 3347 imagine what they would feel by examining the position of their body
rlm@572 3348 and relating it to your own previous experience.
rlm@572 3349
rlm@572 3350 Empathic representation of physical actions is robust and general.
rlm@572 3351 Because the representation is body-centered, it avoids baking in a
rlm@572 3352 particular viewpoint like you might get from learning from example
rlm@572 3353 videos. Because empathic representation relies on all of a
rlm@572 3354 creature's senses, it can describe exactly what an action \emph{feels
rlm@572 3355 like} without getting caught up in irrelevant details such as visual
rlm@572 3356 appearance. I think it is important that a correct description of
rlm@572 3357 jumping (for example) should not include irrelevant details such as
rlm@572 3358 the color of a person's clothes or skin; empathic representation can
rlm@572 3359 get right to the heart of what jumping is by describing it in terms
rlm@572 3360 of touch, muscle contractions, and a brief feeling of
rlm@572 3361 weightlessness. Empathic representation is very low-level in that it
rlm@572 3362 describes actions using concrete sensory data with little
rlm@572 3363 abstraction, but it has the generality of much more abstract
rlm@572 3364 representations!
rlm@572 3365
rlm@572 3366 Another important contribution of this thesis is the development of
rlm@572 3367 the \texttt{CORTEX} system, a complete environment for creating simulated
rlm@572 3368 creatures. You have seen how to implement five senses: touch,
rlm@572 3369 proprioception, hearing, vision, and muscle tension. You have seen
rlm@572 3370 how to create new creatures using Blender, a 3D modeling tool.
rlm@572 3371
rlm@572 3372 As a minor digression, you also saw how I used \texttt{CORTEX} to enable a
rlm@572 3373 tiny worm to discover the topology of its skin simply by rolling on
rlm@572 3374 the ground. You also saw how to detect objects using only embodied
rlm@572 3375 predicates.
rlm@572 3376
rlm@572 3377 In conclusion, for this thesis I:
rlm@572 3378
rlm@572 3379 \begin{itemize}
rlm@572 3380 \item Developed the idea of embodied representation, which describes
rlm@572 3381 actions that a creature can do in terms of first-person sensory
rlm@572 3382 data.
rlm@572 3383
rlm@572 3384 \item Developed a method of empathic action recognition which uses
rlm@572 3385 previous embodied experience and embodied representation of
rlm@572 3386 actions to greatly constrain the possible interpretations of an
rlm@572 3387 action.
rlm@572 3388
rlm@572 3389 \item Created \texttt{EMPATH}, a program which uses empathic action
rlm@572 3390 recognition to recognize physical actions in a simple model
rlm@572 3391 involving segmented worm-like creatures.
rlm@572 3392
rlm@572 3393 \item Created \texttt{CORTEX}, a comprehensive platform for embodied AI
rlm@572 3394 experiments. It is the base on which \texttt{EMPATH} is built.
rlm@572 3395 \end{itemize}
rlm@572 3396
rlm@572 3397 \clearpage
rlm@572 3398 \appendix
rlm@572 3399
rlm@572 3400 \section{Appendix: \texttt{CORTEX} User Guide}
rlm@572 3401 \label{sec-5}
rlm@572 3402
rlm@572 3403 Those who write a thesis should endeavor to make their code not only
rlm@572 3404 accessible, but actually usable, as a way to pay back the community
rlm@572 3405 that made the thesis possible in the first place. This thesis would
rlm@572 3406 not be possible without Free Software such as jMonkeyEngine3,
rlm@572 3407 Blender, clojure, \texttt{emacs}, \texttt{ffmpeg}, and many other tools. That is
rlm@572 3408 why I have included this user guide, in the hope that someone else
rlm@572 3409 might find \texttt{CORTEX} useful.
rlm@572 3410
rlm@572 3411 \subsection{Obtaining \texttt{CORTEX}}
rlm@572 3412 \label{sec-5-1}
rlm@572 3413
rlm@572 3414 You can get cortex from its mercurial repository at
rlm@572 3415 \url{http://hg.bortreb.com/cortex}. You may also download \texttt{CORTEX}
rlm@572 3416 releases at \url{http://aurellem.org/cortex/releases/}. As a condition of
rlm@572 3417 making this thesis, I have also provided Professor Winston the
rlm@572 3418 \texttt{CORTEX} source, and he knows how to run the demos and get started.
rlm@572 3419 You may also email me at \texttt{cortex@aurellem.org} and I may help where
rlm@572 3420 I can.
rlm@572 3421
rlm@572 3422 \subsection{Running \texttt{CORTEX}}
rlm@572 3423 \label{sec-5-2}
rlm@572 3424
rlm@572 3425 \texttt{CORTEX} comes with README and INSTALL files that will guide you
rlm@572 3426 through installation and running the test suite. In particular you
rlm@572 3427 should look at test \texttt{cortex.test} which contains test suites that
rlm@572 3428 run through all senses and multiple creatures.
rlm@572 3429
rlm@572 3430 \subsection{Creating creatures}
rlm@572 3431 \label{sec-5-3}
rlm@572 3432
rlm@572 3433 Creatures are created using \emph{Blender}, a free 3D modeling program.
rlm@572 3434 You will need Blender version 2.6 when using the \texttt{CORTEX} included
rlm@572 3435 in this thesis. You create a \texttt{CORTEX} creature in a similar manner
rlm@572 3436 to modeling anything in Blender, except that you also create
rlm@572 3437 several trees of empty nodes which define the creature's senses.
rlm@572 3438
rlm@572 3439 \subsubsection{Mass}
rlm@572 3440 \label{sec-5-3-1}
rlm@572 3441
rlm@572 3442 To give an object mass in \texttt{CORTEX}, add a ``mass'' metadata label
rlm@572 3443 to the object with the mass in jMonkeyEngine units. Note that
rlm@572 3444 setting the mass to 0 causes the object to be immovable.
rlm@572 3445
rlm@572 3446 \subsubsection{Joints}
rlm@572 3447 \label{sec-5-3-2}
rlm@572 3448
rlm@572 3449 Joints are created by creating an empty node named \texttt{joints} and
rlm@572 3450 then creating any number of empty child nodes to represent your
rlm@572 3451 creature's joints. The joint will automatically connect the
rlm@572 3452 closest two physical objects. It will help to set the empty node's
rlm@572 3453 display mode to ``Arrows'' so that you can clearly see the
rlm@572 3454 direction of the axes.
rlm@572 3455
rlm@572 3456 Joint nodes should have the following metadata under the ``joint''
rlm@572 3457 label:
rlm@572 3458
rlm@572 3459 \begin{verbatim}
rlm@572 3460 ;; ONE of the following, under the label "joint":
rlm@572 3461 {:type :point}
rlm@572 3462
rlm@572 3463 ;; OR
rlm@572 3464
rlm@572 3465 {:type :hinge
rlm@572 3466 :limit [<limit-low> <limit-high>]
rlm@572 3467 :axis (Vector3f. <x> <y> <z>)}
rlm@572 3468 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572 3469
rlm@572 3470 ;; OR
rlm@572 3471
rlm@572 3472 {:type :cone
rlm@572 3473 :limit-xz <lim-xz>
rlm@572 3474 :limit-xy <lim-xy>
rlm@572 3475 :twist <lim-twist>} ;(use XZY rotation mode in Blender!)
rlm@572 3476 \end{verbatim}
rlm@572 3477
rlm@572 3478 \subsubsection{Eyes}
rlm@572 3479 \label{sec-5-3-3}
rlm@572 3480
rlm@572 3481 Eyes are created by creating an empty node named \texttt{eyes} and then
rlm@572 3482 creating any number of empty child nodes to represent your
rlm@572 3483 creature's eyes.
rlm@572 3484
rlm@572 3485 Eye nodes should have the following metadata under the ``eye''
rlm@572 3486 label:
rlm@572 3487
rlm@572 3488 \begin{verbatim}
rlm@572 3489 {:red <red-retina-definition>
rlm@572 3490 :blue <blue-retina-definition>
rlm@572 3491 :green <green-retina-definition>
rlm@572 3492 :all <all-retina-definition>
rlm@572 3493 (<0xrrggbb> <custom-retina-image>)...
rlm@572 3494 }
rlm@572 3495 \end{verbatim}
rlm@572 3496
rlm@572 3497 Any of the color channels may be omitted. You may also include
rlm@572 3498 your own color selectors, and in fact :red is equivalent to
rlm@572 3499 0xFF0000 and so forth. The eye will be placed at the same position
rlm@572 3500 as the empty node and will bind to the neatest physical object.
rlm@572 3501 The eye will point outward from the X-axis of the node, and ``up''
rlm@572 3502 will be in the direction of the X-axis of the node. It will help
rlm@572 3503 to set the empty node's display mode to ``Arrows'' so that you can
rlm@572 3504 clearly see the direction of the axes.
rlm@572 3505
rlm@572 3506 Each retina file should contain white pixels wherever you want to be
rlm@572 3507 sensitive to your chosen color. If you want the entire field of
rlm@572 3508 view, specify :all of 0xFFFFFF and a retinal map that is entirely
rlm@572 3509 white.
rlm@572 3510
rlm@572 3511 Here is a sample retinal map:
rlm@572 3512
rlm@572 3513 \begin{figure}[H]
rlm@572 3514 \centering
rlm@572 3515 \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572 3516 \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572 3517 \end{figure}
rlm@572 3518
rlm@572 3519 \subsubsection{Hearing}
rlm@572 3520 \label{sec-5-3-4}
rlm@572 3521
rlm@572 3522 Ears are created by creating an empty node named \texttt{ears} and then
rlm@572 3523 creating any number of empty child nodes to represent your
rlm@572 3524 creature's ears.
rlm@572 3525
rlm@572 3526 Ear nodes do not require any metadata.
rlm@572 3527
rlm@572 3528 The ear will bind to and follow the closest physical node.
rlm@572 3529
rlm@572 3530 \subsubsection{Touch}
rlm@572 3531 \label{sec-5-3-5}
rlm@572 3532
rlm@572 3533 Touch is handled similarly to mass. To make a particular object
rlm@572 3534 touch sensitive, add metadata of the following form under the
rlm@572 3535 object's ``touch'' metadata field:
rlm@572 3536
rlm@572 3537 \begin{verbatim}
rlm@572 3538 <touch-UV-map-file-name>
rlm@572 3539 \end{verbatim}
rlm@572 3540
rlm@572 3541 You may also include an optional ``scale'' metadata number to
rlm@572 3542 specify the length of the touch feelers. The default is \(0.1\),
rlm@572 3543 and this is generally sufficient.
rlm@572 3544
rlm@572 3545 The touch UV should contain white pixels for each touch sensor.
rlm@572 3546
rlm@572 3547 Here is an example touch-uv map that approximates a human finger,
rlm@572 3548 and its corresponding model.
rlm@572 3549
rlm@572 3550 \begin{figure}[htb]
rlm@572 3551 \centering
rlm@572 3552 \includegraphics[width=9cm]{./images/finger-UV.png}
rlm@572 3553 \caption{\label{guide-fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572 3554 \end{figure}
rlm@572 3555
rlm@572 3556 \begin{figure}[htb]
rlm@572 3557 \centering
rlm@572 3558 \includegraphics[width=9cm]{./images/finger-1.png}
rlm@572 3559 \caption{\label{guide-fingertip}The fingertip UV-image form above applied to a simple model of a fingertip.}
rlm@572 3560 \end{figure}
rlm@572 3561
rlm@572 3562 \subsubsection{Proprioception}
rlm@572 3563 \label{sec-5-3-6}
rlm@572 3564
rlm@572 3565 Proprioception is tied to each joint node -- nothing special must
rlm@572 3566 be done in a Blender model to enable proprioception other than
rlm@572 3567 creating joint nodes.
rlm@572 3568
rlm@572 3569 \subsubsection{Muscles}
rlm@572 3570 \label{sec-5-3-7}
rlm@572 3571
rlm@572 3572 Muscles are created by creating an empty node named \texttt{muscles} and
rlm@572 3573 then creating any number of empty child nodes to represent your
rlm@572 3574 creature's muscles.
rlm@572 3575
rlm@572 3576
rlm@572 3577 Muscle nodes should have the following metadata under the
rlm@572 3578 ``muscle'' label:
rlm@572 3579
rlm@572 3580 \begin{verbatim}
rlm@572 3581 <muscle-profile-file-name>
rlm@572 3582 \end{verbatim}
rlm@572 3583
rlm@572 3584 Muscles should also have a ``strength'' metadata entry describing
rlm@572 3585 the muscle's total strength at full activation.
rlm@572 3586
rlm@572 3587 Muscle profiles are simple images that contain the relative amount
rlm@572 3588 of muscle power in each simulated alpha motor neuron. The width of
rlm@572 3589 the image is the total size of the motor pool, and the redness of
rlm@572 3590 each neuron is the relative power of that motor pool.
rlm@572 3591
rlm@572 3592 While the profile image can have any dimensions, only the first
rlm@572 3593 line of pixels is used to define the muscle. Here is a sample
rlm@572 3594 muscle profile image that defines a human-like muscle.
rlm@572 3595
rlm@572 3596 \begin{figure}[htb]
rlm@572 3597 \centering
rlm@572 3598 \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572 3599 \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572 3600 \end{figure}
rlm@572 3601
rlm@572 3602 Muscles twist the nearest physical object about the muscle node's
rlm@572 3603 Z-axis. I recommend using the ``Single Arrow'' display mode for
rlm@572 3604 muscles and using the right hand rule to determine which way the
rlm@572 3605 muscle will twist. To make a segment that can twist in multiple
rlm@572 3606 directions, create multiple, differently aligned muscles.
rlm@572 3607
rlm@572 3608 \subsection{\texttt{CORTEX} API}
rlm@572 3609 \label{sec-5-4}
rlm@572 3610
rlm@572 3611 These are the some functions exposed by \texttt{CORTEX} for creating
rlm@572 3612 worlds and simulating creatures. These are in addition to
rlm@572 3613 jMonkeyEngine3's extensive library, which is documented elsewhere.
rlm@572 3614
rlm@572 3615 \subsubsection{Simulation}
rlm@572 3616 \label{sec-5-4-1}
rlm@572 3617 \begin{description}
rlm@572 3618 \item[{\texttt{(world root-node key-map setup-fn update-fn)}}] create
rlm@572 3619 a simulation.
rlm@572 3620 \begin{description}
rlm@572 3621 \item[{\emph{root-node} }] a \texttt{com.jme3.scene.Node} object which
rlm@572 3622 contains all of the objects that should be in the
rlm@572 3623 simulation.
rlm@572 3624
rlm@572 3625 \item[{\emph{key-map} }] a map from strings describing keys to
rlm@572 3626 functions that should be executed whenever that key is
rlm@572 3627 pressed. the functions should take a SimpleApplication
rlm@572 3628 object and a boolean value. The SimpleApplication is the
rlm@572 3629 current simulation that is running, and the boolean is true
rlm@572 3630 if the key is being pressed, and false if it is being
rlm@572 3631 released. As an example,
rlm@572 3632 \begin{verbatim}
rlm@572 3633 {"key-j" (fn [game value] (if value (println "key j pressed")))}
rlm@572 3634 \end{verbatim}
rlm@572 3635 is a valid key-map which will cause the simulation to print
rlm@572 3636 a message whenever the 'j' key on the keyboard is pressed.
rlm@572 3637
rlm@572 3638 \item[{\emph{setup-fn} }] a function that takes a \texttt{SimpleApplication}
rlm@572 3639 object. It is called once when initializing the simulation.
rlm@572 3640 Use it to create things like lights, change the gravity,
rlm@572 3641 initialize debug nodes, etc.
rlm@572 3642
rlm@572 3643 \item[{\emph{update-fn} }] this function takes a \texttt{SimpleApplication}
rlm@572 3644 object and a float and is called every frame of the
rlm@572 3645 simulation. The float tells how many seconds is has been
rlm@572 3646 since the last frame was rendered, according to whatever
rlm@572 3647 clock jme is currently using. The default is to use IsoTimer
rlm@572 3648 which will result in this value always being the same.
rlm@572 3649 \end{description}
rlm@572 3650
rlm@572 3651 \item[{\texttt{(position-camera world position rotation)}}] set the position
rlm@572 3652 of the simulation's main camera.
rlm@572 3653
rlm@572 3654 \item[{\texttt{(enable-debug world)}}] turn on debug wireframes for each
rlm@572 3655 simulated object.
rlm@572 3656
rlm@572 3657 \item[{\texttt{(set-gravity world gravity)}}] set the gravity of a running
rlm@572 3658 simulation.
rlm@572 3659
rlm@572 3660 \item[{\texttt{(box length width height \& \{options\})}}] create a box in the
rlm@572 3661 simulation. Options is a hash map specifying texture, mass,
rlm@572 3662 etc. Possible options are \texttt{:name}, \texttt{:color}, \texttt{:mass},
rlm@572 3663 \texttt{:friction}, \texttt{:texture}, \texttt{:material}, \texttt{:position},
rlm@572 3664 \texttt{:rotation}, \texttt{:shape}, and \texttt{:physical?}.
rlm@572 3665
rlm@572 3666 \item[{\texttt{(sphere radius \& \{options\})}}] create a sphere in the simulation.
rlm@572 3667 Options are the same as in \texttt{box}.
rlm@572 3668
rlm@572 3669 \item[{\texttt{(load-blender-model file-name)}}] create a node structure
rlm@572 3670 representing the model described in a Blender file.
rlm@572 3671
rlm@572 3672 \item[{\texttt{(light-up-everything world)}}] distribute a standard compliment
rlm@572 3673 of lights throughout the simulation. Should be adequate for most
rlm@572 3674 purposes.
rlm@572 3675
rlm@572 3676 \item[{\texttt{(node-seq node)}}] return a recursive list of the node's
rlm@572 3677 children.
rlm@572 3678
rlm@572 3679 \item[{\texttt{(nodify name children)}}] construct a node given a node-name and
rlm@572 3680 desired children.
rlm@572 3681
rlm@572 3682 \item[{\texttt{(add-element world element)}}] add an object to a running world
rlm@572 3683 simulation.
rlm@572 3684
rlm@572 3685 \item[{\texttt{(set-accuracy world accuracy)}}] change the accuracy of the
rlm@572 3686 world's physics simulator.
rlm@572 3687
rlm@572 3688 \item[{\texttt{(asset-manager)}}] get an \emph{AssetManager}, a jMonkeyEngine
rlm@572 3689 construct that is useful for loading textures and is required
rlm@572 3690 for smooth interaction with jMonkeyEngine library functions.
rlm@572 3691
rlm@572 3692 \item[{\texttt{(load-bullet)} }] unpack native libraries and initialize the
rlm@572 3693 bullet physics subsystem. This function is required before
rlm@572 3694 other world building functions are called.
rlm@572 3695 \end{description}
rlm@572 3696
rlm@572 3697 \subsubsection{Creature Manipulation / Import}
rlm@572 3698 \label{sec-5-4-2}
rlm@572 3699
rlm@572 3700 \begin{description}
rlm@572 3701 \item[{\texttt{(body! creature)}}] give the creature a physical body.
rlm@572 3702
rlm@572 3703 \item[{\texttt{(vision! creature)}}] give the creature a sense of vision.
rlm@572 3704 Returns a list of functions which will each, when called
rlm@572 3705 during a simulation, return the vision data for the channel of
rlm@572 3706 one of the eyes. The functions are ordered depending on the
rlm@572 3707 alphabetical order of the names of the eye nodes in the
rlm@572 3708 Blender file. The data returned by the functions is a vector
rlm@572 3709 containing the eye's \emph{topology}, a vector of coordinates, and
rlm@572 3710 the eye's \emph{data}, a vector of RGB values filtered by the eye's
rlm@572 3711 sensitivity.
rlm@572 3712
rlm@572 3713 \item[{\texttt{(hearing! creature)}}] give the creature a sense of hearing.
rlm@572 3714 Returns a list of functions, one for each ear, that when
rlm@572 3715 called will return a frame's worth of hearing data for that
rlm@572 3716 ear. The functions are ordered depending on the alphabetical
rlm@572 3717 order of the names of the ear nodes in the Blender file. The
rlm@572 3718 data returned by the functions is an array of PCM (pulse code
rlm@572 3719 modulated) wav data.
rlm@572 3720
rlm@572 3721 \item[{\texttt{(touch! creature)}}] give the creature a sense of touch. Returns
rlm@572 3722 a single function that must be called with the \emph{root node} of
rlm@572 3723 the world, and which will return a vector of \emph{touch-data}
rlm@572 3724 one entry for each touch sensitive component, each entry of
rlm@572 3725 which contains a \emph{topology} that specifies the distribution of
rlm@572 3726 touch sensors, and the \emph{data}, which is a vector of
rlm@572 3727 \texttt{[activation, length]} pairs for each touch hair.
rlm@572 3728
rlm@572 3729 \item[{\texttt{(proprioception! creature)}}] give the creature the sense of
rlm@572 3730 proprioception. Returns a list of functions, one for each
rlm@572 3731 joint, that when called during a running simulation will
rlm@572 3732 report the \texttt{[heading, pitch, roll]} of the joint.
rlm@572 3733
rlm@572 3734 \item[{\texttt{(movement! creature)}}] give the creature the power of movement.
rlm@572 3735 Creates a list of functions, one for each muscle, that when
rlm@572 3736 called with an integer, will set the recruitment of that
rlm@572 3737 muscle to that integer, and will report the current power
rlm@572 3738 being exerted by the muscle. Order of muscles is determined by
rlm@572 3739 the alphabetical sort order of the names of the muscle nodes.
rlm@572 3740 \end{description}
rlm@572 3741
rlm@572 3742 \subsubsection{Visualization/Debug}
rlm@572 3743 \label{sec-5-4-3}
rlm@572 3744
rlm@572 3745 \begin{description}
rlm@572 3746 \item[{\texttt{(view-vision)}}] create a function that when called with a list
rlm@572 3747 of visual data returned from the functions made by \texttt{vision!},
rlm@572 3748 will display that visual data on the screen.
rlm@572 3749
rlm@572 3750 \item[{\texttt{(view-hearing)}}] same as \texttt{view-vision} but for hearing.
rlm@572 3751
rlm@572 3752 \item[{\texttt{(view-touch)}}] same as \texttt{view-vision} but for touch.
rlm@572 3753
rlm@572 3754 \item[{\texttt{(view-proprioception)}}] same as \texttt{view-vision} but for
rlm@572 3755 proprioception.
rlm@572 3756
rlm@572 3757 \item[{\texttt{(view-movement)}}] same as \texttt{view-vision} but for muscles.
rlm@572 3758
rlm@572 3759 \item[{\texttt{(view anything)}}] \texttt{view} is a polymorphic function that allows
rlm@572 3760 you to inspect almost anything you could reasonably expect to
rlm@572 3761 be able to ``see'' in \texttt{CORTEX}.
rlm@572 3762
rlm@572 3763 \item[{\texttt{(text anything)}}] \texttt{text} is a polymorphic function that allows
rlm@572 3764 you to convert practically anything into a text string.
rlm@572 3765
rlm@572 3766 \item[{\texttt{(println-repl anything)}}] print messages to clojure's repl
rlm@572 3767 instead of the simulation's terminal window.
rlm@572 3768
rlm@572 3769 \item[{\texttt{(mega-import-jme3)}}] for experimenting at the REPL. This
rlm@572 3770 function will import all jMonkeyEngine3 classes for immediate
rlm@572 3771 use.
rlm@572 3772
rlm@572 3773 \item[{\texttt{(display-dilated-time world timer)}}] Shows the time as it is
rlm@572 3774 flowing in the simulation on a HUD display.
rlm@572 3775 \end{description}