rlm@572: 
rlm@572: \section{Empathy $\backslash$ Embodiment: problem solving strategies}
rlm@572: \label{sec-1}
rlm@572: 
rlm@572: By the time you have read this thesis, you will understand a novel
rlm@572: approach to representing and recognizing physical actions using
rlm@572: embodiment and empathy. You will also see one way to efficiently
rlm@572: implement physical empathy for embodied creatures. Finally, you will
rlm@572: become familiar with \texttt{CORTEX}, a system for designing and simulating
rlm@572: creatures with rich senses, which I have designed as a library that
rlm@572: you can use in your own research. Note that I \emph{do not} process video
rlm@572: directly --- I start with knowledge of the positions of a creature's
rlm@572: body parts and work from there.
rlm@572: 
rlm@572: This is the core vision of my thesis: That one of the important ways
rlm@572: in which we understand others is by imagining ourselves in their
rlm@572: position and empathically feeling experiences relative to our own
rlm@572: bodies. By understanding events in terms of our own previous
rlm@572: corporeal experience, we greatly constrain the possibilities of what
rlm@572: would otherwise be an unwieldy exponential search. This extra
rlm@572: constraint can be the difference between easily understanding what
rlm@572: is happening in a video and being completely lost in a sea of
rlm@572: incomprehensible color and movement.
rlm@572: 
rlm@572: \subsection{The problem: recognizing actions is hard!}
rlm@572: \label{sec-1-1}
rlm@572: 
rlm@572: Examine figure \ref{cat-drink}. What is happening? As you, and
rlm@572: indeed very young children, can easily determine, this is an image
rlm@572: of drinking.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/cat-drinking.jpg}
rlm@572: \caption{\label{cat-drink}A cat drinking some water. Identifying this action is beyond the capabilities of existing computer vision systems.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Nevertheless, it is beyond the state of the art for a computer
rlm@572: vision program to describe what's happening in this image. Part of
rlm@572: the problem is that many computer vision systems focus on
rlm@572: pixel-level details or comparisons to example images (such as
rlm@572: \cite{volume-action-recognition}), but the 3D world is so variable
rlm@572: that it is hard to describe the world in terms of possible images.
rlm@572: 
rlm@572: In fact, the contents of a scene may have much less to do with
rlm@572: pixel probabilities than with recognizing various affordances:
rlm@572: things you can move, objects you can grasp, spaces that can be
rlm@572: filled . For example, what processes might enable you to see the
rlm@572: chair in figure \ref{hidden-chair}?
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/fat-person-sitting-at-desk.jpg}
rlm@572: \caption{\label{hidden-chair}The chair in this image is quite obvious to humans, but it can't be found by any modern computer vision program.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Finally, how is it that you can easily tell the difference between
rlm@572: how the girl's \emph{muscles} are working in figure \ref{girl}?
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/wall-push.png}
rlm@572: \caption{\label{girl}The mysterious ``common sense'' appears here as you are able to discern the difference in how the girl's arm muscles are activated between the two images. When you compare these two images, do you feel something in your own arm muscles?}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Each of these examples tells us something about what might be going
rlm@572: on in our minds as we easily solve these recognition problems:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item The hidden chair shows us that we are strongly triggered by cues
rlm@572: relating to the position of human bodies, and that we can
rlm@572: determine the overall physical configuration of a human body even
rlm@572: if much of that body is occluded.
rlm@572: 
rlm@572: \item The picture of the girl pushing against the wall tells us that we
rlm@572: have common sense knowledge about the kinetics of our own bodies.
rlm@572: We know well how our muscles would have to work to maintain us in
rlm@572: most positions, and we can easily project this self-knowledge to
rlm@572: imagined positions triggered by images of the human body.
rlm@572: 
rlm@572: \item The cat tells us that imagination of some kind plays an important
rlm@572: role in understanding actions. The question is: Can we be more
rlm@572: precise about what sort of imagination is required to understand
rlm@572: these actions?
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \subsection{A step forward: the sensorimotor-centered approach}
rlm@572: \label{sec-1-2}
rlm@572: 
rlm@572: In this thesis, I explore the idea that our knowledge of our own
rlm@572: bodies, combined with our own rich senses, enables us to recognize
rlm@572: the actions of others.
rlm@572: 
rlm@572: For example, I think humans are able to label the cat video as
rlm@572: ``drinking'' because they imagine \emph{themselves} as the cat, and
rlm@572: imagine putting their face up against a stream of water and
rlm@572: sticking out their tongue. In that imagined world, they can feel
rlm@572: the cool water hitting their tongue, and feel the water entering
rlm@572: their body, and are able to recognize that \emph{feeling} as drinking.
rlm@572: So, the label of the action is not really in the pixels of the
rlm@572: image, but is found clearly in a simulation / recollection inspired
rlm@572: by those pixels. An imaginative system, having been trained on
rlm@572: drinking and non-drinking examples and learning that the most
rlm@572: important component of drinking is the feeling of water flowing
rlm@572: down one's throat, would analyze a video of a cat drinking in the
rlm@572: following manner:
rlm@572: 
rlm@572: \begin{enumerate}
rlm@572: \item Create a physical model of the video by putting a ``fuzzy''
rlm@572: model of its own body in place of the cat. Possibly also create
rlm@572: a simulation of the stream of water.
rlm@572: 
rlm@572: \item Play out this simulated scene and generate imagined sensory
rlm@572: experience. This will include relevant muscle contractions, a
rlm@572: close up view of the stream from the cat's perspective, and most
rlm@572: importantly, the imagined feeling of water entering the mouth.
rlm@572: The imagined sensory experience can come from a simulation of
rlm@572: the event, but can also be pattern-matched from previous,
rlm@572: similar embodied experience.
rlm@572: 
rlm@572: \item The action is now easily identified as drinking by the sense of
rlm@572: taste alone. The other senses (such as the tongue moving in and
rlm@572: out) help to give plausibility to the simulated action. Note that
rlm@572: the sense of vision, while critical in creating the simulation,
rlm@572: is not critical for identifying the action from the simulation.
rlm@572: \end{enumerate}
rlm@572: 
rlm@572: For the chair examples, the process is even easier:
rlm@572: 
rlm@572: \begin{enumerate}
rlm@572: \item Align a model of your body to the person in the image.
rlm@572: 
rlm@572: \item Generate proprioceptive sensory data from this alignment.
rlm@572: 
rlm@572: \item Use the imagined proprioceptive data as a key to lookup related
rlm@572: sensory experience associated with that particular proprioceptive
rlm@572: feeling.
rlm@572: 
rlm@572: \item Retrieve the feeling of your bottom resting on a surface, your
rlm@572: knees bent, and your leg muscles relaxed.
rlm@572: 
rlm@572: \item This sensory information is consistent with your \texttt{sitting?}
rlm@572: sensory predicate, so you (and the entity in the image) must be
rlm@572: sitting.
rlm@572: 
rlm@572: \item There must be a chair-like object since you are sitting.
rlm@572: \end{enumerate}
rlm@572: 
rlm@572: Empathy offers yet another alternative to the age-old AI
rlm@572: representation question: ``What is a chair?'' --- A chair is the
rlm@572: feeling of sitting!
rlm@572: 
rlm@572: One powerful advantage of empathic problem solving is that it
rlm@572: factors the action recognition problem into two easier problems. To
rlm@572: use empathy, you need an \emph{aligner}, which takes the video and a
rlm@572: model of your body, and aligns the model with the video. Then, you
rlm@572: need a \emph{recognizer}, which uses the aligned model to interpret the
rlm@572: action. The power in this method lies in the fact that you describe
rlm@572: all actions from a body-centered viewpoint. You are less tied to
rlm@572: the particulars of any visual representation of the actions. If you
rlm@572: teach the system what ``running'' is, and you have a good enough
rlm@572: aligner, the system will from then on be able to recognize running
rlm@572: from any point of view -- even strange points of view like above or
rlm@572: underneath the runner. This is in contrast to action recognition
rlm@572: schemes that try to identify actions using a non-embodied approach.
rlm@572: If these systems learn about running as viewed from the side, they
rlm@572: will not automatically be able to recognize running from any other
rlm@572: viewpoint.
rlm@572: 
rlm@572: Another powerful advantage is that using the language of multiple
rlm@572: body-centered rich senses to describe body-centered actions offers
rlm@572: a massive boost in descriptive capability. Consider how difficult
rlm@572: it would be to compose a set of HOG (Histogram of Oriented
rlm@572: Gradients) filters to describe the action of a simple worm-creature
rlm@572: ``curling'' so that its head touches its tail, and then behold the
rlm@572: simplicity of describing thus action in a language designed for the
rlm@572: task (listing \ref{grand-circle-intro}):
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn grand-circle?
rlm@572:   "Does the worm form a majestic circle (one end touching the other)?"
rlm@572:   [experiences]
rlm@572:   (and (curled? experiences)
rlm@572:        (let [worm-touch (:touch (peek experiences))
rlm@572:              tail-touch (worm-touch 0)
rlm@572:              head-touch (worm-touch 4)]
rlm@572:          (and (< 0.2 (contact worm-segment-bottom-tip tail-touch))
rlm@572:               (< 0.2 (contact worm-segment-top-tip    head-touch))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{grand-circle-intro}Body-centered actions are best expressed in a body-centered language. This code detects when the worm has curled into a full circle. Imagine how you would replicate this functionality using low-level pixel features such as HOG filters!}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsection{\texttt{EMPATH} recognizes actions using empathy}
rlm@572: \label{sec-1-3}
rlm@572: 
rlm@572: Exploring these ideas further demands a concrete implementation, so
rlm@572: first, I built a system for constructing virtual creatures with
rlm@572: physiologically plausible sensorimotor systems and detailed
rlm@572: environments. The result is \texttt{CORTEX}, which I describe in chapter
rlm@572: \ref{sec-2}.
rlm@572: 
rlm@572: Next, I wrote routines which enabled a simple worm-like creature to
rlm@572: infer the actions of a second worm-like creature, using only its
rlm@572: own prior sensorimotor experiences and knowledge of the second
rlm@572: worm's joint positions. This program, \texttt{EMPATH}, is described in
rlm@572: chapter \ref{sec-3}. It's main components are:
rlm@572: 
rlm@572: \begin{description}
rlm@572: \item[{Embodied Action Definitions}] Many otherwise complicated actions
rlm@572: are easily described in the language of a full suite of
rlm@572: body-centered, rich senses and experiences. For example,
rlm@572: drinking is the feeling of water flowing down your throat, and
rlm@572: cooling your insides. It's often accompanied by bringing your
rlm@572: hand close to your face, or bringing your face close to water.
rlm@572: Sitting down is the feeling of bending your knees, activating
rlm@572: your quadriceps, then feeling a surface with your bottom and
rlm@572: relaxing your legs. These body-centered action descriptions
rlm@572: can be either learned or hard coded.
rlm@572: 
rlm@572: \item[{Guided Play     }] The creature moves around and experiences the
rlm@572: world through its unique perspective. As the creature moves,
rlm@572: it gathers experiences that satisfy the embodied action
rlm@572: definitions.
rlm@572: 
rlm@572: \item[{Posture Imitation}] When trying to interpret a video or image,
rlm@572: the creature takes a model of itself and aligns it with
rlm@572: whatever it sees. This alignment might even cross species, as
rlm@572: when humans try to align themselves with things like ponies,
rlm@572: dogs, or other humans with a different body type.
rlm@572: 
rlm@572: \item[{Empathy         }] The alignment triggers associations with
rlm@572: sensory data from prior experiences. For example, the
rlm@572: alignment itself easily maps to proprioceptive data. Any
rlm@572: sounds or obvious skin contact in the video can to a lesser
rlm@572: extent trigger previous experience keyed to hearing or touch.
rlm@572: Segments of previous experiences gained from play are stitched
rlm@572: together to form a coherent and complete sensory portrait of
rlm@572: the scene.
rlm@572: 
rlm@572: \item[{Recognition}] With the scene described in terms of remembered
rlm@572: first person sensory events, the creature can now run its
rlm@572: action-definition programs (such as the one in listing
rlm@572: \ref{grand-circle-intro}) on this synthesized sensory data,
rlm@572: just as it would if it were actually experiencing the scene
rlm@572: first-hand. If previous experience has been accurately
rlm@572: retrieved, and if it is analogous enough to the scene, then
rlm@572: the creature will correctly identify the action in the scene.
rlm@572: \end{description}
rlm@572: 
rlm@572: My program \texttt{EMPATH} uses this empathic problem solving technique
rlm@572: to interpret the actions of a simple, worm-like creature. 
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/worm-intro-white.png}
rlm@572: \caption{\label{worm-intro}The worm performs many actions during free play such as curling, wiggling, and resting.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/worm-poses.png}
rlm@572: \caption{\label{worm-recognition-intro}\texttt{EMPATH} recognized and classified each of these poses by inferring the complete sensory experience from proprioceptive data.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{Main Results}
rlm@572: \label{sec-1-3-1}
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item After one-shot supervised training, \texttt{EMPATH} was able to
rlm@572: recognize a wide variety of static poses and dynamic
rlm@572: actions---ranging from curling in a circle to wiggling with a
rlm@572: particular frequency --- with 95$\backslash$ accuracy.
rlm@572: 
rlm@572: \item These results were completely independent of viewing angle
rlm@572: because the underlying body-centered language fundamentally is
rlm@572: independent; once an action is learned, it can be recognized
rlm@572: equally well from any viewing angle.
rlm@572: 
rlm@572: \item \texttt{EMPATH} is surprisingly short; the sensorimotor-centered
rlm@572: language provided by \texttt{CORTEX} resulted in extremely economical
rlm@572: recognition routines --- about 500 lines in all --- suggesting
rlm@572: that such representations are very powerful, and often
rlm@572: indispensable for the types of recognition tasks considered here.
rlm@572: 
rlm@572: \item For expediency's sake, I relied on direct knowledge of joint
rlm@572: positions in this proof of concept. However, I believe that the
rlm@572: structure of \texttt{EMPATH} and \texttt{CORTEX} will make future work to
rlm@572: enable video analysis much easier than it would otherwise be.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \subsection{\texttt{EMPATH} is built on \texttt{CORTEX}, a creature builder.}
rlm@572: \label{sec-1-4}
rlm@572: 
rlm@572: I built \texttt{CORTEX} to be a general AI research platform for doing
rlm@572: experiments involving multiple rich senses and a wide variety and
rlm@572: number of creatures. I intend it to be useful as a library for many
rlm@572: more projects than just this thesis. \texttt{CORTEX} was necessary to meet
rlm@572: a need among AI researchers at CSAIL and beyond, which is that
rlm@572: people often will invent wonderful ideas that are best expressed in
rlm@572: the language of creatures and senses, but in order to explore those
rlm@572: ideas they must first build a platform in which they can create
rlm@572: simulated creatures with rich senses! There are many ideas that
rlm@572: would be simple to execute (such as \texttt{EMPATH} or Larson's
rlm@572: self-organizing maps (\cite{larson-symbols})), but attached to them
rlm@572: is the multi-month effort to make a good creature simulator. Often,
rlm@572: that initial investment of time proves to be too much, and the
rlm@572: project must make do with a lesser environment or be abandoned
rlm@572: entirely.
rlm@572: 
rlm@572: \texttt{CORTEX} is well suited as an environment for embodied AI research
rlm@572: for three reasons:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item You can design new creatures using Blender (\cite{blender}), a
rlm@572: popular, free 3D modeling program. Each sense can be specified
rlm@572: using special Blender nodes with biologically inspired
rlm@572: parameters. You need not write any code to create a creature, and
rlm@572: can use a wide library of pre-existing Blender models as a base
rlm@572: for your own creatures.
rlm@572: 
rlm@572: \item \texttt{CORTEX} implements a wide variety of senses: touch,
rlm@572: proprioception, vision, hearing, and muscle tension. Complicated
rlm@572: senses like touch and vision involve multiple sensory elements
rlm@572: embedded in a 2D surface. You have complete control over the
rlm@572: distribution of these sensor elements through the use of simple
rlm@572: image files. \texttt{CORTEX} implements more comprehensive hearing than
rlm@572: any other creature simulation system available.
rlm@572: 
rlm@572: \item \texttt{CORTEX} supports any number of creatures and any number of
rlm@572: senses. Time in \texttt{CORTEX} dilates so that the simulated creatures
rlm@572: always perceive a perfectly smooth flow of time, regardless of
rlm@572: the actual computational load.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \texttt{CORTEX} is built on top of \texttt{jMonkeyEngine3}
rlm@572: (\cite{jmonkeyengine}), which is a video game engine designed to
rlm@572: create cross-platform 3D desktop games. \texttt{CORTEX} is mainly written
rlm@572: in clojure, a dialect of \texttt{LISP} that runs on the Java Virtual
rlm@572: Machine (JVM). The API for creating and simulating creatures and
rlm@572: senses is entirely expressed in clojure, though many senses are
rlm@572: implemented at the layer of jMonkeyEngine or below. For example,
rlm@572: for the sense of hearing I use a layer of clojure code on top of a
rlm@572: layer of java JNI bindings that drive a layer of \texttt{C++} code which
rlm@572: implements a modified version of \texttt{OpenAL} to support multiple
rlm@572: listeners. \texttt{CORTEX} is the only simulation environment that I know
rlm@572: of that can support multiple entities that can each hear the world
rlm@572: from their own perspective. Other senses also require a small layer
rlm@572: of Java code. \texttt{CORTEX} also uses \texttt{bullet}, a physics simulator
rlm@572: written in \texttt{C}.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=12cm]{./images/blender-worm.png}
rlm@572: \caption{\label{worm-recognition-intro-2}Here is the worm from figure \ref{worm-intro} modeled in Blender, a free 3D-modeling program. Senses and joints are described using special nodes in Blender.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Here are some things I anticipate that \texttt{CORTEX} might be used for:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item exploring new ideas about sensory integration
rlm@572: \item distributed communication among swarm creatures
rlm@572: \item self-learning using free exploration,
rlm@572: \item evolutionary algorithms involving creature construction
rlm@572: \item exploration of exotic senses and effectors that are not possible
rlm@572: in the real world (such as telekinesis or a semantic sense)
rlm@572: \item imagination using subworlds
rlm@572: \end{itemize}
rlm@572: 
rlm@572: During one test with \texttt{CORTEX}, I created 3,000 creatures each with
rlm@572: its own independent senses and ran them all at only 1/80 real time.
rlm@572: In another test, I created a detailed model of my own hand,
rlm@572: equipped with a realistic distribution of touch (more sensitive at
rlm@572: the fingertips), as well as eyes and ears, and it ran at around 1/4
rlm@572: real time.
rlm@572: 
rlm@572: \begin{sidewaysfigure}
rlm@572: \includegraphics[width=8.5in]{images/full-hand.png}
rlm@572: \caption{
rlm@572: I modeled my own right hand in Blender and rigged it with all the
rlm@572: senses that {\tt CORTEX} supports. My simulated hand has a
rlm@572: biologically inspired distribution of touch sensors. The senses are
rlm@572: displayed on the right (the red/black squares are raw sensory output), 
rlm@572: and the simulation is displayed on the
rlm@572: left. Notice that my hand is curling its fingers, that it can see
rlm@572: its own finger from the eye in its palm, and that it can feel its
rlm@572: own thumb touching its palm.}
rlm@572: \end{sidewaysfigure}
rlm@572: 
rlm@572: \section{Designing \texttt{CORTEX}}
rlm@572: \label{sec-2}
rlm@572: 
rlm@572: In this chapter, I outline the design decisions that went into
rlm@572: making \texttt{CORTEX}, along with some details about its implementation.
rlm@572: (A practical guide to getting started with \texttt{CORTEX}, which skips
rlm@572: over the history and implementation details presented here, is
rlm@572: provided in an appendix at the end of this thesis.)
rlm@572: 
rlm@572: Throughout this project, I intended for \texttt{CORTEX} to be flexible and
rlm@572: extensible enough to be useful for other researchers who want to
rlm@572: test ideas of their own. To this end, wherever I have had to make
rlm@572: architectural choices about \texttt{CORTEX}, I have chosen to give as much
rlm@572: freedom to the user as possible, so that \texttt{CORTEX} may be used for
rlm@572: things I have not foreseen.
rlm@572: 
rlm@572: \subsection{Building in simulation versus reality}
rlm@572: \label{sec-2-1}
rlm@572: The most important architectural decision of all is the choice to
rlm@572: use a computer-simulated environment in the first place! The world
rlm@572: is a vast and rich place, and for now simulations are a very poor
rlm@572: reflection of its complexity. It may be that there is a significant
rlm@572: qualitative difference between dealing with senses in the real
rlm@572: world and dealing with pale facsimiles of them in a simulation
rlm@572: (\cite{brooks-representation}). What are the advantages and
rlm@572: disadvantages of a simulation vs. reality?
rlm@572: 
rlm@572: \subsubsection{Simulation}
rlm@572: \label{sec-2-1-1}
rlm@572: 
rlm@572: The advantages of virtual reality are that when everything is a
rlm@572: simulation, experiments in that simulation are absolutely
rlm@572: reproducible. It's also easier to change the creature and
rlm@572: environment to explore new situations and different sensory
rlm@572: combinations.
rlm@572: 
rlm@572: If the world is to be simulated on a computer, then not only do
rlm@572: you have to worry about whether the creature's senses are rich
rlm@572: enough to learn from the world, but whether the world itself is
rlm@572: rendered with enough detail and realism to give enough working
rlm@572: material to the creature's senses. To name just a few
rlm@572: difficulties facing modern physics simulators: destructibility of
rlm@572: the environment, simulation of water/other fluids, large areas,
rlm@572: nonrigid bodies, lots of objects, smoke. I don't know of any
rlm@572: computer simulation that would allow a creature to take a rock
rlm@572: and grind it into fine dust, then use that dust to make a clay
rlm@572: sculpture, at least not without spending years calculating the
rlm@572: interactions of every single small grain of dust. Maybe a
rlm@572: simulated world with today's limitations doesn't provide enough
rlm@572: richness for real intelligence to evolve.
rlm@572: 
rlm@572: \subsubsection{Reality}
rlm@572: \label{sec-2-1-2}
rlm@572: 
rlm@572: The other approach for playing with senses is to hook your
rlm@572: software up to real cameras, microphones, robots, etc., and let it
rlm@572: loose in the real world. This has the advantage of eliminating
rlm@572: concerns about simulating the world at the expense of increasing
rlm@572: the complexity of implementing the senses. Instead of just
rlm@572: grabbing the current rendered frame for processing, you have to
rlm@572: use an actual camera with real lenses and interact with photons to
rlm@572: get an image. It is much harder to change the creature, which is
rlm@572: now partly a physical robot of some sort, since doing so involves
rlm@572: changing things around in the real world instead of modifying
rlm@572: lines of code. While the real world is very rich and definitely
rlm@572: provides enough stimulation for intelligence to develop (as
rlm@572: evidenced by our own existence), it is also uncontrollable in the
rlm@572: sense that a particular situation cannot be recreated perfectly or
rlm@572: saved for later use. It is harder to conduct Science because it is
rlm@572: harder to repeat an experiment. The worst thing about using the
rlm@572: real world instead of a simulation is the matter of time. Instead
rlm@572: of simulated time you get the constant and unstoppable flow of
rlm@572: real time. This severely limits the sorts of software you can use
rlm@572: to program an AI, because all sense inputs must be handled in real
rlm@572: time. Complicated ideas may have to be implemented in hardware or
rlm@572: may simply be impossible given the current speed of our
rlm@572: processors. Contrast this with a simulation, in which the flow of
rlm@572: time in the simulated world can be slowed down to accommodate the
rlm@572: limitations of the creature's programming. In terms of cost, doing
rlm@572: everything in software is far cheaper than building custom
rlm@572: real-time hardware. All you need is a laptop and some patience.
rlm@572: 
rlm@572: \subsection{Simulated time enables rapid prototyping $\backslash$ simple programs}
rlm@572: \label{sec-2-2}
rlm@572: 
rlm@572: I envision \texttt{CORTEX} being used to support rapid prototyping and
rlm@572: iteration of ideas. Even if I could put together a well constructed
rlm@572: kit for creating robots, it would still not be enough because of
rlm@572: the scourge of real-time processing. Anyone who wants to test their
rlm@572: ideas in the real world must always worry about getting their
rlm@572: algorithms to run fast enough to process information in real time.
rlm@572: The need for real time processing only increases if multiple senses
rlm@572: are involved. In the extreme case, even simple algorithms will have
rlm@572: to be accelerated by ASIC chips or FPGAs, turning what would
rlm@572: otherwise be a few lines of code and a 10x speed penalty into a
rlm@572: multi-month ordeal. For this reason, \texttt{CORTEX} supports
rlm@572: \emph{time-dilation}, which scales back the framerate of the simulation
rlm@572: in proportion to the amount of processing each frame. From the
rlm@572: perspective of the creatures inside the simulation, time always
rlm@572: appears to flow at a constant rate, regardless of how complicated
rlm@572: the environment becomes or how many creatures are in the
rlm@572: simulation. The cost is that \texttt{CORTEX} can sometimes run slower than
rlm@572: real time. Time dilation works both ways, however --- simulations
rlm@572: of very simple creatures in \texttt{CORTEX} generally run at 40x real-time
rlm@572: on my machine!
rlm@572: 
rlm@572: \subsection{All sense organs are two-dimensional surfaces}
rlm@572: \label{sec-2-3}
rlm@572: 
rlm@572: If \texttt{CORTEX} is to support a wide variety of senses, it would help
rlm@572: to have a better understanding of what a sense actually is! While
rlm@572: vision, touch, and hearing all seem like they are quite different
rlm@572: things, I was surprised to learn during the course of this thesis
rlm@572: that they (and all physical senses) can be expressed as exactly the
rlm@572: same mathematical object!
rlm@572: 
rlm@572: Human beings are three-dimensional objects, and the nerves that
rlm@572: transmit data from our various sense organs to our brain are
rlm@572: essentially one-dimensional. This leaves up to two dimensions in
rlm@572: which our sensory information may flow. For example, imagine your
rlm@572: skin: it is a two-dimensional surface around a three-dimensional
rlm@572: object (your body). It has discrete touch sensors embedded at
rlm@572: various points, and the density of these sensors corresponds to the
rlm@572: sensitivity of that region of skin. Each touch sensor connects to a
rlm@572: nerve, all of which eventually are bundled together as they travel
rlm@572: up the spinal cord to the brain. Intersect the spinal nerves with a
rlm@572: guillotining plane and you will see all of the sensory data of the
rlm@572: skin revealed in a roughly circular two-dimensional image which is
rlm@572: the cross section of the spinal cord. Points on this image that are
rlm@572: close together in this circle represent touch sensors that are
rlm@572: \emph{probably} close together on the skin, although there is of course
rlm@572: some cutting and rearrangement that has to be done to transfer the
rlm@572: complicated surface of the skin onto a two dimensional image.
rlm@572: 
rlm@572: Most human senses consist of many discrete sensors of various
rlm@572: properties distributed along a surface at various densities. For
rlm@572: skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
rlm@572: disks, and Ruffini's endings (\cite{textbook901}), which detect
rlm@572: pressure and vibration of various intensities. For ears, it is the
rlm@572: stereocilia distributed along the basilar membrane inside the
rlm@572: cochlea; each one is sensitive to a slightly different frequency of
rlm@572: sound. For eyes, it is rods and cones distributed along the surface
rlm@572: of the retina. In each case, we can describe the sense with a
rlm@572: surface and a distribution of sensors along that surface.
rlm@572: 
rlm@572: In fact, almost every human sense can be effectively described in
rlm@572: terms of a surface containing embedded sensors. If the sense had
rlm@572: any more dimensions, then there wouldn't be enough room in the
rlm@572: spinal cord to transmit the information!
rlm@572: 
rlm@572: Therefore, \texttt{CORTEX} must support the ability to create objects and
rlm@572: then be able to ``paint'' points along their surfaces to describe
rlm@572: each sense. 
rlm@572: 
rlm@572: Fortunately this idea is already a well known computer graphics
rlm@572: technique called \emph{UV-mapping}. In UV-mapping, the three-dimensional
rlm@572: surface of a model is cut and smooshed until it fits on a
rlm@572: two-dimensional image. You paint whatever you want on that image,
rlm@572: and when the three-dimensional shape is rendered in a game the
rlm@572: smooshing and cutting is reversed and the image appears on the
rlm@572: three-dimensional object.
rlm@572: 
rlm@572: To make a sense, interpret the UV-image as describing the
rlm@572: distribution of that senses' sensors. To get different types of
rlm@572: sensors, you can either use a different color for each type of
rlm@572: sensor, or use multiple UV-maps, each labeled with that sensor
rlm@572: type. I generally use a white pixel to mean the presence of a
rlm@572: sensor and a black pixel to mean the absence of a sensor, and use
rlm@572: one UV-map for each sensor-type within a given sense.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/finger-UV.png}
rlm@572: \caption{\label{finger-UV}The UV-map for an elongated icososphere. The white dots each represent a touch sensor. They are dense in the regions that describe the tip of the finger, and less dense along the dorsal side of the finger opposite the tip.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/finger-1.png}
rlm@572: \caption{\label{finger-side-view}Ventral side of the UV-mapped finger. Note the density of touch sensors at the tip.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{Video game engines provide ready-made physics and shading}
rlm@572: \label{sec-2-4}
rlm@572: 
rlm@572: I did not need to write my own physics simulation code or shader to
rlm@572: build \texttt{CORTEX}. Doing so would lead to a system that is impossible
rlm@572: for anyone but myself to use anyway. Instead, I use a video game
rlm@572: engine as a base and modify it to accommodate the additional needs
rlm@572: of \texttt{CORTEX}. Video game engines are an ideal starting point to
rlm@572: build \texttt{CORTEX}, because they are not far from being creature
rlm@572: building systems themselves.
rlm@572: 
rlm@572: First off, general purpose video game engines come with a physics
rlm@572: engine and lighting / sound system. The physics system provides
rlm@572: tools that can be co-opted to serve as touch, proprioception, and
rlm@572: muscles. Because some games support split screen views, a good
rlm@572: video game engine will allow you to efficiently create multiple
rlm@572: cameras in the simulated world that can be used as eyes. Video game
rlm@572: systems offer integrated asset management for things like textures
rlm@572: and creature models, providing an avenue for defining creatures.
rlm@572: They also understand UV-mapping, because this technique is used to
rlm@572: apply a texture to a model. Finally, because video game engines
rlm@572: support a large number of developers, as long as \texttt{CORTEX} doesn't
rlm@572: stray too far from the base system, other researchers can turn to
rlm@572: this community for help when doing their research.
rlm@572: 
rlm@572: \subsection{\texttt{CORTEX} is based on jMonkeyEngine3}
rlm@572: \label{sec-2-5}
rlm@572: 
rlm@572: While preparing to build \texttt{CORTEX} I studied several video game
rlm@572: engines to see which would best serve as a base. The top contenders
rlm@572: were:
rlm@572: 
rlm@572: \begin{description}
rlm@572: \item[{\href{http://www.idsoftware.com}{Quake II}/\href{http://www.bytonic.de/html/jake2.html}{Jake2}}] The Quake II engine was designed by ID software
rlm@572: in 1997. All the source code was released by ID software into
rlm@572: the Public Domain several years ago, and as a result it has
rlm@572: been ported to many different languages. This engine was
rlm@572: famous for its advanced use of realistic shading and it had
rlm@572: decent and fast physics simulation. The main advantage of the
rlm@572: Quake II engine is its simplicity, but I ultimately rejected
rlm@572: it because the engine is too tied to the concept of a
rlm@572: first-person shooter game. One of the problems I had was that
rlm@572: there does not seem to be any easy way to attach multiple
rlm@572: cameras to a single character. There are also several physics
rlm@572: clipping issues that are corrected in a way that only applies
rlm@572: to the main character and do not apply to arbitrary objects.
rlm@572: 
rlm@572: \item[{\href{http://source.valvesoftware.com/}{Source Engine}    }] The Source Engine evolved from the Quake II
rlm@572: and Quake I engines and is used by Valve in the Half-Life
rlm@572: series of games. The physics simulation in the Source Engine
rlm@572: is quite accurate and probably the best out of all the engines
rlm@572: I investigated. There is also an extensive community actively
rlm@572: working with the engine. However, applications that use the
rlm@572: Source Engine must be written in C++, the code is not open, it
rlm@572: only runs on Windows, and the tools that come with the SDK to
rlm@572: handle models and textures are complicated and awkward to use.
rlm@572: 
rlm@572: \item[{\href{http://jmonkeyengine.com/}{jMonkeyEngine3}}] jMonkeyEngine3 is a new library for creating
rlm@572: games in Java. It uses OpenGL to render to the screen and uses
rlm@572: screengraphs to avoid drawing things that do not appear on the
rlm@572: screen. It has an active community and several games in the
rlm@572: pipeline. The engine was not built to serve any particular
rlm@572: game but is instead meant to be used for any 3D game.
rlm@572: \end{description}
rlm@572: 
rlm@572: I chose jMonkeyEngine3 because it had the most features out of all
rlm@572: the free projects I looked at, and because I could then write my
rlm@572: code in clojure, an implementation of \texttt{LISP} that runs on the JVM.
rlm@572: 
rlm@572: \subsection{\texttt{CORTEX} uses Blender to create creature models}
rlm@572: \label{sec-2-6}
rlm@572: 
rlm@572: For the simple worm-like creatures I will use later on in this
rlm@572: thesis, I could define a simple API in \texttt{CORTEX} that would allow
rlm@572: one to create boxes, spheres, etc., and leave that API as the sole
rlm@572: way to create creatures. However, for \texttt{CORTEX} to truly be useful
rlm@572: for other projects, it needs a way to construct complicated
rlm@572: creatures. If possible, it would be nice to leverage work that has
rlm@572: already been done by the community of 3D modelers, or at least
rlm@572: enable people who are talented at modeling but not programming to
rlm@572: design \texttt{CORTEX} creatures.
rlm@572: 
rlm@572: Therefore I use Blender, a free 3D modeling program, as the main
rlm@572: way to create creatures in \texttt{CORTEX}. However, the creatures modeled
rlm@572: in Blender must also be simple to simulate in jMonkeyEngine3's game
rlm@572: engine, and must also be easy to rig with \texttt{CORTEX}'s senses. I
rlm@572: accomplish this with extensive use of Blender's ``empty nodes.''
rlm@572: 
rlm@572: Empty nodes have no mass, physical presence, or appearance, but
rlm@572: they can hold metadata and have names. I use a tree structure of
rlm@572: empty nodes to specify senses in the following manner:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item Create a single top-level empty node whose name is the name of
rlm@572: the sense.
rlm@572: \item Add empty nodes which each contain meta-data relevant to the
rlm@572: sense, including a UV-map describing the number/distribution of
rlm@572: sensors if applicable.
rlm@572: \item Make each empty-node the child of the top-level node.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/empty-sense-nodes.png}
rlm@572: \caption{\label{sense-nodes}An example of annotating a creature model with empty nodes to describe the layout of senses. There are multiple empty nodes which each describe the position of muscles, ears, eyes, or joints.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{Bodies are composed of segments connected by joints}
rlm@572: \label{sec-2-7}
rlm@572: 
rlm@572: Blender is a general purpose animation tool, which has been used in
rlm@572: the past to create high quality movies such as Sintel
rlm@572: (\cite{blender}). Though Blender can model and render even
rlm@572: complicated things like water, it is crucial to keep models that
rlm@572: are meant to be simulated as creatures simple. \texttt{Bullet}, which
rlm@572: \texttt{CORTEX} uses though jMonkeyEngine3, is a rigid-body physics
rlm@572: system. This offers a compromise between the expressiveness of a
rlm@572: game level and the speed at which it can be simulated, and it means
rlm@572: that creatures should be naturally expressed as rigid components
rlm@572: held together by joint constraints.
rlm@572: 
rlm@572: But humans are more like a squishy bag wrapped around some hard
rlm@572: bones which define the overall shape. When we move, our skin bends
rlm@572: and stretches to accommodate the new positions of our bones.
rlm@572: 
rlm@572: One way to make bodies composed of rigid pieces connected by joints
rlm@572: \emph{seem} more human-like is to use an \emph{armature}, (or \emph{rigging})
rlm@572: system, which defines a overall ``body mesh'' and defines how the
rlm@572: mesh deforms as a function of the position of each ``bone'' which
rlm@572: is a standard rigid body. This technique is used extensively to
rlm@572: model humans and create realistic animations. It is not a good
rlm@572: technique for physical simulation because it is a lie -- the skin
rlm@572: is not a physical part of the simulation and does not interact with
rlm@572: any objects in the world or itself. Objects will pass right though
rlm@572: the skin until they come in contact with the underlying bone, which
rlm@572: is a physical object. Without simulating the skin, the sense of
rlm@572: touch has little meaning, and the creature's own vision will lie to
rlm@572: it about the true extent of its body. Simulating the skin as a
rlm@572: physical object requires some way to continuously update the
rlm@572: physical model of the skin along with the movement of the bones,
rlm@572: which is unacceptably slow compared to rigid body simulation.
rlm@572: 
rlm@572: Therefore, instead of using the human-like ``bony meatbag''
rlm@572: approach, I decided to base my body plans on multiple solid objects
rlm@572: that are connected by joints, inspired by the robot \texttt{EVE} from the
rlm@572: movie WALL-E.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/Eve.jpg}
rlm@572: \caption{\texttt{EVE} from the movie WALL-E.  This body plan turns out to be much better suited to my purposes than a more human-like one.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \texttt{EVE}'s body is composed of several rigid components that are held
rlm@572: together by invisible joint constraints. This is what I mean by
rlm@572: \emph{eve-like}. The main reason that I use eve-like bodies is for
rlm@572: simulation efficiency, and so that there will be correspondence
rlm@572: between the AI's senses and the physical presence of its body. Each
rlm@572: individual section is simulated by a separate rigid body that
rlm@572: corresponds exactly with its visual representation and does not
rlm@572: change. Sections are connected by invisible joints that are well
rlm@572: supported in jMonkeyEngine3. Bullet, the physics backend for
rlm@572: jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
rlm@572: connected by joints. Just because sections are rigid does not mean
rlm@572: they have to stay as one piece forever; they can be dynamically
rlm@572: replaced with multiple sections to simulate splitting in two. This
rlm@572: could be used to simulate retractable claws or \texttt{EVE}'s hands, which
rlm@572: are able to coalesce into one object in the movie.
rlm@572: 
rlm@572: \subsubsection{Solidifying/Connecting a body}
rlm@572: \label{sec-2-7-1}
rlm@572: 
rlm@572: \texttt{CORTEX} creates a creature in two steps: first, it traverses the
rlm@572: nodes in the Blender file and creates physical representations for
rlm@572: any of them that have mass defined in their Blender meta-data.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn physical!
rlm@572:   "Iterate through the nodes in creature and make them real physical
rlm@572:    objects in the simulation."
rlm@572:   [#^Node creature]
rlm@572:   (dorun
rlm@572:    (map
rlm@572:     (fn [geom]
rlm@572:       (let [physics-control
rlm@572:             (RigidBodyControl.
rlm@572:              (HullCollisionShape.
rlm@572:               (.getMesh geom))
rlm@572:              (if-let [mass (meta-data geom "mass")]
rlm@572:                (float mass) (float 1)))]
rlm@572:         (.addControl geom physics-control)))
rlm@572:     (filter #(isa? (class %) Geometry )
rlm@572:             (node-seq creature)))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{physical}Program for iterating through the nodes in a Blender file and generating physical jMonkeyEngine3 objects with mass and a matching physics shape.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: The next step to making a proper body is to connect those pieces
rlm@572: together with joints. jMonkeyEngine has a large array of joints
rlm@572: available via \texttt{bullet}, such as Point2Point, Cone, Hinge, and a
rlm@572: generic Six Degree of Freedom joint, with or without spring
rlm@572: restitution. 
rlm@572: 
rlm@572: Joints are treated a lot like proper senses, in that there is a
rlm@572: top-level empty node named ``joints'' whose children each
rlm@572: represent a joint.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/hand-screenshot1.png}
rlm@572: \caption{\label{blender-hand}View of the hand model in Blender showing the main ``joints'' node (highlighted in yellow) and its children which each represent a joint in the hand. Each joint node has metadata specifying what sort of joint it is.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: 
rlm@572: \texttt{CORTEX}'s procedure for binding the creature together with joints
rlm@572: is as follows:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item Find the children of the ``joints'' node.
rlm@572: \item Determine the two spatials the joint is meant to connect.
rlm@572: \item Create the joint based on the meta-data of the empty node.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: The higher order function \texttt{sense-nodes} from \texttt{cortex.sense}
rlm@572: simplifies finding the joints based on their parent ``joints''
rlm@572: node.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn sense-nodes
rlm@572:   "For some senses there is a special empty Blender node whose
rlm@572:    children are considered markers for an instance of that sense. This
rlm@572:    function generates functions to find those children, given the name
rlm@572:    of the special parent node."
rlm@572:   [parent-name]
rlm@572:   (fn [#^Node creature]
rlm@572:     (if-let [sense-node (.getChild creature parent-name)]
rlm@572:       (seq (.getChildren sense-node)) [])))
rlm@572: 
rlm@572: (def
rlm@572:   ^{:doc "Return the children of the creature's \"joints\" node."
rlm@572:     :arglists '([creature])}
rlm@572:   joints
rlm@572:   (sense-nodes "joints"))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{get-empty-nodes}Retrieving the children empty nodes from a single named empty node is a common pattern in \texttt{CORTEX}. Further instances of this technique for the senses will be omitted}
rlm@572: \end{listing}
rlm@572: 
rlm@572: To find a joint's targets, \texttt{CORTEX} creates a small cube, centered
rlm@572: around the empty-node, and grows the cube exponentially until it
rlm@572: intersects two physical objects. The objects are ordered according
rlm@572: to the joint's rotation, with the first one being the object that
rlm@572: has more negative coordinates in the joint's reference frame.
rlm@572: Because the objects must be physical, the empty-node itself
rlm@572: escapes detection. Because the objects must be physical,
rlm@572: \texttt{joint-targets} must be called \emph{after} \texttt{physical!} is called.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn joint-targets
rlm@572:   "Return the two closest two objects to the joint object, ordered
rlm@572:   from bottom to top according to the joint's rotation."
rlm@572:   [#^Node parts #^Node joint]
rlm@572:   (loop [radius (float 0.01)]
rlm@572:     (let [results (CollisionResults.)]
rlm@572:       (.collideWith
rlm@572:        parts
rlm@572:        (BoundingBox. (.getWorldTranslation joint)
rlm@572:                      radius radius radius) results)
rlm@572:       (let [targets
rlm@572:             (distinct
rlm@572:              (map  #(.getGeometry %) results))]
rlm@572:         (if (>= (count targets) 2)
rlm@572:           (sort-by
rlm@572:            #(let [joint-ref-frame-position
rlm@572:                   (jme-to-blender
rlm@572:                    (.mult
rlm@572:                     (.inverse (.getWorldRotation joint))
rlm@572:                     (.subtract (.getWorldTranslation %)
rlm@572:                                (.getWorldTranslation joint))))]
rlm@572:               (.dot (Vector3f. 1 1 1) joint-ref-frame-position))                  
rlm@572:            (take 2 targets))
rlm@572:           (recur (float (* radius 2))))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{joint-targets}Program to find the targets of a joint node by exponentially growth of a search cube.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Once \texttt{CORTEX} finds all joints and targets, it creates them using
rlm@572: a dispatch on the metadata of each joint node.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defmulti joint-dispatch
rlm@572:   "Translate Blender pseudo-joints into real JME joints."
rlm@572:   (fn [constraints & _] 
rlm@572:     (:type constraints)))
rlm@572: 
rlm@572: (defmethod joint-dispatch :point
rlm@572:   [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572:   (doto (SixDofJoint. control-a control-b pivot-a pivot-b false)
rlm@572:     (.setLinearLowerLimit Vector3f/ZERO)
rlm@572:     (.setLinearUpperLimit Vector3f/ZERO)))
rlm@572: 
rlm@572: (defmethod joint-dispatch :hinge
rlm@572:   [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572:   (let [axis (if-let [axis (:axis constraints)] axis Vector3f/UNIT_X)
rlm@572:         [limit-1 limit-2] (:limit constraints)
rlm@572:         hinge-axis (.mult rotation (blender-to-jme axis))]
rlm@572:     (doto (HingeJoint. control-a control-b pivot-a pivot-b 
rlm@572:                        hinge-axis hinge-axis)
rlm@572:       (.setLimit limit-1 limit-2))))
rlm@572: 
rlm@572: (defmethod joint-dispatch :cone
rlm@572:   [constraints control-a control-b pivot-a pivot-b rotation]
rlm@572:   (let [limit-xz (:limit-xz constraints)
rlm@572:         limit-xy (:limit-xy constraints)
rlm@572:         twist    (:twist constraints)]
rlm@572:     (doto (ConeJoint. control-a control-b pivot-a pivot-b
rlm@572:                       rotation rotation)
rlm@572:       (.setLimit (float limit-xz) (float limit-xy)
rlm@572:                  (float twist)))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{joint-dispatch}Program to dispatch on Blender metadata and create joints suitable for physical simulation.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: All that is left for joints is to combine the above pieces into
rlm@572: something that can operate on the collection of nodes that a
rlm@572: Blender file represents.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn connect
rlm@572:   "Create a joint between 'obj-a and 'obj-b at the location of
rlm@572:   'joint. The type of joint is determined by the metadata on 'joint.
rlm@572: 
rlm@572:    Here are some examples:
rlm@572:    {:type :point}
rlm@572:    {:type :hinge  :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
rlm@572:    (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572: 
rlm@572:    {:type :cone :limit-xz 0]
rlm@572:                 :limit-xy 0]
rlm@572:                 :twist 0]}   (use XZY rotation mode in Blender!)"
rlm@572:   [#^Node obj-a #^Node obj-b #^Node joint]
rlm@572:   (let [control-a (.getControl obj-a RigidBodyControl)
rlm@572:         control-b (.getControl obj-b RigidBodyControl)
rlm@572:         joint-center (.getWorldTranslation joint)
rlm@572:         joint-rotation (.toRotationMatrix (.getWorldRotation joint))
rlm@572:         pivot-a (world-to-local obj-a joint-center)
rlm@572:         pivot-b (world-to-local obj-b joint-center)]
rlm@572:     (if-let
rlm@572:         [constraints (map-vals eval (read-string (meta-data joint "joint")))]
rlm@572:       ;; A side-effect of creating a joint registers
rlm@572:       ;; it with both physics objects which in turn
rlm@572:       ;; will register the joint with the physics system
rlm@572:       ;; when the simulation is started.
rlm@572:         (joint-dispatch constraints
rlm@572:                         control-a control-b
rlm@572:                         pivot-a pivot-b
rlm@572:                         joint-rotation))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{connect}Program to completely create a joint given information from a Blender file.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: In general, whenever \texttt{CORTEX} exposes a sense (or in this case
rlm@572: physicality), it provides a function of the type \texttt{sense!}, which
rlm@572: takes in a collection of nodes and augments it to support that
rlm@572: sense. The function returns any controls necessary to use that
rlm@572: sense. In this case \texttt{body!} creates a physical body and returns no
rlm@572: control functions.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn joints!
rlm@572:   "Connect the solid parts of the creature with physical joints. The
rlm@572:    joints are taken from the \"joints\" node in the creature."
rlm@572:   [#^Node creature]
rlm@572:   (dorun
rlm@572:    (map
rlm@572:     (fn [joint]
rlm@572:       (let [[obj-a obj-b] (joint-targets creature joint)]
rlm@572:         (connect obj-a obj-b joint)))
rlm@572:     (joints creature))))
rlm@572: (defn body!
rlm@572:   "Endow the creature with a physical body connected with joints.  The
rlm@572:    particulars of the joints and the masses of each body part are
rlm@572:    determined in Blender."
rlm@572:   [#^Node creature]
rlm@572:   (physical! creature)
rlm@572:   (joints! creature))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{joints}Program to give joints to a creature.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: All of the code you have just seen amounts to only 130 lines, yet
rlm@572: because it builds on top of Blender and jMonkeyEngine3, those few
rlm@572: lines pack quite a punch!
rlm@572: 
rlm@572: The hand from figure \ref{blender-hand}, which was modeled after
rlm@572: my own right hand, can now be given joints and simulated as a
rlm@572: creature.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/physical-hand.png}
rlm@572: \caption{\label{physical-hand}With the ability to create physical creatures from Blender, \texttt{CORTEX} gets one step closer to becoming a full creature simulation environment.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{Sight reuses standard video game components\ldots{}}
rlm@572: \label{sec-2-8}
rlm@572: 
rlm@572: Vision is one of the most important senses for humans, so I need to
rlm@572: build a simulated sense of vision for my AI. I will do this with
rlm@572: simulated eyes. Each eye can be independently moved and should see
rlm@572: its own version of the world depending on where it is.
rlm@572: 
rlm@572: Making these simulated eyes a reality is simple because
rlm@572: jMonkeyEngine already contains extensive support for multiple views
rlm@572: of the same 3D simulated world. The reason jMonkeyEngine has this
rlm@572: support is because the support is necessary to create games with
rlm@572: split-screen views. Multiple views are also used to create
rlm@572: efficient pseudo-reflections by rendering the scene from a certain
rlm@572: perspective and then projecting it back onto a surface in the 3D
rlm@572: world.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/goldeneye-4-player.png}
rlm@572: \caption{\label{goldeneye}jMonkeyEngine supports multiple views to enable split-screen games, like GoldenEye, which was one of the first games to use split-screen views.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{A Brief Description of jMonkeyEngine's Rendering Pipeline}
rlm@572: \label{sec-2-8-1}
rlm@572: 
rlm@572: jMonkeyEngine allows you to create a \texttt{ViewPort}, which represents a
rlm@572: view of the simulated world. You can create as many of these as you
rlm@572: want. Every frame, the \texttt{RenderManager} iterates through each
rlm@572: \texttt{ViewPort}, rendering the scene in the GPU. For each \texttt{ViewPort} there
rlm@572: is a \texttt{FrameBuffer} which represents the rendered image in the GPU.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/diagram_rendermanager2.png}
rlm@572: \caption{\label{rendermanagers}\texttt{ViewPorts} are cameras in the world. During each frame, the \texttt{RenderManager} records a snapshot of what each view is currently seeing; these snapshots are \texttt{FrameBuffer} objects.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Each \texttt{ViewPort} can have any number of attached \texttt{SceneProcessor}
rlm@572: objects, which are called every time a new frame is rendered. A
rlm@572: \texttt{SceneProcessor} receives its \texttt{ViewPort's} \texttt{FrameBuffer} and can do
rlm@572: whatever it wants to the data.  Often this consists of invoking GPU
rlm@572: specific operations on the rendered image.  The \texttt{SceneProcessor} can
rlm@572: also copy the GPU image data to RAM and process it with the CPU.
rlm@572: 
rlm@572: \subsubsection{Appropriating Views for Vision}
rlm@572: \label{sec-2-8-2}
rlm@572: 
rlm@572: Each eye in the simulated creature needs its own \texttt{ViewPort} so
rlm@572: that it can see the world from its own perspective. To this
rlm@572: \texttt{ViewPort}, I add a \texttt{SceneProcessor} that feeds the visual data to
rlm@572: any arbitrary continuation function for further processing. That
rlm@572: continuation function may perform both CPU and GPU operations on
rlm@572: the data. To make this easy for the continuation function, the
rlm@572: \texttt{SceneProcessor} maintains appropriately sized buffers in RAM to
rlm@572: hold the data. It does not do any copying from the GPU to the CPU
rlm@572: itself because it is a slow operation.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn vision-pipeline
rlm@572:   "Create a SceneProcessor object which wraps a vision processing
rlm@572:   continuation function. The continuation is a function that takes 
rlm@572:   [#^Renderer r #^FrameBuffer fb #^ByteBuffer b #^BufferedImage bi],
rlm@572:   each of which has already been appropriately sized."
rlm@572:   [continuation]
rlm@572:   (let [byte-buffer (atom nil)
rlm@572: 	renderer (atom nil)
rlm@572:         image (atom nil)]
rlm@572:   (proxy [SceneProcessor] []
rlm@572:     (initialize
rlm@572:      [renderManager viewPort]
rlm@572:      (let [cam (.getCamera viewPort)
rlm@572: 	   width (.getWidth cam)
rlm@572: 	   height (.getHeight cam)]
rlm@572:        (reset! renderer (.getRenderer renderManager))
rlm@572:        (reset! byte-buffer
rlm@572: 	     (BufferUtils/createByteBuffer
rlm@572: 	      (* width height 4)))
rlm@572:         (reset! image (BufferedImage.
rlm@572:                       width height
rlm@572:                       BufferedImage/TYPE_4BYTE_ABGR))))
rlm@572:     (isInitialized [] (not (nil? @byte-buffer)))
rlm@572:     (reshape [_ _ _])
rlm@572:     (preFrame [_])
rlm@572:     (postQueue [_])
rlm@572:     (postFrame
rlm@572:      [#^FrameBuffer fb]
rlm@572:      (.clear @byte-buffer)
rlm@572:      (continuation @renderer fb @byte-buffer @image))
rlm@572:     (cleanup []))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{pipeline-1}Function to make the rendered scene in jMonkeyEngine available for further processing.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: The continuation function given to \texttt{vision-pipeline} above will be
rlm@572: given a \texttt{Renderer} and three containers for image data. The
rlm@572: \texttt{FrameBuffer} references the GPU image data, but the pixel data
rlm@572: can not be used directly on the CPU. The \texttt{ByteBuffer} and
rlm@572: \texttt{BufferedImage} are initially "empty" but are sized to hold the
rlm@572: data in the \texttt{FrameBuffer}. I call transferring the GPU image data
rlm@572: to the CPU structures "mixing" the image data.
rlm@572: 
rlm@572: \subsubsection{Optical sensor arrays are described with images and referenced with metadata}
rlm@572: \label{sec-2-8-3}
rlm@572: 
rlm@572: The vision pipeline described above handles the flow of rendered
rlm@572: images. Now, \texttt{CORTEX} needs simulated eyes to serve as the source
rlm@572: of these images.
rlm@572: 
rlm@572: An eye is described in Blender in the same way as a joint. They
rlm@572: are zero dimensional empty objects with no geometry whose local
rlm@572: coordinate system determines the orientation of the resulting eye.
rlm@572: All eyes are children of a parent node named "eyes" just as all
rlm@572: joints have a parent named "joints". An eye binds to the nearest
rlm@572: physical object with \texttt{bind-sense}.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn add-eye!
rlm@572:   "Create a Camera centered on the current position of 'eye which
rlm@572:    follows the closest physical node in 'creature. The camera will
rlm@572:    point in the X direction and use the Z vector as up as determined
rlm@572:    by the rotation of these vectors in Blender coordinate space. Use
rlm@572:    XZY rotation for the node in Blender."
rlm@572:   [#^Node creature #^Spatial eye]
rlm@572:   (let [target (closest-node creature eye)
rlm@572:         [cam-width cam-height] 
rlm@572:         ;;[640 480] ;; graphics card on laptop doesn't support
rlm@572:                     ;; arbitrary dimensions.
rlm@572:         (eye-dimensions eye)
rlm@572:         cam (Camera. cam-width cam-height)
rlm@572:         rot (.getWorldRotation eye)]
rlm@572:     (.setLocation cam (.getWorldTranslation eye))
rlm@572:     (.lookAtDirection
rlm@572:      cam                           ; this part is not a mistake and
rlm@572:      (.mult rot Vector3f/UNIT_X)   ; is consistent with using Z in
rlm@572:      (.mult rot Vector3f/UNIT_Y))  ; Blender as the UP vector.
rlm@572:     (.setFrustumPerspective
rlm@572:      cam (float 45)
rlm@572:      (float (/ (.getWidth cam) (.getHeight cam)))
rlm@572:      (float 1)
rlm@572:      (float 1000))
rlm@572:     (bind-sense target cam) cam))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{add-eye}Here, the camera is created based on metadata on the eye-node and attached to the nearest physical object with \texttt{bind-sense}}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{Simulated Retina}
rlm@572: \label{sec-2-8-4}
rlm@572: 
rlm@572: An eye is a surface (the retina) which contains many discrete
rlm@572: sensors to detect light. These sensors can have different
rlm@572: light-sensing properties. In humans, each discrete sensor is
rlm@572: sensitive to red, blue, green, or gray. These different types of
rlm@572: sensors can have different spatial distributions along the retina.
rlm@572: In humans, there is a fovea in the center of the retina which has
rlm@572: a very high density of color sensors, and a blind spot which has
rlm@572: no sensors at all. Sensor density decreases in proportion to
rlm@572: distance from the fovea.
rlm@572: 
rlm@572: I want to be able to model any retinal configuration, so my
rlm@572: eye-nodes in Blender contain metadata pointing to images that
rlm@572: describe the precise position of the individual sensors using
rlm@572: white pixels. The meta-data also describes the precise sensitivity
rlm@572: to light that the sensors described in the image have. An eye can
rlm@572: contain any number of these images. For example, the metadata for
rlm@572: an eye might look like this:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: {0xFF0000 "Models/test-creature/retina-small.png"}
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572: \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Together, the number 0xFF0000 and the image above describe the
rlm@572: placement of red-sensitive sensory elements.
rlm@572: 
rlm@572: Meta-data to very crudely approximate a human eye might be
rlm@572: something like this:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: (let [retinal-profile "Models/test-creature/retina-small.png"]
rlm@572:   {0xFF0000 retinal-profile
rlm@572:    0x00FF00 retinal-profile
rlm@572:    0x0000FF retinal-profile
rlm@572:    0xFFFFFF retinal-profile})
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: The numbers that serve as keys in the map determine a sensor's
rlm@572: relative sensitivity to the channels red, green, and blue. These
rlm@572: sensitivity values are packed into an integer in the order
rlm@572: \texttt{|\_|R|G|B|} in 8-bit fields. The RGB values of a pixel in the
rlm@572: image are added together with these sensitivities as linear
rlm@572: weights. Therefore, 0xFF0000 means sensitive to red only while
rlm@572: 0xFFFFFF means sensitive to all colors equally (gray).
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn vision-kernel
rlm@572:   "Returns a list of functions, each of which will return a color
rlm@572:    channel's worth of visual information when called inside a running
rlm@572:    simulation."
rlm@572:   [#^Node creature #^Spatial eye & {skip :skip :or {skip 0}}]
rlm@572:   (let [retinal-map (retina-sensor-profile eye)
rlm@572:         camera (add-eye! creature eye)
rlm@572:         vision-image
rlm@572:         (atom
rlm@572:          (BufferedImage. (.getWidth camera)
rlm@572:                          (.getHeight camera)
rlm@572:                          BufferedImage/TYPE_BYTE_BINARY))
rlm@572:         register-eye!
rlm@572:         (runonce
rlm@572:          (fn [world]
rlm@572:            (add-camera!
rlm@572:             world camera
rlm@572:             (let [counter  (atom 0)]
rlm@572:               (fn [r fb bb bi]
rlm@572:                 (if (zero? (rem (swap! counter inc) (inc skip)))
rlm@572:                   (reset! vision-image
rlm@572:                           (BufferedImage! r fb bb bi))))))))]
rlm@572:      (vec
rlm@572:       (map
rlm@572:        (fn [[key image]]
rlm@572:          (let [whites (white-coordinates image)
rlm@572:                topology (vec (collapse whites))
rlm@572:                sensitivity (sensitivity-presets key key)]
rlm@572:            (attached-viewport.
rlm@572:             (fn [world]
rlm@572:               (register-eye! world)
rlm@572:               (vector
rlm@572:                topology
rlm@572:                (vec 
rlm@572:                 (for [[x y] whites]
rlm@572:                   (pixel-sense 
rlm@572:                    sensitivity
rlm@572:                    (.getRGB @vision-image x y))))))
rlm@572:             register-eye!)))
rlm@572:          retinal-map))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{vision-kernel}This is the core of vision in \texttt{CORTEX}. A given eye node is converted into a function that returns visual information from the simulation.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Note that because each of the functions generated by
rlm@572: \texttt{vision-kernel} shares the same \texttt{register-eye!} function, the eye
rlm@572: will be registered only once the first time any of the functions
rlm@572: from the list returned by \texttt{vision-kernel} is called. Each of the
rlm@572: functions returned by \texttt{vision-kernel} also allows access to the
rlm@572: \texttt{Viewport} through which it receives images.
rlm@572: 
rlm@572: All the hard work has been done; all that remains is to apply
rlm@572: \texttt{vision-kernel} to each eye in the creature and gather the results
rlm@572: into one list of functions.
rlm@572: 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn vision!
rlm@572:   "Returns a list of functions, each of which returns visual sensory
rlm@572:    data when called inside a running simulation."
rlm@572:   [#^Node creature & {skip :skip :or {skip 0}}]
rlm@572:   (reduce
rlm@572:    concat 
rlm@572:    (for [eye (eyes creature)]
rlm@572:      (vision-kernel creature eye))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{vision}With \texttt{vision!}, \texttt{CORTEX} is already a fine simulation environment for experimenting with different types of eyes.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=13cm]{./images/worm-vision.png}
rlm@572: \caption{\label{worm-vision-test.}Simulated vision with a test creature and the human-like eye approximation. Notice how each channel of the eye responds differently to the differently colored balls.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: The vision code is not much more complicated than the body code,
rlm@572: and enables multiple further paths for simulated vision. For
rlm@572: example, it is quite easy to create bifocal vision -- you just
rlm@572: make two eyes next to each other in Blender! It is also possible
rlm@572: to encode vision transforms in the retinal files. For example, the
rlm@572: human like retina file in figure \ref{retina} approximates a
rlm@572: log-polar transform.
rlm@572: 
rlm@572: This vision code has already been absorbed by the jMonkeyEngine
rlm@572: community and is now (in modified form) part of a system for
rlm@572: capturing in-game video to a file.
rlm@572: 
rlm@572: \subsection{\ldots{}but hearing must be built from scratch}
rlm@572: \label{sec-2-9}
rlm@572: 
rlm@572: At the end of this chapter I will have simulated ears that work the
rlm@572: same way as the simulated eyes in the last chapter. I will be able to
rlm@572: place any number of ear-nodes in a Blender file, and they will bind to
rlm@572: the closest physical object and follow it as it moves around. Each ear
rlm@572: will provide access to the sound data it picks up between every frame.
rlm@572: 
rlm@572: Hearing is one of the more difficult senses to simulate, because there
rlm@572: is less support for obtaining the actual sound data that is processed
rlm@572: by jMonkeyEngine3. There is no "split-screen" support for rendering
rlm@572: sound from different points of view, and there is no way to directly
rlm@572: access the rendered sound data.
rlm@572: 
rlm@572: \texttt{CORTEX}'s hearing is unique because it does not have any
rlm@572: limitations compared to other simulation environments. As far as I
rlm@572: know, there is no other system that supports multiple listeners,
rlm@572: and the sound demo at the end of this chapter is the first time
rlm@572: it's been done in a video game environment.
rlm@572: 
rlm@572: \subsubsection{Brief Description of jMonkeyEngine's Sound System}
rlm@572: \label{sec-2-9-1}
rlm@572: 
rlm@572: jMonkeyEngine's sound system works as follows:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item jMonkeyEngine uses the \texttt{AppSettings} for the particular
rlm@572: application to determine what sort of \texttt{AudioRenderer} should be
rlm@572: used.
rlm@572: \item Although some support is provided for multiple AudioRenderer
rlm@572: backends, jMonkeyEngine at the time of this writing will either
rlm@572: pick no \texttt{AudioRenderer} at all, or the \texttt{LwjglAudioRenderer}.
rlm@572: \item jMonkeyEngine tries to figure out what sort of system you're
rlm@572: running and extracts the appropriate native libraries.
rlm@572: \item The \texttt{LwjglAudioRenderer} uses the \href{http://lwjgl.org/}{\texttt{LWJGL}} (LightWeight Java Game
rlm@572: Library) bindings to interface with a C library called \href{http://kcat.strangesoft.net/openal.html}{\texttt{OpenAL}}
rlm@572: \item \texttt{OpenAL} renders the 3D sound and feeds the rendered sound
rlm@572: directly to any of various sound output devices with which it
rlm@572: knows how to communicate.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: A consequence of this is that there's no way to access the actual
rlm@572: sound data produced by \texttt{OpenAL}. Even worse, \texttt{OpenAL} only supports
rlm@572: one \emph{listener} (it renders sound data from only one perspective),
rlm@572: which normally isn't a problem for games, but becomes a problem
rlm@572: when trying to make multiple AI creatures that can each hear the
rlm@572: world from a different perspective.
rlm@572: 
rlm@572: To make many AI creatures in jMonkeyEngine that can each hear the
rlm@572: world from their own perspective, or to make a single creature with
rlm@572: many ears, it is necessary to go all the way back to \texttt{OpenAL} and
rlm@572: implement support for simulated hearing there.
rlm@572: 
rlm@572: \subsubsection{Extending \texttt{OpenAl}}
rlm@572: \label{sec-2-9-2}
rlm@572: 
rlm@572: Extending \texttt{OpenAL} to support multiple listeners requires 500
rlm@572: lines of \texttt{C} code and is too complicated to mention here. Instead,
rlm@572: I will show a small amount of extension code and go over the high
rlm@572: level strategy. Full source is of course available with the
rlm@572: \texttt{CORTEX} distribution if you're interested.
rlm@572: 
rlm@572: \texttt{OpenAL} goes to great lengths to support many different systems,
rlm@572: all with different sound capabilities and interfaces. It
rlm@572: accomplishes this difficult task by providing code for many
rlm@572: different sound backends in pseudo-objects called \emph{Devices}.
rlm@572: There's a device for the Linux Open Sound System and the Advanced
rlm@572: Linux Sound Architecture, there's one for Direct Sound on Windows,
rlm@572: and there's even one for Solaris. \texttt{OpenAL} solves the problem of
rlm@572: platform independence by providing all these Devices.
rlm@572: 
rlm@572: Wrapper libraries such as LWJGL are free to examine the system on
rlm@572: which they are running and then select an appropriate device for
rlm@572: that system.
rlm@572: 
rlm@572: There are also a few "special" devices that don't interface with
rlm@572: any particular system. These include the Null Device, which
rlm@572: doesn't do anything, and the Wave Device, which writes whatever
rlm@572: sound it receives to a file, if everything has been set up
rlm@572: correctly when configuring \texttt{OpenAL}.
rlm@572: 
rlm@572: Actual mixing (Doppler shift and distance.environment-based
rlm@572: attenuation) of the sound data happens in the Devices, and they
rlm@572: are the only point in the sound rendering process where this data
rlm@572: is available.
rlm@572: 
rlm@572: Therefore, in order to support multiple listeners, and get the
rlm@572: sound data in a form that the AIs can use, it is necessary to
rlm@572: create a new Device which supports this feature.
rlm@572: 
rlm@572: Adding a device to OpenAL is rather tricky -- there are five
rlm@572: separate files in the \texttt{OpenAL} source tree that must be modified
rlm@572: to do so. I named my device the "Multiple Audio Send" Device, or
rlm@572: \texttt{Send} Device for short, since it sends audio data back to the
rlm@572: calling application like an Aux-Send cable on a mixing board.
rlm@572: 
rlm@572: The main idea behind the Send device is to take advantage of the
rlm@572: fact that LWJGL only manages one \emph{context} when using OpenAL. A
rlm@572: \emph{context} is like a container that holds samples and keeps track
rlm@572: of where the listener is. In order to support multiple listeners,
rlm@572: the Send device identifies the LWJGL context as the master
rlm@572: context, and creates any number of slave contexts to represent
rlm@572: additional listeners. Every time the device renders sound, it
rlm@572: synchronizes every source from the master LWJGL context to the
rlm@572: slave contexts. Then, it renders each context separately, using a
rlm@572: different listener for each one. The rendered sound is made
rlm@572: available via JNI to jMonkeyEngine.
rlm@572: 
rlm@572: Switching between contexts is not the normal operation of a
rlm@572: Device, and one of the problems with doing so is that a Device
rlm@572: normally keeps around a few pieces of state such as the
rlm@572: \texttt{ClickRemoval} array above which will become corrupted if the
rlm@572: contexts are not rendered in parallel. The solution is to create a
rlm@572: copy of this normally global device state for each context, and
rlm@572: copy it back and forth into and out of the actual device state
rlm@572: whenever a context is rendered.
rlm@572: 
rlm@572: The core of the \texttt{Send} device is the \texttt{syncSources} function, which
rlm@572: does the job of copying all relevant data from one context to
rlm@572: another. 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: void syncSources(ALsource *masterSource, ALsource *slaveSource, 
rlm@572: 		 ALCcontext *masterCtx, ALCcontext *slaveCtx){
rlm@572:   ALuint master = masterSource->source;
rlm@572:   ALuint slave = slaveSource->source;
rlm@572:   ALCcontext *current = alcGetCurrentContext();
rlm@572: 
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_PITCH);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_GAIN);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_DISTANCE);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_ROLLOFF_FACTOR);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_REFERENCE_DISTANCE);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_MIN_GAIN);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_MAX_GAIN);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_GAIN);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_INNER_ANGLE);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_CONE_OUTER_ANGLE);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_SEC_OFFSET);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_SAMPLE_OFFSET);
rlm@572:   syncSourcef(master,slave,masterCtx,slaveCtx,AL_BYTE_OFFSET);
rlm@572:     
rlm@572:   syncSource3f(master,slave,masterCtx,slaveCtx,AL_POSITION);
rlm@572:   syncSource3f(master,slave,masterCtx,slaveCtx,AL_VELOCITY);
rlm@572:   syncSource3f(master,slave,masterCtx,slaveCtx,AL_DIRECTION);
rlm@572:   
rlm@572:   syncSourcei(master,slave,masterCtx,slaveCtx,AL_SOURCE_RELATIVE);
rlm@572:   syncSourcei(master,slave,masterCtx,slaveCtx,AL_LOOPING);
rlm@572: 
rlm@572:   alcMakeContextCurrent(masterCtx);
rlm@572:   ALint source_type;
rlm@572:   alGetSourcei(master, AL_SOURCE_TYPE, &source_type);
rlm@572: 
rlm@572:   // Only static sources are currently synchronized! 
rlm@572:   if (AL_STATIC == source_type){
rlm@572:     ALint master_buffer;
rlm@572:     ALint slave_buffer;
rlm@572:     alGetSourcei(master, AL_BUFFER, &master_buffer);
rlm@572:     alcMakeContextCurrent(slaveCtx);
rlm@572:     alGetSourcei(slave, AL_BUFFER, &slave_buffer);
rlm@572:     if (master_buffer != slave_buffer){
rlm@572:       alSourcei(slave, AL_BUFFER, master_buffer);
rlm@572:     }
rlm@572:   }
rlm@572:   
rlm@572:   // Synchronize the state of the two sources.
rlm@572:   alcMakeContextCurrent(masterCtx);
rlm@572:   ALint masterState;
rlm@572:   ALint slaveState;
rlm@572: 
rlm@572:   alGetSourcei(master, AL_SOURCE_STATE, &masterState);
rlm@572:   alcMakeContextCurrent(slaveCtx);
rlm@572:   alGetSourcei(slave, AL_SOURCE_STATE, &slaveState);
rlm@572: 
rlm@572:   if (masterState != slaveState){
rlm@572:     switch (masterState){
rlm@572:     case AL_INITIAL : alSourceRewind(slave); break;
rlm@572:     case AL_PLAYING : alSourcePlay(slave);   break;
rlm@572:     case AL_PAUSED  : alSourcePause(slave);  break;
rlm@572:     case AL_STOPPED : alSourceStop(slave);   break;
rlm@572:     }
rlm@572:   }
rlm@572:   // Restore whatever context was previously active.
rlm@572:   alcMakeContextCurrent(current);
rlm@572: }
rlm@572: \end{verbatim}
rlm@572: \caption{\label{sync-openal-sources}Program for extending \texttt{OpenAL} to support multiple listeners via context copying/switching.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: With this special context-switching device, and some ugly JNI
rlm@572: bindings that are not worth mentioning, \texttt{CORTEX} gains the ability
rlm@572: to access multiple sound streams from \texttt{OpenAL}. 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn add-ear!  
rlm@572:   "Create a Listener centered on the current position of 'ear 
rlm@572:    which follows the closest physical node in 'creature and 
rlm@572:    sends sound data to 'continuation."
rlm@572:   [#^Application world #^Node creature #^Spatial ear continuation]
rlm@572:   (let [target (closest-node creature ear)
rlm@572:         lis (Listener.)
rlm@572:         audio-renderer (.getAudioRenderer world)
rlm@572:         sp (hearing-pipeline continuation)]
rlm@572:     (.setLocation lis (.getWorldTranslation ear))
rlm@572:     (.setRotation lis (.getWorldRotation ear))
rlm@572:     (bind-sense target lis)
rlm@572:     (update-listener-velocity! target lis)
rlm@572:     (.addListener audio-renderer lis)
rlm@572:     (.registerSoundProcessor audio-renderer lis sp)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{add-ear}Program to create an ear from a Blender empty node. The ear follows around the nearest physical object and passes all sensory data to a continuation function.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: The \texttt{Send} device, unlike most of the other devices in \texttt{OpenAL},
rlm@572: does not render sound unless asked. This enables the system to
rlm@572: slow down or speed up depending on the needs of the AIs who are
rlm@572: using it to listen. If the device tried to render samples in
rlm@572: real-time, a complicated AI whose mind takes 100 seconds of
rlm@572: computer time to simulate 1 second of AI-time would miss almost
rlm@572: all of the sound in its environment!
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn hearing-kernel
rlm@572:   "Returns a function which returns auditory sensory data when called
rlm@572:    inside a running simulation."
rlm@572:   [#^Node creature #^Spatial ear]
rlm@572:   (let [hearing-data (atom [])
rlm@572:         register-listener!
rlm@572:         (runonce 
rlm@572:          (fn [#^Application world]
rlm@572:            (add-ear!
rlm@572:             world creature ear
rlm@572:             (comp #(reset! hearing-data %)
rlm@572:                   byteBuffer->pulse-vector))))]
rlm@572:     (fn [#^Application world]
rlm@572:       (register-listener! world)
rlm@572:       (let [data @hearing-data
rlm@572:             topology              
rlm@572:             (vec (map #(vector % 0) (range 0 (count data))))]
rlm@572:         [topology data]))))
rlm@572:     
rlm@572: (defn hearing!
rlm@572:   "Endow the creature in a particular world with the sense of
rlm@572:    hearing. Will return a sequence of functions, one for each ear,
rlm@572:    which when called will return the auditory data from that ear."
rlm@572:   [#^Node creature]
rlm@572:   (for [ear (ears creature)]
rlm@572:     (hearing-kernel creature ear)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{hearing}Program to enable arbitrary hearing in \texttt{CORTEX}}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Armed with these functions, \texttt{CORTEX} is able to test possibly the
rlm@572: first ever instance of multiple listeners in a video game engine
rlm@572: based simulation!
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: /**
rlm@572:  * Respond to sound!  This is the brain of an AI entity that 
rlm@572:  * hears its surroundings and reacts to them.
rlm@572:  */
rlm@572: public void process(ByteBuffer audioSamples, 
rlm@572: 		    int numSamples, AudioFormat format) {
rlm@572:     audioSamples.clear();
rlm@572:     byte[] data = new byte[numSamples];
rlm@572:     float[] out = new float[numSamples];
rlm@572:     audioSamples.get(data);
rlm@572:     FloatSampleTools.
rlm@572: 	byte2floatInterleaved
rlm@572: 	(data, 0, out, 0, numSamples/format.getFrameSize(), format);
rlm@572: 
rlm@572:     float max = Float.NEGATIVE_INFINITY;
rlm@572:     for (float f : out){if (f > max) max = f;}
rlm@572:     audioSamples.clear();
rlm@572: 
rlm@572:     if (max > 0.1){
rlm@572: 	entity.getMaterial().setColor("Color", ColorRGBA.Green);
rlm@572:     }
rlm@572:     else {
rlm@572: 	entity.getMaterial().setColor("Color", ColorRGBA.Gray);
rlm@572:     }
rlm@572: \end{verbatim}
rlm@572: \caption{\label{sound-test}Here a simple creature responds to sound by changing its color from gray to green when the total volume goes over a threshold.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/java-hearing-test.png}
rlm@572: \caption{\label{sound-cubes.}First ever simulation of multiple listeners in \texttt{CORTEX}. Each cube is a creature which processes sound data with the \texttt{process} function from listing \ref{sound-test}. the ball is constantly emitting a pure tone of constant volume. As it approaches the cubes, they each change color in response to the sound.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: This system of hearing has also been co-opted by the
rlm@572: jMonkeyEngine3 community and is used to record audio for demo
rlm@572: videos.
rlm@572: 
rlm@572: \subsection{Hundreds of hair-like elements provide a sense of touch}
rlm@572: \label{sec-2-10}
rlm@572: 
rlm@572: Touch is critical to navigation and spatial reasoning and as such I
rlm@572: need a simulated version of it to give to my AI creatures.
rlm@572: 
rlm@572: Human skin has a wide array of touch sensors, each of which
rlm@572: specialize in detecting different vibrational modes and pressures.
rlm@572: These sensors can integrate a vast expanse of skin (i.e. your
rlm@572: entire palm), or a tiny patch of skin at the tip of your finger.
rlm@572: The hairs of the skin help detect objects before they even come
rlm@572: into contact with the skin proper.
rlm@572: 
rlm@572: However, touch in my simulated world can not exactly correspond to
rlm@572: human touch because my creatures are made out of completely rigid
rlm@572: segments that don't deform like human skin.
rlm@572: 
rlm@572: Instead of measuring deformation or vibration, I surround each
rlm@572: rigid part with a plenitude of hair-like objects (\emph{feelers}) which
rlm@572: do not interact with the physical world. Physical objects can pass
rlm@572: through them with no effect. The feelers are able to tell when
rlm@572: other objects pass through them, and they constantly report how
rlm@572: much of their extent is covered. So even though the creature's body
rlm@572: parts do not deform, the feelers create a margin around those body
rlm@572: parts which achieves a sense of touch which is a hybrid between a
rlm@572: human's sense of deformation and sense from hairs.
rlm@572: 
rlm@572: Implementing touch in jMonkeyEngine follows a different technical
rlm@572: route than vision and hearing. Those two senses piggybacked off
rlm@572: jMonkeyEngine's 3D audio and video rendering subsystems. To
rlm@572: simulate touch, I use jMonkeyEngine's physics system to execute
rlm@572: many small collision detections, one for each feeler. The placement
rlm@572: of the feelers is determined by a UV-mapped image which shows where
rlm@572: each feeler should be on the 3D surface of the body.
rlm@572: 
rlm@572: \subsubsection{Defining Touch Meta-Data in Blender}
rlm@572: \label{sec-2-10-1}
rlm@572: 
rlm@572: Each geometry can have a single UV map which describes the
rlm@572: position of the feelers which will constitute its sense of touch.
rlm@572: This image path is stored under the ``touch'' key. The image itself
rlm@572: is black and white, with black meaning a feeler length of 0 (no
rlm@572: feeler is present) and white meaning a feeler length of \texttt{scale},
rlm@572: which is a float stored under the key "scale".
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn tactile-sensor-profile
rlm@572:   "Return the touch-sensor distribution image in BufferedImage format,
rlm@572:    or nil if it does not exist."
rlm@572:   [#^Geometry obj]
rlm@572:   (if-let [image-path (meta-data obj "touch")]
rlm@572:     (load-image image-path)))
rlm@572: 
rlm@572: (defn tactile-scale
rlm@572:   "Return the length of each feeler. Default scale is 0.01
rlm@572:   jMonkeyEngine units."
rlm@572:   [#^Geometry obj]
rlm@572:   (if-let [scale (meta-data obj "scale")]
rlm@572:     scale 0.1))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{touch-meta-data}Touch does not use empty nodes, to store metadata, because the metadata of each solid part of a creature's body is sufficient.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Here is an example of a UV-map which specifies the position of
rlm@572: touch sensors along the surface of the upper segment of a fingertip.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=13cm]{./images/finger-UV.png}
rlm@572: \caption{\label{fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{Implementation Summary}
rlm@572: \label{sec-2-10-2}
rlm@572: 
rlm@572: To simulate touch there are three conceptual steps. For each solid
rlm@572: object in the creature, you first have to get UV image and scale
rlm@572: parameter which define the position and length of the feelers.
rlm@572: Then, you use the triangles which comprise the mesh and the UV
rlm@572: data stored in the mesh to determine the world-space position and
rlm@572: orientation of each feeler. Then once every frame, update these
rlm@572: positions and orientations to match the current position and
rlm@572: orientation of the object, and use physics collision detection to
rlm@572: gather tactile data.
rlm@572: 
rlm@572: Extracting the meta-data has already been described. The third
rlm@572: step, physics collision detection, is handled in \texttt{touch-kernel}.
rlm@572: Translating the positions and orientations of the feelers from the
rlm@572: UV-map to world-space is itself a three-step process.
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item Find the triangles which make up the mesh in pixel-space and in
rlm@572: world-space. $\backslash$(\texttt{triangles}, \texttt{pixel-triangles}).
rlm@572: 
rlm@572: \item Find the coordinates of each feeler in world-space. These are
rlm@572: the origins of the feelers. (\texttt{feeler-origins}).
rlm@572: 
rlm@572: \item Calculate the normals of the triangles in world space, and add
rlm@572: them to each of the origins of the feelers. These are the
rlm@572: normalized coordinates of the tips of the feelers.
rlm@572: (\texttt{feeler-tips}).
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \subsubsection{Triangle Math}
rlm@572: \label{sec-2-10-3}
rlm@572: 
rlm@572: The rigid objects which make up a creature have an underlying
rlm@572: \texttt{Geometry}, which is a \texttt{Mesh} plus a \texttt{Material} and other
rlm@572: important data involved with displaying the object.
rlm@572: 
rlm@572: A \texttt{Mesh} is composed of \texttt{Triangles}, and each \texttt{Triangle} has three
rlm@572: vertices which have coordinates in world space and UV space.
rlm@572: 
rlm@572: Here, \texttt{triangles} gets all the world-space triangles which
rlm@572: comprise a mesh, while \texttt{pixel-triangles} gets those same triangles
rlm@572: expressed in pixel coordinates (which are UV coordinates scaled to
rlm@572: fit the height and width of the UV image).
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn triangle
rlm@572:   "Get the triangle specified by triangle-index from the mesh."
rlm@572:   [#^Geometry geo triangle-index]
rlm@572:   (triangle-seq
rlm@572:    (let [scratch (Triangle.)]
rlm@572:      (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
rlm@572: 
rlm@572: (defn triangles
rlm@572:   "Return a sequence of all the Triangles which comprise a given
rlm@572:    Geometry." 
rlm@572:   [#^Geometry geo]
rlm@572:   (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
rlm@572: 
rlm@572: (defn triangle-vertex-indices
rlm@572:   "Get the triangle vertex indices of a given triangle from a given
rlm@572:    mesh."
rlm@572:   [#^Mesh mesh triangle-index]
rlm@572:   (let [indices (int-array 3)]
rlm@572:     (.getTriangle mesh triangle-index indices)
rlm@572:     (vec indices)))
rlm@572: 
rlm@572:     (defn vertex-UV-coord
rlm@572:   "Get the UV-coordinates of the vertex named by vertex-index"
rlm@572:   [#^Mesh mesh vertex-index]
rlm@572:   (let [UV-buffer
rlm@572:         (.getData
rlm@572:          (.getBuffer
rlm@572:           mesh
rlm@572:           VertexBuffer$Type/TexCoord))]
rlm@572:     [(.get UV-buffer (* vertex-index 2))
rlm@572:      (.get UV-buffer (+ 1 (* vertex-index 2)))]))
rlm@572: 
rlm@572: (defn pixel-triangle [#^Geometry geo image index]
rlm@572:   (let [mesh (.getMesh geo)
rlm@572:         width (.getWidth image)
rlm@572:         height (.getHeight image)]
rlm@572:     (vec (map (fn [[u v]] (vector (* width u) (* height v)))
rlm@572:               (map (partial vertex-UV-coord mesh)
rlm@572:                    (triangle-vertex-indices mesh index))))))
rlm@572: 
rlm@572: (defn pixel-triangles 
rlm@572:   "The pixel-space triangles of the Geometry, in the same order as
rlm@572:    (triangles geo)"
rlm@572:   [#^Geometry geo image]
rlm@572:   (let [height (.getHeight image)
rlm@572:         width (.getWidth image)]
rlm@572:     (map (partial pixel-triangle geo image)
rlm@572:          (range (.getTriangleCount (.getMesh geo))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{get-triangles}Programs to extract triangles from a geometry and get their vertices in both world and UV-coordinates.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{The Affine Transform from one Triangle to Another}
rlm@572: \label{sec-2-10-4}
rlm@572: 
rlm@572: \texttt{pixel-triangles} gives us the mesh triangles expressed in pixel
rlm@572: coordinates and \texttt{triangles} gives us the mesh triangles expressed
rlm@572: in world coordinates. The tactile-sensor-profile gives the
rlm@572: position of each feeler in pixel-space. In order to convert
rlm@572: pixel-space coordinates into world-space coordinates we need
rlm@572: something that takes coordinates on the surface of one triangle
rlm@572: and gives the corresponding coordinates on the surface of another
rlm@572: triangle.
rlm@572: 
rlm@572: Triangles are \href{http://mathworld.wolfram.com/AffineTransformation.html }{affine}, which means any triangle can be transformed
rlm@572: into any other by a combination of translation, scaling, and
rlm@572: rotation. The affine transformation from one triangle to another
rlm@572: is readily computable if the triangle is expressed in terms of a
rlm@572: \(4x4\) matrix.
rlm@572: 
rlm@572: $$
rlm@572: \begin{bmatrix}
rlm@572: x_1 & x_2 & x_3 & n_x \\
rlm@572: y_1 & y_2 & y_3 & n_y \\ 
rlm@572: z_1 & z_2 & z_3 & n_z \\
rlm@572: 1 & 1 & 1 & 1 
rlm@572: \end{bmatrix}
rlm@572: $$
rlm@572: 
rlm@572: Here, the first three columns of the matrix are the vertices of
rlm@572: the triangle. The last column is the right-handed unit normal of
rlm@572: the triangle.
rlm@572: 
rlm@572: With two triangles \(T_{1}\) and \(T_{2}\) each expressed as a
rlm@572: matrix like above, the affine transform from \(T_{1}\) to \(T_{2}\)
rlm@572: is \(T_{2}T_{1}^{-1}\).
rlm@572: 
rlm@572: The clojure code below recapitulates the formulas above, using
rlm@572: jMonkeyEngine's \texttt{Matrix4f} objects, which can describe any affine
rlm@572: transformation.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn triangle->matrix4f
rlm@572:   "Converts the triangle into a 4x4 matrix: The first three columns
rlm@572:    contain the vertices of the triangle; the last contains the unit
rlm@572:    normal of the triangle. The bottom row is filled with 1s."
rlm@572:   [#^Triangle t]
rlm@572:   (let [mat (Matrix4f.)
rlm@572:         [vert-1 vert-2 vert-3]
rlm@572:         (mapv #(.get t %) (range 3))
rlm@572:         unit-normal (do (.calculateNormal t)(.getNormal t))
rlm@572:         vertices [vert-1 vert-2 vert-3 unit-normal]]
rlm@572:     (dorun 
rlm@572:      (for [row (range 4) col (range 3)]
rlm@572:        (do
rlm@572:          (.set mat col row (.get (vertices row) col))
rlm@572:          (.set mat 3 row 1)))) mat))
rlm@572: 
rlm@572: (defn triangles->affine-transform
rlm@572:   "Returns the affine transformation that converts each vertex in the
rlm@572:    first triangle into the corresponding vertex in the second
rlm@572:    triangle."
rlm@572:   [#^Triangle tri-1 #^Triangle tri-2]
rlm@572:   (.mult 
rlm@572:    (triangle->matrix4f tri-2)
rlm@572:    (.invert (triangle->matrix4f tri-1))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{triangle-affine}Program to interpret triangles as affine transforms.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{Triangle Boundaries}
rlm@572: \label{sec-2-10-5}
rlm@572: 
rlm@572: For efficiency's sake I will divide the tactile-profile image into
rlm@572: small squares which inscribe each pixel-triangle, then extract the
rlm@572: points which lie inside the triangle and map them to 3D-space using
rlm@572: \texttt{triangle-transform} above. To do this I need a function,
rlm@572: \texttt{convex-bounds} which finds the smallest box which inscribes a 2D
rlm@572: triangle.
rlm@572: 
rlm@572: \texttt{inside-triangle?} determines whether a point is inside a triangle
rlm@572: in 2D pixel-space.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn convex-bounds
rlm@572:   "Returns the smallest square containing the given vertices, as a
rlm@572:    vector of integers [left top width height]."
rlm@572:   [verts]
rlm@572:   (let [xs (map first verts)
rlm@572:         ys (map second verts)
rlm@572:         x0 (Math/floor (apply min xs))
rlm@572:         y0 (Math/floor (apply min ys))
rlm@572:         x1 (Math/ceil (apply max xs))
rlm@572:         y1 (Math/ceil (apply max ys))]
rlm@572:     [x0 y0 (- x1 x0) (- y1 y0)]))
rlm@572: 
rlm@572: (defn same-side?
rlm@572:   "Given the points p1 and p2 and the reference point ref, is point p
rlm@572:   on the same side of the line that goes through p1 and p2 as ref is?" 
rlm@572:   [p1 p2 ref p]
rlm@572:   (<=
rlm@572:    0
rlm@572:    (.dot 
rlm@572:     (.cross (.subtract p2 p1) (.subtract p p1))
rlm@572:     (.cross (.subtract p2 p1) (.subtract ref p1)))))
rlm@572: 
rlm@572: (defn inside-triangle?
rlm@572:   "Is the point inside the triangle?"
rlm@572:   {:author "Dylan Holmes"}
rlm@572:   [#^Triangle tri #^Vector3f p]
rlm@572:   (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
rlm@572:     (and
rlm@572:      (same-side? vert-1 vert-2 vert-3 p)
rlm@572:      (same-side? vert-2 vert-3 vert-1 p)
rlm@572:      (same-side? vert-3 vert-1 vert-2 p))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{in-triangle}Program to efficiently determine point inclusion in a triangle.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{Feeler Coordinates}
rlm@572: \label{sec-2-10-6}
rlm@572: 
rlm@572: The triangle-related functions above make short work of
rlm@572: calculating the positions and orientations of each feeler in
rlm@572: world-space.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn feeler-pixel-coords
rlm@572:  "Returns the coordinates of the feelers in pixel space in lists, one
rlm@572:   list for each triangle, ordered in the same way as (triangles) and
rlm@572:   (pixel-triangles)."
rlm@572:  [#^Geometry geo image]
rlm@572:  (map 
rlm@572:   (fn [pixel-triangle]
rlm@572:     (filter
rlm@572:      (fn [coord]
rlm@572:        (inside-triangle? (->triangle pixel-triangle)
rlm@572:                          (->vector3f coord)))
rlm@572:        (white-coordinates image (convex-bounds pixel-triangle))))
rlm@572:   (pixel-triangles geo image)))
rlm@572: 
rlm@572: (defn feeler-world-coords 
rlm@572:  "Returns the coordinates of the feelers in world space in lists, one
rlm@572:   list for each triangle, ordered in the same way as (triangles) and
rlm@572:   (pixel-triangles)."
rlm@572:  [#^Geometry geo image]
rlm@572:  (let [transforms
rlm@572:        (map #(triangles->affine-transform
rlm@572:               (->triangle %1) (->triangle %2))
rlm@572:             (pixel-triangles geo image)
rlm@572:             (triangles geo))]
rlm@572:    (map (fn [transform coords]
rlm@572:           (map #(.mult transform (->vector3f %)) coords))
rlm@572:         transforms (feeler-pixel-coords geo image))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{feeler-coordinates}Program to get the coordinates of ``feelers '' in both world and UV-coordinates.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn feeler-origins
rlm@572:   "The world space coordinates of the root of each feeler."
rlm@572:   [#^Geometry geo image]
rlm@572:    (reduce concat (feeler-world-coords geo image)))
rlm@572: 
rlm@572: (defn feeler-tips
rlm@572:   "The world space coordinates of the tip of each feeler."
rlm@572:   [#^Geometry geo image]
rlm@572:   (let [world-coords (feeler-world-coords geo image)
rlm@572:         normals
rlm@572:         (map
rlm@572:          (fn [triangle]
rlm@572:            (.calculateNormal triangle)
rlm@572:            (.clone (.getNormal triangle)))
rlm@572:          (map ->triangle (triangles geo)))]
rlm@572: 
rlm@572:     (mapcat (fn [origins normal]
rlm@572:               (map #(.add % normal) origins))
rlm@572:             world-coords normals)))
rlm@572: 
rlm@572: (defn touch-topology
rlm@572:   [#^Geometry geo image]
rlm@572:   (collapse (reduce concat (feeler-pixel-coords geo image))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{feeler-tips}Program to get the position of the base and tip of each ``feeler''}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{Simulated Touch}
rlm@572: \label{sec-2-10-7}
rlm@572: 
rlm@572: Now that the functions to construct feelers are complete,
rlm@572: \texttt{touch-kernel} generates functions to be called from within a
rlm@572: simulation that perform the necessary physics collisions to
rlm@572: collect tactile data, and \texttt{touch!} recursively applies it to every
rlm@572: node in the creature.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn set-ray [#^Ray ray #^Matrix4f transform
rlm@572:                #^Vector3f origin #^Vector3f tip]
rlm@572:   ;; Doing everything locally reduces garbage collection by enough to
rlm@572:   ;; be worth it.
rlm@572:   (.mult transform origin (.getOrigin ray))
rlm@572:   (.mult transform tip (.getDirection ray))
rlm@572:   (.subtractLocal (.getDirection ray) (.getOrigin ray))
rlm@572:   (.normalizeLocal (.getDirection ray)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{set-ray}Efficient program to transform a ray from one position to another.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn touch-kernel
rlm@572:   "Constructs a function which will return tactile sensory data from
rlm@572:    'geo when called from inside a running simulation"
rlm@572:   [#^Geometry geo]
rlm@572:   (if-let
rlm@572:       [profile (tactile-sensor-profile geo)]
rlm@572:     (let [ray-reference-origins (feeler-origins geo profile)
rlm@572:           ray-reference-tips (feeler-tips geo profile)
rlm@572:           ray-length (tactile-scale geo)
rlm@572:           current-rays (map (fn [_] (Ray.)) ray-reference-origins)
rlm@572:           topology (touch-topology geo profile)
rlm@572:           correction (float (* ray-length -0.2))]
rlm@572:       ;; slight tolerance for very close collisions.
rlm@572:       (dorun
rlm@572:        (map (fn [origin tip]
rlm@572:               (.addLocal origin (.mult (.subtract tip origin)
rlm@572:                                        correction)))
rlm@572:             ray-reference-origins ray-reference-tips))
rlm@572:       (dorun (map #(.setLimit % ray-length) current-rays))
rlm@572:       (fn [node]
rlm@572:         (let [transform (.getWorldMatrix geo)]
rlm@572:           (dorun
rlm@572:            (map (fn [ray ref-origin ref-tip]
rlm@572:                   (set-ray ray transform ref-origin ref-tip))
rlm@572:                 current-rays ray-reference-origins
rlm@572:                 ray-reference-tips))
rlm@572:           (vector
rlm@572:            topology
rlm@572:            (vec
rlm@572:             (for [ray current-rays]
rlm@572:               (do
rlm@572:                 (let [results (CollisionResults.)]
rlm@572:                   (.collideWith node ray results)
rlm@572:                   (let [touch-objects
rlm@572:                         (filter #(not (= geo (.getGeometry %)))
rlm@572:                                 results)
rlm@572:                         limit (.getLimit ray)]
rlm@572:                     [(if (empty? touch-objects)
rlm@572:                        limit
rlm@572:                        (let [response
rlm@572:                              (apply min (map #(.getDistance %)
rlm@572:                                              touch-objects))]
rlm@572:                          (FastMath/clamp
rlm@572:                           (float 
rlm@572:                            (if (> response limit) (float 0.0)
rlm@572:                                (+ response correction)))
rlm@572:                            (float 0.0)
rlm@572:                            limit)))
rlm@572:                      limit])))))))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{touch-kernel}This is the core of touch in \texttt{CORTEX} each feeler follows the object it is bound to, reporting any collisions that may happen.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Armed with the \texttt{touch!} function, \texttt{CORTEX} becomes capable of
rlm@572: giving creatures a sense of touch. A simple test is to create a
rlm@572: cube that is outfitted with a uniform distribution of touch
rlm@572: sensors. It can feel the ground and any balls that it touches.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn touch! 
rlm@572:   "Endow the creature with the sense of touch. Returns a sequence of
rlm@572:    functions, one for each body part with a tactile-sensor-profile,
rlm@572:    each of which when called returns sensory data for that body part."
rlm@572:   [#^Node creature]
rlm@572:   (filter
rlm@572:    (comp not nil?)
rlm@572:    (map touch-kernel
rlm@572:         (filter #(isa? (class %) Geometry)
rlm@572:                 (node-seq creature)))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{touch}\texttt{CORTEX} interface for creating touch in a simulated creature.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: The tactile-sensor-profile image for the touch cube is a simple
rlm@572: cross with a uniform distribution of touch sensors:
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/touch-profile.png}
rlm@572: \caption{\label{touch-cube-uv-map}The touch profile for the touch-cube. Each pure white pixel defines a touch sensitive feeler.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/touch-cube.png}
rlm@572: \caption{\label{touch-cube-uv-map-2}The touch cube reacts to cannonballs. The black, red, and white cross on the right is a visual display of the creature's touch. White means that it is feeling something strongly, black is not feeling anything, and gray is in-between. The cube can feel both the floor and the ball. Notice that when the ball causes the cube to tip, that the bottom face can still feel part of the ground.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{Proprioception provides knowledge of your own body's position}
rlm@572: \label{sec-2-11}
rlm@572: 
rlm@572: Close your eyes, and touch your nose with your right index finger.
rlm@572: How did you do it? You could not see your hand, and neither your
rlm@572: hand nor your nose could use the sense of touch to guide the path
rlm@572: of your hand. There are no sound cues, and Taste and Smell
rlm@572: certainly don't provide any help. You know where your hand is
rlm@572: without your other senses because of Proprioception.
rlm@572: 
rlm@572: Humans can sometimes loose this sense through viral infections or
rlm@572: damage to the spinal cord or brain, and when they do, they loose
rlm@572: the ability to control their own bodies without looking directly at
rlm@572: the parts they want to move. In \href{http://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat}{The Man Who Mistook His Wife for a
rlm@572: Hat} (\cite{man-wife-hat}), a woman named Christina looses this
rlm@572: sense and has to learn how to move by carefully watching her arms
rlm@572: and legs. She describes proprioception as the "eyes of the body,
rlm@572: the way the body sees itself".
rlm@572: 
rlm@572: Proprioception in humans is mediated by \href{http://en.wikipedia.org/wiki/Articular_capsule}{joint capsules}, \href{http://en.wikipedia.org/wiki/Muscle_spindle}{muscle
rlm@572: spindles}, and the \href{http://en.wikipedia.org/wiki/Golgi_tendon_organ}{Golgi tendon organs}. These measure the relative
rlm@572: positions of each body part by monitoring muscle strain and length.
rlm@572: 
rlm@572: It's clear that this is a vital sense for fluid, graceful movement.
rlm@572: It's also particularly easy to implement in jMonkeyEngine.
rlm@572: 
rlm@572: My simulated proprioception calculates the relative angles of each
rlm@572: joint from the rest position defined in the Blender file. This
rlm@572: simulates the muscle-spindles and joint capsules. I will deal with
rlm@572: Golgi tendon organs, which calculate muscle strain, in the next
rlm@572: section (2.12).
rlm@572: 
rlm@572: \subsubsection{Helper functions}
rlm@572: \label{sec-2-11-1}
rlm@572: 
rlm@572: \texttt{absolute-angle} calculates the angle between two vectors,
rlm@572: relative to a third axis vector. This angle is the number of
rlm@572: radians you have to move counterclockwise around the axis vector
rlm@572: to get from the first to the second vector. It is not commutative
rlm@572: like a normal dot-product angle is.
rlm@572: 
rlm@572: The purpose of these functions is to build a system of angle
rlm@572: measurement that is biologically plausible.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn right-handed?
rlm@572:   "true iff the three vectors form a right handed coordinate
rlm@572:    system. The three vectors do not have to be normalized or
rlm@572:    orthogonal."
rlm@572:   [vec1 vec2 vec3]
rlm@572:   (pos? (.dot (.cross vec1 vec2) vec3)))
rlm@572: 
rlm@572: (defn absolute-angle
rlm@572:   "The angle between 'vec1 and 'vec2 around 'axis. In the range 
rlm@572:    [0 (* 2 Math/PI)]."
rlm@572:   [vec1 vec2 axis]
rlm@572:   (let [angle (.angleBetween vec1 vec2)]
rlm@572:     (if (right-handed? vec1 vec2 axis)
rlm@572:       angle (- (* 2 Math/PI) angle))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{helpers}Program to measure angles along a vector}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsubsection{Proprioception Kernel}
rlm@572: \label{sec-2-11-2}
rlm@572: 
rlm@572: Given a joint, \texttt{proprioception-kernel} produces a function that
rlm@572: calculates the Euler angles between the objects the joint
rlm@572: connects. The only tricky part here is making the angles relative
rlm@572: to the joint's initial ``straightness''.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn proprioception-kernel
rlm@572:   "Returns a function which returns proprioceptive sensory data when
rlm@572:   called inside a running simulation."
rlm@572:   [#^Node parts #^Node joint]
rlm@572:   (let [[obj-a obj-b] (joint-targets parts joint)
rlm@572:         joint-rot (.getWorldRotation joint)
rlm@572:         x0 (.mult joint-rot Vector3f/UNIT_X)
rlm@572:         y0 (.mult joint-rot Vector3f/UNIT_Y)
rlm@572:         z0 (.mult joint-rot Vector3f/UNIT_Z)]
rlm@572:     (fn []
rlm@572:       (let [rot-a (.clone (.getWorldRotation obj-a))
rlm@572:             rot-b (.clone (.getWorldRotation obj-b))
rlm@572:             x (.mult rot-a x0)
rlm@572:             y (.mult rot-a y0)
rlm@572:             z (.mult rot-a z0)
rlm@572: 
rlm@572:             X (.mult rot-b x0)
rlm@572:             Y (.mult rot-b y0)
rlm@572:             Z (.mult rot-b z0)
rlm@572:             heading  (Math/atan2 (.dot X z) (.dot X x))
rlm@572:             pitch  (Math/atan2 (.dot X y) (.dot X x))
rlm@572: 
rlm@572:             ;; rotate x-vector back to origin
rlm@572:             reverse
rlm@572:             (doto (Quaternion.)
rlm@572:               (.fromAngleAxis
rlm@572:                (.angleBetween X x)
rlm@572:                (let [cross (.normalize (.cross X x))]
rlm@572:                  (if (= 0 (.length cross)) y cross))))
rlm@572:             roll (absolute-angle (.mult reverse Y) y x)]
rlm@572:         [heading pitch roll]))))
rlm@572: 
rlm@572: (defn proprioception!
rlm@572:   "Endow the creature with the sense of proprioception. Returns a
rlm@572:    sequence of functions, one for each child of the \"joints\" node in
rlm@572:    the creature, which each report proprioceptive information about
rlm@572:    that joint."
rlm@572:   [#^Node creature]
rlm@572:   ;; extract the body's joints
rlm@572:   (let [senses (map (partial proprioception-kernel creature)
rlm@572:                     (joints creature))]
rlm@572:     (fn []
rlm@572:       (map #(%) senses))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{proprioception}Program to return biologically reasonable proprioceptive data for each joint.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \texttt{proprioception!} maps \texttt{proprioception-kernel} across all the
rlm@572: joints of the creature. It uses the same list of joints that
rlm@572: \texttt{joints} uses. Proprioception is the easiest sense to implement in
rlm@572: \texttt{CORTEX}, and it will play a crucial role when efficiently
rlm@572: implementing empathy.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=11cm]{./images/proprio.png}
rlm@572: \caption{\label{proprio}In the upper right corner, the three proprioceptive angle measurements are displayed. Red is yaw, Green is pitch, and White is roll.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{Muscles contain both sensors and effectors}
rlm@572: \label{sec-2-12}
rlm@572: 
rlm@572: Surprisingly enough, terrestrial creatures only move by using
rlm@572: torque applied about their joints. There's not a single straight
rlm@572: line of force in the human body at all! (A straight line of force
rlm@572: would correspond to some sort of jet or rocket propulsion.)
rlm@572: 
rlm@572: In humans, muscles are composed of muscle fibers which can contract
rlm@572: to exert force. The muscle fibers which compose a muscle are
rlm@572: partitioned into discrete groups which are each controlled by a
rlm@572: single alpha motor neuron. A single alpha motor neuron might
rlm@572: control as little as three or as many as one thousand muscle
rlm@572: fibers. When the alpha motor neuron is engaged by the spinal cord,
rlm@572: it activates all of the muscle fibers to which it is attached. The
rlm@572: spinal cord generally engages the alpha motor neurons which control
rlm@572: few muscle fibers before the motor neurons which control many
rlm@572: muscle fibers. This recruitment strategy allows for precise
rlm@572: movements at low strength. The collection of all motor neurons that
rlm@572: control a muscle is called the motor pool. The brain essentially
rlm@572: says "activate 30\% of the motor pool" and the spinal cord recruits
rlm@572: motor neurons until 30\% are activated. Since the distribution of
rlm@572: power among motor neurons is unequal and recruitment goes from
rlm@572: weakest to strongest, the first 30\% of the motor pool might be 5\%
rlm@572: of the strength of the muscle.
rlm@572: 
rlm@572: My simulated muscles follow a similar design: Each muscle is
rlm@572: defined by a 1-D array of numbers (the "motor pool"). Each entry in
rlm@572: the array represents a motor neuron which controls a number of
rlm@572: muscle fibers equal to the value of the entry. Each muscle has a
rlm@572: scalar strength factor which determines the total force the muscle
rlm@572: can exert when all motor neurons are activated. The effector
rlm@572: function for a muscle takes a number to index into the motor pool,
rlm@572: and then "activates" all the motor neurons whose index is lower or
rlm@572: equal to the number. Each motor-neuron will apply force in
rlm@572: proportion to its value in the array. Lower values cause less
rlm@572: force. The lower values can be put at the "beginning" of the 1-D
rlm@572: array to simulate the layout of actual human muscles, which are
rlm@572: capable of more precise movements when exerting less force. Or, the
rlm@572: motor pool can simulate more exotic recruitment strategies which do
rlm@572: not correspond to human muscles.
rlm@572: 
rlm@572: This 1D array is defined in an image file for ease of
rlm@572: creation/visualization. Here is an example muscle profile image.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572: \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{Muscle meta-data}
rlm@572: \label{sec-2-12-1}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn muscle-profile-image
rlm@572:   "Get the muscle-profile image from the node's Blender meta-data."
rlm@572:   [#^Node muscle]
rlm@572:   (if-let [image (meta-data muscle "muscle")]
rlm@572:     (load-image image)))
rlm@572: 
rlm@572: (defn muscle-strength
rlm@572:   "Return the strength of this muscle, or 1 if it is not defined."
rlm@572:   [#^Node muscle]
rlm@572:   (if-let [strength (meta-data muscle "strength")]
rlm@572:     strength 1))
rlm@572: 
rlm@572: (defn motor-pool
rlm@572:   "Return a vector where each entry is the strength of the \"motor
rlm@572:    neuron\" at that part in the muscle."
rlm@572:   [#^Node muscle]
rlm@572:   (let [profile (muscle-profile-image muscle)]
rlm@572:     (vec
rlm@572:      (let [width (.getWidth profile)]
rlm@572:        (for [x (range width)]
rlm@572:        (- 255
rlm@572:           (bit-and
rlm@572:            0x0000FF
rlm@572:            (.getRGB profile x 0))))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{motor-pool}Program to deal with loading muscle data from a Blender file's metadata.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Of note here is \texttt{motor-pool} which interprets the muscle-profile
rlm@572: image in a way that allows me to use gradients between white and
rlm@572: red, instead of shades of gray as I've been using for all the
rlm@572: other senses. This is purely an aesthetic touch.
rlm@572: 
rlm@572: \subsubsection{Creating muscles}
rlm@572: \label{sec-2-12-2}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn movement-kernel
rlm@572:   "Returns a function which when called with a integer value inside a
rlm@572:    running simulation will cause movement in the creature according
rlm@572:    to the muscle's position and strength profile. Each function
rlm@572:    returns the amount of force applied / max force."
rlm@572:   [#^Node creature #^Node muscle]
rlm@572:   (let [target (closest-node creature muscle)
rlm@572:         axis
rlm@572:         (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
rlm@572:         strength (muscle-strength muscle)
rlm@572:         
rlm@572:         pool (motor-pool muscle)
rlm@572:         pool-integral (reductions + pool)
rlm@572:         forces
rlm@572:         (vec (map  #(float (* strength (/ % (last pool-integral))))
rlm@572:               pool-integral))
rlm@572:         control (.getControl target RigidBodyControl)]
rlm@572:     (fn [n]
rlm@572:       (let [pool-index (max 0 (min n (dec (count pool))))
rlm@572:             force (forces pool-index)]
rlm@572:         (.applyTorque control (.mult axis force))
rlm@572:         (float (/ force strength))))))
rlm@572: 
rlm@572: (defn movement!
rlm@572:   "Endow the creature with the power of movement. Returns a sequence
rlm@572:    of functions, each of which accept an integer value and will
rlm@572:    activate their corresponding muscle."
rlm@572:   [#^Node creature]
rlm@572:     (for [muscle (muscles creature)]
rlm@572:       (movement-kernel creature muscle)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{muscle-kernel}This is the core movement function in \texttt{CORTEX}, which implements muscles that report on their activation.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: 
rlm@572: \texttt{movement-kernel} creates a function that controls the movement
rlm@572: of the nearest physical node to the muscle node. The muscle exerts
rlm@572: a rotational force dependent on it's orientation to the object in
rlm@572: the Blender file. The function returned by \texttt{movement-kernel} is
rlm@572: also a sense function: it returns the percent of the total muscle
rlm@572: strength that is currently being employed. This is analogous to
rlm@572: muscle tension in humans and completes the sense of proprioception
rlm@572: begun in the last chapter.
rlm@572: 
rlm@572: \subsection{\texttt{CORTEX} brings complex creatures to life!}
rlm@572: \label{sec-2-13}
rlm@572: 
rlm@572: The ultimate test of \texttt{CORTEX} is to create a creature with the full
rlm@572: gamut of senses and put it though its paces. 
rlm@572: 
rlm@572: With all senses enabled, my right hand model looks like an
rlm@572: intricate marionette hand with several strings for each finger:
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=11cm]{./images/hand-with-all-senses2.png}
rlm@572: \caption{\label{hand-nodes-1}View of the hand model with all sense nodes. You can see the joint, muscle, ear, and eye nodes here.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/hand-with-all-senses3.png}
rlm@572: \caption{\label{hand-nodes-2}An alternate view of the hand.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: With the hand fully rigged with senses, I can run it though a test
rlm@572: that will test everything. 
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=15cm]{./images/integration.png}
rlm@572: \caption{\label{integration}Selected frames from a full test of the hand with all senses. Note especially the interactions the hand has with itself: it feels its own palm and fingers, and when it curls its fingers, it sees them with its eye (which is located in the center of the palm. The red block appears with a pure tone sound. The hand then uses its muscles to launch the cube!}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsection{\texttt{CORTEX} enables many possibilities for further research}
rlm@572: \label{sec-2-14}
rlm@572: 
rlm@572: Often times, the hardest part of building a system involving
rlm@572: creatures is dealing with physics and graphics. \texttt{CORTEX} removes
rlm@572: much of this initial difficulty and leaves researchers free to
rlm@572: directly pursue their ideas. I hope that even novices with a
rlm@572: passing curiosity about simulated touch or creature evolution will
rlm@572: be able to use cortex for experimentation. \texttt{CORTEX} is a completely
rlm@572: simulated world, and far from being a disadvantage, its simulated
rlm@572: nature enables you to create senses and creatures that would be
rlm@572: impossible to make in the real world.
rlm@572: 
rlm@572: While not by any means a complete list, here are some paths
rlm@572: \texttt{CORTEX} is well suited to help you explore:
rlm@572: 
rlm@572: \begin{description}
rlm@572: \item[{Empathy        }] my empathy program leaves many areas for
rlm@572: improvement, among which are using vision to infer
rlm@572: proprioception and looking up sensory experience with imagined
rlm@572: vision, touch, and sound.
rlm@572: \item[{Evolution}] Karl Sims created a rich environment for simulating
rlm@572: the evolution of creatures on a Connection Machine
rlm@572: (\cite{sims-evolving-creatures}). Today, this can be redone
rlm@572: and expanded with \texttt{CORTEX} on an ordinary computer.
rlm@572: \item[{Exotic senses }] Cortex enables many fascinating senses that are
rlm@572: not possible to build in the real world. For example,
rlm@572: telekinesis is an interesting avenue to explore. You can also
rlm@572: make a ``semantic'' sense which looks up metadata tags on
rlm@572: objects in the environment the metadata tags might contain
rlm@572: other sensory information.
rlm@572: \item[{Imagination via subworlds}] this would involve a creature with
rlm@572: an effector which creates an entire new sub-simulation where
rlm@572: the creature has direct control over placement/creation of
rlm@572: objects via simulated telekinesis. The creature observes this
rlm@572: sub-world through its normal senses and uses its observations
rlm@572: to make predictions about its top level world.
rlm@572: \item[{Simulated prescience}] step the simulation forward a few ticks,
rlm@572: gather sensory data, then supply this data for the creature as
rlm@572: one of its actual senses. The cost of prescience is slowing
rlm@572: the simulation down by a factor proportional to however far
rlm@572: you want the entities to see into the future. What happens
rlm@572: when two evolved creatures that can each see into the future
rlm@572: fight each other?
rlm@572: \item[{Swarm creatures}] Program a group of creatures that cooperate
rlm@572: with each other. Because the creatures would be simulated, you
rlm@572: could investigate computationally complex rules of behavior
rlm@572: which still, from the group's point of view, would happen in
rlm@572: real time. Interactions could be as simple as cellular
rlm@572: organisms communicating via flashing lights, or as complex as
rlm@572: humanoids completing social tasks, etc.
rlm@572: \item[{\texttt{HACKER} for writing muscle-control programs}] Presented with a
rlm@572: low-level muscle control / sense API, generate higher level
rlm@572: programs for accomplishing various stated goals. Example goals
rlm@572: might be "extend all your fingers" or "move your hand into the
rlm@572: area with blue light" or "decrease the angle of this joint".
rlm@572: It would be like Sussman's HACKER, except it would operate
rlm@572: with much more data in a more realistic world. Start off with
rlm@572: "calisthenics" to develop subroutines over the motor control
rlm@572: API. The low level programming code might be a turning machine
rlm@572: that could develop programs to iterate over a "tape" where
rlm@572: each entry in the tape could control recruitment of the fibers
rlm@572: in a muscle.
rlm@572: \item[{Sense fusion}] There is much work to be done on sense
rlm@572: integration -- building up a coherent picture of the world and
rlm@572: the things in it. With \texttt{CORTEX} as a base, you can explore
rlm@572: concepts like self-organizing maps or cross modal clustering
rlm@572: in ways that have never before been tried.
rlm@572: \item[{Inverse kinematics}] experiments in sense guided motor control
rlm@572: are easy given \texttt{CORTEX}'s support -- you can get right to the
rlm@572: hard control problems without worrying about physics or
rlm@572: senses.
rlm@572: \end{description}
rlm@572: 
rlm@572: \newpage
rlm@572: 
rlm@572: \section{\texttt{EMPATH}: action recognition in a simulated worm}
rlm@572: \label{sec-3}
rlm@572: 
rlm@572: Here I develop a computational model of empathy, using \texttt{CORTEX} as a
rlm@572: base. Empathy in this context is the ability to observe another
rlm@572: creature and infer what sorts of sensations that creature is
rlm@572: feeling. My empathy algorithm involves multiple phases. First is
rlm@572: free-play, where the creature moves around and gains sensory
rlm@572: experience. From this experience I construct a representation of the
rlm@572: creature's sensory state space, which I call \(\Phi\)-space. Using
rlm@572: \(\Phi\)-space, I construct an efficient function which takes the
rlm@572: limited data that comes from observing another creature and enriches
rlm@572: it with a full compliment of imagined sensory data. I can then use
rlm@572: the imagined sensory data to recognize what the observed creature is
rlm@572: doing and feeling, using straightforward embodied action predicates.
rlm@572: This is all demonstrated with using a simple worm-like creature, and
rlm@572: recognizing worm-actions based on limited data.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/basic-worm-view.png}
rlm@572: \caption{\label{basic-worm-view}Here is the worm with which we will be working. It is composed of 5 segments. Each segment has a pair of extensor and flexor muscles. Each of the worm's four joints is a hinge joint which allows about 30 degrees of rotation to either side. Each segment of the worm is touch-capable and has a uniform distribution of touch sensors on each of its faces. Each joint has a proprioceptive sense to detect relative positions. The worm segments are all the same except for the first one, which has a much higher weight than the others to allow for easy manual motor control.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn worm []
rlm@572:   (let [model (load-blender-model "Models/worm/worm.blend")]
rlm@572:     {:body (doto model (body!))
rlm@572:      :touch (touch! model)
rlm@572:      :proprioception (proprioception! model)
rlm@572:      :muscles (movement! model)}))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{get-worm}Program for reading a worm from a Blender file and outfitting it with the senses of proprioception, touch, and the ability to move, as specified in the Blender file.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsection{Embodiment factors action recognition into manageable parts}
rlm@572: \label{sec-3-1}
rlm@572: 
rlm@572: Using empathy, I divide the problem of action recognition into a
rlm@572: recognition process expressed in the language of a full compliment
rlm@572: of senses, and an imaginative process that generates full sensory
rlm@572: data from partial sensory data. Splitting the action recognition
rlm@572: problem in this manner greatly reduces the total amount of work to
rlm@572: recognize actions: The imaginative process is mostly just matching
rlm@572: previous experience, and the recognition process gets to use all
rlm@572: the senses to directly describe any action.
rlm@572: 
rlm@572: \subsection{Action recognition is easy with a full gamut of senses}
rlm@572: \label{sec-3-2}
rlm@572: 
rlm@572: Embodied representation using multiple senses such as touch,
rlm@572: proprioception, and muscle tension turns out be exceedingly
rlm@572: efficient at describing body-centered actions. It is the right
rlm@572: language for the job. For example, it takes only around 5 lines of
rlm@572: clojure code to describe the action of curling using embodied
rlm@572: primitives. It takes about 10 lines to describe the seemingly
rlm@572: complicated action of wiggling.
rlm@572: 
rlm@572: The following action predicates each take a stream of sensory
rlm@572: experience, observe however much of it they desire, and decide
rlm@572: whether the worm is doing the action they describe. \texttt{curled?}
rlm@572: relies on proprioception, \texttt{resting?} relies on touch, \texttt{wiggling?}
rlm@572: relies on a Fourier analysis of muscle contraction, and
rlm@572: \texttt{grand-circle?} relies on touch and reuses \texttt{curled?} in its
rlm@572: definition, showing how embodied predicates can be composed.
rlm@572: 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn curled?
rlm@572:   "Is the worm curled up?"
rlm@572:   [experiences]
rlm@572:   (every?
rlm@572:    (fn [[_ _ bend]]
rlm@572:      (> (Math/sin bend) 0.64))
rlm@572:    (:proprioception (peek experiences))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{curled}Program for detecting whether the worm is curled. This is the simplest action predicate, because it only uses the last frame of sensory experience, and only uses proprioceptive data. Even this simple predicate, however, is automatically frame independent and ignores vermopomorphic\protect\footnotemark \space differences such as worm textures and colors.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \footnotetext{Like \emph{anthropomorphic} except for worms instead of humans.}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn contact
rlm@572:   "Determine how much contact a particular worm segment has with
rlm@572:    other objects. Returns a value between 0 and 1, where 1 is full
rlm@572:    contact and 0 is no contact."
rlm@572:   [touch-region [coords contact :as touch]]
rlm@572:   (-> (zipmap coords contact)
rlm@572:       (select-keys touch-region)
rlm@572:       (vals)
rlm@572:       (#(map first %))
rlm@572:       (average)
rlm@572:       (* 10)
rlm@572:       (- 1)
rlm@572:       (Math/abs)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{touch-summary}Program for summarizing the touch information in a patch of skin.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (def worm-segment-bottom (rect-region [8 15] [14 22]))
rlm@572: 
rlm@572: (defn resting?
rlm@572:   "Is the worm resting on the ground?"
rlm@572:   [experiences]
rlm@572:   (every?
rlm@572:    (fn [touch-data]
rlm@572:      (< 0.9 (contact worm-segment-bottom touch-data)))
rlm@572:    (:touch (peek experiences))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{resting}Program for detecting whether the worm is at rest. This program uses a summary of the tactile information from the underbelly of the worm, and is only true if every segment is touching the floor. Note that this function contains no references to proprioception at all.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
rlm@572: 
rlm@572: (def worm-segment-top-tip (rect-region [0 15] [7 22]))
rlm@572: 
rlm@572: (defn grand-circle?
rlm@572:   "Does the worm form a majestic circle (one end touching the other)?"
rlm@572:   [experiences]
rlm@572:   (and (curled? experiences)
rlm@572:        (let [worm-touch (:touch (peek experiences))
rlm@572:              tail-touch (worm-touch 0)
rlm@572:              head-touch (worm-touch 4)]
rlm@572:          (and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
rlm@572:               (< 0.55 (contact worm-segment-top-tip    head-touch))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{grand-circle}Program for detecting whether the worm is curled up into a full circle. Here the embodied approach begins to shine, as I am able to both use a previous action predicate (\texttt{curled?}) as well as the direct tactile experience of the head and tail.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn fft [nums]
rlm@572:   (map
rlm@572:    #(.getReal %)
rlm@572:    (.transform
rlm@572:     (FastFourierTransformer. DftNormalization/STANDARD)
rlm@572:     (double-array nums) TransformType/FORWARD)))
rlm@572: 
rlm@572: (def indexed (partial map-indexed vector))
rlm@572: 
rlm@572: (defn max-indexed [s]
rlm@572:   (first (sort-by (comp - second) (indexed s))))
rlm@572: 
rlm@572: (defn wiggling?
rlm@572:   "Is the worm wiggling?"
rlm@572:   [experiences]
rlm@572:   (let [analysis-interval 0x40]
rlm@572:     (when (> (count experiences) analysis-interval)
rlm@572:       (let [a-flex 3
rlm@572:             a-ex   2
rlm@572:             muscle-activity
rlm@572:             (map :muscle (vector:last-n experiences analysis-interval))
rlm@572:             base-activity
rlm@572:             (map #(- (% a-flex) (% a-ex)) muscle-activity)]
rlm@572:         (= 2
rlm@572:            (first
rlm@572:             (max-indexed
rlm@572:              (map #(Math/abs %)
rlm@572:                   (take 20 (fft base-activity))))))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{wiggling}Program for detecting whether the worm has been wiggling for the last few frames. It uses a Fourier analysis of the muscle contractions of the worm's tail to determine wiggling. This is significant because there is no particular frame that clearly indicates that the worm is wiggling --- only when multiple frames are analyzed together is the wiggling revealed. Defining wiggling this way also gives the worm an opportunity to learn and recognize ``frustrated wiggling'', where the worm tries to wiggle but can't. Frustrated wiggling is very visually different from actual wiggling, but this definition gives it to us for free.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: With these action predicates, I can now recognize the actions of
rlm@572: the worm while it is moving under my control and I have access to
rlm@572: all the worm's senses.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn debug-experience
rlm@572:   [experiences text]
rlm@572:   (cond
rlm@572:    (grand-circle? experiences) (.setText text "Grand Circle")
rlm@572:    (curled? experiences)       (.setText text "Curled")
rlm@572:    (wiggling? experiences)     (.setText text "Wiggling")
rlm@572:    (resting? experiences)      (.setText text "Resting")))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{report-worm-activity}Use the action predicates defined earlier to report on what the worm is doing while in simulation.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/worm-identify-init.png}
rlm@572: \caption{\label{basic-worm-view}Using \texttt{debug-experience}, the body-centered predicates work together to classify the behavior of the worm. the predicates are operating with access to the worm's full sensory data.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: These action predicates satisfy the recognition requirement of an
rlm@572: empathic recognition system. There is power in the simplicity of
rlm@572: the action predicates. They describe their actions without getting
rlm@572: confused in visual details of the worm. Each one is independent of
rlm@572: position and rotation, but more than that, they are each
rlm@572: independent of irrelevant visual details of the worm and the
rlm@572: environment. They will work regardless of whether the worm is a
rlm@572: different color or heavily textured, or if the environment has
rlm@572: strange lighting.
rlm@572: 
rlm@572: Consider how the human act of jumping might be described with
rlm@572: body-centered action predicates: You might specify that jumping is
rlm@572: mainly the feeling of your knees bending, your thigh muscles
rlm@572: contracting, and your inner ear experiencing a certain sort of back
rlm@572: and forth acceleration. This representation is a very concrete
rlm@572: description of jumping, couched in terms of muscles and senses, but
rlm@572: it also has the ability to describe almost all kinds of jumping, a
rlm@572: generality that you might think could only be achieved by a very
rlm@572: abstract description. The body centered jumping predicate does not
rlm@572: have terms that consider the color of a person's skin or whether
rlm@572: they are male or female, instead it gets right to the meat of what
rlm@572: jumping actually \emph{is}.
rlm@572: 
rlm@572: Of course, the action predicates are not directly applicable to
rlm@572: video data, which lacks the advanced sensory information which they
rlm@572: require!
rlm@572: 
rlm@572: The trick now is to make the action predicates work even when the
rlm@572: sensory data on which they depend is absent!
rlm@572: 
rlm@572: \subsection{\(\Phi\)-space describes the worm's experiences}
rlm@572: \label{sec-3-3}
rlm@572: 
rlm@572: As a first step towards building empathy, I need to gather all of
rlm@572: the worm's experiences during free play. I use a simple vector to
rlm@572: store all the experiences. 
rlm@572: 
rlm@572: Each element of the experience vector exists in the vast space of
rlm@572: all possible worm-experiences. Most of this vast space is actually
rlm@572: unreachable due to physical constraints of the worm's body. For
rlm@572: example, the worm's segments are connected by hinge joints that put
rlm@572: a practical limit on the worm's range of motions without limiting
rlm@572: its degrees of freedom. Some groupings of senses are impossible;
rlm@572: the worm can not be bent into a circle so that its ends are
rlm@572: touching and at the same time not also experience the sensation of
rlm@572: touching itself.
rlm@572: 
rlm@572: As the worm moves around during free play and its experience vector
rlm@572: grows larger, the vector begins to define a subspace which is all
rlm@572: the sensations the worm can practically experience during normal
rlm@572: operation. I call this subspace \(\Phi\)-space, short for
rlm@572: physical-space. The experience vector defines a path through
rlm@572: \(\Phi\)-space. This path has interesting properties that all derive
rlm@572: from physical embodiment. The proprioceptive components of the path
rlm@572: vary smoothly, because in order for the worm to move from one
rlm@572: position to another, it must pass through the intermediate
rlm@572: positions. The path invariably forms loops as common actions are
rlm@572: repeated. Finally and most importantly, proprioception alone
rlm@572: actually gives very strong inference about the other senses. For
rlm@572: example, when the worm is proprioceptively flat over several
rlm@572: frames, you can infer that it is touching the ground and that its
rlm@572: muscles are not active, because if the muscles were active, the
rlm@572: worm would be moving and would not remain perfectly flat. In order
rlm@572: to stay flat, the worm has to be touching the ground, or it would
rlm@572: again be moving out of the flat position due to gravity. If the
rlm@572: worm is positioned in such a way that it interacts with itself,
rlm@572: then it is very likely to be feeling the same tactile feelings as
rlm@572: the last time it was in that position, because it has the same body
rlm@572: as then. As you observe multiple frames of proprioceptive data, you
rlm@572: can become increasingly confident about the exact activations of
rlm@572: the worm's muscles, because it generally takes a unique combination
rlm@572: of muscle contractions to transform the worm's body along a
rlm@572: specific path through \(\Phi\)-space.
rlm@572: 
rlm@572: The worm's total life experience is a long looping path through
rlm@572: \(\Phi\)-space. I will now introduce simple way of taking that
rlm@572: experience path and building a function that can infer complete
rlm@572: sensory experience given only a stream of proprioceptive data. This
rlm@572: \emph{empathy} function will provide a bridge to use the body centered
rlm@572: action predicates on video-like streams of information.
rlm@572: 
rlm@572: \subsection{Empathy is the process of building paths in \(\Phi\)-space}
rlm@572: \label{sec-3-4}
rlm@572: 
rlm@572: Here is the core of a basic empathy algorithm, starting with an
rlm@572: experience vector:
rlm@572: 
rlm@572: An \emph{experience-index} is an index into the grand experience vector
rlm@572: that defines the worm's life. It is a time-stamp for each set of
rlm@572: sensations the worm has experienced.
rlm@572: 
rlm@572: First, I group the experience-indices into bins according to the
rlm@572: similarity of their proprioceptive data. I organize my bins into a
rlm@572: 3 level hierarchy. The smallest bins have an approximate size of
rlm@572: 0.001 radians in all proprioceptive dimensions. Each higher level
rlm@572: is 10x bigger than the level below it.
rlm@572: 
rlm@572: The bins serve as a hashing function for proprioceptive data. Given
rlm@572: a single piece of proprioceptive experience, the bins allow me to
rlm@572: rapidly find all other similar experience-indices of past
rlm@572: experience that had a very similar proprioceptive configuration.
rlm@572: When looking up a proprioceptive experience, if the smallest bin
rlm@572: does not match any previous experience, then I use successively
rlm@572: larger bins until a match is found or I reach the largest bin.
rlm@572: 
rlm@572: Given a sequence of proprioceptive input, I use the bins to
rlm@572: generate a set of similar experiences for each input using the
rlm@572: tiered proprioceptive bins.
rlm@572: 
rlm@572: Finally, to infer sensory data, I select the longest consecutive
rlm@572: chain of experiences that threads through the sets of similar
rlm@572: experiences, starting with the current moment as a root and going
rlm@572: backwards. Consecutive experience means that the experiences appear
rlm@572: next to each other in the experience vector.
rlm@572: 
rlm@572: A stream of proprioceptive input might be:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: [ flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: The worm's previous experience of lying on the ground and lifting
rlm@572: its head generates possible interpretations for each frame (the
rlm@572: numbers are experience-indices):
rlm@572: 
rlm@572: \clearpage
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572:    1     1     1     1     1     1     1     4     
rlm@572:    2     2     2     2     2     2     2   
rlm@572:    3     3     3     3     3     3     3
rlm@572:    6     6     6     6     6     6     6
rlm@572:    7     7     7     7     7     7     7
rlm@572:    8     8     8     8     8     8     8
rlm@572:    9     9     9     9     9     9     9
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: These interpretations suggest a new path through phi space:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
rlm@572:    6     7     8     9     1     2     3     4
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: The new path through \(\Phi\)-space is synthesized from two actual
rlm@572: paths that the creature has experienced: the "1-2-3-4" chain and
rlm@572: the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
rlm@572: ends with the worm lifting its head. It originated from a short
rlm@572: training session where the worm rested on the floor for a brief
rlm@572: while and then raised its head. The "6-7-8-9" chain is part of a
rlm@572: longer chain of inactivity where the worm simply rested on the
rlm@572: floor without moving. It is preferred over a "1-2-3" chain (which
rlm@572: also describes inactivity) because it is longer. The main ideas
rlm@572: again:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item Imagined \(\Phi\)-space paths are synthesized by looping and mixing
rlm@572: previous experiences.
rlm@572: 
rlm@572: \item Longer experience paths (less edits) are preferred.
rlm@572: 
rlm@572: \item The present is more important than the past --- more recent
rlm@572: events take precedence in interpretation.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: This algorithm has three advantages: 
rlm@572: 
rlm@572: \begin{enumerate}
rlm@572: \item It's simple
rlm@572: 
rlm@572: \item It's very fast -- retrieving possible interpretations takes
rlm@572: constant time. Tracing through chains of interpretations takes
rlm@572: time proportional to the average number of experiences in a
rlm@572: proprioceptive bin. Redundant experiences in \(\Phi\)-space can be
rlm@572: merged to save computation.
rlm@572: 
rlm@572: \item It protects from wrong interpretations of transient ambiguous
rlm@572: proprioceptive data. For example, if the worm is flat for just
rlm@572: an instant, this flatness will not be interpreted as implying
rlm@572: that the worm has its muscles relaxed, since the flatness is
rlm@572: part of a longer chain which includes a distinct pattern of
rlm@572: muscle activation. Markov chains or other memoryless statistical
rlm@572: models that operate on individual frames may very well make this
rlm@572: mistake.
rlm@572: \end{enumerate}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn bin [digits]
rlm@572:   (fn [angles]
rlm@572:     (->> angles
rlm@572:          (flatten)
rlm@572:          (map (juxt #(Math/sin %) #(Math/cos %)))
rlm@572:          (flatten)
rlm@572:          (mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
rlm@572: 
rlm@572: (defn gen-phi-scan 
rlm@572:   "Nearest-neighbors with binning. Only returns a result if
rlm@572:    the proprioceptive data is within 10% of a previously recorded
rlm@572:    result in all dimensions."
rlm@572:   [phi-space]
rlm@572:   (let [bin-keys (map bin [3 2 1])
rlm@572:         bin-maps
rlm@572:         (map (fn [bin-key]
rlm@572:                (group-by
rlm@572:                 (comp bin-key :proprioception phi-space)
rlm@572:                 (range (count phi-space)))) bin-keys)
rlm@572:         lookups (map (fn [bin-key bin-map]
rlm@572:                        (fn [proprio] (bin-map (bin-key proprio))))
rlm@572:                      bin-keys bin-maps)]
rlm@572:     (fn lookup [proprio-data]
rlm@572:       (set (some #(% proprio-data) lookups)))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{bin}Program to convert an experience vector into a proprioceptively binned lookup function.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/film-of-imagination.png}
rlm@572: \caption{\label{phi-space-history-scan}\texttt{longest-thread} finds the longest path of consecutive past experiences to explain proprioceptive worm data from previous data. Here, the film strip represents the creature's previous experience. Sort sequences of memories are spliced together to match the proprioceptive data. Their carry the other senses along with them.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \texttt{longest-thread} infers sensory data by stitching together pieces
rlm@572: from previous experience. It prefers longer chains of previous
rlm@572: experience to shorter ones. For example, during training the worm
rlm@572: might rest on the ground for one second before it performs its
rlm@572: exercises. If during recognition the worm rests on the ground for
rlm@572: five seconds, \texttt{longest-thread} will accommodate this five second
rlm@572: rest period by looping the one second rest chain five times.
rlm@572: 
rlm@572: \texttt{longest-thread} takes time proportional to the average number of
rlm@572: entries in a proprioceptive bin, because for each element in the
rlm@572: starting bin it performs a series of set lookups in the preceding
rlm@572: bins. If the total history is limited, then this takes time
rlm@572: proportional to a only a constant multiple of the number of entries
rlm@572: in the starting bin. This analysis also applies, even if the action
rlm@572: requires multiple longest chains -- it's still the average number
rlm@572: of entries in a proprioceptive bin times the desired chain length.
rlm@572: Because \texttt{longest-thread} is so efficient and simple, I can
rlm@572: interpret worm-actions in real time.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn longest-thread
rlm@572:   "Find the longest thread from phi-index-sets. The index sets should
rlm@572:    be ordered from most recent to least recent."
rlm@572:   [phi-index-sets]
rlm@572:   (loop [result '()
rlm@572:          [thread-bases & remaining :as phi-index-sets] phi-index-sets]
rlm@572:     (if (empty? phi-index-sets)
rlm@572:       (vec result)
rlm@572:       (let [threads
rlm@572:             (for [thread-base thread-bases]
rlm@572:               (loop [thread (list thread-base)
rlm@572:                      remaining remaining]
rlm@572:                 (let [next-index (dec (first thread))]
rlm@572:                   (cond (empty? remaining) thread
rlm@572:                         (contains? (first remaining) next-index)
rlm@572:                         (recur
rlm@572:                          (cons next-index thread) (rest remaining))
rlm@572:                         :else thread))))
rlm@572:             longest-thread
rlm@572:             (reduce (fn [thread-a thread-b]
rlm@572:                       (if (> (count thread-a) (count thread-b))
rlm@572:                         thread-a thread-b))
rlm@572:                     '(nil)
rlm@572:                     threads)]
rlm@572:         (recur (concat longest-thread result)
rlm@572:                (drop (count longest-thread) phi-index-sets))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{longest-thread}Program to calculate empathy by tracing though \(\Phi\)-space and finding the longest (ie. most coherent) interpretation of the data.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: There is one final piece, which is to replace missing sensory data
rlm@572: with a best-guess estimate. While I could fill in missing data by
rlm@572: using a gradient over the closest known sensory data points,
rlm@572: averages can be misleading. It is certainly possible to create an
rlm@572: impossible sensory state by averaging two possible sensory states.
rlm@572: For example, consider moving your hand in an arc over your head. If
rlm@572: for some reason you only have the initial and final positions of
rlm@572: this movement in your \(\Phi\)-space, averaging them together will
rlm@572: produce the proprioceptive sensation of having your hand \emph{inside}
rlm@572: your head, which is physically impossible to ever experience
rlm@572: (barring motor adaption illusions). Therefore I simply replicate
rlm@572: the most recent sensory experience to fill in the gaps.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn infer-nils
rlm@572:   "Replace nils with the next available non-nil element in the
rlm@572:    sequence, or barring that, 0."
rlm@572:   [s]
rlm@572:   (loop [i (dec (count s))
rlm@572:          v (transient s)]
rlm@572:     (if (zero? i) (persistent! v)
rlm@572:         (if-let [cur (v i)]
rlm@572:           (if (get v (dec i) 0)
rlm@572:             (recur (dec i) v)
rlm@572:             (recur (dec i) (assoc! v (dec i) cur)))
rlm@572:           (recur i (assoc! v i 0))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{infer-nils}Fill in blanks in sensory experience by replicating the most recent experience.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \subsection{\texttt{EMPATH} recognizes actions efficiently}
rlm@572: \label{sec-3-5}
rlm@572: 
rlm@572: To use \texttt{EMPATH} with the worm, I first need to gather a set of
rlm@572: experiences from the worm that includes the actions I want to
rlm@572: recognize. The \texttt{generate-phi-space} program (listing
rlm@572: \ref{generate-phi-space} runs the worm through a series of
rlm@572: exercises and gathers those experiences into a vector. The
rlm@572: \texttt{do-all-the-things} program is a routine expressed in a simple
rlm@572: muscle contraction script language for automated worm control. It
rlm@572: causes the worm to rest, curl, and wiggle over about 700 frames
rlm@572: (approx. 11 seconds).
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (def do-all-the-things 
rlm@572:   (concat
rlm@572:    curl-script
rlm@572:    [[300 :d-ex 40]
rlm@572:     [320 :d-ex 0]]
rlm@572:    (shift-script 280 (take 16 wiggle-script))))
rlm@572: 
rlm@572: (defn generate-phi-space []
rlm@572:   (let [experiences (atom [])]
rlm@572:     (run-world
rlm@572:      (apply-map 
rlm@572:       worm-world
rlm@572:       (merge
rlm@572:        (worm-world-defaults)
rlm@572:        {:end-frame 700
rlm@572:         :motor-control
rlm@572:         (motor-control-program worm-muscle-labels do-all-the-things)
rlm@572:         :experiences experiences})))
rlm@572:     @experiences))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{generate-phi-space}Program to gather the worm's experiences into a vector for further processing. The \texttt{motor-control-program} line uses a motor control script that causes the worm to execute a series of ``exercises'' that include all the action predicates.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn init []
rlm@572:   (def phi-space (generate-phi-space))
rlm@572:   (def phi-scan (gen-phi-scan phi-space)))
rlm@572: 
rlm@572: (defn empathy-demonstration []
rlm@572:   (let [proprio (atom ())]
rlm@572:     (fn
rlm@572:       [experiences text]
rlm@572:       (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572:         (swap! proprio (partial cons phi-indices))
rlm@572:         (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572:               empathy (mapv phi-space (infer-nils exp-thread))]
rlm@572:           (println-repl (vector:last-n exp-thread 22))
rlm@572:           (cond
rlm@572:            (grand-circle? empathy) (.setText text "Grand Circle")
rlm@572:            (curled? empathy)       (.setText text "Curled")
rlm@572:            (wiggling? empathy)     (.setText text "Wiggling")
rlm@572:            (resting? empathy)      (.setText text "Resting")
rlm@572:            :else                   (.setText text "Unknown")))))))
rlm@572: 
rlm@572: (defn empathy-experiment [record]
rlm@572:   (.start (worm-world :experience-watch (debug-experience-phi)
rlm@572:                       :record record :worm worm*)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{empathy-debug}Use \texttt{longest-thread} and a \(\Phi\)-space generated from a short exercise routine to interpret actions during free play.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: These programs create a test for the empathy system. First, the
rlm@572: worm's \(\Phi\)-space is generated from a simple motor script. Then the
rlm@572: worm is re-created in an environment almost exactly identical to
rlm@572: the testing environment for the action-predicates, with one major
rlm@572: difference : the only sensory information available to the system
rlm@572: is proprioception. From just the proprioception data and
rlm@572: \(\Phi\)-space, \texttt{longest-thread} synthesizes a complete record the last
rlm@572: 300 sensory experiences of the worm. These synthesized experiences
rlm@572: are fed directly into the action predicates \texttt{grand-circle?},
rlm@572: \texttt{curled?}, \texttt{wiggling?}, and \texttt{resting?} and their outputs are
rlm@572: printed to the screen at each frame.
rlm@572: 
rlm@572: The result of running \texttt{empathy-experiment} is that the system is
rlm@572: generally able to interpret worm actions using the action-predicates
rlm@572: on simulated sensory data just as well as with actual data. Figure
rlm@572: \ref{empathy-debug-image} was generated using \texttt{empathy-experiment}:
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=10cm]{./images/empathy-1.png}
rlm@572: \caption{\label{empathy-debug-image}From only proprioceptive data, \texttt{EMPATH} was able to infer the complete sensory experience and classify four poses (The last panel shows a composite image of \emph{wiggling}, a dynamic pose.)}
rlm@572: \end{figure}
rlm@572: 
rlm@572: One way to measure the performance of \texttt{EMPATH} is to compare the
rlm@572: suitability of the imagined sense experience to trigger the same
rlm@572: action predicates as the real sensory experience. 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (def worm-action-label
rlm@572:   (juxt grand-circle? curled? wiggling?))
rlm@572: 
rlm@572: (defn compare-empathy-with-baseline [matches]
rlm@572:   (let [proprio (atom ())]
rlm@572:     (fn
rlm@572:       [experiences text]
rlm@572:       (let [phi-indices (phi-scan (:proprioception (peek experiences)))]
rlm@572:         (swap! proprio (partial cons phi-indices))
rlm@572:         (let [exp-thread (longest-thread (take 300 @proprio))
rlm@572:               empathy (mapv phi-space (infer-nils exp-thread))
rlm@572:               experience-matches-empathy
rlm@572:               (= (worm-action-label experiences)
rlm@572:                  (worm-action-label empathy))]
rlm@572:           (println-repl experience-matches-empathy)
rlm@572:           (swap! matches #(conj % experience-matches-empathy)))))))
rlm@572:               
rlm@572: (defn accuracy [v]
rlm@572:   (float (/ (count (filter true? v)) (count v))))
rlm@572: 
rlm@572: (defn test-empathy-accuracy []
rlm@572:   (let [res (atom [])]
rlm@572:     (run-world
rlm@572:      (worm-world :experience-watch
rlm@572:                  (compare-empathy-with-baseline res)
rlm@572:                  :worm worm*))
rlm@572:     (accuracy @res)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{test-empathy-accuracy}Determine how closely empathy approximates actual sensory data.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Running \texttt{test-empathy-accuracy} using the very short exercise
rlm@572: program \texttt{do-all-the-things} defined in listing
rlm@572: \ref{generate-phi-space}, and then doing a similar pattern of
rlm@572: activity using manual control of the worm, yields an accuracy of
rlm@572: around 73\%. This is based on very limited worm experience, and
rlm@572: almost all errors are due to the worm's \(\Phi\)-space being too
rlm@572: incomplete to properly interpret common poses. By manually training
rlm@572: the worm for longer using \texttt{init-interactive} defined in listing
rlm@572: \ref{manual-phi-space}, the accuracy dramatically improves:
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn init-interactive []
rlm@572:   (def phi-space
rlm@572:     (let [experiences (atom [])]
rlm@572:       (run-world
rlm@572:        (apply-map 
rlm@572:         worm-world
rlm@572:         (merge
rlm@572:          (worm-world-defaults)
rlm@572:          {:experiences experiences})))
rlm@572:       @experiences))
rlm@572:   (def phi-scan (gen-phi-scan phi-space)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{manual-phi-space}Program to generate \(\Phi\)-space using manual training.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \texttt{init-interactive} allows me to take direct control of the worm's
rlm@572: muscles and run it through each characteristic movement I care
rlm@572: about. After about 1 minute of manual training, I was able to
rlm@572: achieve 95\% accuracy on manual testing of the worm using
rlm@572: \texttt{test-empathy-accuracy}. The majority of disagreements are near the
rlm@572: transition boundaries from one type of action to another. During
rlm@572: these transitions the exact label for the action is often unclear,
rlm@572: and disagreement between empathy and experience is practically
rlm@572: irrelevant. Thus, the system's effective identification accuracy is
rlm@572: even higher than 95\%. When I watch this system myself, I generally
rlm@572: see no errors in action identification compared to my own judgment
rlm@572: of what the worm is doing.
rlm@572: 
rlm@572: \subsection{Digression: Learning touch sensor layout through free play}
rlm@572: \label{sec-3-6}
rlm@572: 
rlm@572: In the previous chapter I showed how to compute actions in terms of
rlm@572: body-centered predicates, but some of those predicates relied on
rlm@572: the average touch activation of pre-defined regions of the worm's
rlm@572: skin. What if, instead of receiving touch pre-grouped into the six
rlm@572: faces of each worm segment, the true partitioning of the worm's
rlm@572: skin was unknown? This is more similar to how a nerve fiber bundle
rlm@572: might be arranged inside an animal. While two fibers that are close
rlm@572: in a nerve bundle \emph{might} correspond to two touch sensors that are
rlm@572: close together on the skin, the process of taking a complicated
rlm@572: surface and forcing it into essentially a 2D circle requires that
rlm@572: some regions of skin that are close together in the animal end up
rlm@572: far apart in the nerve bundle.
rlm@572: 
rlm@572: In this chapter I show how to automatically learn the
rlm@572: skin-partitioning of a worm segment by free exploration. As the
rlm@572: worm rolls around on the floor, large sections of its surface get
rlm@572: activated. If the worm has stopped moving, then whatever region of
rlm@572: skin that is touching the floor is probably an important region,
rlm@572: and should be recorded. The code I provide relies on the worm
rlm@572: segment having flat faces, but still demonstrates a primitive kind
rlm@572: of multi-sensory bootstrapping that I find appealing.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (def full-contact [(float 0.0) (float 0.1)])
rlm@572: 
rlm@572: (defn pure-touch?
rlm@572:   "This is worm specific code to determine if a large region of touch
rlm@572:    sensors is either all on or all off."
rlm@572:   [[coords touch :as touch-data]]
rlm@572:   (= (set (map first touch)) (set full-contact)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{pure-touch}Program to detect whether the worm is in a resting state with one face touching the floor.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: After collecting these important regions, there will many nearly
rlm@572: similar touch regions. While for some purposes the subtle
rlm@572: differences between these regions will be important, for my
rlm@572: purposes I collapse them into mostly non-overlapping sets using
rlm@572: \texttt{remove-similar} in listing \ref{remove-similar}
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn remove-similar
rlm@572:   [coll]
rlm@572:   (loop [result () coll (sort-by (comp - count) coll)]
rlm@572:     (if (empty? coll) result
rlm@572:         (let  [[x & xs] coll
rlm@572:                c (count x)]
rlm@572:           (if (some
rlm@572:                (fn [other-set]
rlm@572:                  (let [oc (count other-set)]
rlm@572:                    (< (- (count (union other-set x)) c) (* oc 0.1))))
rlm@572:                xs)
rlm@572:             (recur result xs)
rlm@572:             (recur (cons x result) xs))))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{remove-similar}Program to take a list of sets of points and ``collapse them'' so that the remaining sets in the list are significantly different from each other. Prefer smaller sets to larger ones.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: Actually running this simulation is easy given \texttt{CORTEX}'s facilities.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn learn-touch-regions []
rlm@572:   (let [experiences (atom [])
rlm@572:         world (apply-map
rlm@572:                worm-world
rlm@572:                (assoc (worm-segment-defaults)
rlm@572:                  :experiences experiences))]
rlm@572:     (run-world world)
rlm@572:     (->>
rlm@572:      @experiences
rlm@572:      (drop 175)
rlm@572:      ;; access the single segment's touch data
rlm@572:      (map (comp first :touch))
rlm@572:      ;; only deal with "pure" touch data to determine surfaces
rlm@572:      (filter pure-touch?)
rlm@572:      ;; associate coordinates with touch values
rlm@572:      (map (partial apply zipmap))
rlm@572:      ;; select those regions where contact is being made
rlm@572:      (map (partial group-by second))
rlm@572:      (map #(get % full-contact))
rlm@572:      (map (partial map first))
rlm@572:      ;; remove redundant/subset regions
rlm@572:      (map set)
rlm@572:      remove-similar)))
rlm@572: 
rlm@572: (defn learn-and-view-touch-regions []
rlm@572:   (map view-touch-region
rlm@572:        (learn-touch-regions)))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{learn-touch}Collect experiences while the worm moves around. Filter the touch sensations by stable ones, collapse similar ones together, and report the regions learned.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: The only thing remaining to define is the particular motion the worm
rlm@572: must take. I accomplish this with a simple motor control program.
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn touch-kinesthetics []
rlm@572:   [[170 :lift-1 40]
rlm@572:    [190 :lift-1 19]
rlm@572:    [206 :lift-1  0]
rlm@572: 
rlm@572:    [400 :lift-2 40]
rlm@572:    [410 :lift-2  0]
rlm@572: 
rlm@572:    [570 :lift-2 40]
rlm@572:    [590 :lift-2 21]
rlm@572:    [606 :lift-2  0]
rlm@572: 
rlm@572:    [800 :lift-1 30]
rlm@572:    [809 :lift-1 0]
rlm@572: 
rlm@572:    [900 :roll-2 40]
rlm@572:    [905 :roll-2 20]
rlm@572:    [910 :roll-2  0]
rlm@572: 
rlm@572:    [1000 :roll-2 40]
rlm@572:    [1005 :roll-2 20]
rlm@572:    [1010 :roll-2  0]
rlm@572:    
rlm@572:    [1100 :roll-2 40]
rlm@572:    [1105 :roll-2 20]
rlm@572:    [1110 :roll-2  0]
rlm@572:    ])
rlm@572: \end{verbatim}
rlm@572: \caption{\label{worm-roll}Motor control program for making the worm roll on the ground. This could also be replaced with random motion.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=12cm]{./images/worm-roll.png}
rlm@572: \caption{\label{worm-roll}The small worm rolls around on the floor, driven by the motor control program in listing \ref{worm-roll}.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=12cm]{./images/touch-learn.png}
rlm@572: \caption{\label{worm-touch-map}After completing its adventures, the worm now knows how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were deemed important by \texttt{learn-touch-regions}. Each white square in the rectangles above is a cluster of ``related" touch nodes as determined by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: While simple, \texttt{learn-touch-regions} exploits regularities in both
rlm@572: the worm's physiology and the worm's environment to correctly
rlm@572: deduce that the worm has six sides. Note that \texttt{learn-touch-regions}
rlm@572: would work just as well even if the worm's touch sense data were
rlm@572: completely scrambled. The cross shape is just for convenience. This
rlm@572: example justifies the use of pre-defined touch regions in \texttt{EMPATH}.
rlm@572: 
rlm@572: \subsection{Recognizing an object using embodied representation}
rlm@572: \label{sec-3-7}
rlm@572: 
rlm@572: At the beginning of the thesis, I suggested that we might recognize
rlm@572: the chair in Figure \ref{hidden-chair} by imagining ourselves in
rlm@572: the position of the man and realizing that he must be sitting on
rlm@572: something in order to maintain that position. Here, I present a
rlm@572: brief elaboration on how to this might be done.
rlm@572: 
rlm@572: First, I need the feeling of leaning or resting \emph{on} some other
rlm@572: object that is not the floor. This feeling is easy to describe
rlm@572: using an embodied representation. 
rlm@572: 
rlm@572: \begin{listing}
rlm@572: \begin{verbatim}
rlm@572: (defn draped?
rlm@572:   "Is the worm:
rlm@572:     -- not flat (the floor is not a 'chair')
rlm@572:     -- supported (not using its muscles to hold its position)
rlm@572:     -- stable (not changing its position)
rlm@572:     -- touching something (must register contact)"
rlm@572:   [experiences]
rlm@572:   (let [b2-hash (bin 2)
rlm@572:         touch (:touch (peek experiences))
rlm@572:         total-contact
rlm@572:         (reduce
rlm@572:          +
rlm@572:          (map #(contact all-touch-coordinates %)
rlm@572:               (rest touch)))]
rlm@572:     (println total-contact)
rlm@572:     (and (not (resting? experiences))
rlm@572:          (every?
rlm@572:           zero?
rlm@572:           (-> experiences
rlm@572:               (vector:last-n 25)
rlm@572:               (#(map :muscle %))
rlm@572:               (flatten)))
rlm@572:          (-> experiences
rlm@572:              (vector:last-n 20)
rlm@572:              (#(map (comp b2-hash flatten :proprioception) %))
rlm@572:              (set)
rlm@572:              (count) (= 1))
rlm@572:          (< 0.03 total-contact))))
rlm@572: \end{verbatim}
rlm@572: \caption{\label{draped}Program describing the sense of leaning or resting on something. This involves a relaxed posture, the feeling of touching something, and a period of stability where the worm does not move.}
rlm@572: \end{listing}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=13cm]{./images/draped.png}
rlm@572: \caption{\label{draped-video}The \texttt{draped?} predicate detects the presence of the cube whenever the worm interacts with it. The details of the cube are irrelevant; only the way it influences the worm's body matters. The ``unknown'' label on the fifth frame is due to the fact that the worm is not stationary. \texttt{draped?} will only declare that the worm is draped if it has been still for a while.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Though this is a simple example, using the \texttt{draped?} predicate to
rlm@572: detect a cube has interesting advantages. The \texttt{draped?} predicate
rlm@572: describes the cube not in terms of properties that the cube has,
rlm@572: but instead in terms of how the worm interacts with it physically.
rlm@572: This means that the cube can still be detected even if it is not
rlm@572: visible, as long as its influence on the worm's body is visible.
rlm@572: 
rlm@572: This system will also see the virtual cube created by a
rlm@572: ``mimeworm", which uses its muscles in a very controlled way to
rlm@572: mimic the appearance of leaning on a cube. The system will
rlm@572: anticipate that there is an actual invisible cube that provides
rlm@572: support!
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=6cm]{./images/pablo-the-mime.png}
rlm@572: \caption{\label{mime}Can you see the thing that this person is leaning on? What properties does it have, other than how it makes the man's elbow and shoulder feel? I wonder if people who can actually maintain this pose easily still see the support?}
rlm@572: \end{figure}
rlm@572: 
rlm@572: This makes me wonder about the psychology of actual mimes. Suppose
rlm@572: for a moment that people have something analogous to \(\Phi\)-space and
rlm@572: that one of the ways that they find objects in a scene is by their
rlm@572: relation to other people's bodies. Suppose that a person watches a
rlm@572: person miming an invisible wall. For a person with no experience
rlm@572: with miming, their \(\Phi\)-space will only have entries that describe
rlm@572: the scene with the sensation of their hands touching a wall. This
rlm@572: sensation of touch will create a strong impression of a wall, even
rlm@572: though the wall would have to be invisible. A person with
rlm@572: experience in miming however, will have entries in their \(\Phi\)-space
rlm@572: that describe the wall-miming position without a sense of touch. It
rlm@572: will not seem to such as person that an invisible wall is present,
rlm@572: but merely that the mime is holding out their hands in a special
rlm@572: way. Thus, the theory that humans use something like \(\Phi\)-space
rlm@572: weakly predicts that learning how to mime should break the power of
rlm@572: miming illusions. Most optical illusions still work no matter how
rlm@572: much you know about them, so this proposal would be quite
rlm@572: interesting to test, as it predicts a non-standard result!
rlm@572: 
rlm@572: 
rlm@572: \clearpage
rlm@572: 
rlm@572: \section{Contributions}
rlm@572: \label{sec-4}
rlm@572: 
rlm@572: The big idea behind this thesis is a new way to represent and
rlm@572: recognize physical actions, which I call \emph{empathic representation}.
rlm@572: Actions are represented as predicates which have access to the
rlm@572: totality of a creature's sensory abilities. To recognize the
rlm@572: physical actions of another creature similar to yourself, you
rlm@572: imagine what they would feel by examining the position of their body
rlm@572: and relating it to your own previous experience.
rlm@572: 
rlm@572: Empathic representation of physical actions is robust and general.
rlm@572: Because the representation is body-centered, it avoids baking in a
rlm@572: particular viewpoint like you might get from learning from example
rlm@572: videos. Because empathic representation relies on all of a
rlm@572: creature's senses, it can describe exactly what an action \emph{feels
rlm@572: like} without getting caught up in irrelevant details such as visual
rlm@572: appearance. I think it is important that a correct description of
rlm@572: jumping (for example) should not include irrelevant details such as
rlm@572: the color of a person's clothes or skin; empathic representation can
rlm@572: get right to the heart of what jumping is by describing it in terms
rlm@572: of touch, muscle contractions, and a brief feeling of
rlm@572: weightlessness. Empathic representation is very low-level in that it
rlm@572: describes actions using concrete sensory data with little
rlm@572: abstraction, but it has the generality of much more abstract
rlm@572: representations!
rlm@572: 
rlm@572: Another important contribution of this thesis is the development of
rlm@572: the \texttt{CORTEX} system, a complete environment for creating simulated
rlm@572: creatures. You have seen how to implement five senses: touch,
rlm@572: proprioception, hearing, vision, and muscle tension. You have seen
rlm@572: how to create new creatures using Blender, a 3D modeling tool.
rlm@572: 
rlm@572: As a minor digression, you also saw how I used \texttt{CORTEX} to enable a
rlm@572: tiny worm to discover the topology of its skin simply by rolling on
rlm@572: the ground.  You also saw how to detect objects using only embodied
rlm@572: predicates. 
rlm@572: 
rlm@572: In conclusion, for this thesis I:
rlm@572: 
rlm@572: \begin{itemize}
rlm@572: \item Developed the idea of embodied representation, which describes
rlm@572: actions that a creature can do in terms of first-person sensory
rlm@572: data.
rlm@572: 
rlm@572: \item Developed a method of empathic action recognition which uses
rlm@572: previous embodied experience and embodied representation of
rlm@572: actions to greatly constrain the possible interpretations of an
rlm@572: action.
rlm@572: 
rlm@572: \item Created \texttt{EMPATH}, a program which uses empathic action
rlm@572: recognition to recognize physical actions in a simple model
rlm@572: involving segmented worm-like creatures.
rlm@572: 
rlm@572: \item Created \texttt{CORTEX}, a comprehensive platform for embodied AI
rlm@572: experiments. It is the base on which \texttt{EMPATH} is built.
rlm@572: \end{itemize}
rlm@572: 
rlm@572: \clearpage
rlm@572: \appendix
rlm@572: 
rlm@572: \section{Appendix: \texttt{CORTEX} User Guide}
rlm@572: \label{sec-5}
rlm@572: 
rlm@572: Those who write a thesis should endeavor to make their code not only
rlm@572: accessible, but actually usable, as a way to pay back the community
rlm@572: that made the thesis possible in the first place. This thesis would
rlm@572: not be possible without Free Software such as jMonkeyEngine3,
rlm@572: Blender, clojure, \texttt{emacs}, \texttt{ffmpeg}, and many other tools. That is
rlm@572: why I have included this user guide, in the hope that someone else
rlm@572: might find \texttt{CORTEX} useful.
rlm@572: 
rlm@572: \subsection{Obtaining \texttt{CORTEX}}
rlm@572: \label{sec-5-1}
rlm@572: 
rlm@572: You can get cortex from its mercurial repository at
rlm@572: \url{http://hg.bortreb.com/cortex}. You may also download \texttt{CORTEX}
rlm@572: releases at \url{http://aurellem.org/cortex/releases/}. As a condition of
rlm@572: making this thesis, I have also provided Professor Winston the
rlm@572: \texttt{CORTEX} source, and he knows how to run the demos and get started.
rlm@572: You may also email me at \texttt{cortex@aurellem.org} and I may help where
rlm@572: I can.
rlm@572: 
rlm@572: \subsection{Running \texttt{CORTEX}}
rlm@572: \label{sec-5-2}
rlm@572: 
rlm@572: \texttt{CORTEX} comes with README and INSTALL files that will guide you
rlm@572: through installation and running the test suite. In particular you
rlm@572: should look at test \texttt{cortex.test} which contains test suites that
rlm@572: run through all senses and multiple creatures.
rlm@572: 
rlm@572: \subsection{Creating creatures}
rlm@572: \label{sec-5-3}
rlm@572: 
rlm@572: Creatures are created using \emph{Blender}, a free 3D modeling program.
rlm@572: You will need Blender version 2.6 when using the \texttt{CORTEX} included
rlm@572: in this thesis. You create a \texttt{CORTEX} creature in a similar manner
rlm@572: to modeling anything in Blender, except that you also create
rlm@572: several trees of empty nodes which define the creature's senses.
rlm@572: 
rlm@572: \subsubsection{Mass}
rlm@572: \label{sec-5-3-1}
rlm@572: 
rlm@572: To give an object mass in \texttt{CORTEX}, add a ``mass'' metadata label
rlm@572: to the object with the mass in jMonkeyEngine units. Note that
rlm@572: setting the mass to 0 causes the object to be immovable.
rlm@572: 
rlm@572: \subsubsection{Joints}
rlm@572: \label{sec-5-3-2}
rlm@572: 
rlm@572: Joints are created by creating an empty node named \texttt{joints} and
rlm@572: then creating any number of empty child nodes to represent your
rlm@572: creature's joints. The joint will automatically connect the
rlm@572: closest two physical objects. It will help to set the empty node's
rlm@572: display mode to ``Arrows'' so that you can clearly see the
rlm@572: direction of the axes.
rlm@572: 
rlm@572: Joint nodes should have the following metadata under the ``joint''
rlm@572: label:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: ;; ONE of the following, under the label "joint":
rlm@572: {:type :point}
rlm@572: 
rlm@572: ;; OR
rlm@572: 
rlm@572: {:type :hinge
rlm@572:  :limit [<limit-low> <limit-high>]
rlm@572:  :axis (Vector3f. <x> <y> <z>)}
rlm@572: ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@572: 
rlm@572: ;; OR
rlm@572: 
rlm@572: {:type :cone
rlm@572:  :limit-xz <lim-xz>
rlm@572:  :limit-xy <lim-xy>
rlm@572:  :twist    <lim-twist>}   ;(use XZY rotation mode in Blender!)
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: \subsubsection{Eyes}
rlm@572: \label{sec-5-3-3}
rlm@572: 
rlm@572: Eyes are created by creating an empty node named \texttt{eyes} and then
rlm@572: creating any number of empty child nodes to represent your
rlm@572: creature's eyes.
rlm@572: 
rlm@572: Eye nodes should have the following metadata under the ``eye''
rlm@572: label:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: {:red    <red-retina-definition>
rlm@572:  :blue   <blue-retina-definition>
rlm@572:  :green  <green-retina-definition>
rlm@572:  :all    <all-retina-definition>
rlm@572:  (<0xrrggbb> <custom-retina-image>)...
rlm@572: }
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: Any of the color channels may be omitted. You may also include
rlm@572: your own color selectors, and in fact :red is equivalent to
rlm@572: 0xFF0000 and so forth. The eye will be placed at the same position
rlm@572: as the empty node and will bind to the neatest physical object.
rlm@572: The eye will point outward from the X-axis of the node, and ``up''
rlm@572: will be in the direction of the X-axis of the node. It will help
rlm@572: to set the empty node's display mode to ``Arrows'' so that you can
rlm@572: clearly see the direction of the axes.
rlm@572: 
rlm@572: Each retina file should contain white pixels wherever you want to be
rlm@572: sensitive to your chosen color. If you want the entire field of
rlm@572: view, specify :all of 0xFFFFFF and a retinal map that is entirely
rlm@572: white. 
rlm@572: 
rlm@572: Here is a sample retinal map:
rlm@572: 
rlm@572: \begin{figure}[H]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/retina-small.png}
rlm@572: \caption{\label{retina}An example retinal profile image. White pixels are photo-sensitive elements. The distribution of white pixels is denser in the middle and falls off at the edges and is inspired by the human retina.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{Hearing}
rlm@572: \label{sec-5-3-4}
rlm@572: 
rlm@572: Ears are created by creating an empty node named \texttt{ears} and then
rlm@572: creating any number of empty child nodes to represent your
rlm@572: creature's ears. 
rlm@572: 
rlm@572: Ear nodes do not require any metadata.
rlm@572: 
rlm@572: The ear will bind to and follow the closest physical node.
rlm@572: 
rlm@572: \subsubsection{Touch}
rlm@572: \label{sec-5-3-5}
rlm@572: 
rlm@572: Touch is handled similarly to mass. To make a particular object
rlm@572: touch sensitive, add metadata of the following form under the
rlm@572: object's ``touch'' metadata field:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: <touch-UV-map-file-name>
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: You may also include an optional ``scale'' metadata number to
rlm@572: specify the length of the touch feelers. The default is \(0.1\),
rlm@572: and this is generally sufficient.
rlm@572: 
rlm@572: The touch UV should contain white pixels for each touch sensor.
rlm@572: 
rlm@572: Here is an example touch-uv map that approximates a human finger,
rlm@572: and its corresponding model.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=9cm]{./images/finger-UV.png}
rlm@572: \caption{\label{guide-fingertip-UV}This is the tactile-sensor-profile for the upper segment of a fingertip. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=9cm]{./images/finger-1.png}
rlm@572: \caption{\label{guide-fingertip}The fingertip UV-image form above applied to a simple model of a fingertip.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: \subsubsection{Proprioception}
rlm@572: \label{sec-5-3-6}
rlm@572: 
rlm@572: Proprioception is tied to each joint node -- nothing special must
rlm@572: be done in a Blender model to enable proprioception other than
rlm@572: creating joint nodes.
rlm@572: 
rlm@572: \subsubsection{Muscles}
rlm@572: \label{sec-5-3-7}
rlm@572: 
rlm@572: Muscles are created by creating an empty node named \texttt{muscles} and
rlm@572: then creating any number of empty child nodes to represent your
rlm@572: creature's muscles.
rlm@572: 
rlm@572: 
rlm@572: Muscle nodes should have the following metadata under the
rlm@572: ``muscle'' label:
rlm@572: 
rlm@572: \begin{verbatim}
rlm@572: <muscle-profile-file-name>
rlm@572: \end{verbatim}
rlm@572: 
rlm@572: Muscles should also have a ``strength'' metadata entry describing
rlm@572: the muscle's total strength at full activation. 
rlm@572: 
rlm@572: Muscle profiles are simple images that contain the relative amount
rlm@572: of muscle power in each simulated alpha motor neuron. The width of
rlm@572: the image is the total size of the motor pool, and the redness of
rlm@572: each neuron is the relative power of that motor pool.
rlm@572: 
rlm@572: While the profile image can have any dimensions, only the first
rlm@572: line of pixels is used to define the muscle. Here is a sample
rlm@572: muscle profile image that defines a human-like muscle.
rlm@572: 
rlm@572: \begin{figure}[htb]
rlm@572: \centering
rlm@572: \includegraphics[width=7cm]{./images/basic-muscle.png}
rlm@572: \caption{\label{muscle-recruit}A muscle profile image that describes the strengths of each motor neuron in a muscle. White is weakest and dark red is strongest. This particular pattern has weaker motor neurons at the beginning, just like human muscle.}
rlm@572: \end{figure}
rlm@572: 
rlm@572: Muscles twist the nearest physical object about the muscle node's
rlm@572: Z-axis. I recommend using the ``Single Arrow'' display mode for
rlm@572: muscles and using the right hand rule to determine which way the
rlm@572: muscle will twist. To make a segment that can twist in multiple
rlm@572: directions, create multiple, differently aligned muscles.
rlm@572: 
rlm@572: \subsection{\texttt{CORTEX} API}
rlm@572: \label{sec-5-4}
rlm@572: 
rlm@572: These are the some functions exposed by \texttt{CORTEX} for creating
rlm@572: worlds and simulating creatures. These are in addition to
rlm@572: jMonkeyEngine3's extensive library, which is documented elsewhere.
rlm@572: 
rlm@572: \subsubsection{Simulation}
rlm@572: \label{sec-5-4-1}
rlm@572: \begin{description}
rlm@572: \item[{\texttt{(world root-node key-map setup-fn update-fn)}}] create
rlm@572: a simulation.
rlm@572: \begin{description}
rlm@572: \item[{\emph{root-node}    }] a \texttt{com.jme3.scene.Node} object which
rlm@572: contains all of the objects that should be in the
rlm@572: simulation.
rlm@572: 
rlm@572: \item[{\emph{key-map}      }] a map from strings describing keys to
rlm@572: functions that should be executed whenever that key is
rlm@572: pressed. the functions should take a SimpleApplication
rlm@572: object and a boolean value. The SimpleApplication is the
rlm@572: current simulation that is running, and the boolean is true
rlm@572: if the key is being pressed, and false if it is being
rlm@572: released. As an example,
rlm@572: \begin{verbatim}
rlm@572:        {"key-j" (fn [game value] (if value (println "key j pressed")))}
rlm@572: \end{verbatim}
rlm@572: is a valid key-map which will cause the simulation to print
rlm@572: a message whenever the 'j' key on the keyboard is pressed.
rlm@572: 
rlm@572: \item[{\emph{setup-fn}     }] a function that takes a \texttt{SimpleApplication}
rlm@572: object. It is called once when initializing the simulation.
rlm@572: Use it to create things like lights, change the gravity,
rlm@572: initialize debug nodes, etc.
rlm@572: 
rlm@572: \item[{\emph{update-fn}    }] this function takes a \texttt{SimpleApplication}
rlm@572: object and a float and is called every frame of the
rlm@572: simulation. The float tells how many seconds is has been
rlm@572: since the last frame was rendered, according to whatever
rlm@572: clock jme is currently using. The default is to use IsoTimer
rlm@572: which will result in this value always being the same.
rlm@572: \end{description}
rlm@572: 
rlm@572: \item[{\texttt{(position-camera world position rotation)}}] set the position
rlm@572: of the simulation's main camera.
rlm@572: 
rlm@572: \item[{\texttt{(enable-debug world)}}] turn on debug wireframes for each
rlm@572: simulated object.
rlm@572: 
rlm@572: \item[{\texttt{(set-gravity world gravity)}}] set the gravity of a running
rlm@572: simulation.
rlm@572: 
rlm@572: \item[{\texttt{(box length width height \& \{options\})}}] create a box in the
rlm@572: simulation. Options is a hash map specifying texture, mass,
rlm@572: etc. Possible options are \texttt{:name}, \texttt{:color}, \texttt{:mass},
rlm@572: \texttt{:friction}, \texttt{:texture}, \texttt{:material}, \texttt{:position},
rlm@572: \texttt{:rotation}, \texttt{:shape}, and \texttt{:physical?}.
rlm@572: 
rlm@572: \item[{\texttt{(sphere radius \& \{options\})}}] create a sphere in the simulation.
rlm@572: Options are the same as in \texttt{box}.
rlm@572: 
rlm@572: \item[{\texttt{(load-blender-model file-name)}}] create a node structure
rlm@572: representing the model described in a Blender file.
rlm@572: 
rlm@572: \item[{\texttt{(light-up-everything world)}}] distribute a standard compliment
rlm@572: of lights throughout the simulation. Should be adequate for most
rlm@572: purposes.
rlm@572: 
rlm@572: \item[{\texttt{(node-seq node)}}] return a recursive list of the node's
rlm@572: children.
rlm@572: 
rlm@572: \item[{\texttt{(nodify name children)}}] construct a node given a node-name and
rlm@572: desired children.
rlm@572: 
rlm@572: \item[{\texttt{(add-element world element)}}] add an object to a running world
rlm@572: simulation.
rlm@572: 
rlm@572: \item[{\texttt{(set-accuracy world accuracy)}}] change the accuracy of the
rlm@572: world's physics simulator.
rlm@572: 
rlm@572: \item[{\texttt{(asset-manager)}}] get an \emph{AssetManager}, a jMonkeyEngine
rlm@572: construct that is useful for loading textures and is required
rlm@572: for smooth interaction with jMonkeyEngine library functions.
rlm@572: 
rlm@572: \item[{\texttt{(load-bullet)}  }] unpack native libraries and initialize the
rlm@572: bullet physics subsystem. This function is required before
rlm@572: other world building functions are called.
rlm@572: \end{description}
rlm@572: 
rlm@572: \subsubsection{Creature Manipulation / Import}
rlm@572: \label{sec-5-4-2}
rlm@572: 
rlm@572: \begin{description}
rlm@572: \item[{\texttt{(body! creature)}}] give the creature a physical body.
rlm@572: 
rlm@572: \item[{\texttt{(vision! creature)}}] give the creature a sense of vision.
rlm@572: Returns a list of functions which will each, when called
rlm@572: during a simulation, return the vision data for the channel of
rlm@572: one of the eyes. The functions are ordered depending on the
rlm@572: alphabetical order of the names of the eye nodes in the
rlm@572: Blender file. The data returned by the functions is a vector
rlm@572: containing the eye's \emph{topology}, a vector of coordinates, and
rlm@572: the eye's \emph{data}, a vector of RGB values filtered by the eye's
rlm@572: sensitivity.
rlm@572: 
rlm@572: \item[{\texttt{(hearing! creature)}}] give the creature a sense of hearing.
rlm@572: Returns a list of functions, one for each ear, that when
rlm@572: called will return a frame's worth of hearing data for that
rlm@572: ear. The functions are ordered depending on the alphabetical
rlm@572: order of the names of the ear nodes in the Blender file. The
rlm@572: data returned by the functions is an array of PCM (pulse code
rlm@572: modulated) wav data.
rlm@572: 
rlm@572: \item[{\texttt{(touch! creature)}}] give the creature a sense of touch. Returns
rlm@572: a single function that must be called with the \emph{root node} of
rlm@572: the world, and which will return a vector of \emph{touch-data}
rlm@572: one entry for each touch sensitive component, each entry of
rlm@572: which contains a \emph{topology} that specifies the distribution of
rlm@572: touch sensors, and the \emph{data}, which is a vector of
rlm@572: \texttt{[activation, length]} pairs for each touch hair.
rlm@572: 
rlm@572: \item[{\texttt{(proprioception! creature)}}] give the creature the sense of
rlm@572: proprioception. Returns a list of functions, one for each
rlm@572: joint, that when called during a running simulation will
rlm@572: report the \texttt{[heading, pitch, roll]} of the joint.
rlm@572: 
rlm@572: \item[{\texttt{(movement! creature)}}] give the creature the power of movement.
rlm@572: Creates a list of functions, one for each muscle, that when
rlm@572: called with an integer, will set the recruitment of that
rlm@572: muscle to that integer, and will report the current power
rlm@572: being exerted by the muscle. Order of muscles is determined by
rlm@572: the alphabetical sort order of the names of the muscle nodes.
rlm@572: \end{description}
rlm@572: 
rlm@572: \subsubsection{Visualization/Debug}
rlm@572: \label{sec-5-4-3}
rlm@572: 
rlm@572: \begin{description}
rlm@572: \item[{\texttt{(view-vision)}}] create a function that when called with a list
rlm@572: of visual data returned from the functions made by \texttt{vision!}, 
rlm@572: will display that visual data on the screen.
rlm@572: 
rlm@572: \item[{\texttt{(view-hearing)}}] same as \texttt{view-vision} but for hearing.
rlm@572: 
rlm@572: \item[{\texttt{(view-touch)}}] same as \texttt{view-vision} but for touch.
rlm@572: 
rlm@572: \item[{\texttt{(view-proprioception)}}] same as \texttt{view-vision} but for
rlm@572: proprioception.
rlm@572: 
rlm@572: \item[{\texttt{(view-movement)}}] same as \texttt{view-vision} but for muscles.
rlm@572: 
rlm@572: \item[{\texttt{(view anything)}}] \texttt{view} is a polymorphic function that allows
rlm@572: you to inspect almost anything you could reasonably expect to
rlm@572: be able to ``see'' in \texttt{CORTEX}.
rlm@572: 
rlm@572: \item[{\texttt{(text anything)}}] \texttt{text} is a polymorphic function that allows
rlm@572: you to convert practically anything into a text string.
rlm@572: 
rlm@572: \item[{\texttt{(println-repl anything)}}] print messages to clojure's repl
rlm@572: instead of the simulation's terminal window.
rlm@572: 
rlm@572: \item[{\texttt{(mega-import-jme3)}}] for experimenting at the REPL. This
rlm@572: function will import all jMonkeyEngine3 classes for immediate
rlm@572: use.
rlm@572: 
rlm@572: \item[{\texttt{(display-dilated-time world timer)}}] Shows the time as it is
rlm@572: flowing in the simulation on a HUD display.
rlm@572: \end{description}