cortex: thesis/cortex.org comparison

comparison thesis/cortex.org @ 449:09b7c8dd4365

first chapter done, half of last chapter done.

author	Robert McIntyre <rlm@mit.edu>
date	Wed, 26 Mar 2014 02:42:01 -0400
parents	af13fc73e851
children	432f2c4646cb

comparison

equal deleted inserted replaced

-:af13fc73e851
+:09b7c8dd4365
 (< 0.55 (contact worm-segment-top-tip    head-touch))))))
 #+end_src
 #+end_listing
-** =CORTEX= is a toolkit for building sensate creatures
+**  =CORTEX= is a toolkit for building sensate creatures
 I built =CORTEX= to be a general AI research platform for doing
 experiments involving multiple rich senses and a wide variety and
 number of creatures. I intend it to be useful as a library for many
 more projects than just this one. =CORTEX= was necessary to meet a
 =CORTEX= is built on top of =jMonkeyEngine3=, which is a video game
 engine designed to create cross-platform 3D desktop games. =CORTEX=
 is mainly written in clojure, a dialect of =LISP= that runs on the
 java virtual machine (JVM). The API for creating and simulating
-creatures is entirely expressed in clojure. Hearing is implemented
+creatures and senses is entirely expressed in clojure, though many
-as a layer of clojure code on top of a layer of java code on top of
+senses are implemented at the layer of jMonkeyEngine or below. For
-a layer of =C++= code which implements a modified version of
+example, for the sense of hearing I use a layer of clojure code on
-=OpenAL= to support multiple listeners. =CORTEX= is the only
+top of a layer of java JNI bindings that drive a layer of =C++=
-simulation environment that I know of that can support multiple
+code which implements a modified version of =OpenAL= to support
-entities that can each hear the world from their own perspective.
+multiple listeners. =CORTEX= is the only simulation environment
-Other senses also require a small layer of Java code. =CORTEX= also
+that I know of that can support multiple entities that can each
-uses =bullet=, a physics simulator written in =C=.
+hear the world from their own perspective. Other senses also
+require a small layer of Java code. =CORTEX= also uses =bullet=, a
+physics simulator written in =C=.
 #+caption: Here is the worm from above modeled in Blender, a free
 #+caption: 3D-modeling program. Senses and joints are described
 #+caption: using special nodes in Blender.
 #+name: worm-recognition-intro
 #+ATTR_LaTeX: :width 12cm
 [[./images/blender-worm.png]]
+Here are some thing I anticipate that =CORTEX= might be used for:
+- exploring new ideas about sensory integration
+- distributed communication among swarm creatures
+- self-learning using free exploration,
+- evolutionary algorithms involving creature construction
+- exploration of exoitic senses and effectors that are not possible
+in the real world (such as telekenisis or a semantic sense)
+- imagination using subworlds
 During one test with =CORTEX=, I created 3,000 entities each with
 their own independent senses and ran them all at only 1/80 real
 time. In another test, I created a detailed model of my own hand,
 equipped with a realistic distribution of touch (more sensitive at
 the fingertips), as well as eyes and ears, and it ran at around 1/4
 real time.
-#+caption: Here is the worm from above modeled in Blender, a free
+#+BEGIN_LaTeX
-#+caption: 3D-modeling program. Senses and joints are described
+\begin{sidewaysfigure}
-#+caption: using special nodes in Blender.
+\includegraphics[width=9.5in]{images/full-hand.png}
-#+name: worm-recognition-intro
+\caption{Here is the worm from above modeled in Blender,
-#+ATTR_LaTeX: :width 15cm
+a free 3D-modeling program. Senses and joints are described
-[[./images/full-hand.png]]
+using special nodes in Blender. The senses are displayed on
+the right, and the simulation is displayed on the left. Notice
+that the hand is curling its fingers, that it can see its own
+finger from the eye in its palm, and thta it can feel its own
+thumb touching its palm.}
+\end{sidewaysfigure}
+#+END_LaTeX
 ** Contributions
+I built =CORTEX=, a comprehensive platform for embodied AI
+experiments. =CORTEX= many new features lacking in other systems,
+such as sound. It is easy to create new creatures using Blender, a
+free 3D modeling program.
+I built =EMPATH=, which uses =CORTEX= to identify the actions of a
+worm-like creature using a computational model of empathy.
 * Building =CORTEX=
 ** To explore embodiment, we need a world, body, and senses
 ** Because of Time, simulation is perferable to reality
 ** =CORTEX= enables many possiblities for further research
 * Empathy in a simulated worm
+Here I develop a computational model of empathy, using =CORTEX= as a
+base. Empathy in this context is the ability to observe another
+creature and infer what sorts of sensations that creature is
+feeling. My empathy algorithm involves multiple phases. First is
+free-play, where the creature moves around and gains sensory
+experience. From this experience I construct a representation of the
+creature's sensory state space, which I call \Phi-space. Using
+\Phi-space, I construct an efficient function which takes the
+limited data that comes from observing another creature and enriches
+it full compliment of imagined sensory data. I can then use the
+imagined sensory data to recognize what the observed creature is
+doing and feeling, using straightforward embodied action predicates.
+This is all demonstrated with using a simple worm-like creature, and
+recognizing worm-actions based on limited data.
+#+caption: Here is the worm with which we will be working.
+#+caption: It is composed of 5 segments. Each segment has a
+#+caption: pair of extensor and flexor muscles. Each of the
+#+caption: worm's four joints is a hinge joint which allows
+#+caption: 30 degrees of rotation to either side. Each segment
+#+caption: of the worm is touch-capable and has a uniform
+#+caption: distribution of touch sensors on each of its faces.
+#+caption: Each joint has a proprioceptive sense to detect
+#+caption: relative positions. The worm segments are all the
+#+caption: same except for the first one, which has a much
+#+caption: higher weight than the others to allow for easy
+#+caption: manual motor control.
+#+name: basic-worm-view
+#+ATTR_LaTeX: :width 10cm
+[[./images/basic-worm-view.png]]
+#+caption: Program for reading a worm from a blender file and
+#+caption: outfitting it with the senses of proprioception,
+#+caption: touch, and the ability to move, as specified in the
+#+caption: blender file.
+#+name: get-worm
+#+begin_listing clojure
+#+begin_src clojure
+(defn worm []
+(let [model (load-blender-model "Models/worm/worm.blend")]
+{:body (doto model (body!))
+:touch (touch! model)
+:proprioception (proprioception! model)
+:muscles (movement! model)}))
+#+end_src
+#+end_listing
 ** Embodiment factors action recognition into managable parts
+Using empathy, I divide the problem of action recognition into a
+recognition process expressed in the language of a full compliment
+of senses, and an imaganitive process that generates full sensory
+data from partial sensory data. Splitting the action recognition
+problem in this manner greatly reduces the total amount of work to
+recognize actions: The imaganitive process is mostly just matching
+previous experience, and the recognition process gets to use all
+the senses to directly describe any action.
 ** Action recognition is easy with a full gamut of senses
-** Digression: bootstrapping touch using free exploration
+Embodied representations using multiple senses such as touch,
+proprioception, and muscle tension turns out be be exceedingly
+efficient at describing body-centered actions. It is the ``right
+language for the job''. For example, it takes only around 5 lines
+of LISP code to describe the action of ``curling'' using embodied
+primitives. It takes about 8 lines to describe the seemingly
+complicated action of wiggling.
+The following action predicates each take a stream of sensory
+experience, observe however much of it they desire, and decide
+whether the worm is doing the action they describe. =curled?=
+relies on proprioception, =resting?= relies on touch, =wiggling?=
+relies on a fourier analysis of muscle contraction, and
+=grand-circle?= relies on touch and reuses =curled?= as a gaurd.
+#+caption: Program for detecting whether the worm is curled. This is the
+#+caption: simplest action predicate, because it only uses the last frame
+#+caption: of sensory experience, and only uses proprioceptive data. Even
+#+caption: this simple predicate, however, is automatically frame
+#+caption: independent and ignores vermopomorphic differences such as
+#+caption: worm textures and colors.
+#+name: curled
+#+begin_listing clojure
+#+begin_src clojure
+(defn curled?
+"Is the worm curled up?"
+[experiences]
+(every?
+(fn [[_ _ bend]]
+(> (Math/sin bend) 0.64))
+(:proprioception (peek experiences))))
+#+end_src
+#+end_listing
+#+caption: Program for summarizing the touch information in a patch
+#+caption: of skin.
+#+name: touch-summary
+#+begin_listing clojure
+#+begin_src clojure
+(defn contact
+"Determine how much contact a particular worm segment has with
+other objects. Returns a value between 0 and 1, where 1 is full
+contact and 0 is no contact."
+[touch-region [coords contact :as touch]]
+(-> (zipmap coords contact)
+(select-keys touch-region)
+(vals)
+(#(map first %))
+(average)
+(* 10)
+(- 1)
+(Math/abs)))
+#+end_src
+#+end_listing
+#+caption: Program for detecting whether the worm is at rest. This program
+#+caption: uses a summary of the tactile information from the underbelly
+#+caption: of the worm, and is only true if every segment is touching the
+#+caption: floor. Note that this function contains no references to
+#+caption: proprioction at all.
+#+name: resting
+#+begin_listing clojure
+#+begin_src clojure
+(def worm-segment-bottom (rect-region [8 15] [14 22]))
+(defn resting?
+"Is the worm resting on the ground?"
+[experiences]
+(every?
+(fn [touch-data]
+(< 0.9 (contact worm-segment-bottom touch-data)))
+(:touch (peek experiences))))
+#+end_src
+#+end_listing
+#+caption: Program for detecting whether the worm is curled up into a
+#+caption: full circle. Here the embodied approach begins to shine, as
+#+caption: I am able to both use a previous action predicate (=curled?=)
+#+caption: as well as the direct tactile experience of the head and tail.
+#+name: grand-circle
+#+begin_listing clojure
+#+begin_src clojure
+(def worm-segment-bottom-tip (rect-region [15 15] [22 22]))
+(def worm-segment-top-tip (rect-region [0 15] [7 22]))
+(defn grand-circle?
+"Does the worm form a majestic circle (one end touching the other)?"
+[experiences]
+(and (curled? experiences)
+(let [worm-touch (:touch (peek experiences))
+tail-touch (worm-touch 0)
+head-touch (worm-touch 4)]
+(and (< 0.55 (contact worm-segment-bottom-tip tail-touch))
+(< 0.55 (contact worm-segment-top-tip    head-touch))))))
+#+end_src
+#+end_listing
+#+caption: Program for detecting whether the worm has been wiggling for
+#+caption: the last few frames. It uses a fourier analysis of the muscle
+#+caption: contractions of the worm's tail to determine wiggling. This is
+#+caption: signigicant because there is no particular frame that clearly
+#+caption: indicates that the worm is wiggling --- only when multiple frames
+#+caption: are analyzed together is the wiggling revealed. Defining
+#+caption: wiggling this way also gives the worm an opportunity to learn
+#+caption: and recognize ``frustrated wiggling'', where the worm tries to
+#+caption: wiggle but can't. Frustrated wiggling is very visually different
+#+caption: from actual wiggling, but this definition gives it to us for free.
+#+name: wiggling
+#+begin_listing clojure
+#+begin_src clojure
+(defn fft [nums]
+(map
+#(.getReal %)
+(.transform
+(FastFourierTransformer. DftNormalization/STANDARD)
+(double-array nums) TransformType/FORWARD)))
+(def indexed (partial map-indexed vector))
+(defn max-indexed [s]
+(first (sort-by (comp - second) (indexed s))))
+(defn wiggling?
+"Is the worm wiggling?"
+[experiences]
+(let [analysis-interval 0x40]
+(when (> (count experiences) analysis-interval)
+(let [a-flex 3
+a-ex   2
+muscle-activity
+(map :muscle (vector:last-n experiences analysis-interval))
+base-activity
+(map #(- (% a-flex) (% a-ex)) muscle-activity)]
+(= 2
+(first
+(max-indexed
+(map #(Math/abs %)
+(take 20 (fft base-activity))))))))))
+#+end_src
+#+end_listing
+With these action predicates, I can now recognize the actions of
+the worm while it is moving under my control and I have access to
+all the worm's senses.
+#+caption: Use the action predicates defined earlier to report on
+#+caption: what the worm is doing while in simulation.
+#+name: report-worm-activity
+#+begin_listing clojure
+#+begin_src clojure
+(defn debug-experience
+[experiences text]
+(cond
+(grand-circle? experiences) (.setText text "Grand Circle")
+(curled? experiences)       (.setText text "Curled")
+(wiggling? experiences)     (.setText text "Wiggling")
+(resting? experiences)      (.setText text "Resting")))
+#+end_src
+#+end_listing
+#+caption: Using =debug-experience=, the body-centered predicates
+#+caption: work together to classify the behaviour of the worm.
+#+caption: while under manual motor control.
+#+name: basic-worm-view
+#+ATTR_LaTeX: :width 10cm
+[[./images/worm-identify-init.png]]
+These action predicates satisfy the recognition requirement of an
+empathic recognition system. There is a lot of power in the
+simplicity of the action predicates. They describe their actions
+without getting confused in visual details of the worm. Each one is
+frame independent, but more than that, they are each indepent of
+irrelevant visual details of the worm and the environment. They
+will work regardless of whether the worm is a different color or
+hevaily textured, or of the environment has strange lighting.
+The trick now is to make the action predicates work even when the
+sensory data on which they depend is absent. If I can do that, then
+I will have gained much,
 ** \Phi-space describes the worm's experiences
+As a first step towards building empathy, I need to gather all of
+the worm's experiences during free play. I use a simple vector to
+store all the experiences.
+#+caption: Program to gather the worm's experiences into a vector for
+#+caption: further processing. The =motor-control-program= line uses
+#+caption: a motor control script that causes the worm to execute a series
+#+caption: of ``exercices'' that include all the action predicates.
+#+name: generate-phi-space
+#+begin_listing clojure
+#+begin_src clojure
+(defn generate-phi-space []
+(let [experiences (atom [])]
+(run-world
+(apply-map
+worm-world
+(merge
+(worm-world-defaults)
+{:end-frame 700
+:motor-control
+(motor-control-program worm-muscle-labels do-all-the-things)
+:experiences experiences})))
+@experiences))
+#+end_src
+#+end_listing
+Each element of the experience vector exists in the vast space of
+all possible worm-experiences. Most of this vast space is actually
+unreachable due to physical constraints of the worm's body. For
+example, the worm's segments are connected by hinge joints that put
+a practical limit on the worm's degrees of freedom. Also, the worm
+can not be bent into a circle so that its ends are touching and at
+the same time not also experience the sensation of touching itself.
+As the worm moves around during free play and the vector grows
+larger, the vector begins to define a subspace which is all the
+practical experiences the worm can experience during normal
+operation, which I call \Phi-space, short for physical-space. The
+vector defines a path through \Phi-space. This path has interesting
+properties that all derive from embodiment. The proprioceptive
+components are completely smooth, because in order for the worm to
+move from one position to another, it must pass through the
+intermediate positions. The path invariably forms loops as actions
+are repeated. Finally and most importantly, proprioception actually
+gives very strong inference about the other senses. For example,
+when the worm is flat, you can infer that it is touching the ground
+and that its muscles are not active, because if the muscles were
+active, the worm would be moving and would not be perfectly flat.
+In order to stay flat, the worm has to be touching the ground, or
+it would again be moving out of the flat position due to gravity.
+If the worm is positioned in such a way that it interacts with
+itself, then it is very likely to be feeling the same tactile
+feelings as the last time it was in that position, because it has
+the same body as then. If you observe multiple frames of
+proprioceptive data, then you can become increasingly confident
+about the exact activations of the worm's muscles, because it
+generally takes a unique combination of muscle contractions to
+transform the worm's body along a specific path through \Phi-space.
+There is a simple way of taking \Phi-space and the total ordering
+provided by an experience vector and reliably infering the rest of
+the senses.
 ** Empathy is the process of tracing though \Phi-space
+(defn bin [digits]
+(fn [angles]
+(->> angles
+(flatten)
+(map (juxt #(Math/sin %) #(Math/cos %)))
+(flatten)
+(mapv #(Math/round (* % (Math/pow 10 (dec digits))))))))
+(defn gen-phi-scan
+"Nearest-neighbors with spatial binning. Only returns a result if
+the propriceptive data is within 10% of a previously recorded
+result in all dimensions."
+[phi-space]
+(let [bin-keys (map bin [3 2 1])
+bin-maps
+(map (fn [bin-key]
+(group-by
+(comp bin-key :proprioception phi-space)
+(range (count phi-space)))) bin-keys)
+lookups (map (fn [bin-key bin-map]
+(fn [proprio] (bin-map (bin-key proprio))))
+bin-keys bin-maps)]
+(fn lookup [proprio-data]
+(set (some #(% proprio-data) lookups)))))
+(defn longest-thread
+"Find the longest thread from phi-index-sets. The index sets should
+be ordered from most recent to least recent."
+[phi-index-sets]
+(loop [result '()
+[thread-bases & remaining :as phi-index-sets] phi-index-sets]
+(if (empty? phi-index-sets)
+(vec result)
+(let [threads
+(for [thread-base thread-bases]
+(loop [thread (list thread-base)
+remaining remaining]
+(let [next-index (dec (first thread))]
+(cond (empty? remaining) thread
+(contains? (first remaining) next-index)
+(recur
+(cons next-index thread) (rest remaining))
+:else thread))))
+longest-thread
+(reduce (fn [thread-a thread-b]
+(if (> (count thread-a) (count thread-b))
+thread-a thread-b))
+'(nil)
+threads)]
+(recur (concat longest-thread result)
+(drop (count longest-thread) phi-index-sets))))))
+There is one final piece, which is to replace missing sensory data
+with a best-guess estimate. While I could fill in missing data by
+using a gradient over the closest known sensory data points, averages
+can be misleading. It is certainly possible to create an impossible
+sensory state by averaging two possible sensory states. Therefore, I
+simply replicate the most recent sensory experience to fill in the
+gaps.
+#+caption: Fill in blanks in sensory experience by replicating the most
+#+caption: recent experience.
+#+name: infer-nils
+#+begin_listing clojure
+#+begin_src clojure
+(defn infer-nils
+"Replace nils with the next available non-nil element in the
+sequence, or barring that, 0."
+[s]
+(loop [i (dec (count s))
+v (transient s)]
+(if (zero? i) (persistent! v)
+(if-let [cur (v i)]
+(if (get v (dec i) 0)
+(recur (dec i) v)
+(recur (dec i) (assoc! v (dec i) cur)))
+(recur i (assoc! v i 0))))))
+#+end_src
+#+end_listing
 ** Efficient action recognition with =EMPATH=
+** Digression: bootstrapping touch using free exploration
 * Contributions
-- Built =CORTEX=, a comprehensive platform for embodied AI
-experiments. Has many new features lacking in other systems, such
-as sound. Easy to model/create new creatures.
-- created a novel concept for action recognition by using artificial
-imagination.
-In the second half of the thesis I develop a computational model of
-empathy, using =CORTEX= as a base. Empathy in this context is the
-ability to observe another creature and infer what sorts of sensations
-that creature is feeling. My empathy algorithm involves multiple
-phases. First is free-play, where the creature moves around and gains
-sensory experience. From this experience I construct a representation
-of the creature's sensory state space, which I call \Phi-space. Using
-\Phi-space, I construct an efficient function for enriching the
-limited data that comes from observing another creature with a full
-compliment of imagined sensory data based on previous experience. I
-can then use the imagined sensory data to recognize what the observed
-creature is doing and feeling, using straightforward embodied action
-predicates. This is all demonstrated with using a simple worm-like
-creature, and recognizing worm-actions based on limited data.
-Embodied representation using multiple senses such as touch,
-proprioception, and muscle tension turns out be be exceedingly
-efficient at describing body-centered actions. It is the ``right
-language for the job''. For example, it takes only around 5 lines of
-LISP code to describe the action of ``curling'' using embodied
-primitives. It takes about 8 lines to describe the seemingly
-complicated action of wiggling.
-* COMMENT names for cortex
-- bioland
 # An anatomical joke:

Mercurial > cortex

comparison thesis/cortex.org @ 449:09b7c8dd4365