cortex: thesis/cortex.org comparison

comparison thesis/cortex.org @ 547:5d89879fc894

couple hours worth of edits.

author	Robert McIntyre <rlm@mit.edu>
date	Mon, 28 Apr 2014 15:10:59 -0400
parents	b2c66ea58c39
children	0b891e0dd809

comparison

equal deleted inserted replaced

-:f4770e3d30ae
+:5d89879fc894
 [[./images/aurellem-gray.png]]
 * Empathy \& Embodiment: problem solving strategies
-By the end of this thesis, you will have seen a novel approach to
+By the end of this thesis, you will have a novel approach to
-interpreting video using embodiment and empathy. You will also see
+representing an recognizing physical actions using embodiment and
-one way to efficiently implement physical empathy for embodied
+empathy. You will also see one way to efficiently implement physical
-creatures. Finally, you will become familiar with =CORTEX=, a system
+empathy for embodied creatures. Finally, you will become familiar
-for designing and simulating creatures with rich senses, which I
+with =CORTEX=, a system for designing and simulating creatures with
-have designed as a library that you can use in your own research.
+rich senses, which I have designed as a library that you can use in
-Note that I /do not/ process video directly --- I start with
+your own research. Note that I /do not/ process video directly --- I
-knowledge of the positions of a creature's body parts and works from
+start with knowledge of the positions of a creature's body parts and
-there.
+works from there.
 This is the core vision of my thesis: That one of the important ways
 in which we understand others is by imagining ourselves in their
 position and emphatically feeling experiences relative to our own
 bodies. By understanding events in terms of our own previous
 the problem is that many computer vision systems focus on
 pixel-level details or comparisons to example images (such as
 \cite{volume-action-recognition}), but the 3D world is so variable
 that it is hard to describe the world in terms of possible images.
-In fact, the contents of scene may have much less to do with pixel
+In fact, the contents of a scene may have much less to do with
-probabilities than with recognizing various affordances: things you
+pixel probabilities than with recognizing various affordances:
-can move, objects you can grasp, spaces that can be filled . For
+things you can move, objects you can grasp, spaces that can be
-example, what processes might enable you to see the chair in figure
+filled . For example, what processes might enable you to see the
-\ref{hidden-chair}?
+chair in figure \ref{hidden-chair}?
 #+caption: The chair in this image is quite obvious to humans, but
 #+caption: it can't be found by any modern computer vision program.
 #+name: hidden-chair
 #+ATTR_LaTeX: :width 10cm
 [[./images/wall-push.png]]
 Each of these examples tells us something about what might be going
 on in our minds as we easily solve these recognition problems:
-The hidden chair shows us that we are strongly triggered by cues
+- The hidden chair shows us that we are strongly triggered by cues
-relating to the position of human bodies, and that we can determine
+relating to the position of human bodies, and that we can
-the overall physical configuration of a human body even if much of
+determine the overall physical configuration of a human body even
-that body is occluded.
+if much of that body is occluded.
-The picture of the girl pushing against the wall tells us that we
+- The picture of the girl pushing against the wall tells us that we
 have common sense knowledge about the kinetics of our own bodies.
 We know well how our muscles would have to work to maintain us in
 most positions, and we can easily project this self-knowledge to
 imagined positions triggered by images of the human body.
-The cat tells us that imagination of some kind plays an important
+- The cat tells us that imagination of some kind plays an important
 role in understanding actions. The question is: Can we be more
 precise about what sort of imagination is required to understand
 these actions?
 ** A step forward: the sensorimotor-centered approach
 In this thesis, I explore the idea that our knowledge of our own
 bodies, combined with our own rich senses, enables us to recognize
 imagine putting their face up against a stream of water and
 sticking out their tongue. In that imagined world, they can feel
 the cool water hitting their tongue, and feel the water entering
 their body, and are able to recognize that /feeling/ as drinking.
 So, the label of the action is not really in the pixels of the
-image, but is found clearly in a simulation inspired by those
+image, but is found clearly in a simulation / recollection inspired
-pixels. An imaginative system, having been trained on drinking and
+by those pixels. An imaginative system, having been trained on
-non-drinking examples and learning that the most important
+drinking and non-drinking examples and learning that the most
-component of drinking is the feeling of water sliding down one's
+important component of drinking is the feeling of water sliding
-throat, would analyze a video of a cat drinking in the following
+down one's throat, would analyze a video of a cat drinking in the
-manner:
+following manner:
 1. Create a physical model of the video by putting a ``fuzzy''
 model of its own body in place of the cat. Possibly also create
 a simulation of the stream of water.
 action. The power in this method lies in the fact that you describe
 all actions from a body-centered viewpoint. You are less tied to
 the particulars of any visual representation of the actions. If you
 teach the system what ``running'' is, and you have a good enough
 aligner, the system will from then on be able to recognize running
-from any point of view, even strange points of view like above or
+from any point of view -- even strange points of view like above or
 underneath the runner. This is in contrast to action recognition
 schemes that try to identify actions using a non-embodied approach.
 If these systems learn about running as viewed from the side, they
 will not automatically be able to recognize running from any other
 viewpoint.
 Another powerful advantage is that using the language of multiple
-body-centered rich senses to describe body-centered actions offers a
+body-centered rich senses to describe body-centered actions offers
-massive boost in descriptive capability. Consider how difficult it
+a massive boost in descriptive capability. Consider how difficult
-would be to compose a set of HOG filters to describe the action of
+it would be to compose a set of HOG (Histogram of Oriented
-a simple worm-creature ``curling'' so that its head touches its
+Gradients) filters to describe the action of a simple worm-creature
-tail, and then behold the simplicity of describing thus action in a
+``curling'' so that its head touches its tail, and then behold the
-language designed for the task (listing \ref{grand-circle-intro}):
+simplicity of describing thus action in a language designed for the
+task (listing \ref{grand-circle-intro}):
 #+caption: Body-centered actions are best expressed in a body-centered
 #+caption: language. This code detects when the worm has curled into a
 #+caption: full circle. Imagine how you would replicate this functionality
 #+caption: using low-level pixel features such as HOG filters!
 extent trigger previous experience keyed to hearing or touch.
 Segments of previous experiences gained from play are stitched
 together to form a coherent and complete sensory portrait of
 the scene.
-- Recognition      :: With the scene described in terms of
+- Recognition :: With the scene described in terms of remembered
-remembered first person sensory events, the creature can now
+first person sensory events, the creature can now run its
-run its action-identified programs (such as the one in listing
+action-definition programs (such as the one in listing
-\ref{grand-circle-intro} on this synthesized sensory data,
+\ref{grand-circle-intro}) on this synthesized sensory data,
 just as it would if it were actually experiencing the scene
 first-hand. If previous experience has been accurately
 retrieved, and if it is analogous enough to the scene, then
 the creature will correctly identify the action in the scene.
 I built =CORTEX= to be a general AI research platform for doing
 experiments involving multiple rich senses and a wide variety and
 number of creatures. I intend it to be useful as a library for many
 more projects than just this thesis. =CORTEX= was necessary to meet
 a need among AI researchers at CSAIL and beyond, which is that
-people often will invent neat ideas that are best expressed in the
+people often will invent wonderful ideas that are best expressed in
-language of creatures and senses, but in order to explore those
+the language of creatures and senses, but in order to explore those
 ideas they must first build a platform in which they can create
 simulated creatures with rich senses! There are many ideas that
-would be simple to execute (such as =EMPATH= or
+would be simple to execute (such as =EMPATH= or Larson's
-\cite{larson-symbols}), but attached to them is the multi-month
+self-organizing maps (\cite{larson-symbols})), but attached to them
-effort to make a good creature simulator. Often, that initial
+is the multi-month effort to make a good creature simulator. Often,
-investment of time proves to be too much, and the project must make
+that initial investment of time proves to be too much, and the
-do with a lesser environment.
+project must make do with a lesser environment or be abandoned
+entirely.
 =CORTEX= is well suited as an environment for embodied AI research
 for three reasons:
-- You can create new creatures using Blender (\cite{blender}), a
+- You can design new creatures using Blender (\cite{blender}), a
 popular 3D modeling program. Each sense can be specified using
 special blender nodes with biologically inspired parameters. You
 need not write any code to create a creature, and can use a wide
 library of pre-existing blender models as a base for your own
 creatures.
 - =CORTEX= implements a wide variety of senses: touch,
 proprioception, vision, hearing, and muscle tension. Complicated
 senses like touch and vision involve multiple sensory elements
 embedded in a 2D surface. You have complete control over the
 distribution of these sensor elements through the use of simple
-png image files. In particular, =CORTEX= implements more
+png image files. =CORTEX= implements more comprehensive hearing
-comprehensive hearing than any other creature simulation system
+than any other creature simulation system available.
-available.
 - =CORTEX= supports any number of creatures and any number of
 senses. Time in =CORTEX= dilates so that the simulated creatures
 always perceive a perfectly smooth flow of time, regardless of
 the actual computational load.
 over the history and implementation details presented here, is
 provided in an appendix at the end of this thesis.)
 Throughout this project, I intended for =CORTEX= to be flexible and
 extensible enough to be useful for other researchers who want to
-test out ideas of their own. To this end, wherever I have had to make
+test ideas of their own. To this end, wherever I have had to make
 architectural choices about =CORTEX=, I have chosen to give as much
 freedom to the user as possible, so that =CORTEX= may be used for
 things I have not foreseen.
 ** Building in simulation versus reality
 use a computer-simulated environment in the first place! The world
 is a vast and rich place, and for now simulations are a very poor
 reflection of its complexity. It may be that there is a significant
 qualitative difference between dealing with senses in the real
 world and dealing with pale facsimiles of them in a simulation
-\cite{brooks-representation}. What are the advantages and
+(\cite{brooks-representation}). What are the advantages and
 disadvantages of a simulation vs. reality?
 *** Simulation
 The advantages of virtual reality are that when everything is a
 simulation, experiments in that simulation are absolutely
-reproducible. It's also easier to change the character and world
+reproducible. It's also easier to change the creature and
-to explore new situations and different sensory combinations.
+environment to explore new situations and different sensory
+combinations.
 If the world is to be simulated on a computer, then not only do
-you have to worry about whether the character's senses are rich
+you have to worry about whether the creature's senses are rich
 enough to learn from the world, but whether the world itself is
 rendered with enough detail and realism to give enough working
-material to the character's senses. To name just a few
+material to the creature's senses. To name just a few
 difficulties facing modern physics simulators: destructibility of
 the environment, simulation of water/other fluids, large areas,
 nonrigid bodies, lots of objects, smoke. I don't know of any
-computer simulation that would allow a character to take a rock
+computer simulation that would allow a creature to take a rock
 and grind it into fine dust, then use that dust to make a clay
 sculpture, at least not without spending years calculating the
 interactions of every single small grain of dust. Maybe a
 simulated world with today's limitations doesn't provide enough
 richness for real intelligence to evolve.
 loose in the real world. This has the advantage of eliminating
 concerns about simulating the world at the expense of increasing
 the complexity of implementing the senses. Instead of just
 grabbing the current rendered frame for processing, you have to
 use an actual camera with real lenses and interact with photons to
-get an image. It is much harder to change the character, which is
+get an image. It is much harder to change the creature, which is
 now partly a physical robot of some sort, since doing so involves
 changing things around in the real world instead of modifying
 lines of code. While the real world is very rich and definitely
-provides enough stimulation for intelligence to develop as
+provides enough stimulation for intelligence to develop (as
-evidenced by our own existence, it is also uncontrollable in the
+evidenced by our own existence), it is also uncontrollable in the
 sense that a particular situation cannot be recreated perfectly or
-saved for later use. It is harder to conduct science because it is
+saved for later use. It is harder to conduct Science because it is
 harder to repeat an experiment. The worst thing about using the
 real world instead of a simulation is the matter of time. Instead
 of simulated time you get the constant and unstoppable flow of
 real time. This severely limits the sorts of software you can use
 to program an AI, because all sense inputs must be handled in real
 time. Complicated ideas may have to be implemented in hardware or
 may simply be impossible given the current speed of our
 processors. Contrast this with a simulation, in which the flow of
 time in the simulated world can be slowed down to accommodate the
-limitations of the character's programming. In terms of cost,
+limitations of the creature's programming. In terms of cost, doing
-doing everything in software is far cheaper than building custom
+everything in software is far cheaper than building custom
 real-time hardware. All you need is a laptop and some patience.
 ** Simulated time enables rapid prototyping \& simple programs
 I envision =CORTEX= being used to support rapid prototyping and
 The need for real time processing only increases if multiple senses
 are involved. In the extreme case, even simple algorithms will have
 to be accelerated by ASIC chips or FPGAs, turning what would
 otherwise be a few lines of code and a 10x speed penalty into a
 multi-month ordeal. For this reason, =CORTEX= supports
-/time-dilation/, which scales back the framerate of the
+/time-dilation/, which scales back the framerate of the simulation
-simulation in proportion to the amount of processing each frame.
+in proportion to the amount of processing each frame. From the
-From the perspective of the creatures inside the simulation, time
+perspective of the creatures inside the simulation, time always
-always appears to flow at a constant rate, regardless of how
+appears to flow at a constant rate, regardless of how complicated
-complicated the environment becomes or how many creatures are in
+the environment becomes or how many creatures are in the
-the simulation. The cost is that =CORTEX= can sometimes run slower
+simulation. The cost is that =CORTEX= can sometimes run slower than
-than real time. This can also be an advantage, however ---
+real time. Time dialation works both ways, however --- simulations
-simulations of very simple creatures in =CORTEX= generally run at
+of very simple creatures in =CORTEX= generally run at 40x real-time
-40x on my machine!
+on my machine!
 ** All sense organs are two-dimensional surfaces
 If =CORTEX= is to support a wide variety of senses, it would help
-to have a better understanding of what a ``sense'' actually is!
+to have a better understanding of what a sense actually is! While
-While vision, touch, and hearing all seem like they are quite
+vision, touch, and hearing all seem like they are quite different
-different things, I was surprised to learn during the course of
+things, I was surprised to learn during the course of this thesis
-this thesis that they (and all physical senses) can be expressed as
+that they (and all physical senses) can be expressed as exactly the
-exactly the same mathematical object due to a dimensional argument!
+same mathematical object!
 Human beings are three-dimensional objects, and the nerves that
 transmit data from our various sense organs to our brain are
 essentially one-dimensional. This leaves up to two dimensions in
 which our sensory information may flow. For example, imagine your
 complicated surface of the skin onto a two dimensional image.
 Most human senses consist of many discrete sensors of various
 properties distributed along a surface at various densities. For
 skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
-disks, and Ruffini's endings \cite{textbook901}, which detect
+disks, and Ruffini's endings (\cite{textbook901}), which detect
 pressure and vibration of various intensities. For ears, it is the
 stereocilia distributed along the basilar membrane inside the
 cochlea; each one is sensitive to a slightly different frequency of
 sound. For eyes, it is rods and cones distributed along the surface
 of the retina. In each case, we can describe the sense with a
 surface and a distribution of sensors along that surface.
 In fact, almost every human sense can be effectively described in
 terms of a surface containing embedded sensors. If the sense had
 any more dimensions, then there wouldn't be enough room in the
-spinal chord to transmit the information!
+spinal cord to transmit the information!
 Therefore, =CORTEX= must support the ability to create objects and
 then be able to ``paint'' points along their surfaces to describe
 each sense.
 Fortunately this idea is already a well known computer graphics
-technique called /UV-mapping/. The three-dimensional surface of a
+technique called /UV-mapping/. In UV-maping, the three-dimensional
-model is cut and smooshed until it fits on a two-dimensional
+surface of a model is cut and smooshed until it fits on a
-image. You paint whatever you want on that image, and when the
+two-dimensional image. You paint whatever you want on that image,
-three-dimensional shape is rendered in a game the smooshing and
+and when the three-dimensional shape is rendered in a game the
-cutting is reversed and the image appears on the three-dimensional
+smooshing and cutting is reversed and the image appears on the
-object.
+three-dimensional object.
 To make a sense, interpret the UV-image as describing the
 distribution of that senses sensors. To get different types of
 sensors, you can either use a different color for each type of
 sensor, or use multiple UV-maps, each labeled with that sensor
 tools that can be co-opted to serve as touch, proprioception, and
 muscles. Since some games support split screen views, a good video
 game engine will allow you to efficiently create multiple cameras
 in the simulated world that can be used as eyes. Video game systems
 offer integrated asset management for things like textures and
-creatures models, providing an avenue for defining creatures. They
+creature models, providing an avenue for defining creatures. They
 also understand UV-mapping, since this technique is used to apply a
 texture to a model. Finally, because video game engines support a
-large number of users, as long as =CORTEX= doesn't stray too far
+large number of developers, as long as =CORTEX= doesn't stray too
-from the base system, other researchers can turn to this community
+far from the base system, other researchers can turn to this
-for help when doing their research.
+community for help when doing their research.
 ** =CORTEX= is based on jMonkeyEngine3
 While preparing to build =CORTEX= I studied several video game
 engines to see which would best serve as a base. The top contenders
 were:
-- [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]]    :: The Quake II engine was designed by ID
+- [[http://www.idsoftware.com][Quake II]]/[[http://www.bytonic.de/html/jake2.html][Jake2]] :: The Quake II engine was designed by ID software
-software in 1997.  All the source code was released by ID
+in 1997. All the source code was released by ID software into
-software into the Public Domain several years ago, and as a
+the Public Domain several years ago, and as a result it has
-result it has been ported to many different languages. This
+been ported to many different languages. This engine was
-engine was famous for its advanced use of realistic shading
+famous for its advanced use of realistic shading and it had
-and had decent and fast physics simulation. The main advantage
+decent and fast physics simulation. The main advantage of the
-of the Quake II engine is its simplicity, but I ultimately
+Quake II engine is its simplicity, but I ultimately rejected
-rejected it because the engine is too tied to the concept of a
+it because the engine is too tied to the concept of a
 first-person shooter game. One of the problems I had was that
 there does not seem to be any easy way to attach multiple
 cameras to a single character. There are also several physics
 clipping issues that are corrected in a way that only applies
 to the main character and do not apply to arbitrary objects.
 creatures. If possible, it would be nice to leverage work that has
 already been done by the community of 3D modelers, or at least
 enable people who are talented at modeling but not programming to
 design =CORTEX= creatures.
-Therefore, I use Blender, a free 3D modeling program, as the main
+Therefore I use Blender, a free 3D modeling program, as the main
 way to create creatures in =CORTEX=. However, the creatures modeled
 in Blender must also be simple to simulate in jMonkeyEngine3's game
 engine, and must also be easy to rig with =CORTEX='s senses. I
 accomplish this with extensive use of Blender's ``empty nodes.''
 Empty nodes have no mass, physical presence, or appearance, but
 they can hold metadata and have names. I use a tree structure of
 empty nodes to specify senses in the following manner:
 ** Bodies are composed of segments connected by joints
 Blender is a general purpose animation tool, which has been used in
 the past to create high quality movies such as Sintel
-\cite{blender}. Though Blender can model and render even complicated
+(\cite{blender}). Though Blender can model and render even
-things like water, it is crucial to keep models that are meant to
+complicated things like water, it is crucial to keep models that
-be simulated as creatures simple. =Bullet=, which =CORTEX= uses
+are meant to be simulated as creatures simple. =Bullet=, which
-though jMonkeyEngine3, is a rigid-body physics system. This offers
+=CORTEX= uses though jMonkeyEngine3, is a rigid-body physics
-a compromise between the expressiveness of a game level and the
+system. This offers a compromise between the expressiveness of a
-speed at which it can be simulated, and it means that creatures
+game level and the speed at which it can be simulated, and it means
-should be naturally expressed as rigid components held together by
+that creatures should be naturally expressed as rigid components
-joint constraints.
+held together by joint constraints.
 But humans are more like a squishy bag wrapped around some hard
 bones which define the overall shape. When we move, our skin bends
 and stretches to accommodate the new positions of our bones.
 it about the true extent of its body. Simulating the skin as a
 physical object requires some way to continuously update the
 physical model of the skin along with the movement of the bones,
 which is unacceptably slow compared to rigid body simulation.
-Therefore, instead of using the human-like ``deformable bag of
+Therefore, instead of using the human-like ``bony meatbag''
-bones'' approach, I decided to base my body plans on multiple solid
+approach, I decided to base my body plans on multiple solid objects
-objects that are connected by joints, inspired by the robot =EVE=
+that are connected by joints, inspired by the robot =EVE= from the
-from the movie WALL-E.
+movie WALL-E.
 #+caption: =EVE= from the movie WALL-E.  This body plan turns
 #+caption: out to be much better suited to my purposes than a more
 #+caption: human-like one.
 #+ATTR_LaTeX: :width 10cm
 [[./images/Eve.jpg]]
 =EVE='s body is composed of several rigid components that are held
 together by invisible joint constraints. This is what I mean by
-``eve-like''. The main reason that I use eve-style bodies is for
+/eve-like/. The main reason that I use eve-like bodies is for
-efficiency, and so that there will be correspondence between the
+simulation efficiency, and so that there will be correspondence
-AI's senses and the physical presence of its body. Each individual
+between the AI's senses and the physical presence of its body. Each
-section is simulated by a separate rigid body that corresponds
+individual section is simulated by a separate rigid body that
-exactly with its visual representation and does not change.
+corresponds exactly with its visual representation and does not
-Sections are connected by invisible joints that are well supported
+change. Sections are connected by invisible joints that are well
-in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
+supported in jMonkeyEngine3. Bullet, the physics backend for
-can efficiently simulate hundreds of rigid bodies connected by
+jMonkeyEngine3, can efficiently simulate hundreds of rigid bodies
-joints. Just because sections are rigid does not mean they have to
+connected by joints. Just because sections are rigid does not mean
-stay as one piece forever; they can be dynamically replaced with
+they have to stay as one piece forever; they can be dynamically
-multiple sections to simulate splitting in two. This could be used
+replaced with multiple sections to simulate splitting in two. This
-to simulate retractable claws or =EVE='s hands, which are able to
+could be used to simulate retractable claws or =EVE='s hands, which
-coalesce into one object in the movie.
+are able to coalesce into one object in the movie.
 *** Solidifying/Connecting a body
 =CORTEX= creates a creature in two steps: first, it traverses the
 nodes in the blender file and creates physical representations for
 - Empathy         :: my empathy program leaves many areas for
 improvement, among which are using vision to infer
 proprioception and looking up sensory experience with imagined
 vision, touch, and sound.
-- Evolution       :: Karl Sims created a rich environment for
+- Evolution :: Karl Sims created a rich environment for simulating
-simulating the evolution of creatures on a connection
+the evolution of creatures on a Connection Machine
-machine. Today, this can be redone and expanded with =CORTEX=
+(\cite{sims-evolving-creatures}). Today, this can be redone
-on an ordinary computer.
+and expanded with =CORTEX= on an ordinary computer.
 - Exotic senses  :: Cortex enables many fascinating senses that are
 not possible to build in the real world. For example,
 telekinesis is an interesting avenue to explore. You can also
 make a ``semantic'' sense which looks up metadata tags on
 objects in the environment the metadata tags might contain
 other sensory information.
 - Imagination via subworlds :: this would involve a creature with
 an effector which creates an entire new sub-simulation where
 the creature has direct control over placement/creation of
 objects via simulated telekinesis. The creature observes this
-sub-world through it's normal senses and uses its observations
+sub-world through its normal senses and uses its observations
 to make predictions about its top level world.
 - Simulated prescience :: step the simulation forward a few ticks,
 gather sensory data, then supply this data for the creature as
 one of its actual senses. The cost of prescience is slowing
 the simulation down by a factor proportional to however far
 fight each other?
 - Swarm creatures :: Program a group of creatures that cooperate
 with each other. Because the creatures would be simulated, you
 could investigate computationally complex rules of behavior
 which still, from the group's point of view, would happen in
-``real time''. Interactions could be as simple as cellular
+real time. Interactions could be as simple as cellular
 organisms communicating via flashing lights, or as complex as
 humanoids completing social tasks, etc.
-- =HACKER= for writing muscle-control programs :: Presented with
+- =HACKER= for writing muscle-control programs :: Presented with a
-low-level muscle control/ sense API, generate higher level
+low-level muscle control / sense API, generate higher level
 programs for accomplishing various stated goals. Example goals
 might be "extend all your fingers" or "move your hand into the
 area with blue light" or "decrease the angle of this joint".
 It would be like Sussman's HACKER, except it would operate
 with much more data in a more realistic world. Start off with
 "calisthenics" to develop subroutines over the motor control
-API. This would be the "spinal chord" of a more intelligent
+API. The low level programming code might be a turning machine
-creature. The low level programming code might be a turning
+that could develop programs to iterate over a "tape" where
-machine that could develop programs to iterate over a "tape"
+each entry in the tape could control recruitment of the fibers
-where each entry in the tape could control recruitment of the
+in a muscle.
-fibers in a muscle.
+- Sense fusion :: There is much work to be done on sense
-- Sense fusion    :: There is much work to be done on sense
 integration -- building up a coherent picture of the world and
-the things in it with =CORTEX= as a base, you can explore
+the things in it. With =CORTEX= as a base, you can explore
 concepts like self-organizing maps or cross modal clustering
 in ways that have never before been tried.
 - Inverse kinematics :: experiments in sense guided motor control
 are easy given =CORTEX='s support -- you can get right to the
 hard control problems without worrying about physics or
 have terms that consider the color of a person's skin or whether
 they are male or female, instead it gets right to the meat of what
 jumping actually /is/.
 Of course, the action predicates are not directly applicable to
-video data which lacks the advanced sensory information which they
+video data, which lacks the advanced sensory information which they
 require!
 The trick now is to make the action predicates work even when the
 sensory data on which they depend is absent. If I can do that, then
 I will have gained much.
 #+BEGIN_EXAMPLE
 [ flat, flat, flat, flat, flat, flat, lift-head ]
 #+END_EXAMPLE
 The worm's previous experience of lying on the ground and lifting
-its head generates possible interpretations for each frame:
+its head generates possible interpretations for each frame (the
+numbers are experience-indices):
 #+BEGIN_EXAMPLE
 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
 1     1     1     1     1     1     1     4
 2     2     2     2     2     2     2
 [ flat, flat, flat, flat, flat, flat, flat, lift-head ]
 6     7     8     9     1     2     3     4
 #+END_EXAMPLE
 The new path through \Phi-space is synthesized from two actual
-paths that the creature actually experiences, the "1-2-3-4" chain
+paths that the creature has experienced: the "1-2-3-4" chain and
-and the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because
+the "6-7-8-9" chain. The "1-2-3-4" chain is necessary because it
-it ends with the worm lifting its head. It originated from a short
+ends with the worm lifting its head. It originated from a short
 training session where the worm rested on the floor for a brief
 while and then raised its head. The "6-7-8-9" chain is part of a
 longer chain of inactivity where the worm simply rested on the
 floor without moving. It is preferred over a "1-2-3" chain (which
 also describes inactivity) because it is longer. The main ideas
 - =(display-dilated-time world timer)= :: Shows the time as it is
 flowing in the simulation on a HUD display.
+TODO -- add a paper about detecting biological motion from only a few dots.

Mercurial > cortex

comparison thesis/cortex.org @ 547:5d89879fc894