cortex: thesis/cortex.org comparison

comparison thesis/cortex.org @ 468:258078f78b33

insert argument about senses.

author	Robert McIntyre <rlm@mit.edu>
date	Fri, 28 Mar 2014 16:25:31 -0400
parents	ade64947d2bf
children	ae10f35022ba

comparison

equal deleted inserted replaced

-:ade64947d2bf
+:258078f78b33
 The most important archetictural decision of all is the choice to
 use a computer-simulated environemnt in the first place! The world
 is a vast and rich place, and for now simulations are a very poor
 reflection of its complexity. It may be that there is a significant
 qualatative difference between dealing with senses in the real
-world and dealing with pale facilimilies of them in a
+world and dealing with pale facilimilies of them in a simulation.
-simulation. What are the advantages and disadvantages of a
+What are the advantages and disadvantages of a simulation vs.
-simulation vs. reality?
+reality?
 *** Simulation
 The advantages of virtual reality are that when everything is a
 simulation, experiments in that simulation are absolutely
 the simulation. The cost is that =CORTEX= can sometimes run slower
 than real time. This can also be an advantage, however ---
 simulations of very simple creatures in =CORTEX= generally run at
 40x on my machine!
+** What is a sense?
+If =CORTEX= is to support a wide variety of senses, it would help
+to have a better understanding of what a ``sense'' actually is!
+While vision, touch, and hearing all seem like they are quite
+different things, I was supprised to learn during the course of
+this thesis that they (and all physical senses) can be expressed as
+exactly the same mathematical object due to a dimensional argument!
+Human beings are three-dimensional objects, and the nerves that
+transmit data from our various sense organs to our brain are
+essentially one-dimensional. This leaves up to two dimensions in
+which our sensory information may flow. For example, imagine your
+skin: it is a two-dimensional surface around a three-dimensional
+object (your body). It has discrete touch sensors embedded at
+various points, and the density of these sensors corresponds to the
+sensitivity of that region of skin. Each touch sensor connects to a
+nerve, all of which eventually are bundled together as they travel
+up the spinal cord to the brain. Intersect the spinal nerves with a
+guillotining plane and you will see all of the sensory data of the
+skin revealed in a roughly circular two-dimensional image which is
+the cross section of the spinal cord. Points on this image that are
+close together in this circle represent touch sensors that are
+/probably/ close together on the skin, although there is of course
+some cutting and rearrangement that has to be done to transfer the
+complicated surface of the skin onto a two dimensional image.
+Most human senses consist of many discrete sensors of various
+properties distributed along a surface at various densities. For
+skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
+disks, and Ruffini's endings, which detect pressure and vibration
+of various intensities. For ears, it is the stereocilia distributed
+along the basilar membrane inside the cochlea; each one is
+sensitive to a slightly different frequency of sound. For eyes, it
+is rods and cones distributed along the surface of the retina. In
+each case, we can describe the sense with a surface and a
+distribution of sensors along that surface.
+The neat idea is that every human sense can be effectively
+described in terms of a surface containing embedded sensors. If the
+sense had any more dimensions, then there wouldn't be enough room
+in the spinal chord to transmit the information!
+Therefore, =CORTEX= must support the ability to create objects and
+then be able to ``paint'' points along their surfaces to describe
+each sense.
+Fortunately this idea is already a well known computer graphics
+technique called called /UV-mapping/. The three-dimensional surface
+of a model is cut and smooshed until it fits on a two-dimensional
+image. You paint whatever you want on that image, and when the
+three-dimensional shape is rendered in a game the smooshing and
+cutting is reversed and the image appears on the three-dimensional
+object.
+To make a sense, interpret the UV-image as describing the
+distribution of that senses sensors. To get different types of
+sensors, you can either use a different color for each type of
+sensor, or use multiple UV-maps, each labeled with that sensor
+type. I generally use a white pixel to mean the presence of a
+sensor and a black pixel to mean the absence of a sensor, and use
+one UV-map for each sensor-type within a given sense.
+#+CAPTION: The UV-map for an elongated icososphere. The white
+#+caption: dots each represent a touch sensor. They are dense
+#+caption: in the regions that describe the tip of the finger,
+#+caption: and less dense along the dorsal side of the finger
+#+caption: opposite the tip.
+#+name: finger-UV
+#+ATTR_latex: :width 10cm
+[[./images/finger-UV.png]]
+#+caption: Ventral side of the UV-mapped finger. Notice the
+#+caption: density of touch sensors at the tip.
+#+name: finger-side-view
+#+ATTR_LaTeX: :width 10cm
+[[./images/finger-1.png]]
 ** COMMENT Video game engines are a great starting point
 I did not need to write my own physics simulation code or shader to
 build =CORTEX=. Doing so would lead to a system that is impossible
 for anyone but myself to use anyway. Instead, I use a video game
 tools that can be co-opted to serve as touch, proprioception, and
 muscles. Since some games support split screen views, a good video
 game engine will allow you to efficiently create multiple cameras
 in the simulated world that can be used as eyes. Video game systems
 offer integrated asset management for things like textures and
-creatures models, providing an avenue for defining creatures.
+creatures models, providing an avenue for defining creatures. They
-Finally, because video game engines support a large number of
+also understand UV-mapping, since this technique is used to apply a
-users, if I don't stray too far from the base system, other
+texture to a model. Finally, because video game engines support a
-researchers can turn to this community for help when doing their
+large number of users, as long as =CORTEX= doesn't stray too far
-research.
+from the base system, other researchers can turn to this community
+for help when doing their research.
 ** COMMENT =CORTEX= is based on jMonkeyEngine3
 While preparing to build =CORTEX= I studied several video game
 engines to see which would best serve as a base. The top contenders
 I chose jMonkeyEngine3 because it because it had the most features
 out of all the free projects I looked at, and because I could then
 write my code in clojure, an implementation of =LISP= that runs on
 the JVM.
-** COMMENT Bodies are composed of segments connected by joints
+** =CORTEX= uses Blender to create creature models
 For the simple worm-like creatures I will use later on in this
 thesis, I could define a simple API in =CORTEX= that would allow
 one to create boxes, spheres, etc., and leave that API as the sole
 way to create creatures. However, for =CORTEX= to truly be useful
-for other projects, it needs to have a way to construct complicated
+for other projects, it needs a way to construct complicated
 creatures. If possible, it would be nice to leverage work that has
 already been done by the community of 3D modelers, or at least
 enable people who are talented at moedling but not programming to
 design =CORTEX= creatures.
 Therefore, I use Blender, a free 3D modeling program, as the main
 way to create creatures in =CORTEX=. However, the creatures modeled
 in Blender must also be simple to simulate in jMonkeyEngine3's game
-engine, and must also be easy to rig with =CORTEX='s senses.
+engine, and must also be easy to rig with =CORTEX='s senses. I
+accomplish this with extensive use of Blender's ``empty nodes.''
-While trying to find a good compromise for body-design, one option
-I ultimately rejected is to use blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea
+Empty nodes have no mass, physical presence, or appearance, but
-would have been to define a mesh which describes the creature's
+they can hold metadata and have names. I use a tree structure of
-entire body. To this you add a skeleton which deforms this mesh
+empty nodes to specify senses in the following manner:
-(called rigging). This technique is used extensively to model
-humans and create realistic animations. It is not a good technique
+- Create a single top-level empty node whose name is the name of
-for physical simulation, because deformable surfaces are hard to
+the sense.
-model. Humans work like a squishy bag with some hard bones to give
+- Add empty nodes which each contain meta-data relevant to the
-it shape. The bones are easy to simulate physically, but they
+sense, including a UV-map describing the number/distribution of
-interact with thr world though the skin, which is contiguous, but
+sensors if applicable.
-does not have a constant shape. In order to simulate skin you need
+- Make each empty-node the child of the top-level node.
-some way to continuously update the physical model of the skin
-along with the movement of the bones. Given that bullet is
+#+caption: An example of annoting a creature model with empty
-optimized for rigid, solid objects, this leads to unmanagable
+#+caption: nodes to describe the layout of senses. There are
-computation and incorrect simulation.
+#+caption: multiple empty nodes which each describe the position
+#+caption: of muscles, ears, eyes, or joints.
-Instead of using the human-like ``deformable bag of bones''
+#+name: sense-nodes
-approach, I decided to base my body plans on multiple solid objects
+#+ATTR_LaTeX: :width 10cm
-that are connected by joints, inspired by the robot =EVE= from the
+[[./images/empty-sense-nodes.png]]
-movie WALL-E.
+** Bodies are composed of segments connected by joints
+Blender is a general purpose animation tool, which has been used in
+the past to create high quality movies such as Sintel
+\cite{sintel}. Though Blender can model and render even complicated
+things like water, it is crucual to keep models that are meant to
+be simulated as creatures simple. =Bullet=, which =CORTEX= uses
+though jMonkeyEngine3, is a rigid-body physics system. This offers
+a compromise between the expressiveness of a game level and the
+speed at which it can be simulated, and it means that creatures
+should be naturally expressed as rigid components held together by
+joint constraints.
+But humans are more like a squishy bag with wrapped around some
+hard bones which define the overall shape. When we move, our skin
+bends and stretches to accomodate the new positions of our bones.
+One way to make bodies composed of rigid pieces connected by joints
+/seem/ more human-like is to use an /armature/, (or /rigging/)
+system, which defines a overall ``body mesh'' and defines how the
+mesh deforms as a function of the position of each ``bone'' which
+is a standard rigid body. This technique is used extensively to
+model humans and create realistic animations. It is not a good
+technique for physical simulation, however because it creates a lie
+-- the skin is not a physical part of the simulation and does not
+interact with any objects in the world or itself. Objects will pass
+right though the skin until they come in contact with the
+underlying bone, which is a physical object. Whithout simulating
+the skin, the sense of touch has little meaning, and the creature's
+own vision will lie to it about the true extent of its body.
+Simulating the skin as a physical object requires some way to
+continuously update the physical model of the skin along with the
+movement of the bones, which is unacceptably slow compared to rigid
+body simulation.
+Therefore, instead of using the human-like ``deformable bag of
+bones'' approach, I decided to base my body plans on multiple solid
+objects that are connected by joints, inspired by the robot =EVE=
+from the movie WALL-E.
 #+caption: =EVE= from the movie WALL-E.  This body plan turns
 #+caption: out to be much better suited to my purposes than a more
 #+caption: human-like one.
 #+ATTR_LaTeX: :width 10cm
 =EVE='s body is composed of several rigid components that are held
 together by invisible joint constraints. This is what I mean by
 ``eve-like''. The main reason that I use eve-style bodies is for
 efficiency, and so that there will be correspondence between the
-AI's vision and the physical presence of its body. Each individual
+AI's semses and the physical presence of its body. Each individual
 section is simulated by a separate rigid body that corresponds
 exactly with its visual representation and does not change.
 Sections are connected by invisible joints that are well supported
 in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
 can efficiently simulate hundreds of rigid bodies connected by
-joints. Sections do not have to stay as one piece forever; they can
+joints. Just because sections are rigid does not mean they have to
-be dynamically replaced with multiple sections to simulate
+stay as one piece forever; they can be dynamically replaced with
-splitting in two. This could be used to simulate retractable claws
+multiple sections to simulate splitting in two. This could be used
-or =EVE='s hands, which are able to coalesce into one object in the
+to simulate retractable claws or =EVE='s hands, which are able to
-movie.
+coalesce into one object in the movie.
 *** Solidifying/Connecting the body
-Importing bodies from =CORTEX= into blender involves encoding
-metadata into the blender file that specifies the mass of each
-component and the joints by which those components are connected. I
-do this in Blender in two ways. First is by using the ``metadata''
-field of each solid object to specify the mass. Second is by using
-Blender ``empty nodes'' to specify the position and type of each
-joint. Empty nodes have no mass, physical presence, or appearance,
-but they can hold metadata and have names. I use a tree structure
-of empty nodes to specify joints. There is a parent node named
-``joints'', and a series of empty child nodes of the ``joints''
-node that each represent a single joint.
 #+caption: View of the hand model in Blender showing the main ``joints''
 #+caption: node (highlighted in yellow) and its children which each
 #+caption: represent a joint in the hand. Each joint node has metadata
 #+caption: specifying what sort of joint it is.
 #+name: blender-hand
 #+ATTR_LaTeX: :width 10cm
 [[./images/hand-screenshot1.png]]
 =CORTEX= creates a creature in two steps: first, it traverses the
 nodes in the blender file and creates physical representations for
 any of them that have mass defined.
 #+caption: =CORTEX= gets one step closer to a full creature simulation
 #+caption: environment.
 #+name: name
 #+ATTR_LaTeX: :width 15cm
 [[./images/physical-hand.png]]
 ** Eyes reuse standard video game components
 ** Hearing is hard; =CORTEX= does it right

Mercurial > cortex

comparison thesis/cortex.org @ 468:258078f78b33