1.1 --- a/thesis/cortex.org	Fri Mar 28 15:30:23 2014 -0400
     1.2 +++ b/thesis/cortex.org	Fri Mar 28 16:25:31 2014 -0400
     1.3 @@ -366,9 +366,9 @@
     1.4     is a vast and rich place, and for now simulations are a very poor
     1.5     reflection of its complexity. It may be that there is a significant
     1.6     qualatative difference between dealing with senses in the real
     1.7 -   world and dealing with pale facilimilies of them in a
     1.8 -   simulation. What are the advantages and disadvantages of a
     1.9 -   simulation vs. reality?
    1.10 +   world and dealing with pale facilimilies of them in a simulation.
    1.11 +   What are the advantages and disadvantages of a simulation vs.
    1.12 +   reality?
    1.13  
    1.14  *** Simulation
    1.15  
    1.16 @@ -445,6 +445,85 @@
    1.17     simulations of very simple creatures in =CORTEX= generally run at
    1.18     40x on my machine!
    1.19  
    1.20 +** What is a sense?
    1.21 +   
    1.22 +   If =CORTEX= is to support a wide variety of senses, it would help
    1.23 +   to have a better understanding of what a ``sense'' actually is!
    1.24 +   While vision, touch, and hearing all seem like they are quite
    1.25 +   different things, I was supprised to learn during the course of
    1.26 +   this thesis that they (and all physical senses) can be expressed as
    1.27 +   exactly the same mathematical object due to a dimensional argument!
    1.28 +
    1.29 +   Human beings are three-dimensional objects, and the nerves that
    1.30 +   transmit data from our various sense organs to our brain are
    1.31 +   essentially one-dimensional. This leaves up to two dimensions in
    1.32 +   which our sensory information may flow. For example, imagine your
    1.33 +   skin: it is a two-dimensional surface around a three-dimensional
    1.34 +   object (your body). It has discrete touch sensors embedded at
    1.35 +   various points, and the density of these sensors corresponds to the
    1.36 +   sensitivity of that region of skin. Each touch sensor connects to a
    1.37 +   nerve, all of which eventually are bundled together as they travel
    1.38 +   up the spinal cord to the brain. Intersect the spinal nerves with a
    1.39 +   guillotining plane and you will see all of the sensory data of the
    1.40 +   skin revealed in a roughly circular two-dimensional image which is
    1.41 +   the cross section of the spinal cord. Points on this image that are
    1.42 +   close together in this circle represent touch sensors that are
    1.43 +   /probably/ close together on the skin, although there is of course
    1.44 +   some cutting and rearrangement that has to be done to transfer the
    1.45 +   complicated surface of the skin onto a two dimensional image.
    1.46 +
    1.47 +   Most human senses consist of many discrete sensors of various
    1.48 +   properties distributed along a surface at various densities. For
    1.49 +   skin, it is Pacinian corpuscles, Meissner's corpuscles, Merkel's
    1.50 +   disks, and Ruffini's endings, which detect pressure and vibration
    1.51 +   of various intensities. For ears, it is the stereocilia distributed
    1.52 +   along the basilar membrane inside the cochlea; each one is
    1.53 +   sensitive to a slightly different frequency of sound. For eyes, it
    1.54 +   is rods and cones distributed along the surface of the retina. In
    1.55 +   each case, we can describe the sense with a surface and a
    1.56 +   distribution of sensors along that surface.
    1.57 +
    1.58 +   The neat idea is that every human sense can be effectively
    1.59 +   described in terms of a surface containing embedded sensors. If the
    1.60 +   sense had any more dimensions, then there wouldn't be enough room
    1.61 +   in the spinal chord to transmit the information!
    1.62 +
    1.63 +   Therefore, =CORTEX= must support the ability to create objects and
    1.64 +   then be able to ``paint'' points along their surfaces to describe
    1.65 +   each sense. 
    1.66 +
    1.67 +   Fortunately this idea is already a well known computer graphics
    1.68 +   technique called called /UV-mapping/. The three-dimensional surface
    1.69 +   of a model is cut and smooshed until it fits on a two-dimensional
    1.70 +   image. You paint whatever you want on that image, and when the
    1.71 +   three-dimensional shape is rendered in a game the smooshing and
    1.72 +   cutting is reversed and the image appears on the three-dimensional
    1.73 +   object.
    1.74 +
    1.75 +   To make a sense, interpret the UV-image as describing the
    1.76 +   distribution of that senses sensors. To get different types of
    1.77 +   sensors, you can either use a different color for each type of
    1.78 +   sensor, or use multiple UV-maps, each labeled with that sensor
    1.79 +   type. I generally use a white pixel to mean the presence of a
    1.80 +   sensor and a black pixel to mean the absence of a sensor, and use
    1.81 +   one UV-map for each sensor-type within a given sense. 
    1.82 +
    1.83 +   #+CAPTION: The UV-map for an elongated icososphere. The white
    1.84 +   #+caption: dots each represent a touch sensor. They are dense 
    1.85 +   #+caption: in the regions that describe the tip of the finger, 
    1.86 +   #+caption: and less dense along the dorsal side of the finger 
    1.87 +   #+caption: opposite the tip.
    1.88 +   #+name: finger-UV
    1.89 +   #+ATTR_latex: :width 10cm
    1.90 +   [[./images/finger-UV.png]]
    1.91 +
    1.92 +   #+caption: Ventral side of the UV-mapped finger. Notice the 
    1.93 +   #+caption: density of touch sensors at the tip.
    1.94 +   #+name: finger-side-view
    1.95 +   #+ATTR_LaTeX: :width 10cm
    1.96 +   [[./images/finger-1.png]]
    1.97 +
    1.98 +
    1.99  ** COMMENT Video game engines are a great starting point
   1.100     
   1.101     I did not need to write my own physics simulation code or shader to
   1.102 @@ -462,11 +541,12 @@
   1.103     game engine will allow you to efficiently create multiple cameras
   1.104     in the simulated world that can be used as eyes. Video game systems
   1.105     offer integrated asset management for things like textures and
   1.106 -   creatures models, providing an avenue for defining creatures.
   1.107 -   Finally, because video game engines support a large number of
   1.108 -   users, if I don't stray too far from the base system, other
   1.109 -   researchers can turn to this community for help when doing their
   1.110 -   research. 
   1.111 +   creatures models, providing an avenue for defining creatures. They
   1.112 +   also understand UV-mapping, since this technique is used to apply a
   1.113 +   texture to a model. Finally, because video game engines support a
   1.114 +   large number of users, as long as =CORTEX= doesn't stray too far
   1.115 +   from the base system, other researchers can turn to this community
   1.116 +   for help when doing their research.
   1.117     
   1.118  ** COMMENT =CORTEX= is based on jMonkeyEngine3
   1.119  
   1.120 @@ -510,43 +590,83 @@
   1.121     write my code in clojure, an implementation of =LISP= that runs on
   1.122     the JVM.
   1.123  
   1.124 -** COMMENT Bodies are composed of segments connected by joints
   1.125 +** =CORTEX= uses Blender to create creature models
   1.126  
   1.127     For the simple worm-like creatures I will use later on in this
   1.128     thesis, I could define a simple API in =CORTEX= that would allow
   1.129     one to create boxes, spheres, etc., and leave that API as the sole
   1.130     way to create creatures. However, for =CORTEX= to truly be useful
   1.131 -   for other projects, it needs to have a way to construct complicated
   1.132 +   for other projects, it needs a way to construct complicated
   1.133     creatures. If possible, it would be nice to leverage work that has
   1.134     already been done by the community of 3D modelers, or at least
   1.135     enable people who are talented at moedling but not programming to
   1.136 -   design =CORTEX= creatures. 
   1.137 +   design =CORTEX= creatures.
   1.138  
   1.139     Therefore, I use Blender, a free 3D modeling program, as the main
   1.140     way to create creatures in =CORTEX=. However, the creatures modeled
   1.141     in Blender must also be simple to simulate in jMonkeyEngine3's game
   1.142 -   engine, and must also be easy to rig with =CORTEX='s senses. 
   1.143 -   
   1.144 -   While trying to find a good compromise for body-design, one option
   1.145 -   I ultimately rejected is to use blender's [[http://wiki.blender.org/index.php/Doc:2.6/Manual/Rigging/Armatures][armature]] system. The idea
   1.146 -   would have been to define a mesh which describes the creature's
   1.147 -   entire body. To this you add a skeleton which deforms this mesh
   1.148 -   (called rigging). This technique is used extensively to model
   1.149 -   humans and create realistic animations. It is not a good technique
   1.150 -   for physical simulation, because deformable surfaces are hard to
   1.151 -   model. Humans work like a squishy bag with some hard bones to give
   1.152 -   it shape. The bones are easy to simulate physically, but they
   1.153 -   interact with thr world though the skin, which is contiguous, but
   1.154 -   does not have a constant shape. In order to simulate skin you need
   1.155 -   some way to continuously update the physical model of the skin
   1.156 -   along with the movement of the bones. Given that bullet is
   1.157 -   optimized for rigid, solid objects, this leads to unmanagable
   1.158 -   computation and incorrect simulation. 
   1.159 +   engine, and must also be easy to rig with =CORTEX='s senses. I
   1.160 +   accomplish this with extensive use of Blender's ``empty nodes.'' 
   1.161  
   1.162 -   Instead of using the human-like ``deformable bag of bones''
   1.163 -   approach, I decided to base my body plans on multiple solid objects
   1.164 -   that are connected by joints, inspired by the robot =EVE= from the
   1.165 -   movie WALL-E.
   1.166 +   Empty nodes have no mass, physical presence, or appearance, but
   1.167 +   they can hold metadata and have names. I use a tree structure of
   1.168 +   empty nodes to specify senses in the following manner:
   1.169 +
   1.170 +   - Create a single top-level empty node whose name is the name of
   1.171 +     the sense.
   1.172 +   - Add empty nodes which each contain meta-data relevant to the
   1.173 +     sense, including a UV-map describing the number/distribution of
   1.174 +     sensors if applicable.
   1.175 +   - Make each empty-node the child of the top-level node.
   1.176 +     
   1.177 +   #+caption: An example of annoting a creature model with empty
   1.178 +   #+caption: nodes to describe the layout of senses. There are 
   1.179 +   #+caption: multiple empty nodes which each describe the position
   1.180 +   #+caption: of muscles, ears, eyes, or joints.
   1.181 +   #+name: sense-nodes
   1.182 +   #+ATTR_LaTeX: :width 10cm
   1.183 +   [[./images/empty-sense-nodes.png]]
   1.184 +
   1.185 +
   1.186 +** Bodies are composed of segments connected by joints
   1.187 +
   1.188 +   Blender is a general purpose animation tool, which has been used in
   1.189 +   the past to create high quality movies such as Sintel
   1.190 +   \cite{sintel}. Though Blender can model and render even complicated
   1.191 +   things like water, it is crucual to keep models that are meant to
   1.192 +   be simulated as creatures simple. =Bullet=, which =CORTEX= uses
   1.193 +   though jMonkeyEngine3, is a rigid-body physics system. This offers
   1.194 +   a compromise between the expressiveness of a game level and the
   1.195 +   speed at which it can be simulated, and it means that creatures
   1.196 +   should be naturally expressed as rigid components held together by
   1.197 +   joint constraints.
   1.198 +
   1.199 +   But humans are more like a squishy bag with wrapped around some
   1.200 +   hard bones which define the overall shape. When we move, our skin
   1.201 +   bends and stretches to accomodate the new positions of our bones. 
   1.202 +
   1.203 +   One way to make bodies composed of rigid pieces connected by joints
   1.204 +   /seem/ more human-like is to use an /armature/, (or /rigging/)
   1.205 +   system, which defines a overall ``body mesh'' and defines how the
   1.206 +   mesh deforms as a function of the position of each ``bone'' which
   1.207 +   is a standard rigid body. This technique is used extensively to
   1.208 +   model humans and create realistic animations. It is not a good
   1.209 +   technique for physical simulation, however because it creates a lie
   1.210 +   -- the skin is not a physical part of the simulation and does not
   1.211 +   interact with any objects in the world or itself. Objects will pass
   1.212 +   right though the skin until they come in contact with the
   1.213 +   underlying bone, which is a physical object. Whithout simulating
   1.214 +   the skin, the sense of touch has little meaning, and the creature's
   1.215 +   own vision will lie to it about the true extent of its body.
   1.216 +   Simulating the skin as a physical object requires some way to
   1.217 +   continuously update the physical model of the skin along with the
   1.218 +   movement of the bones, which is unacceptably slow compared to rigid
   1.219 +   body simulation. 
   1.220 +
   1.221 +   Therefore, instead of using the human-like ``deformable bag of
   1.222 +   bones'' approach, I decided to base my body plans on multiple solid
   1.223 +   objects that are connected by joints, inspired by the robot =EVE=
   1.224 +   from the movie WALL-E.
   1.225     
   1.226     #+caption: =EVE= from the movie WALL-E.  This body plan turns 
   1.227     #+caption: out to be much better suited to my purposes than a more 
   1.228 @@ -558,32 +678,20 @@
   1.229     together by invisible joint constraints. This is what I mean by
   1.230     ``eve-like''. The main reason that I use eve-style bodies is for
   1.231     efficiency, and so that there will be correspondence between the
   1.232 -   AI's vision and the physical presence of its body. Each individual
   1.233 +   AI's semses and the physical presence of its body. Each individual
   1.234     section is simulated by a separate rigid body that corresponds
   1.235     exactly with its visual representation and does not change.
   1.236     Sections are connected by invisible joints that are well supported
   1.237     in jMonkeyEngine3. Bullet, the physics backend for jMonkeyEngine3,
   1.238     can efficiently simulate hundreds of rigid bodies connected by
   1.239 -   joints. Sections do not have to stay as one piece forever; they can
   1.240 -   be dynamically replaced with multiple sections to simulate
   1.241 -   splitting in two. This could be used to simulate retractable claws
   1.242 -   or =EVE='s hands, which are able to coalesce into one object in the
   1.243 -   movie.
   1.244 +   joints. Just because sections are rigid does not mean they have to
   1.245 +   stay as one piece forever; they can be dynamically replaced with
   1.246 +   multiple sections to simulate splitting in two. This could be used
   1.247 +   to simulate retractable claws or =EVE='s hands, which are able to
   1.248 +   coalesce into one object in the movie.
   1.249  
   1.250  *** Solidifying/Connecting the body   
   1.251  
   1.252 -   Importing bodies from =CORTEX= into blender involves encoding
   1.253 -   metadata into the blender file that specifies the mass of each
   1.254 -   component and the joints by which those components are connected. I
   1.255 -   do this in Blender in two ways. First is by using the ``metadata''
   1.256 -   field of each solid object to specify the mass. Second is by using
   1.257 -   Blender ``empty nodes'' to specify the position and type of each
   1.258 -   joint. Empty nodes have no mass, physical presence, or appearance,
   1.259 -   but they can hold metadata and have names. I use a tree structure
   1.260 -   of empty nodes to specify joints. There is a parent node named
   1.261 -   ``joints'', and a series of empty child nodes of the ``joints''
   1.262 -   node that each represent a single joint.
   1.263 -
   1.264     #+caption: View of the hand model in Blender showing the main ``joints''
   1.265     #+caption: node (highlighted in yellow) and its children which each
   1.266     #+caption: represent a joint in the hand. Each joint node has metadata
   1.267 @@ -591,7 +699,6 @@
   1.268     #+name: blender-hand
   1.269     #+ATTR_LaTeX: :width 10cm
   1.270     [[./images/hand-screenshot1.png]]
   1.271 -
   1.272     
   1.273     =CORTEX= creates a creature in two steps: first, it traverses the
   1.274     nodes in the blender file and creates physical representations for
   1.275 @@ -831,6 +938,7 @@
   1.276     [[./images/physical-hand.png]]
   1.277  
   1.278  
   1.279 +
   1.280  ** Eyes reuse standard video game components
   1.281  
   1.282  ** Hearing is hard; =CORTEX= does it right