1.1 --- a/org/cortex.org	Sun Oct 23 12:32:56 2011 -0700
     1.2 +++ b/org/cortex.org	Sun Oct 23 13:49:45 2011 -0700
     1.3 @@ -4,369 +4,65 @@
     1.4  #+description: Simulating senses for AI research using JMonkeyEngine3
     1.5  #+SETUPFILE: ../../aurellem/org/setup.org
     1.6  #+INCLUDE: ../../aurellem/org/level-0.org
     1.7 -#+babel: :mkdirp yes :noweb yes
     1.8 -
     1.9 -* Background
    1.10 -Artificial Intelligence has tried and failed for more than half a
    1.11 -century to produce programs as flexible, creative, and “intelligent”
    1.12 -as the human mind itself. Clearly, we are still missing some important
    1.13 -ideas concerning intelligent programs or we would have strong AI
    1.14 -already. What idea could be missing?
    1.15 -
    1.16 -When Turing first proposed his famous “Turing Test” in the
    1.17 -groundbreaking paper [[./sources/turing.pdf][/Computing Machines and Intelligence/]], he gave
    1.18 -little importance to how a computer program might interact with the
    1.19 -world:
    1.20 -
    1.21 -#+BEGIN_QUOTE
    1.22 -\ldquo{}We need not be too concerned about the legs, eyes, etc. The example of
    1.23 -Miss Helen Keller shows that education can take place provided that
    1.24 -communication in both directions between teacher and pupil can take
    1.25 -place by some means or other.\rdquo{}
    1.26 -#+END_QUOTE
    1.27 -
    1.28 -And from the example of Hellen Keller he went on to assume that the
    1.29 -only thing a fledgling AI program could need by way of communication
    1.30 -is a teletypewriter. But Hellen Keller did possess vision and hearing
    1.31 -for the first few months of her life, and her tactile sense was far
    1.32 -more rich than any text-stream could hope to achieve. She possessed a
    1.33 -body she could move freely, and had continual access to the real world
    1.34 -to learn from her actions.
    1.35 -
    1.36 -I believe that our programs are suffering from too little sensory
    1.37 -input to become really intelligent. Imagine for a moment that you
    1.38 -lived in a world completely cut off form all sensory stimulation. You
    1.39 -have no eyes to see, no ears to hear, no mouth to speak. No body, no
    1.40 -taste, no feeling whatsoever. The only sense you get at all is a
    1.41 -single point of light, flickering on and off in the void. If this was
    1.42 -your life from birth, you would never learn anything, and could never
    1.43 -become intelligent. Actual humans placed in sensory deprivation
    1.44 -chambers experience hallucinations and can begin to loose their sense
    1.45 -of reality in as little as 15 minutes[sensory-deprivation]. Most of
    1.46 -the time, the programs we write are in exactly this situation. They do
    1.47 -not interface with cameras and microphones, and they do not control a
    1.48 -real or simulated body or interact with any sort of world.
    1.49 -
    1.50 -
    1.51 -* Simulation vs. Reality
    1.52 -I want demonstrate that multiple senses are what enable
    1.53 -intelligence. There are two ways of playing around with senses and
    1.54 -computer programs:
    1.55 -
    1.56 -The first is to go entirely with simulation: virtual world, virtual
    1.57 -character, virtual senses. The advantages are that when everything is
    1.58 -a simulation, experiments in that simulation are absolutely
    1.59 -reproducible. It's also easier to change the character and world to
    1.60 -explore new situations and different sensory combinations.
    1.61 -
    1.62 -
    1.63 -** Issues with Simulation
    1.64 -
    1.65 -If the world is to be simulated on a computer, then not only do you
    1.66 -have to worry about whether the character's senses are rich enough to
    1.67 -learn from the world, but whether the world itself is rendered with
    1.68 -enough detail and realism to give enough working material to the
    1.69 -character's senses. To name just a few difficulties facing modern
    1.70 -physics simulators: destructibility of the environment, simulation of
    1.71 -water/other fluids, large areas, nonrigid bodies, lots of objects,
    1.72 -smoke. I don't know of any computer simulation that would allow a
    1.73 -character to take a rock and grind it into fine dust, then use that
    1.74 -dust to make a clay sculpture, at least not without spending years
    1.75 -calculating the interactions of every single small grain of
    1.76 -dust. Maybe a simulated world with today's limitations doesn't provide
    1.77 -enough richness for real intelligence to evolve.
    1.78 -
    1.79 -** Issues with Reality
    1.80 -
    1.81 -The other approach for playing with senses is to hook your software up
    1.82 -to real cameras, microphones, robots, etc., and let it loose in the
    1.83 -real world. This has the advantage of eliminating concerns about
    1.84 -simulating the world at the expense of increasing the complexity of
    1.85 -implementing the senses. Instead of just grabbing the current rendered
    1.86 -frame for processing, you have to use an actual camera with real
    1.87 -lenses and interact with photons to get an image. It is much harder to
    1.88 -change the character, which is now partly a physical robot of some
    1.89 -sort, since doing so involves changing things around in the real world
    1.90 -instead of modifying lines of code. While the real world is very rich
    1.91 -and definitely provides enough stimulation for intelligence to develop
    1.92 -as evidenced by our own existence, it is also uncontrollable in the
    1.93 -sense that a particular situation cannot be recreated perfectly or
    1.94 -saved for later use. It is harder to conduct science because it is
    1.95 -harder to repeat an experiment. The worst thing about using the real
    1.96 -world instead of a simulation is the matter of time. Instead of
    1.97 -simulated time you get the constant and unstoppable flow of real
    1.98 -time. This severely limits the sorts of software you can use to
    1.99 -program the AI because all sense inputs must be handled in real
   1.100 -time. Complicated ideas may have to be implemented in hardware or may
   1.101 -simply be impossible given the current speed of our
   1.102 -processors. Contrast this with a simulation, in which the flow of time
   1.103 -in the simulated world can be slowed down to accommodate the
   1.104 -limitations of the character's programming. In terms of cost, doing
   1.105 -everything in software is far cheaper than building custom real-time
   1.106 -hardware. All you need is a laptop and some patience.
   1.107 -
   1.108 -* Choose a Simulation Engine
   1.109 -
   1.110 -Mainly because of issues with controlling the flow of time, I chose to
   1.111 -simulate both the world and the character. I set out to make a minimal
   1.112 -world in which I could embed a character with multiple senses. My main
   1.113 -goal is to make an environment where I can perform further experiments
   1.114 -in simulated senses.
   1.115 -
   1.116 -As Carl Sagan once said, "If you wish to make an apple pie from
   1.117 -scratch, you must first invent the universe.” I examined many
   1.118 -different 3D environments to try and find something I would use as the
   1.119 -base for my simulation; eventually the choice came down to three
   1.120 -engines: the Quake II engine, the Source Engine, and jMonkeyEngine.
   1.121 -
   1.122 -** Quake II/Jake2
   1.123 -
   1.124 -I spent a bit more than a month working with the Quake II Engine from
   1.125 -ID software to see if I could use it for my purposes. All the source
   1.126 -code was released by ID software into the Public Domain several years
   1.127 -ago, and as a result it has been ported and modified for many
   1.128 -different reasons. This engine was famous for its advanced use of
   1.129 -realistic shading and had decent and fast physics
   1.130 -simulation. Researchers at Princeton [[http://www.nature.com/nature/journal/v461/n7266/pdf/nature08499.pdf][used this code]] to study spatial
   1.131 -information encoding in the hippocampal cells of rats. Those
   1.132 -researchers created a special Quake II level that simulated a maze,
   1.133 -and added an interface where a mouse could run around inside a ball in
   1.134 -various directions to move the character in the simulated maze. They
   1.135 -measured hippocampal activity during this exercise to try and tease
   1.136 -out the method in which spatial data was stored in that area of the
   1.137 -brain. I find this promising because if a real living rat can interact
   1.138 -with a computer simulation of a maze in the same way as it interacts
   1.139 -with a real-world maze, then maybe that simulation is close enough to
   1.140 -reality that a simulated sense of vision and motor control interacting
   1.141 -with that simulation could reveal useful information about the real
   1.142 -thing. It happens that there is a Java port of the original C source
   1.143 -code called Jake2. The port demonstrates Java's OpenGL bindings and
   1.144 -runs anywhere from 90% to 105% as fast as the C version. After
   1.145 -reviewing much of the source of Jake2, I eventually rejected it
   1.146 -because the engine is too tied to the concept of a first-person
   1.147 -shooter game. One of the problems I had was that there does not seem
   1.148 -to be any easy way to attach multiple cameras to a single
   1.149 -character. There are also several physics clipping issues that are
   1.150 -corrected in a way that only applies to the main character and does
   1.151 -not apply to arbitrary objects. While there is a large community of
   1.152 -level modders, I couldn't find a community to support using the engine
   1.153 -to make new things.
   1.154 -
   1.155 -** Source Engine
   1.156 -
   1.157 -The Source Engine evolved from the Quake II and Quake I engines and is
   1.158 -used by Valve in the Half-Life series of games. The physics simulation
   1.159 -in the Source Engine is quite accurate and probably the best out of
   1.160 -all the engines I investigated. There is also an extensive community
   1.161 -actively working with the engine. However, applications that use the
   1.162 -Source Engine must be written in C++, the code is not open, it only
   1.163 -runs on Windows, and the tools that come with the SDK to handle models
   1.164 -and textures are complicated and awkward to use.
   1.165 -
   1.166 -** jMonkeyEngine
   1.167 -
   1.168 -jMonkeyEngine is a new library for creating games in Java. It uses
   1.169 -OpenGL to render to the screen and uses screengraphs to avoid drawing
   1.170 -things that do not appear on the screen. It has an active community
   1.171 -and several games in the pipeline. The engine was not built to serve
   1.172 -any particular game but is instead meant to be used for any 3D
   1.173 -game. After experimenting with each of these three engines and a few
   1.174 -others for about 2 months I settled on jMonkeyEngine. I chose it
   1.175 -because it had the most features out of all the open projects I looked
   1.176 -at, and because I could then write my code in Clojure, an
   1.177 -implementation of LISP that runs on the JVM.
   1.178 -
   1.179 -* Setup
   1.180 -
   1.181 -First, I checked out the source to jMonkeyEngine:
   1.182 -
   1.183 -#+srcname: checkout 
   1.184 -#+begin_src sh :results verbatim
   1.185 -svn checkout http://jmonkeyengine.googlecode.com/svn/trunk/engine jme3
   1.186 -#+end_src
   1.187 -
   1.188 -#+results: checkout
   1.189 -: Checked out revision 7975.
   1.190 -
   1.191 -
   1.192 -Building jMonkeyEngine is easy enough:
   1.193 -
   1.194 -#+srcname: build
   1.195 -#+begin_src sh :results verbatim
   1.196 -cd jme3
   1.197 -ant jar | tail -n 2
   1.198 -#+end_src
   1.199 -
   1.200 -#+results: build
   1.201 -: BUILD SUCCESSFUL
   1.202 -: Total time: 15 seconds
   1.203 -
   1.204 -
   1.205 -Also build the javadoc:
   1.206 -
   1.207 -#+srcname: javadoc
   1.208 -#+begin_src sh :results verbatim
   1.209 -cd jme3
   1.210 -ant javadoc | tail -n 2
   1.211 -#+end_src
   1.212 -
   1.213 -#+results: javadoc
   1.214 -: BUILD SUCCESSFUL
   1.215 -: Total time: 12 seconds
   1.216 -
   1.217 -Now, move the jars from the compilation into the project's lib folder.
   1.218 -
   1.219 -#+srcname: move-jars
   1.220 -#+begin_src sh :results verbatim
   1.221 -mkdir -p lib 
   1.222 -mkdir -p src
   1.223 -cp jme3/dist/jMonkeyEngine3.jar lib/
   1.224 -cp jme3/dist/lib/* lib/
   1.225 -ls lib
   1.226 -#+end_src
   1.227 -
   1.228 -#+results: move-jars
   1.229 -#+begin_example
   1.230 -eventbus-1.4.jar
   1.231 -jbullet.jar
   1.232 -jheora-jst-debug-0.6.0.jar
   1.233 -jinput.jar
   1.234 -jME3-jbullet.jar
   1.235 -jME3-lwjgl-natives.jar
   1.236 -jME3-testdata.jar
   1.237 -jME3-test.jar
   1.238 -jMonkeyEngine3.jar
   1.239 -j-ogg-oggd.jar
   1.240 -j-ogg-vorbisd.jar
   1.241 -lwjgl.jar
   1.242 -nifty-1.3.jar
   1.243 -nifty-default-controls-1.3.jar
   1.244 -nifty-examples-1.3.jar
   1.245 -nifty-lwjgl-renderer-1.3.jar
   1.246 -nifty-openal-soundsystem-1.0.jar
   1.247 -nifty-style-black-1.3.jar
   1.248 -nifty-style-grey-1.0.jar
   1.249 -noise-0.0.1-SNAPSHOT.jar
   1.250 -stack-alloc.jar
   1.251 -vecmath.jar
   1.252 -xmlpull-xpp3-1.1.4c.jar
   1.253 -#+end_example
   1.254 -
   1.255 -It's good to create a =assets= directory in the style that the
   1.256 -=AssetManager= will like.
   1.257 -
   1.258 -#+srcname: create-assets
   1.259 -#+begin_src sh :results verbatim
   1.260 -mkdir -p assets
   1.261 -mkdir -p assets/Interface
   1.262 -mkdir -p assets/Materials
   1.263 -mkdir -p assets/MatDefs
   1.264 -mkdir -p assets/Models
   1.265 -mkdir -p assets/Scenes
   1.266 -mkdir -p assets/Shaders
   1.267 -mkdir -p assets/Sounds
   1.268 -mkdir -p assets/Textures
   1.269 -tree -L 1 assets
   1.270 -#+end_src
   1.271 -
   1.272 -#+results: create-assets
   1.273 -#+begin_example
   1.274 -assets
   1.275 -|-- Interface
   1.276 -|-- MatDefs
   1.277 -|-- Materials
   1.278 -|-- Models
   1.279 -|-- Scenes
   1.280 -|-- Shaders
   1.281 -|-- Sounds
   1.282 -`-- Textures
   1.283 -
   1.284 -8 directories, 0 files
   1.285 -#+end_example
   1.286 -
   1.287 -
   1.288 -The java classpath should have all the jars contained in the =lib=
   1.289 -directory as well as the src directory.
   1.290 -
   1.291 -For example, here is the file I use to run my REPL for clojure.
   1.292 -
   1.293 -#+include: "~/swank-all" src sh :exports code
   1.294 -
   1.295 -The important thing here is that =cortex/lib/*=, =cortex/src=, and
   1.296 -=cortex/assets= appear on the classpath. (=cortex= is the base
   1.297 -directory of this project.)
   1.298 -
   1.299 -#+srcname: pwd
   1.300 -#+begin_src sh 
   1.301 -pwd
   1.302 -#+end_src
   1.303 -
   1.304 -#+results: pwd
   1.305 -: /home/r/cortex
   1.306 +#+babel: :mkdirp yes :noweb yes :exports both
   1.307  
   1.308  
   1.309  * Simulation Base
   1.310    
   1.311  ** Imports
   1.312 -First, I'll import jme core classes. 
   1.313 +jMonkeyEngine has a plethora of classes which can be overwhelming at
   1.314 +first. So that I one can get right to coding, it's good to take the
   1.315 +time right now and make a "import all" function which brings in all of
   1.316 +the important jme3 classes. Once I'm happy with the general structure
   1.317 +of a namespace I can deal with importing only the classes it actually
   1.318 +needs.
   1.319 +
   1.320  #+srcname: import
   1.321  #+begin_src clojure :results silent
   1.322  (ns cortex.import
   1.323    (:require swank.util.class-browse))
   1.324  
   1.325 -(defn import-jme3 []
   1.326 -  (import '[com.jme3.system AppSettings JmeSystem]) 
   1.327 -  (import '[com.jme3.app Application SimpleApplication])
   1.328 -  (import 'com.jme3.material.Material)
   1.329 -  (import '[com.jme3.math Vector3f ColorRGBA Quaternion Transform])
   1.330 -  (import '[com.jme3.scene Node  Geometry])
   1.331 -  (import '[com.jme3.scene.shape Box Sphere Sphere$TextureMode])
   1.332 -  (import 'com.jme3.font.BitmapText)
   1.333 -  (import '[com.jme3.input KeyInput InputManager])
   1.334 -  (import '[com.jme3.input.controls
   1.335 -	    ActionListener AnalogListener KeyTrigger MouseButtonTrigger])
   1.336 -  (import '[com.jme3.asset AssetManager DesktopAssetManager] )
   1.337 -  (import '[com.jme3.asset.plugins HttpZipLocator ZipLocator])
   1.338 -  (import '[com.jme3.light PointLight DirectionalLight])
   1.339 -  (import '[com.jme3.animation AnimControl Skeleton Bone])
   1.340 -  (import '[com.jme3.bullet.collision.shapes
   1.341 -	    MeshCollisionShape SphereCollisionShape BoxCollisionShape])
   1.342 -  (import 'com.jme3.renderer.queue.RenderQueue$ShadowMode)
   1.343 -  (import 'jme3test.TestChooser)
   1.344 -  (import '[com.jme3.bullet PhysicsTickListener PhysicsSpace])
   1.345 -  (import '[com.jme3.bullet.joints SixDofJoint HingeJoint
   1.346 -	    SliderJoint Point2PointJoint ConeJoint]))
   1.347 +(defn permissive-import
   1.348 +  [classname]
   1.349 +  (eval `(try (import '~classname)
   1.350 +              (catch java.lang.Exception e#
   1.351 +                (println "couldn't import " '~classname))))
   1.352 +  classname)
   1.353  
   1.354 +(defn jme-class? [classname]
   1.355 +  (and
   1.356 +   (.startsWith classname "com.jme3.")
   1.357 +   ;; Don't import the Lwjgl stuff since it can throw exceptions
   1.358 +   ;;  upon being loaded.
   1.359 +   (not (re-matches #".*Lwjgl.*" classname))))
   1.360  
   1.361 -(defmacro permissive-import* [class-symbol]
   1.362 -  `(try
   1.363 -       (import ~class-symbol)
   1.364 -       (catch Exception e#
   1.365 -	 (println "can't import " ~class-symbol))))
   1.366 -
   1.367 -(defn permissive-import [class-symbol]
   1.368 -  (eval (list 'cortex.import/permissive-import* class-symbol)))
   1.369 -
   1.370 -(defn selection-import [selection-fn]
   1.371 -  (dorun
   1.372 -   (map (comp permissive-import symbol)
   1.373 -	(filter selection-fn
   1.374 -		(map :name
   1.375 -		     swank.util.class-browse/available-classes)))))
   1.376 +(defn jme-classes
   1.377 +  "returns a list of all jme3 classes"
   1.378 +  []
   1.379 +  (filter
   1.380 +   jme-class?
   1.381 +   (map :name
   1.382 +        swank.util.class-browse/available-classes)))
   1.383    
   1.384  (defn mega-import-jme3
   1.385 -  "ALL the jme classes. For REPL use."
   1.386 +  "Import ALL the jme classes. For REPL use."
   1.387    []
   1.388 -  (selection-import
   1.389 -   #(and
   1.390 -     (.startsWith % "com.jme3.")
   1.391 -     ;; Don't import the Lwjgl stuff since it can throw exceptions
   1.392 -     ;; upon being loaded.
   1.393 -     (not (re-matches #".*Lwjgl.*" %)))))
   1.394 +  (doall
   1.395 +   (map (comp permissive-import symbol) (jme-classes))))
   1.396  #+end_src  
   1.397  
   1.398  The =mega-import-jme3= is quite usefull for debugging purposes since
   1.399 -it allows completion for almost all of JME's classes
   1.400 +it allows completion for almost all of JME's classes.
   1.401 +
   1.402 +Out of curiousity, let's see just how many classes =mega-import-jme3=
   1.403 +imports:
   1.404 +
   1.405 +#+begin_src clojure :exports both
   1.406 +(clojure.core/count (cortex.import/jme-classes))
   1.407 +#+end_src
   1.408 +
   1.409 +#+results:
   1.410 +: 955
   1.411  
   1.412  ** Simplification
   1.413  *** World
   1.414 @@ -732,7 +428,7 @@
   1.415  (require 'cortex.import)
   1.416  (use 'clojure.contrib.def)
   1.417  (rlm.rlm-commands/help)
   1.418 -(cortex.import/import-jme3)
   1.419 +(cortex.import/mega-import-jme3)
   1.420  (use 'cortex.world)
   1.421  
   1.422  
   1.423 @@ -854,7 +550,7 @@
   1.424             (.setLinearVelocity
   1.425              (.getControl cannon-ball RigidBodyControl)
   1.426              (.mult (.getDirection camera) (float 50))) ;50
   1.427 -           (add-element game cannon-ball node)))))
   1.428 +           (add-element game cannon-ball (if node node (.getRootNode game)))))))
   1.429    ([]
   1.430      (fire-cannon-ball false)))
   1.431