# HG changeset patch # User Robert McIntyre # Date 1319402985 25200 # Node ID 01e1427126afffbadcf8825f2fd50fa34c498211 # Parent 67d508a1e34de4a6ec19223da9c58bb95be5fe33 splitting cortex into more manageable pieces diff -r 67d508a1e34d -r 01e1427126af org/cortex.org --- a/org/cortex.org Sun Oct 23 12:32:56 2011 -0700 +++ b/org/cortex.org Sun Oct 23 13:49:45 2011 -0700 @@ -4,369 +4,65 @@ #+description: Simulating senses for AI research using JMonkeyEngine3 #+SETUPFILE: ../../aurellem/org/setup.org #+INCLUDE: ../../aurellem/org/level-0.org -#+babel: :mkdirp yes :noweb yes - -* Background -Artificial Intelligence has tried and failed for more than half a -century to produce programs as flexible, creative, and “intelligent” -as the human mind itself. Clearly, we are still missing some important -ideas concerning intelligent programs or we would have strong AI -already. What idea could be missing? - -When Turing first proposed his famous “Turing Test” in the -groundbreaking paper [[./sources/turing.pdf][/Computing Machines and Intelligence/]], he gave -little importance to how a computer program might interact with the -world: - -#+BEGIN_QUOTE -\ldquo{}We need not be too concerned about the legs, eyes, etc. The example of -Miss Helen Keller shows that education can take place provided that -communication in both directions between teacher and pupil can take -place by some means or other.\rdquo{} -#+END_QUOTE - -And from the example of Hellen Keller he went on to assume that the -only thing a fledgling AI program could need by way of communication -is a teletypewriter. But Hellen Keller did possess vision and hearing -for the first few months of her life, and her tactile sense was far -more rich than any text-stream could hope to achieve. She possessed a -body she could move freely, and had continual access to the real world -to learn from her actions. - -I believe that our programs are suffering from too little sensory -input to become really intelligent. Imagine for a moment that you -lived in a world completely cut off form all sensory stimulation. You -have no eyes to see, no ears to hear, no mouth to speak. No body, no -taste, no feeling whatsoever. The only sense you get at all is a -single point of light, flickering on and off in the void. If this was -your life from birth, you would never learn anything, and could never -become intelligent. Actual humans placed in sensory deprivation -chambers experience hallucinations and can begin to loose their sense -of reality in as little as 15 minutes[sensory-deprivation]. Most of -the time, the programs we write are in exactly this situation. They do -not interface with cameras and microphones, and they do not control a -real or simulated body or interact with any sort of world. - - -* Simulation vs. Reality -I want demonstrate that multiple senses are what enable -intelligence. There are two ways of playing around with senses and -computer programs: - -The first is to go entirely with simulation: virtual world, virtual -character, virtual senses. The advantages are that when everything is -a simulation, experiments in that simulation are absolutely -reproducible. It's also easier to change the character and world to -explore new situations and different sensory combinations. - - -** Issues with Simulation - -If the world is to be simulated on a computer, then not only do you -have to worry about whether the character's senses are rich enough to -learn from the world, but whether the world itself is rendered with -enough detail and realism to give enough working material to the -character's senses. To name just a few difficulties facing modern -physics simulators: destructibility of the environment, simulation of -water/other fluids, large areas, nonrigid bodies, lots of objects, -smoke. I don't know of any computer simulation that would allow a -character to take a rock and grind it into fine dust, then use that -dust to make a clay sculpture, at least not without spending years -calculating the interactions of every single small grain of -dust. Maybe a simulated world with today's limitations doesn't provide -enough richness for real intelligence to evolve. - -** Issues with Reality - -The other approach for playing with senses is to hook your software up -to real cameras, microphones, robots, etc., and let it loose in the -real world. This has the advantage of eliminating concerns about -simulating the world at the expense of increasing the complexity of -implementing the senses. Instead of just grabbing the current rendered -frame for processing, you have to use an actual camera with real -lenses and interact with photons to get an image. It is much harder to -change the character, which is now partly a physical robot of some -sort, since doing so involves changing things around in the real world -instead of modifying lines of code. While the real world is very rich -and definitely provides enough stimulation for intelligence to develop -as evidenced by our own existence, it is also uncontrollable in the -sense that a particular situation cannot be recreated perfectly or -saved for later use. It is harder to conduct science because it is -harder to repeat an experiment. The worst thing about using the real -world instead of a simulation is the matter of time. Instead of -simulated time you get the constant and unstoppable flow of real -time. This severely limits the sorts of software you can use to -program the AI because all sense inputs must be handled in real -time. Complicated ideas may have to be implemented in hardware or may -simply be impossible given the current speed of our -processors. Contrast this with a simulation, in which the flow of time -in the simulated world can be slowed down to accommodate the -limitations of the character's programming. In terms of cost, doing -everything in software is far cheaper than building custom real-time -hardware. All you need is a laptop and some patience. - -* Choose a Simulation Engine - -Mainly because of issues with controlling the flow of time, I chose to -simulate both the world and the character. I set out to make a minimal -world in which I could embed a character with multiple senses. My main -goal is to make an environment where I can perform further experiments -in simulated senses. - -As Carl Sagan once said, "If you wish to make an apple pie from -scratch, you must first invent the universe.” I examined many -different 3D environments to try and find something I would use as the -base for my simulation; eventually the choice came down to three -engines: the Quake II engine, the Source Engine, and jMonkeyEngine. - -** Quake II/Jake2 - -I spent a bit more than a month working with the Quake II Engine from -ID software to see if I could use it for my purposes. All the source -code was released by ID software into the Public Domain several years -ago, and as a result it has been ported and modified for many -different reasons. This engine was famous for its advanced use of -realistic shading and had decent and fast physics -simulation. Researchers at Princeton [[http://www.nature.com/nature/journal/v461/n7266/pdf/nature08499.pdf][used this code]] to study spatial -information encoding in the hippocampal cells of rats. Those -researchers created a special Quake II level that simulated a maze, -and added an interface where a mouse could run around inside a ball in -various directions to move the character in the simulated maze. They -measured hippocampal activity during this exercise to try and tease -out the method in which spatial data was stored in that area of the -brain. I find this promising because if a real living rat can interact -with a computer simulation of a maze in the same way as it interacts -with a real-world maze, then maybe that simulation is close enough to -reality that a simulated sense of vision and motor control interacting -with that simulation could reveal useful information about the real -thing. It happens that there is a Java port of the original C source -code called Jake2. The port demonstrates Java's OpenGL bindings and -runs anywhere from 90% to 105% as fast as the C version. After -reviewing much of the source of Jake2, I eventually rejected it -because the engine is too tied to the concept of a first-person -shooter game. One of the problems I had was that there does not seem -to be any easy way to attach multiple cameras to a single -character. There are also several physics clipping issues that are -corrected in a way that only applies to the main character and does -not apply to arbitrary objects. While there is a large community of -level modders, I couldn't find a community to support using the engine -to make new things. - -** Source Engine - -The Source Engine evolved from the Quake II and Quake I engines and is -used by Valve in the Half-Life series of games. The physics simulation -in the Source Engine is quite accurate and probably the best out of -all the engines I investigated. There is also an extensive community -actively working with the engine. However, applications that use the -Source Engine must be written in C++, the code is not open, it only -runs on Windows, and the tools that come with the SDK to handle models -and textures are complicated and awkward to use. - -** jMonkeyEngine - -jMonkeyEngine is a new library for creating games in Java. It uses -OpenGL to render to the screen and uses screengraphs to avoid drawing -things that do not appear on the screen. It has an active community -and several games in the pipeline. The engine was not built to serve -any particular game but is instead meant to be used for any 3D -game. After experimenting with each of these three engines and a few -others for about 2 months I settled on jMonkeyEngine. I chose it -because it had the most features out of all the open projects I looked -at, and because I could then write my code in Clojure, an -implementation of LISP that runs on the JVM. - -* Setup - -First, I checked out the source to jMonkeyEngine: - -#+srcname: checkout -#+begin_src sh :results verbatim -svn checkout http://jmonkeyengine.googlecode.com/svn/trunk/engine jme3 -#+end_src - -#+results: checkout -: Checked out revision 7975. - - -Building jMonkeyEngine is easy enough: - -#+srcname: build -#+begin_src sh :results verbatim -cd jme3 -ant jar | tail -n 2 -#+end_src - -#+results: build -: BUILD SUCCESSFUL -: Total time: 15 seconds - - -Also build the javadoc: - -#+srcname: javadoc -#+begin_src sh :results verbatim -cd jme3 -ant javadoc | tail -n 2 -#+end_src - -#+results: javadoc -: BUILD SUCCESSFUL -: Total time: 12 seconds - -Now, move the jars from the compilation into the project's lib folder. - -#+srcname: move-jars -#+begin_src sh :results verbatim -mkdir -p lib -mkdir -p src -cp jme3/dist/jMonkeyEngine3.jar lib/ -cp jme3/dist/lib/* lib/ -ls lib -#+end_src - -#+results: move-jars -#+begin_example -eventbus-1.4.jar -jbullet.jar -jheora-jst-debug-0.6.0.jar -jinput.jar -jME3-jbullet.jar -jME3-lwjgl-natives.jar -jME3-testdata.jar -jME3-test.jar -jMonkeyEngine3.jar -j-ogg-oggd.jar -j-ogg-vorbisd.jar -lwjgl.jar -nifty-1.3.jar -nifty-default-controls-1.3.jar -nifty-examples-1.3.jar -nifty-lwjgl-renderer-1.3.jar -nifty-openal-soundsystem-1.0.jar -nifty-style-black-1.3.jar -nifty-style-grey-1.0.jar -noise-0.0.1-SNAPSHOT.jar -stack-alloc.jar -vecmath.jar -xmlpull-xpp3-1.1.4c.jar -#+end_example - -It's good to create a =assets= directory in the style that the -=AssetManager= will like. - -#+srcname: create-assets -#+begin_src sh :results verbatim -mkdir -p assets -mkdir -p assets/Interface -mkdir -p assets/Materials -mkdir -p assets/MatDefs -mkdir -p assets/Models -mkdir -p assets/Scenes -mkdir -p assets/Shaders -mkdir -p assets/Sounds -mkdir -p assets/Textures -tree -L 1 assets -#+end_src - -#+results: create-assets -#+begin_example -assets -|-- Interface -|-- MatDefs -|-- Materials -|-- Models -|-- Scenes -|-- Shaders -|-- Sounds -`-- Textures - -8 directories, 0 files -#+end_example - - -The java classpath should have all the jars contained in the =lib= -directory as well as the src directory. - -For example, here is the file I use to run my REPL for clojure. - -#+include: "~/swank-all" src sh :exports code - -The important thing here is that =cortex/lib/*=, =cortex/src=, and -=cortex/assets= appear on the classpath. (=cortex= is the base -directory of this project.) - -#+srcname: pwd -#+begin_src sh -pwd -#+end_src - -#+results: pwd -: /home/r/cortex +#+babel: :mkdirp yes :noweb yes :exports both * Simulation Base ** Imports -First, I'll import jme core classes. +jMonkeyEngine has a plethora of classes which can be overwhelming at +first. So that I one can get right to coding, it's good to take the +time right now and make a "import all" function which brings in all of +the important jme3 classes. Once I'm happy with the general structure +of a namespace I can deal with importing only the classes it actually +needs. + #+srcname: import #+begin_src clojure :results silent (ns cortex.import (:require swank.util.class-browse)) -(defn import-jme3 [] - (import '[com.jme3.system AppSettings JmeSystem]) - (import '[com.jme3.app Application SimpleApplication]) - (import 'com.jme3.material.Material) - (import '[com.jme3.math Vector3f ColorRGBA Quaternion Transform]) - (import '[com.jme3.scene Node Geometry]) - (import '[com.jme3.scene.shape Box Sphere Sphere$TextureMode]) - (import 'com.jme3.font.BitmapText) - (import '[com.jme3.input KeyInput InputManager]) - (import '[com.jme3.input.controls - ActionListener AnalogListener KeyTrigger MouseButtonTrigger]) - (import '[com.jme3.asset AssetManager DesktopAssetManager] ) - (import '[com.jme3.asset.plugins HttpZipLocator ZipLocator]) - (import '[com.jme3.light PointLight DirectionalLight]) - (import '[com.jme3.animation AnimControl Skeleton Bone]) - (import '[com.jme3.bullet.collision.shapes - MeshCollisionShape SphereCollisionShape BoxCollisionShape]) - (import 'com.jme3.renderer.queue.RenderQueue$ShadowMode) - (import 'jme3test.TestChooser) - (import '[com.jme3.bullet PhysicsTickListener PhysicsSpace]) - (import '[com.jme3.bullet.joints SixDofJoint HingeJoint - SliderJoint Point2PointJoint ConeJoint])) +(defn permissive-import + [classname] + (eval `(try (import '~classname) + (catch java.lang.Exception e# + (println "couldn't import " '~classname)))) + classname) +(defn jme-class? [classname] + (and + (.startsWith classname "com.jme3.") + ;; Don't import the Lwjgl stuff since it can throw exceptions + ;; upon being loaded. + (not (re-matches #".*Lwjgl.*" classname)))) -(defmacro permissive-import* [class-symbol] - `(try - (import ~class-symbol) - (catch Exception e# - (println "can't import " ~class-symbol)))) - -(defn permissive-import [class-symbol] - (eval (list 'cortex.import/permissive-import* class-symbol))) - -(defn selection-import [selection-fn] - (dorun - (map (comp permissive-import symbol) - (filter selection-fn - (map :name - swank.util.class-browse/available-classes))))) +(defn jme-classes + "returns a list of all jme3 classes" + [] + (filter + jme-class? + (map :name + swank.util.class-browse/available-classes))) (defn mega-import-jme3 - "ALL the jme classes. For REPL use." + "Import ALL the jme classes. For REPL use." [] - (selection-import - #(and - (.startsWith % "com.jme3.") - ;; Don't import the Lwjgl stuff since it can throw exceptions - ;; upon being loaded. - (not (re-matches #".*Lwjgl.*" %))))) + (doall + (map (comp permissive-import symbol) (jme-classes)))) #+end_src The =mega-import-jme3= is quite usefull for debugging purposes since -it allows completion for almost all of JME's classes +it allows completion for almost all of JME's classes. + +Out of curiousity, let's see just how many classes =mega-import-jme3= +imports: + +#+begin_src clojure :exports both +(clojure.core/count (cortex.import/jme-classes)) +#+end_src + +#+results: +: 955 ** Simplification *** World @@ -732,7 +428,7 @@ (require 'cortex.import) (use 'clojure.contrib.def) (rlm.rlm-commands/help) -(cortex.import/import-jme3) +(cortex.import/mega-import-jme3) (use 'cortex.world) @@ -854,7 +550,7 @@ (.setLinearVelocity (.getControl cannon-ball RigidBodyControl) (.mult (.getDirection camera) (float 50))) ;50 - (add-element game cannon-ball node))))) + (add-element game cannon-ball (if node node (.getRootNode game))))))) ([] (fire-cannon-ball false)))