Mercurial > cortex
comparison org/cortex.org @ 21:01e1427126af
splitting cortex into more manageable pieces
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Sun, 23 Oct 2011 13:49:45 -0700 |
parents | c32f3eb9fdeb |
children | cab2da252494 |
comparison
equal
deleted
inserted
replaced
20:67d508a1e34d | 21:01e1427126af |
---|---|
2 #+author: Robert McIntyre | 2 #+author: Robert McIntyre |
3 #+email: rlm@mit.edu | 3 #+email: rlm@mit.edu |
4 #+description: Simulating senses for AI research using JMonkeyEngine3 | 4 #+description: Simulating senses for AI research using JMonkeyEngine3 |
5 #+SETUPFILE: ../../aurellem/org/setup.org | 5 #+SETUPFILE: ../../aurellem/org/setup.org |
6 #+INCLUDE: ../../aurellem/org/level-0.org | 6 #+INCLUDE: ../../aurellem/org/level-0.org |
7 #+babel: :mkdirp yes :noweb yes | 7 #+babel: :mkdirp yes :noweb yes :exports both |
8 | |
9 * Background | |
10 Artificial Intelligence has tried and failed for more than half a | |
11 century to produce programs as flexible, creative, and “intelligent” | |
12 as the human mind itself. Clearly, we are still missing some important | |
13 ideas concerning intelligent programs or we would have strong AI | |
14 already. What idea could be missing? | |
15 | |
16 When Turing first proposed his famous “Turing Test” in the | |
17 groundbreaking paper [[./sources/turing.pdf][/Computing Machines and Intelligence/]], he gave | |
18 little importance to how a computer program might interact with the | |
19 world: | |
20 | |
21 #+BEGIN_QUOTE | |
22 \ldquo{}We need not be too concerned about the legs, eyes, etc. The example of | |
23 Miss Helen Keller shows that education can take place provided that | |
24 communication in both directions between teacher and pupil can take | |
25 place by some means or other.\rdquo{} | |
26 #+END_QUOTE | |
27 | |
28 And from the example of Hellen Keller he went on to assume that the | |
29 only thing a fledgling AI program could need by way of communication | |
30 is a teletypewriter. But Hellen Keller did possess vision and hearing | |
31 for the first few months of her life, and her tactile sense was far | |
32 more rich than any text-stream could hope to achieve. She possessed a | |
33 body she could move freely, and had continual access to the real world | |
34 to learn from her actions. | |
35 | |
36 I believe that our programs are suffering from too little sensory | |
37 input to become really intelligent. Imagine for a moment that you | |
38 lived in a world completely cut off form all sensory stimulation. You | |
39 have no eyes to see, no ears to hear, no mouth to speak. No body, no | |
40 taste, no feeling whatsoever. The only sense you get at all is a | |
41 single point of light, flickering on and off in the void. If this was | |
42 your life from birth, you would never learn anything, and could never | |
43 become intelligent. Actual humans placed in sensory deprivation | |
44 chambers experience hallucinations and can begin to loose their sense | |
45 of reality in as little as 15 minutes[sensory-deprivation]. Most of | |
46 the time, the programs we write are in exactly this situation. They do | |
47 not interface with cameras and microphones, and they do not control a | |
48 real or simulated body or interact with any sort of world. | |
49 | |
50 | |
51 * Simulation vs. Reality | |
52 I want demonstrate that multiple senses are what enable | |
53 intelligence. There are two ways of playing around with senses and | |
54 computer programs: | |
55 | |
56 The first is to go entirely with simulation: virtual world, virtual | |
57 character, virtual senses. The advantages are that when everything is | |
58 a simulation, experiments in that simulation are absolutely | |
59 reproducible. It's also easier to change the character and world to | |
60 explore new situations and different sensory combinations. | |
61 | |
62 | |
63 ** Issues with Simulation | |
64 | |
65 If the world is to be simulated on a computer, then not only do you | |
66 have to worry about whether the character's senses are rich enough to | |
67 learn from the world, but whether the world itself is rendered with | |
68 enough detail and realism to give enough working material to the | |
69 character's senses. To name just a few difficulties facing modern | |
70 physics simulators: destructibility of the environment, simulation of | |
71 water/other fluids, large areas, nonrigid bodies, lots of objects, | |
72 smoke. I don't know of any computer simulation that would allow a | |
73 character to take a rock and grind it into fine dust, then use that | |
74 dust to make a clay sculpture, at least not without spending years | |
75 calculating the interactions of every single small grain of | |
76 dust. Maybe a simulated world with today's limitations doesn't provide | |
77 enough richness for real intelligence to evolve. | |
78 | |
79 ** Issues with Reality | |
80 | |
81 The other approach for playing with senses is to hook your software up | |
82 to real cameras, microphones, robots, etc., and let it loose in the | |
83 real world. This has the advantage of eliminating concerns about | |
84 simulating the world at the expense of increasing the complexity of | |
85 implementing the senses. Instead of just grabbing the current rendered | |
86 frame for processing, you have to use an actual camera with real | |
87 lenses and interact with photons to get an image. It is much harder to | |
88 change the character, which is now partly a physical robot of some | |
89 sort, since doing so involves changing things around in the real world | |
90 instead of modifying lines of code. While the real world is very rich | |
91 and definitely provides enough stimulation for intelligence to develop | |
92 as evidenced by our own existence, it is also uncontrollable in the | |
93 sense that a particular situation cannot be recreated perfectly or | |
94 saved for later use. It is harder to conduct science because it is | |
95 harder to repeat an experiment. The worst thing about using the real | |
96 world instead of a simulation is the matter of time. Instead of | |
97 simulated time you get the constant and unstoppable flow of real | |
98 time. This severely limits the sorts of software you can use to | |
99 program the AI because all sense inputs must be handled in real | |
100 time. Complicated ideas may have to be implemented in hardware or may | |
101 simply be impossible given the current speed of our | |
102 processors. Contrast this with a simulation, in which the flow of time | |
103 in the simulated world can be slowed down to accommodate the | |
104 limitations of the character's programming. In terms of cost, doing | |
105 everything in software is far cheaper than building custom real-time | |
106 hardware. All you need is a laptop and some patience. | |
107 | |
108 * Choose a Simulation Engine | |
109 | |
110 Mainly because of issues with controlling the flow of time, I chose to | |
111 simulate both the world and the character. I set out to make a minimal | |
112 world in which I could embed a character with multiple senses. My main | |
113 goal is to make an environment where I can perform further experiments | |
114 in simulated senses. | |
115 | |
116 As Carl Sagan once said, "If you wish to make an apple pie from | |
117 scratch, you must first invent the universe.” I examined many | |
118 different 3D environments to try and find something I would use as the | |
119 base for my simulation; eventually the choice came down to three | |
120 engines: the Quake II engine, the Source Engine, and jMonkeyEngine. | |
121 | |
122 ** Quake II/Jake2 | |
123 | |
124 I spent a bit more than a month working with the Quake II Engine from | |
125 ID software to see if I could use it for my purposes. All the source | |
126 code was released by ID software into the Public Domain several years | |
127 ago, and as a result it has been ported and modified for many | |
128 different reasons. This engine was famous for its advanced use of | |
129 realistic shading and had decent and fast physics | |
130 simulation. Researchers at Princeton [[http://www.nature.com/nature/journal/v461/n7266/pdf/nature08499.pdf][used this code]] to study spatial | |
131 information encoding in the hippocampal cells of rats. Those | |
132 researchers created a special Quake II level that simulated a maze, | |
133 and added an interface where a mouse could run around inside a ball in | |
134 various directions to move the character in the simulated maze. They | |
135 measured hippocampal activity during this exercise to try and tease | |
136 out the method in which spatial data was stored in that area of the | |
137 brain. I find this promising because if a real living rat can interact | |
138 with a computer simulation of a maze in the same way as it interacts | |
139 with a real-world maze, then maybe that simulation is close enough to | |
140 reality that a simulated sense of vision and motor control interacting | |
141 with that simulation could reveal useful information about the real | |
142 thing. It happens that there is a Java port of the original C source | |
143 code called Jake2. The port demonstrates Java's OpenGL bindings and | |
144 runs anywhere from 90% to 105% as fast as the C version. After | |
145 reviewing much of the source of Jake2, I eventually rejected it | |
146 because the engine is too tied to the concept of a first-person | |
147 shooter game. One of the problems I had was that there does not seem | |
148 to be any easy way to attach multiple cameras to a single | |
149 character. There are also several physics clipping issues that are | |
150 corrected in a way that only applies to the main character and does | |
151 not apply to arbitrary objects. While there is a large community of | |
152 level modders, I couldn't find a community to support using the engine | |
153 to make new things. | |
154 | |
155 ** Source Engine | |
156 | |
157 The Source Engine evolved from the Quake II and Quake I engines and is | |
158 used by Valve in the Half-Life series of games. The physics simulation | |
159 in the Source Engine is quite accurate and probably the best out of | |
160 all the engines I investigated. There is also an extensive community | |
161 actively working with the engine. However, applications that use the | |
162 Source Engine must be written in C++, the code is not open, it only | |
163 runs on Windows, and the tools that come with the SDK to handle models | |
164 and textures are complicated and awkward to use. | |
165 | |
166 ** jMonkeyEngine | |
167 | |
168 jMonkeyEngine is a new library for creating games in Java. It uses | |
169 OpenGL to render to the screen and uses screengraphs to avoid drawing | |
170 things that do not appear on the screen. It has an active community | |
171 and several games in the pipeline. The engine was not built to serve | |
172 any particular game but is instead meant to be used for any 3D | |
173 game. After experimenting with each of these three engines and a few | |
174 others for about 2 months I settled on jMonkeyEngine. I chose it | |
175 because it had the most features out of all the open projects I looked | |
176 at, and because I could then write my code in Clojure, an | |
177 implementation of LISP that runs on the JVM. | |
178 | |
179 * Setup | |
180 | |
181 First, I checked out the source to jMonkeyEngine: | |
182 | |
183 #+srcname: checkout | |
184 #+begin_src sh :results verbatim | |
185 svn checkout http://jmonkeyengine.googlecode.com/svn/trunk/engine jme3 | |
186 #+end_src | |
187 | |
188 #+results: checkout | |
189 : Checked out revision 7975. | |
190 | |
191 | |
192 Building jMonkeyEngine is easy enough: | |
193 | |
194 #+srcname: build | |
195 #+begin_src sh :results verbatim | |
196 cd jme3 | |
197 ant jar | tail -n 2 | |
198 #+end_src | |
199 | |
200 #+results: build | |
201 : BUILD SUCCESSFUL | |
202 : Total time: 15 seconds | |
203 | |
204 | |
205 Also build the javadoc: | |
206 | |
207 #+srcname: javadoc | |
208 #+begin_src sh :results verbatim | |
209 cd jme3 | |
210 ant javadoc | tail -n 2 | |
211 #+end_src | |
212 | |
213 #+results: javadoc | |
214 : BUILD SUCCESSFUL | |
215 : Total time: 12 seconds | |
216 | |
217 Now, move the jars from the compilation into the project's lib folder. | |
218 | |
219 #+srcname: move-jars | |
220 #+begin_src sh :results verbatim | |
221 mkdir -p lib | |
222 mkdir -p src | |
223 cp jme3/dist/jMonkeyEngine3.jar lib/ | |
224 cp jme3/dist/lib/* lib/ | |
225 ls lib | |
226 #+end_src | |
227 | |
228 #+results: move-jars | |
229 #+begin_example | |
230 eventbus-1.4.jar | |
231 jbullet.jar | |
232 jheora-jst-debug-0.6.0.jar | |
233 jinput.jar | |
234 jME3-jbullet.jar | |
235 jME3-lwjgl-natives.jar | |
236 jME3-testdata.jar | |
237 jME3-test.jar | |
238 jMonkeyEngine3.jar | |
239 j-ogg-oggd.jar | |
240 j-ogg-vorbisd.jar | |
241 lwjgl.jar | |
242 nifty-1.3.jar | |
243 nifty-default-controls-1.3.jar | |
244 nifty-examples-1.3.jar | |
245 nifty-lwjgl-renderer-1.3.jar | |
246 nifty-openal-soundsystem-1.0.jar | |
247 nifty-style-black-1.3.jar | |
248 nifty-style-grey-1.0.jar | |
249 noise-0.0.1-SNAPSHOT.jar | |
250 stack-alloc.jar | |
251 vecmath.jar | |
252 xmlpull-xpp3-1.1.4c.jar | |
253 #+end_example | |
254 | |
255 It's good to create a =assets= directory in the style that the | |
256 =AssetManager= will like. | |
257 | |
258 #+srcname: create-assets | |
259 #+begin_src sh :results verbatim | |
260 mkdir -p assets | |
261 mkdir -p assets/Interface | |
262 mkdir -p assets/Materials | |
263 mkdir -p assets/MatDefs | |
264 mkdir -p assets/Models | |
265 mkdir -p assets/Scenes | |
266 mkdir -p assets/Shaders | |
267 mkdir -p assets/Sounds | |
268 mkdir -p assets/Textures | |
269 tree -L 1 assets | |
270 #+end_src | |
271 | |
272 #+results: create-assets | |
273 #+begin_example | |
274 assets | |
275 |-- Interface | |
276 |-- MatDefs | |
277 |-- Materials | |
278 |-- Models | |
279 |-- Scenes | |
280 |-- Shaders | |
281 |-- Sounds | |
282 `-- Textures | |
283 | |
284 8 directories, 0 files | |
285 #+end_example | |
286 | |
287 | |
288 The java classpath should have all the jars contained in the =lib= | |
289 directory as well as the src directory. | |
290 | |
291 For example, here is the file I use to run my REPL for clojure. | |
292 | |
293 #+include: "~/swank-all" src sh :exports code | |
294 | |
295 The important thing here is that =cortex/lib/*=, =cortex/src=, and | |
296 =cortex/assets= appear on the classpath. (=cortex= is the base | |
297 directory of this project.) | |
298 | |
299 #+srcname: pwd | |
300 #+begin_src sh | |
301 pwd | |
302 #+end_src | |
303 | |
304 #+results: pwd | |
305 : /home/r/cortex | |
306 | 8 |
307 | 9 |
308 * Simulation Base | 10 * Simulation Base |
309 | 11 |
310 ** Imports | 12 ** Imports |
311 First, I'll import jme core classes. | 13 jMonkeyEngine has a plethora of classes which can be overwhelming at |
14 first. So that I one can get right to coding, it's good to take the | |
15 time right now and make a "import all" function which brings in all of | |
16 the important jme3 classes. Once I'm happy with the general structure | |
17 of a namespace I can deal with importing only the classes it actually | |
18 needs. | |
19 | |
312 #+srcname: import | 20 #+srcname: import |
313 #+begin_src clojure :results silent | 21 #+begin_src clojure :results silent |
314 (ns cortex.import | 22 (ns cortex.import |
315 (:require swank.util.class-browse)) | 23 (:require swank.util.class-browse)) |
316 | 24 |
317 (defn import-jme3 [] | 25 (defn permissive-import |
318 (import '[com.jme3.system AppSettings JmeSystem]) | 26 [classname] |
319 (import '[com.jme3.app Application SimpleApplication]) | 27 (eval `(try (import '~classname) |
320 (import 'com.jme3.material.Material) | 28 (catch java.lang.Exception e# |
321 (import '[com.jme3.math Vector3f ColorRGBA Quaternion Transform]) | 29 (println "couldn't import " '~classname)))) |
322 (import '[com.jme3.scene Node Geometry]) | 30 classname) |
323 (import '[com.jme3.scene.shape Box Sphere Sphere$TextureMode]) | 31 |
324 (import 'com.jme3.font.BitmapText) | 32 (defn jme-class? [classname] |
325 (import '[com.jme3.input KeyInput InputManager]) | 33 (and |
326 (import '[com.jme3.input.controls | 34 (.startsWith classname "com.jme3.") |
327 ActionListener AnalogListener KeyTrigger MouseButtonTrigger]) | 35 ;; Don't import the Lwjgl stuff since it can throw exceptions |
328 (import '[com.jme3.asset AssetManager DesktopAssetManager] ) | 36 ;; upon being loaded. |
329 (import '[com.jme3.asset.plugins HttpZipLocator ZipLocator]) | 37 (not (re-matches #".*Lwjgl.*" classname)))) |
330 (import '[com.jme3.light PointLight DirectionalLight]) | 38 |
331 (import '[com.jme3.animation AnimControl Skeleton Bone]) | 39 (defn jme-classes |
332 (import '[com.jme3.bullet.collision.shapes | 40 "returns a list of all jme3 classes" |
333 MeshCollisionShape SphereCollisionShape BoxCollisionShape]) | 41 [] |
334 (import 'com.jme3.renderer.queue.RenderQueue$ShadowMode) | 42 (filter |
335 (import 'jme3test.TestChooser) | 43 jme-class? |
336 (import '[com.jme3.bullet PhysicsTickListener PhysicsSpace]) | 44 (map :name |
337 (import '[com.jme3.bullet.joints SixDofJoint HingeJoint | 45 swank.util.class-browse/available-classes))) |
338 SliderJoint Point2PointJoint ConeJoint])) | |
339 | |
340 | |
341 (defmacro permissive-import* [class-symbol] | |
342 `(try | |
343 (import ~class-symbol) | |
344 (catch Exception e# | |
345 (println "can't import " ~class-symbol)))) | |
346 | |
347 (defn permissive-import [class-symbol] | |
348 (eval (list 'cortex.import/permissive-import* class-symbol))) | |
349 | |
350 (defn selection-import [selection-fn] | |
351 (dorun | |
352 (map (comp permissive-import symbol) | |
353 (filter selection-fn | |
354 (map :name | |
355 swank.util.class-browse/available-classes))))) | |
356 | 46 |
357 (defn mega-import-jme3 | 47 (defn mega-import-jme3 |
358 "ALL the jme classes. For REPL use." | 48 "Import ALL the jme classes. For REPL use." |
359 [] | 49 [] |
360 (selection-import | 50 (doall |
361 #(and | 51 (map (comp permissive-import symbol) (jme-classes)))) |
362 (.startsWith % "com.jme3.") | |
363 ;; Don't import the Lwjgl stuff since it can throw exceptions | |
364 ;; upon being loaded. | |
365 (not (re-matches #".*Lwjgl.*" %))))) | |
366 #+end_src | 52 #+end_src |
367 | 53 |
368 The =mega-import-jme3= is quite usefull for debugging purposes since | 54 The =mega-import-jme3= is quite usefull for debugging purposes since |
369 it allows completion for almost all of JME's classes | 55 it allows completion for almost all of JME's classes. |
56 | |
57 Out of curiousity, let's see just how many classes =mega-import-jme3= | |
58 imports: | |
59 | |
60 #+begin_src clojure :exports both | |
61 (clojure.core/count (cortex.import/jme-classes)) | |
62 #+end_src | |
63 | |
64 #+results: | |
65 : 955 | |
370 | 66 |
371 ** Simplification | 67 ** Simplification |
372 *** World | 68 *** World |
373 | 69 |
374 It is comvienent to wrap the JME elements that deal with creating a | 70 It is comvienent to wrap the JME elements that deal with creating a |
730 #+begin_src clojure :results silent | 426 #+begin_src clojure :results silent |
731 (ns hello.hello-simple-app) | 427 (ns hello.hello-simple-app) |
732 (require 'cortex.import) | 428 (require 'cortex.import) |
733 (use 'clojure.contrib.def) | 429 (use 'clojure.contrib.def) |
734 (rlm.rlm-commands/help) | 430 (rlm.rlm-commands/help) |
735 (cortex.import/import-jme3) | 431 (cortex.import/mega-import-jme3) |
736 (use 'cortex.world) | 432 (use 'cortex.world) |
737 | 433 |
738 | 434 |
739 (def cube (Box. Vector3f/ZERO 1 1 1)) | 435 (def cube (Box. Vector3f/ZERO 1 1 1)) |
740 | 436 |
852 :mass 3)] ;200 0.05 | 548 :mass 3)] ;200 0.05 |
853 (.setShadowMode cannon-ball RenderQueue$ShadowMode/CastAndReceive) | 549 (.setShadowMode cannon-ball RenderQueue$ShadowMode/CastAndReceive) |
854 (.setLinearVelocity | 550 (.setLinearVelocity |
855 (.getControl cannon-ball RigidBodyControl) | 551 (.getControl cannon-ball RigidBodyControl) |
856 (.mult (.getDirection camera) (float 50))) ;50 | 552 (.mult (.getDirection camera) (float 50))) ;50 |
857 (add-element game cannon-ball node))))) | 553 (add-element game cannon-ball (if node node (.getRootNode game))))))) |
858 ([] | 554 ([] |
859 (fire-cannon-ball false))) | 555 (fire-cannon-ball false))) |
860 | 556 |
861 | 557 |
862 (defn floor* [] | 558 (defn floor* [] |