Mercurial > cortex
comparison thesis/cortex.org @ 432:1e5ea711857d
abstract first draft.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Sun, 23 Mar 2014 16:33:01 -0400 |
parents | 7410f0d8011c |
children | ae3bfc82ac7c |
comparison
equal
deleted
inserted
replaced
431:7410f0d8011c | 432:1e5ea711857d |
---|---|
2 #+author: Robert McIntyre | 2 #+author: Robert McIntyre |
3 #+email: rlm@mit.edu | 3 #+email: rlm@mit.edu |
4 #+description: Using embodied AI to facilitate Artificial Imagination. | 4 #+description: Using embodied AI to facilitate Artificial Imagination. |
5 #+keywords: AI, clojure, embodiment | 5 #+keywords: AI, clojure, embodiment |
6 | 6 |
7 * Artificial Imagination | 7 * Vision |
8 | 8 |
9 Imagine watching a video of someone skateboarding. When you watch | 9 System for understanding what the actors in a video are doing -- |
10 the video, you can imagine yourself skateboarding, and your | 10 Action Recognition. |
11 knowledge of the human body and its dynamics guides your | 11 |
12 interpretation of the scene. For example, even if the skateboarder | 12 Separate action recognition into three components: |
13 is partially occluded, you can infer the positions of his arms and | |
14 body from your own knowledge of how your body would be positioned if | |
15 you were skateboarding. If the skateboarder suffers an accident, you | |
16 wince in sympathy, imagining the pain your own body would experience | |
17 if it were in the same situation. This empathy with other people | |
18 guides our understanding of whatever they are doing because it is a | |
19 powerful constraint on what is probable and possible. In order to | |
20 make use of this powerful empathy constraint, I need a system that | |
21 can generate and make sense of sensory data from the many different | |
22 senses that humans possess. The two key proprieties of such a system | |
23 are /embodiment/ and /imagination/. | |
24 | 13 |
25 ** What is imagination? | 14 - free play |
15 - embodied action predicates | |
16 - model alignment | |
17 - sensory imagination | |
26 | 18 |
27 One kind of imagination is /sympathetic/ imagination: you imagine | 19 * Steps |
28 yourself in the position of something/someone you are | 20 |
29 observing. This type of imagination comes into play when you follow | 21 - Build cortex, a simulated environment for sensate AI |
30 along visually when watching someone perform actions, or when you | 22 - solid bodies w/ joints |
31 sympathetically grimace when someone hurts themselves. This type of | 23 - vision |
32 imagination uses the constraints you have learned about your own | 24 - touch |
33 body to highly constrain the possibilities in whatever you are | 25 - vision |
34 seeing. It uses all your senses to including your senses of touch, | 26 - hearing |
35 proprioception, etc. Humans are flexible when it comes to "putting | 27 - proprioception |
36 themselves in another's shoes," and can sympathetically understand | 28 - muscle contraction |
37 not only other humans, but entities ranging from animals to cartoon | 29 |
38 characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen! | 30 - Build experimental framework for worm-actions |
31 - embodied stream predicates | |
32 - \phi-space | |
33 - \phi-scan | |
34 | |
35 * News | |
36 | |
37 Experimental results: | |
38 | |
39 - \phi-space actually works very well for the worm! | |
40 - self organizing touch map | |
39 | 41 |
40 | 42 |
41 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers. | 43 * Contributions |
42 #+ATTR_LaTeX: :width 5cm | 44 - Built =CORTEX=, a comprehensive platform for embodied AI |
43 [[./images/cat-drinking.jpg]] | 45 experiments. Has many new features lacking in other systems, such |
46 as sound. Easy to model/create new creatures. | |
47 - created a novel concept for action recognition by using artificial | |
48 imagination. | |
44 | 49 |
45 | 50 |
46 #+begin_listing | |
47 \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} | |
48 #+name: test-1 | |
49 #+begin_src clojure | |
50 (defn test-pipeline | |
51 "Testing vision: | |
52 Tests the vision system by creating two views of the same rotating | |
53 object from different angles and displaying both of those views in | |
54 JFrames. | |
55 | 51 |
56 You should see a rotating cube, and two windows, | |
57 each displaying a different view of the cube." | |
58 ([] (test-pipeline false)) | |
59 ([record?] | |
60 (let [candy | |
61 (box 1 1 1 :physical? false :color ColorRGBA/Blue)] | |
62 (world | |
63 (doto (Node.) | |
64 (.attachChild candy)) | |
65 {} | |
66 (fn [world] | |
67 (let [cam (.clone (.getCamera world)) | |
68 width (.getWidth cam) | |
69 height (.getHeight cam)] | |
70 (add-camera! world cam | |
71 (comp | |
72 (view-image | |
73 (if record? | |
74 (File. "/home/r/proj/cortex/render/vision/1"))) | |
75 BufferedImage!)) | |
76 (add-camera! world | |
77 (doto (.clone cam) | |
78 (.setLocation (Vector3f. -10 0 0)) | |
79 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) | |
80 (comp | |
81 (view-image | |
82 (if record? | |
83 (File. "/home/r/proj/cortex/render/vision/2"))) | |
84 BufferedImage!)) | |
85 (let [timer (IsoTimer. 60)] | |
86 (.setTimer world timer) | |
87 (display-dilated-time world timer)) | |
88 ;; This is here to restore the main view | |
89 ;; after the other views have completed processing | |
90 (add-camera! world (.getCamera world) no-op))) | |
91 (fn [world tpf] | |
92 (.rotate candy (* tpf 0.2) 0 0)))))) | |
93 #+end_src | |
94 #+end_listing | |
95 | 52 |
96 - This is test1 \cite{Tappert77}. | |
97 | 53 |
98 \cite{Tappert77} | 54 |
99 lol | 55 |
100 \cite{Tappert77} |