Mercurial > cortex

comparison thesis/cortex.org @ 432:1e5ea711857d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
abstract first draft.
author Robert McIntyre <rlm@mit.edu>
date Sun, 23 Mar 2014 16:33:01 -0400
parents 7410f0d8011c
children ae3bfc82ac7c
comparison
equal deleted inserted replaced
431:7410f0d8011c 432:1e5ea711857d
2 #+author: Robert McIntyre 2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu 3 #+email: rlm@mit.edu
4 #+description: Using embodied AI to facilitate Artificial Imagination. 4 #+description: Using embodied AI to facilitate Artificial Imagination.
5 #+keywords: AI, clojure, embodiment 5 #+keywords: AI, clojure, embodiment
6 6
7 * Artificial Imagination 7 * Vision
8 8
9 Imagine watching a video of someone skateboarding. When you watch 9 System for understanding what the actors in a video are doing --
10 the video, you can imagine yourself skateboarding, and your 10 Action Recognition.
11 knowledge of the human body and its dynamics guides your 11
12 interpretation of the scene. For example, even if the skateboarder 12 Separate action recognition into three components:
13 is partially occluded, you can infer the positions of his arms and
14 body from your own knowledge of how your body would be positioned if
15 you were skateboarding. If the skateboarder suffers an accident, you
16 wince in sympathy, imagining the pain your own body would experience
17 if it were in the same situation. This empathy with other people
18 guides our understanding of whatever they are doing because it is a
19 powerful constraint on what is probable and possible. In order to
20 make use of this powerful empathy constraint, I need a system that
21 can generate and make sense of sensory data from the many different
22 senses that humans possess. The two key proprieties of such a system
23 are /embodiment/ and /imagination/.
24 13
25 ** What is imagination? 14 - free play
15 - embodied action predicates
16 - model alignment
17 - sensory imagination
26 18
27 One kind of imagination is /sympathetic/ imagination: you imagine 19 * Steps
28 yourself in the position of something/someone you are 20
29 observing. This type of imagination comes into play when you follow 21 - Build cortex, a simulated environment for sensate AI
30 along visually when watching someone perform actions, or when you 22 - solid bodies w/ joints
31 sympathetically grimace when someone hurts themselves. This type of 23 - vision
32 imagination uses the constraints you have learned about your own 24 - touch
33 body to highly constrain the possibilities in whatever you are 25 - vision
34 seeing. It uses all your senses to including your senses of touch, 26 - hearing
35 proprioception, etc. Humans are flexible when it comes to "putting 27 - proprioception
36 themselves in another's shoes," and can sympathetically understand 28 - muscle contraction
37 not only other humans, but entities ranging from animals to cartoon 29
38 characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen! 30 - Build experimental framework for worm-actions
31 - embodied stream predicates
32 - \phi-space
33 - \phi-scan
34
35 * News
36
37 Experimental results:
38
39 - \phi-space actually works very well for the worm!
40 - self organizing touch map
39 41
40 42
41 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers. 43 * Contributions
42 #+ATTR_LaTeX: :width 5cm 44 - Built =CORTEX=, a comprehensive platform for embodied AI
43 [[./images/cat-drinking.jpg]] 45 experiments. Has many new features lacking in other systems, such
46 as sound. Easy to model/create new creatures.
47 - created a novel concept for action recognition by using artificial
48 imagination.
44 49
45 50
46 #+begin_listing
47 \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.}
48 #+name: test-1
49 #+begin_src clojure
50 (defn test-pipeline
51 "Testing vision:
52 Tests the vision system by creating two views of the same rotating
53 object from different angles and displaying both of those views in
54 JFrames.
55 51
56 You should see a rotating cube, and two windows,
57 each displaying a different view of the cube."
58 ([] (test-pipeline false))
59 ([record?]
60 (let [candy
61 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
62 (world
63 (doto (Node.)
64 (.attachChild candy))
65 {}
66 (fn [world]
67 (let [cam (.clone (.getCamera world))
68 width (.getWidth cam)
69 height (.getHeight cam)]
70 (add-camera! world cam
71 (comp
72 (view-image
73 (if record?
74 (File. "/home/r/proj/cortex/render/vision/1")))
75 BufferedImage!))
76 (add-camera! world
77 (doto (.clone cam)
78 (.setLocation (Vector3f. -10 0 0))
79 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
80 (comp
81 (view-image
82 (if record?
83 (File. "/home/r/proj/cortex/render/vision/2")))
84 BufferedImage!))
85 (let [timer (IsoTimer. 60)]
86 (.setTimer world timer)
87 (display-dilated-time world timer))
88 ;; This is here to restore the main view
89 ;; after the other views have completed processing
90 (add-camera! world (.getCamera world) no-op)))
91 (fn [world tpf]
92 (.rotate candy (* tpf 0.2) 0 0))))))
93 #+end_src
94 #+end_listing
95 52
96 - This is test1 \cite{Tappert77}.
97 53
98 \cite{Tappert77} 54
99 lol 55
100 \cite{Tappert77}