Mercurial > cortex
comparison thesis/cortex.tex @ 432:1e5ea711857d
abstract first draft.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Sun, 23 Mar 2014 16:33:01 -0400 |
parents | 7410f0d8011c |
children |
comparison
equal
deleted
inserted
replaced
431:7410f0d8011c | 432:1e5ea711857d |
---|---|
1 | 1 |
2 \section{Artificial Imagination} | 2 \section{Vision} |
3 \label{sec-1} | 3 \label{sec-1} |
4 | 4 |
5 Imagine watching a video of someone skateboarding. When you watch | 5 System for understanding what the actors in a video are doing -- |
6 the video, you can imagine yourself skateboarding, and your | 6 Action Recognition. |
7 knowledge of the human body and its dynamics guides your | |
8 interpretation of the scene. For example, even if the skateboarder | |
9 is partially occluded, you can infer the positions of his arms and | |
10 body from your own knowledge of how your body would be positioned if | |
11 you were skateboarding. If the skateboarder suffers an accident, you | |
12 wince in sympathy, imagining the pain your own body would experience | |
13 if it were in the same situation. This empathy with other people | |
14 guides our understanding of whatever they are doing because it is a | |
15 powerful constraint on what is probable and possible. In order to | |
16 make use of this powerful empathy constraint, I need a system that | |
17 can generate and make sense of sensory data from the many different | |
18 senses that humans possess. The two key proprieties of such a system | |
19 are \emph{embodiment} and \emph{imagination}. | |
20 | 7 |
21 \subsection{What is imagination?} | 8 Separate action recognition into three components: |
22 \label{sec-1-1} | |
23 | |
24 One kind of imagination is \emph{sympathetic} imagination: you imagine | |
25 yourself in the position of something/someone you are | |
26 observing. This type of imagination comes into play when you follow | |
27 along visually when watching someone perform actions, or when you | |
28 sympathetically grimace when someone hurts themselves. This type of | |
29 imagination uses the constraints you have learned about your own | |
30 body to highly constrain the possibilities in whatever you are | |
31 seeing. It uses all your senses to including your senses of touch, | |
32 proprioception, etc. Humans are flexible when it comes to "putting | |
33 themselves in another's shoes," and can sympathetically understand | |
34 not only other humans, but entities ranging from animals to cartoon | |
35 characters to \href{http://www.youtube.com/watch?v=0jz4HcwTQmU}{single dots} on a screen! | |
36 | |
37 | |
38 \begin{figure}[htb] | |
39 \centering | |
40 \includegraphics[width=5cm]{./images/cat-drinking.jpg} | |
41 \caption{A cat drinking some water. Identifying this action is beyond the state of the art for computers.} | |
42 \end{figure} | |
43 | |
44 | |
45 \begin{listing} | |
46 \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} | |
47 \begin{clojurecode} | |
48 (defn test-pipeline | |
49 "Testing vision: | |
50 Tests the vision system by creating two views of the same rotating | |
51 object from different angles and displaying both of those views in | |
52 JFrames. | |
53 | |
54 You should see a rotating cube, and two windows, | |
55 each displaying a different view of the cube." | |
56 ([] (test-pipeline false)) | |
57 ([record?] | |
58 (let [candy | |
59 (box 1 1 1 :physical? false :color ColorRGBA/Blue)] | |
60 (world | |
61 (doto (Node.) | |
62 (.attachChild candy)) | |
63 {} | |
64 (fn [world] | |
65 (let [cam (.clone (.getCamera world)) | |
66 width (.getWidth cam) | |
67 height (.getHeight cam)] | |
68 (add-camera! world cam | |
69 (comp | |
70 (view-image | |
71 (if record? | |
72 (File. "/home/r/proj/cortex/render/vision/1"))) | |
73 BufferedImage!)) | |
74 (add-camera! world | |
75 (doto (.clone cam) | |
76 (.setLocation (Vector3f. -10 0 0)) | |
77 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) | |
78 (comp | |
79 (view-image | |
80 (if record? | |
81 (File. "/home/r/proj/cortex/render/vision/2"))) | |
82 BufferedImage!)) | |
83 (let [timer (IsoTimer. 60)] | |
84 (.setTimer world timer) | |
85 (display-dilated-time world timer)) | |
86 ;; This is here to restore the main view | |
87 ;; after the other views have completed processing | |
88 (add-camera! world (.getCamera world) no-op))) | |
89 (fn [world tpf] | |
90 (.rotate candy (* tpf 0.2) 0 0)))))) | |
91 \end{clojurecode} | |
92 \end{listing} | |
93 | 9 |
94 \begin{itemize} | 10 \begin{itemize} |
95 \item This is test1 \cite{Tappert77}. | 11 \item free play |
12 \item embodied action predicates | |
13 \item model alignment | |
14 \item sensory imagination | |
15 \end{itemize} | |
16 \section{Steps} | |
17 \label{sec-2} | |
18 | |
19 \begin{itemize} | |
20 \item Build cortex, a simulated environment for sensate AI | |
21 \begin{itemize} | |
22 \item solid bodies w/ joints | |
23 \item vision | |
24 \item touch | |
25 \item vision | |
26 \item hearing | |
27 \item proprioception | |
28 \item muscle contraction | |
96 \end{itemize} | 29 \end{itemize} |
97 | 30 |
98 \cite{Tappert77} | 31 \item Build experimental framework for worm-actions |
99 lol | 32 \begin{itemize} |
100 \cite{Tappert77} | 33 \item embodied stream predicates |
34 \item \(\phi\)-space | |
35 \item \(\phi\)-scan | |
36 \end{itemize} | |
37 \end{itemize} | |
38 \section{News} | |
39 \label{sec-3} | |
40 | |
41 Experimental results: | |
42 | |
43 \begin{itemize} | |
44 \item \(\phi\)-space actually works very well for the worm! | |
45 \item self organizing touch map | |
46 \end{itemize} | |
47 | |
48 \section{Contributions} | |
49 \label{sec-4} | |
50 \begin{itemize} | |
51 \item Built \texttt{CORTEX}, a comprehensive platform for embodied AI | |
52 experiments. Has many new features lacking in other systems, such | |
53 as sound. Easy to model/create new creatures. | |
54 \item created a novel concept for action recognition by using artificial | |
55 imagination. | |
56 \end{itemize} |