|
rlm@422
|
1
|
|
rlm@425
|
2 \section{Artificial Imagination}
|
|
rlm@425
|
3 \label{sec-1}
|
|
rlm@422
|
4
|
|
rlm@425
|
5 Imagine watching a video of someone skateboarding. When you watch
|
|
rlm@425
|
6 the video, you can imagine yourself skateboarding, and your
|
|
rlm@425
|
7 knowledge of the human body and its dynamics guides your
|
|
rlm@425
|
8 interpretation of the scene. For example, even if the skateboarder
|
|
rlm@425
|
9 is partially occluded, you can infer the positions of his arms and
|
|
rlm@425
|
10 body from your own knowledge of how your body would be positioned if
|
|
rlm@425
|
11 you were skateboarding. If the skateboarder suffers an accident, you
|
|
rlm@425
|
12 wince in sympathy, imagining the pain your own body would experience
|
|
rlm@425
|
13 if it were in the same situation. This empathy with other people
|
|
rlm@425
|
14 guides our understanding of whatever they are doing because it is a
|
|
rlm@425
|
15 powerful constraint on what is probable and possible. In order to
|
|
rlm@425
|
16 make use of this powerful empathy constraint, I need a system that
|
|
rlm@425
|
17 can generate and make sense of sensory data from the many different
|
|
rlm@425
|
18 senses that humans possess. The two key proprieties of such a system
|
|
rlm@425
|
19 are \emph{embodiment} and \emph{imagination}.
|
|
rlm@422
|
20
|
|
rlm@425
|
21 \subsection{What is imagination?}
|
|
rlm@425
|
22 \label{sec-1-1}
|
|
rlm@425
|
23
|
|
rlm@425
|
24 One kind of imagination is \emph{sympathetic} imagination: you imagine
|
|
rlm@425
|
25 yourself in the position of something/someone you are
|
|
rlm@425
|
26 observing. This type of imagination comes into play when you follow
|
|
rlm@425
|
27 along visually when watching someone perform actions, or when you
|
|
rlm@425
|
28 sympathetically grimace when someone hurts themselves. This type of
|
|
rlm@425
|
29 imagination uses the constraints you have learned about your own
|
|
rlm@425
|
30 body to highly constrain the possibilities in whatever you are
|
|
rlm@425
|
31 seeing. It uses all your senses to including your senses of touch,
|
|
rlm@425
|
32 proprioception, etc. Humans are flexible when it comes to "putting
|
|
rlm@425
|
33 themselves in another's shoes," and can sympathetically understand
|
|
rlm@425
|
34 not only other humans, but entities ranging from animals to cartoon
|
|
rlm@425
|
35 characters to \href{http://www.youtube.com/watch?v=0jz4HcwTQmU}{single dots} on a screen!
|
|
rlm@425
|
36
|
|
rlm@425
|
37
|
|
rlm@425
|
38 \begin{figure}[htb]
|
|
rlm@425
|
39 \centering
|
|
rlm@425
|
40 \includegraphics[width=5cm]{./images/cat-drinking.jpg}
|
|
rlm@425
|
41 \caption{A cat drinking some water. Identifying this action is beyond the state of the art for computers.}
|
|
rlm@425
|
42 \end{figure}
|
|
rlm@425
|
43
|
|
rlm@425
|
44
|
|
rlm@425
|
45 This is a basic test for the vision system. It only tests the
|
|
rlm@425
|
46 vision-pipeline and does not deal with loading eyes from a blender
|
|
rlm@425
|
47 file. The code creates two videos of the same rotating cube from
|
|
rlm@425
|
48 different angles.
|
|
rlm@425
|
49
|
|
rlm@425
|
50
|
|
rlm@425
|
51 \begin{clojurecode}
|
|
rlm@425
|
52 (in-ns 'cortex.test.vision)
|
|
rlm@425
|
53
|
|
rlm@425
|
54 (defn test-pipeline
|
|
rlm@425
|
55 "Testing vision:
|
|
rlm@425
|
56 Tests the vision system by creating two views of the same rotating
|
|
rlm@425
|
57 object from different angles and displaying both of those views in
|
|
rlm@425
|
58 JFrames.
|
|
rlm@425
|
59
|
|
rlm@425
|
60 You should see a rotating cube, and two windows,
|
|
rlm@425
|
61 each displaying a different view of the cube."
|
|
rlm@425
|
62 ([] (test-pipeline false))
|
|
rlm@425
|
63 ([record?]
|
|
rlm@425
|
64 (let [candy
|
|
rlm@425
|
65 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
|
|
rlm@425
|
66 (world
|
|
rlm@425
|
67 (doto (Node.)
|
|
rlm@425
|
68 (.attachChild candy))
|
|
rlm@425
|
69 {}
|
|
rlm@425
|
70 (fn [world]
|
|
rlm@425
|
71 (let [cam (.clone (.getCamera world))
|
|
rlm@425
|
72 width (.getWidth cam)
|
|
rlm@425
|
73 height (.getHeight cam)]
|
|
rlm@425
|
74 (add-camera! world cam
|
|
rlm@425
|
75 (comp
|
|
rlm@425
|
76 (view-image
|
|
rlm@425
|
77 (if record?
|
|
rlm@425
|
78 (File. "/home/r/proj/cortex/render/vision/1")))
|
|
rlm@425
|
79 BufferedImage!))
|
|
rlm@425
|
80 (add-camera! world
|
|
rlm@425
|
81 (doto (.clone cam)
|
|
rlm@425
|
82 (.setLocation (Vector3f. -10 0 0))
|
|
rlm@425
|
83 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
|
|
rlm@425
|
84 (comp
|
|
rlm@425
|
85 (view-image
|
|
rlm@425
|
86 (if record?
|
|
rlm@425
|
87 (File. "/home/r/proj/cortex/render/vision/2")))
|
|
rlm@425
|
88 BufferedImage!))
|
|
rlm@425
|
89 (let [timer (IsoTimer. 60)]
|
|
rlm@425
|
90 (.setTimer world timer)
|
|
rlm@425
|
91 (display-dilated-time world timer))
|
|
rlm@425
|
92 ;; This is here to restore the main view
|
|
rlm@425
|
93 ;; after the other views have completed processing
|
|
rlm@425
|
94 (add-camera! world (.getCamera world) no-op)))
|
|
rlm@425
|
95 (fn [world tpf]
|
|
rlm@425
|
96 (.rotate candy (* tpf 0.2) 0 0))))))
|
|
rlm@425
|
97 \end{clojurecode}
|
