Mercurial > cortex

annotate thesis/rlm-cortex-meng.tex @ 427:c670d23003de

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
rename.
author Robert McIntyre <rlm@mit.edu>
date Fri, 21 Mar 2014 15:36:37 -0400
parents
children d53a31969a51
rev   line source
rlm@427 1
rlm@427 2 \section{Artificial Imagination}
rlm@427 3 \label{sec-1}
rlm@427 4
rlm@427 5 Imagine watching a video of someone skateboarding. When you watch
rlm@427 6 the video, you can imagine yourself skateboarding, and your
rlm@427 7 knowledge of the human body and its dynamics guides your
rlm@427 8 interpretation of the scene. For example, even if the skateboarder
rlm@427 9 is partially occluded, you can infer the positions of his arms and
rlm@427 10 body from your own knowledge of how your body would be positioned if
rlm@427 11 you were skateboarding. If the skateboarder suffers an accident, you
rlm@427 12 wince in sympathy, imagining the pain your own body would experience
rlm@427 13 if it were in the same situation. This empathy with other people
rlm@427 14 guides our understanding of whatever they are doing because it is a
rlm@427 15 powerful constraint on what is probable and possible. In order to
rlm@427 16 make use of this powerful empathy constraint, I need a system that
rlm@427 17 can generate and make sense of sensory data from the many different
rlm@427 18 senses that humans possess. The two key proprieties of such a system
rlm@427 19 are \emph{embodiment} and \emph{imagination}.
rlm@427 20
rlm@427 21 \subsection{What is imagination?}
rlm@427 22 \label{sec-1-1}
rlm@427 23
rlm@427 24 One kind of imagination is \emph{sympathetic} imagination: you imagine
rlm@427 25 yourself in the position of something/someone you are
rlm@427 26 observing. This type of imagination comes into play when you follow
rlm@427 27 along visually when watching someone perform actions, or when you
rlm@427 28 sympathetically grimace when someone hurts themselves. This type of
rlm@427 29 imagination uses the constraints you have learned about your own
rlm@427 30 body to highly constrain the possibilities in whatever you are
rlm@427 31 seeing. It uses all your senses to including your senses of touch,
rlm@427 32 proprioception, etc. Humans are flexible when it comes to "putting
rlm@427 33 themselves in another's shoes," and can sympathetically understand
rlm@427 34 not only other humans, but entities ranging from animals to cartoon
rlm@427 35 characters to \href{http://www.youtube.com/watch?v=0jz4HcwTQmU}{single dots} on a screen!
rlm@427 36
rlm@427 37
rlm@427 38 \begin{figure}[htb]
rlm@427 39 \centering
rlm@427 40 \includegraphics[width=5cm]{./images/cat-drinking.jpg}
rlm@427 41 \caption{A cat drinking some water. Identifying this action is beyond the state of the art for computers.}
rlm@427 42 \end{figure}
rlm@427 43
rlm@427 44
rlm@427 45 This is a basic test for the vision system. It only tests the
rlm@427 46 vision-pipeline and does not deal with loading eyes from a blender
rlm@427 47 file. The code creates two videos of the same rotating cube from
rlm@427 48 different angles.
rlm@427 49
rlm@427 50
rlm@427 51 \begin{clojurecode}
rlm@427 52 (in-ns 'cortex.test.vision)
rlm@427 53
rlm@427 54 (defn test-pipeline
rlm@427 55 "Testing vision:
rlm@427 56 Tests the vision system by creating two views of the same rotating
rlm@427 57 object from different angles and displaying both of those views in
rlm@427 58 JFrames.
rlm@427 59
rlm@427 60 You should see a rotating cube, and two windows,
rlm@427 61 each displaying a different view of the cube."
rlm@427 62 ([] (test-pipeline false))
rlm@427 63 ([record?]
rlm@427 64 (let [candy
rlm@427 65 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@427 66 (world
rlm@427 67 (doto (Node.)
rlm@427 68 (.attachChild candy))
rlm@427 69 {}
rlm@427 70 (fn [world]
rlm@427 71 (let [cam (.clone (.getCamera world))
rlm@427 72 width (.getWidth cam)
rlm@427 73 height (.getHeight cam)]
rlm@427 74 (add-camera! world cam
rlm@427 75 (comp
rlm@427 76 (view-image
rlm@427 77 (if record?
rlm@427 78 (File. "/home/r/proj/cortex/render/vision/1")))
rlm@427 79 BufferedImage!))
rlm@427 80 (add-camera! world
rlm@427 81 (doto (.clone cam)
rlm@427 82 (.setLocation (Vector3f. -10 0 0))
rlm@427 83 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@427 84 (comp
rlm@427 85 (view-image
rlm@427 86 (if record?
rlm@427 87 (File. "/home/r/proj/cortex/render/vision/2")))
rlm@427 88 BufferedImage!))
rlm@427 89 (let [timer (IsoTimer. 60)]
rlm@427 90 (.setTimer world timer)
rlm@427 91 (display-dilated-time world timer))
rlm@427 92 ;; This is here to restore the main view
rlm@427 93 ;; after the other views have completed processing
rlm@427 94 (add-camera! world (.getCamera world) no-op)))
rlm@427 95 (fn [world tpf]
rlm@427 96 (.rotate candy (* tpf 0.2) 0 0))))))
rlm@427 97 \end{clojurecode}
rlm@427 98
rlm@427 99
rlm@427 100 \begin{itemize}
rlm@427 101 \item This is test1 \cite{Tappert77}.
rlm@427 102 \end{itemize}