Mercurial > cortex
diff thesis/cortex.tex @ 432:1e5ea711857d
abstract first draft.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Sun, 23 Mar 2014 16:33:01 -0400 |
parents | 7410f0d8011c |
children |
line wrap: on
line diff
1.1 --- a/thesis/cortex.tex Sat Mar 22 23:31:07 2014 -0400 1.2 +++ b/thesis/cortex.tex Sun Mar 23 16:33:01 2014 -0400 1.3 @@ -1,100 +1,56 @@ 1.4 1.5 -\section{Artificial Imagination} 1.6 +\section{Vision} 1.7 \label{sec-1} 1.8 1.9 -Imagine watching a video of someone skateboarding. When you watch 1.10 -the video, you can imagine yourself skateboarding, and your 1.11 -knowledge of the human body and its dynamics guides your 1.12 -interpretation of the scene. For example, even if the skateboarder 1.13 -is partially occluded, you can infer the positions of his arms and 1.14 -body from your own knowledge of how your body would be positioned if 1.15 -you were skateboarding. If the skateboarder suffers an accident, you 1.16 -wince in sympathy, imagining the pain your own body would experience 1.17 -if it were in the same situation. This empathy with other people 1.18 -guides our understanding of whatever they are doing because it is a 1.19 -powerful constraint on what is probable and possible. In order to 1.20 -make use of this powerful empathy constraint, I need a system that 1.21 -can generate and make sense of sensory data from the many different 1.22 -senses that humans possess. The two key proprieties of such a system 1.23 -are \emph{embodiment} and \emph{imagination}. 1.24 +System for understanding what the actors in a video are doing -- 1.25 +Action Recognition. 1.26 1.27 -\subsection{What is imagination?} 1.28 -\label{sec-1-1} 1.29 - 1.30 -One kind of imagination is \emph{sympathetic} imagination: you imagine 1.31 -yourself in the position of something/someone you are 1.32 -observing. This type of imagination comes into play when you follow 1.33 -along visually when watching someone perform actions, or when you 1.34 -sympathetically grimace when someone hurts themselves. This type of 1.35 -imagination uses the constraints you have learned about your own 1.36 -body to highly constrain the possibilities in whatever you are 1.37 -seeing. It uses all your senses to including your senses of touch, 1.38 -proprioception, etc. Humans are flexible when it comes to "putting 1.39 -themselves in another's shoes," and can sympathetically understand 1.40 -not only other humans, but entities ranging from animals to cartoon 1.41 -characters to \href{http://www.youtube.com/watch?v=0jz4HcwTQmU}{single dots} on a screen! 1.42 - 1.43 - 1.44 -\begin{figure}[htb] 1.45 -\centering 1.46 -\includegraphics[width=5cm]{./images/cat-drinking.jpg} 1.47 -\caption{A cat drinking some water. Identifying this action is beyond the state of the art for computers.} 1.48 -\end{figure} 1.49 - 1.50 - 1.51 -\begin{listing} 1.52 -\caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} 1.53 -\begin{clojurecode} 1.54 -(defn test-pipeline 1.55 - "Testing vision: 1.56 - Tests the vision system by creating two views of the same rotating 1.57 - object from different angles and displaying both of those views in 1.58 - JFrames. 1.59 - 1.60 - You should see a rotating cube, and two windows, 1.61 - each displaying a different view of the cube." 1.62 - ([] (test-pipeline false)) 1.63 - ([record?] 1.64 - (let [candy 1.65 - (box 1 1 1 :physical? false :color ColorRGBA/Blue)] 1.66 - (world 1.67 - (doto (Node.) 1.68 - (.attachChild candy)) 1.69 - {} 1.70 - (fn [world] 1.71 - (let [cam (.clone (.getCamera world)) 1.72 - width (.getWidth cam) 1.73 - height (.getHeight cam)] 1.74 - (add-camera! world cam 1.75 - (comp 1.76 - (view-image 1.77 - (if record? 1.78 - (File. "/home/r/proj/cortex/render/vision/1"))) 1.79 - BufferedImage!)) 1.80 - (add-camera! world 1.81 - (doto (.clone cam) 1.82 - (.setLocation (Vector3f. -10 0 0)) 1.83 - (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) 1.84 - (comp 1.85 - (view-image 1.86 - (if record? 1.87 - (File. "/home/r/proj/cortex/render/vision/2"))) 1.88 - BufferedImage!)) 1.89 - (let [timer (IsoTimer. 60)] 1.90 - (.setTimer world timer) 1.91 - (display-dilated-time world timer)) 1.92 - ;; This is here to restore the main view 1.93 - ;; after the other views have completed processing 1.94 - (add-camera! world (.getCamera world) no-op))) 1.95 - (fn [world tpf] 1.96 - (.rotate candy (* tpf 0.2) 0 0)))))) 1.97 -\end{clojurecode} 1.98 -\end{listing} 1.99 +Separate action recognition into three components: 1.100 1.101 \begin{itemize} 1.102 -\item This is test1 \cite{Tappert77}. 1.103 +\item free play 1.104 +\item embodied action predicates 1.105 +\item model alignment 1.106 +\item sensory imagination 1.107 +\end{itemize} 1.108 +\section{Steps} 1.109 +\label{sec-2} 1.110 + 1.111 +\begin{itemize} 1.112 +\item Build cortex, a simulated environment for sensate AI 1.113 +\begin{itemize} 1.114 +\item solid bodies w/ joints 1.115 +\item vision 1.116 +\item touch 1.117 +\item vision 1.118 +\item hearing 1.119 +\item proprioception 1.120 +\item muscle contraction 1.121 \end{itemize} 1.122 1.123 -\cite{Tappert77} 1.124 -lol 1.125 -\cite{Tappert77} 1.126 +\item Build experimental framework for worm-actions 1.127 +\begin{itemize} 1.128 +\item embodied stream predicates 1.129 +\item \(\phi\)-space 1.130 +\item \(\phi\)-scan 1.131 +\end{itemize} 1.132 +\end{itemize} 1.133 +\section{News} 1.134 +\label{sec-3} 1.135 + 1.136 +Experimental results: 1.137 + 1.138 +\begin{itemize} 1.139 +\item \(\phi\)-space actually works very well for the worm! 1.140 +\item self organizing touch map 1.141 +\end{itemize} 1.142 + 1.143 +\section{Contributions} 1.144 +\label{sec-4} 1.145 +\begin{itemize} 1.146 +\item Built \texttt{CORTEX}, a comprehensive platform for embodied AI 1.147 +experiments. Has many new features lacking in other systems, such 1.148 +as sound. Easy to model/create new creatures. 1.149 +\item created a novel concept for action recognition by using artificial 1.150 +imagination. 1.151 +\end{itemize}