# HG changeset patch # User Robert McIntyre # Date 1395606781 14400 # Node ID 1e5ea711857df93d3d1a5536d7cfd0fd2682ea1d # Parent 7410f0d8011c7cece5dcac93d706be7b51603f2e abstract first draft. diff -r 7410f0d8011c -r 1e5ea711857d thesis/abstract.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/thesis/abstract.org Sun Mar 23 16:33:01 2014 -0400 @@ -0,0 +1,31 @@ +Here I explore the design and capabilities of my system (called +=CORTEX=) which enables experiments in /embodied artificial +intelligence/ -- that is, AI which uses a physical simulation of +reality accompanied by a simulated body to solve problems. + +In the first half of the thesis I describe the construction of +=CORTEX= and the rationale behind my architecture choices. =CORTEX= is +a complete platform for embodied AI research. It provides multiple +senses for simulated creatures, including vision, touch, +proprioception, muscle tension, and hearing. Each of these senses +provides a wealth of parameters that are biologically +inspired. =CORTEX= is able to simulate any number of creatures and +senses, and provides facilities for easily modeling and creating new +creatures. As a research platform it is more complete than any other +system currently available. + +In the second half of the thesis I develop a computational model of +empathy, using =CORTEX= as a base. Empathy in this context is the +ability to observe another creature and infer what sorts of sensations +that creature is feeling. My empathy algorithm involves multiple +phases. First is free-play, where the creature moves around and gains +sensory experience. From this experience I construct a representation +of the creature's sensory state space, which I call \phi-space. Using +\phi-space, I construct an efficient function for enriching the +limited data that comes from observing another creature with a full +compliment of imagined sensory data based on previous experience. I +can then use the imagined sensory data to recognize what the observed +creature is doing and feeling, using straightforward embodied action +predicates. This is all demonstrated with using a simple worm-like +creature, recognizing worm-actions in video. + diff -r 7410f0d8011c -r 1e5ea711857d thesis/abstract.tex --- a/thesis/abstract.tex Sat Mar 22 23:31:07 2014 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,22 +0,0 @@ -% $Log: abstract.tex,v $ -% Revision 1.1 93/05/14 14:56:25 starflt -% Initial revision -% -% Revision 1.1 90/05/04 10:41:01 lwvanels -% Initial revision -% -% -%% The text of your abstract and nothing else (other than comments) goes here. -%% It will be single-spaced and the rest of the text that is supposed to go on -%% the abstract page will be generated by the abstractpage environment. This -%% file should be \input (not \include 'd) from cover.tex. -In this thesis, I designed and implemented a compiler which performs -optimizations that reduce the number of low-level floating point operations -necessary for a specific task; this involves the optimization of chains of -floating point operations as well as the implementation of a ``fixed'' point -data type that allows some floating point operations to simulated with integer -arithmetic. The source language of the compiler is a subset of C, and the -destination language is assembly language for a micro-floating point CPU. An -instruction-level simulator of the CPU was written to allow testing of the -code. A series of test pieces of codes was compiled, both with and without -optimization, to determine how effective these optimizations were. diff -r 7410f0d8011c -r 1e5ea711857d thesis/cortex.org --- a/thesis/cortex.org Sat Mar 22 23:31:07 2014 -0400 +++ b/thesis/cortex.org Sun Mar 23 16:33:01 2014 -0400 @@ -4,97 +4,52 @@ #+description: Using embodied AI to facilitate Artificial Imagination. #+keywords: AI, clojure, embodiment -* Artificial Imagination +* Vision - Imagine watching a video of someone skateboarding. When you watch - the video, you can imagine yourself skateboarding, and your - knowledge of the human body and its dynamics guides your - interpretation of the scene. For example, even if the skateboarder - is partially occluded, you can infer the positions of his arms and - body from your own knowledge of how your body would be positioned if - you were skateboarding. If the skateboarder suffers an accident, you - wince in sympathy, imagining the pain your own body would experience - if it were in the same situation. This empathy with other people - guides our understanding of whatever they are doing because it is a - powerful constraint on what is probable and possible. In order to - make use of this powerful empathy constraint, I need a system that - can generate and make sense of sensory data from the many different - senses that humans possess. The two key proprieties of such a system - are /embodiment/ and /imagination/. + System for understanding what the actors in a video are doing -- + Action Recognition. + + Separate action recognition into three components: -** What is imagination? + - free play + - embodied action predicates + - model alignment + - sensory imagination - One kind of imagination is /sympathetic/ imagination: you imagine - yourself in the position of something/someone you are - observing. This type of imagination comes into play when you follow - along visually when watching someone perform actions, or when you - sympathetically grimace when someone hurts themselves. This type of - imagination uses the constraints you have learned about your own - body to highly constrain the possibilities in whatever you are - seeing. It uses all your senses to including your senses of touch, - proprioception, etc. Humans are flexible when it comes to "putting - themselves in another's shoes," and can sympathetically understand - not only other humans, but entities ranging from animals to cartoon - characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen! +* Steps + + - Build cortex, a simulated environment for sensate AI + - solid bodies w/ joints + - vision + - touch + - vision + - hearing + - proprioception + - muscle contraction + - Build experimental framework for worm-actions + - embodied stream predicates + - \phi-space + - \phi-scan - #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers. - #+ATTR_LaTeX: :width 5cm - [[./images/cat-drinking.jpg]] +* News + + Experimental results: + - \phi-space actually works very well for the worm! + - self organizing touch map -#+begin_listing -\caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} -#+name: test-1 -#+begin_src clojure -(defn test-pipeline - "Testing vision: - Tests the vision system by creating two views of the same rotating - object from different angles and displaying both of those views in - JFrames. - You should see a rotating cube, and two windows, - each displaying a different view of the cube." - ([] (test-pipeline false)) - ([record?] - (let [candy - (box 1 1 1 :physical? false :color ColorRGBA/Blue)] - (world - (doto (Node.) - (.attachChild candy)) - {} - (fn [world] - (let [cam (.clone (.getCamera world)) - width (.getWidth cam) - height (.getHeight cam)] - (add-camera! world cam - (comp - (view-image - (if record? - (File. "/home/r/proj/cortex/render/vision/1"))) - BufferedImage!)) - (add-camera! world - (doto (.clone cam) - (.setLocation (Vector3f. -10 0 0)) - (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) - (comp - (view-image - (if record? - (File. "/home/r/proj/cortex/render/vision/2"))) - BufferedImage!)) - (let [timer (IsoTimer. 60)] - (.setTimer world timer) - (display-dilated-time world timer)) - ;; This is here to restore the main view - ;; after the other views have completed processing - (add-camera! world (.getCamera world) no-op))) - (fn [world tpf] - (.rotate candy (* tpf 0.2) 0 0)))))) -#+end_src -#+end_listing +* Contributions + - Built =CORTEX=, a comprehensive platform for embodied AI + experiments. Has many new features lacking in other systems, such + as sound. Easy to model/create new creatures. + - created a novel concept for action recognition by using artificial + imagination. -- This is test1 \cite{Tappert77}. -\cite{Tappert77} -lol -\cite{Tappert77} \ No newline at end of file + + + + + diff -r 7410f0d8011c -r 1e5ea711857d thesis/cortex.tex --- a/thesis/cortex.tex Sat Mar 22 23:31:07 2014 -0400 +++ b/thesis/cortex.tex Sun Mar 23 16:33:01 2014 -0400 @@ -1,100 +1,56 @@ -\section{Artificial Imagination} +\section{Vision} \label{sec-1} -Imagine watching a video of someone skateboarding. When you watch -the video, you can imagine yourself skateboarding, and your -knowledge of the human body and its dynamics guides your -interpretation of the scene. For example, even if the skateboarder -is partially occluded, you can infer the positions of his arms and -body from your own knowledge of how your body would be positioned if -you were skateboarding. If the skateboarder suffers an accident, you -wince in sympathy, imagining the pain your own body would experience -if it were in the same situation. This empathy with other people -guides our understanding of whatever they are doing because it is a -powerful constraint on what is probable and possible. In order to -make use of this powerful empathy constraint, I need a system that -can generate and make sense of sensory data from the many different -senses that humans possess. The two key proprieties of such a system -are \emph{embodiment} and \emph{imagination}. +System for understanding what the actors in a video are doing -- +Action Recognition. -\subsection{What is imagination?} -\label{sec-1-1} - -One kind of imagination is \emph{sympathetic} imagination: you imagine -yourself in the position of something/someone you are -observing. This type of imagination comes into play when you follow -along visually when watching someone perform actions, or when you -sympathetically grimace when someone hurts themselves. This type of -imagination uses the constraints you have learned about your own -body to highly constrain the possibilities in whatever you are -seeing. It uses all your senses to including your senses of touch, -proprioception, etc. Humans are flexible when it comes to "putting -themselves in another's shoes," and can sympathetically understand -not only other humans, but entities ranging from animals to cartoon -characters to \href{http://www.youtube.com/watch?v=0jz4HcwTQmU}{single dots} on a screen! - - -\begin{figure}[htb] -\centering -\includegraphics[width=5cm]{./images/cat-drinking.jpg} -\caption{A cat drinking some water. Identifying this action is beyond the state of the art for computers.} -\end{figure} - - -\begin{listing} -\caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} -\begin{clojurecode} -(defn test-pipeline - "Testing vision: - Tests the vision system by creating two views of the same rotating - object from different angles and displaying both of those views in - JFrames. - - You should see a rotating cube, and two windows, - each displaying a different view of the cube." - ([] (test-pipeline false)) - ([record?] - (let [candy - (box 1 1 1 :physical? false :color ColorRGBA/Blue)] - (world - (doto (Node.) - (.attachChild candy)) - {} - (fn [world] - (let [cam (.clone (.getCamera world)) - width (.getWidth cam) - height (.getHeight cam)] - (add-camera! world cam - (comp - (view-image - (if record? - (File. "/home/r/proj/cortex/render/vision/1"))) - BufferedImage!)) - (add-camera! world - (doto (.clone cam) - (.setLocation (Vector3f. -10 0 0)) - (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) - (comp - (view-image - (if record? - (File. "/home/r/proj/cortex/render/vision/2"))) - BufferedImage!)) - (let [timer (IsoTimer. 60)] - (.setTimer world timer) - (display-dilated-time world timer)) - ;; This is here to restore the main view - ;; after the other views have completed processing - (add-camera! world (.getCamera world) no-op))) - (fn [world tpf] - (.rotate candy (* tpf 0.2) 0 0)))))) -\end{clojurecode} -\end{listing} +Separate action recognition into three components: \begin{itemize} -\item This is test1 \cite{Tappert77}. +\item free play +\item embodied action predicates +\item model alignment +\item sensory imagination +\end{itemize} +\section{Steps} +\label{sec-2} + +\begin{itemize} +\item Build cortex, a simulated environment for sensate AI +\begin{itemize} +\item solid bodies w/ joints +\item vision +\item touch +\item vision +\item hearing +\item proprioception +\item muscle contraction \end{itemize} -\cite{Tappert77} -lol -\cite{Tappert77} +\item Build experimental framework for worm-actions +\begin{itemize} +\item embodied stream predicates +\item \(\phi\)-space +\item \(\phi\)-scan +\end{itemize} +\end{itemize} +\section{News} +\label{sec-3} + +Experimental results: + +\begin{itemize} +\item \(\phi\)-space actually works very well for the worm! +\item self organizing touch map +\end{itemize} + +\section{Contributions} +\label{sec-4} +\begin{itemize} +\item Built \texttt{CORTEX}, a comprehensive platform for embodied AI +experiments. Has many new features lacking in other systems, such +as sound. Easy to model/create new creatures. +\item created a novel concept for action recognition by using artificial +imagination. +\end{itemize} diff -r 7410f0d8011c -r 1e5ea711857d thesis/garbage_cortex.org --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/thesis/garbage_cortex.org Sun Mar 23 16:33:01 2014 -0400 @@ -0,0 +1,100 @@ +#+title: =CORTEX= +#+author: Robert McIntyre +#+email: rlm@mit.edu +#+description: Using embodied AI to facilitate Artificial Imagination. +#+keywords: AI, clojure, embodiment + +* Artificial Imagination + + Imagine watching a video of someone skateboarding. When you watch + the video, you can imagine yourself skateboarding, and your + knowledge of the human body and its dynamics guides your + interpretation of the scene. For example, even if the skateboarder + is partially occluded, you can infer the positions of his arms and + body from your own knowledge of how your body would be positioned if + you were skateboarding. If the skateboarder suffers an accident, you + wince in sympathy, imagining the pain your own body would experience + if it were in the same situation. This empathy with other people + guides our understanding of whatever they are doing because it is a + powerful constraint on what is probable and possible. In order to + make use of this powerful empathy constraint, I need a system that + can generate and make sense of sensory data from the many different + senses that humans possess. The two key proprieties of such a system + are /embodiment/ and /imagination/. + +** What is imagination? + + One kind of imagination is /sympathetic/ imagination: you imagine + yourself in the position of something/someone you are + observing. This type of imagination comes into play when you follow + along visually when watching someone perform actions, or when you + sympathetically grimace when someone hurts themselves. This type of + imagination uses the constraints you have learned about your own + body to highly constrain the possibilities in whatever you are + seeing. It uses all your senses to including your senses of touch, + proprioception, etc. Humans are flexible when it comes to "putting + themselves in another's shoes," and can sympathetically understand + not only other humans, but entities ranging from animals to cartoon + characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen! + + + #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers. + #+ATTR_LaTeX: :width 5cm + [[./images/cat-drinking.jpg]] + + +#+begin_listing clojure +\caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} +#+name: test-1 +#+begin_src clojure +(defn test-pipeline + "Testing vision: + Tests the vision system by creating two views of the same rotating + object from different angles and displaying both of those views in + JFrames. + + You should see a rotating cube, and two windows, + each displaying a different view of the cube." + ([] (test-pipeline false)) + ([record?] + (let [candy + (box 1 1 1 :physical? false :color ColorRGBA/Blue)] + (world + (doto (Node.) + (.attachChild candy)) + {} + (fn [world] + (let [cam (.clone (.getCamera world)) + width (.getWidth cam) + height (.getHeight cam)] + (add-camera! world cam + (comp + (view-image + (if record? + (File. "/home/r/proj/cortex/render/vision/1"))) + BufferedImage!)) + (add-camera! world + (doto (.clone cam) + (.setLocation (Vector3f. -10 0 0)) + (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) + (comp + (view-image + (if record? + (File. "/home/r/proj/cortex/render/vision/2"))) + BufferedImage!)) + (let [timer (IsoTimer. 60)] + (.setTimer world timer) + (display-dilated-time world timer)) + ;; This is here to restore the main view + ;; after the other views have completed processing + (add-camera! world (.getCamera world) no-op))) + (fn [world tpf] + (.rotate candy (* tpf 0.2) 0 0)))))) +#+end_src +#+end_listing + +- This is test1 \cite{Tappert77}. + +\cite{Tappert77} +lol +\cite{Tappert77} \ No newline at end of file diff -r 7410f0d8011c -r 1e5ea711857d thesis/rlm-cortex-meng.tex --- a/thesis/rlm-cortex-meng.tex Sat Mar 22 23:31:07 2014 -0400 +++ b/thesis/rlm-cortex-meng.tex Sun Mar 23 16:33:01 2014 -0400 @@ -41,7 +41,6 @@ \usepackage{wasysym} \usepackage{amssymb} \usepackage{hyperref} -%\usepackage{natbib} \usepackage{libertine} \usepackage{inconsolata} diff -r 7410f0d8011c -r 1e5ea711857d thesis/weave-thesis.sh --- a/thesis/weave-thesis.sh Sat Mar 22 23:31:07 2014 -0400 +++ b/thesis/weave-thesis.sh Sun Mar 23 16:33:01 2014 -0400 @@ -7,9 +7,13 @@ --batch \ --eval " (progn - (find-file \"$1.org\") + (find-file \"cortex.org\") + (org-latex-export-to-latex nil nil nil t nil)) \ +(progn + (find-file \"abstract.org\") (org-latex-export-to-latex nil nil nil t nil))" \ \ 2>&1 -rm $1.tex~ \ No newline at end of file +rm cortex.tex~ +rm abstract.tex~