#+title: =CORTEX=

 #+author: Robert McIntyre

 #+email: rlm@mit.edu

 #+description: Using embodied AI to facilitate Artificial Imagination.

 #+keywords: AI, clojure, embodiment

 

 * Artificial Imagination

 

   Imagine watching a video of someone skateboarding. When you watch

   the video, you can imagine yourself skateboarding, and your

   knowledge of the human body and its dynamics guides your

   interpretation of the scene. For example, even if the skateboarder

   is partially occluded, you can infer the positions of his arms and

   body from your own knowledge of how your body would be positioned if

   you were skateboarding. If the skateboarder suffers an accident, you

   wince in sympathy, imagining the pain your own body would experience

   if it were in the same situation. This empathy with other people

   guides our understanding of whatever they are doing because it is a

   powerful constraint on what is probable and possible. In order to

   make use of this powerful empathy constraint, I need a system that

   can generate and make sense of sensory data from the many different

   senses that humans possess. The two key proprieties of such a system

   are /embodiment/ and /imagination/.

 

 ** What is imagination?

 

    One kind of imagination is /sympathetic/ imagination: you imagine

    yourself in the position of something/someone you are

    observing. This type of imagination comes into play when you follow

    along visually when watching someone perform actions, or when you

    sympathetically grimace when someone hurts themselves. This type of

    imagination uses the constraints you have learned about your own

    body to highly constrain the possibilities in whatever you are

    seeing. It uses all your senses to including your senses of touch,

    proprioception, etc. Humans are flexible when it comes to "putting

    themselves in another's shoes," and can sympathetically understand

    not only other humans, but entities ranging from animals to cartoon

    characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen!

 

 

    #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.

    #+ATTR_LaTeX: :width 5cm

    [[./images/cat-drinking.jpg]]

 

 

 #+begin_listing clojure

 \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.}

 #+name: test-1

 #+begin_src clojure

 (defn test-pipeline

   "Testing vision:

    Tests the vision system by creating two views of the same rotating

    object from different angles and displaying both of those views in

    JFrames.

 

    You should see a rotating cube, and two windows,

    each displaying a different view of the cube."

   ([] (test-pipeline false))

   ([record?]

      (let [candy

            (box 1 1 1 :physical? false :color ColorRGBA/Blue)]

        (world

         (doto (Node.)

           (.attachChild candy))

         {}

         (fn [world]

           (let [cam (.clone (.getCamera world))

                 width (.getWidth cam)

                 height (.getHeight cam)]

             (add-camera! world cam 

                          (comp

                           (view-image

                            (if record?

                              (File. "/home/r/proj/cortex/render/vision/1")))

                           BufferedImage!))

             (add-camera! world

                          (doto (.clone cam)

                            (.setLocation (Vector3f. -10 0 0))

                            (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))

                          (comp

                           (view-image

                            (if record?

                              (File. "/home/r/proj/cortex/render/vision/2")))

                           BufferedImage!))

             (let [timer (IsoTimer. 60)]

               (.setTimer world timer)

               (display-dilated-time world timer))

             ;; This is here to restore the main view

             ;; after the other views have completed processing

             (add-camera! world (.getCamera world) no-op)))

         (fn [world tpf]

           (.rotate candy (* tpf 0.2) 0 0))))))

 #+end_src

 #+end_listing

 

 - This is test1 \cite{Tappert77}.

 

 \cite{Tappert77}

 lol

 \cite{Tappert77}