rlm@432: #+title: =CORTEX= rlm@432: #+author: Robert McIntyre rlm@432: #+email: rlm@mit.edu rlm@432: #+description: Using embodied AI to facilitate Artificial Imagination. rlm@432: #+keywords: AI, clojure, embodiment rlm@432: rlm@432: * Artificial Imagination rlm@432: rlm@432: Imagine watching a video of someone skateboarding. When you watch rlm@432: the video, you can imagine yourself skateboarding, and your rlm@432: knowledge of the human body and its dynamics guides your rlm@432: interpretation of the scene. For example, even if the skateboarder rlm@432: is partially occluded, you can infer the positions of his arms and rlm@432: body from your own knowledge of how your body would be positioned if rlm@432: you were skateboarding. If the skateboarder suffers an accident, you rlm@432: wince in sympathy, imagining the pain your own body would experience rlm@432: if it were in the same situation. This empathy with other people rlm@432: guides our understanding of whatever they are doing because it is a rlm@432: powerful constraint on what is probable and possible. In order to rlm@432: make use of this powerful empathy constraint, I need a system that rlm@432: can generate and make sense of sensory data from the many different rlm@432: senses that humans possess. The two key proprieties of such a system rlm@432: are /embodiment/ and /imagination/. rlm@432: rlm@432: ** What is imagination? rlm@432: rlm@432: One kind of imagination is /sympathetic/ imagination: you imagine rlm@432: yourself in the position of something/someone you are rlm@432: observing. This type of imagination comes into play when you follow rlm@432: along visually when watching someone perform actions, or when you rlm@432: sympathetically grimace when someone hurts themselves. This type of rlm@432: imagination uses the constraints you have learned about your own rlm@432: body to highly constrain the possibilities in whatever you are rlm@432: seeing. It uses all your senses to including your senses of touch, rlm@432: proprioception, etc. Humans are flexible when it comes to "putting rlm@432: themselves in another's shoes," and can sympathetically understand rlm@432: not only other humans, but entities ranging from animals to cartoon rlm@432: characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen! rlm@432: rlm@432: rlm@432: #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers. rlm@432: #+ATTR_LaTeX: :width 5cm rlm@432: [[./images/cat-drinking.jpg]] rlm@432: rlm@432: rlm@432: #+begin_listing clojure rlm@432: \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.} rlm@432: #+name: test-1 rlm@432: #+begin_src clojure rlm@432: (defn test-pipeline rlm@432: "Testing vision: rlm@432: Tests the vision system by creating two views of the same rotating rlm@432: object from different angles and displaying both of those views in rlm@432: JFrames. rlm@432: rlm@432: You should see a rotating cube, and two windows, rlm@432: each displaying a different view of the cube." rlm@432: ([] (test-pipeline false)) rlm@432: ([record?] rlm@432: (let [candy rlm@432: (box 1 1 1 :physical? false :color ColorRGBA/Blue)] rlm@432: (world rlm@432: (doto (Node.) rlm@432: (.attachChild candy)) rlm@432: {} rlm@432: (fn [world] rlm@432: (let [cam (.clone (.getCamera world)) rlm@432: width (.getWidth cam) rlm@432: height (.getHeight cam)] rlm@432: (add-camera! world cam rlm@432: (comp rlm@432: (view-image rlm@432: (if record? rlm@432: (File. "/home/r/proj/cortex/render/vision/1"))) rlm@432: BufferedImage!)) rlm@432: (add-camera! world rlm@432: (doto (.clone cam) rlm@432: (.setLocation (Vector3f. -10 0 0)) rlm@432: (.lookAt Vector3f/ZERO Vector3f/UNIT_Y)) rlm@432: (comp rlm@432: (view-image rlm@432: (if record? rlm@432: (File. "/home/r/proj/cortex/render/vision/2"))) rlm@432: BufferedImage!)) rlm@432: (let [timer (IsoTimer. 60)] rlm@432: (.setTimer world timer) rlm@432: (display-dilated-time world timer)) rlm@432: ;; This is here to restore the main view rlm@432: ;; after the other views have completed processing rlm@432: (add-camera! world (.getCamera world) no-op))) rlm@432: (fn [world tpf] rlm@432: (.rotate candy (* tpf 0.2) 0 0)))))) rlm@432: #+end_src rlm@432: #+end_listing rlm@432: rlm@432: - This is test1 \cite{Tappert77}. rlm@432: rlm@432: \cite{Tappert77} rlm@432: lol rlm@523: \cite{Tappert77} rlm@523: rlm@523: rlm@523: rlm@523: rlm@523: #+caption: This sensory predicate detects when the worm is resting on the rlm@523: #+caption: ground using the worm's sense of touch. rlm@523: #+name: resting-intro rlm@523: #+begin_listing clojure rlm@523: #+begin_src clojure rlm@523: (defn resting? rlm@523: "Is the worm resting on the ground?" rlm@523: [experiences] rlm@523: (every? rlm@523: (fn [touch-data] rlm@523: (< 0.9 (contact worm-segment-bottom touch-data))) rlm@523: (:touch (peek experiences)))) rlm@523: #+end_src rlm@523: #+end_listing rlm@523: rlm@523: #+caption: Even complicated actions such as ``wiggling'' are fairly simple rlm@523: #+caption: to describe with a rich enough language. rlm@523: #+name: wiggling-intro rlm@523: #+begin_listing clojure rlm@523: #+begin_src clojure rlm@523: (defn wiggling? rlm@523: "Is the worm wiggling?" rlm@523: [experiences] rlm@523: (let [analysis-interval 0x40] rlm@523: (when (> (count experiences) analysis-interval) rlm@523: (let [a-flex 3 rlm@523: a-ex 2 rlm@523: muscle-activity rlm@523: (map :muscle (vector:last-n experiences analysis-interval)) rlm@523: base-activity rlm@523: (map #(- (% a-flex) (% a-ex)) muscle-activity)] rlm@523: (= 2 rlm@523: (first rlm@523: (max-indexed rlm@523: (map #(Math/abs %) rlm@523: (take 20 (fft base-activity)))))))))) rlm@523: #+end_src rlm@523: #+end_listing