Mercurial > cortex

annotate thesis/garbage_cortex.org @ 553:20f64a70f8c5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
saving image and caption mods suggested by winston.
author Robert McIntyre <rlm@mit.edu>
date Fri, 02 May 2014 14:08:09 -0400
parents 90b236381642
children
rev   line source
rlm@432 1 #+title: =CORTEX=
rlm@432 2 #+author: Robert McIntyre
rlm@432 3 #+email: rlm@mit.edu
rlm@432 4 #+description: Using embodied AI to facilitate Artificial Imagination.
rlm@432 5 #+keywords: AI, clojure, embodiment
rlm@432 6
rlm@432 7 * Artificial Imagination
rlm@432 8
rlm@432 9 Imagine watching a video of someone skateboarding. When you watch
rlm@432 10 the video, you can imagine yourself skateboarding, and your
rlm@432 11 knowledge of the human body and its dynamics guides your
rlm@432 12 interpretation of the scene. For example, even if the skateboarder
rlm@432 13 is partially occluded, you can infer the positions of his arms and
rlm@432 14 body from your own knowledge of how your body would be positioned if
rlm@432 15 you were skateboarding. If the skateboarder suffers an accident, you
rlm@432 16 wince in sympathy, imagining the pain your own body would experience
rlm@432 17 if it were in the same situation. This empathy with other people
rlm@432 18 guides our understanding of whatever they are doing because it is a
rlm@432 19 powerful constraint on what is probable and possible. In order to
rlm@432 20 make use of this powerful empathy constraint, I need a system that
rlm@432 21 can generate and make sense of sensory data from the many different
rlm@432 22 senses that humans possess. The two key proprieties of such a system
rlm@432 23 are /embodiment/ and /imagination/.
rlm@432 24
rlm@432 25 ** What is imagination?
rlm@432 26
rlm@432 27 One kind of imagination is /sympathetic/ imagination: you imagine
rlm@432 28 yourself in the position of something/someone you are
rlm@432 29 observing. This type of imagination comes into play when you follow
rlm@432 30 along visually when watching someone perform actions, or when you
rlm@432 31 sympathetically grimace when someone hurts themselves. This type of
rlm@432 32 imagination uses the constraints you have learned about your own
rlm@432 33 body to highly constrain the possibilities in whatever you are
rlm@432 34 seeing. It uses all your senses to including your senses of touch,
rlm@432 35 proprioception, etc. Humans are flexible when it comes to "putting
rlm@432 36 themselves in another's shoes," and can sympathetically understand
rlm@432 37 not only other humans, but entities ranging from animals to cartoon
rlm@432 38 characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen!
rlm@432 39
rlm@432 40
rlm@432 41 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.
rlm@432 42 #+ATTR_LaTeX: :width 5cm
rlm@432 43 [[./images/cat-drinking.jpg]]
rlm@432 44
rlm@432 45
rlm@432 46 #+begin_listing clojure
rlm@432 47 \caption{This is a basic test for the vision system. It only tests the vision-pipeline and does not deal with loading eyes from a blender file. The code creates two videos of the same rotating cube from different angles.}
rlm@432 48 #+name: test-1
rlm@432 49 #+begin_src clojure
rlm@432 50 (defn test-pipeline
rlm@432 51 "Testing vision:
rlm@432 52 Tests the vision system by creating two views of the same rotating
rlm@432 53 object from different angles and displaying both of those views in
rlm@432 54 JFrames.
rlm@432 55
rlm@432 56 You should see a rotating cube, and two windows,
rlm@432 57 each displaying a different view of the cube."
rlm@432 58 ([] (test-pipeline false))
rlm@432 59 ([record?]
rlm@432 60 (let [candy
rlm@432 61 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@432 62 (world
rlm@432 63 (doto (Node.)
rlm@432 64 (.attachChild candy))
rlm@432 65 {}
rlm@432 66 (fn [world]
rlm@432 67 (let [cam (.clone (.getCamera world))
rlm@432 68 width (.getWidth cam)
rlm@432 69 height (.getHeight cam)]
rlm@432 70 (add-camera! world cam
rlm@432 71 (comp
rlm@432 72 (view-image
rlm@432 73 (if record?
rlm@432 74 (File. "/home/r/proj/cortex/render/vision/1")))
rlm@432 75 BufferedImage!))
rlm@432 76 (add-camera! world
rlm@432 77 (doto (.clone cam)
rlm@432 78 (.setLocation (Vector3f. -10 0 0))
rlm@432 79 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@432 80 (comp
rlm@432 81 (view-image
rlm@432 82 (if record?
rlm@432 83 (File. "/home/r/proj/cortex/render/vision/2")))
rlm@432 84 BufferedImage!))
rlm@432 85 (let [timer (IsoTimer. 60)]
rlm@432 86 (.setTimer world timer)
rlm@432 87 (display-dilated-time world timer))
rlm@432 88 ;; This is here to restore the main view
rlm@432 89 ;; after the other views have completed processing
rlm@432 90 (add-camera! world (.getCamera world) no-op)))
rlm@432 91 (fn [world tpf]
rlm@432 92 (.rotate candy (* tpf 0.2) 0 0))))))
rlm@432 93 #+end_src
rlm@432 94 #+end_listing
rlm@432 95
rlm@432 96 - This is test1 \cite{Tappert77}.
rlm@432 97
rlm@432 98 \cite{Tappert77}
rlm@432 99 lol
rlm@523 100 \cite{Tappert77}
rlm@523 101
rlm@523 102
rlm@523 103
rlm@523 104
rlm@523 105 #+caption: This sensory predicate detects when the worm is resting on the
rlm@523 106 #+caption: ground using the worm's sense of touch.
rlm@523 107 #+name: resting-intro
rlm@523 108 #+begin_listing clojure
rlm@523 109 #+begin_src clojure
rlm@523 110 (defn resting?
rlm@523 111 "Is the worm resting on the ground?"
rlm@523 112 [experiences]
rlm@523 113 (every?
rlm@523 114 (fn [touch-data]
rlm@523 115 (< 0.9 (contact worm-segment-bottom touch-data)))
rlm@523 116 (:touch (peek experiences))))
rlm@523 117 #+end_src
rlm@523 118 #+end_listing
rlm@523 119
rlm@523 120 #+caption: Even complicated actions such as ``wiggling'' are fairly simple
rlm@523 121 #+caption: to describe with a rich enough language.
rlm@523 122 #+name: wiggling-intro
rlm@523 123 #+begin_listing clojure
rlm@523 124 #+begin_src clojure
rlm@523 125 (defn wiggling?
rlm@523 126 "Is the worm wiggling?"
rlm@523 127 [experiences]
rlm@523 128 (let [analysis-interval 0x40]
rlm@523 129 (when (> (count experiences) analysis-interval)
rlm@523 130 (let [a-flex 3
rlm@523 131 a-ex 2
rlm@523 132 muscle-activity
rlm@523 133 (map :muscle (vector:last-n experiences analysis-interval))
rlm@523 134 base-activity
rlm@523 135 (map #(- (% a-flex) (% a-ex)) muscle-activity)]
rlm@523 136 (= 2
rlm@523 137 (first
rlm@523 138 (max-indexed
rlm@523 139 (map #(Math/abs %)
rlm@523 140 (take 20 (fft base-activity))))))))))
rlm@523 141 #+end_src
rlm@523 142 #+end_listing