Mercurial > cortex

annotate thesis/cortex.org @ 425:efba8526a662

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
happy with code formatting.
author Robert McIntyre <rlm@mit.edu>
date Fri, 21 Mar 2014 02:48:23 -0400
parents 6b0f77df0e53
children 435b5e22d72a
rev   line source
rlm@425 1 #+title: =CORTEX=
rlm@425 2 #+author: Robert McIntyre
rlm@425 3 #+email: rlm@mit.edu
rlm@425 4 #+description: Using embodied AI to facilitate Artificial Imagination.
rlm@425 5 #+keywords: AI, clojure, embodiment
rlm@422 6
rlm@425 7 * Artificial Imagination
rlm@422 8
rlm@425 9 Imagine watching a video of someone skateboarding. When you watch
rlm@425 10 the video, you can imagine yourself skateboarding, and your
rlm@425 11 knowledge of the human body and its dynamics guides your
rlm@425 12 interpretation of the scene. For example, even if the skateboarder
rlm@425 13 is partially occluded, you can infer the positions of his arms and
rlm@425 14 body from your own knowledge of how your body would be positioned if
rlm@425 15 you were skateboarding. If the skateboarder suffers an accident, you
rlm@425 16 wince in sympathy, imagining the pain your own body would experience
rlm@425 17 if it were in the same situation. This empathy with other people
rlm@425 18 guides our understanding of whatever they are doing because it is a
rlm@425 19 powerful constraint on what is probable and possible. In order to
rlm@425 20 make use of this powerful empathy constraint, I need a system that
rlm@425 21 can generate and make sense of sensory data from the many different
rlm@425 22 senses that humans possess. The two key proprieties of such a system
rlm@425 23 are /embodiment/ and /imagination/.
rlm@422 24
rlm@425 25 ** What is imagination?
rlm@422 26
rlm@425 27 One kind of imagination is /sympathetic/ imagination: you imagine
rlm@425 28 yourself in the position of something/someone you are
rlm@425 29 observing. This type of imagination comes into play when you follow
rlm@425 30 along visually when watching someone perform actions, or when you
rlm@425 31 sympathetically grimace when someone hurts themselves. This type of
rlm@425 32 imagination uses the constraints you have learned about your own
rlm@425 33 body to highly constrain the possibilities in whatever you are
rlm@425 34 seeing. It uses all your senses to including your senses of touch,
rlm@425 35 proprioception, etc. Humans are flexible when it comes to "putting
rlm@425 36 themselves in another's shoes," and can sympathetically understand
rlm@425 37 not only other humans, but entities ranging from animals to cartoon
rlm@425 38 characters to [[http://www.youtube.com/watch?v=0jz4HcwTQmU][single dots]] on a screen!
rlm@425 39
rlm@425 40
rlm@425 41 #+caption: A cat drinking some water. Identifying this action is beyond the state of the art for computers.
rlm@425 42 #+ATTR_LaTeX: :width 5cm
rlm@425 43 [[./images/cat-drinking.jpg]]
rlm@425 44
rlm@425 45
rlm@425 46 This is a basic test for the vision system. It only tests the
rlm@425 47 vision-pipeline and does not deal with loading eyes from a blender
rlm@425 48 file. The code creates two videos of the same rotating cube from
rlm@425 49 different angles.
rlm@425 50
rlm@425 51
rlm@425 52 #+name: test-1
rlm@425 53 #+begin_src clojure
rlm@425 54 (in-ns 'cortex.test.vision)
rlm@425 55
rlm@425 56 (defn test-pipeline
rlm@425 57 "Testing vision:
rlm@425 58 Tests the vision system by creating two views of the same rotating
rlm@425 59 object from different angles and displaying both of those views in
rlm@425 60 JFrames.
rlm@425 61
rlm@425 62 You should see a rotating cube, and two windows,
rlm@425 63 each displaying a different view of the cube."
rlm@425 64 ([] (test-pipeline false))
rlm@425 65 ([record?]
rlm@425 66 (let [candy
rlm@425 67 (box 1 1 1 :physical? false :color ColorRGBA/Blue)]
rlm@425 68 (world
rlm@425 69 (doto (Node.)
rlm@425 70 (.attachChild candy))
rlm@425 71 {}
rlm@425 72 (fn [world]
rlm@425 73 (let [cam (.clone (.getCamera world))
rlm@425 74 width (.getWidth cam)
rlm@425 75 height (.getHeight cam)]
rlm@425 76 (add-camera! world cam
rlm@425 77 (comp
rlm@425 78 (view-image
rlm@425 79 (if record?
rlm@425 80 (File. "/home/r/proj/cortex/render/vision/1")))
rlm@425 81 BufferedImage!))
rlm@425 82 (add-camera! world
rlm@425 83 (doto (.clone cam)
rlm@425 84 (.setLocation (Vector3f. -10 0 0))
rlm@425 85 (.lookAt Vector3f/ZERO Vector3f/UNIT_Y))
rlm@425 86 (comp
rlm@425 87 (view-image
rlm@425 88 (if record?
rlm@425 89 (File. "/home/r/proj/cortex/render/vision/2")))
rlm@425 90 BufferedImage!))
rlm@425 91 (let [timer (IsoTimer. 60)]
rlm@425 92 (.setTimer world timer)
rlm@425 93 (display-dilated-time world timer))
rlm@425 94 ;; This is here to restore the main view
rlm@425 95 ;; after the other views have completed processing
rlm@425 96 (add-camera! world (.getCamera world) no-op)))
rlm@425 97 (fn [world tpf]
rlm@425 98 (.rotate candy (* tpf 0.2) 0 0))))))
rlm@425 99 #+end_src