# HG changeset patch # User Robert McIntyre # Date 1399002101 14400 # Node ID 0b891e0dd8096f77094502ffd81f997c481b808e # Parent 5d89879fc894358ffe44d45d8a303a26b11b95ad version 0.2 of thesis complete. diff -r 5d89879fc894 -r 0b891e0dd809 org/self_organizing_touch.clj --- a/org/self_organizing_touch.clj Mon Apr 28 15:10:59 2014 -0400 +++ b/org/self_organizing_touch.clj Thu May 01 23:41:41 2014 -0400 @@ -99,11 +99,6 @@ (recur result xs) (recur (cons x result) xs)))))) -(def all-touch-coordinates - (concat - (rect-region [0 15] [7 22]) - (rect-region [8 0] [14 29]) - (rect-region [15 15] [22 22]))) (defn view-touch-region ([coords out] diff -r 5d89879fc894 -r 0b891e0dd809 org/worm_learn.clj --- a/org/worm_learn.clj Mon Apr 28 15:10:59 2014 -0400 +++ b/org/worm_learn.clj Thu May 01 23:41:41 2014 -0400 @@ -20,8 +20,18 @@ (dorun (cortex.import/mega-import-jme3)) (rlm.rlm-commands/help) + (load-bullet) +(defn bin [digits] + (fn [angles] + (->> angles + (flatten) + (map (juxt #(Math/sin %) #(Math/cos %))) + (flatten) + (mapv #(Math/round (* % (Math/pow 10 (dec digits)))))))) + + (def hand "Models/test-creature/hand.blend") (defn worm-model [] @@ -162,6 +172,12 @@ y (range y0 (inc y1))] [x y]))) +(def all-touch-coordinates + (concat + (rect-region [0 15] [7 22]) + (rect-region [8 0] [14 29]) + (rect-region [15 15] [22 22]))) + (def worm-segment-bottom (rect-region [8 15] [14 22])) (defn contact @@ -221,8 +237,6 @@ (or (accept? (take 64 base-activity)) (accept? (take 64 (drop 20 base-activity)))))))) - - (def worm-segment-bottom-tip (rect-region [15 15] [22 22])) (def worm-segment-top-tip (rect-region [0 15] [7 22])) @@ -238,6 +252,36 @@ (< 0.1 (contact worm-segment-top-tip head-touch)))))) +(defn draped? + "Is the worm: + -- not flat (the floor is not a 'chair') + -- supported (not using its muscles to hold its position) + -- stable (not changing its position) + -- touching something (must register contact)" + [experiences] + (let [b2-hash (bin 2) + touch (:touch (peek experiences)) + total-contact + (reduce + + + (map #(contact all-touch-coordinates %) + (rest touch)))] + (println total-contact) + (and (not (resting? experiences)) + (every? + zero? + (-> experiences + (vector:last-n 25) + (#(map :muscle %)) + (flatten))) + (-> experiences + (vector:last-n 20) + (#(map (comp b2-hash flatten :proprioception) %)) + (set) + (count) (= 1)) + (< 0.03 total-contact)))) + + (declare phi-space phi-scan debug-experience) @@ -254,6 +298,13 @@ [(Vector3f. -0.0708936, -8.570261, 2.6487997) (Quaternion. -2.318909E-4, 0.9985348, 0.053941682, 0.004291452)]) +(defn summon-chair + "Create a chair in the world for the worm" + [world] + (let [chair (box 0.5 0.5 0.5 :position (Vector3f. 0 -5 -2) + :mass 350. :color ColorRGBA/Pink)] + (add-element world chair (.getRootNode world)))) + (defn worm-world-defaults [] (let [direct-control (worm-direct-control worm-muscle-labels 40)] (merge direct-control @@ -262,7 +313,11 @@ :experiences (atom []) :experience-watch debug-experience :worm worm - :end-frame nil}))) + :end-frame nil + :keybindings + (merge (:keybindings direct-control) + {"key-b" (fn [world pressed?] + (if pressed? (summon-chair world)))})}))) (defn dir! [file] (if-not (.exists file) @@ -291,6 +346,7 @@ (defn debug-experience [experiences text] (cond + (draped? experiences) (.setText text "Draped") (grand-circle? experiences) (.setText text "Grand Circle") (curled? experiences) (.setText text "Curled") (wiggling? experiences) (.setText text "Wiggling") @@ -390,14 +446,6 @@ :experiences experiences}))) @experiences)) -(defn bin [digits] - (fn [angles] - (->> angles - (flatten) - (map (juxt #(Math/sin %) #(Math/cos %))) - (flatten) - (mapv #(Math/round (* % (Math/pow 10 (dec digits)))))))) - ;; k-nearest neighbors with spatial binning. Only returns a result if ;; the propriceptive data is within 10% of a previously recorded ;; result in all dimensions. @@ -467,7 +515,7 @@ ;;(infer-nils [1 nil 1 1]) [1 1 1 1] ;;(infer-nils [1 1 1 nil]) [1 1 1 0] ;;(infer-nils [nil 2 1 1]) [2 2 1 1] - + (defn empathy-demonstration [] (let [proprio (atom ())] @@ -479,6 +527,7 @@ empathy (mapv phi-space (infer-nils exp-thread))] (println-repl (vector:last-n exp-thread 22)) (cond + (draped? empathy) (.setText text "Draped") (grand-circle? empathy) (.setText text "Grand Circle") (curled? empathy) (.setText text "Curled") (wiggling? empathy) (.setText text "Wiggling") @@ -497,6 +546,11 @@ @experiences)) (def phi-scan (gen-phi-scan phi-space))) +(defn empathy-experiment-0 [record] + (.start (worm-world :record record))) + + + (defn empathy-experiment-1 [record] (.start (worm-world :experience-watch (empathy-demonstration) :record record :worm worm*))) diff -r 5d89879fc894 -r 0b891e0dd809 thesis/Makefile --- a/thesis/Makefile Mon Apr 28 15:10:59 2014 -0400 +++ b/thesis/Makefile Thu May 01 23:41:41 2014 -0400 @@ -14,4 +14,5 @@ scp: scp "r@aurellem.org:/home/r/thesis/$(THESIS_NAME).pdf" . -all: thesis scp +all: thesis + make scp diff -r 5d89879fc894 -r 0b891e0dd809 thesis/cortex.org --- a/thesis/cortex.org Mon Apr 28 15:10:59 2014 -0400 +++ b/thesis/cortex.org Thu May 01 23:41:41 2014 -0400 @@ -513,7 +513,7 @@ appears to flow at a constant rate, regardless of how complicated the environment becomes or how many creatures are in the simulation. The cost is that =CORTEX= can sometimes run slower than - real time. Time dialation works both ways, however --- simulations + real time. Time dilation works both ways, however --- simulations of very simple creatures in =CORTEX= generally run at 40x real-time on my machine! @@ -565,7 +565,7 @@ each sense. Fortunately this idea is already a well known computer graphics - technique called /UV-mapping/. In UV-maping, the three-dimensional + technique called /UV-mapping/. In UV-mapping, the three-dimensional surface of a model is cut and smooshed until it fits on a two-dimensional image. You paint whatever you want on that image, and when the three-dimensional shape is rendered in a game the @@ -2814,7 +2814,7 @@ The worm's total life experience is a long looping path through \Phi-space. I will now introduce simple way of taking that - experiece path and building a function that can infer complete + experience path and building a function that can infer complete sensory experience given only a stream of proprioceptive data. This /empathy/ function will provide a bridge to use the body centered action predicates on video-like streams of information. @@ -2972,7 +2972,7 @@ entries in a proprioceptive bin, because for each element in the starting bin it performs a series of set lookups in the preceding bins. If the total history is limited, then this takes time - proprotional to a only a constant multiple of the number of entries + proportional to a only a constant multiple of the number of entries in the starting bin. This analysis also applies, even if the action requires multiple longest chains -- it's still the average number of entries in a proprioceptive bin times the desired chain length. @@ -3125,7 +3125,7 @@ the testing environment for the action-predicates, with one major difference : the only sensory information available to the system is proprioception. From just the proprioception data and - \Phi-space, =longest-thread= synthesises a complete record the last + \Phi-space, =longest-thread= synthesizes a complete record the last 300 sensory experiences of the worm. These synthesized experiences are fed directly into the action predicates =grand-circle?=, =curled?=, =wiggling?=, and =resting?= from before and their output @@ -3365,13 +3365,11 @@ [[./images/worm-roll.png]] #+caption: After completing its adventures, the worm now knows - #+caption: how its touch sensors are arranged along its skin. These - #+caption: are the regions that were deemed important by + #+caption: how its touch sensors are arranged along its skin. Each of these six rectangles are touch sensory patterns that were + #+caption: deemed important by #+caption: =learn-touch-regions=. Each white square in the rectangles #+caption: above is a cluster of ``related" touch nodes as determined - #+caption: by the system. Since each square in the ``cross" corresponds - #+caption: to a face, the worm has correctly discovered that it has - #+caption: six faces. + #+caption: by the system. The worm has correctly discovered that it has six faces, and has partitioned its sensory map into these six faces. #+name: worm-touch-map #+ATTR_LaTeX: :width 12cm [[./images/touch-learn.png]] @@ -3383,29 +3381,133 @@ completely scrambled. The cross shape is just for convenience. This example justifies the use of pre-defined touch regions in =EMPATH=. +** Recognizing an object using embodied representation + + At the beginning of the thesis, I suggested that we might recognize + the chair in Figure \ref{hidden-chair} by imagining ourselves in + the position of the man and realizing that he must be sitting on + something in order to maintain that position. Here, I present a + brief elaboration on how to this might be done. + + First, I need the feeling of leaning or resting /on/ some other + object that is not the floor. This feeling is easy to describe + using an embodied representation. + + #+caption: Program describing the sense of leaning or resting on something. + #+caption: This involves a relaxed posture, the feeling of touching something, + #+caption: and a period of stability where the worm does not move. + #+name: draped + #+begin_listing clojure + #+begin_src clojure +(defn draped? + "Is the worm: + -- not flat (the floor is not a 'chair') + -- supported (not using its muscles to hold its position) + -- stable (not changing its position) + -- touching something (must register contact)" + [experiences] + (let [b2-hash (bin 2) + touch (:touch (peek experiences)) + total-contact + (reduce + + + (map #(contact all-touch-coordinates %) + (rest touch)))] + (println total-contact) + (and (not (resting? experiences)) + (every? + zero? + (-> experiences + (vector:last-n 25) + (#(map :muscle %)) + (flatten))) + (-> experiences + (vector:last-n 20) + (#(map (comp b2-hash flatten :proprioception) %)) + (set) + (count) (= 1)) + (< 0.03 total-contact)))) + #+end_src + #+end_listing + + #+caption: The =draped?= predicate detects the presence of the + #+caption: cube whenever the worm interacts with it. The details of the + #+caption: cube are irrelevant; only the way it influences the worm's + #+caption: body matters. + #+name: draped-video + #+ATTR_LaTeX: :width 13cm + [[./images/draped.png]] + + Though this is a simple example, using the =draped?= predicate to + detect the cube has interesting advantages. The =draped?= predicate + describes the cube not in terms of properties that the cube has, + but instead in terms of how the worm interacts with it physically. + This means that the cube can still be detected even if it is not + visible, as long as its influence on the worm's body is visible. + + This system will also see the virtual cube created by a + ``mimeworm", which uses its muscles in a very controlled way to + mimic the appearance of leaning on a cube. The system will + anticipate that there is an actual invisible cube that provides + support! + + #+caption: Can you see the thing that this person is leaning on? + #+caption: What properties does it have, other than how it makes the man's + #+caption: elbow and shoulder feel? I wonder if people who can actually + #+caption: maintain this pose easily still see the support? + #+name: mime + #+ATTR_LaTeX: :width 6cm + [[./images/pablo-the-mime.png]] + + This makes me wonder about the psychology of actual mimes. Suppose + for a moment that people have something analogous to \Phi-space and + that one of the ways that they find objects in a scene is by their + relation to other people's bodies. Suppose that a person watches a + person miming an invisible wall. For a person with no experience + with miming, their \Phi-space will only have entries that describe + the scene with the sensation of their hands touching a wall. This + sensation of touch will create a strong impression of a wall, even + though the wall would have to be invisible. A person with + experience in miming however, will have entries in their \Phi-space + that describe the wall-miming position without a sense of touch. It + will not seem to such as person that an invisible wall is present, + but merely that the mime is holding out their hands in a special + way. Thus, the theory that humans use something like \Phi-space + weakly predicts that learning how to mime should break the power of + miming illusions. Most optical illusions still work no matter how + much you know about them, so this proposal would be quite + interesting to test, as it predicts a non-standard result! + + +#+BEGIN_LaTeX +\clearpage +#+END_LaTeX + * Contributions + + The big idea behind this thesis is a new way to represent and + recognize physical actions, which I call /empathic representation/. + Actions are represented as predicates which have access to the + totality of a creature's sensory abilities. To recognize the + physical actions of another creature similar to yourself, you + imagine what they would feel by examining the position of their body + and relating it to your own previous experience. - The big idea behind this thesis is a new way to represent and - recognize physical actions -- empathic representation. Actions are - represented as predicates which have available the totality of a - creature's sensory abilities. To recognize the physical actions of - another creature similar to yourself, you imagine what they would - feel by examining the position of their body and relating it to your - own previous experience. - - Empathic description of physical actions is very robust and general. - Because the representation is body-centered, it avoids the fragility - of learning from example videos. Because it relies on all of a + Empathic representation of physical actions is robust and general. + Because the representation is body-centered, it avoids baking in a + particular viewpoint like you might get from learning from example + videos. Because empathic representation relies on all of a creature's senses, it can describe exactly what an action /feels like/ without getting caught up in irrelevant details such as visual appearance. I think it is important that a correct description of - jumping (for example) should not waste even a single bit on the - color of a person's clothes or skin; empathic representation can - avoid this waste by describing jumping in terms of touch, muscle - contractions, and the brief feeling of weightlessness. Empathic - representation is very low-level in that it describes actions using - concrete sensory data with little abstraction, but it has the - generality of much more abstract representations! + jumping (for example) should not include irrelevant details such as + the color of a person's clothes or skin; empathic representation can + get right to the heart of what jumping is by describing it in terms + of touch, muscle contractions, and a brief feeling of + weightlessness. Empathic representation is very low-level in that it + describes actions using concrete sensory data with little + abstraction, but it has the generality of much more abstract + representations! Another important contribution of this thesis is the development of the =CORTEX= system, a complete environment for creating simulated @@ -3413,29 +3515,31 @@ proprioception, hearing, vision, and muscle tension. You have seen how to create new creatures using blender, a 3D modeling tool. - I hope that =CORTEX= will be useful in further research projects. To - this end I have included the full source to =CORTEX= along with a - large suite of tests and examples. I have also created a user guide - for =CORTEX= which is included in an appendix to this thesis. - As a minor digression, you also saw how I used =CORTEX= to enable a tiny worm to discover the topology of its skin simply by rolling on - the ground. - - In conclusion, the main contributions of this thesis are: - - - =CORTEX=, a comprehensive platform for embodied AI experiments. - =CORTEX= supports many features lacking in other systems, such - proper simulation of hearing. It is easy to create new =CORTEX= - creatures using Blender, a free 3D modeling program. - - - =EMPATH=, which uses =CORTEX= to identify the actions of a - worm-like creature using a computational model of empathy. This - empathic representation of actions is an important new kind of - representation for physical actions. + the ground. You also saw how to detect objects using only embodied + predicates. + + In conclusion, for this thesis I: + + - Developed the idea of embodied representation, which describes + actions that a creature can do in terms of first-person sensory + data. + + - Developed a method of empathic action recognition which uses + previous embodied experience and embodied representation of + actions to greatly constrain the possible interpretations of an + action. + + - Created =EMPATH=, a program which uses empathic action + recognition to recognize physical actions in a simple model + involving segmented worm-like creatures. + + - Created =CORTEX=, a comprehensive platform for embodied AI + experiments. It is the base on which =EMPATH= is built. #+BEGIN_LaTeX -\newpage +\clearpage \appendix #+END_LaTeX diff -r 5d89879fc894 -r 0b891e0dd809 thesis/images/draped.png Binary file thesis/images/draped.png has changed diff -r 5d89879fc894 -r 0b891e0dd809 thesis/images/pablo-the-mime.png Binary file thesis/images/pablo-the-mime.png has changed diff -r 5d89879fc894 -r 0b891e0dd809 thesis/rlm-cortex-meng.tex --- a/thesis/rlm-cortex-meng.tex Mon Apr 28 15:10:59 2014 -0400 +++ b/thesis/rlm-cortex-meng.tex Thu May 01 23:41:41 2014 -0400 @@ -56,6 +56,7 @@ \usepackage{hyperref} \fancyhead[RO,LE]{\bfseries \nouppercase{\rightmark}} \renewcommand{\subsectionmark}[1]{\markright{#1}} +\renewcommand{\sectionmark}[1]{\markright{#1}} \usepackage[backend=bibtex,style=authoryear]{biblatex} %\usepackage[section]{placeins}