Mercurial > cortex

annotate thesis/user-guide.org @ 486:6d460ac3f5d0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
working on user guide.
author Robert McIntyre <rlm@mit.edu>
date Sat, 29 Mar 2014 20:12:52 -0400
parents c1e6b7221b2f
children 19b55aaf4462
rev   line source
rlm@437 1 * Appendix: =CORTEX= User Guide
rlm@437 2
rlm@486 3 Those who write a thesis should endeavor to make their code not only
rlm@486 4 accessable, but actually useable, as a way to pay back the community
rlm@486 5 that made the thesis possible in the first place. This thesis would
rlm@486 6 not be possible without Free Software such as jMonkeyEngine3,
rlm@486 7 Blender, clojure, emacs, ffmpeg, and many other tools. That is why I
rlm@486 8 have included this user guide, in the hope that someone else might
rlm@486 9 find =CORTEX= useful.
rlm@437 10
rlm@486 11 ** Obtaining =CORTEX=
rlm@437 12
rlm@486 13 You can get cortex from its mercurial repository at
rlm@486 14 http://hg.bortreb.com/cortex. You may also download =CORTEX=
rlm@486 15 releases at http://aurellem.org/cortex/releases/. As a condition of
rlm@486 16 making this thesis, I have also provided Professor Winston the
rlm@486 17 =CORTEX= source, and he knows how to run the demos and get started.
rlm@486 18 You may also email me at =cortex@aurellem.org= and I may help where
rlm@486 19 I can.
rlm@486 20
rlm@486 21 ** Running =CORTEX=
rlm@486 22
rlm@486 23 =CORTEX= comes with README and INSTALL files that will guide you
rlm@486 24 through installation and running the test suite. In particular you
rlm@486 25 should look at test =cortex.test= which contains test suites that
rlm@486 26 run through all senses and multiple creatures.
rlm@486 27
rlm@486 28 ** Creating creatures
rlm@486 29
rlm@486 30 Creatures are created using /Blender/, a free 3D modeling program.
rlm@486 31 You will need Blender version 2.6 when using the =CORTEX= included
rlm@486 32 in this thesis. You create a =CORTEX= creature in a similiar manner
rlm@486 33 to modeling anything in Blender, except that you also create
rlm@486 34 several trees of empty nodes which define the creature's senses.
rlm@486 35
rlm@486 36 *** Mass
rlm@486 37
rlm@486 38 To give an object mass in =CORTEX=, add a ``mass'' metadata label
rlm@486 39 to the object with the mass in jMonkeyEngine units. Note that
rlm@486 40 setting the mass to 0 causes the object to be immovable.
rlm@486 41
rlm@486 42 *** Joints
rlm@486 43
rlm@486 44 Joints are created by creating an empty node named =joints= and
rlm@486 45 then creating any number of empty child nodes to represent your
rlm@486 46 creature's joints. The joint will automatically connect the
rlm@486 47 closest two physical objects. It will help to set the empty node's
rlm@486 48 display mode to ``Arrows'' so that you can clearly see the
rlm@486 49 direction of the axes.
rlm@486 50
rlm@486 51 Joint nodes should have the following metadata under the ``joint''
rlm@486 52 label:
rlm@486 53
rlm@486 54 #+BEGIN_SRC clojure
rlm@486 55 ;; ONE OF the following, under the label "joint":
rlm@486 56 {:type :point}
rlm@486 57
rlm@486 58 ;; OR
rlm@486 59
rlm@486 60 {:type :hinge
rlm@486 61 :limit [<limit-low> <limit-high>]
rlm@486 62 :axis (Vector3f. <x> <y> <z>)}
rlm@486 63 ;;(:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
rlm@486 64
rlm@486 65 ;; OR
rlm@486 66
rlm@486 67 {:type :cone
rlm@486 68 :limit-xz <lim-xz>
rlm@486 69 :limit-xy <lim-xy>
rlm@486 70 :twist <lim-twist>} ;(use XZY rotation mode in blender!)
rlm@486 71 #+END_SRC
rlm@486 72
rlm@486 73 *** Eyes
rlm@486 74
rlm@486 75 Eyes are created by creating an empty node named =eyes= and then
rlm@486 76 creating any number of empty child nodes to represent your
rlm@486 77 creature's eyes.
rlm@486 78
rlm@486 79 Eye nodes should have the following metadata under the ``eye''
rlm@486 80 label:
rlm@486 81
rlm@486 82 #+BEGIN_SRC clojure
rlm@486 83 {:red <red-retina-definition>
rlm@486 84 :blue <blue-retina-definition>
rlm@486 85 :green <green-retina-definition>
rlm@486 86 :all <all-retina-definition>
rlm@486 87 (<0xrrggbb> <custom-retina-image>)...
rlm@486 88 }
rlm@486 89 #+END_SRC
rlm@486 90
rlm@486 91 Any of the color channels may be omitted. You may also include
rlm@486 92 your own color selectors, and in fact :red is equivalent to
rlm@486 93 0xFF0000 and so forth. The eye will be placed at the same position
rlm@486 94 as the empty node and will bind to the neatest physical object.
rlm@486 95 The eye will point outward from the X-axis of the node, and ``up''
rlm@486 96 will be in the direction of the X-axis of the node. It will help
rlm@486 97 to set the empty node's display mode to ``Arrows'' so that you can
rlm@486 98 clearly see the direction of the axes.
rlm@486 99
rlm@486 100 Each retina file should contain white pixels whever you want to be
rlm@486 101 sensitive to your chosen color. If you want the entire field of
rlm@486 102 view, specify :all of 0xFFFFFF and a retinal map that is entirely
rlm@486 103 white.
rlm@486 104
rlm@486 105 Here is a sample retinal map:
rlm@486 106
rlm@486 107 #+caption: An example retinal profile image. White pixels are
rlm@486 108 #+caption: photo-sensitive elements. The distribution of white
rlm@486 109 #+caption: pixels is denser in the middle and falls off at the
rlm@486 110 #+caption: edges and is inspired by the human retina.
rlm@486 111 #+name: retina
rlm@486 112 #+ATTR_LaTeX: :width 7cm :placement [H]
rlm@486 113 [[./images/retina-small.png]]
rlm@486 114
rlm@486 115 *** Hearing
rlm@486 116
rlm@486 117 Ears are created by creating an empty node named =ears= and then
rlm@486 118 creating any number of empty child nodes to represent your
rlm@486 119 creature's ears.
rlm@486 120
rlm@486 121 Ear nodes do not require any metadata.
rlm@486 122
rlm@486 123 The ear will bind to and follow the closest physical node.
rlm@486 124
rlm@486 125 *** Touch
rlm@486 126
rlm@486 127 Touch is handled similarly to mass. To make a particular object
rlm@486 128 touch sensitive, add metadata of the following form under the
rlm@486 129 object's ``touch'' metadata field:
rlm@486 130
rlm@486 131 #+BEGIN_EXAMPLE
rlm@486 132 <touch-UV-map-file-name>
rlm@486 133 #+END_EXAMPLE
rlm@486 134
rlm@486 135 You may also include an optional ``scale'' metadata number to
rlm@486 136 specifiy the length of the touch feelers. The default is $0.1$,
rlm@486 137 and this is generally sufficient.
rlm@486 138
rlm@486 139 The touch UV should contain white pixels for each touch sensor.
rlm@486 140
rlm@486 141 Here is an example touch-uv map that approximates a human finger,
rlm@486 142 and its corresponding model.
rlm@486 143
rlm@486 144 #+caption: This is the tactile-sensor-profile for the upper segment
rlm@486 145 #+caption: of a fingertip. It defines regions of high touch sensitivity
rlm@486 146 #+caption: (where there are many white pixels) and regions of low
rlm@486 147 #+caption: sensitivity (where white pixels are sparse).
rlm@486 148 #+name: guide-fingertip-UV
rlm@486 149 #+ATTR_LaTeX: :width 9cm :placement [H]
rlm@486 150 [[./images/finger-UV.png]]
rlm@486 151
rlm@486 152 #+caption: The fingertip UV-image form above applied to a simple
rlm@486 153 #+caption: model of a fingertip.
rlm@486 154 #+name: guide-fingertip
rlm@486 155 #+ATTR_LaTeX: :width 9cm :placement [H]
rlm@486 156 [[./images/finger-2.png]]
rlm@486 157
rlm@486 158 *** Propriocepotion
rlm@486 159
rlm@486 160 Proprioception is tied to each joint node -- nothing special must
rlm@486 161 be done in a blender model to enable proprioception other than
rlm@486 162 creating joint nodes.
rlm@486 163
rlm@486 164 *** Muscles
rlm@486 165
rlm@486 166 Muscles are created by creating an empty node named =muscles= and
rlm@486 167 then creating any number of empty child nodes to represent your
rlm@486 168 creature's muscles.
rlm@486 169
rlm@486 170
rlm@486 171 Muscle nodes should have the following metadata under the
rlm@486 172 ``muscle'' label:
rlm@486 173
rlm@486 174 #+BEGIN_EXAMPLE
rlm@486 175 <muscle-profile-file-name>
rlm@486 176 #+END_EXAMPLE
rlm@486 177
rlm@486 178 Muscles should also have a ``strength'' metadata entry describing
rlm@486 179 the muscle's total strength at full activation.
rlm@486 180
rlm@486 181 Muscle profiles are simple images that contain the relative amount
rlm@486 182 of muscle power in each simulated alpha motor neuron. The width of
rlm@486 183 the image is the total size of the motor pool, and the redness of
rlm@486 184 each neuron is the relative power of that motor pool.
rlm@486 185
rlm@486 186 While the profile image can have any dimensions, only the first
rlm@486 187 line of pixels is used to define the muscle. Here is a sample
rlm@486 188 muscle profile image that defines a human-like muscle.
rlm@486 189
rlm@486 190 #+caption: A muscle profile image that describes the strengths
rlm@486 191 #+caption: of each motor neuron in a muscle. White is weakest
rlm@486 192 #+caption: and dark red is strongest. This particular pattern
rlm@486 193 #+caption: has weaker motor neurons at the beginning, just
rlm@486 194 #+caption: like human muscle.
rlm@486 195 #+name: muscle-recruit
rlm@486 196 #+ATTR_LaTeX: :width 7cm :placement [H]
rlm@486 197 [[./images/basic-muscle.png]]
rlm@486 198
rlm@486 199 Muscles twist the nearest physical object about the muscle node's
rlm@486 200 Z-axis. I recommend using the ``Single Arrow'' display mode for
rlm@486 201 muscles and using the right hand rule to determine which way the
rlm@486 202 muscle will twist. To make a segment that can twist in multiple
rlm@486 203 directions, create multiple, differently aligned muscles.
rlm@486 204
rlm@486 205 ** =CORTEX= API
rlm@486 206
rlm@486 207 These are the some functions exposed by =CORTEX= for creating
rlm@486 208 worlds and simulating creatures. These are in addition to
rlm@486 209 jMonkeyEngine3's extensive library, which is documented elsewhere.
rlm@486 210
rlm@486 211 *** Simulation
rlm@486 212 - =(world root-node key-map setup-fn update-fn)= :: create
rlm@486 213 a simulation.
rlm@486 214 - /root-node/ :: a =com.jme3.scene.Node= object which
rlm@486 215 contains all of the objects that should be in the
rlm@486 216 simulation.
rlm@486 217
rlm@486 218 - /key-map/ :: a map from strings describing keys to
rlm@486 219 functions that should be executed whenever that key is
rlm@486 220 pressed. the functions should take a SimpleApplication
rlm@486 221 object and a boolean value. The SimpleApplication is the
rlm@486 222 current simulation that is running, and the boolean is true
rlm@486 223 if the key is being pressed, and false if it is being
rlm@486 224 released. As an example,
rlm@486 225 #+BEGIN_SRC clojure
rlm@486 226 {"key-j" (fn [game value] (if value (println "key j pressed")))}
rlm@486 227 #+END_SRC
rlm@486 228 is a valid key-map which will cause the simulation to print
rlm@486 229 a message whenever the 'j' key on the keyboard is pressed.
rlm@486 230
rlm@486 231 - /setup-fn/ :: a function that takes a =SimpleApplication=
rlm@486 232 object. It is called once when initializing the simulation.
rlm@486 233 Use it to create things like lights, change the gravity,
rlm@486 234 initialize debug nodes, etc.
rlm@486 235
rlm@486 236 - /update-fn/ :: this function takes a =SimpleApplication=
rlm@486 237 object and a float and is called every frame of the
rlm@486 238 simulation. The float tells how many seconds is has been
rlm@486 239 since the last frame was rendered, according to whatever
rlm@486 240 clock jme is currently using. The default is to use IsoTimer
rlm@486 241 which will result in this value always being the same.
rlm@486 242
rlm@486 243
rlm@486 244 - =(position-camera world position rotation)= :: set the position
rlm@486 245 of the simulation's main camera.
rlm@486 246 - =(enable-debug world)= :: turn on debug wireframes for each
rlm@486 247 simulated object.
rlm@486 248 - =(set-gravity world gravity)= :: set the gravity of a running
rlm@486 249 simulation.
rlm@486 250 - =(box length width height & {options})= :: create a box in the
rlm@486 251 simulation. Options is a hash map specifying texture, mass,
rlm@486 252 etc. Possible options are =:name=, =:color=, =:mass=,
rlm@486 253 =:friction=, =:texture=, =:material=, =:position=,
rlm@486 254 =:rotation=, =:shape=, and =:physical?=.
rlm@486 255 - =(sphere radius & {options})= :: create a sphere in the simulation.
rlm@486 256 Options are the same as in =box=.
rlm@486 257 - =(load-blender-model file-name)= :: create a node structure
rlm@486 258 representing that described in a blender file.
rlm@486 259 - =(light-up-everything world)= :: distribute a standard compliment
rlm@486 260 of lights throught the simulation. Should be adequate for most
rlm@486 261 purposes.
rlm@486 262 - =(node-seq node)= :: return a recursuve list of the node's
rlm@486 263 children.
rlm@486 264 - =(nodify name children)= :: construct a node given a node-name and
rlm@486 265 desired children.
rlm@486 266 - =(add-element world element)= :: add an object to a running world
rlm@486 267 simulation.
rlm@486 268 - =(set-accuracy world accuracy)= :: change the accuracy of the
rlm@486 269 world's physics simulator.
rlm@486 270 - =(asset-manager)= :: get an /AssetManager/, a jMonkeyEngine
rlm@486 271 construct that is useful for loading textures and is required
rlm@486 272 for smooth interaction with jMonkeyEngine library functions.
rlm@486 273
rlm@486 274
rlm@486 275 *** Creature Manipulation / Import
rlm@486 276
rlm@486 277 - =(body! creature)= :: give the creature a physical body.
rlm@486 278 - =(vision! creature)= :: give the creature a sense of vision.
rlm@486 279 Returns a list of functions which will each, when called
rlm@486 280 during a simulation, return the vision data for the channel of
rlm@486 281 one of the eyes. The functions are ordered depending on the
rlm@486 282 alphabetical order of the names of the eye nodes in the
rlm@486 283 blender file. The data returned by the functions is a vector
rlm@486 284 containing the eye's /topology/, a vector of coordinates, and
rlm@486 285 the eye's /data/, a vector of RGB values filtered by the eye's
rlm@486 286 sensitivity.
rlm@486 287 - =(hearing! creature)= :: give the creature a sense of hearing.
rlm@486 288 Returns a list of functions, one for each ear, that when
rlm@486 289 called will return a frame's worth of hearing data for that
rlm@486 290 ear. The functions are ordered depending on the alphabetical
rlm@486 291 order of the names of the ear nodes in the blender file. The
rlm@486 292 data returned by the functions is an array PCM encoded wav
rlm@486 293 data.
rlm@486 294 - =(touch! creature)= :: give the creature a sense of touch. Returns
rlm@486 295 a single function that must be called with the /root node/ of
rlm@486 296 the world, and which will return a vector of /touch-data/
rlm@486 297 one entry for each touch sensitive component, each entry of
rlm@486 298 which contains a /topology/ that specifies the distribution of
rlm@486 299 touch sensors, and the /data/, which is a vector of
rlm@486 300 =[activation, length]= pairs for each touch hair.
rlm@486 301 - =(proprioception! creature)= :: give the creature the sense of
rlm@486 302 proprioception. Returns a list of functions, one for each
rlm@486 303 joint, that when called during a running simulation will
rlm@486 304 report the =[headnig, pitch, roll]= of the joint.
rlm@486 305 - =(movement! creature)= :: give the creature the power of movement.
rlm@486 306 Creates a list of functions, one for each muscle, that when
rlm@486 307 called with an integer, will set the recruitment of that
rlm@486 308 muscle to that integer, and will report the current power
rlm@486 309 being exerted by the muscle. Order of muscles is determined by
rlm@486 310 the alphabetical sort order of the names of the muscle nodes.
rlm@486 311
rlm@486 312 *** Visualization/Debug
rlm@486 313
rlm@486 314 - =(view-vision)= :: create a function that when called with a list
rlm@486 315 of visual data returned from the functions made by =vision!=,
rlm@486 316 will display that visual data on the screen.
rlm@486 317 - =(view-hearing)= :: same as =view-vision= but for hearing.
rlm@486 318 - =(view-touch)= :: same as =view-vision= but for touch.
rlm@486 319 - =(view-proprioception)= :: same as =view-vision= but for
rlm@486 320 proprioception.
rlm@486 321 - =(view-movement)= :: same as =view-vision= but for
rlm@486 322 proprioception.
rlm@486 323 - =(view anything)= :: =view= is a polymorphic function that allows
rlm@486 324 you to inspect almost anything you could reasonably expect to
rlm@486 325 be able to ``see'' in =CORTEX=.
rlm@486 326 - =(text anything)= :: =text= is a polymorphic function that allows
rlm@486 327 you to convert practically anything into a text string.
rlm@486 328 - =(println-repl anything)= :: print messages to clojure's repl
rlm@486 329 instead of the simulation's terminal window.
rlm@486 330
rlm@486 331 *** Misc
rlm@486 332
rlm@486 333 - =(load-bullet)= :: unpack native libraries and initialize
rlm@486 334 blender. This function is required before other world building
rlm@486 335 functions are called.
rlm@486 336 - =(mega-import-jme3)= :: for experimenting at the REPL. This
rlm@486 337 function will import all jMonkeyEngine3 classes for immediate
rlm@486 338 use.