comparison thesis/cortex.org @ 495:c816594fbbe6

changes from Dylan.
author Robert McIntyre <rlm@mit.edu>
date Sat, 29 Mar 2014 23:01:49 -0400
parents 21b9dcec8d71
children 609356a82760
comparison
equal deleted inserted replaced
494:092b60381af0 495:c816594fbbe6
362 new =CORTEX= creatures using Blender, a free 3D modeling program. 362 new =CORTEX= creatures using Blender, a free 3D modeling program.
363 363
364 - I built =EMPATH=, which uses =CORTEX= to identify the actions of 364 - I built =EMPATH=, which uses =CORTEX= to identify the actions of
365 a worm-like creature using a computational model of empathy. 365 a worm-like creature using a computational model of empathy.
366 366
367 * Building =CORTEX= 367 * COMMENT Building =CORTEX=
368 368
369 I intend for =CORTEX= to be used as a general-purpose library for 369 I intend for =CORTEX= to be used as a general-purpose library for
370 building creatures and outfitting them with senses, so that it will 370 building creatures and outfitting them with senses, so that it will
371 be useful for other researchers who want to test out ideas of their 371 be useful for other researchers who want to test out ideas of their
372 own. To this end, wherver I have had to make archetictural choices 372 own. To this end, wherver I have had to make archetictural choices
373 about =CORTEX=, I have chosen to give as much freedom to the user as 373 about =CORTEX=, I have chosen to give as much freedom to the user as
374 possible, so that =CORTEX= may be used for things I have not 374 possible, so that =CORTEX= may be used for things I have not
375 forseen. 375 forseen.
376 376
377 ** COMMENT Simulation or Reality? 377 ** Simulation or Reality?
378 378
379 The most important archetictural decision of all is the choice to 379 The most important archetictural decision of all is the choice to
380 use a computer-simulated environemnt in the first place! The world 380 use a computer-simulated environemnt in the first place! The world
381 is a vast and rich place, and for now simulations are a very poor 381 is a vast and rich place, and for now simulations are a very poor
382 reflection of its complexity. It may be that there is a significant 382 reflection of its complexity. It may be that there is a significant
435 time in the simulated world can be slowed down to accommodate the 435 time in the simulated world can be slowed down to accommodate the
436 limitations of the character's programming. In terms of cost, 436 limitations of the character's programming. In terms of cost,
437 doing everything in software is far cheaper than building custom 437 doing everything in software is far cheaper than building custom
438 real-time hardware. All you need is a laptop and some patience. 438 real-time hardware. All you need is a laptop and some patience.
439 439
440 ** COMMENT Because of Time, simulation is perferable to reality 440 ** Because of Time, simulation is perferable to reality
441 441
442 I envision =CORTEX= being used to support rapid prototyping and 442 I envision =CORTEX= being used to support rapid prototyping and
443 iteration of ideas. Even if I could put together a well constructed 443 iteration of ideas. Even if I could put together a well constructed
444 kit for creating robots, it would still not be enough because of 444 kit for creating robots, it would still not be enough because of
445 the scourge of real-time processing. Anyone who wants to test their 445 the scourge of real-time processing. Anyone who wants to test their
458 the simulation. The cost is that =CORTEX= can sometimes run slower 458 the simulation. The cost is that =CORTEX= can sometimes run slower
459 than real time. This can also be an advantage, however --- 459 than real time. This can also be an advantage, however ---
460 simulations of very simple creatures in =CORTEX= generally run at 460 simulations of very simple creatures in =CORTEX= generally run at
461 40x on my machine! 461 40x on my machine!
462 462
463 ** COMMENT What is a sense? 463 ** What is a sense?
464 464
465 If =CORTEX= is to support a wide variety of senses, it would help 465 If =CORTEX= is to support a wide variety of senses, it would help
466 to have a better understanding of what a ``sense'' actually is! 466 to have a better understanding of what a ``sense'' actually is!
467 While vision, touch, and hearing all seem like they are quite 467 While vision, touch, and hearing all seem like they are quite
468 different things, I was supprised to learn during the course of 468 different things, I was supprised to learn during the course of
536 #+caption: density of touch sensors at the tip. 536 #+caption: density of touch sensors at the tip.
537 #+name: finger-side-view 537 #+name: finger-side-view
538 #+ATTR_LaTeX: :width 10cm 538 #+ATTR_LaTeX: :width 10cm
539 [[./images/finger-1.png]] 539 [[./images/finger-1.png]]
540 540
541 ** COMMENT Video game engines provide ready-made physics and shading 541 ** Video game engines provide ready-made physics and shading
542 542
543 I did not need to write my own physics simulation code or shader to 543 I did not need to write my own physics simulation code or shader to
544 build =CORTEX=. Doing so would lead to a system that is impossible 544 build =CORTEX=. Doing so would lead to a system that is impossible
545 for anyone but myself to use anyway. Instead, I use a video game 545 for anyone but myself to use anyway. Instead, I use a video game
546 engine as a base and modify it to accomodate the additional needs 546 engine as a base and modify it to accomodate the additional needs
560 texture to a model. Finally, because video game engines support a 560 texture to a model. Finally, because video game engines support a
561 large number of users, as long as =CORTEX= doesn't stray too far 561 large number of users, as long as =CORTEX= doesn't stray too far
562 from the base system, other researchers can turn to this community 562 from the base system, other researchers can turn to this community
563 for help when doing their research. 563 for help when doing their research.
564 564
565 ** COMMENT =CORTEX= is based on jMonkeyEngine3 565 ** =CORTEX= is based on jMonkeyEngine3
566 566
567 While preparing to build =CORTEX= I studied several video game 567 While preparing to build =CORTEX= I studied several video game
568 engines to see which would best serve as a base. The top contenders 568 engines to see which would best serve as a base. The top contenders
569 were: 569 were:
570 570
602 I chose jMonkeyEngine3 because it because it had the most features 602 I chose jMonkeyEngine3 because it because it had the most features
603 out of all the free projects I looked at, and because I could then 603 out of all the free projects I looked at, and because I could then
604 write my code in clojure, an implementation of =LISP= that runs on 604 write my code in clojure, an implementation of =LISP= that runs on
605 the JVM. 605 the JVM.
606 606
607 ** COMMENT =CORTEX= uses Blender to create creature models 607 ** =CORTEX= uses Blender to create creature models
608 608
609 For the simple worm-like creatures I will use later on in this 609 For the simple worm-like creatures I will use later on in this
610 thesis, I could define a simple API in =CORTEX= that would allow 610 thesis, I could define a simple API in =CORTEX= that would allow
611 one to create boxes, spheres, etc., and leave that API as the sole 611 one to create boxes, spheres, etc., and leave that API as the sole
612 way to create creatures. However, for =CORTEX= to truly be useful 612 way to create creatures. However, for =CORTEX= to truly be useful
639 #+caption: of muscles, ears, eyes, or joints. 639 #+caption: of muscles, ears, eyes, or joints.
640 #+name: sense-nodes 640 #+name: sense-nodes
641 #+ATTR_LaTeX: :width 10cm 641 #+ATTR_LaTeX: :width 10cm
642 [[./images/empty-sense-nodes.png]] 642 [[./images/empty-sense-nodes.png]]
643 643
644 ** COMMENT Bodies are composed of segments connected by joints 644 ** Bodies are composed of segments connected by joints
645 645
646 Blender is a general purpose animation tool, which has been used in 646 Blender is a general purpose animation tool, which has been used in
647 the past to create high quality movies such as Sintel 647 the past to create high quality movies such as Sintel
648 \cite{sintel}. Though Blender can model and render even complicated 648 \cite{sintel}. Though Blender can model and render even complicated
649 things like water, it is crucual to keep models that are meant to 649 things like water, it is crucual to keep models that are meant to
955 #+caption: simulation environment. 955 #+caption: simulation environment.
956 #+name: name 956 #+name: name
957 #+ATTR_LaTeX: :width 15cm 957 #+ATTR_LaTeX: :width 15cm
958 [[./images/physical-hand.png]] 958 [[./images/physical-hand.png]]
959 959
960 ** COMMENT Eyes reuse standard video game components 960 ** Eyes reuse standard video game components
961 961
962 Vision is one of the most important senses for humans, so I need to 962 Vision is one of the most important senses for humans, so I need to
963 build a simulated sense of vision for my AI. I will do this with 963 build a simulated sense of vision for my AI. I will do this with
964 simulated eyes. Each eye can be independently moved and should see 964 simulated eyes. Each eye can be independently moved and should see
965 its own version of the world depending on where it is. 965 its own version of the world depending on where it is.
1252 1252
1253 This vision code has already been absorbed by the jMonkeyEngine 1253 This vision code has already been absorbed by the jMonkeyEngine
1254 community and is now (in modified form) part of a system for 1254 community and is now (in modified form) part of a system for
1255 capturing in-game video to a file. 1255 capturing in-game video to a file.
1256 1256
1257 ** COMMENT Hearing is hard; =CORTEX= does it right 1257 ** Hearing is hard; =CORTEX= does it right
1258 1258
1259 At the end of this section I will have simulated ears that work the 1259 At the end of this section I will have simulated ears that work the
1260 same way as the simulated eyes in the last section. I will be able to 1260 same way as the simulated eyes in the last section. I will be able to
1261 place any number of ear-nodes in a blender file, and they will bind to 1261 place any number of ear-nodes in a blender file, and they will bind to
1262 the closest physical object and follow it as it moves around. Each ear 1262 the closest physical object and follow it as it moves around. Each ear
1553 1553
1554 This system of hearing has also been co-opted by the 1554 This system of hearing has also been co-opted by the
1555 jMonkeyEngine3 community and is used to record audio for demo 1555 jMonkeyEngine3 community and is used to record audio for demo
1556 videos. 1556 videos.
1557 1557
1558 ** COMMENT Touch uses hundreds of hair-like elements 1558 ** Touch uses hundreds of hair-like elements
1559 1559
1560 Touch is critical to navigation and spatial reasoning and as such I 1560 Touch is critical to navigation and spatial reasoning and as such I
1561 need a simulated version of it to give to my AI creatures. 1561 need a simulated version of it to give to my AI creatures.
1562 1562
1563 Human skin has a wide array of touch sensors, each of which 1563 Human skin has a wide array of touch sensors, each of which
2032 cross with a unifom distribution of touch sensors: 2032 cross with a unifom distribution of touch sensors:
2033 2033
2034 #+caption: The touch profile for the touch-cube. Each pure white 2034 #+caption: The touch profile for the touch-cube. Each pure white
2035 #+caption: pixel defines a touch sensitive feeler. 2035 #+caption: pixel defines a touch sensitive feeler.
2036 #+name: touch-cube-uv-map 2036 #+name: touch-cube-uv-map
2037 #+ATTR_LaTeX: :width 10cm 2037 #+ATTR_LaTeX: :width 7cm
2038 [[./images/touch-profile.png]] 2038 [[./images/touch-profile.png]]
2039 2039
2040 #+caption: The touch cube reacts to canonballs. The black, red, 2040 #+caption: The touch cube reacts to canonballs. The black, red,
2041 #+caption: and white cross on the right is a visual display of 2041 #+caption: and white cross on the right is a visual display of
2042 #+caption: the creature's touch. White means that it is feeling 2042 #+caption: the creature's touch. White means that it is feeling
2047 #+caption: part of the ground. 2047 #+caption: part of the ground.
2048 #+name: touch-cube-uv-map 2048 #+name: touch-cube-uv-map
2049 #+ATTR_LaTeX: :width 15cm 2049 #+ATTR_LaTeX: :width 15cm
2050 [[./images/touch-cube.png]] 2050 [[./images/touch-cube.png]]
2051 2051
2052 ** COMMENT Proprioception is the sense that makes everything ``real'' 2052 ** Proprioception is the sense that makes everything ``real''
2053 2053
2054 Close your eyes, and touch your nose with your right index finger. 2054 Close your eyes, and touch your nose with your right index finger.
2055 How did you do it? You could not see your hand, and neither your 2055 How did you do it? You could not see your hand, and neither your
2056 hand nor your nose could use the sense of touch to guide the path 2056 hand nor your nose could use the sense of touch to guide the path
2057 of your hand. There are no sound cues, and Taste and Smell 2057 of your hand. There are no sound cues, and Taste and Smell
2181 #+caption: pitch, and White is roll. 2181 #+caption: pitch, and White is roll.
2182 #+name: proprio 2182 #+name: proprio
2183 #+ATTR_LaTeX: :width 11cm 2183 #+ATTR_LaTeX: :width 11cm
2184 [[./images/proprio.png]] 2184 [[./images/proprio.png]]
2185 2185
2186 ** COMMENT Muscles are both effectors and sensors 2186 ** Muscles are both effectors and sensors
2187 2187
2188 Surprisingly enough, terrestrial creatures only move by using 2188 Surprisingly enough, terrestrial creatures only move by using
2189 torque applied about their joints. There's not a single straight 2189 torque applied about their joints. There's not a single straight
2190 line of force in the human body at all! (A straight line of force 2190 line of force in the human body at all! (A straight line of force
2191 would correspond to some sort of jet or rocket propulsion.) 2191 would correspond to some sort of jet or rocket propulsion.)
2324 function: it returns the percent of the total muscle strength that 2324 function: it returns the percent of the total muscle strength that
2325 is currently being employed. This is analogous to muscle tension 2325 is currently being employed. This is analogous to muscle tension
2326 in humans and completes the sense of proprioception begun in the 2326 in humans and completes the sense of proprioception begun in the
2327 last section. 2327 last section.
2328 2328
2329 ** COMMENT =CORTEX= brings complex creatures to life! 2329 ** =CORTEX= brings complex creatures to life!
2330 2330
2331 The ultimate test of =CORTEX= is to create a creature with the full 2331 The ultimate test of =CORTEX= is to create a creature with the full
2332 gamut of senses and put it though its paces. 2332 gamut of senses and put it though its paces.
2333 2333
2334 With all senses enabled, my right hand model looks like an 2334 With all senses enabled, my right hand model looks like an
2347 2347
2348 With the hand fully rigged with senses, I can run it though a test 2348 With the hand fully rigged with senses, I can run it though a test
2349 that will test everything. 2349 that will test everything.
2350 2350
2351 #+caption: A full test of the hand with all senses. Note expecially 2351 #+caption: A full test of the hand with all senses. Note expecially
2352 #+caption: the interactions the hand has with itself. It feels 2352 #+caption: the interactions the hand has with itself: it feels
2353 #+caption: its own palm and fingers, and when it curls its fingers, 2353 #+caption: its own palm and fingers, and when it curls its fingers,
2354 #+caption: it sees them with its eye (which is located in the center 2354 #+caption: it sees them with its eye (which is located in the center
2355 #+caption: of the palm. The red block appears with a pure tone sound. 2355 #+caption: of the palm. The red block appears with a pure tone sound.
2356 #+caption: The hand then uses its muscles to launch the cube! 2356 #+caption: The hand then uses its muscles to launch the cube!
2357 #+name: integration 2357 #+name: integration
2358 #+ATTR_LaTeX: :width 16cm 2358 #+ATTR_LaTeX: :width 16cm
2359 [[./images/integration.png]] 2359 [[./images/integration.png]]
2360 2360
2361 ** COMMENT =CORTEX= enables many possiblities for further research 2361 ** =CORTEX= enables many possiblities for further research
2362 2362
2363 Often times, the hardest part of building a system involving 2363 Often times, the hardest part of building a system involving
2364 creatures is dealing with physics and graphics. =CORTEX= removes 2364 creatures is dealing with physics and graphics. =CORTEX= removes
2365 much of this initial difficulty and leaves researchers free to 2365 much of this initial difficulty and leaves researchers free to
2366 directly pursue their ideas. I hope that even undergrads with a 2366 directly pursue their ideas. I hope that even undergrads with a