Mercurial > cortex
comparison thesis/cortex.org @ 495:c816594fbbe6
changes from Dylan.
author | Robert McIntyre <rlm@mit.edu> |
---|---|
date | Sat, 29 Mar 2014 23:01:49 -0400 |
parents | 21b9dcec8d71 |
children | 609356a82760 |
comparison
equal
deleted
inserted
replaced
494:092b60381af0 | 495:c816594fbbe6 |
---|---|
362 new =CORTEX= creatures using Blender, a free 3D modeling program. | 362 new =CORTEX= creatures using Blender, a free 3D modeling program. |
363 | 363 |
364 - I built =EMPATH=, which uses =CORTEX= to identify the actions of | 364 - I built =EMPATH=, which uses =CORTEX= to identify the actions of |
365 a worm-like creature using a computational model of empathy. | 365 a worm-like creature using a computational model of empathy. |
366 | 366 |
367 * Building =CORTEX= | 367 * COMMENT Building =CORTEX= |
368 | 368 |
369 I intend for =CORTEX= to be used as a general-purpose library for | 369 I intend for =CORTEX= to be used as a general-purpose library for |
370 building creatures and outfitting them with senses, so that it will | 370 building creatures and outfitting them with senses, so that it will |
371 be useful for other researchers who want to test out ideas of their | 371 be useful for other researchers who want to test out ideas of their |
372 own. To this end, wherver I have had to make archetictural choices | 372 own. To this end, wherver I have had to make archetictural choices |
373 about =CORTEX=, I have chosen to give as much freedom to the user as | 373 about =CORTEX=, I have chosen to give as much freedom to the user as |
374 possible, so that =CORTEX= may be used for things I have not | 374 possible, so that =CORTEX= may be used for things I have not |
375 forseen. | 375 forseen. |
376 | 376 |
377 ** COMMENT Simulation or Reality? | 377 ** Simulation or Reality? |
378 | 378 |
379 The most important archetictural decision of all is the choice to | 379 The most important archetictural decision of all is the choice to |
380 use a computer-simulated environemnt in the first place! The world | 380 use a computer-simulated environemnt in the first place! The world |
381 is a vast and rich place, and for now simulations are a very poor | 381 is a vast and rich place, and for now simulations are a very poor |
382 reflection of its complexity. It may be that there is a significant | 382 reflection of its complexity. It may be that there is a significant |
435 time in the simulated world can be slowed down to accommodate the | 435 time in the simulated world can be slowed down to accommodate the |
436 limitations of the character's programming. In terms of cost, | 436 limitations of the character's programming. In terms of cost, |
437 doing everything in software is far cheaper than building custom | 437 doing everything in software is far cheaper than building custom |
438 real-time hardware. All you need is a laptop and some patience. | 438 real-time hardware. All you need is a laptop and some patience. |
439 | 439 |
440 ** COMMENT Because of Time, simulation is perferable to reality | 440 ** Because of Time, simulation is perferable to reality |
441 | 441 |
442 I envision =CORTEX= being used to support rapid prototyping and | 442 I envision =CORTEX= being used to support rapid prototyping and |
443 iteration of ideas. Even if I could put together a well constructed | 443 iteration of ideas. Even if I could put together a well constructed |
444 kit for creating robots, it would still not be enough because of | 444 kit for creating robots, it would still not be enough because of |
445 the scourge of real-time processing. Anyone who wants to test their | 445 the scourge of real-time processing. Anyone who wants to test their |
458 the simulation. The cost is that =CORTEX= can sometimes run slower | 458 the simulation. The cost is that =CORTEX= can sometimes run slower |
459 than real time. This can also be an advantage, however --- | 459 than real time. This can also be an advantage, however --- |
460 simulations of very simple creatures in =CORTEX= generally run at | 460 simulations of very simple creatures in =CORTEX= generally run at |
461 40x on my machine! | 461 40x on my machine! |
462 | 462 |
463 ** COMMENT What is a sense? | 463 ** What is a sense? |
464 | 464 |
465 If =CORTEX= is to support a wide variety of senses, it would help | 465 If =CORTEX= is to support a wide variety of senses, it would help |
466 to have a better understanding of what a ``sense'' actually is! | 466 to have a better understanding of what a ``sense'' actually is! |
467 While vision, touch, and hearing all seem like they are quite | 467 While vision, touch, and hearing all seem like they are quite |
468 different things, I was supprised to learn during the course of | 468 different things, I was supprised to learn during the course of |
536 #+caption: density of touch sensors at the tip. | 536 #+caption: density of touch sensors at the tip. |
537 #+name: finger-side-view | 537 #+name: finger-side-view |
538 #+ATTR_LaTeX: :width 10cm | 538 #+ATTR_LaTeX: :width 10cm |
539 [[./images/finger-1.png]] | 539 [[./images/finger-1.png]] |
540 | 540 |
541 ** COMMENT Video game engines provide ready-made physics and shading | 541 ** Video game engines provide ready-made physics and shading |
542 | 542 |
543 I did not need to write my own physics simulation code or shader to | 543 I did not need to write my own physics simulation code or shader to |
544 build =CORTEX=. Doing so would lead to a system that is impossible | 544 build =CORTEX=. Doing so would lead to a system that is impossible |
545 for anyone but myself to use anyway. Instead, I use a video game | 545 for anyone but myself to use anyway. Instead, I use a video game |
546 engine as a base and modify it to accomodate the additional needs | 546 engine as a base and modify it to accomodate the additional needs |
560 texture to a model. Finally, because video game engines support a | 560 texture to a model. Finally, because video game engines support a |
561 large number of users, as long as =CORTEX= doesn't stray too far | 561 large number of users, as long as =CORTEX= doesn't stray too far |
562 from the base system, other researchers can turn to this community | 562 from the base system, other researchers can turn to this community |
563 for help when doing their research. | 563 for help when doing their research. |
564 | 564 |
565 ** COMMENT =CORTEX= is based on jMonkeyEngine3 | 565 ** =CORTEX= is based on jMonkeyEngine3 |
566 | 566 |
567 While preparing to build =CORTEX= I studied several video game | 567 While preparing to build =CORTEX= I studied several video game |
568 engines to see which would best serve as a base. The top contenders | 568 engines to see which would best serve as a base. The top contenders |
569 were: | 569 were: |
570 | 570 |
602 I chose jMonkeyEngine3 because it because it had the most features | 602 I chose jMonkeyEngine3 because it because it had the most features |
603 out of all the free projects I looked at, and because I could then | 603 out of all the free projects I looked at, and because I could then |
604 write my code in clojure, an implementation of =LISP= that runs on | 604 write my code in clojure, an implementation of =LISP= that runs on |
605 the JVM. | 605 the JVM. |
606 | 606 |
607 ** COMMENT =CORTEX= uses Blender to create creature models | 607 ** =CORTEX= uses Blender to create creature models |
608 | 608 |
609 For the simple worm-like creatures I will use later on in this | 609 For the simple worm-like creatures I will use later on in this |
610 thesis, I could define a simple API in =CORTEX= that would allow | 610 thesis, I could define a simple API in =CORTEX= that would allow |
611 one to create boxes, spheres, etc., and leave that API as the sole | 611 one to create boxes, spheres, etc., and leave that API as the sole |
612 way to create creatures. However, for =CORTEX= to truly be useful | 612 way to create creatures. However, for =CORTEX= to truly be useful |
639 #+caption: of muscles, ears, eyes, or joints. | 639 #+caption: of muscles, ears, eyes, or joints. |
640 #+name: sense-nodes | 640 #+name: sense-nodes |
641 #+ATTR_LaTeX: :width 10cm | 641 #+ATTR_LaTeX: :width 10cm |
642 [[./images/empty-sense-nodes.png]] | 642 [[./images/empty-sense-nodes.png]] |
643 | 643 |
644 ** COMMENT Bodies are composed of segments connected by joints | 644 ** Bodies are composed of segments connected by joints |
645 | 645 |
646 Blender is a general purpose animation tool, which has been used in | 646 Blender is a general purpose animation tool, which has been used in |
647 the past to create high quality movies such as Sintel | 647 the past to create high quality movies such as Sintel |
648 \cite{sintel}. Though Blender can model and render even complicated | 648 \cite{sintel}. Though Blender can model and render even complicated |
649 things like water, it is crucual to keep models that are meant to | 649 things like water, it is crucual to keep models that are meant to |
955 #+caption: simulation environment. | 955 #+caption: simulation environment. |
956 #+name: name | 956 #+name: name |
957 #+ATTR_LaTeX: :width 15cm | 957 #+ATTR_LaTeX: :width 15cm |
958 [[./images/physical-hand.png]] | 958 [[./images/physical-hand.png]] |
959 | 959 |
960 ** COMMENT Eyes reuse standard video game components | 960 ** Eyes reuse standard video game components |
961 | 961 |
962 Vision is one of the most important senses for humans, so I need to | 962 Vision is one of the most important senses for humans, so I need to |
963 build a simulated sense of vision for my AI. I will do this with | 963 build a simulated sense of vision for my AI. I will do this with |
964 simulated eyes. Each eye can be independently moved and should see | 964 simulated eyes. Each eye can be independently moved and should see |
965 its own version of the world depending on where it is. | 965 its own version of the world depending on where it is. |
1252 | 1252 |
1253 This vision code has already been absorbed by the jMonkeyEngine | 1253 This vision code has already been absorbed by the jMonkeyEngine |
1254 community and is now (in modified form) part of a system for | 1254 community and is now (in modified form) part of a system for |
1255 capturing in-game video to a file. | 1255 capturing in-game video to a file. |
1256 | 1256 |
1257 ** COMMENT Hearing is hard; =CORTEX= does it right | 1257 ** Hearing is hard; =CORTEX= does it right |
1258 | 1258 |
1259 At the end of this section I will have simulated ears that work the | 1259 At the end of this section I will have simulated ears that work the |
1260 same way as the simulated eyes in the last section. I will be able to | 1260 same way as the simulated eyes in the last section. I will be able to |
1261 place any number of ear-nodes in a blender file, and they will bind to | 1261 place any number of ear-nodes in a blender file, and they will bind to |
1262 the closest physical object and follow it as it moves around. Each ear | 1262 the closest physical object and follow it as it moves around. Each ear |
1553 | 1553 |
1554 This system of hearing has also been co-opted by the | 1554 This system of hearing has also been co-opted by the |
1555 jMonkeyEngine3 community and is used to record audio for demo | 1555 jMonkeyEngine3 community and is used to record audio for demo |
1556 videos. | 1556 videos. |
1557 | 1557 |
1558 ** COMMENT Touch uses hundreds of hair-like elements | 1558 ** Touch uses hundreds of hair-like elements |
1559 | 1559 |
1560 Touch is critical to navigation and spatial reasoning and as such I | 1560 Touch is critical to navigation and spatial reasoning and as such I |
1561 need a simulated version of it to give to my AI creatures. | 1561 need a simulated version of it to give to my AI creatures. |
1562 | 1562 |
1563 Human skin has a wide array of touch sensors, each of which | 1563 Human skin has a wide array of touch sensors, each of which |
2032 cross with a unifom distribution of touch sensors: | 2032 cross with a unifom distribution of touch sensors: |
2033 | 2033 |
2034 #+caption: The touch profile for the touch-cube. Each pure white | 2034 #+caption: The touch profile for the touch-cube. Each pure white |
2035 #+caption: pixel defines a touch sensitive feeler. | 2035 #+caption: pixel defines a touch sensitive feeler. |
2036 #+name: touch-cube-uv-map | 2036 #+name: touch-cube-uv-map |
2037 #+ATTR_LaTeX: :width 10cm | 2037 #+ATTR_LaTeX: :width 7cm |
2038 [[./images/touch-profile.png]] | 2038 [[./images/touch-profile.png]] |
2039 | 2039 |
2040 #+caption: The touch cube reacts to canonballs. The black, red, | 2040 #+caption: The touch cube reacts to canonballs. The black, red, |
2041 #+caption: and white cross on the right is a visual display of | 2041 #+caption: and white cross on the right is a visual display of |
2042 #+caption: the creature's touch. White means that it is feeling | 2042 #+caption: the creature's touch. White means that it is feeling |
2047 #+caption: part of the ground. | 2047 #+caption: part of the ground. |
2048 #+name: touch-cube-uv-map | 2048 #+name: touch-cube-uv-map |
2049 #+ATTR_LaTeX: :width 15cm | 2049 #+ATTR_LaTeX: :width 15cm |
2050 [[./images/touch-cube.png]] | 2050 [[./images/touch-cube.png]] |
2051 | 2051 |
2052 ** COMMENT Proprioception is the sense that makes everything ``real'' | 2052 ** Proprioception is the sense that makes everything ``real'' |
2053 | 2053 |
2054 Close your eyes, and touch your nose with your right index finger. | 2054 Close your eyes, and touch your nose with your right index finger. |
2055 How did you do it? You could not see your hand, and neither your | 2055 How did you do it? You could not see your hand, and neither your |
2056 hand nor your nose could use the sense of touch to guide the path | 2056 hand nor your nose could use the sense of touch to guide the path |
2057 of your hand. There are no sound cues, and Taste and Smell | 2057 of your hand. There are no sound cues, and Taste and Smell |
2181 #+caption: pitch, and White is roll. | 2181 #+caption: pitch, and White is roll. |
2182 #+name: proprio | 2182 #+name: proprio |
2183 #+ATTR_LaTeX: :width 11cm | 2183 #+ATTR_LaTeX: :width 11cm |
2184 [[./images/proprio.png]] | 2184 [[./images/proprio.png]] |
2185 | 2185 |
2186 ** COMMENT Muscles are both effectors and sensors | 2186 ** Muscles are both effectors and sensors |
2187 | 2187 |
2188 Surprisingly enough, terrestrial creatures only move by using | 2188 Surprisingly enough, terrestrial creatures only move by using |
2189 torque applied about their joints. There's not a single straight | 2189 torque applied about their joints. There's not a single straight |
2190 line of force in the human body at all! (A straight line of force | 2190 line of force in the human body at all! (A straight line of force |
2191 would correspond to some sort of jet or rocket propulsion.) | 2191 would correspond to some sort of jet or rocket propulsion.) |
2324 function: it returns the percent of the total muscle strength that | 2324 function: it returns the percent of the total muscle strength that |
2325 is currently being employed. This is analogous to muscle tension | 2325 is currently being employed. This is analogous to muscle tension |
2326 in humans and completes the sense of proprioception begun in the | 2326 in humans and completes the sense of proprioception begun in the |
2327 last section. | 2327 last section. |
2328 | 2328 |
2329 ** COMMENT =CORTEX= brings complex creatures to life! | 2329 ** =CORTEX= brings complex creatures to life! |
2330 | 2330 |
2331 The ultimate test of =CORTEX= is to create a creature with the full | 2331 The ultimate test of =CORTEX= is to create a creature with the full |
2332 gamut of senses and put it though its paces. | 2332 gamut of senses and put it though its paces. |
2333 | 2333 |
2334 With all senses enabled, my right hand model looks like an | 2334 With all senses enabled, my right hand model looks like an |
2347 | 2347 |
2348 With the hand fully rigged with senses, I can run it though a test | 2348 With the hand fully rigged with senses, I can run it though a test |
2349 that will test everything. | 2349 that will test everything. |
2350 | 2350 |
2351 #+caption: A full test of the hand with all senses. Note expecially | 2351 #+caption: A full test of the hand with all senses. Note expecially |
2352 #+caption: the interactions the hand has with itself. It feels | 2352 #+caption: the interactions the hand has with itself: it feels |
2353 #+caption: its own palm and fingers, and when it curls its fingers, | 2353 #+caption: its own palm and fingers, and when it curls its fingers, |
2354 #+caption: it sees them with its eye (which is located in the center | 2354 #+caption: it sees them with its eye (which is located in the center |
2355 #+caption: of the palm. The red block appears with a pure tone sound. | 2355 #+caption: of the palm. The red block appears with a pure tone sound. |
2356 #+caption: The hand then uses its muscles to launch the cube! | 2356 #+caption: The hand then uses its muscles to launch the cube! |
2357 #+name: integration | 2357 #+name: integration |
2358 #+ATTR_LaTeX: :width 16cm | 2358 #+ATTR_LaTeX: :width 16cm |
2359 [[./images/integration.png]] | 2359 [[./images/integration.png]] |
2360 | 2360 |
2361 ** COMMENT =CORTEX= enables many possiblities for further research | 2361 ** =CORTEX= enables many possiblities for further research |
2362 | 2362 |
2363 Often times, the hardest part of building a system involving | 2363 Often times, the hardest part of building a system involving |
2364 creatures is dealing with physics and graphics. =CORTEX= removes | 2364 creatures is dealing with physics and graphics. =CORTEX= removes |
2365 much of this initial difficulty and leaves researchers free to | 2365 much of this initial difficulty and leaves researchers free to |
2366 directly pursue their ideas. I hope that even undergrads with a | 2366 directly pursue their ideas. I hope that even undergrads with a |