Mercurial > cortex

     1 In order for this to be a reasonable thesis that I can be proud of,

     2 what are the /minimum/ number of things I need to get done?

     3 

     4 

     5 * worm OR hand registration

     6   - training from a few examples (2 to start out)

     7   - aligning the body with the scene

     8   - generating sensory data

     9   - matching previous labeled examples using dot-products or some

    10     other basic thing

    11   - showing that it works with different views

    12 

    13 * first draft

    14   - draft of thesis without bibliography or formatting

    15   - should have basic experiment and have full description of

    16     framework with code

    17   - review with Winston

    18   

    19 * final draft

    20   - implement stretch goals from Winston if possible

    21   - complete final formatting and submit

    22 

    23 <<<<<<< local

    24 * DONE CORTEX

    25   CLOSED: [2014-05-29 Thu 17:01] DEADLINE: <2014-05-09 Fri>

    26   SHIT THAT'S IN 67 DAYS!!!

    27 

    28 ** program simple feature matching code for the worm's segments

    29 

    30 Subgoals:

    31 *** DONE Get cortex working again, run tests, no jmonkeyengine updates

    32     CLOSED: [2014-03-03 Mon 22:07] SCHEDULED: <2014-03-03 Mon>

    33 *** DONE get blender working again

    34     CLOSED: [2014-03-03 Mon 22:43] SCHEDULED: <2014-03-03 Mon>

    35 *** DONE make sparce touch worm segment in blender

    36     CLOSED: [2014-03-03 Mon 23:16] SCHEDULED: <2014-03-03 Mon>

    37     CLOCK: [2014-03-03 Mon 22:44]--[2014-03-03 Mon 23:16] =>  0:32

    38 *** DONE make multi-segment touch worm with touch sensors and display

    39     CLOSED: [2014-03-03 Mon 23:54] SCHEDULED: <2014-03-03 Mon>

    40 

    41 *** DONE Make a worm wiggle and curl

    42     CLOSED: [2014-03-04 Tue 23:03] SCHEDULED: <2014-03-04 Tue>

    43 

    44 

    45 ** First draft

    46 

    47 Subgoals:

    48 *** Writeup new worm experiments.

    49 *** Triage implementation code and get it into chapter form.

    50 

    51 

    52 

    53  

    54 

    55 ** for today

    56 

    57 - guided worm :: control the worm with the keyboard. Useful for

    58                  testing the body-centered recog scripts, and for

    59                  preparing a cool demo video.

    60 

    61 - body-centered recognition :: detect actions using hard coded

    62      body-centered scripts. 

    63 

    64 - cool demo video of the worm being moved and recognizing things ::

    65      will be a neat part of the thesis.

    66 

    67 - thesis export :: refactoring and organization of code so that it

    68                    spits out a thesis in addition to the web page.

    69 

    70 - video alignment :: analyze the frames of a video in order to align

    71      the worm. Requires body-centered recognition. Can "cheat".

    72 

    73 - smoother actions :: use debugging controls to directly influence the

    74      demo actions, and to generate recoginition procedures.

    75 

    76 - degenerate video demonstration :: show the system recognizing a

    77      curled worm from dead on. Crowning achievement of thesis.

    78 

    79 ** Ordered from easiest to hardest

    80 

    81 Just report the positions of everything. I don't think that this

    82 necessairly shows anything usefull.

    83 

    84 Worm-segment vision -- you initialize a view of the worm, but instead

    85 of pixels you use labels via ray tracing. Has the advantage of still

    86 allowing for visual occlusion, but reliably identifies the objects,

    87 even without rainbow coloring. You can code this as an image. 

    88 

    89 Same as above, except just with worm/non-worm labels.

    90 

    91 Color code each worm segment and then recognize them using blob

    92 detectors. Then you solve for the perspective and the action

    93 simultaneously.

    94 

    95 The entire worm can be colored the same, high contrast color against a

    96 nearly black background.

    97 

    98 "Rooted" vision. You give the exact coordinates of ONE piece of the

    99 worm, but the algorithm figures out the rest.

   100 

   101 More rooted vision -- start off the entire worm with one posistion.

   102 

   103 The right way to do alignment is to use motion over multiple frames to

   104 snap individual pieces of the model into place sharing and

   105 propragating the individual alignments over the whole model. We also

   106 want to limit the alignment search to just those actions we are

   107 prepared to identify. This might mean that I need some small "micro

   108 actions" such as the individual movements of the worm pieces.

   109 

   110 Get just the centers of each segment projected onto the imaging

   111 plane. (best so far).

   112 

   113 

   114 Repertoire of actions  +  video frames -->

   115    directed multi-frame-search alg

   116 

   117 

   118 

   119 

   120 

   121 

   122 !! Could also have a bounding box around the worm provided by

   123 filtering the worm/non-worm render, and use bbbgs. As a bonus, I get

   124 to include bbbgs in my thesis! Could finally do that recursive things

   125 where I make bounding boxes be those things that give results that

   126 give good bounding boxes. If I did this I could use a disruptive

   127 pattern on the worm.

   128 

   129 Re imagining using default textures is very simple for this system,

   130 but hard for others.

   131 

   132 

   133 Want to demonstrate, at minimum, alignment of some model of the worm

   134 to the video, and a lookup of the action by simulated perception.

   135 

   136 note: the purple/white points is a very beautiful texture, because

   137 when it moves slightly, the white dots look like they're

   138 twinkling. Would look even better if it was a darker purple. Also

   139 would look better more spread out.

   140 

   141 

   142 embed assumption of one frame of view, search by moving around in

   143 simulated world.

   144 

   145 Allowed to limit search by setting limits to a hemisphere around the

   146 imagined worm! This limits scale also.

   147 

   148 

   149 

   150 

   151 

   152 !! Limited search with worm/non-worm rendering. 

   153 How much inverse kinematics do we have to do?

   154 What about cached (allowed state-space) paths, derived from labeled

   155 training. You have to lead from one to another.

   156 

   157 What about initial state? Could start the input videos at a specific

   158 state, then just match that explicitly.

   159 

   160 !! The training doesn't have to be labeled -- you can just move around

   161 for a while!!

   162 

   163 !! Limited search with motion based alignment.

   164 

   165 

   166 

   167 

   168 "play arounds" can establish a chain of linked sensoriums. Future

   169 matches must fall into one of the already experienced things, and once

   170 they do, it greatly limits the things that are possible in the future.

   171 

   172 

   173 frame differences help to detect muscle exertion.

   174 

   175 Can try to match on a few "representative" frames. Can also just have

   176 a few "bodies" in various states which we try to match.

   177 

   178 

   179 

   180 Paths through state-space have the exact same signature as

   181 simulation. BUT, these can be searched in parallel and don't interfere

   182 with each other.

   183 

   184 

   185 

   186 

   187 ** Final stretch up to First Draft

   188 

   189 *** DONE complete debug control of worm

   190     CLOSED: [2014-03-17 Mon 17:29] SCHEDULED: <2014-03-17 Mon>

   191     CLOCK: [2014-03-17 Mon 14:01]--[2014-03-17 Mon 17:29] =>  3:28

   192 *** DONE add phi-space output to debug control

   193     CLOSED: [2014-03-17 Mon 17:42] SCHEDULED: <2014-03-17 Mon>

   194     CLOCK: [2014-03-17 Mon 17:31]--[2014-03-17 Mon 17:42] =>  0:11

   195 

   196 *** DONE complete automatic touch partitioning

   197     CLOSED: [2014-03-18 Tue 21:43] SCHEDULED: <2014-03-18 Tue>

   198 *** DONE complete cyclic predicate

   199     CLOSED: [2014-03-19 Wed 16:34] SCHEDULED: <2014-03-18 Tue>

   200     CLOCK: [2014-03-19 Wed 13:16]--[2014-03-19 Wed 16:34] =>  3:18

   201 *** DONE complete three phi-stream action predicatates; test them with debug control

   202     CLOSED: [2014-03-19 Wed 16:35] SCHEDULED: <2014-03-17 Mon>

   203     CLOCK: [2014-03-18 Tue 18:36]--[2014-03-18 Tue 21:43] =>  3:07

   204     CLOCK: [2014-03-18 Tue 18:34]--[2014-03-18 Tue 18:36] =>  0:02

   205     CLOCK: [2014-03-17 Mon 19:19]--[2014-03-17 Mon 21:19] =>  2:00

   206 *** DONE build an automatic "do all the things" sequence.

   207     CLOSED: [2014-03-19 Wed 16:55] SCHEDULED: <2014-03-19 Wed>

   208     CLOCK: [2014-03-19 Wed 16:53]--[2014-03-19 Wed 16:55] =>  0:02

   209 *** DONE implement proprioception based movement lookup in phi-space

   210     CLOSED: [2014-03-19 Wed 22:04] SCHEDULED: <2014-03-19 Wed>

   211     CLOCK: [2014-03-19 Wed 19:32]--[2014-03-19 Wed 22:04] =>  2:32

   212 *** DONE make proprioception reference phi-space indexes

   213     CLOSED: [2014-03-19 Wed 22:47] SCHEDULED: <2014-03-19 Wed>

   214 

   215 

   216 *** DONE create test videos, also record positions of worm segments

   217     CLOSED: [2014-03-20 Thu 22:02] SCHEDULED: <2014-03-19 Wed>

   218 

   219 *** TODO Collect intro, worm-learn and cortex creation into draft thesis. 

   220
author	Robert McIntyre <rlm@mit.edu>
date	Mon, 02 Mar 2015 10:04:16 -0800
parents	9647f0168287
children