Mercurial > cortex

annotate thesis/org/roadmap.org @ 402:a533a0038bd7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
now have overall plan.
author Robert McIntyre <rlm@mit.edu>
date Mon, 17 Mar 2014 00:26:15 -0400
parents 7ee735a836da
children 92acbe7e5c91
rev   line source
rlm@401 1 In order for this to be a reasonable thesis that I can be proud of,
rlm@401 2 what are the /minimum/ number of things I need to get done?
rlm@401 3
rlm@401 4
rlm@401 5 * worm OR hand registration
rlm@401 6 - training from a few examples (2 to start out)
rlm@401 7 - aligning the body with the scene
rlm@401 8 - generating sensory data
rlm@401 9 - matching previous labeled examples using dot-products or some
rlm@401 10 other basic thing
rlm@401 11 - showing that it works with different views
rlm@401 12
rlm@401 13 * first draft
rlm@401 14 - draft of thesis without bibliography or formatting
rlm@401 15 - should have basic experiment and have full description of
rlm@401 16 framework with code
rlm@401 17 - review with Winston
rlm@401 18
rlm@401 19 * final draft
rlm@401 20 - implement stretch goals from Winston if possible
rlm@401 21 - complete final formatting and submit
rlm@401 22
rlm@401 23
rlm@401 24
rlm@401 25
rlm@401 26 * CORTEX
rlm@401 27 DEADLINE: <2014-05-09 Fri>
rlm@401 28 SHIT THAT'S IN 67 DAYS!!!
rlm@401 29
rlm@401 30 ** TODO program simple feature matching code for the worm's segments
rlm@401 31 DEADLINE: <2014-03-11 Tue>
rlm@401 32 Subgoals:
rlm@401 33 *** DONE Get cortex working again, run tests, no jmonkeyengine updates
rlm@401 34 CLOSED: [2014-03-03 Mon 22:07] SCHEDULED: <2014-03-03 Mon>
rlm@401 35 *** DONE get blender working again
rlm@401 36 CLOSED: [2014-03-03 Mon 22:43] SCHEDULED: <2014-03-03 Mon>
rlm@401 37 *** DONE make sparce touch worm segment in blender
rlm@401 38 CLOSED: [2014-03-03 Mon 23:16] SCHEDULED: <2014-03-03 Mon>
rlm@401 39 CLOCK: [2014-03-03 Mon 22:44]--[2014-03-03 Mon 23:16] => 0:32
rlm@401 40 *** DONE make multi-segment touch worm with touch sensors and display
rlm@401 41 CLOSED: [2014-03-03 Mon 23:54] SCHEDULED: <2014-03-03 Mon>
rlm@401 42 CLOCK: [2014-03-03 Mon 23:17]--[2014-03-03 Mon 23:54] => 0:37
rlm@401 43
rlm@401 44
rlm@401 45 *** DONE Make a worm wiggle and curl
rlm@401 46 CLOSED: [2014-03-04 Tue 23:03] SCHEDULED: <2014-03-04 Tue>
rlm@401 47 *** TODO work on alignment for the worm (can "cheat")
rlm@401 48 SCHEDULED: <2014-03-05 Wed>
rlm@401 49
rlm@401 50 ** First draft
rlm@401 51 DEADLINE: <2014-03-14 Fri>
rlm@401 52 Subgoals:
rlm@401 53 *** Writeup new worm experiments.
rlm@401 54 *** Triage implementation code and get it into chapter form.
rlm@401 55
rlm@401 56
rlm@401 57
rlm@401 58
rlm@401 59
rlm@401 60 ** for today
rlm@401 61
rlm@401 62 - guided worm :: control the worm with the keyboard. Useful for
rlm@401 63 testing the body-centered recog scripts, and for
rlm@401 64 preparing a cool demo video.
rlm@401 65
rlm@401 66 - body-centered recognition :: detect actions using hard coded
rlm@401 67 body-centered scripts.
rlm@401 68
rlm@401 69 - cool demo video of the worm being moved and recognizing things ::
rlm@401 70 will be a neat part of the thesis.
rlm@401 71
rlm@401 72 - thesis export :: refactoring and organization of code so that it
rlm@401 73 spits out a thesis in addition to the web page.
rlm@401 74
rlm@401 75 - video alignment :: analyze the frames of a video in order to align
rlm@401 76 the worm. Requires body-centered recognition. Can "cheat".
rlm@401 77
rlm@401 78 - smoother actions :: use debugging controls to directly influence the
rlm@401 79 demo actions, and to generate recoginition procedures.
rlm@401 80
rlm@401 81 - degenerate video demonstration :: show the system recognizing a
rlm@401 82 curled worm from dead on. Crowning achievement of thesis.
rlm@401 83
rlm@401 84 ** Ordered from easiest to hardest
rlm@401 85
rlm@401 86 Just report the positions of everything. I don't think that this
rlm@401 87 necessairly shows anything usefull.
rlm@401 88
rlm@401 89 Worm-segment vision -- you initialize a view of the worm, but instead
rlm@401 90 of pixels you use labels via ray tracing. Has the advantage of still
rlm@401 91 allowing for visual occlusion, but reliably identifies the objects,
rlm@401 92 even without rainbow coloring. You can code this as an image.
rlm@401 93
rlm@401 94 Same as above, except just with worm/non-worm labels.
rlm@401 95
rlm@401 96 Color code each worm segment and then recognize them using blob
rlm@401 97 detectors. Then you solve for the perspective and the action
rlm@401 98 simultaneously.
rlm@401 99
rlm@401 100 The entire worm can be colored the same, high contrast color against a
rlm@401 101 nearly black background.
rlm@401 102
rlm@401 103 "Rooted" vision. You give the exact coordinates of ONE piece of the
rlm@401 104 worm, but the algorithm figures out the rest.
rlm@401 105
rlm@401 106 More rooted vision -- start off the entire worm with one posistion.
rlm@401 107
rlm@401 108 The right way to do alignment is to use motion over multiple frames to
rlm@401 109 snap individual pieces of the model into place sharing and
rlm@401 110 propragating the individual alignments over the whole model. We also
rlm@401 111 want to limit the alignment search to just those actions we are
rlm@401 112 prepared to identify. This might mean that I need some small "micro
rlm@401 113 actions" such as the individual movements of the worm pieces.
rlm@401 114
rlm@401 115 Get just the centers of each segment projected onto the imaging
rlm@401 116 plane. (best so far).
rlm@401 117
rlm@401 118
rlm@401 119 Repertoire of actions + video frames -->
rlm@401 120 directed multi-frame-search alg
rlm@401 121
rlm@401 122
rlm@401 123
rlm@401 124
rlm@401 125
rlm@401 126
rlm@401 127 !! Could also have a bounding box around the worm provided by
rlm@401 128 filtering the worm/non-worm render, and use bbbgs. As a bonus, I get
rlm@401 129 to include bbbgs in my thesis! Could finally do that recursive things
rlm@401 130 where I make bounding boxes be those things that give results that
rlm@401 131 give good bounding boxes. If I did this I could use a disruptive
rlm@401 132 pattern on the worm.
rlm@401 133
rlm@401 134 Re imagining using default textures is very simple for this system,
rlm@401 135 but hard for others.
rlm@401 136
rlm@401 137
rlm@401 138 Want to demonstrate, at minimum, alignment of some model of the worm
rlm@401 139 to the video, and a lookup of the action by simulated perception.
rlm@401 140
rlm@401 141 note: the purple/white points is a very beautiful texture, because
rlm@401 142 when it moves slightly, the white dots look like they're
rlm@401 143 twinkling. Would look even better if it was a darker purple. Also
rlm@401 144 would look better more spread out.
rlm@401 145
rlm@401 146
rlm@401 147 embed assumption of one frame of view, search by moving around in
rlm@401 148 simulated world.
rlm@401 149
rlm@401 150 Allowed to limit search by setting limits to a hemisphere around the
rlm@401 151 imagined worm! This limits scale also.
rlm@401 152
rlm@401 153
rlm@401 154
rlm@401 155
rlm@401 156
rlm@401 157 !! Limited search with worm/non-worm rendering.
rlm@401 158 How much inverse kinematics do we have to do?
rlm@401 159 What about cached (allowed state-space) paths, derived from labeled
rlm@401 160 training. You have to lead from one to another.
rlm@401 161
rlm@401 162 What about initial state? Could start the input videos at a specific
rlm@401 163 state, then just match that explicitly.
rlm@401 164
rlm@401 165 !! The training doesn't have to be labeled -- you can just move around
rlm@401 166 for a while!!
rlm@401 167
rlm@401 168 !! Limited search with motion based alignment.
rlm@401 169
rlm@401 170
rlm@401 171
rlm@401 172
rlm@401 173 "play arounds" can establish a chain of linked sensoriums. Future
rlm@401 174 matches must fall into one of the already experienced things, and once
rlm@401 175 they do, it greatly limits the things that are possible in the future.
rlm@401 176
rlm@401 177
rlm@401 178 frame differences help to detect muscle exertion.
rlm@401 179
rlm@401 180 Can try to match on a few "representative" frames. Can also just have
rlm@401 181 a few "bodies" in various states which we try to match.
rlm@401 182
rlm@401 183
rlm@401 184
rlm@401 185 Paths through state-space have the exact same signature as
rlm@401 186 simulation. BUT, these can be searched in parallel and don't interfere
rlm@401 187 with each other.
rlm@401 188
rlm@401 189
rlm@402 190
rlm@402 191
rlm@402 192 ** Final stretch up to First Draft
rlm@402 193
rlm@402 194 - complete debug control of worm
rlm@402 195 - add phi-space output to debug control
rlm@402 196 - complete phi-stream action predicatates; test them with debug
rlm@402 197 control
rlm@402 198 - create test videos, also record positions of worm segments
rlm@402 199 - complete proprioception based movement lookup in phi-space
rlm@402 200 - Collect intro, worm-learn and cortex creation into draft thesis.
rlm@402 201
rlm@402 202