rlm@401: In order for this to be a reasonable thesis that I can be proud of, rlm@401: what are the /minimum/ number of things I need to get done? rlm@401: rlm@401: rlm@401: * worm OR hand registration rlm@401: - training from a few examples (2 to start out) rlm@401: - aligning the body with the scene rlm@401: - generating sensory data rlm@401: - matching previous labeled examples using dot-products or some rlm@401: other basic thing rlm@401: - showing that it works with different views rlm@401: rlm@401: * first draft rlm@401: - draft of thesis without bibliography or formatting rlm@401: - should have basic experiment and have full description of rlm@401: framework with code rlm@401: - review with Winston rlm@401: rlm@401: * final draft rlm@401: - implement stretch goals from Winston if possible rlm@401: - complete final formatting and submit rlm@401: rlm@401: * CORTEX rlm@401: DEADLINE: <2014-05-09 Fri> rlm@401: SHIT THAT'S IN 67 DAYS!!! rlm@401: rlm@403: ** program simple feature matching code for the worm's segments rlm@403: rlm@401: Subgoals: rlm@401: *** DONE Get cortex working again, run tests, no jmonkeyengine updates rlm@401: CLOSED: [2014-03-03 Mon 22:07] SCHEDULED: <2014-03-03 Mon> rlm@401: *** DONE get blender working again rlm@401: CLOSED: [2014-03-03 Mon 22:43] SCHEDULED: <2014-03-03 Mon> rlm@401: *** DONE make sparce touch worm segment in blender rlm@401: CLOSED: [2014-03-03 Mon 23:16] SCHEDULED: <2014-03-03 Mon> rlm@401: CLOCK: [2014-03-03 Mon 22:44]--[2014-03-03 Mon 23:16] => 0:32 rlm@401: *** DONE make multi-segment touch worm with touch sensors and display rlm@401: CLOSED: [2014-03-03 Mon 23:54] SCHEDULED: <2014-03-03 Mon> rlm@401: rlm@401: *** DONE Make a worm wiggle and curl rlm@401: CLOSED: [2014-03-04 Tue 23:03] SCHEDULED: <2014-03-04 Tue> rlm@403: rlm@401: rlm@401: ** First draft rlm@403: rlm@401: Subgoals: rlm@401: *** Writeup new worm experiments. rlm@401: *** Triage implementation code and get it into chapter form. rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: ** for today rlm@401: rlm@401: - guided worm :: control the worm with the keyboard. Useful for rlm@401: testing the body-centered recog scripts, and for rlm@401: preparing a cool demo video. rlm@401: rlm@401: - body-centered recognition :: detect actions using hard coded rlm@401: body-centered scripts. rlm@401: rlm@401: - cool demo video of the worm being moved and recognizing things :: rlm@401: will be a neat part of the thesis. rlm@401: rlm@401: - thesis export :: refactoring and organization of code so that it rlm@401: spits out a thesis in addition to the web page. rlm@401: rlm@401: - video alignment :: analyze the frames of a video in order to align rlm@401: the worm. Requires body-centered recognition. Can "cheat". rlm@401: rlm@401: - smoother actions :: use debugging controls to directly influence the rlm@401: demo actions, and to generate recoginition procedures. rlm@401: rlm@401: - degenerate video demonstration :: show the system recognizing a rlm@401: curled worm from dead on. Crowning achievement of thesis. rlm@401: rlm@401: ** Ordered from easiest to hardest rlm@401: rlm@401: Just report the positions of everything. I don't think that this rlm@401: necessairly shows anything usefull. rlm@401: rlm@401: Worm-segment vision -- you initialize a view of the worm, but instead rlm@401: of pixels you use labels via ray tracing. Has the advantage of still rlm@401: allowing for visual occlusion, but reliably identifies the objects, rlm@401: even without rainbow coloring. You can code this as an image. rlm@401: rlm@401: Same as above, except just with worm/non-worm labels. rlm@401: rlm@401: Color code each worm segment and then recognize them using blob rlm@401: detectors. Then you solve for the perspective and the action rlm@401: simultaneously. rlm@401: rlm@401: The entire worm can be colored the same, high contrast color against a rlm@401: nearly black background. rlm@401: rlm@401: "Rooted" vision. You give the exact coordinates of ONE piece of the rlm@401: worm, but the algorithm figures out the rest. rlm@401: rlm@401: More rooted vision -- start off the entire worm with one posistion. rlm@401: rlm@401: The right way to do alignment is to use motion over multiple frames to rlm@401: snap individual pieces of the model into place sharing and rlm@401: propragating the individual alignments over the whole model. We also rlm@401: want to limit the alignment search to just those actions we are rlm@401: prepared to identify. This might mean that I need some small "micro rlm@401: actions" such as the individual movements of the worm pieces. rlm@401: rlm@401: Get just the centers of each segment projected onto the imaging rlm@401: plane. (best so far). rlm@401: rlm@401: rlm@401: Repertoire of actions + video frames --> rlm@401: directed multi-frame-search alg rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: !! Could also have a bounding box around the worm provided by rlm@401: filtering the worm/non-worm render, and use bbbgs. As a bonus, I get rlm@401: to include bbbgs in my thesis! Could finally do that recursive things rlm@401: where I make bounding boxes be those things that give results that rlm@401: give good bounding boxes. If I did this I could use a disruptive rlm@401: pattern on the worm. rlm@401: rlm@401: Re imagining using default textures is very simple for this system, rlm@401: but hard for others. rlm@401: rlm@401: rlm@401: Want to demonstrate, at minimum, alignment of some model of the worm rlm@401: to the video, and a lookup of the action by simulated perception. rlm@401: rlm@401: note: the purple/white points is a very beautiful texture, because rlm@401: when it moves slightly, the white dots look like they're rlm@401: twinkling. Would look even better if it was a darker purple. Also rlm@401: would look better more spread out. rlm@401: rlm@401: rlm@401: embed assumption of one frame of view, search by moving around in rlm@401: simulated world. rlm@401: rlm@401: Allowed to limit search by setting limits to a hemisphere around the rlm@401: imagined worm! This limits scale also. rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: !! Limited search with worm/non-worm rendering. rlm@401: How much inverse kinematics do we have to do? rlm@401: What about cached (allowed state-space) paths, derived from labeled rlm@401: training. You have to lead from one to another. rlm@401: rlm@401: What about initial state? Could start the input videos at a specific rlm@401: state, then just match that explicitly. rlm@401: rlm@401: !! The training doesn't have to be labeled -- you can just move around rlm@401: for a while!! rlm@401: rlm@401: !! Limited search with motion based alignment. rlm@401: rlm@401: rlm@401: rlm@401: rlm@401: "play arounds" can establish a chain of linked sensoriums. Future rlm@401: matches must fall into one of the already experienced things, and once rlm@401: they do, it greatly limits the things that are possible in the future. rlm@401: rlm@401: rlm@401: frame differences help to detect muscle exertion. rlm@401: rlm@401: Can try to match on a few "representative" frames. Can also just have rlm@401: a few "bodies" in various states which we try to match. rlm@401: rlm@401: rlm@401: rlm@401: Paths through state-space have the exact same signature as rlm@401: simulation. BUT, these can be searched in parallel and don't interfere rlm@401: with each other. rlm@401: rlm@401: rlm@402: rlm@402: rlm@402: ** Final stretch up to First Draft rlm@402: rlm@404: *** DONE complete debug control of worm rlm@404: CLOSED: [2014-03-17 Mon 17:29] SCHEDULED: <2014-03-17 Mon> rlm@404: CLOCK: [2014-03-17 Mon 14:01]--[2014-03-17 Mon 17:29] => 3:28 rlm@405: *** DONE add phi-space output to debug control rlm@405: CLOSED: [2014-03-17 Mon 17:42] SCHEDULED: <2014-03-17 Mon> rlm@405: CLOCK: [2014-03-17 Mon 17:31]--[2014-03-17 Mon 17:42] => 0:11 rlm@407: rlm@409: *** DONE complete automatic touch partitioning rlm@409: CLOSED: [2014-03-18 Tue 21:43] SCHEDULED: <2014-03-18 Tue> rlm@415: *** DONE complete cyclic predicate rlm@415: CLOSED: [2014-03-19 Wed 16:34] SCHEDULED: <2014-03-18 Tue> rlm@415: CLOCK: [2014-03-19 Wed 13:16]--[2014-03-19 Wed 16:34] => 3:18 rlm@415: *** DONE complete three phi-stream action predicatates; test them with debug control rlm@415: CLOSED: [2014-03-19 Wed 16:35] SCHEDULED: <2014-03-17 Mon> rlm@415: CLOCK: [2014-03-18 Tue 18:36]--[2014-03-18 Tue 21:43] => 3:07 rlm@407: CLOCK: [2014-03-18 Tue 18:34]--[2014-03-18 Tue 18:36] => 0:02 rlm@407: CLOCK: [2014-03-17 Mon 19:19]--[2014-03-17 Mon 21:19] => 2:00 rlm@415: *** DONE build an automatic "do all the things" sequence. rlm@415: CLOSED: [2014-03-19 Wed 16:55] SCHEDULED: <2014-03-19 Wed> rlm@415: CLOCK: [2014-03-19 Wed 16:53]--[2014-03-19 Wed 16:55] => 0:02 rlm@417: *** DONE implement proprioception based movement lookup in phi-space rlm@417: CLOSED: [2014-03-19 Wed 22:04] SCHEDULED: <2014-03-19 Wed> rlm@417: CLOCK: [2014-03-19 Wed 19:32]--[2014-03-19 Wed 22:04] => 2:32 rlm@418: *** DONE make proprioception reference phi-space indexes rlm@418: CLOSED: [2014-03-19 Wed 22:47] SCHEDULED: <2014-03-19 Wed> rlm@417: rlm@415: rlm@420: *** DONE create test videos, also record positions of worm segments rlm@420: CLOSED: [2014-03-20 Thu 22:02] SCHEDULED: <2014-03-19 Wed> rlm@402: rlm@403: *** TODO Collect intro, worm-learn and cortex creation into draft thesis. rlm@405: