1.1 --- a/org/ideas.org	Thu Jan 16 10:30:23 2014 -0500
     1.2 +++ b/org/ideas.org	Thu Jan 16 10:37:11 2014 -0500
     1.3 @@ -31,14 +31,133 @@
     1.4  getting credit.
     1.5  #+end_quote
     1.6  
     1.7 +<<<<<<< local
     1.8  - pronuncation guide :: a simple webpage where you type in a word and
     1.9       it returns a simpe, English sentence describing exactly how to
    1.10       pronounce the word. For people who don't want to learn IPA. 
    1.11 +=======
    1.12 +- learning to teleport :: scifi idea, this is a story about a person
    1.13 +     who is struggling with his/her society's ideas about
    1.14 +     teleportation. It's considered a fundamental part of being a
    1.15 +     member of that society (after all, the difference between animals
    1.16 +     and humans is that humans are creatures of pure information while
    1.17 +     animals are burdened with base matter, "that's how you travel the
    1.18 +     stars, etc") Humans are born normally, grow up, and then
    1.19 +     eventually transcend via destructive upload. Analogies to jumping
    1.20 +     off a diving board into a pool (which I simply /could not do/ for
    1.21 +     a long time), etc.     
    1.22 +>>>>>>> other
    1.23  
    1.24 +<<<<<<< local
    1.25  - touch vision :: inspired by GelSight, I want to reexamine cortex and
    1.26                    see if I could implement touch as a very low range
    1.27                    form of vision.
    1.28 +=======
    1.29 +- bitcoins for immigrants :: A common case with Mexican immigrants
    1.30 +     (illegal or not) is that they want to send money they've earned
    1.31 +     in the US back to their families in Mexico. They currently do this
    1.32 +     through things like Money Gram or Western Union, and they get
    1.33 +     fleeced in the process with fees. Bitcoin could greatly reduce
    1.34 +     the cost of sending money from America to Mexico, but I don't
    1.35 +     believe that it's currently used for that among Mexican
    1.36 +     immigrants currently due to lack of knowledge. I bet you could
    1.37 +     set up physical locations like those obnoxious Western Union huts
    1.38 +     in places like Texas, Arizona, etc, and greatly undercut
    1.39 +     them. Or, perhaps some educational seminars about bitcoin might
    1.40 +     be in order. There's some money to be made there because there is
    1.41 +     great demand, and it's a good thing to boot!
    1.42 +>>>>>>> other
    1.43  
    1.44 +<<<<<<< local
    1.45 +=======
    1.46 +- reverse eye-tracking :: A painting that is actually a digital screen
    1.47 +     with a camera. It records people's eye tracks permanently. It's
    1.48 +     "artistic" because paintings are normally these things that you
    1.49 +     look at without changing, but this one is changed the second you
    1.50 +     look at it, recording where /you/ looked forever for others to
    1.51 +     see. Make it be a painting of a woman and see the trolling as the
    1.52 +     breasts and groin area light up with interest from all the males
    1.53 +     passing by.
    1.54 +
    1.55 +- smart toilets :: Instead of using indirect measures like infrared
    1.56 +                   detectors of the presence of a person, use computer
    1.57 +                   vision to directly measure whether the toilet needs
    1.58 +                   to be flushed. I think a lot of things will end up
    1.59 +                   going this way as we get better computer vision.
    1.60 +
    1.61 +- validate chemopreservation :: chemopreservation is difficult to
    1.62 +     validate because it destroys the functionality of a brain, and
    1.63 +     brain simulation will take a long time to mature as a
    1.64 +     technology. However, one very powerful way to validate
    1.65 +     chemopreservation would be to have a person/animal learn
    1.66 +     something with high complexity such as a number or the solution
    1.67 +     to a maze, or a flashbulb memory. Then you preserve their brain
    1.68 +     chemically, slice it up, and read /that specific memory/ from the
    1.69 +     detailed brain scan. Much more difficult, but much more doable.
    1.70 +
    1.71 +- candy screw :: edible candy screw with candy nuts that you can screw
    1.72 +                 as well. 
    1.73 +
    1.74 +- better bibliography :: when writing a thesis or paper, have the
    1.75 +     bibliography not just be an opaque list of resources, but have it
    1.76 +     be a list of /summaries/ and /qualities/ that each paper has in
    1.77 +     the context of the paper being written. When examining a
    1.78 +     bibliography, I want to know if reading the papers in the
    1.79 +     bibliography are worth my time, and I also am probably also
    1.80 +     interested in exactly the things that are being discussed in the
    1.81 +     paper I'm reading. The bibliography is the perfect place to
    1.82 +     provide information about the referenced papers from the
    1.83 +     author's perspective. I will use this biographic form in my own
    1.84 +     thesis. 
    1.85 +
    1.86 +- digital inter-library loan :: libraries at universities already do
    1.87 +     inter-library loans for books, so why not do the same for access
    1.88 +     to stupid paywalled digital papers? All the universities could
    1.89 +     allow access to articles for registered students to all the files
    1.90 +     available through any participating university. This could be
    1.91 +     achieved by sending requests through proxies at participating
    1.92 +     universities. Each university would decide who at the university
    1.93 +     can access the proxy network. Access to the proxy network could
    1.94 +     be made easy using something like http://libx.org/.
    1.95 +
    1.96 +- chess visual :: to show the vast size of the game trees considered
    1.97 +                  by computers, show two people playing chess in a
    1.98 +                  void. They are floating in space, and there is a
    1.99 +                  simple chess board between them. Then, as they play,
   1.100 +                  the game tree's they are considering are drawn
   1.101 +                  behind him. The root of the tree starts centered in
   1.102 +                  their heads or whatever they use to think, and the
   1.103 +                  tree grows out from behind, never crossing the
   1.104 +                  dividing plane between the two players. Each
   1.105 +                  player's tree is a different color. As they grow,
   1.106 +                  there are animations for pruning, etc. Eventually,
   1.107 +                  they look like the hemispheres of a brain, wings,
   1.108 +                  etc. A human's tree might occasionally have a long
   1.109 +                  chain, while the computer tree would be more
   1.110 +                  uniform. You could compare deep blue and a modern
   1.111 +                  laptop. Use actual data when fighting two computers! 
   1.112 +
   1.113 +- time verification :: some standard way to verify that some piece of
   1.114 +     data was recorded at a specific time. Might involve a time
   1.115 +     server, a key for each time period, something like that.
   1.116 +
   1.117 +- tamper proof gold bars :: [[http://www.tungsten-alloy.com/gold-plated-tungsten-alloy-bar.html][this site]] offers gold plated tungsten bars
   1.118 +     as "novelty" items. One reason to prefer coins is because they
   1.119 +     are much harder to counterfeit because there is less surface area
   1.120 +     to mass ratio. However, gold bars are still a great design
   1.121 +     because they can hold a lot of value in a small space. A gold bar
   1.122 +     could be given the same protections (and more) that gold coins
   1.123 +     have to offer by changing it into a "gold book", which would have
   1.124 +     hundreds of "pages" of gold bound together. This could be
   1.125 +     implemented with multiple steel rods going through the book which
   1.126 +     can be removed, or some more classier mechanism for holding the
   1.127 +     pages. The point is that the bar can be EASILY subdivided (and
   1.128 +     people would perform this test before buying), thus guaranteeing
   1.129 +     it's authenticity.
   1.130 +
   1.131 +- aurellem shirt :: I should make an aurellem star symbol tee-shirt. 
   1.132 +
   1.133 +>>>>>>> other
   1.134  - high school science :: this is a lesson in scientific ethics. The
   1.135       goal is to calculate /g/, the local gravitational
   1.136       acceleration. The students are told that the textbook says it's
   1.137 @@ -63,7 +182,7 @@
   1.138                     wipe job you have done / watch how your excretion
   1.139                     system works.
   1.140  
   1.141 -- test dummies :: why don't we clone encephalic humans and use then to
   1.142 +- test dummies :: why don't we clone anencephalic humans and use then to
   1.143                    test /in vivo/ human organ systems and drugs? It
   1.144                    would be ethical as long as there are women who are
   1.145                    willing to host the clones, and it would be a
   1.146 @@ -75,7 +194,7 @@
   1.147  - X-ray telepresence :: given that a doctor is operating on a patient
   1.148       via telepresence, one cool things you can do is shine X-rays into
   1.149       the patient to view the insides during real time. If the system
   1.150 -     was coupled with a Baysean model of the layout of the structure,
   1.151 +     was coupled with a Bayesian model of the layout of the structure,
   1.152       and the x-rays were only fired whenever the uncertainty of the
   1.153       model reached a certain threshold, then the radiation damage
   1.154       and surgery risk could be minimized.
   1.155 @@ -138,8 +257,8 @@
   1.156  - image compression :: use a library like gimp or opencv to process an
   1.157       image to make it have less entropy, then store the reverse of
   1.158       those operations along with the compressed simpler image as a
   1.159 -     super-compressed image file (possibly sccapting some
   1.160 -     lesses). Trades file size for decompression time, and allows one
   1.161 +     super-compressed image file (possibly accepting some
   1.162 +     losses). Trades file size for decompression time, and allows one
   1.163       to cheat by using information in gimp/opencv to compress the
   1.164       image. 
   1.165  
   1.166 @@ -318,7 +437,7 @@
   1.167       by the sensor array).
   1.168  
   1.169  - intelligent microwave :: it learns where the hot nodes of its fields
   1.170 -  are, and uses them to evenly heat any food item. It has an infared
   1.171 +  are, and uses them to evenly heat any food item. It has an infrared
   1.172    camera or something to keep track of how hot the food is. That way,
   1.173    you don't get bowls where the edges are boiling, while the center is
   1.174    still frozen. Requires a little bit of intelligence/vision, since
   1.175 @@ -362,7 +481,7 @@
   1.176  
   1.177  - UROP :: magnet gear/metal teeth tape 
   1.178  
   1.179 -- rod of moses :: device to distill urine through evaporation and
   1.180 +- Rod of Moses :: device to distill urine through evaporation and
   1.181                    easily dispose of urea crystals for use in desert --
   1.182                    produce drinkable water.
   1.183  

     2.1 --- a/org/steph.org	Thu Jan 16 10:30:23 2014 -0500
     2.2 +++ b/org/steph.org	Thu Jan 16 10:37:11 2014 -0500
     2.3 @@ -4,8 +4,6 @@
     2.4  #+setupfile: ../../aurellem/org/setup.org
     2.5  #+include: ../../aurellem/org/level-0.org
     2.6  
     2.7 -
     2.8 -
     2.9  [[../images/steph.jpg]]
    2.10  
    2.11  #+begin_src clojure :results output :exports both
    2.12 @@ -25,5 +23,5 @@
    2.13  #+results:
    2.14  : 7906 days.
    2.15  
    2.16 -Goodbye Stephanie. I will miss you always.
    2.17 +Goodbye Stephanie. I will miss you.
    2.18  

     3.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     3.2 +++ b/org/svm.org	Thu Jan 16 10:37:11 2014 -0500
     3.3 @@ -0,0 +1,193 @@
     3.4 +#+TITLE: Notes on SVMs
     3.5 +#+AUTHOR:Dylan Holmes
     3.6 +#+SETUPFILE: ../../aurellem/org/setup.org
     3.7 +#+INCLUDE:   ../../aurellem/org/level-0.org
     3.8 +#+MATHJAX: align:"left" mathml:t path:"http://www.aurellem.org/MathJax/MathJax.js"
     3.9 +
    3.10 +* Question: How do you find $\vec{w}$, $b$, and the \alpha{}'s?
    3.11 +
    3.12 +You should take a look at the class notes, for starters:
    3.13 +http://ai6034.mit.edu/fall12/images/SVM_and_Boosting.pdf
    3.14 +
    3.15 +You can also take a look at @388 to see an example of (part of) a
    3.16 +worked problem.
    3.17 +
    3.18 +
    3.19 +When solving SVM problems, there are some useful equations to keep in
    3.20 +mind:
    3.21 +
    3.22 +1. $\vec{w}\cdot \vec{x} + b = 0$ defines the boundary, and in
    3.23 +   particular $\vec{w}\cdot \vec{x} + b \geq 0$ defines the positive
    3.24 +   side of the boundary.
    3.25 +2. $\vec{w}\cdot \vec{x} + b = \pm 1$ defines the positive and
    3.26 +   negative gutters.  
    3.27 +3. The distance between the gutters (the width of the margin) is
    3.28 +   $\text{margin-width} = \frac{2}{||\vec{w}||} =
    3.29 +   \frac{2}{\sqrt{\vec{w}\cdot \vec{w}}}$ 
    3.30 +4. $\sum_{\text{training data}} y_i \alpha_i =
    3.31 +   0$ 
    3.32 +5. $\sum_{\text{training data}} y_i \alpha_i \vec{x}_i = \vec{w}$
    3.33 +
    3.34 +The last two equations are the ones that can help you find the
    3.35 +  $\alpha_i$. You may also find it useful to think of the $\alpha_i$
    3.36 +  as measuring \ldquo{}supportiveness\rdquo{}. This means, for
    3.37 +  example, that: - $\alpha_i$ is zero for non-support vectors,
    3.38 +  i.e. training points that do not determine the boundary, and which
    3.39 +  would not affect the placement of the boundary if deleted - When you
    3.40 +  compare two separate SVM problems, where the first has support
    3.41 +  vectors that are far from the boundary, and the second has support
    3.42 +  vectors very close to the boundary, the latter support vectors have
    3.43 +  comparatively higher $\alpha_i$ values.
    3.44 +
    3.45 +** Can \alpha{} be zero for support vectors?
    3.46 +No, \alpha{} is zero for a point if and only if it's not a support
    3.47 +vector. You can see this because in equation (5), just the points
    3.48 +with nonzero alphas contribute to $\vec{w}$.
    3.49 +
    3.50 +** Are all points on the gutter support vectors? How do you tell whether a point is a support vector?
    3.51 +
    3.52 +Not all points on the gutter are support vectors. To tell if a
    3.53 +training point is a support vector, imagine what would happen if you
    3.54 +deleted it --- would the decision boundary be different? If it would,
    3.55 +the point is a support vector. If it wouldn't, the point isn't a
    3.56 +support vector.
    3.57 +
    3.58 +* 2011 Q2 Part A
    3.59 +
    3.60 +The equation for the line you drew in part A is $y = 0$, and points
    3.61 +will be classified as positive if $y\geq 0$.
    3.62 +
    3.63 +# shouldn't points be classified as positive only if leq 1 + b dot w x? 
    3.64 +Now in general, the equation for the decision boundary is
    3.65 +$\vec{w}\cdot \vec{x} + b = 0$, and points $\vec{x}$ are
    3.66 +classified as positive if $\vec{w}\cdot \vec{x} + b \geq 0$. To
    3.67 +solve for $\vec{w}$ and $b$, you just manipulate your inequation
    3.68 +for the boundary into this form:
    3.69 +
    3.70 +$$y\geq 0$$
    3.71 +$$0x + 1y + 0 \geq 0$$
    3.72 +$$\begin{bmatrix}0&1\end{bmatrix}\cdot \begin{bmatrix}x\\y\end{bmatrix} + 0 \geq 0$$
    3.73 +
    3.74 +And so we have:
    3.75 +
    3.76 +$$\underbrace{\begin{bmatrix}0&1\end{bmatrix}}_{\vec{w}}\cdot \underbrace{\begin{bmatrix}x\\y\end{bmatrix}}_{\vec{x}} + \underbrace{0}_{b} \geq 0$$
    3.77 +
    3.78 +...which is /almost/ right. The trouble is that we can multiply this
    3.79 +inequation by any positive constant $c>0$ and it will still be true
    3.80 +--- so, the right answer might not be the $\vec{w}$ and $b$ we found
    3.81 +above, but instead a /multiple/ of them. To fix this, we multiply the
    3.82 +inequation by $c>0$, and solve for $c$:
    3.83 +
    3.84 +$$c\cdot (0x + 1y + 0) \geq c\cdot0$$
    3.85 +$$\begin{bmatrix}0&c\end{bmatrix}\cdot \begin{bmatrix}x\\y\end{bmatrix} +
    3.86 +0 \geq 0$$
    3.87 +
    3.88 +One formula that you can use to solve for $c$ is the margin-width
    3.89 +formula; by examining the lines you drew, you see that the margin
    3.90 +width is *4*. Now, in general, the equation for margin width is
    3.91 +
    3.92 +$$\text{margin-width} = \frac{2}{||\vec{w}||} = \frac{2}{\sqrt{\vec{w}\cdot \vec{w}}}$$
    3.93 +
    3.94 +So, for the second time, we use the technique of setting the quantity
    3.95 +you can see (margin width = 4) equal to the general formula (margin
    3.96 +width = 2/||w||) and solving for the parameters in the general
    3.97 +formula:
    3.98 +
    3.99 +$$\frac{2}{||w||} = 4$$
   3.100 +
   3.101 +$$\frac{2}{\sqrt{\vec{w}\cdot \vec{w}}} = 4$$
   3.102 +
   3.103 +$$\frac{4}{\vec{w}\cdot \vec{w}} = 16$$ (squaring both sides)
   3.104 +
   3.105 +$$\vec{w}\cdot \vec{w} = \frac{1}{4}$$
   3.106 +
   3.107 +$$\begin{bmatrix}0&c\end{bmatrix}\cdot \begin{bmatrix}0\\c\end{bmatrix}
   3.108 += \frac{1}{4}$$ 
   3.109 +
   3.110 +(This uses the expression for $\vec{w}$ we got after
   3.111 +we multiplied by $c$ above.)
   3.112 +
   3.113 +$$\begin{eqnarray*}
   3.114 +c^2 &=& \frac{1}{4}\\
   3.115 +c &=& \pm \frac{1}{2}\\
   3.116 +c &=& \frac{1}{2}\end{eqnarray*}$$...because we require $c>0$ so that it doesn't flip the inequality.
   3.117 +
   3.118 +
   3.119 +Returning to our inequality, we have 
   3.120 +
   3.121 +$$\begin{eqnarray*}\begin{bmatrix}0&c\end{bmatrix}\cdot \begin{bmatrix}x\\y\end{bmatrix} + 0 &\geq& 0\\
   3.122 +\begin{bmatrix}0&\frac{1}{2}\end{bmatrix}\cdot \begin{bmatrix}x\\y\end{bmatrix} + 0 &\geq& 0\end{eqnarray*}$$
   3.123 +
   3.124 +$$\underbrace{\begin{bmatrix}0&\frac{1}{2}\end{bmatrix}}_{\vec{w}}\cdot \underbrace{\begin{bmatrix}x\\y\end{bmatrix}}_{\vec{x}} + \underbrace{0}_{b} \geq 0$$
   3.125 +
   3.126 +And this is truly the right answer, now that we have solved for $c$.
   3.127 +
   3.128 +
   3.129 +* What's the use of gutters?
   3.130 +The gutters are the regions where you have no hard data supporting
   3.131 +your classification, but the decision boundary is, as its name
   3.132 +implies, the cutoff for making a decision between the two
   3.133 +classifications.  As such, you would likely have lower confidence in
   3.134 +any decision within the gutters, but a classifier needs to make a
   3.135 +decision one way or the other for all possible test data (there's no
   3.136 +reason not to do so, as making no decision means you're automatically
   3.137 +wrong in all cases, whereas even a lousy classifier will get some of
   3.138 +them right).
   3.139 +
   3.140 +----
   3.141 +
   3.142 +Here are two more ways to look at it. First of all, and simplest, SVMs
   3.143 +are supposed to find the boundary with the widest margin. The width of
   3.144 +the margin is just the perpendicular distance between the gutters, so
   3.145 +the gutters are important for that reason.
   3.146 +
   3.147 + Second of all, if you attempt to define the SVM problem
   3.148 + geometrically, i.e. put it into a form that can be solved by a
   3.149 + computer, it looks something like the following.
   3.150 +
   3.151 +#+BEGIN_QUOTE
   3.152 +Find a boundary --- which is defined by a normal vector
   3.153 +$\vec{w}$ and an offset $b$ --- such that:
   3.154 +1. The boundary \ldquo{} $\vec{w}\cdot \vec{x} + b = 0$ \rdquo{}
   3.155 +   separates the positive and negative points
   3.156 +2. The width of the margin is as large as possible. Remember, the
   3.157 +   margin width is twice the distance between the boundary and the
   3.158 +   training point that is closest to the boundary; in other words,
   3.159 +   $$\text{margin-width} = \min_{\text{training data}} 2 y_i
   3.160 +   (\vec{w}\cdot \vec{x}_i + b)$$
   3.161 +#+END_QUOTE
   3.162 +
   3.163 +Unfortunately, the problem as stated has no unique solution, since if
   3.164 +you find a satisfactory pair $\langle \vec{w}, b \rangle$ , then the
   3.165 +pair $\langle 3\vec{w}, 3b\rangle$ (for example) defines the same
   3.166 +boundary but has a larger margin. The problem is that any multiple of
   3.167 +a satisfactory pair yields another satisfactory pair. In essence,
   3.168 +we're just changing the units of measurement without materially
   3.169 +affecting the boundary.
   3.170 +
   3.171 +We can /remove this additional degree of freedom by adding another
   3.172 +constraint/ to the problem which establishes a sense of scale. For
   3.173 +example, we could require $\vec{w}$ to be a /unit/ normal vector,
   3.174 +i.e. we could require that $||\vec{w}|| = 1$. This fixes the problem
   3.175 +and gives SVMs a unique solution.
   3.176 +
   3.177 +In 6.034, and elsewhere, we use an alternate constraint; we impose the /gutter conditions/:  
   3.178 +
   3.179 +$\vec{w}\cdot \vec{x_+} + b = + 1$ for positive training points
   3.180 +$\vec{x_+}$.  
   3.181 +
   3.182 +$\vec{w}\cdot \vec{x_-} + b = -1$ for negative
   3.183 +training points $\vec{x_-}$.
   3.184 +
   3.185 +which we often combine into a single inequality: $y_i (\vec{w}\cdot
   3.186 +\vec{x}_i + b) \geqslant 1$ for all training points $\langle
   3.187 +\vec{x}_i, y_i\rangle$.
   3.188 +
   3.189 +
   3.190 +This constraint is enough to eliminate the extra degree of freedom and give SVMs a unique solution. Moreover, you get a nice formula for the margin width by requiring that it holds:
   3.191 +
   3.192 +$\text{margin-width} = \frac{2}{||\vec{w}||}$ because the gutter
   3.193 +conditions are enforced.
   3.194 +
   3.195 +
   3.196 +That's why gutters are important.