Mercurial > cortex

annotate org/gabor.org @ 526:01934317b25b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
changes from conference.
author Robert McIntyre <rlm@mit.edu>
date Mon, 21 Apr 2014 02:11:29 -0400
parents 42ddfe406c0a
children
rev   line source
rlm@356 1 #+title: Fun with Gabor Filters
rlm@356 2 #+author: Robert McIntyre
rlm@356 3 #+email: rlm@mit.edu
rlm@356 4 #+description: gabor filters in clojure with opencv
rlm@356 5 #+keywords: computer vision, jMonkeyEngine3, clojure, opencv
rlm@356 6 #+SETUPFILE: ../../aurellem/org/setup.org
rlm@356 7 #+INCLUDE: ../../aurellem/org/level-0.org
rlm@356 8 #+babel: :mkdirp yes :noweb yes :exports both
rlm@356 9
rlm@356 10
rlm@368 11 Gabor filters were invented by the same guy who invented holograms.
rlm@368 12
rlm@368 13 They work well as edge detectors and are related to the human visual
rlm@368 14 system.
rlm@356 15
rlm@356 16 #+name: gabor
rlm@356 17 #+begin_src clojure
rlm@357 18 (ns cortex.gabor
rlm@357 19 (:import org.opencv.core.CvType)
rlm@357 20 (:import java.awt.image.BufferedImage)
rlm@357 21 (:import ij.ImagePlus)
rlm@358 22 (:import org.opencv.core.Mat)
rlm@364 23 (:use (cortex world sense util vision import))
rlm@363 24 (:import com.jme3.post.SceneProcessor)
rlm@363 25 (:import (com.jme3.util BufferUtils Screenshots))
rlm@363 26 (:import java.nio.ByteBuffer)
rlm@363 27 (:import java.awt.image.BufferedImage)
rlm@363 28 (:import (com.jme3.renderer ViewPort Camera))
rlm@363 29 (:import (com.jme3.math ColorRGBA Vector3f Matrix3f Vector2f))
rlm@363 30 (:import com.jme3.renderer.Renderer)
rlm@363 31 (:import com.jme3.app.Application)
rlm@363 32 (:import com.jme3.texture.FrameBuffer)
rlm@363 33 (:import (com.jme3.scene Node Spatial)))
rlm@363 34
rlm@363 35
rlm@363 36 (cortex.import/mega-import-jme3)
rlm@364 37 (use 'clojure.math.numeric-tower)
rlm@356 38 (defn load-opencv
rlm@356 39 "Load the opencv native library. Must be called before any OpenCV
rlm@356 40 stuff is used."
rlm@356 41 []
rlm@356 42 (clojure.lang.RT/loadLibrary "opencv_java249"))
rlm@356 43
rlm@364 44 (load-opencv)
rlm@364 45
rlm@364 46 (defn gabor-kernel
rlm@364 47 ([sigma wavelength theta]
rlm@364 48 (gabor-kernel sigma wavelength theta 1 0))
rlm@364 49 ([sigma wavelength]
rlm@364 50 (gabor-kernel sigma wavelength 0 1 0))
rlm@364 51 ([sigma wavelength theta aspect-ratio phase-offset]
rlm@359 52
rlm@359 53 ;; first, find the size of the kernel which is required
rlm@359 54 (let [square #(expt % 2)
rlm@359 55 rotated (fn [[x y]]
rlm@359 56 [(+ (* x (Math/cos theta)) (* y (Math/sin theta)))
rlm@359 57 (- (* y (Math/cos theta)) (* x (Math/sin theta)))])
rlm@359 58
rlm@359 59 gaussian (fn [[x y]]
rlm@359 60 (let [[x' y'] (rotated [x y])]
rlm@359 61 (Math/exp (- (/ (+ (square x')
rlm@359 62 (square (* aspect-ratio y')))
rlm@359 63 (* 2 (square sigma)))))))
rlm@359 64 sinusoid (fn [[x y]]
rlm@359 65 (let [[x' y'] (rotated [x y])]
rlm@359 66 (Math/cos
rlm@359 67 (+ (* 2 Math/PI (/ x' wavelength))
rlm@359 68 phase-offset))))
rlm@359 69
rlm@361 70 half-width
rlm@361 71 (let [std-dev-capture 5]
rlm@361 72 (max
rlm@361 73 (int (* std-dev-capture (/ sigma aspect-ratio)))
rlm@361 74 (int (* std-dev-capture sigma))
rlm@361 75 (int (* std-dev-capture (/ aspect-ratio sigma)))))
rlm@359 76
rlm@359 77 grid (let [axis (range (- half-width) (inc half-width))]
rlm@359 78 (for [y (reverse axis) x axis] (vector x y)))
rlm@359 79
rlm@359 80 scale (reduce + (map gaussian grid))
rlm@359 81
rlm@359 82 gabor (fn [[x y :as coord]]
rlm@360 83 (* (sinusoid coord) (gaussian coord) (/ scale)))
rlm@359 84
rlm@359 85 mat-width (+ 1 (* 2 half-width))
rlm@359 86 mat (Mat. mat-width mat-width CvType/CV_32F)]
rlm@359 87
rlm@359 88 (.put mat 0 0 (float-array (map gabor grid)))
rlm@364 89 mat)))
rlm@359 90
rlm@359 91
rlm@361 92 (defn draw-kernel! [kernel img-path]
rlm@361 93 (let [output img-path
rlm@360 94 size (.size kernel)
rlm@360 95 width (int (.width size))
rlm@360 96 height (int (.height size))
rlm@360 97 tmp-array (float-array (* width height))]
rlm@360 98
rlm@360 99 ;; read values from matrix.
rlm@360 100 (.get kernel 0 0 tmp-array)
rlm@360 101
rlm@360 102 ;; find overall dynamic range of the filter
rlm@360 103 (let [vals (vec tmp-array)
rlm@360 104 low (apply min vals)
rlm@360 105 high (apply max vals)
rlm@360 106 scaled-vals (map #(* 255 (- % low) (/ (- high low))) vals)
rlm@360 107 new-mat (Mat. height width CvType/CV_32F)]
rlm@360 108 (.put new-mat 0 0 (float-array scaled-vals))
rlm@361 109 (org.opencv.highgui.Highgui/imwrite output new-mat))))
rlm@361 110
rlm@364 111 ;; some cool examples
rlm@364 112 #+end_src
rlm@364 113
rlm@364 114
rlm@364 115
rlm@364 116
rlm@364 117 [[../images/gabor-50-10.png]]
rlm@364 118
rlm@364 119 #+begin_src clojure
rlm@364 120 (def img-base "/home/r/proj/cortex/images/")
rlm@364 121
rlm@364 122 (draw-kernel! (gabor-kernel 50 10 0 1 0)
rlm@364 123 (str img-base "gabor-50-10.png"))
rlm@364 124 #+end_src
rlm@364 125
rlm@364 126
rlm@364 127 [[../images/gabor-50-10-pi-over-4.png]]
rlm@364 128
rlm@364 129 #+begin_src clojure
rlm@364 130 (draw-kernel! (gabor-kernel 50 10 (/ Math/PI 4) 1 0)
rlm@364 131 (str img-base "gabor-50-10-pi-over-4.png"))
rlm@364 132 #+end_src
rlm@364 133
rlm@364 134
rlm@364 135 [[../images/gabor-50-10-pi-over-2.png]]
rlm@364 136
rlm@364 137 #+begin_src clojure
rlm@364 138 (draw-kernel! (gabor-kernel 50 10 (/ Math/PI 2) 1 0)
rlm@364 139 (str img-base "gabor-50-10-pi-over-2.png"))
rlm@364 140 #+end_src
rlm@364 141
rlm@364 142
rlm@364 143 [[../images/gabor-50-50.png]]
rlm@364 144
rlm@364 145
rlm@364 146
rlm@364 147 #+begin_src clojure
rlm@364 148 (draw-kernel! (gabor-kernel 50 50 0 1 0)
rlm@364 149 (str img-base "gabor-50-50.png"))
rlm@364 150
rlm@364 151 #+end_src
rlm@364 152
rlm@364 153 [[../images/gabor-50-10-0-3.png]]
rlm@364 154
rlm@364 155 #+begin_src clojure
rlm@364 156 (draw-kernel! (gabor-kernel 50 10 0 3 0)
rlm@364 157 (str img-base "gabor-50-10-0-3.png"))
rlm@364 158 #+end_src
rlm@364 159
rlm@364 160
rlm@364 161
rlm@364 162 [[../images/gabor-50-4-pi-over3-3.png]]
rlm@364 163 #+begin_src clojure
rlm@364 164 (draw-kernel! (gabor-kernel 50 4 (/ Math/PI 3) 3 0)
rlm@364 165 (str img-base "gabor-50-4-pi-over3-3.png"))
rlm@364 166 #+end_src
rlm@376 167
rlm@364 168
rlm@375 169 #+name: gabor-tail
rlm@364 170 #+begin_src clojure
rlm@361 171 (defn show-kernel [kernel]
rlm@361 172 (let [img-path "/home/r/proj/cortex/tmp/kernel.png"]
rlm@362 173 (draw-kernel! kernel img-path)
rlm@364 174 (view (ImagePlus. img-path))))
rlm@359 175
rlm@359 176 (defn print-kernel [kernel]
rlm@359 177 (println (.dump kernel)))
rlm@359 178
rlm@363 179
rlm@363 180 (import com.aurellem.capture.Capture)
rlm@363 181
rlm@363 182 (import java.io.File)
rlm@363 183
rlm@364 184 (def base "/home/r/proj/cortex/render/gabor-1/")
rlm@363 185
rlm@363 186 (defn brick-wall-game-run [record?]
rlm@364 187 (let [capture-dir (File. base "main")]
rlm@364 188
rlm@364 189 (.mkdir (File. base "main"))
rlm@364 190 (doto
rlm@364 191 (world
rlm@364 192 (doto (Node.) (.attachChild (floor*))
rlm@364 193 (.attachChild (brick-wall*))
rlm@364 194 )
rlm@364 195 {"key-f" (fn [game value]
rlm@364 196 (if (not value) (add-element game (brick-wall*))))
rlm@364 197 "key-space" (fire-cannon-ball )}
rlm@364 198 (fn [world]
rlm@366 199 (position-camera
rlm@366 200 world
rlm@366 201 (Vector3f. 1.382548, 4.0383573, 5.994235)
rlm@366 202 (Quaternion. 0.0013082094, 0.98581666,
rlm@366 203 -0.1676442, 0.0076932586))
rlm@363 204
rlm@364 205 ;;(speed-up world)
rlm@364 206
rlm@364 207 (if record?
rlm@364 208 (Capture/captureVideo
rlm@364 209 world capture-dir))
rlm@364 210
rlm@363 211 (add-camera! world (.getCamera world) no-op))
rlm@364 212 (fn [& _]))
rlm@364 213 (.start))))
rlm@363 214
rlm@364 215 (defn convolve-preview [kernel]
rlm@363 216 (let [input "/home/r/proj/cortex/render/gabor-1/main/0000032.png"
rlm@357 217
rlm@357 218
rlm@357 219 output "/home/r/ppp.png"
rlm@356 220
rlm@357 221 i (org.opencv.highgui.Highgui/imread input)
rlm@358 222
rlm@364 223 ;;kernel (gabor-kernel 10 1 (/ Math/PI 2) 10 0)
rlm@358 224
rlm@358 225 new-mat (Mat.)
rlm@358 226
rlm@357 227 ]
rlm@356 228
rlm@373 229 (org.opencv.imgproc.Imgproc/filter2D
rlm@373 230 i new-mat CvType/CV_32F kernel)
rlm@358 231
rlm@358 232 (org.opencv.highgui.Highgui/imwrite "/home/r/ppp.png" new-mat)
rlm@358 233
rlm@357 234 (view (ImagePlus. input))
rlm@361 235 (view (ImagePlus. output))))
rlm@357 236
rlm@364 237 (use 'clojure.java.shell)
rlm@363 238
rlm@363 239
rlm@364 240 (defn apply-gabor [kernel source dest]
rlm@364 241 (let [i (org.opencv.highgui.Highgui/imread source)
rlm@364 242 new-mat (Mat.)]
rlm@364 243
rlm@364 244 (println dest)
rlm@364 245 (if (not (.exists (File. dest)))
rlm@364 246 (do
rlm@373 247 (org.opencv.imgproc.Imgproc/filter2D
rlm@373 248 i new-mat CvType/CV_32F kernel)
rlm@364 249 (org.opencv.highgui.Highgui/imwrite dest new-mat)
rlm@364 250 (println "mogrify" "-modulate" "1000%" dest)
rlm@364 251 (sh "mogrify" "-modulate" "1000%" dest)))))
rlm@363 252
rlm@363 253
rlm@364 254 (import java.io.File)
rlm@363 255
rlm@364 256 (defn images [path]
rlm@364 257 (sort (rest (file-seq (File. path)))))
rlm@364 258
rlm@364 259
rlm@364 260
rlm@364 261 (defn pics [file]
rlm@364 262 (images (str base file)))
rlm@364 263
rlm@364 264 (defn generate-gabor-images [kernel name]
rlm@364 265 (draw-kernel! kernel (str base name ".png"))
rlm@364 266
rlm@364 267 (.mkdir (File. (str base name)))
rlm@364 268
rlm@364 269 (let [main (map #(.getCanonicalPath %) (pics "main"))
rlm@364 270 targets (map #(str base name "/" (format "%07d.png" %))
rlm@364 271 (range 0 (count main)))]
rlm@364 272 (dorun (pmap (partial apply-gabor kernel) main targets))))
rlm@364 273
rlm@364 274
rlm@364 275 (def banks
rlm@364 276 [[(gabor-kernel 2.8 3.5) "bank-1-1"]
rlm@364 277 [(gabor-kernel 2.8 3.5 (/ Math/PI 2)) "bank-1-1-rot"]
rlm@364 278
rlm@364 279 ;; [(gabor-kernel 3.6 4.6) "bank-1-2"]
rlm@364 280 ;; [(gabor-kernel 4.5 5.6) "bank-2-1"]
rlm@364 281 ;; [(gabor-kernel 6.3 7.9) "bank-3-1"]
rlm@364 282 ;; [(gabor-kernel 7.3 9.1) "bank-3-2"]
rlm@364 283
rlm@364 284 [(gabor-kernel 12.3 15.4) "bank-6-1"]
rlm@364 285
rlm@364 286
rlm@364 287 ;; [(gabor-kernel 17 21.2) "bank-8-1"]
rlm@364 288 ;; [(gabor-kernel 18.2 22.8) "bank-8-2"]
rlm@364 289 ])
rlm@364 290
rlm@364 291
rlm@364 292 (defn make-all-images []
rlm@364 293 (dorun (map (partial apply generate-gabor-images) banks)))
rlm@364 294
rlm@364 295
rlm@364 296
rlm@364 297 (defn compile-left-right []
rlm@364 298 (.mkdir (File. (str base "left-right")))
rlm@364 299 (let [main (pics "main")
rlm@364 300 left (pics "bank-1-1")
rlm@364 301 right (pics "bank-1-1-rot")
rlm@364 302 left-kernel (repeat 20000 (File. (str base "bank-1-1.png")))
rlm@364 303 right-kernel (repeat 20000 (File. (str base "bank-1-1-rot.png")))
rlm@364 304 targets (map
rlm@364 305 #(File. (str base "left-right/" (format "%07d.png" %)))
rlm@364 306 (range 0 (count main)))]
rlm@364 307
rlm@364 308 (dorun
rlm@364 309 (pmap
rlm@364 310 (comp
rlm@364 311 (fn [[main left right left-kernel right-kernel target]]
rlm@364 312 (println target)
rlm@364 313 (if (not (.exists (File. target)))
rlm@364 314 (sh "convert"
rlm@364 315 "-size" "1940x515" "xc:white"
rlm@364 316 main "-geometry" "+0+0" "-composite"
rlm@364 317 left "-geometry" "+650+0" "-composite"
rlm@364 318 right "-geometry" "+1300+0" "-composite"
rlm@364 319 left-kernel "-geometry" "+960+485" "-composite"
rlm@364 320 right-kernel "-geometry" "+1610+485" "-composite"
rlm@364 321 target)))
rlm@364 322 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@364 323 main left right left-kernel right-kernel targets))))
rlm@364 324
rlm@364 325
rlm@364 326 (defn compile-big-small []
rlm@364 327 (.mkdir (File. (str base "big-small")))
rlm@364 328 (let [main (pics "main")
rlm@364 329 left (pics "bank-1-1")
rlm@364 330 right (pics "bank-6-1")
rlm@364 331 small-kernel (repeat 20000 (File. (str base "bank-1-1.png")))
rlm@364 332 big-kernel (repeat 20000 (File. (str base "bank-6-1.png")))
rlm@364 333 targets (map
rlm@364 334 #(File. (str base "big-small/" (format "%07d.png" %)))
rlm@364 335 (range 0 (count main)))]
rlm@364 336
rlm@364 337 (dorun
rlm@364 338 (pmap
rlm@364 339 (comp
rlm@364 340 (fn [[main left right small-kernel big-kernel target]]
rlm@364 341 (println target)
rlm@364 342 (if (not (.exists (File. target)))
rlm@364 343 (sh "convert"
rlm@364 344 "-size" "1940x610" "xc:white"
rlm@364 345 main "-geometry" "+0+0" "-composite"
rlm@364 346 left "-geometry" "+650+0" "-composite"
rlm@364 347 right "-geometry" "+1300+0" "-composite"
rlm@364 348 small-kernel "-geometry" "+960+485" "-composite"
rlm@364 349 big-kernel "-geometry" "+1560+485" "-composite"
rlm@364 350 target)))
rlm@364 351 (fn [& args] (map #(.getCanonicalPath %) args)))
rlm@364 352 main left right small-kernel big-kernel targets))))
rlm@364 353
rlm@364 354
rlm@364 355 (defn regen-everything []
rlm@364 356 (make-all-images)
rlm@364 357 (compile-left-right)
rlm@364 358 (compile-big-small))
rlm@364 359
rlm@364 360
rlm@364 361 #+end_src
rlm@364 362
rlm@364 363
rlm@370 364 #+name: make-videos
rlm@370 365 #+begin_src makefile
rlm@370 366 scale:
rlm@370 367 ffmpeg -framerate 60 -i ./big-small/%07d.png -b:v 9000k\
rlm@370 368 -c:v theora -r 60 gabor-scale.ogg
rlm@364 369
rlm@370 370 rotation:
rlm@370 371 ffmpeg -framerate 60 -i ./left-right/%07d.png -b:v 9000k\
rlm@370 372 -c:v theora -r 60 gabor-rotation.ogg
rlm@370 373
rlm@370 374 all: rotation scale
rlm@370 375
rlm@370 376 clean:
rlm@370 377 rm gabor-rotation.org gabor-scale.ogg
rlm@364 378 #+end_src
rlm@364 379
rlm@364 380
rlm@365 381 #+begin_html
rlm@365 382 <div class="figure">
rlm@370 383 <video controls="controls">
rlm@372 384 <source src="../video/gabor-rotation.ogg" type="video/ogg"
rlm@365 385 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@365 386 </video>
rlm@367 387 <br> <a href="http://youtu.be/bxujjO97B-U"> YouTube </a>
rlm@365 388 <p>Two gabor filters with different values of theta are compared. The
rlm@365 389 horizontally aligned one does better in this example.</p>
rlm@365 390 </div>
rlm@365 391 #+end_html
rlm@365 392
rlm@365 393
rlm@365 394
rlm@365 395 #+begin_html
rlm@365 396 <div class="figure">
rlm@370 397 <video controls="controls">
rlm@370 398 <source src="../video/gabor-scale.ogg" type="video/ogg"
rlm@365 399 preload="none" poster="../images/aurellem-1280x480.png" />
rlm@365 400 </video>
rlm@367 401 <br> <a href="http://youtu.be/-EsfA2ceWUk"> YouTube </a>
rlm@365 402 <p>Here we compare a gabor filter from bank 1 with one from bank 6.</p>
rlm@365 403 </div>
rlm@365 404 #+end_html
rlm@365 405
rlm@374 406 * Source Listing
rlm@374 407 - [[../src/cortex/gabor.clj][cortex.gabor]]
rlm@374 408 #+html: <ul> <li> <a href="../org/gabor.org">This org file</a> </li> </ul>
rlm@374 409 - [[http://hg.bortreb.com ][source-repository]]
rlm@374 410
rlm@365 411
rlm@356 412
rlm@356 413 * COMMENT Generate Source
rlm@356 414 #+begin_src clojure :tangle ../src/cortex/gabor.clj
rlm@356 415 <<gabor>>
rlm@364 416 <<gabor-tail>>
rlm@356 417 #+end_src
rlm@364 418
rlm@370 419 #+begin_src Makefile :tangle ../render/gabor-1/Makefile
rlm@370 420 <<make-videos>>
rlm@370 421 #+end_src