view org/touch.org @ 331:0c9e972c85a4

maybe try #including relevant sections?
author Robert McIntyre <rlm@mit.edu>
date Thu, 19 Jul 2012 19:17:52 -0500
parents 5c398b65d01b
children 698d48b91cd5
line wrap: on
line source
1 #+title: Simulated Sense of Touch
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description: Simulated touch for AI research using JMonkeyEngine and clojure.
5 #+keywords: simulation, tactile sense, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
9 * Touch
11 Touch is critical to navigation and spatial reasoning and as such I
12 need a simulated version of it to give to my AI creatures.
14 Human skin has a wide array of touch sensors, each of which specialize
15 in detecting different vibrational modes and pressures. These sensors
16 can integrate a vast expanse of skin (i.e. your entire palm), or a
17 tiny patch of skin at the tip of your finger. The hairs of the skin
18 help detect objects before they even come into contact with the skin
19 proper.
21 However, touch in my simulated world can not exactly correspond to
22 human touch because my creatures are made out of completely rigid
23 segments that don't deform like human skin.
25 Instead of measuring deformation or vibration, I surround each rigid
26 part with a plenitude of hair-like objects (/feelers/) which do not
27 interact with the physical world. Physical objects can pass through
28 them with no effect. The feelers are able to tell when other objects
29 pass through them, and they constantly report how much of their extent
30 is covered. So even though the creature's body parts do not deform,
31 the feelers create a margin around those body parts which achieves a
32 sense of touch which is a hybrid between a human's sense of
33 deformation and sense from hairs.
35 Implementing touch in jMonkeyEngine follows a different technical route
36 than vision and hearing. Those two senses piggybacked off
37 jMonkeyEngine's 3D audio and video rendering subsystems. To simulate
38 touch, I use jMonkeyEngine's physics system to execute many small
39 collision detections, one for each feeler. The placement of the
40 feelers is determined by a UV-mapped image which shows where each
41 feeler should be on the 3D surface of the body.
43 * Defining Touch Meta-Data in Blender
45 Each geometry can have a single UV map which describes the position of
46 the feelers which will constitute its sense of touch. This image path
47 is stored under the "touch" key. The image itself is black and white,
48 with black meaning a feeler length of 0 (no feeler is present) and
49 white meaning a feeler length of =scale=, which is a float stored
50 under the key "scale".
52 #+name: meta-data
53 #+begin_src clojure
54 (defn tactile-sensor-profile
55 "Return the touch-sensor distribution image in BufferedImage format,
56 or nil if it does not exist."
57 [#^Geometry obj]
58 (if-let [image-path (meta-data obj "touch")]
59 (load-image image-path)))
61 (defn tactile-scale
62 "Return the length of each feeler. Default scale is 0.01
63 jMonkeyEngine units."
64 [#^Geometry obj]
65 (if-let [scale (meta-data obj "scale")]
66 scale 0.1))
67 #+end_src
69 Here is an example of a UV-map which specifies the position of touch
70 sensors along the surface of the upper segment of the worm.
72 #+attr_html: width=755
73 #+caption: This is the tactile-sensor-profile for the upper segment of the worm. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).
74 [[../images/finger-UV.png]]
76 * Implementation Summary
78 To simulate touch there are three conceptual steps. For each solid
79 object in the creature, you first have to get UV image and scale
80 parameter which define the position and length of the feelers. Then,
81 you use the triangles which compose the mesh and the UV data stored in
82 the mesh to determine the world-space position and orientation of each
83 feeler. Then once every frame, update these positions and orientations
84 to match the current position and orientation of the object, and use
85 physics collision detection to gather tactile data.
87 Extracting the meta-data has already been described. The third step,
88 physics collision detection, is handled in =touch-kernel=.
89 Translating the positions and orientations of the feelers from the
90 UV-map to world-space is itself a three-step process.
92 - Find the triangles which make up the mesh in pixel-space and in
93 world-space. =triangles= =pixel-triangles=.
95 - Find the coordinates of each feeler in world-space. These are the
96 origins of the feelers. =feeler-origins=.
98 - Calculate the normals of the triangles in world space, and add
99 them to each of the origins of the feelers. These are the
100 normalized coordinates of the tips of the feelers. =feeler-tips=.
102 * Triangle Math
103 ** Shrapnel Conversion Functions
105 #+name: triangles-1
106 #+begin_src clojure
107 (defn vector3f-seq [#^Vector3f v]
108 [(.getX v) (.getY v) (.getZ v)])
110 (defn triangle-seq [#^Triangle tri]
111 [(vector3f-seq (.get1 tri))
112 (vector3f-seq (.get2 tri))
113 (vector3f-seq (.get3 tri))])
115 (defn ->vector3f
116 ([coords] (Vector3f. (nth coords 0 0)
117 (nth coords 1 0)
118 (nth coords 2 0))))
120 (defn ->triangle [points]
121 (apply #(Triangle. %1 %2 %3) (map ->vector3f points)))
122 #+end_src
124 It is convenient to treat a =Triangle= as a vector of vectors, and a
125 =Vector2f= or =Vector3f= as vectors of floats. (->vector3f) and
126 (->triangle) undo the operations of =vector3f-seq= and
127 =triangle-seq=. If these classes implemented =Iterable= then =seq=
128 would work on them automatically.
130 ** Decomposing a 3D shape into Triangles
132 The rigid objects which make up a creature have an underlying
133 =Geometry=, which is a =Mesh= plus a =Material= and other important
134 data involved with displaying the object.
136 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
137 vertices which have coordinates in world space and UV space.
139 Here, =triangles= gets all the world-space triangles which compose a
140 mesh, while =pixel-triangles= gets those same triangles expressed in
141 pixel coordinates (which are UV coordinates scaled to fit the height
142 and width of the UV image).
144 #+name: triangles-2
145 #+begin_src clojure
146 (in-ns 'cortex.touch)
147 (defn triangle
148 "Get the triangle specified by triangle-index from the mesh."
149 [#^Geometry geo triangle-index]
150 (triangle-seq
151 (let [scratch (Triangle.)]
152 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
154 (defn triangles
155 "Return a sequence of all the Triangles which compose a given
156 Geometry."
157 [#^Geometry geo]
158 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
160 (defn triangle-vertex-indices
161 "Get the triangle vertex indices of a given triangle from a given
162 mesh."
163 [#^Mesh mesh triangle-index]
164 (let [indices (int-array 3)]
165 (.getTriangle mesh triangle-index indices)
166 (vec indices)))
168 (defn vertex-UV-coord
169 "Get the UV-coordinates of the vertex named by vertex-index"
170 [#^Mesh mesh vertex-index]
171 (let [UV-buffer
172 (.getData
173 (.getBuffer
174 mesh
175 VertexBuffer$Type/TexCoord))]
176 [(.get UV-buffer (* vertex-index 2))
177 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
179 (defn pixel-triangle [#^Geometry geo image index]
180 (let [mesh (.getMesh geo)
181 width (.getWidth image)
182 height (.getHeight image)]
183 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
184 (map (partial vertex-UV-coord mesh)
185 (triangle-vertex-indices mesh index))))))
187 (defn pixel-triangles
188 "The pixel-space triangles of the Geometry, in the same order as
189 (triangles geo)"
190 [#^Geometry geo image]
191 (let [height (.getHeight image)
192 width (.getWidth image)]
193 (map (partial pixel-triangle geo image)
194 (range (.getTriangleCount (.getMesh geo))))))
195 #+end_src
196 ** The Affine Transform from one Triangle to Another
198 =pixel-triangles= gives us the mesh triangles expressed in pixel
199 coordinates and =triangles= gives us the mesh triangles expressed in
200 world coordinates. The tactile-sensor-profile gives the position of
201 each feeler in pixel-space. In order to convert pixel-space
202 coordinates into world-space coordinates we need something that takes
203 coordinates on the surface of one triangle and gives the corresponding
204 coordinates on the surface of another triangle.
206 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed into
207 any other by a combination of translation, scaling, and
208 rotation. The affine transformation from one triangle to another
209 is readily computable if the triangle is expressed in terms of a $4x4$
210 matrix.
212 \begin{bmatrix}
213 x_1 & x_2 & x_3 & n_x \\
214 y_1 & y_2 & y_3 & n_y \\
215 z_1 & z_2 & z_3 & n_z \\
216 1 & 1 & 1 & 1
217 \end{bmatrix}
219 Here, the first three columns of the matrix are the vertices of the
220 triangle. The last column is the right-handed unit normal of the
221 triangle.
223 With two triangles $T_{1}$ and $T_{2}$ each expressed as a matrix like
224 above, the affine transform from $T_{1}$ to $T_{2}$ is
226 $T_{2}T_{1}^{-1}$
228 The clojure code below recapitulates the formulas above, using
229 jMonkeyEngine's =Matrix4f= objects, which can describe any affine
230 transformation.
232 #+name: triangles-3
233 #+begin_src clojure
234 (in-ns 'cortex.touch)
236 (defn triangle->matrix4f
237 "Converts the triangle into a 4x4 matrix: The first three columns
238 contain the vertices of the triangle; the last contains the unit
239 normal of the triangle. The bottom row is filled with 1s."
240 [#^Triangle t]
241 (let [mat (Matrix4f.)
242 [vert-1 vert-2 vert-3]
243 ((comp vec map) #(.get t %) (range 3))
244 unit-normal (do (.calculateNormal t)(.getNormal t))
245 vertices [vert-1 vert-2 vert-3 unit-normal]]
246 (dorun
247 (for [row (range 4) col (range 3)]
248 (do
249 (.set mat col row (.get (vertices row) col))
250 (.set mat 3 row 1)))) mat))
252 (defn triangles->affine-transform
253 "Returns the affine transformation that converts each vertex in the
254 first triangle into the corresponding vertex in the second
255 triangle."
256 [#^Triangle tri-1 #^Triangle tri-2]
257 (.mult
258 (triangle->matrix4f tri-2)
259 (.invert (triangle->matrix4f tri-1))))
260 #+end_src
261 ** Triangle Boundaries
263 For efficiency's sake I will divide the tactile-profile image into
264 small squares which inscribe each pixel-triangle, then extract the
265 points which lie inside the triangle and map them to 3D-space using
266 =triangle-transform= above. To do this I need a function,
267 =convex-bounds= which finds the smallest box which inscribes a 2D
268 triangle.
270 =inside-triangle?= determines whether a point is inside a triangle
271 in 2D pixel-space.
273 #+name: triangles-4
274 #+begin_src clojure
275 (defn convex-bounds
276 "Returns the smallest square containing the given vertices, as a
277 vector of integers [left top width height]."
278 [verts]
279 (let [xs (map first verts)
280 ys (map second verts)
281 x0 (Math/floor (apply min xs))
282 y0 (Math/floor (apply min ys))
283 x1 (Math/ceil (apply max xs))
284 y1 (Math/ceil (apply max ys))]
285 [x0 y0 (- x1 x0) (- y1 y0)]))
287 (defn same-side?
288 "Given the points p1 and p2 and the reference point ref, is point p
289 on the same side of the line that goes through p1 and p2 as ref is?"
290 [p1 p2 ref p]
291 (<=
292 0
293 (.dot
294 (.cross (.subtract p2 p1) (.subtract p p1))
295 (.cross (.subtract p2 p1) (.subtract ref p1)))))
297 (defn inside-triangle?
298 "Is the point inside the triangle?"
299 {:author "Dylan Holmes"}
300 [#^Triangle tri #^Vector3f p]
301 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
302 (and
303 (same-side? vert-1 vert-2 vert-3 p)
304 (same-side? vert-2 vert-3 vert-1 p)
305 (same-side? vert-3 vert-1 vert-2 p))))
306 #+end_src
308 * Feeler Coordinates
310 The triangle-related functions above make short work of calculating
311 the positions and orientations of each feeler in world-space.
313 #+name: sensors
314 #+begin_src clojure
315 (in-ns 'cortex.touch)
317 (defn feeler-pixel-coords
318 "Returns the coordinates of the feelers in pixel space in lists, one
319 list for each triangle, ordered in the same way as (triangles) and
320 (pixel-triangles)."
321 [#^Geometry geo image]
322 (map
323 (fn [pixel-triangle]
324 (filter
325 (fn [coord]
326 (inside-triangle? (->triangle pixel-triangle)
327 (->vector3f coord)))
328 (white-coordinates image (convex-bounds pixel-triangle))))
329 (pixel-triangles geo image)))
331 (defn feeler-world-coords
332 "Returns the coordinates of the feelers in world space in lists, one
333 list for each triangle, ordered in the same way as (triangles) and
334 (pixel-triangles)."
335 [#^Geometry geo image]
336 (let [transforms
337 (map #(triangles->affine-transform
338 (->triangle %1) (->triangle %2))
339 (pixel-triangles geo image)
340 (triangles geo))]
341 (map (fn [transform coords]
342 (map #(.mult transform (->vector3f %)) coords))
343 transforms (feeler-pixel-coords geo image))))
345 (defn feeler-origins
346 "The world space coordinates of the root of each feeler."
347 [#^Geometry geo image]
348 (reduce concat (feeler-world-coords geo image)))
350 (defn feeler-tips
351 "The world space coordinates of the tip of each feeler."
352 [#^Geometry geo image]
353 (let [world-coords (feeler-world-coords geo image)
354 normals
355 (map
356 (fn [triangle]
357 (.calculateNormal triangle)
358 (.clone (.getNormal triangle)))
359 (map ->triangle (triangles geo)))]
361 (mapcat (fn [origins normal]
362 (map #(.add % normal) origins))
363 world-coords normals)))
365 (defn touch-topology
366 "touch-topology? is not a function."
367 [#^Geometry geo image]
368 (collapse (reduce concat (feeler-pixel-coords geo image))))
369 #+end_src
370 * Simulated Touch
372 =touch-kernel= generates functions to be called from within a
373 simulation that perform the necessary physics collisions to collect
374 tactile data, and =touch!= recursively applies it to every node in
375 the creature.
377 #+name: kernel
378 #+begin_src clojure
379 (in-ns 'cortex.touch)
381 (defn set-ray [#^Ray ray #^Matrix4f transform
382 #^Vector3f origin #^Vector3f tip]
383 ;; Doing everything locally reduces garbage collection by enough to
384 ;; be worth it.
385 (.mult transform origin (.getOrigin ray))
386 (.mult transform tip (.getDirection ray))
387 (.subtractLocal (.getDirection ray) (.getOrigin ray))
388 (.normalizeLocal (.getDirection ray)))
390 (import com.jme3.math.FastMath)
393 (defn touch-kernel
394 "Constructs a function which will return tactile sensory data from
395 'geo when called from inside a running simulation"
396 [#^Geometry geo]
397 (if-let
398 [profile (tactile-sensor-profile geo)]
399 (let [ray-reference-origins (feeler-origins geo profile)
400 ray-reference-tips (feeler-tips geo profile)
401 ray-length (tactile-scale geo)
402 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
403 topology (touch-topology geo profile)
404 correction (float (* ray-length -0.2))]
406 ;; slight tolerance for very close collisions.
407 (dorun
408 (map (fn [origin tip]
409 (.addLocal origin (.mult (.subtract tip origin)
410 correction)))
411 ray-reference-origins ray-reference-tips))
412 (dorun (map #(.setLimit % ray-length) current-rays))
413 (fn [node]
414 (let [transform (.getWorldMatrix geo)]
415 (dorun
416 (map (fn [ray ref-origin ref-tip]
417 (set-ray ray transform ref-origin ref-tip))
418 current-rays ray-reference-origins
419 ray-reference-tips))
420 (vector
421 topology
422 (vec
423 (for [ray current-rays]
424 (do
425 (let [results (CollisionResults.)]
426 (.collideWith node ray results)
427 (let [touch-objects
428 (filter #(not (= geo (.getGeometry %)))
429 results)
430 limit (.getLimit ray)]
431 [(if (empty? touch-objects)
432 limit
433 (let [response
434 (apply min (map #(.getDistance %)
435 touch-objects))]
436 (FastMath/clamp
437 (float
438 (if (> response limit) (float 0.0)
439 (+ response correction)))
440 (float 0.0)
441 limit)))
442 limit])))))))))))
444 (defn touch!
445 "Endow the creature with the sense of touch. Returns a sequence of
446 functions, one for each body part with a tactile-sensor-profile,
447 each of which when called returns sensory data for that body part."
448 [#^Node creature]
449 (filter
450 (comp not nil?)
451 (map touch-kernel
452 (filter #(isa? (class %) Geometry)
453 (node-seq creature)))))
454 #+end_src
456 #+results: kernel
457 : #'cortex.touch/touch!
459 * Visualizing Touch
461 Each feeler is represented in the image as a single pixel. The
462 greyscale value of each pixel represents how deep the feeler
463 represented by that pixel is inside another object. Black means that
464 nothing is touching the feeler, while white means that the feeler is
465 completely inside another object, which is presumably flush with the
466 surface of the triangle from which the feeler originates.
468 #+name: visualization
469 #+begin_src clojure
470 (in-ns 'cortex.touch)
472 (defn touch->gray
473 "Convert a pair of [distance, max-distance] into a gray-scale pixel."
474 [distance max-distance]
475 (gray (- 255 (rem (int (* 255 (/ distance max-distance))) 256))))
477 (defn view-touch
478 "Creates a function which accepts a list of touch sensor-data and
479 displays each element to the screen."
480 []
481 (view-sense
482 (fn [[coords sensor-data]]
483 (let [image (points->image coords)]
484 (dorun
485 (for [i (range (count coords))]
486 (.setRGB image ((coords i) 0) ((coords i) 1)
487 (apply touch->gray (sensor-data i)))))
488 image))))
489 #+end_src
491 #+results: visualization
492 : #'cortex.touch/view-touch
494 * Basic Test of Touch
496 The worm's sense of touch is a bit complicated, so for this basic test
497 I'll use a new creature --- a simple cube which has touch sensors
498 evenly distributed along each of its sides.
500 #+name: test-touch-0
501 #+begin_src clojure
502 (in-ns 'cortex.test.touch)
504 (defn touch-cube []
505 (load-blender-model "Models/test-touch/touch-cube.blend"))
506 #+end_src
508 ** The Touch Cube
509 #+begin_html
510 <div class="figure">
511 <center>
512 <video controls="controls" width="500">
513 <source src="../video/touch-cube.ogg" type="video/ogg"
514 preload="none" poster="../images/aurellem-1280x480.png" />
515 </video>
516 <br> <a href="http://youtu.be/aEao4m8meAI"> YouTube </a>
517 </center>
518 <p>A simple creature with evenly distributed touch sensors.</p>
519 </div>
520 #+end_html
522 The tactile-sensor-profile image for this simple creature looks like
523 this:
525 #+attr_html: width=500
526 #+caption: The distribution of feelers along the touch-cube. The colors of the faces are irrelevant; only the white pixels specify feelers.
527 [[../images/touch-profile.png]]
529 #+name: test-touch-1
530 #+begin_src clojure
531 (in-ns 'cortex.test.touch)
533 (defn test-basic-touch
534 "Testing touch:
535 You should see a cube fall onto a table. There is a cross-shaped
536 display which reports the cube's sensation of touch. This display
537 should change when the cube hits the table, and whenever you hit
538 the cube with balls.
540 Keys:
541 <space> : fire ball"
542 ([] (test-basic-touch false))
543 ([record?]
544 (let [the-cube (doto (touch-cube) (body!))
545 touch (touch! the-cube)
546 touch-display (view-touch)]
547 (world
548 (nodify [the-cube
549 (box 10 1 10 :position (Vector3f. 0 -10 0)
550 :color ColorRGBA/Gray :mass 0)])
552 standard-debug-controls
554 (fn [world]
555 (if record?
556 (Capture/captureVideo
557 world
558 (File. "/home/r/proj/cortex/render/touch-cube/main-view/")))
559 (speed-up world)
560 (light-up-everything world))
562 (fn [world tpf]
563 (touch-display
564 (map #(% (.getRootNode world)) touch)
565 (if record?
566 (File. "/home/r/proj/cortex/render/touch-cube/touch/"))))))))
567 #+end_src
569 ** Basic Touch Demonstration
571 #+begin_html
572 <div class="figure">
573 <center>
574 <video controls="controls" width="755">
575 <source src="../video/basic-touch.ogg" type="video/ogg"
576 preload="none" poster="../images/aurellem-1280x480.png" />
577 </video>
578 <br> <a href="http://youtu.be/8xNEtD-a8f0"> YouTube </a>
579 </center>
580 <p>The simple creature responds to touch.</p>
581 </div>
582 #+end_html
584 ** Generating the Basic Touch Video
585 #+name: magick4
586 #+begin_src clojure
587 (ns cortex.video.magick4
588 (:import java.io.File)
589 (:use clojure.java.shell))
591 (defn images [path]
592 (sort (rest (file-seq (File. path)))))
594 (def base "/home/r/proj/cortex/render/touch-cube/")
596 (defn pics [file]
597 (images (str base file)))
599 (defn combine-images []
600 (let [main-view (pics "main-view")
601 touch (pics "touch/0")
602 background (repeat 9001 (File. (str base "background.png")))
603 targets (map
604 #(File. (str base "out/" (format "%07d.png" %)))
605 (range 0 (count main-view)))]
606 (dorun
607 (pmap
608 (comp
609 (fn [[background main-view touch target]]
610 (println target)
611 (sh "convert"
612 touch
613 "-resize" "x300"
614 "-rotate" "180"
615 background
616 "-swap" "0,1"
617 "-geometry" "+776+129"
618 "-composite"
619 main-view "-geometry" "+66+21" "-composite"
620 target))
621 (fn [& args] (map #(.getCanonicalPath %) args)))
622 background main-view touch targets))))
623 #+end_src
625 #+begin_src sh :results silent
626 cd ~/proj/cortex/render/touch-cube/
627 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora basic-touch.ogg
628 #+end_src
630 #+begin_src sh :results silent
631 cd ~/proj/cortex/render/touch-cube/
632 ffmpeg -r 30 -i blender-intro/%07d.png -b:v 9000k -c:v libtheora touch-cube.ogg
633 #+end_src
635 * Adding Touch to the Worm
637 #+name: test-touch-2
638 #+begin_src clojure
639 (in-ns 'cortex.test.touch)
641 (defn test-worm-touch
642 "Testing touch:
643 You will see the worm fall onto a table. There is a display which
644 reports the worm's sense of touch. It should change when the worm
645 hits the table and when you hit it with balls.
647 Keys:
648 <space> : fire ball"
649 ([] (test-worm-touch false))
650 ([record?]
651 (let [the-worm (doto (worm) (body!))
652 touch (touch! the-worm)
653 touch-display (view-touch)]
654 (world
655 (nodify [the-worm (floor)])
656 standard-debug-controls
658 (fn [world]
659 (if record?
660 (Capture/captureVideo
661 world
662 (File. "/home/r/proj/cortex/render/worm-touch/main-view/")))
663 (speed-up world)
664 (light-up-everything world))
666 (fn [world tpf]
667 (touch-display
668 (map #(% (.getRootNode world)) touch)
669 (if record?
670 (File. "/home/r/proj/cortex/render/worm-touch/touch/"))))))))
671 #+end_src
673 ** Worm Touch Demonstration
674 #+begin_html
675 <div class="figure">
676 <center>
677 <video controls="controls" width="550">
678 <source src="../video/worm-touch.ogg" type="video/ogg"
679 preload="none" poster="../images/aurellem-1280x480.png" />
680 </video>
681 <br> <a href="http://youtu.be/RHx2wqzNVcU"> YouTube </a>
682 </center>
683 <p>The worm responds to touch.</p>
684 </div>
685 #+end_html
688 ** Generating the Worm Touch Video
689 #+name: magick5
690 #+begin_src clojure
691 (ns cortex.video.magick5
692 (:import java.io.File)
693 (:use clojure.java.shell))
695 (defn images [path]
696 (sort (rest (file-seq (File. path)))))
698 (def base "/home/r/proj/cortex/render/worm-touch/")
700 (defn pics [file]
701 (images (str base file)))
703 (defn combine-images []
704 (let [main-view (pics "main-view")
705 touch (pics "touch/0")
706 targets (map
707 #(File. (str base "out/" (format "%07d.png" %)))
708 (range 0 (count main-view)))]
709 (dorun
710 (pmap
711 (comp
712 (fn [[ main-view touch target]]
713 (println target)
714 (sh "convert"
715 main-view
716 touch "-geometry" "+0+0" "-composite"
717 target))
718 (fn [& args] (map #(.getCanonicalPath %) args)))
719 main-view touch targets))))
720 #+end_src
722 #+begin_src sh :results silent
723 cd ~/proj/cortex/render/worm-touch
724 ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora worm-touch.ogg
725 #+end_src
727 * Headers
729 #+name: touch-header
730 #+begin_src clojure
731 (ns cortex.touch
732 "Simulate the sense of touch in jMonkeyEngine3. Enables any Geometry
733 to be outfitted with touch sensors with density determined by a UV
734 image. In this way a Geometry can know what parts of itself are
735 touching nearby objects. Reads specially prepared blender files to
736 construct this sense automatically."
737 {:author "Robert McIntyre"}
738 (:use (cortex world util sense))
739 (:import (com.jme3.scene Geometry Node Mesh))
740 (:import com.jme3.collision.CollisionResults)
741 (:import com.jme3.scene.VertexBuffer$Type)
742 (:import (com.jme3.math Triangle Vector3f Vector2f Ray Matrix4f)))
743 #+end_src
745 #+name: test-touch-header
746 #+begin_src clojure
747 (ns cortex.test.touch
748 (:use (cortex world util sense body touch))
749 (:use cortex.test.body)
750 (:import com.aurellem.capture.Capture)
751 (:import java.io.File)
752 (:import (com.jme3.math Vector3f ColorRGBA)))
753 #+end_src
755 * Source Listing
756 - [[../src/cortex/touch.clj][cortex.touch]]
757 - [[../src/cortex/test/touch.clj][cortex.test.touch]]
758 - [[../src/cortex/video/magick4.clj][cortex.video.magick4]]
759 - [[../src/cortex/video/magick5.clj][cortex.video.magick5]]
760 - [[../assets/Models/test-touch/touch-cube.blend][touch-cube.blend]]
761 #+html: <ul> <li> <a href="../org/touch.org">This org file</a> </li> </ul>
762 - [[http://hg.bortreb.com ][source-repository]]
764 * Next
765 So far I've implemented simulated Vision, Hearing, and Touch, the most
766 obvious and prominent senses that humans have. Smell and Taste shall
767 remain unimplemented for now. This accounts for the "five senses" that
768 feature so prominently in our lives. But humans have far more than the
769 five main senses. There are internal chemical senses, pain (which is
770 *not* the same as touch), heat sensitivity, and our sense of balance,
771 among others. One extra sense is so important that I must implement it
772 to have a hope of making creatures that can gracefully control their
773 own bodies. It is Proprioception, which is the sense of the location
774 of each body part in relation to the other body parts.
776 Close your eyes, and touch your nose with your right index finger. How
777 did you do it? You could not see your hand, and neither your hand nor
778 your nose could use the sense of touch to guide the path of your hand.
779 There are no sound cues, and Taste and Smell certainly don't provide
780 any help. You know where your hand is without your other senses
781 because of Proprioception.
783 Onward to [[./proprioception.org][proprioception]]!
785 * COMMENT Code Generation
786 #+begin_src clojure :tangle ../src/cortex/touch.clj
787 <<touch-header>>
788 <<meta-data>>
789 <<triangles-1>>
790 <<triangles-2>>
791 <<triangles-3>>
792 <<triangles-4>>
793 <<sensors>>
794 <<kernel>>
795 <<visualization>>
796 #+end_src
798 #+begin_src clojure :tangle ../src/cortex/test/touch.clj
799 <<test-touch-header>>
800 <<test-touch-0>>
801 <<test-touch-1>>
802 <<test-touch-2>>
803 #+end_src
805 #+begin_src clojure :tangle ../src/cortex/video/magick4.clj
806 <<magick4>>
807 #+end_src
809 #+begin_src clojure :tangle ../src/cortex/video/magick5.clj
810 <<magick5>>
811 #+end_src