|
rlm@37
|
1 #+title: Simulated Sense of Touch
|
|
rlm@0
|
2 #+author: Robert McIntyre
|
|
rlm@0
|
3 #+email: rlm@mit.edu
|
|
rlm@37
|
4 #+description: Simulated touch for AI research using JMonkeyEngine and clojure.
|
|
rlm@37
|
5 #+keywords: simulation, tactile sense, jMonkeyEngine3, clojure
|
|
rlm@4
|
6 #+SETUPFILE: ../../aurellem/org/setup.org
|
|
rlm@4
|
7 #+INCLUDE: ../../aurellem/org/level-0.org
|
|
rlm@0
|
8
|
|
rlm@37
|
9 * Touch
|
|
rlm@0
|
10
|
|
rlm@226
|
11 Touch is critical to navigation and spatial reasoning and as such I
|
|
rlm@226
|
12 need a simulated version of it to give to my AI creatures.
|
|
rlm@0
|
13
|
|
rlm@228
|
14 However, touch in my virtual can not exactly correspond to human touch
|
|
rlm@228
|
15 because my creatures are made out of completely rigid segments that
|
|
rlm@228
|
16 don't deform like human skin.
|
|
rlm@228
|
17
|
|
rlm@228
|
18 Human skin has a wide array of touch sensors, each of which speciliaze
|
|
rlm@228
|
19 in detecting different vibrational modes and pressures. These sensors
|
|
rlm@228
|
20 can integrate a vast expanse of skin (i.e. your entire palm), or a
|
|
rlm@228
|
21 tiny patch of skin at the tip of your finger. The hairs of the skin
|
|
rlm@228
|
22 help detect objects before they even come into contact with the skin
|
|
rlm@247
|
23 proper.
|
|
rlm@228
|
24
|
|
rlm@228
|
25 Instead of measuring deformation or vibration, I surround each rigid
|
|
rlm@247
|
26 part with a plenitude of hair-like objects (/feelers/) which do not
|
|
rlm@247
|
27 interact with the physical world. Physical objects can pass through
|
|
rlm@247
|
28 them with no effect. The feelers are able to measure contact with
|
|
rlm@247
|
29 other objects, and constantly report how much of their extent is
|
|
rlm@247
|
30 covered. So, even though the creature's body parts do not deform, the
|
|
rlm@247
|
31 feelers create a margin around those body parts which achieves a sense
|
|
rlm@247
|
32 of touch which is a hybrid between a human's sense of deformation and
|
|
rlm@247
|
33 sense from hairs.
|
|
rlm@228
|
34
|
|
rlm@228
|
35 Implementing touch in jMonkeyEngine follows a different techinal route
|
|
rlm@228
|
36 than vision and hearing. Those two senses piggybacked off
|
|
rlm@228
|
37 jMonkeyEngine's 3D audio and video rendering subsystems. To simulate
|
|
rlm@247
|
38 touch, I use jMonkeyEngine's physics system to execute many small
|
|
rlm@247
|
39 collision detections, one for each feeler. The placement of the
|
|
rlm@247
|
40 feelers is determined by a UV-mapped image which shows where each
|
|
rlm@247
|
41 feeler should be on the 3D surface of the body.
|
|
rlm@228
|
42
|
|
rlm@229
|
43 * Defining Touch Meta-Data in Blender
|
|
rlm@229
|
44
|
|
rlm@245
|
45 Each geometry can have a single UV map which describes the position of
|
|
rlm@247
|
46 the feelers which will constitute its sense of touch. This image path
|
|
rlm@245
|
47 is stored under the "touch" key. The image itself is black and white,
|
|
rlm@247
|
48 with black meaning a feeler length of 0 (no feeler is present) and
|
|
rlm@247
|
49 white meaning a feeler length of =scale=, which is a float stored
|
|
rlm@247
|
50 under the key "scale".
|
|
rlm@229
|
51
|
|
rlm@231
|
52 #+name: meta-data
|
|
rlm@0
|
53 #+begin_src clojure
|
|
rlm@229
|
54 (defn tactile-sensor-profile
|
|
rlm@229
|
55 "Return the touch-sensor distribution image in BufferedImage format,
|
|
rlm@229
|
56 or nil if it does not exist."
|
|
rlm@229
|
57 [#^Geometry obj]
|
|
rlm@229
|
58 (if-let [image-path (meta-data obj "touch")]
|
|
rlm@229
|
59 (load-image image-path)))
|
|
rlm@233
|
60
|
|
rlm@233
|
61 (defn tactile-scale
|
|
rlm@247
|
62 "Return the length of each feeler. Default scale is 0.01
|
|
rlm@247
|
63 jMonkeyEngine units."
|
|
rlm@233
|
64 [#^Geometry obj]
|
|
rlm@233
|
65 (if-let [scale (meta-data obj "scale")]
|
|
rlm@233
|
66 scale 0.1))
|
|
rlm@228
|
67 #+end_src
|
|
rlm@156
|
68
|
|
rlm@246
|
69 Here is an example of a UV-map which specifies the position of touch
|
|
rlm@247
|
70 sensors along the surface of the upper segment of the worm.
|
|
rlm@229
|
71
|
|
rlm@246
|
72 #+attr_html: width=755
|
|
rlm@246
|
73 #+caption: This is the tactile-sensor-profile for the upper segment of the worm. It defines regions of high touch sensitivity (where there are many white pixels) and regions of low sensitivity (where white pixels are sparse).
|
|
rlm@246
|
74 [[../images/finger-UV.png]]
|
|
rlm@234
|
75
|
|
rlm@247
|
76 * Implementation Summary
|
|
rlm@247
|
77
|
|
rlm@247
|
78 To simulate touch there are three conceptual steps. For each solid
|
|
rlm@247
|
79 object in the creature, you first have to get UV image and scale
|
|
rlm@247
|
80 paramater which define the position and length of the feelers. Then,
|
|
rlm@247
|
81 you use the triangles which compose the mesh and the UV data stored in
|
|
rlm@247
|
82 the mesh to determine the world-space position and orientation of each
|
|
rlm@247
|
83 feeler. Once every frame, update these positions and orientations to
|
|
rlm@247
|
84 match the current position and orientation of the object, and use
|
|
rlm@247
|
85 physics collision detection to gather tactile data.
|
|
rlm@238
|
86
|
|
rlm@247
|
87 Extracting the meta-data has already been described. The third step,
|
|
rlm@247
|
88 physics collision detection, is handled in =(touch-kernel)=.
|
|
rlm@247
|
89 Translating the positions and orientations of the feelers from the
|
|
rlm@247
|
90 UV-map to world-space is also a three-step process.
|
|
rlm@239
|
91
|
|
rlm@238
|
92 - Find the triangles which make up the mesh in pixel-space and in
|
|
rlm@247
|
93 world-space. =(triangles)= =(pixel-triangles)=.
|
|
rlm@239
|
94
|
|
rlm@247
|
95 - Find the coordinates of each feeler in world-space. These are the
|
|
rlm@247
|
96 origins of the feelers. =(feeler-origins)=.
|
|
rlm@239
|
97
|
|
rlm@238
|
98 - Calculate the normals of the triangles in world space, and add
|
|
rlm@238
|
99 them to each of the origins of the feelers. These are the
|
|
rlm@247
|
100 normalized coordinates of the tips of the feelers. =(feeler-tips)=.
|
|
rlm@239
|
101
|
|
rlm@247
|
102 * Triangle Math
|
|
rlm@247
|
103 ** Schrapnel Conversion Functions
|
|
rlm@239
|
104
|
|
rlm@247
|
105 #+name: triangles-1
|
|
rlm@247
|
106 #+begin_src clojure
|
|
rlm@247
|
107 (defn vector3f-seq [#^Vector3f v]
|
|
rlm@247
|
108 [(.getX v) (.getY v) (.getZ v)])
|
|
rlm@247
|
109
|
|
rlm@247
|
110 (defn triangle-seq [#^Triangle tri]
|
|
rlm@247
|
111 [(vector3f-seq (.get1 tri))
|
|
rlm@247
|
112 (vector3f-seq (.get2 tri))
|
|
rlm@247
|
113 (vector3f-seq (.get3 tri))])
|
|
rlm@247
|
114
|
|
rlm@247
|
115 (defn ->vector3f
|
|
rlm@247
|
116 ([coords] (Vector3f. (nth coords 0 0)
|
|
rlm@247
|
117 (nth coords 1 0)
|
|
rlm@247
|
118 (nth coords 2 0))))
|
|
rlm@247
|
119
|
|
rlm@247
|
120 (defn ->triangle [points]
|
|
rlm@247
|
121 (apply #(Triangle. %1 %2 %3) (map ->vector3f points)))
|
|
rlm@247
|
122 #+end_src
|
|
rlm@247
|
123
|
|
rlm@247
|
124 It is convienent to treat a =Triangle= as a sequence of verticies, and
|
|
rlm@247
|
125 a =Vector2f= and =Vector3f= as a sequence of floats. These conversion
|
|
rlm@247
|
126 functions make this easy. If these classes implemented =Iterable= then
|
|
rlm@247
|
127 =(seq)= would work on them automitacally.
|
|
rlm@247
|
128 ** Decomposing a 3D shape into Triangles
|
|
rlm@247
|
129
|
|
rlm@247
|
130 The rigid bodies which make up a creature have an underlying
|
|
rlm@247
|
131 =Geometry=, which is a =Mesh= plus a =Material= and other important
|
|
rlm@247
|
132 data involved with displaying the body.
|
|
rlm@247
|
133
|
|
rlm@247
|
134 A =Mesh= is composed of =Triangles=, and each =Triangle= has three
|
|
rlm@247
|
135 verticies which have coordinates in world space and UV space.
|
|
rlm@247
|
136
|
|
rlm@247
|
137 Here, =(triangles)= gets all the world-space triangles which compose a
|
|
rlm@247
|
138 mesh, while =(pixel-triangles)= gets those same triangles expressed in
|
|
rlm@247
|
139 pixel coordinates (which are UV coordinates scaled to fit the height
|
|
rlm@247
|
140 and width of the UV image).
|
|
rlm@247
|
141
|
|
rlm@247
|
142 #+name: triangles-2
|
|
rlm@247
|
143 #+begin_src clojure
|
|
rlm@247
|
144 (in-ns 'cortex.touch)
|
|
rlm@247
|
145 (defn triangle
|
|
rlm@247
|
146 "Get the triangle specified by triangle-index from the mesh."
|
|
rlm@247
|
147 [#^Geometry geo triangle-index]
|
|
rlm@247
|
148 (triangle-seq
|
|
rlm@247
|
149 (let [scratch (Triangle.)]
|
|
rlm@247
|
150 (.getTriangle (.getMesh geo) triangle-index scratch) scratch)))
|
|
rlm@247
|
151
|
|
rlm@247
|
152 (defn triangles
|
|
rlm@247
|
153 "Return a sequence of all the Triangles which compose a given
|
|
rlm@247
|
154 Geometry."
|
|
rlm@247
|
155 [#^Geometry geo]
|
|
rlm@247
|
156 (map (partial triangle geo) (range (.getTriangleCount (.getMesh geo)))))
|
|
rlm@247
|
157
|
|
rlm@247
|
158 (defn triangle-vertex-indices
|
|
rlm@247
|
159 "Get the triangle vertex indices of a given triangle from a given
|
|
rlm@247
|
160 mesh."
|
|
rlm@247
|
161 [#^Mesh mesh triangle-index]
|
|
rlm@247
|
162 (let [indices (int-array 3)]
|
|
rlm@247
|
163 (.getTriangle mesh triangle-index indices)
|
|
rlm@247
|
164 (vec indices)))
|
|
rlm@247
|
165
|
|
rlm@247
|
166 (defn vertex-UV-coord
|
|
rlm@247
|
167 "Get the UV-coordinates of the vertex named by vertex-index"
|
|
rlm@247
|
168 [#^Mesh mesh vertex-index]
|
|
rlm@247
|
169 (let [UV-buffer
|
|
rlm@247
|
170 (.getData
|
|
rlm@247
|
171 (.getBuffer
|
|
rlm@247
|
172 mesh
|
|
rlm@247
|
173 VertexBuffer$Type/TexCoord))]
|
|
rlm@247
|
174 [(.get UV-buffer (* vertex-index 2))
|
|
rlm@247
|
175 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
|
|
rlm@247
|
176
|
|
rlm@247
|
177 (defn pixel-triangle [#^Geometry geo image index]
|
|
rlm@247
|
178 (let [mesh (.getMesh geo)
|
|
rlm@247
|
179 width (.getWidth image)
|
|
rlm@247
|
180 height (.getHeight image)]
|
|
rlm@247
|
181 (vec (map (fn [[u v]] (vector (* width u) (* height v)))
|
|
rlm@247
|
182 (map (partial vertex-UV-coord mesh)
|
|
rlm@247
|
183 (triangle-vertex-indices mesh index))))))
|
|
rlm@247
|
184
|
|
rlm@247
|
185 (defn pixel-triangles [#^Geometry geo image]
|
|
rlm@247
|
186 (let [height (.getHeight image)
|
|
rlm@247
|
187 width (.getWidth image)]
|
|
rlm@247
|
188 (map (partial pixel-triangle geo image)
|
|
rlm@247
|
189 (range (.getTriangleCount (.getMesh geo))))))
|
|
rlm@247
|
190 #+end_src
|
|
rlm@247
|
191 ** The Affine Transform from one Triangle to Another
|
|
rlm@247
|
192
|
|
rlm@247
|
193 =(pixel-triangles)= gives us the mesh triangles expressed in pixel
|
|
rlm@247
|
194 coordinates and =(triangles)= gives us the mesh triangles expressed in
|
|
rlm@247
|
195 world coordinates. The tactile-sensor-profile gives the position of
|
|
rlm@247
|
196 each feeler in pixel-space. In order to convert pixel-dpace
|
|
rlm@247
|
197 coordinates into world-space coordinates we need something that takes
|
|
rlm@247
|
198 coordinates on the surface of one triangle and gives the corresponding
|
|
rlm@247
|
199 coordinates on the surface of another triangle.
|
|
rlm@247
|
200
|
|
rlm@247
|
201 Triangles are [[http://mathworld.wolfram.com/AffineTransformation.html ][affine]], which means any triangle can be transformed into
|
|
rlm@247
|
202 any other by a combination of translation, scaling, and
|
|
rlm@247
|
203 rotation. jMonkeyEngine's =Matrix4f= objects can describe any affine
|
|
rlm@247
|
204 transformation. The affine transformation from one triangle to another
|
|
rlm@247
|
205 is readily computable if the triangle is expressed in terms of a $4x4$
|
|
rlm@247
|
206 matrix.
|
|
rlm@247
|
207
|
|
rlm@247
|
208 \begin{bmatrix}
|
|
rlm@247
|
209 x_1 & x_2 & x_3 & n_x \\
|
|
rlm@247
|
210 y_1 & y_2 & y_3 & n_y \\
|
|
rlm@247
|
211 z_1 & z_2 & z_3 & n_z \\
|
|
rlm@247
|
212 1 & 1 & 1 & 1
|
|
rlm@247
|
213 \end{bmatrix}
|
|
rlm@247
|
214
|
|
rlm@247
|
215 Here, the first three columns of the matrix are the verticies of the
|
|
rlm@247
|
216 triangle. The last column is the right-handed unit normal of the
|
|
rlm@247
|
217 triangle.
|
|
rlm@247
|
218
|
|
rlm@247
|
219 With two triangles $T_{1}$ and $T_{2}$ each expressed as a matrix like
|
|
rlm@247
|
220 above, the affine transform from $T_{1}$ to $T_{2}$ is
|
|
rlm@247
|
221
|
|
rlm@247
|
222 $T_{2}T_{1}^{-1}$
|
|
rlm@247
|
223
|
|
rlm@247
|
224 The clojure code below recaptiulates the formulas above.
|
|
rlm@247
|
225
|
|
rlm@247
|
226 #+name: triangles-3
|
|
rlm@247
|
227 #+begin_src clojure
|
|
rlm@247
|
228 (in-ns 'cortex.touch)
|
|
rlm@247
|
229
|
|
rlm@247
|
230 (defn triangle->matrix4f
|
|
rlm@247
|
231 "Converts the triangle into a 4x4 matrix: The first three columns
|
|
rlm@247
|
232 contain the vertices of the triangle; the last contains the unit
|
|
rlm@247
|
233 normal of the triangle. The bottom row is filled with 1s."
|
|
rlm@247
|
234 [#^Triangle t]
|
|
rlm@247
|
235 (let [mat (Matrix4f.)
|
|
rlm@247
|
236 [vert-1 vert-2 vert-3]
|
|
rlm@247
|
237 ((comp vec map) #(.get t %) (range 3))
|
|
rlm@247
|
238 unit-normal (do (.calculateNormal t)(.getNormal t))
|
|
rlm@247
|
239 vertices [vert-1 vert-2 vert-3 unit-normal]]
|
|
rlm@247
|
240 (dorun
|
|
rlm@247
|
241 (for [row (range 4) col (range 3)]
|
|
rlm@247
|
242 (do
|
|
rlm@247
|
243 (.set mat col row (.get (vertices row) col))
|
|
rlm@247
|
244 (.set mat 3 row 1)))) mat))
|
|
rlm@247
|
245
|
|
rlm@247
|
246 (defn triangles->affine-transform
|
|
rlm@247
|
247 "Returns the affine transformation that converts each vertex in the
|
|
rlm@247
|
248 first triangle into the corresponding vertex in the second
|
|
rlm@247
|
249 triangle."
|
|
rlm@247
|
250 [#^Triangle tri-1 #^Triangle tri-2]
|
|
rlm@247
|
251 (.mult
|
|
rlm@247
|
252 (triangle->matrix4f tri-2)
|
|
rlm@247
|
253 (.invert (triangle->matrix4f tri-1))))
|
|
rlm@247
|
254 #+end_src
|
|
rlm@247
|
255 ** Triangle Boundaries
|
|
rlm@247
|
256
|
|
rlm@247
|
257 For efficiency's sake I will divide the tactile-profile image into
|
|
rlm@247
|
258 small squares which inscribe each pixel-triangle, then extract the
|
|
rlm@247
|
259 points which lie inside the triangle and map them to 3D-space using
|
|
rlm@247
|
260 =(triangle-transform)= above. To do this I need a function,
|
|
rlm@247
|
261 =(convex-bounds)= which finds the smallest box which inscribes a 2D
|
|
rlm@247
|
262 triangle.
|
|
rlm@247
|
263
|
|
rlm@247
|
264 =(inside-triangle?)= determines whether a point is inside a triangle
|
|
rlm@247
|
265 in 2D pixel-space.
|
|
rlm@247
|
266
|
|
rlm@247
|
267 #+name: triangles-4
|
|
rlm@247
|
268 #+begin_src clojure
|
|
rlm@247
|
269 (defn convex-bounds
|
|
rlm@247
|
270 "Returns the smallest square containing the given vertices, as a
|
|
rlm@247
|
271 vector of integers [left top width height]."
|
|
rlm@247
|
272 [verts]
|
|
rlm@247
|
273 (let [xs (map first verts)
|
|
rlm@247
|
274 ys (map second verts)
|
|
rlm@247
|
275 x0 (Math/floor (apply min xs))
|
|
rlm@247
|
276 y0 (Math/floor (apply min ys))
|
|
rlm@247
|
277 x1 (Math/ceil (apply max xs))
|
|
rlm@247
|
278 y1 (Math/ceil (apply max ys))]
|
|
rlm@247
|
279 [x0 y0 (- x1 x0) (- y1 y0)]))
|
|
rlm@247
|
280
|
|
rlm@247
|
281 (defn same-side?
|
|
rlm@247
|
282 "Given the points p1 and p2 and the reference point ref, is point p
|
|
rlm@247
|
283 on the same side of the line that goes through p1 and p2 as ref is?"
|
|
rlm@247
|
284 [p1 p2 ref p]
|
|
rlm@247
|
285 (<=
|
|
rlm@247
|
286 0
|
|
rlm@247
|
287 (.dot
|
|
rlm@247
|
288 (.cross (.subtract p2 p1) (.subtract p p1))
|
|
rlm@247
|
289 (.cross (.subtract p2 p1) (.subtract ref p1)))))
|
|
rlm@247
|
290
|
|
rlm@247
|
291 (defn inside-triangle?
|
|
rlm@247
|
292 "Is the point inside the triangle?"
|
|
rlm@247
|
293 {:author "Dylan Holmes"}
|
|
rlm@247
|
294 [#^Triangle tri #^Vector3f p]
|
|
rlm@247
|
295 (let [[vert-1 vert-2 vert-3] [(.get1 tri) (.get2 tri) (.get3 tri)]]
|
|
rlm@247
|
296 (and
|
|
rlm@247
|
297 (same-side? vert-1 vert-2 vert-3 p)
|
|
rlm@247
|
298 (same-side? vert-2 vert-3 vert-1 p)
|
|
rlm@247
|
299 (same-side? vert-3 vert-1 vert-2 p))))
|
|
rlm@247
|
300 #+end_src
|
|
rlm@247
|
301
|
|
rlm@247
|
302 * Feeler Coordinates
|
|
rlm@247
|
303
|
|
rlm@247
|
304 The triangle-related functions above make short work of calculating
|
|
rlm@247
|
305 the positions and orientations of each feeler in world-space.
|
|
rlm@247
|
306
|
|
rlm@247
|
307 #+name: sensors
|
|
rlm@247
|
308 #+begin_src clojure
|
|
rlm@247
|
309 (in-ns 'cortex.touch)
|
|
rlm@247
|
310
|
|
rlm@247
|
311 (defn feeler-pixel-coords
|
|
rlm@247
|
312 "Returns the coordinates of the feelers in pixel space in lists, one
|
|
rlm@247
|
313 list for each triangle, ordered in the same way as (triangles) and
|
|
rlm@247
|
314 (pixel-triangles)."
|
|
rlm@247
|
315 [#^Geometry geo image]
|
|
rlm@247
|
316 (map
|
|
rlm@247
|
317 (fn [pixel-triangle]
|
|
rlm@247
|
318 (filter
|
|
rlm@247
|
319 (fn [coord]
|
|
rlm@247
|
320 (inside-triangle? (->triangle pixel-triangle)
|
|
rlm@247
|
321 (->vector3f coord)))
|
|
rlm@247
|
322 (white-coordinates image (convex-bounds pixel-triangle))))
|
|
rlm@247
|
323 (pixel-triangles geo image)))
|
|
rlm@247
|
324
|
|
rlm@247
|
325 (defn feeler-world-coords
|
|
rlm@247
|
326 "Returns the coordinates of the feelers in world space in lists, one
|
|
rlm@247
|
327 list for each triangle, ordered in the same way as (triangles) and
|
|
rlm@247
|
328 (pixel-triangles)."
|
|
rlm@247
|
329 [#^Geometry geo image]
|
|
rlm@247
|
330 (let [transforms
|
|
rlm@247
|
331 (map #(triangles->affine-transform
|
|
rlm@247
|
332 (->triangle %1) (->triangle %2))
|
|
rlm@247
|
333 (pixel-triangles geo image)
|
|
rlm@247
|
334 (triangles geo))]
|
|
rlm@247
|
335 (map (fn [transform coords]
|
|
rlm@247
|
336 (map #(.mult transform (->vector3f %)) coords))
|
|
rlm@247
|
337 transforms (feeler-pixel-coords geo image))))
|
|
rlm@247
|
338
|
|
rlm@247
|
339 (defn feeler-origins
|
|
rlm@247
|
340 "The world space coordinates of the root of each feeler."
|
|
rlm@247
|
341 [#^Geometry geo image]
|
|
rlm@247
|
342 (reduce concat (feeler-world-coords geo image)))
|
|
rlm@247
|
343
|
|
rlm@247
|
344 (defn feeler-tips
|
|
rlm@247
|
345 "The world space coordinates of the tip of each feeler."
|
|
rlm@247
|
346 [#^Geometry geo image]
|
|
rlm@247
|
347 (let [world-coords (feeler-world-coords geo image)
|
|
rlm@247
|
348 normals
|
|
rlm@247
|
349 (map
|
|
rlm@247
|
350 (fn [triangle]
|
|
rlm@247
|
351 (.calculateNormal triangle)
|
|
rlm@247
|
352 (.clone (.getNormal triangle)))
|
|
rlm@247
|
353 (map ->triangle (triangles geo)))]
|
|
rlm@247
|
354
|
|
rlm@247
|
355 (mapcat (fn [origins normal]
|
|
rlm@247
|
356 (map #(.add % normal) origins))
|
|
rlm@247
|
357 world-coords normals)))
|
|
rlm@247
|
358
|
|
rlm@247
|
359 (defn touch-topology
|
|
rlm@247
|
360 "touch-topology? is not a function."
|
|
rlm@247
|
361 [#^Geometry geo image]
|
|
rlm@247
|
362 (collapse (reduce concat (feeler-pixel-coords geo image))))
|
|
rlm@247
|
363 #+end_src
|
|
rlm@247
|
364 * Simulated Touch
|
|
rlm@247
|
365
|
|
rlm@247
|
366 =(touch-kernel)= generates functions to be called from within a
|
|
rlm@247
|
367 simulation that perform the necessary physics collisions to collect
|
|
rlm@247
|
368 tactile data, and =(touch!)= recursively applies it to every node in
|
|
rlm@247
|
369 the creature.
|
|
rlm@238
|
370
|
|
rlm@233
|
371 #+name: kernel
|
|
rlm@233
|
372 #+begin_src clojure
|
|
rlm@233
|
373 (in-ns 'cortex.touch)
|
|
rlm@233
|
374
|
|
rlm@244
|
375 (defn set-ray [#^Ray ray #^Matrix4f transform
|
|
rlm@244
|
376 #^Vector3f origin #^Vector3f tip]
|
|
rlm@243
|
377 ;; Doing everything locally recduces garbage collection by enough to
|
|
rlm@243
|
378 ;; be worth it.
|
|
rlm@243
|
379 (.mult transform origin (.getOrigin ray))
|
|
rlm@243
|
380
|
|
rlm@243
|
381 (.mult transform tip (.getDirection ray))
|
|
rlm@244
|
382 (.subtractLocal (.getDirection ray) (.getOrigin ray)))
|
|
rlm@242
|
383
|
|
rlm@233
|
384 (defn touch-kernel
|
|
rlm@234
|
385 "Constructs a function which will return tactile sensory data from
|
|
rlm@234
|
386 'geo when called from inside a running simulation"
|
|
rlm@234
|
387 [#^Geometry geo]
|
|
rlm@243
|
388 (if-let
|
|
rlm@243
|
389 [profile (tactile-sensor-profile geo)]
|
|
rlm@243
|
390 (let [ray-reference-origins (feeler-origins geo profile)
|
|
rlm@243
|
391 ray-reference-tips (feeler-tips geo profile)
|
|
rlm@244
|
392 ray-length (tactile-scale geo)
|
|
rlm@243
|
393 current-rays (map (fn [_] (Ray.)) ray-reference-origins)
|
|
rlm@243
|
394 topology (touch-topology geo profile)]
|
|
rlm@244
|
395 (dorun (map #(.setLimit % ray-length) current-rays))
|
|
rlm@233
|
396 (fn [node]
|
|
rlm@243
|
397 (let [transform (.getWorldMatrix geo)]
|
|
rlm@243
|
398 (dorun
|
|
rlm@244
|
399 (map (fn [ray ref-origin ref-tip]
|
|
rlm@244
|
400 (set-ray ray transform ref-origin ref-tip))
|
|
rlm@243
|
401 current-rays ray-reference-origins
|
|
rlm@244
|
402 ray-reference-tips))
|
|
rlm@233
|
403 (vector
|
|
rlm@243
|
404 topology
|
|
rlm@233
|
405 (vec
|
|
rlm@243
|
406 (for [ray current-rays]
|
|
rlm@233
|
407 (do
|
|
rlm@233
|
408 (let [results (CollisionResults.)]
|
|
rlm@233
|
409 (.collideWith node ray results)
|
|
rlm@233
|
410 (let [touch-objects
|
|
rlm@233
|
411 (filter #(not (= geo (.getGeometry %)))
|
|
rlm@233
|
412 results)]
|
|
rlm@233
|
413 [(if (empty? touch-objects)
|
|
rlm@243
|
414 (.getLimit ray)
|
|
rlm@243
|
415 (.getDistance (first touch-objects)))
|
|
rlm@243
|
416 (.getLimit ray)])))))))))))
|
|
rlm@233
|
417
|
|
rlm@233
|
418 (defn touch!
|
|
rlm@233
|
419 "Endow the creature with the sense of touch. Returns a sequence of
|
|
rlm@233
|
420 functions, one for each body part with a tactile-sensor-proile,
|
|
rlm@233
|
421 each of which when called returns sensory data for that body part."
|
|
rlm@233
|
422 [#^Node creature]
|
|
rlm@233
|
423 (filter
|
|
rlm@233
|
424 (comp not nil?)
|
|
rlm@233
|
425 (map touch-kernel
|
|
rlm@233
|
426 (filter #(isa? (class %) Geometry)
|
|
rlm@233
|
427 (node-seq creature)))))
|
|
rlm@233
|
428 #+end_src
|
|
rlm@233
|
429
|
|
rlm@247
|
430 * Visualizing Touch
|
|
rlm@238
|
431
|
|
rlm@233
|
432 #+name: visualization
|
|
rlm@233
|
433 #+begin_src clojure
|
|
rlm@233
|
434 (in-ns 'cortex.touch)
|
|
rlm@233
|
435
|
|
rlm@233
|
436 (defn touch->gray
|
|
rlm@245
|
437 "Convert a pair of [distance, max-distance] into a grayscale pixel."
|
|
rlm@233
|
438 [distance max-distance]
|
|
rlm@245
|
439 (gray (- 255 (rem (int (* 255 (/ distance max-distance))) 256))))
|
|
rlm@233
|
440
|
|
rlm@233
|
441 (defn view-touch
|
|
rlm@245
|
442 "Creates a function which accepts a list of touch sensor-data and
|
|
rlm@233
|
443 displays each element to the screen."
|
|
rlm@233
|
444 []
|
|
rlm@233
|
445 (view-sense
|
|
rlm@246
|
446 (fn [[coords sensor-data]]
|
|
rlm@233
|
447 (let [image (points->image coords)]
|
|
rlm@233
|
448 (dorun
|
|
rlm@233
|
449 (for [i (range (count coords))]
|
|
rlm@233
|
450 (.setRGB image ((coords i) 0) ((coords i) 1)
|
|
rlm@246
|
451 (apply touch->gray (sensor-data i))))) image))))
|
|
rlm@233
|
452 #+end_src
|
|
rlm@232
|
453 * Adding Touch to the Worm
|
|
rlm@232
|
454
|
|
rlm@232
|
455 #+name: test-touch
|
|
rlm@232
|
456 #+begin_src clojure
|
|
rlm@232
|
457 (ns cortex.test.touch
|
|
rlm@232
|
458 (:use (cortex world util sense body touch))
|
|
rlm@232
|
459 (:use cortex.test.body))
|
|
rlm@232
|
460
|
|
rlm@232
|
461 (cortex.import/mega-import-jme3)
|
|
rlm@232
|
462
|
|
rlm@232
|
463 (defn test-touch []
|
|
rlm@232
|
464 (let [the-worm (doto (worm) (body!))
|
|
rlm@232
|
465 touch (touch! the-worm)
|
|
rlm@232
|
466 touch-display (view-touch)]
|
|
rlm@232
|
467 (world (nodify [the-worm (floor)])
|
|
rlm@232
|
468 standard-debug-controls
|
|
rlm@232
|
469
|
|
rlm@232
|
470 (fn [world]
|
|
rlm@244
|
471 (speed-up world)
|
|
rlm@232
|
472 (light-up-everything world))
|
|
rlm@232
|
473
|
|
rlm@232
|
474 (fn [world tpf]
|
|
rlm@246
|
475 (touch-display
|
|
rlm@246
|
476 (map #(% (.getRootNode world)) touch))))))
|
|
rlm@232
|
477 #+end_src
|
|
rlm@247
|
478
|
|
rlm@247
|
479 * Headers
|
|
rlm@247
|
480
|
|
rlm@247
|
481 #+name: touch-header
|
|
rlm@247
|
482 #+begin_src clojure
|
|
rlm@247
|
483 (ns cortex.touch
|
|
rlm@247
|
484 "Simulate the sense of touch in jMonkeyEngine3. Enables any Geometry
|
|
rlm@247
|
485 to be outfitted with touch sensors with density determined by a UV
|
|
rlm@247
|
486 image. In this way a Geometry can know what parts of itself are
|
|
rlm@247
|
487 touching nearby objects. Reads specially prepared blender files to
|
|
rlm@247
|
488 construct this sense automatically."
|
|
rlm@247
|
489 {:author "Robert McIntyre"}
|
|
rlm@247
|
490 (:use (cortex world util sense))
|
|
rlm@247
|
491 (:use clojure.contrib.def)
|
|
rlm@247
|
492 (:import (com.jme3.scene Geometry Node Mesh))
|
|
rlm@247
|
493 (:import com.jme3.collision.CollisionResults)
|
|
rlm@247
|
494 (:import com.jme3.scene.VertexBuffer$Type)
|
|
rlm@247
|
495 (:import (com.jme3.math Triangle Vector3f Vector2f Ray Matrix4f)))
|
|
rlm@247
|
496 #+end_src
|
|
rlm@247
|
497
|
|
rlm@228
|
498 * Source Listing
|
|
rlm@228
|
499 * Next
|
|
rlm@228
|
500
|
|
rlm@228
|
501
|
|
rlm@226
|
502 * COMMENT Code Generation
|
|
rlm@39
|
503 #+begin_src clojure :tangle ../src/cortex/touch.clj
|
|
rlm@231
|
504 <<touch-header>>
|
|
rlm@231
|
505 <<meta-data>>
|
|
rlm@231
|
506 <<triangles-1>>
|
|
rlm@247
|
507 <<triangles-2>>
|
|
rlm@231
|
508 <<triangles-3>>
|
|
rlm@231
|
509 <<triangles-4>>
|
|
rlm@231
|
510 <<sensors>>
|
|
rlm@231
|
511 <<kernel>>
|
|
rlm@231
|
512 <<visualization>>
|
|
rlm@0
|
513 #+end_src
|
|
rlm@0
|
514
|
|
rlm@232
|
515
|
|
rlm@68
|
516 #+begin_src clojure :tangle ../src/cortex/test/touch.clj
|
|
rlm@232
|
517 <<test-touch>>
|
|
rlm@39
|
518 #+end_src
|
|
rlm@39
|
519
|
|
rlm@0
|
520
|
|
rlm@0
|
521
|
|
rlm@0
|
522
|
|
rlm@32
|
523
|
|
rlm@32
|
524
|
|
rlm@226
|
525
|
