view org/test-creature.org @ 151:aaacf087504c

refactored vision code
author Robert McIntyre <rlm@mit.edu>
date Fri, 03 Feb 2012 05:52:18 -0700
parents 1e6beed24cec
children c95179907951
line wrap: on
line source
1 #+title: First attempt at a creature!
2 #+author: Robert McIntyre
3 #+email: rlm@mit.edu
4 #+description:
5 #+keywords: simulation, jMonkeyEngine3, clojure
6 #+SETUPFILE: ../../aurellem/org/setup.org
7 #+INCLUDE: ../../aurellem/org/level-0.org
11 * Brainstorming different sensors and effectors.
13 Every sense that we have should have an effector that changes what
14 that sense (or others who have that sense) experiences.
16 ** Classic Senses
17 | Sense | Effector |
18 |------------------------------+---------------------------------|
19 | Vision | Variable Coloration |
20 | Hearing | Speech |
21 | Proprioception | Movement |
22 | Smell/Taste (Chemoreception) | Pheremones |
23 | Touch | Movement / Controllable Texture |
24 | Acceleration | Movement |
25 | Balance (sense gravity) | Movement |
26 | | |
28 - New Senses/Effectors
29 - Levitation
30 - Telekenesis
31 - control of gravity within a certain radius
32 - speed up/slow time
33 - object creation/destruction
35 - Symbol Sense
36 Where objects in the world can be queried for description /
37 symbols.
39 - Symbol Marking
40 The ability to mark objects in the world with your own descriptions
41 and symbols.
43 - Vision
44 Distinguish the polarization of light
45 Color
46 Movement
48 * project ideas
49 - HACKER for writing muscle-control programs : Presented with
50 low-level muscle control/ sense API, generate higher level programs
51 for accomplishing various stated goals. Example goals might be
52 "extend all your fingers" or "move your hand into the area with
53 blue light" or "decrease the angle of this joint". It would be
54 like Sussman's HACKER, except it would operate with much more data
55 in a more realistic world. Start off with "calestanthics" to
56 develop subrouitines over the motor control API. This would be the
57 "spinal chord" of a more intelligent creature. The low level
58 programming code might be a turning machine that could develop
59 programs to iterate over a "tape" where each entry in the tape
60 could control recruitment of the fibers in a muscle.
61 - Make a virtual computer in the virtual world which with which the
62 creature interacts using its fingers to press keys on a virtual
63 keyboard. The creature can access the internet, watch videos, take
64 over the world, anything it wants.
65 - Make virtual insturments like pianos, drumbs, etc that it learns to
66 play.
67 - make a joint that figures out what type of joint it is (range of
68 motion)
74 * goals
76 ** have to get done before winston
77 - [ ] write an explination for why greyscale bitmaps for senses is
78 appropiate -- 1/2 day
79 - [X] muscle control -- day
80 - [X] proprioception sensor map in the style of the other senses -- day
81 - [ ] refactor integration code to distribute to each of the senses
82 -- day
83 - [ ] create video showing all the senses for Winston -- 2 days
84 - [ ] send package to friends for critiques -- 2 days
85 - [ ] write summary of project for Winston \
86 - [ ] project proposals for Winston \
87 - [ ] additional senses to be implemented for Winston | -- 2 days
88 - [ ] send Winston package /
90 ** would be cool to get done before winston
91 - [X] enable greyscale bitmaps for touch -- 2 hours
92 - [X] use sawfish to auto-tile sense windows -- 6 hours
93 - [X] sawfish keybinding to automatically delete all sense windows
94 - [ ] directly change the UV-pixels to show sensor activation -- 2
95 days
96 - [ ] proof of concept C sense manipulation -- 2 days
97 - [ ] proof of concept GPU sense manipulation -- week
98 - [ ] fourier view of sound -- 2 or 3 days
99 - [ ] dancing music listener -- 1 day, depends on fourier
101 ** don't have to get done before winston
102 - [ ] write tests for integration -- 3 days
103 - [ ] usertime/gametime clock HUD display -- day
104 - [ ] find papers for each of the senses justifying my own
105 representation -- week
106 - [ ] show sensor maps in HUD display? -- 4 days
107 - [ ] show sensor maps in AWT display? -- 2 days
110 * Intro
111 So far, I've made the following senses --
112 - Vision
113 - Hearing
114 - Touch
115 - Proprioception
117 And one effector:
118 - Movement
120 However, the code so far has only enabled these senses, but has not
121 actually implemented them. For example, there is still a lot of work
122 to be done for vision. I need to be able to create an /eyeball/ in
123 simulation that can be moved around and see the world from different
124 angles. I also need to determine weather to use log-polar or cartesian
125 for the visual input, and I need to determine how/wether to
126 disceritise the visual input.
128 I also want to be able to visualize both the sensors and the
129 effectors in pretty pictures. This semi-retarted creature will be my
130 first attempt at bringing everything together.
132 * The creature's body
134 Still going to do an eve-like body in blender, but due to problems
135 importing the joints, etc into jMonkeyEngine3, I'm going to do all
136 the connecting here in clojure code, using the names of the individual
137 components and trial and error. Later, I'll maybe make some sort of
138 creature-building modifications to blender that support whatever
139 discreitized senses I'm going to make.
141 #+name: body-1
142 #+begin_src clojure
143 (ns cortex.silly
144 "let's play!"
145 {:author "Robert McIntyre"})
147 ;; TODO remove this!
148 (require 'cortex.import)
149 (cortex.import/mega-import-jme3)
150 (use '(cortex world util body hearing touch vision sense))
152 (rlm.rlm-commands/help)
153 (import java.awt.image.BufferedImage)
154 (import javax.swing.JPanel)
155 (import javax.swing.SwingUtilities)
156 (import java.awt.Dimension)
157 (import javax.swing.JFrame)
158 (import java.awt.Dimension)
159 (import com.aurellem.capture.RatchetTimer)
160 (declare joint-create)
161 (use 'clojure.contrib.def)
164 (defn load-bullet []
165 (let [sim (world (Node.) {} no-op no-op)]
166 (doto sim
167 (.enqueue
168 (fn []
169 (.stop sim)))
170 (.start))))
172 (defn load-blender-model
173 "Load a .blend file using an asset folder relative path."
174 [^String model]
175 (.loadModel
176 (doto (asset-manager)
177 (.registerLoader BlenderModelLoader (into-array String ["blend"])))
178 model))
180 (defn blender-to-jme
181 "Convert from Blender coordinates to JME coordinates"
182 [#^Vector3f in]
183 (Vector3f. (.getX in)
184 (.getZ in)
185 (- (.getY in))))
191 (defn world-to-local
192 "Convert the world coordinates into coordinates relative to the
193 object (i.e. local coordinates), taking into account the rotation
194 of object."
195 [#^Spatial object world-coordinate]
196 (let [out (Vector3f.)]
197 (.worldToLocal object world-coordinate out) out))
199 (defn local-to-world
200 "Convert the local coordinates into coordinates into world relative
201 coordinates"
202 [#^Spatial object local-coordinate]
203 (let [world-coordinate (Vector3f.)]
204 (.localToWorld object local-coordinate world-coordinate)
205 world-coordinate))
207 (defmulti joint-dispatch
208 "Translate blender pseudo-joints into real JME joints."
209 (fn [constraints & _]
210 (:type constraints)))
212 (defmethod joint-dispatch :point
213 [constraints control-a control-b pivot-a pivot-b rotation]
214 (println-repl "creating POINT2POINT joint")
215 ;; bullet's point2point joints are BROKEN, so we must use the
216 ;; generic 6DOF joint instead of an actual Point2Point joint!
218 ;; should be able to do this:
219 (comment
220 (Point2PointJoint.
221 control-a
222 control-b
223 pivot-a
224 pivot-b))
226 ;; but instead we must do this:
227 (println-repl "substuting 6DOF joint for POINT2POINT joint!")
228 (doto
229 (SixDofJoint.
230 control-a
231 control-b
232 pivot-a
233 pivot-b
234 false)
235 (.setLinearLowerLimit Vector3f/ZERO)
236 (.setLinearUpperLimit Vector3f/ZERO)
237 ;;(.setAngularLowerLimit (Vector3f. 1 1 1))
238 ;;(.setAngularUpperLimit (Vector3f. 0 0 0))
240 ))
243 (defmethod joint-dispatch :hinge
244 [constraints control-a control-b pivot-a pivot-b rotation]
245 (println-repl "creating HINGE joint")
246 (let [axis
247 (if-let
248 [axis (:axis constraints)]
249 axis
250 Vector3f/UNIT_X)
251 [limit-1 limit-2] (:limit constraints)
252 hinge-axis
253 (.mult
254 rotation
255 (blender-to-jme axis))]
256 (doto
257 (HingeJoint.
258 control-a
259 control-b
260 pivot-a
261 pivot-b
262 hinge-axis
263 hinge-axis)
264 (.setLimit limit-1 limit-2))))
266 (defmethod joint-dispatch :cone
267 [constraints control-a control-b pivot-a pivot-b rotation]
268 (let [limit-xz (:limit-xz constraints)
269 limit-xy (:limit-xy constraints)
270 twist (:twist constraints)]
272 (println-repl "creating CONE joint")
273 (println-repl rotation)
274 (println-repl
275 "UNIT_X --> " (.mult rotation (Vector3f. 1 0 0)))
276 (println-repl
277 "UNIT_Y --> " (.mult rotation (Vector3f. 0 1 0)))
278 (println-repl
279 "UNIT_Z --> " (.mult rotation (Vector3f. 0 0 1)))
280 (doto
281 (ConeJoint.
282 control-a
283 control-b
284 pivot-a
285 pivot-b
286 rotation
287 rotation)
288 (.setLimit (float limit-xz)
289 (float limit-xy)
290 (float twist)))))
292 (defn connect
293 "here are some examples:
294 {:type :point}
295 {:type :hinge :limit [0 (/ Math/PI 2)] :axis (Vector3f. 0 1 0)}
296 (:axis defaults to (Vector3f. 1 0 0) if not provided for hinge joints)
298 {:type :cone :limit-xz 0]
299 :limit-xy 0]
300 :twist 0]} (use XZY rotation mode in blender!)"
301 [#^Node obj-a #^Node obj-b #^Node joint]
302 (let [control-a (.getControl obj-a RigidBodyControl)
303 control-b (.getControl obj-b RigidBodyControl)
304 joint-center (.getWorldTranslation joint)
305 joint-rotation (.toRotationMatrix (.getWorldRotation joint))
306 pivot-a (world-to-local obj-a joint-center)
307 pivot-b (world-to-local obj-b joint-center)]
309 (if-let [constraints
310 (map-vals
311 eval
312 (read-string
313 (meta-data joint "joint")))]
314 ;; A side-effect of creating a joint registers
315 ;; it with both physics objects which in turn
316 ;; will register the joint with the physics system
317 ;; when the simulation is started.
318 (do
319 (println-repl "creating joint between"
320 (.getName obj-a) "and" (.getName obj-b))
321 (joint-dispatch constraints
322 control-a control-b
323 pivot-a pivot-b
324 joint-rotation))
325 (println-repl "could not find joint meta-data!"))))
330 (defn assemble-creature [#^Node pieces joints]
331 (dorun
332 (map
333 (fn [geom]
334 (let [physics-control
335 (RigidBodyControl.
336 (HullCollisionShape.
337 (.getMesh geom))
338 (if-let [mass (meta-data geom "mass")]
339 (do
340 (println-repl
341 "setting" (.getName geom) "mass to" (float mass))
342 (float mass))
343 (float 1)))]
345 (.addControl geom physics-control)))
346 (filter #(isa? (class %) Geometry )
347 (node-seq pieces))))
348 (dorun
349 (map
350 (fn [joint]
351 (let [[obj-a obj-b] (joint-targets pieces joint)]
352 (connect obj-a obj-b joint)))
353 joints))
354 pieces)
356 (declare blender-creature)
358 (def hand "Models/creature1/one.blend")
360 (def worm "Models/creature1/try-again.blend")
362 (def touch "Models/creature1/touch.blend")
364 (defn worm-model [] (load-blender-model worm))
366 (defn x-ray [#^ColorRGBA color]
367 (doto (Material. (asset-manager)
368 "Common/MatDefs/Misc/Unshaded.j3md")
369 (.setColor "Color" color)
370 (-> (.getAdditionalRenderState)
371 (.setDepthTest false))))
373 (defn colorful []
374 (.getChild (worm-model) "worm-21"))
376 (import jme3tools.converters.ImageToAwt)
378 (import ij.ImagePlus)
380 ;; Every Mesh has many triangles, each with its own index.
381 ;; Every vertex has its own index as well.
383 (defn tactile-sensor-image
384 "Return the touch-sensor distribution image in BufferedImage format,
385 or nil if it does not exist."
386 [#^Geometry obj]
387 (if-let [image-path (meta-data obj "touch")]
388 (ImageToAwt/convert
389 (.getImage
390 (.loadTexture
391 (asset-manager)
392 image-path))
393 false false 0)))
397 (defn triangle
398 "Get the triangle specified by triangle-index from the mesh within
399 bounds."
400 [#^Mesh mesh triangle-index]
401 (let [scratch (Triangle.)]
402 (.getTriangle mesh triangle-index scratch)
403 scratch))
405 (defn triangle-vertex-indices
406 "Get the triangle vertex indices of a given triangle from a given
407 mesh."
408 [#^Mesh mesh triangle-index]
409 (let [indices (int-array 3)]
410 (.getTriangle mesh triangle-index indices)
411 (vec indices)))
413 (defn vertex-UV-coord
414 "Get the uv-coordinates of the vertex named by vertex-index"
415 [#^Mesh mesh vertex-index]
416 (let [UV-buffer
417 (.getData
418 (.getBuffer
419 mesh
420 VertexBuffer$Type/TexCoord))]
421 [(.get UV-buffer (* vertex-index 2))
422 (.get UV-buffer (+ 1 (* vertex-index 2)))]))
424 (defn triangle-UV-coord
425 "Get the uv-cooridnates of the triangle's verticies."
426 [#^Mesh mesh width height triangle-index]
427 (map (fn [[u v]] (vector (* width u) (* height v)))
428 (map (partial vertex-UV-coord mesh)
429 (triangle-vertex-indices mesh triangle-index))))
431 (defn same-side?
432 "Given the points p1 and p2 and the reference point ref, is point p
433 on the same side of the line that goes through p1 and p2 as ref is?"
434 [p1 p2 ref p]
435 (<=
436 0
437 (.dot
438 (.cross (.subtract p2 p1) (.subtract p p1))
439 (.cross (.subtract p2 p1) (.subtract ref p1)))))
441 (defn triangle-seq [#^Triangle tri]
442 [(.get1 tri) (.get2 tri) (.get3 tri)])
444 (defn vector3f-seq [#^Vector3f v]
445 [(.getX v) (.getY v) (.getZ v)])
447 (defn inside-triangle?
448 "Is the point inside the triangle?"
449 {:author "Dylan Holmes"}
450 [#^Triangle tri #^Vector3f p]
451 (let [[vert-1 vert-2 vert-3] (triangle-seq tri)]
452 (and
453 (same-side? vert-1 vert-2 vert-3 p)
454 (same-side? vert-2 vert-3 vert-1 p)
455 (same-side? vert-3 vert-1 vert-2 p))))
457 (defn triangle->matrix4f
458 "Converts the triangle into a 4x4 matrix: The first three columns
459 contain the vertices of the triangle; the last contains the unit
460 normal of the triangle. The bottom row is filled with 1s."
461 [#^Triangle t]
462 (let [mat (Matrix4f.)
463 [vert-1 vert-2 vert-3]
464 ((comp vec map) #(.get t %) (range 3))
465 unit-normal (do (.calculateNormal t)(.getNormal t))
466 vertices [vert-1 vert-2 vert-3 unit-normal]]
467 (dorun
468 (for [row (range 4) col (range 3)]
469 (do
470 (.set mat col row (.get (vertices row)col))
471 (.set mat 3 row 1))))
472 mat))
474 (defn triangle-transformation
475 "Returns the affine transformation that converts each vertex in the
476 first triangle into the corresponding vertex in the second
477 triangle."
478 [#^Triangle tri-1 #^Triangle tri-2]
479 (.mult
480 (triangle->matrix4f tri-2)
481 (.invert (triangle->matrix4f tri-1))))
483 (defn point->vector2f [[u v]]
484 (Vector2f. u v))
486 (defn vector2f->vector3f [v]
487 (Vector3f. (.getX v) (.getY v) 0))
489 (defn map-triangle [f #^Triangle tri]
490 (Triangle.
491 (f 0 (.get1 tri))
492 (f 1 (.get2 tri))
493 (f 2 (.get3 tri))))
495 (defn points->triangle
496 "Convert a list of points into a triangle."
497 [points]
498 (apply #(Triangle. %1 %2 %3)
499 (map (fn [point]
500 (let [point (vec point)]
501 (Vector3f. (get point 0 0)
502 (get point 1 0)
503 (get point 2 0))))
504 (take 3 points))))
506 (defn convex-bounds
507 ;;dylan
508 "Returns the smallest square containing the given
509 vertices, as a vector of integers [left top width height]."
510 ;; "Dimensions of the smallest integer bounding square of the list of
511 ;; 2D verticies in the form: [x y width height]."
512 [uv-verts]
513 (let [xs (map first uv-verts)
514 ys (map second uv-verts)
515 x0 (Math/floor (apply min xs))
516 y0 (Math/floor (apply min ys))
517 x1 (Math/ceil (apply max xs))
518 y1 (Math/ceil (apply max ys))]
519 [x0 y0 (- x1 x0) (- y1 y0)]))
521 (defn sensors-in-triangle
522 ;;dylan
523 "Locate the touch sensors in the triangle, returning a map of their UV and geometry-relative coordinates."
524 ;;"Find the locations of the touch sensors within a triangle in both
525 ;; UV and gemoetry relative coordinates."
526 [image mesh tri-index]
527 (let [width (.getWidth image)
528 height (.getHeight image)
529 UV-vertex-coords (triangle-UV-coord mesh width height tri-index)
530 bounds (convex-bounds UV-vertex-coords)
532 cutout-triangle (points->triangle UV-vertex-coords)
533 UV-sensor-coords
534 (filter (comp (partial inside-triangle? cutout-triangle)
535 (fn [[u v]] (Vector3f. u v 0)))
536 (white-coordinates image bounds))
537 UV->geometry (triangle-transformation
538 cutout-triangle
539 (triangle mesh tri-index))
540 geometry-sensor-coords
541 (map (fn [[u v]] (.mult UV->geometry (Vector3f. u v 0)))
542 UV-sensor-coords)]
543 {:UV UV-sensor-coords :geometry geometry-sensor-coords}))
545 (defn-memo locate-feelers
546 "Search the geometry's tactile UV image for touch sensors, returning
547 their positions in geometry-relative coordinates."
548 [#^Geometry geo]
549 (let [mesh (.getMesh geo)
550 num-triangles (.getTriangleCount mesh)]
551 (if-let [image (tactile-sensor-image geo)]
552 (map
553 (partial sensors-in-triangle image mesh)
554 (range num-triangles))
555 (repeat (.getTriangleCount mesh) {:UV nil :geometry nil}))))
557 (use 'clojure.contrib.def)
559 (defn-memo touch-topology [#^Gemoetry geo]
560 (vec (collapse (reduce concat (map :UV (locate-feelers geo))))))
562 (defn-memo feeler-coordinates [#^Geometry geo]
563 (vec (map :geometry (locate-feelers geo))))
565 (defn enable-touch [#^Geometry geo]
566 (let [feeler-coords (feeler-coordinates geo)
567 tris (triangles geo)
568 limit 0.1
569 ;;results (CollisionResults.)
570 ]
571 (if (empty? (touch-topology geo))
572 nil
573 (fn [node]
574 (let [sensor-origins
575 (map
576 #(map (partial local-to-world geo) %)
577 feeler-coords)
578 triangle-normals
579 (map (partial get-ray-direction geo)
580 tris)
581 rays
582 (flatten
583 (map (fn [origins norm]
584 (map #(doto (Ray. % norm)
585 (.setLimit limit)) origins))
586 sensor-origins triangle-normals))]
587 (vector
588 (touch-topology geo)
589 (vec
590 (for [ray rays]
591 (do
592 (let [results (CollisionResults.)]
593 (.collideWith node ray results)
594 (let [touch-objects
595 (filter #(not (= geo (.getGeometry %)))
596 results)]
597 (- 255
598 (if (empty? touch-objects) 255
599 (rem
600 (int
601 (* 255 (/ (.getDistance
602 (first touch-objects)) limit)))
603 256))))))))))))))
606 (defn touch [#^Node pieces]
607 (filter (comp not nil?)
608 (map enable-touch
609 (filter #(isa? (class %) Geometry)
610 (node-seq pieces)))))
613 (defn test-eye []
614 (.getChild
615 (.getChild (worm-model) "eyes")
616 "eye"))
620 ;; Ears work the same way as vision.
622 ;; (hearing creature) will return [init-functions
623 ;; sensor-functions]. The init functions each take the world and
624 ;; register a SoundProcessor that does foureier transforms on the
625 ;; incommong sound data, making it available to each sensor function.
627 (defn creature-ears
628 "Return the children of the creature's \"ears\" node."
629 ;;dylan
630 ;;"The ear nodes which are children of the \"ears\" node in the
631 ;;creature."
632 [#^Node creature]
633 (if-let [ear-node (.getChild creature "ears")]
634 (seq (.getChildren ear-node))
635 (do (println-repl "could not find ears node") [])))
638 ;;dylan (defn follow-sense, adjoin-sense, attach-stimuli,
639 ;;anchor-qualia, augment-organ, with-organ
642 (defn update-listener-velocity
643 "Update the listener's velocity every update loop."
644 [#^Spatial obj #^Listener lis]
645 (let [old-position (atom (.getLocation lis))]
646 (.addControl
647 obj
648 (proxy [AbstractControl] []
649 (controlUpdate [tpf]
650 (let [new-position (.getLocation lis)]
651 (.setVelocity
652 lis
653 (.mult (.subtract new-position @old-position)
654 (float (/ tpf))))
655 (reset! old-position new-position)))
656 (controlRender [_ _])))))
658 (import com.aurellem.capture.audio.AudioSendRenderer)
660 (defn attach-ear
661 [#^Application world #^Node creature #^Spatial ear continuation]
662 (let [target (closest-node creature ear)
663 lis (Listener.)
664 audio-renderer (.getAudioRenderer world)
665 sp (sound-processor continuation)]
666 (.setLocation lis (.getWorldTranslation ear))
667 (.setRotation lis (.getWorldRotation ear))
668 (bind-sense target lis)
669 (update-listener-velocity target lis)
670 (.addListener audio-renderer lis)
671 (.registerSoundProcessor audio-renderer lis sp)))
673 (defn enable-hearing
674 [#^Node creature #^Spatial ear]
675 (let [hearing-data (atom [])]
676 [(fn [world]
677 (attach-ear world creature ear
678 (fn [data]
679 (reset! hearing-data (vec data)))))
680 [(fn []
681 (let [data @hearing-data
682 topology
683 (vec (map #(vector % 0) (range 0 (count data))))
684 scaled-data
685 (vec
686 (map
687 #(rem (int (* 255 (/ (+ 1 %) 2))) 256)
688 data))]
689 [topology scaled-data]))
690 ]]))
692 (defn hearing
693 [#^Node creature]
694 (reduce
695 (fn [[init-a senses-a]
696 [init-b senses-b]]
697 [(conj init-a init-b)
698 (into senses-a senses-b)])
699 [[][]]
700 (for [ear (creature-ears creature)]
701 (enable-hearing creature ear))))
708 ;; lower level --- nodes
709 ;; closest-node "parse/compile-x" -> makes organ, which is spatial, fn pair
711 ;; higher level -- organs
712 ;;
714 ;; higher level --- sense/effector
715 ;; these are the functions that provide world i/o, chinese-room style
720 (defn blender-creature
721 "Return a creature with all joints in place."
722 [blender-path]
723 (let [model (load-blender-model blender-path)
724 joints (creature-joints model)]
725 (assemble-creature model joints)))
727 (defn gray-scale [num]
728 (+ num
729 (bit-shift-left num 8)
730 (bit-shift-left num 16)))
732 (defn debug-touch-window
733 "creates function that offers a debug view of sensor data"
734 []
735 (let [vi (view-image)]
736 (fn
737 [[coords sensor-data]]
738 (let [image (points->image coords)]
739 (dorun
740 (for [i (range (count coords))]
741 (.setRGB image ((coords i) 0) ((coords i) 1)
742 (gray-scale (sensor-data i)))))
745 (vi image)))))
747 (defn debug-vision-window
748 "creates function that offers a debug view of sensor data"
749 []
750 (let [vi (view-image)]
751 (fn
752 [[coords sensor-data]]
753 (let [image (points->image coords)]
754 (dorun
755 (for [i (range (count coords))]
756 (.setRGB image ((coords i) 0) ((coords i) 1)
757 (sensor-data i))))
758 (vi image)))))
760 (defn debug-hearing-window
761 "view audio data"
762 [height]
763 (let [vi (view-image)]
764 (fn [[coords sensor-data]]
765 (let [image (BufferedImage. (count coords) height
766 BufferedImage/TYPE_INT_RGB)]
767 (dorun
768 (for [x (range (count coords))]
769 (dorun
770 (for [y (range height)]
771 (let [raw-sensor (sensor-data x)]
772 (.setRGB image x y (gray-scale raw-sensor)))))))
774 (vi image)))))
778 ;;(defn test-touch [world creature]
783 ;; here's how motor-control/ proprioception will work: Each muscle is
784 ;; defined by a 1-D array of numbers (the "motor pool") each of which
785 ;; represent muscle fibers. A muscle also has a scalar :strength
786 ;; factor which determines how strong the muscle as a whole is.
787 ;; The effector function for a muscle takes a number < (count
788 ;; motor-pool) and that number is said to "activate" all the muscle
789 ;; fibers whose index is lower than the number. Each fiber will apply
790 ;; force in proportion to its value in the array. Lower values cause
791 ;; less force. The lower values can be put at the "beginning" of the
792 ;; 1-D array to simulate the layout of actual human muscles, which are
793 ;; capable of more percise movements when exerting less force.
795 ;; I don't know how to encode proprioception, so for now, just return
796 ;; a function for each joint that returns a triplet of floats which
797 ;; represent relative roll, pitch, and yaw. Write display code for
798 ;; this though.
800 (defn muscle-fiber-values
801 "get motor pool strengths"
802 [#^BufferedImage image]
803 (vec
804 (let [width (.getWidth image)]
805 (for [x (range width)]
806 (- 255
807 (bit-and
808 0x0000FF
809 (.getRGB image x 0)))))))
812 (defn creature-muscles
813 "Return the children of the creature's \"muscles\" node."
814 [#^Node creature]
815 (if-let [muscle-node (.getChild creature "muscles")]
816 (seq (.getChildren muscle-node))
817 (do (println-repl "could not find muscles node") [])))
819 (defn single-muscle [#^Node parts #^Node muscle]
820 (let [target (closest-node parts muscle)
821 axis
822 (.mult (.getWorldRotation muscle) Vector3f/UNIT_Y)
823 strength (meta-data muscle "strength")
824 image-name (read-string (meta-data muscle "muscle"))
825 image
826 (ImageToAwt/convert
827 (.getImage (.loadTexture (asset-manager) image-name))
828 false false 0)
829 fibers (muscle-fiber-values image)
830 fiber-integral (reductions + fibers)
831 force-index (vec
832 (map
833 #(float (* strength (/ % (last
834 fiber-integral))))
835 fiber-integral))
836 control (.getControl target RigidBodyControl)]
837 (fn [n]
838 (let [pool-index (min n (count fibers))]
839 (.applyTorque control (.mult axis (force-index n)))))))
842 (defn enable-muscles
843 "Must be called on a creature after RigidBodyControls have been
844 created."
845 [#^Node creature]
846 (let [muscles (creature-muscles creature)]
847 (for [muscle muscles]
848 (single-muscle creature muscle))))
850 (defn test-creature [thing]
851 (let [x-axis
852 (box 1 0.01 0.01 :physical? false :color ColorRGBA/Red)
853 y-axis
854 (box 0.01 1 0.01 :physical? false :color ColorRGBA/Green)
855 z-axis
856 (box 0.01 0.01 1 :physical? false :color ColorRGBA/Blue)
857 creature (blender-creature thing)
858 touch-nerves (touch creature)
859 touch-debug-windows (map (fn [_] (debug-touch-window)) touch-nerves)
860 [init-vision-fns vision-data] (vision creature)
861 vision-debug (map (fn [_] (debug-vision-window)) vision-data)
862 me (sphere 0.5 :color ColorRGBA/Blue :physical? false)
863 [init-hearing-fns hearing-senses] (hearing creature)
864 hearing-windows (map (fn [_] (debug-hearing-window 50))
865 hearing-senses)
866 bell (AudioNode. (asset-manager)
867 "Sounds/pure.wav" false)
868 prop (proprioception creature)
869 prop-debug (proprioception-debug-window)
871 muscle-fns (enable-muscles creature)
872 ;; dream
874 ]
877 (apply
878 world
879 (with-movement
880 (.getChild creature "worm-21")
881 ["key-r" "key-t"
882 "key-f" "key-g"
883 "key-v" "key-b"]
884 [10 10 10 10 1 1]
885 [(nodify [creature
886 (box 10 2 10 :position (Vector3f. 0 -9 0)
887 :color ColorRGBA/Gray :mass 0)
888 x-axis y-axis z-axis
889 me
890 ])
891 (merge standard-debug-controls
892 {"key-return"
893 (fn [_ value]
894 (if value
895 (do
896 (println-repl "play-sound")
897 (.play bell))))
898 "key-h"
899 (fn [_ value]
900 (if value
901 (do
902 (println-repl "muscle activating!")
903 ((first muscle-fns) 199))))
905 })
906 (fn [world]
907 (light-up-everything world)
908 (enable-debug world)
909 (dorun (map #(% world) init-vision-fns))
910 (dorun (map #(% world) init-hearing-fns))
912 (add-eye world
913 (attach-eye creature (test-eye))
914 (comp (view-image) BufferedImage!))
916 (add-eye world (.getCamera world) no-op)
917 ;;(set-gravity world (Vector3f. 0 0 0))
918 ;;(com.aurellem.capture.Capture/captureVideo
919 ;; world (file-str "/home/r/proj/ai-videos/hand"))
920 ;;(.setTimer world (RatchetTimer. 60))
921 (speed-up world)
922 (set-gravity world (Vector3f. 0 0 0))
923 )
924 (fn [world tpf]
925 ;;(dorun
926 ;; (map #(%1 %2) touch-nerves (repeat (.getRootNode world))))
928 (prop-debug (prop))
930 (dorun
931 (map #(%1 (%2 (.getRootNode world)))
932 touch-debug-windows touch-nerves))
934 (dorun
935 (map #(%1 (%2))
936 vision-debug vision-data))
937 (dorun
938 (map #(%1 (%2)) hearing-windows hearing-senses))
941 ;;(println-repl (vision-data))
942 (.setLocalTranslation me (.getLocation (.getCamera world)))
945 )]
946 ;;(let [timer (atom 0)]
947 ;; (fn [_ _]
948 ;; (swap! timer inc)
949 ;; (if (= (rem @timer 60) 0)
950 ;; (println-repl (float (/ @timer 60))))))
951 ))))
961 ;;; experiments in collisions
965 (defn collision-test []
966 (let [b-radius 1
967 b-position (Vector3f. 0 0 0)
968 obj-b (box 1 1 1 :color ColorRGBA/Blue
969 :position b-position
970 :mass 0)
971 node (nodify [obj-b])
972 bounds-b
973 (doto (Picture.)
974 (.setHeight 50)
975 (.setWidth 50)
976 (.setImage (asset-manager)
977 "Models/creature1/hand.png"
978 false
979 ))
981 ;;(Ray. (Vector3f. 0 -5 0) (.normalize (Vector3f. 0 1 0)))
983 collisions
984 (let [cr (CollisionResults.)]
985 (.collideWith node bounds-b cr)
986 (println (map #(.getContactPoint %) cr))
987 cr)
989 ;;collision-points
990 ;;(map #(sphere 0.1 :position (.getContactPoint %))
991 ;; collisions)
993 ;;node (nodify (conj collision-points obj-b))
995 sim
996 (world node
997 {"key-space"
998 (fn [_ value]
999 (if value
1000 (let [cr (CollisionResults.)]
1001 (.collideWith node bounds-b cr)
1002 (println-repl (map #(.getContactPoint %) cr))
1003 cr)))}
1004 no-op
1005 no-op)
1008 sim
1010 ))
1013 ;; the camera will stay in its initial position/rotation with relation
1014 ;; to the spatial.
1017 (defn follow-test
1018 "show a camera that stays in the same relative position to a blue cube."
1019 []
1020 (let [camera-pos (Vector3f. 0 30 0)
1021 rock (box 1 1 1 :color ColorRGBA/Blue
1022 :position (Vector3f. 0 10 0)
1023 :mass 30
1025 rot (.getWorldRotation rock)
1027 table (box 3 1 10 :color ColorRGBA/Gray :mass 0
1028 :position (Vector3f. 0 -3 0))]
1030 (world
1031 (nodify [rock table])
1032 standard-debug-controls
1033 (fn [world]
1034 (let
1035 [cam (doto (.clone (.getCamera world))
1036 (.setLocation camera-pos)
1037 (.lookAt Vector3f/ZERO
1038 Vector3f/UNIT_X))]
1039 (bind-sense rock cam)
1041 (.setTimer world (RatchetTimer. 60))
1042 (add-eye world cam (comp (view-image) BufferedImage!))
1043 (add-eye world (.getCamera world) no-op))
1045 (fn [_ _] (println-repl rot)))))
1049 #+end_src
1051 #+results: body-1
1052 : #'cortex.silly/follow-test
1055 * COMMENT purgatory
1056 #+begin_src clojure
1057 (defn bullet-trans []
1058 (let [obj-a (sphere 0.5 :color ColorRGBA/Red
1059 :position (Vector3f. -10 5 0))
1060 obj-b (sphere 0.5 :color ColorRGBA/Blue
1061 :position (Vector3f. -10 -5 0)
1062 :mass 0)
1063 control-a (.getControl obj-a RigidBodyControl)
1064 control-b (.getControl obj-b RigidBodyControl)
1065 swivel
1066 (.toRotationMatrix
1067 (doto (Quaternion.)
1068 (.fromAngleAxis (/ Math/PI 2)
1069 Vector3f/UNIT_X)))]
1070 (doto
1071 (ConeJoint.
1072 control-a control-b
1073 (Vector3f. 0 5 0)
1074 (Vector3f. 0 -5 0)
1075 swivel swivel)
1076 (.setLimit (* 0.6 (/ Math/PI 4))
1077 (/ Math/PI 4)
1078 (* Math/PI 0.8)))
1079 (world (nodify
1080 [obj-a obj-b])
1081 standard-debug-controls
1082 enable-debug
1083 no-op)))
1086 (defn bullet-trans* []
1087 (let [obj-a (box 1.5 0.5 0.5 :color ColorRGBA/Red
1088 :position (Vector3f. 5 0 0)
1089 :mass 90)
1090 obj-b (sphere 0.5 :color ColorRGBA/Blue
1091 :position (Vector3f. -5 0 0)
1092 :mass 0)
1093 control-a (.getControl obj-a RigidBodyControl)
1094 control-b (.getControl obj-b RigidBodyControl)
1095 move-up? (atom nil)
1096 move-down? (atom nil)
1097 move-left? (atom nil)
1098 move-right? (atom nil)
1099 roll-left? (atom nil)
1100 roll-right? (atom nil)
1101 force 100
1102 swivel
1103 (.toRotationMatrix
1104 (doto (Quaternion.)
1105 (.fromAngleAxis (/ Math/PI 2)
1106 Vector3f/UNIT_X)))
1107 x-move
1108 (doto (Matrix3f.)
1109 (.fromStartEndVectors Vector3f/UNIT_X
1110 (.normalize (Vector3f. 1 1 0))))
1112 timer (atom 0)]
1113 (doto
1114 (ConeJoint.
1115 control-a control-b
1116 (Vector3f. -8 0 0)
1117 (Vector3f. 2 0 0)
1118 ;;swivel swivel
1119 ;;Matrix3f/IDENTITY Matrix3f/IDENTITY
1120 x-move Matrix3f/IDENTITY
1122 (.setCollisionBetweenLinkedBodys false)
1123 (.setLimit (* 1 (/ Math/PI 4)) ;; twist
1124 (* 1 (/ Math/PI 4)) ;; swing span in X-Y plane
1125 (* 0 (/ Math/PI 4)))) ;; swing span in Y-Z plane
1126 (world (nodify
1127 [obj-a obj-b])
1128 (merge standard-debug-controls
1129 {"key-r" (fn [_ pressed?] (reset! move-up? pressed?))
1130 "key-t" (fn [_ pressed?] (reset! move-down? pressed?))
1131 "key-f" (fn [_ pressed?] (reset! move-left? pressed?))
1132 "key-g" (fn [_ pressed?] (reset! move-right? pressed?))
1133 "key-v" (fn [_ pressed?] (reset! roll-left? pressed?))
1134 "key-b" (fn [_ pressed?] (reset! roll-right? pressed?))})
1136 (fn [world]
1137 (enable-debug world)
1138 (set-gravity world Vector3f/ZERO)
1141 (fn [world _]
1143 (if @move-up?
1144 (.applyForce control-a
1145 (Vector3f. force 0 0)
1146 (Vector3f. 0 0 0)))
1147 (if @move-down?
1148 (.applyForce control-a
1149 (Vector3f. (- force) 0 0)
1150 (Vector3f. 0 0 0)))
1151 (if @move-left?
1152 (.applyForce control-a
1153 (Vector3f. 0 force 0)
1154 (Vector3f. 0 0 0)))
1155 (if @move-right?
1156 (.applyForce control-a
1157 (Vector3f. 0 (- force) 0)
1158 (Vector3f. 0 0 0)))
1160 (if @roll-left?
1161 (.applyForce control-a
1162 (Vector3f. 0 0 force)
1163 (Vector3f. 0 0 0)))
1164 (if @roll-right?
1165 (.applyForce control-a
1166 (Vector3f. 0 0 (- force))
1167 (Vector3f. 0 0 0)))
1169 (if (zero? (rem (swap! timer inc) 100))
1170 (.attachChild
1171 (.getRootNode world)
1172 (sphere 0.05 :color ColorRGBA/Yellow
1173 :physical? false :position
1174 (.getWorldTranslation obj-a)))))
1176 ))
1178 (defn transform-trianglesdsd
1179 "Transform that converts each vertex in the first triangle
1180 into the corresponding vertex in the second triangle."
1181 [#^Triangle tri-1 #^Triangle tri-2]
1182 (let [in [(.get1 tri-1)
1183 (.get2 tri-1)
1184 (.get3 tri-1)]
1185 out [(.get1 tri-2)
1186 (.get2 tri-2)
1187 (.get3 tri-2)]]
1188 (let [translate (doto (Matrix4f.) (.setTranslation (.negate (in 0))))
1189 in* [(.mult translate (in 0))
1190 (.mult translate (in 1))
1191 (.mult translate (in 2))]
1192 final-translation
1193 (doto (Matrix4f.)
1194 (.setTranslation (out 1)))
1196 rotate-1
1197 (doto (Matrix3f.)
1198 (.fromStartEndVectors
1199 (.normalize
1200 (.subtract
1201 (in* 1) (in* 0)))
1202 (.normalize
1203 (.subtract
1204 (out 1) (out 0)))))
1205 in** [(.mult rotate-1 (in* 0))
1206 (.mult rotate-1 (in* 1))
1207 (.mult rotate-1 (in* 2))]
1208 scale-factor-1
1209 (.mult
1210 (.normalize
1211 (.subtract
1212 (out 1)
1213 (out 0)))
1214 (/ (.length
1215 (.subtract (out 1)
1216 (out 0)))
1217 (.length
1218 (.subtract (in** 1)
1219 (in** 0)))))
1220 scale-1 (doto (Matrix4f.) (.setScale scale-factor-1))
1221 in*** [(.mult scale-1 (in** 0))
1222 (.mult scale-1 (in** 1))
1223 (.mult scale-1 (in** 2))]
1231 (dorun (map println in))
1232 (println)
1233 (dorun (map println in*))
1234 (println)
1235 (dorun (map println in**))
1236 (println)
1237 (dorun (map println in***))
1238 (println)
1240 ))))
1243 (defn world-setup [joint]
1244 (let [joint-position (Vector3f. 0 0 0)
1245 joint-rotation
1246 (.toRotationMatrix
1247 (.mult
1248 (doto (Quaternion.)
1249 (.fromAngleAxis
1250 (* 1 (/ Math/PI 4))
1251 (Vector3f. -1 0 0)))
1252 (doto (Quaternion.)
1253 (.fromAngleAxis
1254 (* 1 (/ Math/PI 2))
1255 (Vector3f. 0 0 1)))))
1256 top-position (.mult joint-rotation (Vector3f. 8 0 0))
1258 origin (doto
1259 (sphere 0.1 :physical? false :color ColorRGBA/Cyan
1260 :position top-position))
1261 top (doto
1262 (sphere 0.1 :physical? false :color ColorRGBA/Yellow
1263 :position top-position)
1265 (.addControl
1266 (RigidBodyControl.
1267 (CapsuleCollisionShape. 0.5 1.5 1) (float 20))))
1268 bottom (doto
1269 (sphere 0.1 :physical? false :color ColorRGBA/DarkGray
1270 :position (Vector3f. 0 0 0))
1271 (.addControl
1272 (RigidBodyControl.
1273 (CapsuleCollisionShape. 0.5 1.5 1) (float 0))))
1274 table (box 10 2 10 :position (Vector3f. 0 -20 0)
1275 :color ColorRGBA/Gray :mass 0)
1276 a (.getControl top RigidBodyControl)
1277 b (.getControl bottom RigidBodyControl)]
1279 (cond
1280 (= joint :cone)
1282 (doto (ConeJoint.
1283 a b
1284 (world-to-local top joint-position)
1285 (world-to-local bottom joint-position)
1286 joint-rotation
1287 joint-rotation
1291 (.setLimit (* (/ 10) Math/PI)
1292 (* (/ 4) Math/PI)
1293 0)))
1294 [origin top bottom table]))
1296 (defn test-joint [joint]
1297 (let [[origin top bottom floor] (world-setup joint)
1298 control (.getControl top RigidBodyControl)
1299 move-up? (atom false)
1300 move-down? (atom false)
1301 move-left? (atom false)
1302 move-right? (atom false)
1303 roll-left? (atom false)
1304 roll-right? (atom false)
1305 timer (atom 0)]
1307 (world
1308 (nodify [top bottom floor origin])
1309 (merge standard-debug-controls
1310 {"key-r" (fn [_ pressed?] (reset! move-up? pressed?))
1311 "key-t" (fn [_ pressed?] (reset! move-down? pressed?))
1312 "key-f" (fn [_ pressed?] (reset! move-left? pressed?))
1313 "key-g" (fn [_ pressed?] (reset! move-right? pressed?))
1314 "key-v" (fn [_ pressed?] (reset! roll-left? pressed?))
1315 "key-b" (fn [_ pressed?] (reset! roll-right? pressed?))})
1317 (fn [world]
1318 (light-up-everything world)
1319 (enable-debug world)
1320 (set-gravity world (Vector3f. 0 0 0))
1323 (fn [world _]
1324 (if (zero? (rem (swap! timer inc) 100))
1325 (do
1326 ;; (println-repl @timer)
1327 (.attachChild (.getRootNode world)
1328 (sphere 0.05 :color ColorRGBA/Yellow
1329 :position (.getWorldTranslation top)
1330 :physical? false))
1331 (.attachChild (.getRootNode world)
1332 (sphere 0.05 :color ColorRGBA/LightGray
1333 :position (.getWorldTranslation bottom)
1334 :physical? false))))
1336 (if @move-up?
1337 (.applyTorque control
1338 (.mult (.getPhysicsRotation control)
1339 (Vector3f. 0 0 10))))
1340 (if @move-down?
1341 (.applyTorque control
1342 (.mult (.getPhysicsRotation control)
1343 (Vector3f. 0 0 -10))))
1344 (if @move-left?
1345 (.applyTorque control
1346 (.mult (.getPhysicsRotation control)
1347 (Vector3f. 0 10 0))))
1348 (if @move-right?
1349 (.applyTorque control
1350 (.mult (.getPhysicsRotation control)
1351 (Vector3f. 0 -10 0))))
1352 (if @roll-left?
1353 (.applyTorque control
1354 (.mult (.getPhysicsRotation control)
1355 (Vector3f. -1 0 0))))
1356 (if @roll-right?
1357 (.applyTorque control
1358 (.mult (.getPhysicsRotation control)
1359 (Vector3f. 1 0 0))))))))
1363 (defprotocol Frame
1364 (frame [this]))
1366 (extend-type BufferedImage
1367 Frame
1368 (frame [image]
1369 (merge
1370 (apply
1371 hash-map
1372 (interleave
1373 (doall (for [x (range (.getWidth image)) y (range (.getHeight image))]
1374 (vector x y)))
1375 (doall (for [x (range (.getWidth image)) y (range (.getHeight image))]
1376 (let [data (.getRGB image x y)]
1377 (hash-map :r (bit-shift-right (bit-and 0xff0000 data) 16)
1378 :g (bit-shift-right (bit-and 0x00ff00 data) 8)
1379 :b (bit-and 0x0000ff data)))))))
1380 {:width (.getWidth image) :height (.getHeight image)})))
1383 (extend-type ImagePlus
1384 Frame
1385 (frame [image+]
1386 (frame (.getBufferedImage image+))))
1389 #+end_src
1392 * COMMENT generate source
1393 #+begin_src clojure :tangle ../src/cortex/silly.clj
1394 <<body-1>>
1395 #+end_src