AR/MR Stuff

---

Handtracking and physical Controllers

02/2021 I was experimenting with using hand tracking on the Quest2 and a midi controller. Pressing the physical button aligns the virutal controller to the finger tip that pressed the button.

Multiple 3d cameras

07/2020 This experiment was to visualize when part of the body is occluded and the app knows the parts to fill in. Uses Unity's VFX graph and 3 kinect time-of-flight cameras.

Mixed Reality Demo with Google ARCore

6/2018 This is an experiment with Google ARCore and Unity 2018. My goal was "the floor is made of lava" but in a more unique way. To me, one of the most important things about making mixed reality is occlusion. ARCore detects the ground (actually arcore 1.2 detects vertical now), so I place geometry on the ground that occluded the other parts, such as the lava river and volcano underground. I havent seen this before so I thought it would be a fun experiment.
My other goal was to use Houdini to do some real time VFX. The floor crumbling, river of lava's flow map and the particle fluid lava fountain are made with Houdini. This is mostly done (6/1/2018), there are some things I want to improve like the resolution/appearance of the fluid sims and the underground volcanic lightning and a video example.
(Fixed Link 10/21/2021) Download the APK for Android 7.0+ if your phone supports Google ARCore
chrisrogers3d.graphics/lavagin.apk (70megs)
You also need ARCore from the Google Play store.
Made with Unity 2018.1 and I used Maya, Arnold (for a lightmap), Zbrush, Houdini, Photoshop and Substance Painter.

Augmented/Mixed reality experiment using Mapbox's Unity SDK and Google ARCore.

(3/2018) This experiment uses Android's ARCore, GPS and 3D maps (from Mapbox's Unity SDK) to do "world scale" augmented reality. To fine tune the users position and orientation I allow the user to fine tune the GPS location with buttons, then align known landmarks (in this case Sutro Tower, antennas on Twin Peaks) to fine tune direction. The buildings are rendered as a mask so I can show these characters being occluded by the buildings. I could get good results with the occlusion if I fine tuned the location. This experiment got me interested in computer vision again.

CV Stuff

---

Dance With Vector

When I worked at Anki, I did some computer vision related R&D. In this video, I am using Posenet for installations and Anki's Vector SDK to control the robot. Lyrics by Outkast.

Drawing With Vector

OpenCV and Unity on Android

This is an experiment (1/2017) at animating characters in a mobile app using computer vision as the input. I'm using Unity5.5 and OpenCV on Android for the app. The incidental background music is Talvin Singh and the plastic character that I have attached an ARUCO marker to is Heathrow the Hedgehog by Frank Kozik.

TensorFlow and Object Recognition (4/2018)

People using machine learning with computer vision are starting to use synthetic 3D data sets instead of /in addition to real photographs. I recently created a set of images of billiard balls and followed a tutorial on how to retrain a TensorFlow network to recognize my objects. I didnt get good results, will try again.

using a phone as a 3D input device proof-of-concept (12/2015)

After seeing Tiltbrush and not having any 3D input devices of my own, I started an experiment to see if it was possible to use image tracking on a phone to extract the phone's position in 3D space to be used as an input for an application. This proof-of-concept uses Vuforia's image tracking, Unity5, Unity's new low level network interface (NetworkTransport), and a Nexus6p. It sort of worked, it was noisy and had a limited tracking space. Besides sending the position, there is a UI on the phone with sliders.

Arduino Stuff

---

This project was to determine the input latency for Unity with various game controllers for a Super Smash Brothers type game on PC. There was an Arduino that controlled a relay and had a photoresistor as an input. In Unity, I press a GUI start button on the PC screen and that sends a message to the Arduino program. Although the main goal was to test different controllers' latency, these experiements have the Arduino send a button press signal over USB (simulating the game controller button press instead of the 3d printed solenoid i made to attach to a physical controller) So... the signal is sent over USB to Unity. Unity then flashes the screen white after it was black. The photoresitor, taped to the monitor with packing tape, also hooked up to the Arduino sends a signal when the screen lights up. I measured the difference between a Unity build and the editor, and it was 90ms to 60ms. That's 2 frames at 60 fps. Human response time is at best 200ms. I thought the editor's speed advantage was being multithreaded, but didnt pursue the issue.

VR Stuff

---

Daydream in the neighborhood (11/2015)

This is photographed running on a Nexus6P. With this shader, fog density is influenced by absolute X world position to make the fog denser

Inner Richmond Demo

If you've spent much time in the Inner Richmond in the summer, this should look familiar. Several custom shaders. This is just a sketch of an environment to see how the fog and clouds and sun felt in an VR environment. I thought it was effective.

Be Bots

I started on the idea 10 years ago as an animated show. You are Blue, a kid that is the leader of a robot jazz band. He rides a Schwinn Stingray to the jazz club, Chez Bot.
If you follow the YouTube link, you can watch these videos at 1080p60. For the most part they all run at 90fps when I am not capturing video at 60fps at the same time.

Bass Sequencer in space

This is an experiment with a string bass and a music sequencer, like a drum machine but the third dimension is the note. This is triggering midi notes, I was experimenting with controlling Ableton from a VR interface.

Vive Drums

This is a drum kit for the band. not having physical feedback with drums is weird, and the Vive controllers arent balanced like a drum stick, but its an experiment.

Vive Bass

Early bass demo with a very unrealistic note reproduction (ie: what should be on the fretboard an A is not an A). I liked the different sounds I could assign to the bow (pluck, regular bow and a sustaining bow) interacting with the string.

Drum Sequencer in VR

This video is a drum sequencer in 3D with a "gaze-interaction" interface for devices that only have a direction to look, and a button, like Google Cardboard. Although this is on a PC with a DK2.

Close