Skip to main content

Interactive digital sandbox

This is one of the installations I developed for a digital entertainment theme park. It is an interactive digital sandbox with normal white sand and I used a projector and depth sensor to project topological colors and virtual objects depending on the height and shape of the sand. I also used QR markers on 3D printed objects to interact with the virtual content.

Overview of installation

A sandbox of 2 x 1.5 meters with projected content

As shown in the image, depending on the height of the sand, the projected colors change from deep water, shallow water, beach, green grass, dark green forest, rocky mountains and finally snowy mountains. These are the 7 different height levels defined by the system. In addition to the topological colors, it also has fishes and boats in the waters, lava coming out of a crater and other interactive elements.


The sand depth is detected real time and projection updated

Fish cannot go on land


Hardware setup

  1. Sandbox made of wood. But this can be anything as long as it can contain sand.
  2. Projector. Make sure the projection covers the sandbox entirely and some more. It is better to find a projector with a projection area somewhat larger than the area of the sandbox. This is because due to the height, the projected area will shrink a little.
  3. Depth sensor. There are several readily available such as the Kinect, or Intel's Realsense depth sensors. In this particular setup I used the Kinect. A little word of caution though, Asus Xtion's accuracy and resolution was not good enough to differentiate the height levels.
  4. RGB Imager. Ideally this is integrated into the depth sensor, so the transformation is known. Since I used the Kinect, this is already part of the setup.
  5. AR markers. For this setup I used hard cardboard material cut with laser cutter.
  6. 3D printed objects shaped like magic wands to interact with the content.
  7. Computer. To run the detection and rendering of the contents.
  8. HDMI cable to connect the computer and projector.
  9. Lens polarizer to reduce glare and reflection from projector light. Since the room is dark and the projector emits the strongest light, the reflection from the sand surface "blinds" the RGB imager and causes the AR markers detection to fail. Something like this.
  10. 3D printed attachment to attach the lens polarizer to the Kinect.

Software setup


  1. Unity to develop the contents
  2. OpenCV (Aruco) for AR marker detection and positioning (calibration)
  3. OS. I used Windows 10 in this installation
  4. (Optional) Jenkins, CMake, vcpkg, and other CI and package managers

3D printed objects and AR markers

A 3D printed magnifying glass with AR markers to zoom in on virtual animals.

The AR marker is detected by the RGB camera and after aligning it's position to the virtual content, it determines if there are animals in the vicinity of the magnifying glass. If there are, those animals will be zoomed in.

Another use of the marker is to implement a kind of treasure hunt where there are hidden treasures or characters that players can find, such as in the video below.


Or we can use several markers and realize different effects and interactions.




Lens polarizer

As mentioned, due to the room being dark and the projector the strongest light source around, the reflected light from the sand surface is enough to blind the RGB camera such that the image is just one big white image. Unfortunately, the Kinect does not have a mechanism to adjust the aperture with code so I had to attach an adjustable lens polarizer to adjust the amount of light coming through to the imager.

A lens polarizer adjusts the amount of light

The lens polarizer attached to the Kinect

It's a little hard to see but the lens polarizer is attached to the Kinect using a 3D printed attachment. The attachment was designed on Autodesk Fusion 360 with careful measurements of the Kinect's outer casing. Adjusting the dial on the polarizer allowed the imager to capture better images and detect the AR markers accordingly.

Volcano detection

On snowy mountains, if players dig out a small hole, that will become a crater and lava will spew out. The algorithm to detect a crater is simply checking for points that is at the bottom of a concave shaped area.

Crater detection process

For every depth point, check it's surrounding neighbors and if it is the lowest/deepest point, define it as the crater. This simple algorithm proved reliable enough in my testing.

Gallery

Here are a few extra images of some users painstakingly adjusting the sand height to resemble shapes such as fish bone and a house.


Creative players forms a fish bone

A house and a car, and the moon in the top right


This version has since been updated and this was taken on the day of decommisioning

Comments

Post a Comment

Popular posts from this blog

Installing a custom ROM on Android (on the GT-N8013)

It's been a while since my last entry and since it is a new start in 2019, I thought I'd write something about "gone with the old and in with the new". I've had my Samsung Galaxy Note 10.1 (pnotewifi) since 2014, and it's one of the early Galaxy Note tablet series. It has served me well all this years but now it just sits there collecting dust. My old Samsung GT-N8013 I've known a long time about custom Android ROMs like CyanogenMod but has never had the motivation to try them out, until now ! Overview of the process For beginners like me, I didn't have an understanding of the installation process and so it looked complicated and it was one of the reasons I was put off in trying the custom ROM. I just want to say, it's not complicated at all!   Basically you will need to Prepare an SD card and install Android SDK (you need adb ). Install a custom boot loader ( TWRP is the de facto tool at the moment). Use adb to copy custom...
15 comments
Read more

Pitfalls during Training and Object Detection with TensorFlow for Absolute Beginners

This article is based on the great tutorial here on how to train and detect custom objects with Tensorflow. I also referred to the official documentations here and  here  for running Tensorflow model building locally. It was my first custom detection project and I faced some hiccups along the way and this article is to log and share my finding so it can help other beginners like me. In the end, I managed to train a tensorflow model to detect Batsumaru , a character from Sanrio. This is how the detection will look like. The tools Windows 10 Pro 64 Tensorflow originally 1.7.1 and upgraded to 1.12.0. I will share the reason later. Python 3.5.4 LabelImg for image labeling PyCharm IDE Steps and Pitfalls Some of the mistakes I made and other discoveries when following the guide. I will not repeat the steps mentioned in the original guide, but only the parts where I had to deviate from the walkthrough and found out things by myself. The training and...
1 comment
Read more

Unity Best Practices for Beginners

Unity is a fabulous tool for not only games but also interactive entertainment, simulations, etc. And there are many good tutorials on tips and tricks to optimize performance and code readability. I compiled this list as a memo to myself when starting out on Unity, so I can always refer back to the basics and hoping that this can help someone else too. Object pooling. Object pooling is a pretty cool trick and it improves performance because you can reduce the number of Initiate() and Destroy() calls. To illustrate why Destroy can be bad, I attached a screenshot of a project I was building with Unity. I'm not going to go into the details of Object Pooling because there are already many good tutorials out there. One of them being this one.  One thing to always keep in mind is that it is easy to get MissingReferenceException if you accidentally Destroy() the pooled objects. It happened to me once where I attached one script to multiple prefabs which behave the same except that so...
Post a Comment
Read more