Skip to main content

Interactive digital sandbox

This is one of the installations I developed for a digital entertainment theme park. It is an interactive digital sandbox with normal white sand and I used a projector and depth sensor to project topological colors and virtual objects depending on the height and shape of the sand. I also used QR markers on 3D printed objects to interact with the virtual content.

Overview of installation

A sandbox of 2 x 1.5 meters with projected content

As shown in the image, depending on the height of the sand, the projected colors change from deep water, shallow water, beach, green grass, dark green forest, rocky mountains and finally snowy mountains. These are the 7 different height levels defined by the system. In addition to the topological colors, it also has fishes and boats in the waters, lava coming out of a crater and other interactive elements.


The sand depth is detected real time and projection updated

Fish cannot go on land


Hardware setup

  1. Sandbox made of wood. But this can be anything as long as it can contain sand.
  2. Projector. Make sure the projection covers the sandbox entirely and some more. It is better to find a projector with a projection area somewhat larger than the area of the sandbox. This is because due to the height, the projected area will shrink a little.
  3. Depth sensor. There are several readily available such as the Kinect, or Intel's Realsense depth sensors. In this particular setup I used the Kinect. A little word of caution though, Asus Xtion's accuracy and resolution was not good enough to differentiate the height levels.
  4. RGB Imager. Ideally this is integrated into the depth sensor, so the transformation is known. Since I used the Kinect, this is already part of the setup.
  5. AR markers. For this setup I used hard cardboard material cut with laser cutter.
  6. 3D printed objects shaped like magic wands to interact with the content.
  7. Computer. To run the detection and rendering of the contents.
  8. HDMI cable to connect the computer and projector.
  9. Lens polarizer to reduce glare and reflection from projector light. Since the room is dark and the projector emits the strongest light, the reflection from the sand surface "blinds" the RGB imager and causes the AR markers detection to fail. Something like this.
  10. 3D printed attachment to attach the lens polarizer to the Kinect.

Software setup


  1. Unity to develop the contents
  2. OpenCV (Aruco) for AR marker detection and positioning (calibration)
  3. OS. I used Windows 10 in this installation
  4. (Optional) Jenkins, CMake, vcpkg, and other CI and package managers

3D printed objects and AR markers

A 3D printed magnifying glass with AR markers to zoom in on virtual animals.

The AR marker is detected by the RGB camera and after aligning it's position to the virtual content, it determines if there are animals in the vicinity of the magnifying glass. If there are, those animals will be zoomed in.

Another use of the marker is to implement a kind of treasure hunt where there are hidden treasures or characters that players can find, such as in the video below.


Or we can use several markers and realize different effects and interactions.




Lens polarizer

As mentioned, due to the room being dark and the projector the strongest light source around, the reflected light from the sand surface is enough to blind the RGB camera such that the image is just one big white image. Unfortunately, the Kinect does not have a mechanism to adjust the aperture with code so I had to attach an adjustable lens polarizer to adjust the amount of light coming through to the imager.

A lens polarizer adjusts the amount of light

The lens polarizer attached to the Kinect

It's a little hard to see but the lens polarizer is attached to the Kinect using a 3D printed attachment. The attachment was designed on Autodesk Fusion 360 with careful measurements of the Kinect's outer casing. Adjusting the dial on the polarizer allowed the imager to capture better images and detect the AR markers accordingly.

Volcano detection

On snowy mountains, if players dig out a small hole, that will become a crater and lava will spew out. The algorithm to detect a crater is simply checking for points that is at the bottom of a concave shaped area.

Crater detection process

For every depth point, check it's surrounding neighbors and if it is the lowest/deepest point, define it as the crater. This simple algorithm proved reliable enough in my testing.

Gallery

Here are a few extra images of some users painstakingly adjusting the sand height to resemble shapes such as fish bone and a house.


Creative players forms a fish bone

A house and a car, and the moon in the top right


This version has since been updated and this was taken on the day of decommisioning

Comments

Post a Comment

Popular posts from this blog

Installing a custom ROM on Android (on the GT-N8013)

It's been a while since my last entry and since it is a new start in 2019, I thought I'd write something about "gone with the old and in with the new". I've had my Samsung Galaxy Note 10.1 (pnotewifi) since 2014, and it's one of the early Galaxy Note tablet series. It has served me well all this years but now it just sits there collecting dust. My old Samsung GT-N8013 I've known a long time about custom Android ROMs like CyanogenMod but has never had the motivation to try them out, until now ! Overview of the process For beginners like me, I didn't have an understanding of the installation process and so it looked complicated and it was one of the reasons I was put off in trying the custom ROM. I just want to say, it's not complicated at all!   Basically you will need to Prepare an SD card and install Android SDK (you need adb ). Install a custom boot loader ( TWRP is the de facto tool at the moment). Use adb to copy custom...
15 comments
Read more

Building a native plugin for Intel Realsense D415 for Unity

Based on a previous post , I decided to write a plugin for the Intel Realsense SDK methods so we can use these methods from within Unity. FYI Intel also has their own Unity wrapper in their Github repository , but for our projects, I needed to perform image processing with OpenCV and passing the results to Unity instead of just the raw image/depth data. There is a plugin called OpenCVForUnity to use OpenCV functions from Unity but previous experiments indicate the image processing inside Unity can take a long time. I hope this post can help someone else who wants to use Intel's cameras or any other devices natively in Unity. Test Environment Windows 10 64bit Unity 2017.2.0f3 x64 bit Realsense SDK from Intel CMake 3.0 or higher Steps Checkout the native plugin code here . Don't worry about the other projects in the same repository. The relevant code is in the link above. Checkout the Unity sample project here . However, instead of master you need to go to the br...
5 comments
Read more

OpenCV native plugin for Unity in IOS

In one of my recent projects, I needed to use OpenCV from within Unity, in IOS. The asset called OpenCVForUnity is overkill because I didn't need the whole OpenCV library, just a few functions. In addition, this asset does not implement the whole OpenCV library so unless you know that what you need is included you may find it lacking when you discover it does not support some functions you need. As my project involves some trial and error and mixing algorithms together I decided to go with a native plugin. Overview In IOS, a native library is built as a bundle . We need to put this bundle inside Unity's Plugins/OSX folder to use it. Therefore, we need to create two projects. An XCode project to build the native plugin. A Unity project to use the plugin. Dependencies Of course, since we need to use OpenCV we will have to install it first. Tutorials on installing OpenCV on IOS are abundant and I will not include them here. Assuming you have installed OpenCV go to t...
2 comments
Read more