Time is a limited resource

This is an interactive installation that aims to combine learning words and fun in devising strategies to gather letters. Collect letters to complete the word The letters are dangling at various locations, and only the yellow colored characters can collect them. But these characters cannot jump that high, so players need to put blocks as stepping stones to help these cute yellow fellas to reach the letters. The blocks are generated by touching the screen (wall) and can be removed by touching it again. System overview The system is simply a projector and a laser range finder to detect touches on the wall. You may realize the black object in the photo above at the top of the wall. This is the laser range finder and for this project I used Hokuyo's UST10-LX. Content is developed with Unity. How the touch detection works Readings from the sensor This image is the visualization of the data points detected. With the sensor placed on top of the wall looking down, ...

This is one of the installations I developed for a digital entertainment theme park. It is an interactive digital sandbox with normal white sand and I used a projector and depth sensor to project topological colors and virtual objects depending on the height and shape of the sand. I also used QR markers on 3D printed objects to interact with the virtual content. Overview of installation A sandbox of 2 x 1.5 meters with projected content As shown in the image, depending on the height of the sand, the projected colors change from deep water, shallow water, beach, green grass, dark green forest, rocky mountains and finally snowy mountains. These are the 7 different height levels defined by the system. In addition to the topological colors, it also has fishes and boats in the waters, lava coming out of a crater and other interactive elements. The sand depth is detected real time and projection updated Fish cannot go on land Hardware setup Sandbox made of w...

This post is a side note for anyone trying to setup a Jenkins job to build an Android application. Error because google-services.json is not found google-services.json is used to access Google services, APIs etc. and is probably the most used file in almost all applications. However this is automatically ignored if you use gitignore and it won't be available in the build machine where Jenkins is run. The solution is obviously to un-ignore this file, push and try again. Cannot run nohup.exe error This was solved by adding git command to Windows env path. Merge debug resources failed error The full error is as follows: java.util.concurrent.ExecutionException:com.android.builder.internal.aapt.v2.Aapt2Exception: AAPT2 error: check logs for details :app:mergeDebugResources FAILED This can be solved by setting the  GRADLE_USER_HOME  Global properties in  Jenkins > Manage jenkins > Configure System JDK error Kotlin could not find the required JDK tools ...

In this third and final entry, I will share how we use Google's Protocol Buffer to package the data structure when sharing and transmitting between the PCs. Basic information on Protocol Buffer can be found on Google's official page . But to put it briefly it is a serialization library that manages your structured data to and from a variety of data streams. In my case, since the detection result is a person's position on a 2D image, my data (proto file) looks like this. message people_position { repeated float x = 1 ; repeated float y = 2 ; repeated string device_number = 3 ; } x, y of course is the detected person's position on the image, and device_number is serial number of the device that detected it. The field rule "repeated" means that there will be multiple instances of this field, which in my case applies to when there are multiple people detected. Protocol Buffer Compilation In my project, the detection algorithm is written...

In this second part, I will share how the data between several Realsense devices are merged. First, an overview on how the devices are connected together. The data flow is as follows PC 1 and PC 2 gathers detection data from the devices attached to them. PC 2 has another process that merges the data and sends them to PC 3. All PCs have knowledge of the device layout and transformation Matrix to transform coordinates into Unity coordinates.* A little more on this later. PC 2 transfers the transformed coordinates to PC 3 through TCP/IP connection. As mentioned above, the devices need to know its position in the layout and transforms it detection results into the larger area axes as shown in the figure below. Each dotted rectangle denotes the view of one Realsense depth camera. Each camera is assigned an identifier (a, b) that reflects its position in the layout. Both PC 1 and PC 2 holds a settings file that tells them which devices are attached to each PC, ma...

This is a three part post on using Intel's Realsense depth cameras in a digital theme park, particularly for detecting people from a top-view position. The detected positions are used as input to provide visual effects projected back onto the floor, or in some cases, onto a ball pool. The following image will probably better explain the setup used. If you look closely, there are three Realsense cameras attached to the ceiling close to the projectors. The targets of the project are: We want to project wave ripple effects whenever someone is present. The effect center is where the person is standing and ripples out just like a ripple in the water. In this first part, I will share how we decided to detect a person using Realsense D415 cameras. Person Detection Process There are several ways to detect people, but since we do not need a very accurate position information I found that one of the easiest ways is to utilize Realsense's depth sensor to detect ...