Time is a limited resource

Some of my recent clients are interested in image classification using limited learning data. A major use case is in detecting defective products on the manufacturing line. Defective products are normally caught by human operators watching the production line. It requires constant concentration and effort so if this can be automated it will bring a lot of productivity boost to manufacturers. One method to overcome this issue is by creating a network to compare input images with the training dataset. For defective products, the images will look different from the passing products. Experiments For this experiment, I used a Siamese Network to generate embeddings for images of Pokemon. The Pokemon has 4 classes, Pikachu, Squirtle, Bulbasaur and Charmander. The images are scraped from the internet using the Bing API. The images in the training dataset is trained to find the set of weights that will clearly separate the 4 classes. If a new image which is not included in the 4 classes is used...

In this post I'd like to share my experiments in calibrating a projector using a camera with known intrinsic and extrinsic parameters. No math, just concepts. The final result is a projected image that follows a marker's position and orientation, and adjusts the image. The calibration process enables us to automatically setup the relationship between the projector and the camera that detects the marker. In short, these are the steps performed Calibrate camera and projector relative position and orientation Detect the marker using the camera Adjust the image position and orientation based on the marker position and orientation Experiment setup The following videos show the results of the experiments.   As you can see, the projected dots follow the position and orientation of the chessboard marker. This setup will be very useful for projection mapping installations. Of course, at the current setup it will be useful only on small sizes but by using a larger marker we can theoretic...

Just a short post today on Intel's Openvino toolkit. It is really a great collection of machine learning models that can speed up development. I played around with their pose estimation model and the results are quite good. Final result of the pose detection system This system combined with the Intel Realsense depth cameras offer a promising real world solution to the pose estimation problem. As shown in the video, it works reasonably well on my laptop with an NVIDIA GPU (1060) Graphics card. The white spheres indicate the feet limbs and thanks to the Realsense depth camera we can estimate the positions of the limbs in 3D space. Although I haven't tried it yet but the models can be converted to work on edge platforms such as the Movidius device. Obviously this opens up a whole array of applications which I hope to share again in a future post.

In one of my projects, I needed to reconstruct an object for automatic painting by a robot arm. The target object is a hollow electricity meter box and the robot needs to paint the insides of this object. It is easy to paint the exterior of an object, but to paint the insides of a hollow object we need a decent enough point cloud to make sure our path planning works properly. An illustration of the object to draw is given below As indicated by the red arrows, the area just behind the small area is the target of the painter. Since I don't have electricity boxes lying around, I tried my experiments on a cereal carton. This is a long winding road of trials and errors and is divided into several parts. The first part is about the point cloud generation. Followed by experiments on camera movement tracking and scene and object reconstruction from point clouds. Finally a method to determine path points for the robot painter will be discussed. Preliminary research One of the first things t...

In this article, I'd like to share my project to count the number of people walking past a store/shop etc. The motivation was when I was part of a family oriented digital theme park company, we would want to know if daily sales (visitors) are good enough.  First, some background. Our theme park (store) is part of a department store/shopping complex. So we did not have a large piece of land like Disneyland, where all visitors are definitely there to visit the park. In our case, visitors to the shopping complex will consider taking their children to our entertainment theme park, so we would get a certain percentage of the total visitors of the shopping complex. So for example if we have 100 visitors on one single day, how do we know if this number is good or bad? If we know the number of people passing through our store we would have a better context. If the total number of people is say around 500 then we would be getting 20%, which doesn't seem so bad. But if the shopping compl...

When trying to send FCM notifications I found out that Google has changed their API specifications. The legacy API still works but if you want to use the latest v1 API you need to make several changes. The list of changes is listed on their site so I won't be repeating them again but I'll just mention some of the things that caused some trial and error on my project. The official guide from Google is here : Official Migration Guide to v1 . The request must have a Body with the JSON containing the message data. Most importantly it needs to have "message" field which must contain the target of the notification. Usually this is a Topic, or Device IDs. Since my previous project was using GAS, my request had a field called "payload" instead of "body". Using the request from my previous project, my request in Node JS was as follows: request ({ url: 'https://fcm.googleapis.com/v1/projects/safe-door-278108/messages:send' , method: ...

In this post I'd like to share about one of my projects involving doodles and bringing them to live with Unity. We prepare doodle papers and some crayons and let children color them. After they're done, we scan the images and they appear on the screen. The screen is projected on walls using projectors. Doodles come alive on the screen Project flow I utilized document scanners readily available such as the following. A document scanner The scanner has many helpful features, such as cropping and rotating the scanned images so they are nice and straight even if the paper is slightly rotated or not aligned properly. The scanned images are stored in a server and a Unity application polls the server for new images every few seconds. For the server I initially used AWS S3 for image storage and later on we switched to a local image server with Node JS. Attaching 2D Texture to a 3D Model I no longer have access to the actual doodle papers but they look l...

This is just a memo of the challenges I faced when running a model training on Google Colab, while following a great tutorial here . Mind the versions Tensorflow is currently at version 2.2.0 but most tutorials are still using the contrib package, and there is no known easy way to update the code to remove dependency on contrib. So my best bet is to downgrade the tensorflow version to 1.x. Since Google Colab only gives the options of either 1.x or 2.x and we cannot specify the exact version, I ended up with version 1.15.2. Even with the command :  %tensorflow_version  1.15.0 I ended up with : 1.15.2 Another pitfall was the version of numpy. Installing numpy gives us the version 1.18.3 but for some reason this generates the error : TypeError: 'numpy.float64' object cannot be interpreted as an integer Downgrading numpy to version 1.17.4 solved this for me. It seems we don't need ngrok for tensorboard With the command :  %load_ext tensorboard W...

In one of my recent projects, I needed to use OpenCV from within Unity, in IOS. The asset called OpenCVForUnity is overkill because I didn't need the whole OpenCV library, just a few functions. In addition, this asset does not implement the whole OpenCV library so unless you know that what you need is included you may find it lacking when you discover it does not support some functions you need. As my project involves some trial and error and mixing algorithms together I decided to go with a native plugin. Overview In IOS, a native library is built as a bundle . We need to put this bundle inside Unity's Plugins/OSX folder to use it. Therefore, we need to create two projects. An XCode project to build the native plugin. A Unity project to use the plugin. Dependencies Of course, since we need to use OpenCV we will have to install it first. Tutorials on installing OpenCV on IOS are abundant and I will not include them here. Assuming you have installed OpenCV go to t...

This is an interactive installation that aims to combine learning words and fun in devising strategies to gather letters. Collect letters to complete the word The letters are dangling at various locations, and only the yellow colored characters can collect them. But these characters cannot jump that high, so players need to put blocks as stepping stones to help these cute yellow fellas to reach the letters. The blocks are generated by touching the screen (wall) and can be removed by touching it again. System overview The system is simply a projector and a laser range finder to detect touches on the wall. You may realize the black object in the photo above at the top of the wall. This is the laser range finder and for this project I used Hokuyo's UST10-LX. Content is developed with Unity. How the touch detection works Readings from the sensor This image is the visualization of the data points detected. With the sensor placed on top of the wall looking down, ...

This is one of the installations I developed for a digital entertainment theme park. It is an interactive digital sandbox with normal white sand and I used a projector and depth sensor to project topological colors and virtual objects depending on the height and shape of the sand. I also used QR markers on 3D printed objects to interact with the virtual content. Overview of installation A sandbox of 2 x 1.5 meters with projected content As shown in the image, depending on the height of the sand, the projected colors change from deep water, shallow water, beach, green grass, dark green forest, rocky mountains and finally snowy mountains. These are the 7 different height levels defined by the system. In addition to the topological colors, it also has fishes and boats in the waters, lava coming out of a crater and other interactive elements. The sand depth is detected real time and projection updated Fish cannot go on land Hardware setup Sandbox made of w...

This post is a side note for anyone trying to setup a Jenkins job to build an Android application. Error because google-services.json is not found google-services.json is used to access Google services, APIs etc. and is probably the most used file in almost all applications. However this is automatically ignored if you use gitignore and it won't be available in the build machine where Jenkins is run. The solution is obviously to un-ignore this file, push and try again. Cannot run nohup.exe error This was solved by adding git command to Windows env path. Merge debug resources failed error The full error is as follows: java.util.concurrent.ExecutionException:com.android.builder.internal.aapt.v2.Aapt2Exception: AAPT2 error: check logs for details :app:mergeDebugResources FAILED This can be solved by setting the  GRADLE_USER_HOME  Global properties in  Jenkins > Manage jenkins > Configure System JDK error Kotlin could not find the required JDK tools ...

In this third and final entry, I will share how we use Google's Protocol Buffer to package the data structure when sharing and transmitting between the PCs. Basic information on Protocol Buffer can be found on Google's official page . But to put it briefly it is a serialization library that manages your structured data to and from a variety of data streams. In my case, since the detection result is a person's position on a 2D image, my data (proto file) looks like this. message people_position { repeated float x = 1 ; repeated float y = 2 ; repeated string device_number = 3 ; } x, y of course is the detected person's position on the image, and device_number is serial number of the device that detected it. The field rule "repeated" means that there will be multiple instances of this field, which in my case applies to when there are multiple people detected. Protocol Buffer Compilation In my project, the detection algorithm is written...

In this second part, I will share how the data between several Realsense devices are merged. First, an overview on how the devices are connected together. The data flow is as follows PC 1 and PC 2 gathers detection data from the devices attached to them. PC 2 has another process that merges the data and sends them to PC 3. All PCs have knowledge of the device layout and transformation Matrix to transform coordinates into Unity coordinates.* A little more on this later. PC 2 transfers the transformed coordinates to PC 3 through TCP/IP connection. As mentioned above, the devices need to know its position in the layout and transforms it detection results into the larger area axes as shown in the figure below. Each dotted rectangle denotes the view of one Realsense depth camera. Each camera is assigned an identifier (a, b) that reflects its position in the layout. Both PC 1 and PC 2 holds a settings file that tells them which devices are attached to each PC, ma...

This is a three part post on using Intel's Realsense depth cameras in a digital theme park, particularly for detecting people from a top-view position. The detected positions are used as input to provide visual effects projected back onto the floor, or in some cases, onto a ball pool. The following image will probably better explain the setup used. If you look closely, there are three Realsense cameras attached to the ceiling close to the projectors. The targets of the project are: We want to project wave ripple effects whenever someone is present. The effect center is where the person is standing and ripples out just like a ripple in the water. In this first part, I will share how we decided to detect a person using Realsense D415 cameras. Person Detection Process There are several ways to detect people, but since we do not need a very accurate position information I found that one of the easiest ways is to utilize Realsense's depth sensor to detect ...

Normally apps use database and a backend to operate. Here I'd like to share another way of using Google Sheets as a database to replace SQL, Mongo and the likes. It is not very customizable or extendable but it is a very good way to prototype or for small scale services or internal tools. Whenever a change happens on the sheets, the app gets a notification so it can re-poll the sheets to get new data. Using Google Sheets as a Database Overview What is needed for this to work. Google account, Google Sheets file. Created Firebase project. Created android app and registered to use FCM. The app has access rights to the Google Sheets to read its contents. Reading a Google Sheets file has been covered many times before so I will just refer you to one of them here . Flow of the application Someone in charge of the data will make changes to the Google Sheets file and finally when everything is done, he/she will modify a single cell that acts as the "Apply button...