Skip to main content

3D Reconstruction with markers for multi view object alignment

In one of my projects, I needed to reconstruct an object for automatic painting by a robot arm. The target object is a hollow electricity meter box and the robot needs to paint the insides of this object. It is easy to paint the exterior of an object, but to paint the insides of a hollow object we need a decent enough point cloud to make sure our path planning works properly.

An illustration of the object to draw is given below


As indicated by the red arrows, the area just behind the small area is the target of the painter.

Since I don't have electricity boxes lying around, I tried my experiments on a cereal carton.



This is a long winding road of trials and errors and is divided into several parts. The first part is about the point cloud generation. Followed by experiments on camera movement tracking and scene and object reconstruction from point clouds. Finally a method to determine path points for the robot painter will be discussed.

Preliminary research

One of the first things to try was 3D reconstruction of the scene. Hopefully we can get a good representation of the object that is enough for the path finding algorithm.

3D Scene Reconstruction

Using a Realsense D415 depth camera, I attempted to capture the cereal box as point cloud.

Point cloud representation of the cereal box

This approach has a few problems, mainly:
  • Edges have a lot of noise
  • Sides that are not visible from the camera are not detected
To solve these problems, I looked into ways to merge multiple view point clouds so we can get a full representation of the box. I tried first with ICP and performing iterative point registration but the accuracy is not good enough so I tried using a marker to capture various views and merge the point clouds together.

Marker based multi-view point cloud reconstruction

The following is my attempt to get a better point cloud reconstruction from multi view using a marker of known pattern and size. The marker chosen was the chessboard calibration pattern widely used in camera calibrations. The following images depict the scene setup.







Three views of a sofa in the living room along with the chessboard pattern

The usage of a marker improved the reconstruction a lot and I managed to get the reconstructed sofa as shown below.


3D point cloud representation of the sofa



Color image of the sofa and its surroundings from the same view

Implementing marker based multi-view registration on the cereal box

So the next step will be to try the reconstruction process on the cereal box. The marker has also to be visible to the camera so this poses a small challenge in camera placement but overall I found that overcoming this is not so hard.




Two views of the cereal box next to a known marker

With this method, there is no need to incrementally register point clouds and we can just rely on the transformation matrix of the camera relative to the marker.

The result reconstruction is shown below.


We can see that the transformation matrix from the marker really helped in aligning the two views, even though they are far apart and we don't have incremental positions in between. The accuracy needs more work however and I believe this can be overcome by better hardware, and also more thorough calibration of the camera parameters. Using multiple markers may also improve the transformation matrix estimation.

Future work

A better calibration process, along with multiple markers seem to be promising to increase accuracy of alignment.

Comments

Post a Comment

Popular posts from this blog

Installing a custom ROM on Android (on the GT-N8013)

It's been a while since my last entry and since it is a new start in 2019, I thought I'd write something about "gone with the old and in with the new". I've had my Samsung Galaxy Note 10.1 (pnotewifi) since 2014, and it's one of the early Galaxy Note tablet series. It has served me well all this years but now it just sits there collecting dust. My old Samsung GT-N8013 I've known a long time about custom Android ROMs like CyanogenMod but has never had the motivation to try them out, until now ! Overview of the process For beginners like me, I didn't have an understanding of the installation process and so it looked complicated and it was one of the reasons I was put off in trying the custom ROM. I just want to say, it's not complicated at all!   Basically you will need to Prepare an SD card and install Android SDK (you need adb ). Install a custom boot loader ( TWRP is the de facto tool at the moment). Use adb to copy custom...
15 comments
Read more

Building a native plugin for Intel Realsense D415 for Unity

Based on a previous post , I decided to write a plugin for the Intel Realsense SDK methods so we can use these methods from within Unity. FYI Intel also has their own Unity wrapper in their Github repository , but for our projects, I needed to perform image processing with OpenCV and passing the results to Unity instead of just the raw image/depth data. There is a plugin called OpenCVForUnity to use OpenCV functions from Unity but previous experiments indicate the image processing inside Unity can take a long time. I hope this post can help someone else who wants to use Intel's cameras or any other devices natively in Unity. Test Environment Windows 10 64bit Unity 2017.2.0f3 x64 bit Realsense SDK from Intel CMake 3.0 or higher Steps Checkout the native plugin code here . Don't worry about the other projects in the same repository. The relevant code is in the link above. Checkout the Unity sample project here . However, instead of master you need to go to the br...
5 comments
Read more

OpenCV native plugin for Unity in IOS

In one of my recent projects, I needed to use OpenCV from within Unity, in IOS. The asset called OpenCVForUnity is overkill because I didn't need the whole OpenCV library, just a few functions. In addition, this asset does not implement the whole OpenCV library so unless you know that what you need is included you may find it lacking when you discover it does not support some functions you need. As my project involves some trial and error and mixing algorithms together I decided to go with a native plugin. Overview In IOS, a native library is built as a bundle . We need to put this bundle inside Unity's Plugins/OSX folder to use it. Therefore, we need to create two projects. An XCode project to build the native plugin. A Unity project to use the plugin. Dependencies Of course, since we need to use OpenCV we will have to install it first. Tutorials on installing OpenCV on IOS are abundant and I will not include them here. Assuming you have installed OpenCV go to t...
2 comments
Read more