Image Processing with Intel Realsense for Digital Interactive Installations (3/3)

In this third and final entry, I will share how we use Google's Protocol Buffer to package the data structure when sharing and transmitting between the PCs.

Basic information on Protocol Buffer can be found on Google's official page. But to put it briefly it is a serialization library that manages your structured data to and from a variety of data streams.

In my case, since the detection result is a person's position on a 2D image, my data (proto file) looks like this.

message people_position{
	repeated float x = 1;
	repeated float y = 2;
	repeated string device_number = 3;
	}

x, y of course is the detected person's position on the image, and device_number is serial number of the device that detected it. The field rule "repeated" means that there will be multiple instances of this field, which in my case applies to when there are multiple people detected.

Protocol Buffer Compilation

In my project, the detection algorithm is written in C++, while the contents are made with the game engine Unity, which uses C#.  Although Google, provides documentation on compiling with C#, they do not specifically target Unity. However, I found a repository that handles compiling in Unity here.

Pitfalls

The most common mistake I make when using protocol buffers is not updating all the proto files whenever there is a change in  the data structure. Any change requires all proto files in all project to be recompiled.

Afterthoughts

It has been a great experience building this multi device system with Realsense depth cameras. Although the individual components are not too complicated (my proto structures are simple, and the image processing was greatly simplified with depth thresholding) but combined together they managed to provide realtime effects to our digital ball pool. 

Finally, a big thanks to Intel for a great product, Google for sharing Protobuf, and of course OpenCV for their great image processing library.

See Part 1 here : http://www.dailytinkerer.com/2020/04/image-processing-with-intel-realsense.html

See Part 2 here : http://www.dailytinkerer.com/2020/04/image-processing-with-intel-realsense_12.html