Skip to main content

Image Processing with Intel Realsense for Digital Interactive Installations (2/3)

In this second part, I will share how the data between several Realsense devices are merged. First, an overview on how the devices are connected together.


The data flow is as follows

  1. PC 1 and PC 2 gathers detection data from the devices attached to them.
  2. PC 2 has another process that merges the data and sends them to PC 3.
  3. All PCs have knowledge of the device layout and transformation Matrix to transform coordinates into Unity coordinates.* A little more on this later.
  4. PC 2 transfers the transformed coordinates to PC 3 through TCP/IP connection.
As mentioned above, the devices need to know its position in the layout and transforms it detection results into the larger area axes as shown in the figure below.


Each dotted rectangle denotes the view of one Realsense depth camera. Each camera is assigned an identifier (a, b) that reflects its position in the layout.
Both PC 1 and PC 2 holds a settings file that tells them which devices are attached to each PC, mapped to the Realsense device ID, which is unique. With this information, the detected positions from PC 1 and PC 2 are transformed to the top left axes of the total area.

The layout of devices is stored as JSON file on all PCs. The JSON structure is as follows.

{
"devices": [
{
"serial_number": "747612060144",
"pos_x": 0,
"pos_y": 2,
"max_limit_x": 640,
"max_limit_y": 720,
"min_limit_x": 300,
"min_limit_y": 300,
"overlap_x": 0,
"overlap_y": 0
},
{
"serial_number": "811112060489",
"pos_x": 1,
"pos_y": 0,
"max_limit_x": 1280,
"max_limit_y": 720,
"min_limit_x": 0,
"min_limit_y": 0,
"overlap_x": 0,
"overlap_y": 0
}
]
}

The members are

  • serial_number. This is the device's unique serial number as returned by dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER) 
  • pos_x and pos_y are the layout identifiers as shown on the device layout figure above.
  • max_limit and min_limit denotes the active areas to perform detection.
  • overlap is currently not used but I plan to consider the case where the devices are not perfectly adjacent.
This concludes the data sharing part. In the next entry, I will share how we use Google's Protocol Buffer to package the detection data when sharing and transmitting from and to the three PCs.

Comments

Post a Comment

Popular posts from this blog

Using FCM with the new HTTP v1 API and NodeJS

When trying to send FCM notifications I found out that Google has changed their API specifications. The legacy API still works but if you want to use the latest v1 API you need to make several changes. The list of changes is listed on their site so I won't be repeating them again but I'll just mention some of the things that caused some trial and error on my project. The official guide from Google is here : Official Migration Guide to v1 . The request must have a Body with the JSON containing the message data. Most importantly it needs to have "message" field which must contain the target of the notification. Usually this is a Topic, or Device IDs. Since my previous project was using GAS, my request had a field called "payload" instead of "body". Using the request from my previous project, my request in Node JS was as follows: request ({ url: 'https://fcm.googleapis.com/v1/projects/safe-door-278108/messages:send' , method: ...
Post a Comment
Read more

Object detection with Google Colab and Tensorflow

This is just a memo of the challenges I faced when running a model training on Google Colab, while following a great tutorial here . Mind the versions Tensorflow is currently at version 2.2.0 but most tutorials are still using the contrib package, and there is no known easy way to update the code to remove dependency on contrib. So my best bet is to downgrade the tensorflow version to 1.x. Since Google Colab only gives the options of either 1.x or 2.x and we cannot specify the exact version, I ended up with version 1.15.2. Even with the command :  %tensorflow_version  1.15.0 I ended up with : 1.15.2 Another pitfall was the version of numpy. Installing numpy gives us the version 1.18.3 but for some reason this generates the error : TypeError: 'numpy.float64' object cannot be interpreted as an integer Downgrading numpy to version 1.17.4 solved this for me. It seems we don't need ngrok for tensorboard With the command :  %load_ext tensorboard W...
Post a Comment
Read more

Microsoft Azure Face API and Unity

During one of my projects, I came across Microsoft's face recognition API (Azure Face API) and it looked good enough to recognize people's faces and detect if a person is a newcomer or a repeating customer to our store. As our installations mainly use the game engine Unity, I wanted to be able to use the Face API from Unity. Face API does not have an SDK for Unity but their requests are just HTTP requests so the Networking classes in Unity can be wrapped into methods to make it easy to call these APIs. First of all, to those who just want to see the code, here it is . My tests focus on the identification of a face in an input image. The full tutorial I followed can be found here . The Main scene goes through the steps in the tutorial such as creating a PersonGroup and adding Persons to the group if it is not created yet. Just make sure you: Change the API key. I used a free trial key which is no longer valid. Use whatever images you want. I don't mind you us...
Post a Comment
Read more