Skip to main content

Building a native plugin for the Asus Xtion2 for Unity

During one of the projects in my company I needed to build a native plugin to let Unity communicate with the Asus Xtion2, specifically to get its depth data. We used to be able to do this pretty easily with the Kinect but since Microsoft discontinued it, we need to start looking for alternatives.

Test Environment

  • Windows 10 64 bit.
  • Unity 2017.2.0f3 x64. Important! Choose x64 or x86 to match your Unity installation.
  • OpenNI For Xtion2 SDK. The official SDK is somewhat different from the OpenNI SDK provided by Asus but it should behave the same. The one provided by Asus can be downloaded here. Make sure you choose the latest one.
  • CMake 3.0 or higher

Who is it for

Someone who has been using Unity for some time and is comfortable with the concept of classes and objects. It will be very helpful if you know C++ and pointers too.

Steps

Since the code is provided, I will only go over the major steps. Let me know in the comments if I miss anything.
  1. Build and install OpenNI. Since there are already many tutorials on building and installing libraries I will not go into the details here.
  2. Assuming OpenNI is installed in {OPENNI_INSTALL_PATH}, make sure that you have {OPENNI_INSTALL_PATH}/lib/OpenNI2.lib and you can also run the samples located in {OPENNI_INSTALL_PATH}/Samples/Bin
  3. You can find the code for the native plugin here. 
  4. The Unity project can be found here.
  5. Both lack documentation so in the meantime please check out the code and let me know in the comments if anything is unclear.
  6. You can find the CMakeList file for the native code under src/ folder.
  7. Adjust the install paths and parameters to suit your environment and build with CMake.
  8. If everything goes well with CMake you should see a solution (.sln) file in the build folder
  9. Open this solution file with Visual Studio and build the project.
  10. Don't forget to set the build settings to either Debug or Release and make sure to build x86 or x64 according to your architecture.
  11. If the build is successful, you should see an XtionCapture.dll in the build folder.
  12. Copy this dll to the Unity project in Assets/Plugins.
  13. You will also need to copy some file from the OpenNI SDK. Go to {OPENNI_INSTALL_PATH}/Redist and copy all the files there into the Assets/Plugins folder.
  14. If you managed to come this far then just press Play and enjoy!

Overview of plugin interface

The details can be found in the code but I'd like to share the idea behind the plugin.
  1. Instantiate a new capture class (xtion_capture) and let this class hold references to the Xtion Device. So pointers to device handlers, depth frames etc. are registered as this class' members.
  2. When xtion_capture is instantiated it returns a pointer to its object instance to Unity. This pointer is how Unity calls the capture class and retrieves data from its (member) pointers.
The above design is illustrated below.


Future work

I will definitely want to add documentations, and clean up the repository. If I have the time I'd try getting the color image too. Best of luck and I would be happy to help of anyone needs to have the same system setup.

Comments

Post a Comment

Popular posts from this blog

Using FCM with the new HTTP v1 API and NodeJS

When trying to send FCM notifications I found out that Google has changed their API specifications. The legacy API still works but if you want to use the latest v1 API you need to make several changes. The list of changes is listed on their site so I won't be repeating them again but I'll just mention some of the things that caused some trial and error on my project. The official guide from Google is here : Official Migration Guide to v1 . The request must have a Body with the JSON containing the message data. Most importantly it needs to have "message" field which must contain the target of the notification. Usually this is a Topic, or Device IDs. Since my previous project was using GAS, my request had a field called "payload" instead of "body". Using the request from my previous project, my request in Node JS was as follows: request ({ url: 'https://fcm.googleapis.com/v1/projects/safe-door-278108/messages:send' , method: ...
Post a Comment
Read more

Object detection with Google Colab and Tensorflow

This is just a memo of the challenges I faced when running a model training on Google Colab, while following a great tutorial here . Mind the versions Tensorflow is currently at version 2.2.0 but most tutorials are still using the contrib package, and there is no known easy way to update the code to remove dependency on contrib. So my best bet is to downgrade the tensorflow version to 1.x. Since Google Colab only gives the options of either 1.x or 2.x and we cannot specify the exact version, I ended up with version 1.15.2. Even with the command :  %tensorflow_version  1.15.0 I ended up with : 1.15.2 Another pitfall was the version of numpy. Installing numpy gives us the version 1.18.3 but for some reason this generates the error : TypeError: 'numpy.float64' object cannot be interpreted as an integer Downgrading numpy to version 1.17.4 solved this for me. It seems we don't need ngrok for tensorboard With the command :  %load_ext tensorboard W...
Post a Comment
Read more

Building a native plugin for Intel Realsense D415 for Unity

Based on a previous post , I decided to write a plugin for the Intel Realsense SDK methods so we can use these methods from within Unity. FYI Intel also has their own Unity wrapper in their Github repository , but for our projects, I needed to perform image processing with OpenCV and passing the results to Unity instead of just the raw image/depth data. There is a plugin called OpenCVForUnity to use OpenCV functions from Unity but previous experiments indicate the image processing inside Unity can take a long time. I hope this post can help someone else who wants to use Intel's cameras or any other devices natively in Unity. Test Environment Windows 10 64bit Unity 2017.2.0f3 x64 bit Realsense SDK from Intel CMake 3.0 or higher Steps Checkout the native plugin code here . Don't worry about the other projects in the same repository. The relevant code is in the link above. Checkout the Unity sample project here . However, instead of master you need to go to the br...
5 comments
Read more