Skip to main content

Building a native plugin for Intel Realsense D415 for Unity

Based on a previous post, I decided to write a plugin for the Intel Realsense SDK methods so we can use these methods from within Unity. FYI Intel also has their own Unity wrapper in their Github repository, but for our projects, I needed to perform image processing with OpenCV and passing the results to Unity instead of just the raw image/depth data. There is a plugin called OpenCVForUnity to use OpenCV functions from Unity but previous experiments indicate the image processing inside Unity can take a long time. I hope this post can help someone else who wants to use Intel's cameras or any other devices natively in Unity.

Test Environment

  • Windows 10 64bit
  • Unity 2017.2.0f3 x64 bit
  • Realsense SDK from Intel
  • CMake 3.0 or higher

Steps

  1. Checkout the native plugin code here. Don't worry about the other projects in the same repository. The relevant code is in the link above.
  2. Checkout the Unity sample project here. However, instead of master you need to go to the branch feature/realsense to see the sample scene for Realsense cameras.
  3. Build and install Realsense SDK. Make sure you match the architecture of the OS and Unity.
  4. Modify the CMakelist file in the native code repository to match your installation paths and generate the native code solution file. If you don't change the project name, the solution file generated will be named camera_vision.sln.
  5. Build the solution file and you should find the dll generated in build/uplugin/
  6. Copy this DLL file (if you didn't rename the project it should be uplugin_realsense_d415.dll) and put this in the Assets/Plugins/x64 folder of the Unity project.
  7. Go to the test scene in Assets/WindowsNativePlugin/Scenes/RealsenseD415.unity
  8. Press Instantiate -> List Devices -> Get Depth Data in this order.
  9. You should see the depth data being passed back to Unity.
I plan to add documentation and also reorganize/refactor the scripts but in the meantime please see the code and comment here if there are any questions.

Overview of plugin interface

This plugin borrows heavily from the previous Asus Xtion2 plugin.
  1. Unity calls the _Create function, which is a wrapped to the DLL method which instantiates a new realsense_capture instance. The memory address of this instance is returned to Unity as an IntPtr and as long as Unity holds this pointer we can interact with the DLL to call its methods.
  2. The native plugin returns data in the form of pointers to buffers which hold the actual data. These pointers are in the form of IntPtr and we use Unity's Marshaling methods to convert the values back to C# variables.
Refer to the illustration below for an overview.

Future work

Documentation, and refactoring and reorganizing of scripts. Sorry the repos are a mess right now. Please comment if you have any problems and I will try to help as much as possible.

Comments

  1. Daniel BrownFebruary 5, 2020 at 1:39 PM

    Where are you performing the image processing with OpenCV? Also did you ever get the ReaslSense camera working with OpenCV inside of Unity?

    ReplyDelete
    Replies
    1. Sonny KurniawanMay 1, 2020 at 10:03 AM

      Daniel, sorry for the late reply. I perform the image processing on the native plugin side. Meaning I pass all the parameters from Unity, process them on the native side and return the results. Maybe you can check here : https://github.com/sonnyky/WindowsNativePlugin.
      In one of the sample scenes called "Detect Shape", I take video stream with the native plugin, detects the shapes and sends the results as Texture2D to Unity. Effectively running the videos captured and processed by OpenCV on Unity. Is this what you're looking for?

      Delete
  2. ziteFebruary 27, 2020 at 11:45 PM

    I'd also like to know if you got realsense working with opencv in unity. I haven't been able to make this work yet. Do you have an example project perhaps?

    ReplyDelete
    Replies
    1. Sonny KurniawanMay 1, 2020 at 10:05 AM

      Hi Zite, I had the link also on the steps outlined above. Maybe it was a little hard to find among all those text. https://github.com/sonnyky/WindowsNativePlugin

      You can check the Detect Shape scene. There I took video stream with the native plugin and does shape detection and returns the results to Unity

      Delete
  3. Sonny KurniawanMay 1, 2020 at 10:11 AM

    Hi, the Unity project github link is here:
    https://github.com/sonnyky/WindowsNativePlugin.

    It seems it is easy to miss with all the text in the article. I'm still working on the documentation, so if you have any problems please comment here or on the github page directly. Thank you.

    ReplyDelete

Post a Comment

Popular posts from this blog

Installing a custom ROM on Android (on the GT-N8013)

It's been a while since my last entry and since it is a new start in 2019, I thought I'd write something about "gone with the old and in with the new". I've had my Samsung Galaxy Note 10.1 (pnotewifi) since 2014, and it's one of the early Galaxy Note tablet series. It has served me well all this years but now it just sits there collecting dust. My old Samsung GT-N8013 I've known a long time about custom Android ROMs like CyanogenMod but has never had the motivation to try them out, until now ! Overview of the process For beginners like me, I didn't have an understanding of the installation process and so it looked complicated and it was one of the reasons I was put off in trying the custom ROM. I just want to say, it's not complicated at all!   Basically you will need to Prepare an SD card and install Android SDK (you need adb ). Install a custom boot loader ( TWRP is the de facto tool at the moment). Use adb to copy custom...
15 comments
Read more

Pitfalls during Training and Object Detection with TensorFlow for Absolute Beginners

This article is based on the great tutorial here on how to train and detect custom objects with Tensorflow. I also referred to the official documentations here and  here  for running Tensorflow model building locally. It was my first custom detection project and I faced some hiccups along the way and this article is to log and share my finding so it can help other beginners like me. In the end, I managed to train a tensorflow model to detect Batsumaru , a character from Sanrio. This is how the detection will look like. The tools Windows 10 Pro 64 Tensorflow originally 1.7.1 and upgraded to 1.12.0. I will share the reason later. Python 3.5.4 LabelImg for image labeling PyCharm IDE Steps and Pitfalls Some of the mistakes I made and other discoveries when following the guide. I will not repeat the steps mentioned in the original guide, but only the parts where I had to deviate from the walkthrough and found out things by myself. The training and...
1 comment
Read more

Unity Best Practices for Beginners

Unity is a fabulous tool for not only games but also interactive entertainment, simulations, etc. And there are many good tutorials on tips and tricks to optimize performance and code readability. I compiled this list as a memo to myself when starting out on Unity, so I can always refer back to the basics and hoping that this can help someone else too. Object pooling. Object pooling is a pretty cool trick and it improves performance because you can reduce the number of Initiate() and Destroy() calls. To illustrate why Destroy can be bad, I attached a screenshot of a project I was building with Unity. I'm not going to go into the details of Object Pooling because there are already many good tutorials out there. One of them being this one.  One thing to always keep in mind is that it is easy to get MissingReferenceException if you accidentally Destroy() the pooled objects. It happened to me once where I attached one script to multiple prefabs which behave the same except that so...
Post a Comment
Read more