IoT+Hololens = Awesomeness

IoT+Hololens = Awesomeness

A couple weeks ago, after Yondu’s Arrow project, I was thinking about my next project. There were few ideas running through my mind, though I couldn’t decide on one. It was around that time I had a discussion with Myke – another friend\colleague\mentor and also my manager – on an IoT project of his. It brought me to the idea of controlling my real world entities from my virtual world. So this article is about how to do just that with hololens and a raspberry Pi using Azure IoT hub.

One more reason to choose this topic is, I have seen few Iot projects on hololens in YouTube before. Although, they are mostly app demos with no tutorial or know-how. I believe this could help people who are stuck or trying to get started on IoT using Hololens. Fair warning – it is a bit tricky. We are going to do some tweaking along the way. I had to do few trial and error on it before I nabbed it. Alright, let’s get down to business.

Final Implementation – for the lazy ones like me 😉

Prerequisites (used for this article)

  • An Azure account with an IoT Hub in it. Here is how to do it (it is free of cost).
  • A device added to the IoT hub. (Note the name of the device added since you need to use it in the code to send messages to the device)
  • Visual Studio 2017
  • Unity 2017.1.2p1
  • Raspberry Pi 3 Model B
  • Sunfounder Dual-Color LED

Creating the Hololens application

Obtain the Azure libraries
  • Create a temporary folder somewhere in your system.
  • Use nuget.exe to download the “Microsoft. Azure. Devices” (version as of this writing is 1.4.1) package to the folder.

nuget package

  • Now get the dlls from each folder (from uap sub folder or portable-net45 subfolder if uap is not available) to a common folder – except Newtonsoft.Json. I’ll explain the reason later. To make it easy for you I have included the dlls in my git here.
  • This is the final list.

nuget files

Set up the project
  • Start a new Unity 3D project and import the MixedRealityToolkit-Unity to it.
  • Configure the project for Hololens. Make sure to enable the Private Network Client Server and Internet Client Server Capabilities

 

UWP settings

  • Import the following prefabs from the toolkit
    • HololensCamera
    • InputManager
    • CursorWithFeedback
  • Save the scene. It would look like this when you play.

hololens scene

  • Go to player settings from File->Build Settings->Player Settings
  • Select Other settings section in it. Change the API Compatibility Level to .NET 4.6
  • Now change Scripting Runtime Version to Experimental (.NET 4.6 Equivalent). At this point Unity will ask for a restart to apply the changes. Click Restart.

dotnet settings

  • Once the project is restarted, create a folder called Plugin in the project.
  • Now copy all the azure library and its dependency files we created in the previous section to this folder. Wait for Unity to generate the meta files for them
  • At this point you might see few errors in your Unity project. Ignore it for now.
  • Go to Plugin folder in Unity and click one of the dll in it. This is how the settings would be in the inspector panel.

iot before

  • Change it to look like this. Make sure the Don’t Process option is checked.

iot after

  • Do the same to all the dlls in the Plugin folder.
Complete the Hololens App
  • Lets do the real work. Create a cube into the scene and add a c# script to it called IoTController.cs.
  • Now edit the code in Visual Studio and add this snippet to it
using HoloToolkit.Unity.InputModule;
using System.Text;
using UnityEngine;

#if NETFX_CORE
using System.Threading.Tasks;
using Microsoft.Azure.Devices;
#endif

public class IoTController : MonoBehaviour, IInputClickHandler
{
#if NETFX_CORE
    ServiceClient service;
#endif
    static string connectionUri = "<IoT Hub Service ConnectionString>";

    // Use this for initialization
    void Start()
    {
#if NETFX_CORE
        service = ServiceClient.CreateFromConnectionString(connectionUri, TransportType.Amqp);
#endif
    }

    public void OnInputClicked(InputClickedEventData eventData)
    {
#if NETFX_CORE
        try
        {
            SendCommand("red");
        }
        catch (System.Exception ex)
        {
            Debug.Log("Failed to send message" + ex.Message);
        }
#else
        Debug.Log("Tapped");
#endif
    }

#if NETFX_CORE
    public void SendCommand(string command)
    {
        byte[] bytes = Encoding.ASCII.GetBytes(command);
        Message message = new Message(bytes);
        service.SendAsync("<Device Id of the device you created in the IoT hub>", message);
    }
#endif
}
  • Save everything and build the project.
  • When you try to build the project you’ll get and error which practically means “interface ICloneable exists in both dlls Microsoft.Azure.Amqp and WinRTLegacy”
  • Now, the reason is that in the Amqp dll that we added to the plugin they have defined the interface under System namespace manually. Hence the collision. There are two ways to solve this
    • The hard and correct way – use extern alias and modify the mcp.rcs file to add the reference
    • The easy and not so correct way – Simply comment out the ICloneable interface from “SpatialUnderstandingDllObjectPlacement.cs” inside HoloToolkit scripts.
  • For this article I am taking the second route since the other process is a bit lengthy, and it did not break anything on my testing with Spatial Mapping included. Here is the change

hololens iot build

  • Now save it and build again. The build would succeed and generate a UWP app project.
  • Open the project in Visual Studio. Change the Architecture from ARM to x86. Select the Hololens emulator and press F5 or click Debug button.
  • When the emulator is loaded and the app is deployed, try tapping on the cube.
  • Go to the Azure portal and select the IoT Hub that you configured previously. In the overview section you can see the message count increasing every time you tap the cube, like this

iot portal

Yaaayy!! Now we can send commands\messages from our hololens to our IoT hub. That completes our first segment. Now let’s do the second segment where we create our IoT app for the Raspberry Pi and execute the commands sent from Hololens. Now, I have expanded this application a little bit and posted the final code here in order to give a full demonstration of the connected application. The final app demonstration video is posted here.

Note: In the hololens app code you need to change the connection string placeholder with your hub’s connection string and also pass the device name in the SendCommand function.

Creating a IoT app for Raspberry Pi 3

For our case, I have created a simple dual-LED application on raspberry Pi. I am not going to describe this process in detail. There are tons of tutorials out there on creating a UWP app for Windows IoT Core and receiving cloud to device messages from Azure IoT Hub. Moreover, you can download the app I created for the article from my git here.

Schematic details for the IoT application in Raspberry Pi
  • Dual-Color LED module diagram

  • Schematic diagram for the Dual Color LED in Raspberry PI 3

  • Connect your Pi based on the schema provided and deploy the IoT app into the Pi.
  • Start the IoT app. Start the hololens application in the emulator. Tap on the red or blue buttons and watch your LED in the real world change color.

Note: In the IoT app code you need to change the connection string placeholder with your device’s connection string registered with the hub.

Here is a snap

hololens iot

Well, that’s it!!! If you have any questions feel free to post it in the comments section and I’ll try to answer.

Mistakes I made – so that you don’t repeat 🙂

Here are few things that I have tried and failed before I reached the aforementioned solution.

  • Tried creating a .Net Standard library for the Azure communication – failed since Unity doesn’t support .Net Standard yet according to Unity Team update.
  • Tried a UWP library for the aforementioned point – failed since Unity couldn’t create the project solution due to nuget package resolution for the library. Though, I am yet to do some more tinkering with this to see if I can make it work. Like, enabling “Don’t Process” option for the library.

 

P.S. This article goes to Myke for that thought provoking discussion.

6 Replies to “IoT+Hololens = Awesomeness”

  1. Hello, you have already implemented something that communicates directly with the hololens. I’m trying to make an ESP32 card send a string to hololens via bluetooth as I need real-time response.

  2. I haven’t seen your source code yet. Can you explain in simple terms how you got the HoloLens to talk with Raspberry Pi using Azure IoT Hub? As far as I know, device to device communication is not allowed directly in Azure IoT Hub. Thanks in advance.

  3. Hi! I want to deploy an IoT app on Hololens Emulator. This Unity app is made using Unity and Azure IoT Hub SDK. The big picture is to create a telemetry application using Raspberry Pi, Unity and Azure Cloud. I want to receive sensors data on my Unity app through Azure Cloud, and specially Azure IoT Hub the dedicated service for IoT system.

    Sensors + raspberry pi => Azure Cloud => Unity app.

    According to the Azure SDK, the IoT app needs a WiFi connection.

    But the Hololens emulator is running using Ethernet connection. To deploy the build project on the emulator, it seems that the Ethernet connection is required. When I used WiFi, and error appears during the deployment.

    After some tests, I conclude that my app can access to internet on the emulator, but the way using Ethernet or Wifi is the issue.

    Test 0: Deploy the Unity app on Unity editor using Ethernet connection Internet on the app: Yes Connection to Azure Cloud: No

    Test 1: Deploy the Unity app on Unity editor using Wifi connection Internet on the app: Yes Connection to Azure Cloud: Yes

    Test 2: Deploy the Unity app on Hololens emulator using Visual Studio and Ethernet connection. Deployment: success Internet on Hololens: Yes Internet on the app: Yes Connection to Azure Cloud: No

    Test 3: Deploy the Unity app on Hololens emulator using Visual Studio and Wifi connection. Deployment: failed

    Does the emulator support wifi connection? Once the app is deployed on the emulator, it is possible to configure wifi?

    Thanks for helping!

Leave a Reply

Your email address will not be published. Required fields are marked *