Mixedrealitytoolkit-unity: The Configure menu lacks a way to configure HoloLens Capabilities

[jbienzms]

I'd like to start a conversation about which capabilities should be applied automatically when developers choose HoloToolkit -> Configure -> Apply HoloLens Project Settings in Unity.

For example, I'm seeing a lot of users getting tripped up by the fact that Spatial Perception is not checked for them automatically. They get exported all the way out to the device and then they don't see a wireframe or holograms fall through the ground and they don't understand why. I've had this happen myself and I've lost track of how many new projects I've created by now.

I for one feel that at the very least Spatial Perception should be checked when Project Settings are applied. But I also feel we might want to consider turning on the following capabilities as well:

1. WebCam
2. Microphone
3. PicturesLibrary
4. VideosLibrary

It's my understanding that all of those are required in order to be able to capture mixed reality photos and videos. (MusicLibrary was also checked in a lot of the samples but never explained why. It may or may not be needed.)

I'd almost argue that InternetClient should be checked too, but that one's more debatable. Not every project will need it but I would say most will. I think this one is up for debate.

Any thoughts on which capabilities (if any) we should enable when the users applies scene settings?

Thanks!

[riverar]

Spatial Perception is automagically checked but there seems to be problems when a SLN exists on disk prior to this setting being changed. Subsequent builds don't seem to propagate this through to the appx manifest. I sunk a few hours troubleshooting this.

Repro:

• Start a new project, build a D3D project on disk
• Add Spatial Perception ability in Unity
• Rebuild D3D project, note appx manifest is missing spatial perception feature

[jwittner]

For reference, my original goal for this menu option was to align the project and scene to the settings specified here:

https://developer.microsoft.com/en-us/windows/holographic/unity_development_overview#Configuring_a_Unity_project_for_HoloLens

This gets you setup with the barest of bones for HoloLens development and my preference would be that we keep this page and the default behavior of these menu options aligned. However, I can definitely see a place for a new option under Configure that perhaps prompts a window asking the user what types of HoloLens features they want and then applying the changes as appropriate.

[jbienzms]

Thank you for being part of the conversation @jwittner. I looked at the link you posted above and I understand what you based your design on. I do think that makes sense. Having said that, if you look at the bottom of that page you linked to you'll see a link to a page titled "Recommended settings for Unity". That page states

For an app to take advantage of certain functionality on HoloLens, it must declare certain capabilities in its manifest. The manifest declarations can be made in Unity so they are included in every subsequent project export. The setting are found in Player Settings > Windows Store > Publishing Settings > Capabilities. The applicable capabilities for enabling the commonly used APIs for Holographic apps are:

• WebCam
• SpatialPerception
• Microphone
• picturesLibrary / videosLibrary / musicLibrary

If you feel strongly that there should be a 3rd menu option separate from "Apply HoloLens Project Settings" I can get behind that. I would just sincerely like to see it added.

What do you think it would be called? Maybe "Apply HoloLens Capabilities"?

[jwittner]

I like "Apply HoloLens Capabilities"! Maybe we can rename this issue to state something like the following, "The Configure menu lacks a way to configure HoloLens Capabilities"

The window could offer checkboxes for the APIs listed at the link you provided and apply the necessary Capabilities for what apis are checked.

[jbienzms]

Renamed!

[jwittner]

Want to assign yourself and take this task up?

[NeerajW]

One minor comment, when we say HoloLens capabilities, it means to me things like Spatial Perception. The other capabilities @jbienzms quoted like Pictures, Video which are AppData like capabilities are generic ones. InternetClientServer is a good one for profiling with Unity as well. I am all for adding this.
Perhaps named Apply Application Capabilities or something akin.

[jbienzms]

other capabilities @jbienzms quoted like Pictures, Video which are AppData like capabilities are generic ones

That's not entirely accurate. Be sure to review the Recommended settings for Unity page. picturesLibrary / videosLibrary / musicLibrary are all required to support PhotoCapture and VideoCapture (for still photos, video, and audio respectively). Without them, using the hardware button shortcuts or saying things like "Hey Cortana, take a photo" don't work.

Want to assign yourself and take this task up?

Yes @jwittner, I think it's a good idea to assign this to myself. It will force me to learn how to write extensions to the Unity editor - something I've been putting off for a long time. It just may take a while, especially since I'm out all next week for the USDX offsite. If you have any pointers on how to show checkboxes or any good editor extension tutorials in general I'd love links.

[NeerajW]

@jbienzms even that page does not claim them to be HoloLens capabilities :)

[jbienzms]

Sorry if I wasn't clear. I don't mean they're HoloLens-specific. I just mean that core HoloLens features that users expect to work in every app will not function if they are not enabled.

[jwittner]

I was imagining the options we provided were from the APIs column from the Capabilities section at @jbienzms link, and the code would map those to the appropriate WSA Capabilities in Unity.

[keveleigh]

picturesLibrary / videosLibrary / musicLibrary are all required to support PhotoCapture and VideoCapture (for still photos, video, and audio respectively). Without them, using the hardware button shortcuts or saying things like "Hey Cortana, take a photo" don't work.

@jbienzms Is that true? I'm out without my HoloLens and unable to verify at the moment, but I thought I'd used "Hey Cortana, take a photo" in my app without these capabilities before (I have images in my OneDrive from apps I've written and have never set these capabilities). It's very possible I'm misremembering though. I also wonder if "Select" works without the microphone capability, as these are all system-level actions and not app-level.

[jbienzms]

I'm 90% sure you can't do those things without these caps. Or at least I'm 90% sure I had problems without them before Anniversary Update. I can try to prove that later this week, or if you have time please try it out. It would be good to know for sure once and for all.

As for Select, I seem to remember that was one you couldn't opt out of or override. I was under the impression it was a OS-level thing. But it would be good to test and verify that too.

First one to test and verify wins! (Uh, my admiration. :))

[lalo13]

Hi everyone, I did a test to check it out and it is possible to take a photo from the App by saying "Hey Cortana, take a photo", or mixed reality recording video without having enable the capabilities,

(Web Cam,Spatial Perception, Microphone, Pictures Library, Videos Library)

but if you want to use speech commands in your App (something like "Select") you must enable the Microphone capability. This makes sense for me because the App is not listening the microphone but Cortana it is a separate App that is running and listening in the background.

Also you must enable the Spatial Perception capability manually if you need that you App use the spatial mapping.

I hope this is helpful.

[lalo13]

If you decide to add this option on the HoloToolKit menu this script could help you.

using UnityEditor;
using UnityEngine;

[InitializeOnLoad]
public class CheckmarkMenuItem
{
private const string MENU_MICROPHONE = "HoloToolkit/Configure/Capabilities/Microphone";
private const string MENU_SPATIALPERCEPTION = "HoloToolkit/Configure/Capabilities/Spatial Perception";

/// Called on load thanks to the InitializeOnLoad attribute
static CheckmarkMenuItem()
{       
    EditorApplication.update += () =>
    {
        Menu.SetChecked(MENU_MICROPHONE, PlayerSettings.WSA.GetCapability(PlayerSettings.WSACapability.Microphone));
        Menu.SetChecked(MENU_SPATIALPERCEPTION, PlayerSettings.WSA.GetCapability(PlayerSettings.WSACapability.SpatialPerception));
    };
}

[MenuItem(CheckmarkMenuItem.MENU_MICROPHONE)]
private static void MicrophoneToggleAction()
{        
    switchCapability(PlayerSettings.WSACapability.Microphone);
}

[MenuItem(CheckmarkMenuItem.MENU_SPATIALPERCEPTION)]
private static void SpatialPerceptionToggleAction()
{
    switchCapability(PlayerSettings.WSACapability.SpatialPerception);
}

private static void switchCapability(PlayerSettings.WSACapability mCap)
{
    PlayerSettings.WSA.SetCapability(mCap, !PlayerSettings.WSA.GetCapability(mCap));
}

}

[StephenHodgson]

@jbienzms If this is merged is this issue closed?

[jbienzms]

Yes, we can close the issue. I still want to improve the UI but the main issue itself has been resolved.

I'll close it now.

[Dheerajmentor]

is there any way to set user information capability on hololens developing using Unity ?