Azure-docs: Managed Service Identity

[tvperez76]

Is it possible to use MSI from within a Windows container hosted on Service Fabric? Is so, can you please provide the steps necessary to activate this feature.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: d7df6f76-be53-ff44-5d81-aa1c2c2a1c51
• Version Independent ID: 9e3b3112-bcd1-cee5-48ed-69e3debbd5a3
• Content: Create an Azure Service Fabric container application
• Content Source: articles/service-fabric/service-fabric-get-started-containers.md
• Service: service-fabric
• GitHub Login: @rwike77
• Microsoft Alias: ryanwi

[BryanTrach-MSFT]

@tvperez76 Thanks for the feedback! We are currently investigating and will update you shortly.

[mimckitt]

@tvperez76 based on the following Microsoft Doc Service Fabric is not listed as supported:

https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/overview

@srrengar would you be able to confirm on this?

[srrengar]

I believe so but I'm adding @dkkapur to be sure

Additionally, Service Fabric runs on VMSS, and you should be able to add MSI to an existing VMSS according to this https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/qs-configure-portal-windows-vmss#enable-msi-on-an-existing-azure-vmms

[dkkapur]

Pretty sure @srrengar is right!

[mimckitt]

Thanks @srrengar and @dkkapur! Good point that since Service Fabric is running on VMSS then it should work no problem.

@tvperez76 you should be able to use MSI on service fabric. Suggest following the steps here to enable it.

I will close this out but if you feel more discussion is needed just let me know and we can reopen and continue.

[tvperez76]

Great. However, that only answers half of my question. Is it possible to get MSI working on Containers hosted on Service Fabric? If so, what are the steps necessary activating this feature? Documentation would really help early adopters. Thank you.

[mimckitt]

@tvperez76 yes the logic behind getting it to work is the same as used in VMSS. Hence, that doc is the reference we were able to provide. We do not have any specific documentation specifically to service fabric at this point in time.

So all you need to do after creating your SF cluster is navigate to Virtual Machine ScaleSets in the portal, select the scaleset that was created when you made your SF and under configuration you will see you can enable MSI:

[dkkapur]

@tvperez76 Ah yes - sorry I missed that part of the original question. Let us get back to you on enabling this inside Windows Containers!

My hunch is that it should be possible, since if the OS / other processes in your container image support MSI (which I assume is the case), we as Service Fabric should be agnostic to it and it should work.

Tagging @srrengar and @RajeetN to follow up here.

[mimckitt]

Thanks @dkkapur. I have assigned @srrengar and @RajeetN along with myself to this issue so they can easily track it.

[jeeshnair]

Hi,

I am wondering if there is any follow up to this issue because I am also trying to solve the same problem.

[mimckitt]

Thanks for reaching out @jeeshnair I am working offline to get some traction on this issue. Will update you shortly.

[mimckitt]

@jeeshnair the general consensus is if MSI works on Windows Containers than it should work in Service Fabric. As far as installing MSI on a windows container I don't believe it is any different than the normal install. Accessing the container and installing the package.

[jeeshnair]

@MicahMcKittrick-MSFT , Thanks for swift response. Few of my team members have noted that VM extensions or IMDS end points are not accessible from containers. They have implemented a MSIAccess proxy to workaround. What you are saying is that MSI should be accessible from containers normally and nothing special is required. Correct?

[jeeshnair]

Just to be clear MSI = Managed Service Identity

[mimckitt]

@jeeshnair based on my knowledge I am not aware of any additional setup. However I am not finding any actual documentation on running MSI in containers so it would take some testing to 100% confirm

[jeeshnair]

How soon can you provide results of your testing? I am also doing some testing in parallel and can update you on what I find.

[mimckitt]

@jeeshnair in all honesty I am not not overly familiar with MSI so I am not sure I would be the best person to fully test the scenario. In all reality I would suggest you reach out to the developer community on Stack Overflow to see if anyone else has accomplished this on Service Fabric. It is a bit out of scope for our documentation comments.

We have CC'd @srrengar and @dkkapur on this so they are aware of the ask and the content surrounding this. However at this time there has not been a large ask to use MSI with service fabric so as far as drafting content it is not something that would be coming shortly.

If you like you can post this ask on Stack Overflow and share the link here. That way we can track it and help where we can. As far as specific action on this document there are not actual issues and adding information MSI to this specific doc is not something we will be doing at this time. Hence I will close out this issue as no action is required on the doc at this time.

[jeeshnair]

Thanks @MicahMcKittrick-MSFT . MSI with service fabric is a key scenario for same reasons MSI was introduced. So I am surprised that you feel it is not a large ask. If containers had to access key vault without storing secrets then MSI is the only way out.

By the way I tested and it seems like call the IMDS end point failed with timeout. So clearly something is a miss here. All I want to know is whether MSI is supported or not and if not , when can we expect the full support.
["An error occurred while sending the request. The operation timed out \n\nAn error occurred while sending the request."]

[RajeetN]

@jeeshnair, to clarify what @MicahMcKittrick-MSFT meant this is not an issue with Service Fabric itself. Is the IMDS endpoint address add to routes inside container so the address is routable?

[mimckitt]

thanks for clarifying that @RajeetN.

I did some extra reserach and it seems it is possible:

https://stackoverflow.com/questions/40345712/install-an-msi-into-azure-service-fabric-node
https://social.msdn.microsoft.com/Forums/en-US/8a6d9437-4fd1-410f-9b43-46741c85142b/possible-to-install-msi-packages-in-service-fabric

So yes it should work but not sure it is consider "supportable" at this point in time.

For full product integration I would also suggest adding it to
UserVoice

@RajeetN can confirm on this and we of course can reopen and Rajeet can add it to the backlog

[jeeshnair]

The stack overflow thread talks about MSI installers not managed service
identity . Totally different things and unrelated .

On Thu, Jun 7, 2018 at 6:22 PM Micah [e-mail] wrote:

thanks for clarifying that @RajeetN https://github.com/RajeetN.

I did some extra reserach and it seems it is possible:

https://stackoverflow.com/questions/40345712/install-an-msi-into-azure-service-fabric-node

https://social.msdn.microsoft.com/Forums/en-US/8a6d9437-4fd1-410f-9b43-46741c85142b/possible-to-install-msi-packages-in-service-fabric

So yes it should work but not sure it is consider "supportable" at this
point in time.

For full product integration I would also suggest adding it to
UserVoice https://feedback.azure.com/forums/34192--general-feedback

@RajeetN https://github.com/RajeetN can confirm on this and we of
course can reopen and Rajeet can add it to the backlog

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/6500#issuecomment-395616001,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AkbrcCy8PmIY9kydT40kXmDobISN_k-_ks5t6dHfgaJpZM4TBMMa
.

[jeeshnair]

@RajeetN , How do I do the below , are there any instructions/documentation you can point me to?
"Is the IMDS endpoint address add to routes inside container so the address is routable?"

[RajeetN]

What is the IMDS endpoint address?

[jeeshnair]

I was under impression that 69.254.169.254 should be callable from container.

[jeeshnair]

Here you go. That is a non routable ip on the VM and I can confirm that I can hit this end point and get an access token from VM. Not from the container
http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/

[RajeetN]

Exactly, that is what I meant. the 169.* is not routable. Do "route add 169.254.169.0/24 172.22.160.1" inside the container to make the address routable.

[jeeshnair]

I can try and update you on what I find

[mimckitt]

Thanks @jeeshnair. As an FYI I am working offline with the Service Fabric team and the Containers team to see if we can get it all figured out. From there I would like to get some documentation created around the subject. It might take a few days to fully sort out but wanted to let you know I am actively looking into it :)

[mimckitt]

@jeeshnair I worked with the Windows Containers team and MSI team offline and have confirmed it is possible to run MSI from within a Windows Container. The last suggestion from @RajeetN appears to be correct on what you need to do in order to get it to work.

In addition, the containers team is working now to create the content for running MSI in Containers. Seeing the number requests coming in they have upped the priority for creation time. I do not have an exact ETA on when it is available but do know that they are working on that content now and we do plan an making it publicly available.

Let me know if there is anything else w can help with.

[jeeshnair]

Thanks Micah . I am still working on trying the route addition . I will
report this week what I find , can we keep the issue open until then ?
Thanks

On Sun, Jun 10, 2018 at 8:57 AM Micah [e-mail] wrote:

@jeeshnair https://github.com/jeeshnair I worked with the Windows
Containers team and MSI team offline and have confirmed it is possible to
run MSI from within a Windows Container. The last suggestion from @RajeetN
https://github.com/RajeetN appears to be correct on what you need to do
in order to get it to work.

In addition, the containers team is working now to create the content for
running MSI in Containers. Seeing the number requests coming in they have
upped the priority for creation time. I do not have an exact ETA on when it
is available but do know that they are working on that content now and we
do plan an making it publicly available.

Let me know if there is anything else w can help with.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/6500#issuecomment-396060045,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AkbrcPT_rVUxJ9mKOFpd5oVuhgw6Vytwks5t7UIBgaJpZM4TBMMa
.

[mimckitt]

Can do :)

[mimckitt]

Oops accidently close. Sorry :)

[BuddhaBuddy1]

I would like to request that the documentation you create also addresses local, non-Azure-hosted Service Fabric clusters.

[mimckitt]

Thanks @BuddhaBuddy1 I will forward that information.

@jeeshnair any update?

[jeeshnair]

@MicahMcKittrick-MSFT , route addition works. However using AzureServiceProviderClass out of the box does not work from container and fails with below exception. I am also curious to know the local dev box story.

"Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried the following 3 methods to get an access token, but none of them worked.\r\nParameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.\r\nParameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at \"C:\Users\ContainerAdministrator\AppData\Local\.IdentityService\AzureServiceAuth\tokenprovider.json\"\r\nParameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n\r\n \n\nCertificates could not be retrieved \n\n at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.d__14.MoveNext()\r\n--- End of stack trace from

[jeeshnair]

I think the Azureserviceprovider requires some environment variables to be set which gets provisioned in VM and not on container.

[mimckitt]

Thanks for the update @jeeshnair. I am seeing if anyone on my end has any suggestions.

[jeeshnair]

There is one more problem to be solved for this to work seamlessly. It seems like the AzureServiceProvider class uses MSI_ENDPOINT and MSI_SECRET environment variable to get MSI token. I can set MSI_ENDPOINT , what goes into secret? Right now Azureserviceprovider fails to get MSI token. I can explicitly get it by making a http request.

Microsoft.Azure.KeyVault.KeyVaultClient
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://lo
gin.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried the following 3 methods to get an access
token, but none of them worked.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://lo
gin.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Managed Service Identi
ty. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure reso
urce that has MSI setup.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://lo
gin.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Visual Studio. Access
token could not be acquired. Visual Studio Token provider file not found at "C:\Users\ContainerAdministrator\AppData\Loc
al.IdentityService\AzureServiceAuth\tokenprovider.json"
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://lo
gin.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried to get token using Azure CLI. Access toke
n could not be acquired. 'az' is not recognized as an internal or external command,
operable program or batch file.

[mimckitt]

thanks @jeeshnair. I forwarded this information internally and am seeing if I can get a suggestion for you.

[mimckitt]

@tvperez76 couple of things I heard back:

if you are using SF on a local box then MSI is not expected to work. It is only available for resources running Azure. I am not sure this is applicable in your case but it was some feedback I got.

Where is AzureServiceProvidor class coming from? I am not finding much online refernecing this. It is presumed this class is not working as it is looking for items within the cluster itself.

[varunsh-msft]

You can find documentation about AzureServiceTokenProvider here. This class is in the Microsoft.Azure.Services.AppAuthentication Nuget.

This is how you can use AzureServiceTokenProvider on local box, where MSI is not available:
https://stackoverflow.com/questions/49859477/access-key-vault-from-local-service-fabric-cluster-with-msi

W.r.t use of IMDS, please ensure you are using latest version 1.0.3 of Microsoft.Azure.Services.AppAuthentication Nuget. That version uses IMDS. The earlier versions using the VM extension.

[mimckitt]

@jeeshnair any update on this?

[mimckitt]

@jeeshnair
I will close this out for now. If you need additional help please let me know and we can reopen and continue.

[AlbertoDePena]

@MicahMcKittrick-MSFT

I was trying something similar to this.... I was able to poc doing some CRUD operations from python app that requests files from azure storage using MSI (works great!). I need to prove out the same but having that python app within docker container. I tried @RajeetN approach for the ip routing but no luck. I always get the error about trying to connect to 169.254.169.254. We have lost of component that need to talk to azure storage / keyvalut / azure sql. We would love to use MSI but most of those components are containerized (docker).

Any help with this would be greatly appreciated =)

[KaiWalter]

I engaged with the product group again yesterday because the approach with the route add was not working for me.

[rhummelmose]

Why is this issue closes? It is clearly not working.

[KaiWalter]

@rhummelmose - Many build VM minutes died to bring us this information: check my StackOverflow How To post
I guess the key is the dynamic determination of the gateway address inside the container.

[chanes-cds]

For .NET core based images (e.g. microsoft/dotnet:2.1-aspnetcore-runtime), the route add workaround does not work due to insufficient permissions (route add yields "The requested operation requires elevation."). Perhaps I am missing something - has anyone had luck hitting the 169.254 endpoint with .NET core images? I have been testing with Docker version 18.09.0, build 33a45cd0a2.

[chanes-cds]

I figured it out and was able to add a permanent route in my Dockerfile with the following:
USER ContainerAdministrator RUN route -p add 169.254.169.0 mask 255.255.255.0 0.0.0.0 USER ContainerUser ENTRYPOINT ["dotnet", "myapp.dll"]

[KaiWalter]

With your approach I also can add the route but the request to the metadata endpoint did not work.
I figured out that USER ContainerAdministrator right before the CMD will help the entry-script to run elevated.

Dockerfile

# escape=`

# --------------------------------------------------------------------------------
# PowerShell
FROM mcr.microsoft.com/powershell:nanoserver as ps

# --------------------------------------------------------------------------------
# Runtime image
FROM microsoft/dotnet:2.1.6-aspnetcore-runtime-nanoserver-1803

COPY --from=ps ["C:\\Program Files\\PowerShell", "C:\\PowerShell"]

ADD entry.PS1 C:\\entry.PS1

USER ContainerAdministrator
CMD ["C:\\PowerShell\\pwsh.exe","C:\\entry.PS1"]

entry.PS1

Write-Host "adding route for Managed Service Identity"
$gateway = (route print | ?{$_ -like "*0.0.0.0*0.0.0.0*"} | %{$_ -split " "} | ?{$_.trim() -ne "" } | ?{$_ -ne "0.0.0.0" })[0]
$arguments = 'add','169.254.169.0','mask','255.255.255.0',$gateway
&'route' $arguments

$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net%2F' -Method GET -Headers @{Metadata="true"} -UseBasicParsing
Write-Host "MSI StatusCode :" $response.StatusCode

[chanes-cds]

Thanks for trying it out. You are right about my route not working. Turns out it was another change I made that was causing things to work. When I add a persistent route inside the container, that persistent route is surprisingly added to the host routing table and then becomes available to both existing and subsequently created containers such that they are suddenly able to reach the IMDS endpoint.
See 20190104-demo-sanitized.txt

[TheReaLee]

Just ran into this issue... what is the workaround? or has it not been solved yet?

[KaiWalter]

@TheReaLee which of the approaches above ( @chanes-cds or me ) did you try and how does the problem manifest on your end?

[TheReaLee]

I tried @chanes-cds suggestion:

USER ContainerAdministrator
RUN route -p add 169.254.169.0 mask 255.255.255.0 0.0.0.0
USER ContainerUser
ENTRYPOINT ["dotnet", "myapp.dll"]```

[KaiWalter]

@TheReaLee I'm running my approach (see my post from Jan 4 above) successfully for months. Imho the route needs to be placed on container startup / entry and not on container build.