Azure-functions-host: Error when calling Invoke-AzStorageSyncChangedDetection

[thiagolunardi]

I have a Timer Triggerd powershell function which just do a single command call

Invoke-AzStorageSyncChangeDetection `
 -ResourceGroupName $resourceGroupName `
 -StorageSyncServiceName $storageSyncServiceName `
 -SyncGroupName $syncGroupName `
 -CloudEndpointName $cloudEndpointName `
 -DirectoryPath $directoryPath

The function has a User Assigned as Identity set. The only error we have is the following output:

Result: Failure
Exception: Value cannot be null.
Parameter name: value
Stack:    at Google.Protobuf.ProtoPreconditions.CheckNotNull[T](T value, String name)
   at Microsoft.Azure.WebJobs.Script.Grpc.Messages.StreamingMessage.set_RequestId(String value) in C:\projects\azure-functions-powershell-worker\src\Messaging\protobuf\FunctionRpc.cs:line 332
   at Microsoft.Azure.Functions.PowerShellWorker.Utility.RpcLogger.Log(Boolean isUserOnlyLog, Level logLevel, String message, Exception exception) in C:\projects\azure-functions-powershell-worker\src\Logging\RpcLogger.cs:line 45
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeProfile(String profilePath) in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManager.cs:line 191
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.Initialize() in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManager.cs:line 111
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManagerPool.CheckoutIdleWorker(StreamingMessage request, AzFunctionInfo functionInfo) in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManagerPool.cs:line 93
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessInvocationRequest(StreamingMessage request) in C:\projects\azure-functions-powershell-worker\src\RequestProcessor.cs:line 257

• Timestamp: 2020-03-16 14:17:19.323
• Function App version: ~3
• Function App name: sync-order-gh-app
• Function name(s) (as appropriate): "AzStorageSyncChangeDetection"
• Invocation ID: Id=a26cd7eb-7051-4e2c-be4c-0cdfb3dbe0a6
• Region: WE-Europe

[ankitkumarr]

@AnatoliB, From the stack trace, does this seem that Powershell worker did not set RequestID or something like that, when making the RPC call?

I could be completely off. :)
Do you know what this stack trace means?

[AnatoliB]

Known issue: https://github.com/Azure/azure-functions-powershell-worker/issues/337
Fixed, will be deployed soon, the workaround is to temporarily move the code from profile.ps1 to run.ps1.

[thiagolunardi]

Hello, @AnatoliB

My function app has no profile.ps1 generated and the error occurs even when no code inside run.ps1 body but param($Timer).

2020-03-19T08:08:15.049 [Information] Executing 'Functions.AzStorageSyncChangeDetection' (Reason='This function was programmatically called via the host APIs.', Id=93ecfbfd-46dd-43f7-8dbc-f713a86540bd)
2020-03-19T08:08:37.358 [Error] Executed 'Functions.AzStorageSyncChangeDetection' (Failed, Id=93ecfbfd-46dd-43f7-8dbc-f713a86540bd)
Result: Failure
Exception: Value cannot be null.
Parameter name: value
Stack:    at Google.Protobuf.ProtoPreconditions.CheckNotNull[T](T value, String name)
   at Microsoft.Azure.WebJobs.Script.Grpc.Messages.StreamingMessage.set_RequestId(String value) in C:\projects\azure-functions-powershell-worker\src\Messaging\protobuf\FunctionRpc.cs:line 332
   at Microsoft.Azure.Functions.PowerShellWorker.Utility.RpcLogger.Log(Boolean isUserOnlyLog, Level logLevel, String message, Exception exception) in C:\projects\azure-functions-powershell-worker\src\Logging\RpcLogger.cs:line 45
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.InvokeProfile(String profilePath) in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManager.cs:line 191
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManager.Initialize() in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManager.cs:line 111
   at Microsoft.Azure.Functions.PowerShellWorker.PowerShell.PowerShellManagerPool.CheckoutIdleWorker(StreamingMessage request, AzFunctionInfo functionInfo) in C:\projects\azure-functions-powershell-worker\src\PowerShell\PowerShellManagerPool.cs:line 93
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessInvocationRequest(StreamingMessage request) in C:\projects\azure-functions-powershell-worker\src\RequestProcessor.cs:line 257

I tried to add the following line to my run.ps1, but I still get the same reported error.

if ($env:MSI_SECRET -and (Get-Module -ListAvailable Az.Accounts)) {
    Connect-AzAccount -Identity
}

[AnatoliB]

@thiagolunardi I don't know how you would get this call stack without having a profile.ps1. Are you sure your app does not have it? Please note that profile.ps1 is located one level above the function (in the same folder with host.json), as it applies to the entire function app. It is always generated for new PowerShell apps. You can delete it, of course. But, if you haven't delete it, it should be there. Please double check. If you confirm there is no profile.ps1, we'll need a deeper investigation, as I don't have a reasonable explanation for this behavior.

[thiagolunardi]

@AnatoliB you are totally right, the profile.ps1 is there. I have its code commented out and moved the below lines to run.ps1 this like:

if ($env:MSI_SECRET -and (Get-Module -ListAvailable Az.Accounts)) {
    Connect-AzAccount -Identity # line 5
}

Now I'm getting the following message:

Executing 'Functions.AzStorageSyncChangeDetection' (Reason='This function was programmatically called via the host APIs.', Id=51477f65-a593-4ac5-9e6e-c0aa08a6112c)

ERROR: Connect-AzAccount : An attempt was made to access a socket in a way forbidden by its access permissions
At D:\home\site\wwwroot\AzStorageSyncChangeDetection\run.ps1:5 char:5
+     Connect-AzAccount -Identity
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [Connect-AzAccount], HttpRequestException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

Script stack trace:
   at <ScriptBlock>, D:\home\site\wwwroot\AzStorageSyncChangeDetection\run.ps1: line 5

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.Azure.Commands.Common.Authentication.HttpClientWithRetry.SendAsync(HttpRequestMessage request, CancellationToken token)
   at Microsoft.Azure.Commands.Common.Authentication.HttpClientOperationsFactory.HttpClientOperations`1.SafeSendRequestAsync(HttpRequestMessage request, CancellationToken token)
   at Microsoft.Azure.Commands.Common.Authentication.HttpClientOperationsFactory.HttpClientOperations`1.GetAsync(String requestUri, CancellationToken token)
   at Microsoft.Azure.Commands.Common.Authentication.ManagedServiceAccessTokenBase`1.GetOrRenewAuthentication()
   at Microsoft.Azure.Commands.Common.Authentication.ManagedServiceAccessTokenBase`1.get_AccessToken()
   at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.ListAccountTenants(IAzureAccount account, IAzureEnvironment environment, SecureString password, String promptBehavior, Action`1 promptAction)
   at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.Login(IAzureAccount account, IAzureEnvironment environment, String tenantId, String subscriptionId, String subscriptionName, SecureString password, Boolean skipValidation, Action`1 promptAction, String name, Boolean shouldPopulateContextList)
   at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass92_0.<ExecuteCmdlet>b__0(AzureRmProfile localProfile, RMProfileClient profileClient, String name)
   at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass94_0.<SetContextWithOverwritePrompt>b__0(AzureRmProfile prof, RMProfileClient client)
   at Microsoft.Azure.Commands.Profile.Common.AzureContextModificationCmdlet.ModifyContext(Action`2 contextAction)
   at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.SetContextWithOverwritePrompt(Action`3 setContextAction)
   at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()

Inner exception: System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

[AnatoliB]

@thiagolunardi Are you trying to use a user-assigned identity?

[thiagolunardi]

Yes. The Function App has already a User Assigned configured.

Am I missing something?

[AnatoliB]

Using a user-assigned identity requires a bit more work. The easiest thing to do is to find the “Client ID” property of the user-managed identity and pass it to the Connect-AzAccount cmdlet as an AccountId value, for example:

Connect-AzAccount -Identity -AccountId ‘506f5541-5c3b-40bc-93ab-d24878674dc4’

If you don't want to hardcode the ID, here is another option:

1. Enable the system assigned identity as well.
2. Make the system assigned identity a Reader on the user assigned identity.
3. In the profile.ps1 file, replace:

    Connect-AzAccount -Identity

with:

    Connect-AzAccount -Identity
    $identity = Get-AzUserAssignedIdentity -ResourceGroupName … -Name … # retrieve the specific user assigned identity
    Connect-AzAccount -Identity -AccountId $identity.Id

See also: https://github.com/Azure/azure-functions-powershell-worker/issues/235

[thiagolunardi]

Worked like a charm. Thanks for the guidance through this workaround. I will keep a track on this issue til the fix is GA released.

[ankitkumarr]

Closing this as Anatoli has linked the relevant issues in the Powershell worker repository.