Azure-cosmos-dotnet-v3: Issue with where clause when using a client restricted to a partition key

[BenLuts]

We are experiencing an issue in the following scenario.
We have a container with a partitionkey defined. We create users with permissions specific to a single partitionkey. We create a client specific for this user, based on his token. When we query a single item like this:
container.ReadItemAsync<T>(itemId, PartitionKey)
Everything works as expected, the item is correctly retreived.

However when trying to retrieve a filtered list as such:
container.GetItemLinqQueryable<T>(requestOptions: new QueryRequestOptions { PartitionKey = PartitionKey }).Where(query).ToFeedIterator(); var result = new List<T>(); while (iterator.HasMoreResults) result.AddRange(await iterator.ReadNextAsync());
We receive an unauthorized response: 403.

It appears as though the where clause does not respect the QueryRequestOptions.
If we remove .Where(query) from the above code, then the issue does not present itself.

Environment summary
SDK Version: 3.4.1
.Net Core 2.2 running on Azure App Services

[j82w]

Are you running your app as x64?

[j82w]

@bchong95 any suggestions?

[BenLuts]

Our Azure App Service is currently running x86

[j82w]

My theory on why this is failing is the SDK needs to go to the gateway to get the query plan. The call to get the query plan is failing with the 403. @bchong95 can you confirm this and if so what is the fix?

[BenLuts]

Note that the issue did not occur with sdk V2.

[j82w]

@sboshra any suggestions?

[wouterdeman]

Any update on this issue ? Seems like a showstopper for Resource token based security allowing only read and/or write access on a given partition key ?

[bchong95]

Can we get a stack trace for the 403?

[BenLuts]

Please find the stacktrace below.

Microsoft.Azure.Cosmos.CosmosException: Response status code does not indicate success: 403 Substatus: 0 Reason: (Response status code does not indicate success: 403 Substatus: 0 Reason: (Insufficient permissions provided in the authorization header for the corresponding request. Please retry with another authorization header.ActivityId: e937cff4-5074-422a-be97-765c5b1ae05c, Microsoft.Azure.Documents.Common/2.7.0, Windows/10.0.14393 cosmos-netstandard-sdk/3.4.1).). at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode (Microsoft.Azure.Cosmos.Client, Version=3.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponseHelper (Microsoft.Azure.Cosmos.Client, Version=3.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponse (Microsoft.Azure.Cosmos.Client, Version=3.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.Azure.Cosmos.FeedIteratorCore1+d__5.MoveNext (Microsoft.Azure.Cosmos.Client, Version=3.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Clients.DbClient.DbClient+<CreateItemQueryAsync>d__81.MoveNext (DbClient.cs:71) (Infrastructure.Shared, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Repositories.DeclarationRepository+<GetNoneDeletedDeclarationsAsync>d__5.MoveNext (DeclarationRepository.cs:37) (Declarations.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Services.DeclarationService+<g__func|6_0>d1.MoveNext (DeclarationService.cs:39) (Declarations.Business, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Services.Shared.Base.BaseService+d__51.MoveNext (BaseService.cs:61) (Business.Shared, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Api.Controllers.Base.BaseController+d__31.MoveNext (BaseController.cs:60) (Api, Version=9.9.9.9, Culture=neutral, PublicKeyToken=null)

[wouterdeman]

Did the stack trace help out to find where the issue/bug occurs ?

[Xiaoliangzhang]

We are experiencing the same error with SDK V3. We have reverted back to V2 because of this. Please update if there is any solution/workaround.

[ealsur]

@BenLuts @Xiaoliangzhang Are you both using a Where(query)? Can you share which is the actual query?

@Xiaoliangzhang Can you share which is the V2 code you are using?

[BenLuts]

@ealsur I am using the following query:
.Where(x => !x.IsDeleted && x.ParentId == parentId)

For completion I also tried with the following, but got the same error:
.Where(x => !x.IsDeleted && x.ParentId == parentId && x.PartitionKey == partitionKey)

[Xiaoliangzhang]

A little summary of my findings after some investigation:

1. We use a web service as a token broker and a Xamarin form app as the client.
2. We got "Response status code does not indicate success: 403 Substatus: 0 Reason: (Insufficient permissions provided in the authorization header for the corresponding request." error message when we update Both Web services and Client to SDK V3.
3. However, it works when we Only update Client to V3

Here is some code example:

1. This is the WORKING client code in V3:

            CosmosClientOptions cosmosClientOptions = new CosmosClientOptions()
            {
                ConnectionMode = ConnectionMode.Gateway
            };
            var cosmosClient = new CosmosClient(cosmosDbUri, _CosmosToken_, cosmosClientOptions);
            CosmosContainer cosmosContainer = cosmosClient.GetContainer(databaseId, collectionId);

            QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM c WHERE c.DocumentType = 1");

            QueryRequestOptions queryRequestOptions = new QueryRequestOptions()
            {
                PartitionKey = new PartitionKey(_someKey_)
            };

            List<TDocument> results = new List<TDocument>();

            FeedIterator<TDocument> resultSetIterator = cosmosContainer.GetItemQueryIterator<TDocument>(
                queryDefinition: queryDefinition,
                continuationToken: null,
                requestOptions: queryRequestOptions);

            while (resultSetIterator.HasMoreResults)
            {
                FeedResponse<TDocument> response = await resultSetIterator.ReadNextAsync(cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
                results.AddRange(response);
            }

            return results;

1. This is the None working Code for service to generate a token with V3:

            Database database = await GetCosmosDatabaseAsync().ConfigureAwait(continueOnCapturedContext: false);
            Container container = await GetCosmosContainerAsync().ConfigureAwait(continueOnCapturedContext: false);
            User user = await database.UpsertUserAsync(userId).ConfigureAwait(continueOnCapturedContext: false);

            PartitionKey partitionKey = new PartitionKey(userId);
            PermissionProperties permissionProperties = new PermissionProperties(PermissionIds.AllAccess, PermissionMode.All, container, partitionKey);
            PermissionResponse response = await user.UpsertPermissionAsync(permissionProperties, s_tokenExpiryInSeconds).ConfigureAwait(continueOnCapturedContext: false);
            return response.Resource.Token;

1. This is the Working Code for service to generate a token with V2:

            Permission permission = await GetCosmosClient().ReadPermissionAsync(
                    UriFactory.CreatePermissionUri(databaseId, userId, PermissionIds.AllAccess)).ConfigureAwait(false);

            return permission.Token;

Hence, I am wondering if the token generated by V3 is not right...

[j82w]

@Xiaoliangzhang does the token work for other operations like ReadItemAsync on the partition or do all operations including query fail?

For 3 can you provide the create permission logic? That method provided is only doing a read.

[Xiaoliangzhang]

@j82w Here is the create permission logic:

        private async Task<Permission> CreatePermission(string userId, Permission permission)
        {
            Permission newPermission = new Permission
            {
                PermissionMode = PermissionMode.All,
                ResourceLink = (await GetCosmosContainerAsync().ConfigureAwait(continueOnCapturedContext: false)).SelfLink,
                ResourcePartitionKey = new PartitionKey(userId),
                Id = PermissionIds.AllAccess
            };

            await CreateUserIfNotExistAsync(userId).ConfigureAwait(false);

            permission = await GetCosmosClient().CreatePermissionAsync(
                UriFactory.CreateUserUri(databaseId, userId), newPermission).ConfigureAwait(false);
            return permission;
        }

        private async Task CreateUserIfNotExistAsync(string userId)
        {
            try
            {
                await GetCosmosClient().ReadUserAsync(
                    UriFactory.CreateUserUri(databaseId, userId)).ConfigureAwait(false);
            }
            catch (DocumentClientException ex)
            {
                if (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
                {
                    await GetCosmosClient().CreateUserAsync(
                        UriFactory.CreateDatabaseUri(databaseId), new User { Id = userId }).ConfigureAwait(false);
                }
            }
        }

I'll do a test later and see if other operation works, as the query is our first operations and it fails, we never get to other operations actually.

I'll reply to you after some more tests.

[Xiaoliangzhang]

@j82w Hi, just changed some code and tested CreateItemAsync(), which works fine for the same token generated by V3 SDK.

[ealsur]

Are all the failing queries using Linq? @Xiaoliangzhang are you also using Linq when it fails?

[Xiaoliangzhang]

@ealsur Nope, as you can see in code example 1 (V3 SDK), I was using sql query, not linq.

[Xiaoliangzhang]

The working code 1 stopped working today magically... so SDK V3 doesn't work at all.

[j82w]

@Xiaoliangzhang any way you can provide a basic console app with a repo?

I created a PR #1143 with a test that has the same functionality and it is passing against both prod and the local emulator.

[BenLuts]

@j82w I would like to point out that in my original case the issue only occurs when deployed to our Azure App Service. In local development I don't have an issue.
So maybe a issue with different architectures?

[j82w]

@BenLuts any chance you can try running it with the latest 3.5.1? There was several query fixes in the latest version. The test I wrote validates both scenarios with and without the service interop so there shouldn't be any difference.

Are you running the app x32 locally and in app services?

[Xiaoliangzhang]

@j82w Hi, Thanks for your PR. I am just wondering if you tried to run it with Xamarin.Android or Xamarin.iOS? I did some similar tests. In my case, the tests are passed with a .net core console application however failed in Xamarin.Android , iOS and UWP. All my tests use the same code which is very similar to yours.

[j82w]

@Xiaoliangzhang I was able to get a repo and root cause. The partition key isn't getting passed into the gateway plan retriever which is causing the validation to fail. I'm working on a fix now.

[j82w]

PR #1143 has the fix

[BenLuts]

Looking forward to the release!

[Xiaoliangzhang]

@j82w Thanks for the quick fix, will give a try.

[BenLuts]

Is there a timeline for when the latest version will be released?

[j82w]

@BenLuts doing the release now. Should be available within the hour.

[j82w]

@BenLuts it's available now.

[BenLuts]

@j82w Thanks. Updated my packages and dit some tests. Seems to work like expected now!