Azure-cosmos-dotnet-v2: Intra-document join CreateDocumentQuery<T> fails in unit test mode

[chriswill]

I run unit tests using the test runner in Resharper. A test which previously passed is now failing while running the 1.9.2 client. I know it last succeeded in 1.9.0; I didn't test 1.9.1.

The error is a NotSupportedException, and the error message is "Partition routing information cannot be extracted from the query when running in a 32-bit preferred process. To complete your query and avoid this exception, uncheck the "Prefer 32-bit" option in your project properties window, on the Build tab."

This query succeeds in my main app, but fails in the unit test. Both are built using Any CPU.

The query looks like this:
select a.id, a.documentType from Accounts a join t in a.tenants where a.documentType = 0 and t.tenantId = '123456'

to query a document that looks like this:
{
"createdDateUTC": 1469682631,
"modifiedDateUTC": null,
"pendingDelete": false,
"tenants": [
{
"createdDateUTC": 1469682631,
"displayName": null,
"domain": null,
"enabled": true,
"issValue": null,
"pendingDelete": false,
"products": [],
"tenantId": "123456"
}
],
"stripeAccount": null,
"id": "3aea7052-0658-4ded-9b6e-dcb2c12f1942",
"documentType": 0
}

The code is trivial. I've implemented a wrapper around the CreateDocumentQuery command, but the calling code looks something like this:

string sql = string.Format("select a.id, a.documentType from Accounts a join t in a.tenants where a.documentType = {0} and t.tenantId = '{1}'",
                                (int)DocumentType.Account, "123456");

IQueryable<DocumentEntity> accounts = dataService.GetDocumentsAsQueryable<DocumentEntity>(sql);

DocumentEntity documentEntity = accounts.AsEnumerable().FirstOrDefault();

DocumentEntity is a simple class that is the base for my entities;

public class DocumentEntity
    {
        [JsonProperty(PropertyName = "id")]
        public string Id { get; set; }

        [JsonProperty(PropertyName = "documentType")]
        public DocumentType DocumentType { get; set; }

        protected DocumentEntity()
        {
            Id = Guid.NewGuid().ToString();
        }
    }

I've also attached a screen shot of the build tab of the test project. 32-bit is greyed out.

As I've said, this code runs in a production app but fails in the unit tests. I've love to get it working in unit tests, because, yes, unit tests are important.

[rnagpal]

Hi Chris,

Please make sure that the test runner itself is running in a 64bit process. I'll get the message updated to be clear. Let me know if that works.

Regards,
Rajesh

[chriswill]

The test works if I change the code from Any CPU to x64, which I can do I suppose, at least for testing. Do you see this restriction being lifted in the future or is this how it will likely stay?

[rnagpal]

@chriswill : Not sure why "Prefer 32-bit" is disabled for you when targeting AnyCPU platform. Are you using VS 2013 or VS 2015? What's the target Framework version you are using for your test project?

We don't have any plans to lift this restriction in future, so I'd consider moving it to x64 but I'm interested to know why AnyCPU is not working for you(with Prefer 32-bit unchecked). This seems to work for me for any project I create using VS 2015.

Can you also please run ildasm or ilspy on the generated test assembly(with AnyCPU target selected) and let me know if it's tells it 32-bit or not ?

[chriswill]

According to SO (http://stackoverflow.com/questions/16546886/why-is-the-checkbox-prefer-32-bit-disabled-in-visual-studio-2012) "Prefer 32-bit" is only enabled for executables. My application is a Web Project (MVC), so it doesn't generate an .exe. Test projects, also, aren't executables. They generate a .dll that gets executed by a test runner, which in my case is Resharper.

Anyway, I'm running VS 2015 with update 3. My projects are targeting .net 4.6.

I encountered the same error that I found running my test projects while running my MVC web project. Are you really going to tell me that I can't develop in Any CPU anymore? That seems excessive.

I'm not trying to be difficult, but I'm questioning the level to which this has really been tested and thought through. This all used to work fine.

Can you create a stock MVC app targeting .net 4.6 and using Any CPU and get it to work? I'll try the same. But this seems like a major deal to me.

[chriswill]

The issue is related to FeedOptions with EnableCrossPartitionQuery set to true.

I created a new MVC web app using the default template and pasted a simple query in the Home controller.

This failed:

Uri accountEndPoint = new Uri("https://xxxxx.documents.azure.com:443/");
string accountKey = "xxxx";

string database = "accounts";

string collection = "collection1";

ConnectionPolicy connectionPolicy = new ConnectionPolicy();
connectionPolicy.RetryOptions = new RetryOptions
{
       MaxRetryAttemptsOnThrottledRequests = 20,
       MaxRetryWaitTimeInSeconds = 2
 };

 //Setting read region selection preference 
 connectionPolicy.PreferredLocations.Add("West US");
 connectionPolicy.PreferredLocations.Add("West Europe");

 DocumentClient documentClient = new DocumentClient(
        accountEndPoint,
        accountKey,
        connectionPolicy);

await documentClient.OpenAsync();

Uri documentCollectionUri = UriFactory.CreateDocumentCollectionUri(database, collection);

FeedOptions defaultOptions = new FeedOptions { EnableCrossPartitionQuery = true };

string sql = "select * from c where c.id = 'systemTask-Reports'";

var results = documentClient.CreateDocumentQuery(documentCollectionUri, sql, defaultOptions).ToList();

However, if you null out the FeedOptions setting, the query succeeds.

var results = documentClient.CreateDocumentQuery(documentCollectionUri, sql).ToList();

[rnagpal]

@chriswill :

1. Regarding unit test failure:
   I believe that your Resharper test runner is running in 32-bit process. I haven't used that before but on with VSTest.Console(that comes inbuilt in VS) the default processor architecture is X86(under Test->Test Settings->Default Processor Architecture). I can repro this for a unit test project in VS with default settings. Once I change the processor architecture to X64, it works. Note that my test project itself is targeting AnyCPU(which is same as your case). So you can still keep your unit test project as AnyCPU but you need to change the architecture of the host process(Resharper test runner in your case) to be 64 bit. I'm assuming that changing the target platform of your test project from AnyCPU to x64, triggers Resharper to change it to run in 64-bit somehow. I tried to repro this on VS Unit test project by changing the target platform of the project to x64 and building the solution as well in x64, Test Explorer was smart enough to not show me any tests(with default processor arch set to X86). Only after I changed the default processor architecture to x64 in Test Settings, it started showing and ran successfully. So, in summary you shouldn't have to change the target platform of your test project to x64 to make it work. Let me know if you are able to figure this out what's going on in the Resharper test runner.
2. Regarding Stock MVC Web project:
   The project by default targets AnyCPU platform but when you run it, it runs in a 32 bit process of IIS Express(which gets installed by default with VS). Same explanation as before. The host process is 32 bit. How to you change that: Go to Tools->Options->Projects and Solutions->Web Projects and check:
   Use the 64 bit version of IIS Express for web sites and projects(snapshot below):

That should make your MVC projects work as expected without having to change the target platform of your project from AnyCPU to X64.

1. Regarding the code:
   FeedOptions defaultOptions = new FeedOptions { EnableCrossPartitionQuery = true };
   string sql = "select * from c where c.id = 'systemTask-Reports'";
   var results = documentClient.CreateDocumentQuery(documentCollectionUri, sql, defaultOptions).ToList();

This looks like a cross partition query and hence you require the 64 bit host process(the code path that loads the x64 bit native assembly is this).

var results = documentClient.CreateDocumentQuery(documentCollectionUri, sql).ToList();

This is a single partition query(no EnableCrossPartitionKeyQuery header passed) and so the code path that loads the x64 native assembly is not hit and hence you don't get the error.

Please let me know if you have any more queries. For now you have to make these one time changes to your host processes to ensure that are 64 bit.

Regards,
Rajesh

[rnagpal]

@chriswill Just an update: As of .NET SDK 1.11.2 release we now support all APIs on 32-bit host process as well, though 64-bit host processing is recommended for improved performance.

[ansario]

@rnagpal There is no version of the DocumentDB Client SDK that matches 1.11.2. There is 1.11.3 but I am running into the issue described above with that version.

2017-02-01T21:35:42.100 Error processing message: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.NotSupportedException: Partition routing information cannot be extracted from the query when running in a 32-bit process. To complete your query and avoid this exception, ensure that your host process is 64-bit.
For Executable applications, this can be done by unchecking the "Prefer 32-bit" option in the project properties window, on the Build tab. 
For VSTest based test projects, this can be done by selecting Test->Test Settings->Default Processor Architecture as X64, from Visual Studio Test menu option.
For locally deployed ASP.NET Web applications, this can be done by checking the "Use the 64 bit version of IIS Express for web sites and projects", under Tools->Options->Projects and Solutions->Web Projects.

   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<GetPartitionedQueryExecutionInfoAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<CreateDocumentQueryExecutionContextAsync>d__12.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<GetEnumerator>d__b.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Sample.Functions.Data.DocumentDbRepository`1.<GetOneAsync>d__9.MoveNext() in D:\CurrentProjects\DigitalMerchandising\Sample.AzureFunctions\trunk\src\Sample.Functions\Data\DocumentDbRepository.cs:line 68
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Sample.Functions.Data.DocumentDbRepository`1.<AddOrUpdateAsync>d__11.MoveNext() in D:\CurrentProjects\DigitalMerchandising\Sample.AzureFunctions\trunk\src\Sample.Functions\Data\DocumentDbRepository.cs:line 0
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Sample.Functions.VehicleImageDataTrigger.Run(BrokeredMessage message, TraceWriter log) in D:\CurrentProjects\DigitalMerchandising\Sample.AzureFunctions\trunk\src\Sample.Functions\VehicleImageDataTrigger.cs:line 79
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.NotSupportedException: Partition routing information cannot be extracted from the query when running in a 32-bit process. To complete your query and avoid this exception, ensure that your host process is 64-bit.
For Executable applications, this can be done by unchecking the "Prefer 32-bit" option in the project properties window, on the Build tab. 
For VSTest based test projects, this can be done by selecting Test->Test Settings->Default Processor Architecture as X64, from Visual Studio Test menu option.
For locally deployed ASP.NET Web applications, this can be done by checking the "Use the 64 bit version of IIS Express for web sites and projects", under Tools->Options->Projects and Solutions->Web Projects.

   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<GetPartitionedQueryExecutionInfoAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<CreateDocumentQueryExecutionContextAsync>d__12.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<GetEnumerator>d__b.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Sample.Functions.Data.DocumentDbRepository`1.<GetOneAsync>d__9.MoveNext() in D:\CurrentProjects\DigitalMerchandising\Sample.AzureFunctions\trunk\src\Sample.Functions\Data\DocumentDbRepository.cs:line 68
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Sample.Functions.Data.DocumentDbRepository`1.<AddOrUpdateAsync>d__11.MoveNext() in D:\CurrentProjects\DigitalMerchandising\Sample.AzureFunctions\trunk\src\Sample.Functions\Data\DocumentDbRepository.cs:line 87
---> (Inner Exception #0) System.NotSupportedException: Partition routing information cannot be extracted from the query when running in a 32-bit process. To complete your query and avoid this exception, ensure that your host process is 64-bit.
For Executable applications, this can be done by unchecking the "Prefer 32-bit" option in the project properties window, on the Build tab. 
For VSTest based test projects, this can be done by selecting Test->Test Settings->Default Processor Architecture as X64, from Visual Studio Test menu option.
For locally deployed ASP.NET Web applications, this can be done by checking the "Use the 64 bit version of IIS Express for web sites and projects", under Tools->Options->Projects and Solutions->Web Projects.

   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<GetPartitionedQueryExecutionInfoAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<CreateDocumentQueryExecutionContextAsync>d__12.MoveNext()<---
<---

[rnagpal]

@ansario We had to pull back 1.11.2 due to an unrelated issue but if you are using 1.11.3, you shouldn't see 32-bit related error. I'm pretty sure about it since I removed this error message from the code itself, so if you still see this message, you are probably running an older client. Can you please double check that?

[gmhewett]

Hi @rnagpal. I just updated to 1.11.3, but I'm still getting a bad image exception when dong certain queries. For instance, if I do a query without any constraints:

var query = this.client.CreateDocumentQuery<T>(
                UriFactory.CreateDocumentCollectionUri(this.databaseId, this.collectionName),
                new FeedOptions { EnableCrossPartitionQuery = true });

var results = query.ToList();

everything works great. However, if I try
var results = query.Where(x => x.DeviceProperties.DeviceID == deviceId).ToList();
I get the bad image exception. The SDK is version 1.11.3. Any idea what might be happening?
Thanks!

[rnagpal]

Reopening the issue for investigation.
@gmhewett Can you attach the stack trace of the exception and I'll investigate that?
I'm assuming that you are running in 32 bit mode.

[gmhewett]

@rnagpal you bet, thanks for getting back so quickly. Yes, my platform target is x86, and I'm using the 64-bit version of IIS.

DocDBTrace Information: 0 : QueryPartitionProvider returned something wrong {"query":"SELECT * FROM root WHERE (root[\"DeviceProperties\"][\"DeviceID\"] = \"7e886061-0ac7-410b-98f9-cfa521adde23\") "}, , System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)
   at Microsoft.Azure.Documents.ServiceInteropWrapper.CreateServiceProvider(String configJsonString, IntPtr& serviceProvider)
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.Initialize()
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfoInternal(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Routing.PartitionRoutingHelper.GetProvidedPartitionKeyRanges(SqlQuerySpec querySpec, Boolean enableCrossPartitionQuery, Boolean parallelizeCrossPartitionQuery, PartitionKeyDefinition partitionKeyDefinition, QueryPartitionProvider queryPartitionProvider, String clientApiVersion, QueryInfo& queryInfo)
DocDBTrace Error: 0 : Operation will NOT be retried. Exception: System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)
   at Microsoft.Azure.Documents.ServiceInteropWrapper.CreateServiceProvider(String configJsonString, IntPtr& serviceProvider)
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.Initialize()
   at Microsoft.Azure.Documents.Query.QueryPartitionProvider.GetPartitionedQueryExecutionInfoInternal(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected)
   at Microsoft.Azure.Documents.Routing.PartitionRoutingHelper.GetProvidedPartitionKeyRanges(SqlQuerySpec querySpec, Boolean enableCrossPartitionQuery, Boolean parallelizeCrossPartitionQuery, PartitionKeyDefinition partitionKeyDefinition, QueryPartitionProvider queryPartitionProvider, String clientApiVersion, QueryInfo& queryInfo)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.<TryGetTargetPartitionKeyRangeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.<ExecuteOnceAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2.<<ExecuteAsync>b__0>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
Exception thrown: 'System.AggregateException' in mscorlib.dll

[rnagpal]

@gmhewett I was able to repo this on my end and currently working on a fix which we should release by early next week. Until then, the workaround is to use the x64 target.
Thanks for reporting this to us!

[rnagpal]

We have released a fix for this issue in 1.11.4 released today.
@ansario @gmhewett Please upgrade and confirm.

[ansario]

@rnagpal works for me!

[gmhewett]

@rnagpal works for me! Thanks for the quick turnaround!