Azure-cosmos-dotnet-v2: Local Emulator: Microsoft.Azure.Documents.ServiceUnavailableException - Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.

[MayankSri]

I am getting this error intermittently while the document exists in the collection:

Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.
ActivityId: 676a8665-f218-42c3-9e1a-7a1f7c9cd9ae, documentdb-dotnet-sdk/1.19.1 Host/32-bit MicrosoftWindowsNT/6.2.9200.0 ---> Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: 676a8665-f218-42c3-9e1a-7a1f7c9cd9ae, documentdb-dotnet-sdk/1.19.1 Host/32-bit MicrosoftWindowsNT/6.2.9200.0
   at Microsoft.Azure.Documents.TimeoutHelper.ThrowGoneIfElapsed()
   at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicasInternalAsync>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicaAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyReader.<ReadSessionAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass1.<<InvokeAsync>b__0>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClassf`1.<<ExecuteAsync>b__d>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__18`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Client.DocumentClient.<ReadAsync>d__30c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDocumentPrivateAsync>d__196`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2.<<ExecuteAsync>b__0>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at <Namespace>.<ClassName>.<UpdateFileAsWrittenAsync>d__13.MoveNext() in <Local path>\<ClassName>.cs:line 124

[kirankumarkolli]

@MayankSri could you please send mail to [email protected] with full exception details and account information and we will take a look into it.

[kirankumarkolli]

@MayankSri closing the issue. Feel free to re-active in-case your issues is not addressed.

[mikerains]

Happened to me connecting to local emulator cosmodb service v 1.19.102.5. First time it happened, it was a "ServiceUnavailable" exception. it went on for several hours, and then magically started working again. Second time it happened, several days later, it was throwing GoneException; I performed the "Reset Data" context-menu operation from the systray CosmoDb icon, after which the code was able to connect again.

I don't know the two episodes are truly related. In the first occurrence, I feel it also could not connect to CosmoDb instances hosted on Azure.

[jkonecki]

Same thing happening to me. I susspect this may be related to IP address of the machine changing.

[jkonecki]

Please re-open this issue as the problem persists in version 2.0.0 of the emulator. Resetting data every time the issue occurs is not a workable solution in the long run.

I'm happy to help investigate the issue.

[milismsft]

@jkonecki Can you please follow the instructions for collecting trace logs here and send them to [email protected]? Also can you please look for any DocumentDB*.dmp minidumps in either %LOCALAPPDATA%\CrashDumps or %SystemDrive%\wfroot and attach them them as well?

[jkonecki]

@milismsft I will - please note it may take a few days before the issue to occur again.

[jkonecki]

@milismsft I've just emailed trace files to [email protected]

[jkonecki]

I've just noticed that the issue occurs when Direct / TCP connection policy is used.
I can switch to Garteway / HTTP and connect without problems.
Hope this helps in your investigation.

[devlie]

I just sent a repro trace to askcosmosdb. Thanks.

[kirankumarkolli]

More context about the reported issue (from offline thread):

Sorry for the delay, the issue comes and goes, so it took me a while to repro.

The issue is with Emulator (the MSI-installed version). It seems (I haven’t confirmed) like Emulator is unhappy if I’m trying to access a collection that I created before machine was restarted.

DocumentClientException: Message: {"Errors":["The requested resource is no longer available at the server."]} ActivityId: 335ecfb6-b94d-4a5e-b7f9-c2f9812f4a94, Request URI: /apps/DocDbApp/services/DocDbServer20/partitions/a4cb4960-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: RequestStartTime: 2019-08-14T00:56:44.7105358Z, RequestEndTime: 2019-08-14T00:56:44.7125112Z, Number of regions attempted: 1 ResponseTime: 2019-08-14T00:56:44.7125112Z, StoreResult: StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer20/partitions/a4cb4960-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 541, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 1000, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: null, ResourceType: Collection, OperationType: Read , SDK: Microsoft.Azure.Documents.Common/2.2.0.0, Windows/10.0.18890 documentdb-netcore-sdk/2.4.0