Azure-cosmos-dotnet-v2: Attempting to connect with Protocol.Tcp and ConnectionMode.Direct results in ServiceUnavailableException

Created on 21 Jan 2017  ·  38Comments  ·  Source: Azure/azure-cosmos-dotnet-v2

I've been unable to get connected to document db using version 1.0.0 of the sdk when I set ConnectionMode.Direct and Protocol.Tcp. Connections using Https and Gateway work fine but take some time (~10s) to connect. At first I thought this was a firewall issue on my work network, but have since had the same issue from home and also from a VM on Azure.

Running this code on mac os and linux (haven't tried windows, yet):

ConnectionPolicy connectionPolicy = new ConnectionPolicy
{
    ConnectionMode = ConnectionMode.Direct,
    ConnectionProtocol = Protocol.Tcp
};

this.documentDbClient = new DocumentClient(
    new Uri(documentDbEndpointUrl),
    documentDbAuthKey,
    connectionPolicy
);

await this.documentDbClient.OpenAsync();

Results in the following:

Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.
ActivityId: 5d067c59-a41e-4800-ab94-8f130d50517f ---> Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: 5d067c59-a41e-4800-ab94-8f130d50517f
   at Microsoft.Azure.Documents.StoreReader.<ReadPrimaryAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.QuorumReader.<ReadPrimaryAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.QuorumReader.<ReadStrongAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass14_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2_0`1.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.Client.DocumentClient.<ReadAsync>d__236.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDatabasePrivateAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.Documents.Client.DocumentClient.<CreateDatabaseIfNotExistsAsync>d__95.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MicroEdge.Services.Tenants.Services.TenantDocumentDbService.CreateDatabaseIfNotExists() in /Users/dan.hamlin/Projects/microedge-services-tenants/src/MicroEdge.Services.Tenants/Services/TenantDocumentDbService.cs:line 181
   at MicroEdge.Services.Tenants.Services.TenantDocumentDbService..ctor(IHostingEnvironment env, IConfigurationRoot config, ILogger`1 logger) in /Users/dan.hamlin/Projects/microedge-services-tenants/src/MicroEdge.Services.Tenants/Services/TenantDocumentDbService.cs:line 70
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.ScopedCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)
   at lambda_method(Closure , IServiceProvider , Object[] )
   at Microsoft.AspNetCore.Mvc.Internal.TypeActivatorCache.CreateInstance[TInstance](IServiceProvider serviceProvider, Type implementationType)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeExceptionFilterAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
bug investigating
Source
picture Blackbaud-DanHamlin
👍5

Most helpful comment

I think the fact that Direct mode is not supported in the .NET Standard 2.0 client on non-Windows platform is a pretty large adoption roadblock, especially given how much push for Unix-based containerization we see in the ASP.NET Core world.

It would be very helpful if:

  • this information was added to this article https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips. This is particularly confusing, as the article suggests to use direct mode as the default one, which can easily lead to this error
  • this information was added as known limitation to the .NET Standard SDK release notes https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet-core. At the moment this page states that The Azure Cosmos DB .NET Core SDK has feature parity with the latest version of the Azure Cosmos DB .NET SDK which is simply false.
  • the SDK should throw a PlatformNotSupportedException on macOS / Linux. At the moment the exception is a GoneException with a message that The requested resource is no longer available at the server. which is very difficult to troubleshoot and doesn't really reflect the real situation
  • of course ultimately, the most helpful thing would be if this was fixed, or at least some more concrete plan of how and when this will be dealt with (rather than "first half of 2018", as we are almost there already) was shared here

thanks ⭐️

picture filipw on 18 May 2018
👍10

All 38 comments

Hi Blackbaud-DanHamlin,
I couldn't find 5d067c59-a41e-4800-ab94-8f130d50517f in the server side log, and this looks like client side connectivity issue. To rule out the possibility for .NET core, can you try this tool
https://github.com/mingaliu/DocumentDBStudio/releases (in your home or VM) and using Direct Connectivity, let me know if it work or not.

Thanks!
Ming

mingaliu picture mingaliu on 21 Jan 2017

From my home network on a windows 10 machine, Azure DocumentDb Studio works great:
the response came with x-ms-activity-id: cbf77b22-c5f1-4dd8-891e-850ea12e6009
image

And now that i've tried my same project from windows 10 on my home network i see that it works. I tried from mac os 10.11.6 again and it still didn't work.

dotnet --version
on windows is: 1.0.0-preview2-003131
on os x is: 1.0.0-preview2-003148

going to try getting these on same version.

Blackbaud-DanHamlin picture Blackbaud-DanHamlin on 22 Jan 2017

i should mention, these are the same physical machine, windows 10 is a guest on the mac.

Blackbaud-DanHamlin picture Blackbaud-DanHamlin on 22 Jan 2017

Dan, We will try to repro the issue today and get back.

hemantkr picture hemantkr on 23 Jan 2017

We have a repro of the issue and will fix it. Thanks for flagging the issue Dan. In the interim, please continue using the gateway mode.

hemantkr picture hemantkr on 24 Jan 2017

Thank you @hemantkr

Blackbaud-DanHamlin picture Blackbaud-DanHamlin on 24 Jan 2017

I have the same problem when running my app in a docker container. Gateway works fine but it fails when using TCP. My app is a .NET core app using the Microsoft.Azure.DocumentDB.Core 1.0 sdk. 

docker info

Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false

Is there an ETA on when this will be fixed?

kaladin-storm picture kaladin-storm on 1 Feb 2017
👍2

I also get the same Service is currently unavailable error using Microsoft.Azure.DocumentDB (1.11.3) in a .NET Core project targeting .NET Framework:

ActivityId: 049ad2b7-c7cc-46dd-b1ba-1f876352460f at Microsoft.Azure.Documents.StoreReader.<ReadPrimaryAsync>d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.Documents.QuorumReader.<ReadPrimaryAsync>d__26.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.Documents.QuorumReader.<ReadStrongAsync>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__b.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass1.<<InvokeAsync>b__0>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClassf`1.<<ExecuteAsync>b__d>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

felschr picture felschr on 8 Feb 2017

We are seeing this behavior in our application. We recently moved our data from hundreds of "old school" non-partitioned collections to a single, partitioned collection. Relevant information:

  • This is a .NET 4.5.2 project.
  • DocumentDB SDK package is at 1.11.3. (The latest version caused some problems for us under load with multiple threads reusing the IDocumentClient instance.)
  • The collection is under heavy load. We have a process that is pushing a couple hundred million documents into the collection daily for about 20 hours per day.
  • The queries run for a while before this exception is thrown. They aren't especially fast, but we need to support arbitrary queries that may run for a _long_ time.
  • The retry policy we're using is pretty aggressive. We bump up both maximum retry attempts and maximum wait time.

Here are some affected activity IDs:

  • 1524ba63-8b1b-43ae-b294-a78d8fafa135
  • 393f2cdc-aeac-45ac-bbbb-84154bbf1126
  • e63cbc6c-7782-4a70-a065-eef0fc6baf13
  • 4ea5b233-98ce-488f-9af3-a463f5bb2bb8

EDIT: This is all running within App Services in the same region as the DocumentDB account.

EDIT: Here is a stack trace from one of the exceptions:

Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.
ActivityId: 53f297fe-3f21-4473-a9f2-6bd57dbe9326 ---> Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: 53f297fe-3f21-4473-a9f2-6bd57dbe9326
   at Microsoft.Azure.Documents.TimeoutHelper.ThrowGoneIfElapsed()
   at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicasInternalAsync>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicaAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyReader.<ReadAnyAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass1.<<InvokeAsync>b__0>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClassf`1.<<ExecuteAsync>b__d>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__18`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Client.DocumentClient.<ExecuteQueryAsync>d__312.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryClient.<ExecuteQueryAsync>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<ExecuteQueryRequestInternalAsync>d__40.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<ExecuteQueryRequestAsync>d__25`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.<ExecuteRequestAsync>d__1f`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.ParallelQuery.DocumentProducer`1.<FetchAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.Query.ParallelQuery.DocumentProducer`1.<FetchAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.Query.ParallelQuery.DocumentProducer`1.<MoveNextAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.ParallelDocumentQueryExecutionContext.<DrainAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<ExecuteNextPrivateAsync>d__18`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at SSB.Segmentation.CampaignProcessing.RunObservableCampaignJob.<ReceiveMessageAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at SSB.Segmentation.CampaignProcessing.RunObservableCampaignJob.<ReceiveMessageAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.<InvokeAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<InvokeAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithWatchersAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext()
   --- End of inner exception stack trace ---
davidsidlinger picture davidsidlinger on 24 Apr 2017

Hi David,
We are not able to find any of these activityIds in our logs. Most likely this is an issue on the client side which is affecting the connectivity to our service.
Generally, we have seen such issues when the client is starved for CPU. You could try scaling up your AppService instance and see if it helps. You could try running the application in a standalone VM so that you have better visibility to the resource consumption.
Hope it helps!

shahankur11 picture shahankur11 on 26 Apr 2017

Hello, may be my stack trace would be helpful.
I use Microsoft.Azure.DocumentDB.Core 1.3.2 in Microsoft Visual Studio for Mac
When I call documentDbClient.OpenAsync(); my stacktrace is similar to above ones.
But if I don't call OpenAsync and call ReadDocumentAsync(..), I get:

  Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.
  ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3
  ResponseTime: 2017-06-22T02:40:40.9738740Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:18625/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225503s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:40:41.2802250Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15040/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225502s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:40:42.5247110Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15040/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225502s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:40:44.7818500Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:18625/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225503s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:40:49.0541800Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:40:57.2595530Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
  ResponseTime: 2017-06-22T02:41:10.4898220Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
   ---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
  ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3, Request URI: /apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/ ---> System.PlatformNotSupportedException: Operation is not supported on this platform.
     at System.Net.Sockets.SocketPal.WindowsIoctl(SafeCloseSocket handle, Int32 ioControlCode, Byte[] optionInValue, Byte[] optionOutValue, Int32& optionLength)
     at System.Net.Sockets.Socket.IOControl(Int32 ioControlCode, Byte[] optionInValue, Byte[] optionOutValue)
     at System.Net.Sockets.Socket.IOControl(IOControlCode ioControlCode, Byte[] optionInValue, Byte[] optionOutValue)
     at Microsoft.Azure.Documents.RntbdConnection.SetKeepAlive(Socket socket)
     at Microsoft.Azure.Documents.RntbdConnection.<OpenSocket>d__47.MoveNext()
     --- End of inner exception stack trace ---
     at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicasInternalAsync>d__11.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicaAsync>d__9.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.ConsistencyReader.<ReadSessionAsync>d__14.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__15.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass14_0.<<InvokeAsync>b__0>d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2_0`1.<<ExecuteAsync>b__0>d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
     --- End of inner exception stack trace ---
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__2`1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__14.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__11.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.Client.DocumentClient.<ReadAsync>d__250.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDocumentPrivateAsync>d__165.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetry>d__3.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

Error: Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.
ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3
ResponseTime: 2017-06-22T02:40:40.9738740Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:18625/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225503s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:40:41.2802250Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15040/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225502s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:40:42.5247110Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15040/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225502s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:40:44.7818500Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:18625/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225503s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:40:49.0541800Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:40:57.2595530Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
ResponseTime: 2017-06-22T02:41:10.4898220Z, StoreReadResult: StorePhysicalAddress: rntbd://db5prdddc03-docdb-1.documents.azure.com:15019/apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/, LSN: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 0, IsGone: True, IsNotFound: False, RequestCharge: 0, ResourceType: Document, OperationType: Read
---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3, Request URI: /apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/ ---> System.PlatformNotSupportedException: Operation is not supported on this platform.
at System.Net.Sockets.SocketPal.WindowsIoctl(SafeCloseSocket handle, Int32 ioControlCode, Byte[] optionInValue, Byte[] optionOutValue, Int32& optionLength)
at System.Net.Sockets.Socket.IOControl(Int32 ioControlCode, Byte[] optionInValue, Byte[] optionOutValue)
at System.Net.Sockets.Socket.IOControl(IOControlCode ioControlCode, Byte[] optionInValue, Byte[] optionOutValue)
at Microsoft.Azure.Documents.RntbdConnection.SetKeepAlive(Socket socket)
at Microsoft.Azure.Documents.RntbdConnection.d__47.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.Azure.Documents.StoreReader.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.StoreReader.<ReadMultipleReplicaAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.ConsistencyReader.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.ReplicatedResourceClient.<InvokeAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass14_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.BackoffRetryUtility1.<>c__DisplayClass2_01.<<ExecuteAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.BackoffRetryUtility1.<ExecuteRetry>d__3.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.Documents.BackoffRetryUtility1.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.BackoffRetryUtility1.d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.ReplicatedResourceClient.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.Client.DocumentClient.d__250.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDocumentPrivateAsync>d__165.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.BackoffRetryUtility1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Azure.Documents.BackoffRetryUtility1.<ExecuteRetry>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Azure.Documents.BackoffRetryUtility1.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Azure.Documents.BackoffRetryUtility1.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

gleyba picture gleyba on 22 Jun 2017

So, as I can suppose
ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3, Request URI: /apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/ ---> System.PlatformNotSupportedException: Operation is not supported on this platform.

is the reason of the problem.
Because, I think, System.Net.Sockets.SocketPal.WindowsIoctl is definitely not supported

gleyba picture gleyba on 22 Jun 2017

Looks like it, Rajesh does .Net core SDK work on Mac?

Thanks,
Ankur

From: Gleb Kolobkov <[email protected]notifications@github.com>
Sent: Wednesday, June 21, 2017 7:45 PM
Subject: Re: [Azure/azure-documentdb-dotnet] Attempting to connect with Protocol.Tcp and ConnectionMode.Direct results in ServiceUnavailableException (#194)
To: Azure/azure-documentdb-dotnet <[email protected]azure-documentdb-dotnet@noreply.github.com>
Cc: Ankur Shah <[email protected]ankshah@microsoft.com>, Comment <[email protected]comment@noreply.github.com>

So, as I can suppose
ActivityId: ca070d70-b3c9-409e-b2a4-0082e3bd56b3, Request URI: /apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/a14aac4d-4329-488f-8c37-6e65b34dddb2/partitions/0bbc93da-6e62-48ac-a022-8673b0492d92/replicas/131425683080225504s/ ---> System.PlatformNotSupportedException: Operation is not supported on this platform.

is the root of the problem.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-documentdb-dotnet%2Fissues%2F194%23issuecomment-310260235&data=02%7C01%7Cankshah%40microsoft.com%7Cbaea7d3a8fc346e48fc908d4b918b0c0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336963076875156&sdata=JhWXftk2lPjzUwwp0wlbkOvVKYp5dujXJrRxZc2GC30%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADcJsrWTb2TTvB4eb0VidD8ke3GqDenuks5sGdUvgaJpZM4LqIJ3&data=02%7C01%7Cankshah%40microsoft.com%7Cbaea7d3a8fc346e48fc908d4b918b0c0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636336963076875156&sdata=heY0EwiAnGXL%2BgfbjxT5NRBzTrZW40kw8XexeL1DESc%3D&reserved=0.

shahankur11 picture shahankur11 on 22 Jun 2017

So, ConnectionMode.Direct is only supported on Windows platform? Or there would be fixes in next releases?

gleyba picture gleyba on 22 Jun 2017

@gleyba That's correct. Direct mode is not supported on Mac as of now. This will be supported when we move to .NET Standard 2.0 in future. No timelines yet on that.

Would Gateway mode work for you meanwhile?

rnagpal picture rnagpal on 22 Jun 2017
😕3

Hmm, if consider Mac OS X only as develop platform, I think I can do that trick 

            ConnectionPolicy connectionPolicy = new ConnectionPolicy();
            if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) {
                connectionPolicy.ConnectionMode = ConnectionMode.Direct;
                connectionPolicy.ConnectionProtocol = Protocol.Tcp;
            }
            DocumentClient dbClient = new DocumentClient(hostUri, keyStr, connectionPolicy);

and run DocumentClient in Gateway Mode for Mac and in Direct Mode for Azure.
But I was wonder to know if I could consider Unix as production-ready platform to work with DocumentDb, may be in conjunction with Service Fabric, running on Ubuntu. So, now I know, that I can't do this.

gleyba picture gleyba on 22 Jun 2017
👍2

Just to remember that MacOS and Docker environment still need support to Direct connection

Caldas picture Caldas on 24 Oct 2017
👍4

I seem to have got the same issue on my Windows laptop. Worked fine on my Windows desktop.

Changing my connection policy from new ConnectionPolicy() { ConnectionMode = ConnectionMode.Direct, ConnectionProtocol = Protocol.Tcp }; to simply new ConnectionPolicy(); did the trick but obviously the performance recommendation is to use Direct + TCP. Could it be that I don't have a Windows feature enabled that is required? Like TCP?

benmccallum picture benmccallum on 17 Jan 2018

Just ran into this today. Any ETAs for netstandard2.0 support mentioned by @rnagpal back in June 2017?

Is this documented anywhere beyond this issue? A simple statement right under the recommendation to use TCP+Direct saying it is only supported on windows would have been a huge time saver here.

johlrich picture johlrich on 13 Feb 2018
👍8

@johlrich , in my case it didn't even work on Windows so... Weird. Worked on laptop, not on desktop, both Win10.

benmccallum picture benmccallum on 14 Feb 2018

Just spent a day figuring this out too. Given that Docker (Linux) is the go-to deployment option for individuals and enterprises, is it worth being a little more transparent about the timelines for support?

kieronlanning picture kieronlanning on 28 Feb 2018
👍1

@ausfeldt we should refresh documentation with un-supported scenarios.

Also we are working on direct mode support for core and its planned for first half of 2018.

kirankumarkolli picture kirankumarkolli on 28 Feb 2018
👍1

@kirankumarkolli Thanks for the update, it would be great to see this on a roadmap/ issue so we can track it too...

kieronlanning picture kieronlanning on 28 Feb 2018

Has anything of this changed with the release of 1.21.0?

felschr picture felschr on 6 Mar 2018
👍3

You can try my solution, see https://github.com/Azure/azure-documentdb-dotnet/issues/400

mlankamp picture mlankamp on 1 Apr 2018

I think the fact that Direct mode is not supported in the .NET Standard 2.0 client on non-Windows platform is a pretty large adoption roadblock, especially given how much push for Unix-based containerization we see in the ASP.NET Core world.

It would be very helpful if:

  • this information was added to this article https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips. This is particularly confusing, as the article suggests to use direct mode as the default one, which can easily lead to this error
  • this information was added as known limitation to the .NET Standard SDK release notes https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet-core. At the moment this page states that The Azure Cosmos DB .NET Core SDK has feature parity with the latest version of the Azure Cosmos DB .NET SDK which is simply false.
  • the SDK should throw a PlatformNotSupportedException on macOS / Linux. At the moment the exception is a GoneException with a message that The requested resource is no longer available at the server. which is very difficult to troubleshoot and doesn't really reflect the real situation
  • of course ultimately, the most helpful thing would be if this was fixed, or at least some more concrete plan of how and when this will be dealt with (rather than "first half of 2018", as we are almost there already) was shared here

thanks ⭐️

filipw picture filipw on 18 May 2018
👍10

@filipw This should be fixed in 2.0.0. https://www.nuget.org/packages/Microsoft.Azure.DocumentDB.Core/2.0.0-preview

ausfeldt picture ausfeldt on 29 May 2018

We're facing the same issue blocking our migration to linux containers. What is the estimated release date for 2.0?

eiriktsarpalis picture eiriktsarpalis on 10 Jul 2018

I'm having the same issue with linux containers. It is a total no-go for .Net Core and Linux users... It is incredible that the docs still say feature parity...

Team, is there any updates on that? Its been an year almost... We are using the latest NuGet package here...

galvesribeiro picture galvesribeiro on 12 Jul 2018
👍2

BTW, for the record, 2.0.0-preview still have same issue and it is _not_ targeting netstandard2.0...

galvesribeiro picture galvesribeiro on 12 Jul 2018
😕1

@galvesribeiro is it being built on linux? If not, try "dotnet build -r \ is one of the following linux-rids, or add "\

ausfeldt picture ausfeldt on 13 Jul 2018

Has this been fixed in the stable 2.0.0 release?

UPDATE: It seems it has:
"Added support for Direct Mode TCP for non-Windows clients."
— https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet-core

felschr picture felschr on 17 Sep 2018
👍2

Has anyone verified?

kieronlanning picture kieronlanning on 17 Sep 2018

Works for me, updated to ^2 and it worked on a linux container on Ubuntu vm

martinothamar picture martinothamar on 18 Oct 2018

it works for me on Debian Stretch - thanks!

filipw picture filipw on 18 Oct 2018

I get this error when I dockerize my test app (in Linux mode) and deploy it as an Azure container instance into the same region as my Cosmos DB instance.

If I run my exact same test app on Windows (in an office on the opposite side of the world from my Cosmos instance) it works fine (although very slowly).

If I comment out thusly:

//ConnectionMode = ConnectionMode.Direct,
//ConnectionProtocol = Protocol.Tcp,

And rebuild/publish my container instance, it works fine. The base image is:

FROM microsoft/dotnet:2.1-aspnetcore-runtime

UPDATE: Ignore...

danielearwicker picture danielearwicker on 13 Dec 2018

@danielearwicker Which version of Microsoft.Azure.DocumentDB.Core nuget package are you using? It works fine for us on with 2.0.0 (in Docker, on AKS/ACS).
Base image is microsoft/dotnet:2.1.5-aspnetcore-runtime, not that it should be relevant.

htuomola picture htuomola on 13 Dec 2018
👍1

@htuomola - sigh... forgot I'd downgraded yesterday attempting to work around something else.

danielearwicker picture danielearwicker on 13 Dec 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kspearrin picture kspearrin  ·  4Comments
cgelon picture cgelon  ·  7Comments
brokentriangles picture brokentriangles  ·  8Comments
BenjaBobs picture BenjaBobs  ·  6Comments
rave1502 picture rave1502  ·  4Comments