Iotedge: EdgeHub not reconnecting to IotHub after network failure.
Last night we had a network failure disconnecting the edgeDevice from the internet at 03:07. The device came online again at 06:39 but the edgeHub did not reconnect to iotHub.
The output of iotedge list said that all modules are running (and have been running for a week). The IotEdgeExtension in VSCode displayed the edgeDevice as disconnected.
To reconnect the device to IotHub we had to restart the edgeHub module.
Logging output of edgeAgent:
2019-03-28 23:55:08.600 +00:00 [INF] - Starting compaction of stores
2019-03-28 23:55:08.601 +00:00 [INF] - Starting compaction of store moduleState
2019-03-28 23:55:08.601 +00:00 [INF] - Starting compaction of store default
2019-03-28 23:55:08.602 +00:00 [INF] - Starting compaction of store deploymentConfig
2019-03-29 01:55:08.605 +00:00 [INF] - Starting compaction of stores
2019-03-29 01:55:08.606 +00:00 [INF] - Starting compaction of store moduleState
2019-03-29 01:55:08.606 +00:00 [INF] - Starting compaction of store default
2019-03-29 01:55:08.607 +00:00 [INF] - Starting compaction of store deploymentConfig
2019-03-29 03:25:54.644 +00:00 [ERR] - Error refreshing edge agent configuration from twin.
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeoutTwin>b__64_2(Task`1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.RefreshTwinAsync() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 190
2019-03-29 03:25:58.943 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:26:28.978 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:26:34.063 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:27:04.088 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:27:09.180 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:27:39.207 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:27:44.297 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:28:14.332 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:28:19.416 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:28:49.458 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:28:54.509 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-03-29 03:29:24.531 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-03-29 03:29:29.633 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
The rest of that logfile is the same error over and over again.
Logging output of edgeHub does not reach back to the time of the network failure (because it's set to debug and produces a lot of output) but it is repeating the following error when trying to send messages upstream:
2019-03-29 09:32:35.053 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 1 message(s) upstream.
2019-03-29 09:32:35.191 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation OpenAsync timed out for bewaPi1/OpcDataToMessage
2019-03-29 09:32:35.191 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call timed out
2019-03-29 09:32:35.205 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-03-29 09:32:35.222 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-03-29 09:32:35.230 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.TwinManager] - Processing ConnectionEstablished for device bewaPi1/OpcDataToMessage
2019-03-29 09:32:35.231 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.IDeviceScopeIdentitiesCache] - Getting service identity for bewaPi1/OpcDataToMessage
2019-03-29 09:32:35.232 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Creating cloud connection for client bewaPi1/OpcDataToMessage using EdgeHub credentials
2019-03-29 09:32:35.232 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-03-29 09:32:35.230 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-03-29 09:32:35.255 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
BewaControl-ReneDivossen
All 28 comments
It looks like the network failure also resulted in the loss of data!
Our edgeDevice reads data from an SPS every 2 seconds and also sends messages upstream every 2 seconds.
Looking at the application insight of an azure function processing those messages I noticed that we are missing over ten minutes of data:
BewaControl-ReneDivossen
on 29 Mar 2019
Couple of questions:
- Which version of the edgeHub are you using?
- Are DNS settings configured of the device or all modules? You can do this from deployment like this.
veyalla
on 29 Mar 2019
@veyalla - We are using edgeHub Version - 1.0.6.19913336 (8288bc9bd6f6e15295fea506cd3f99d7f6347a6a)
No we did not have DNS settings configured. I did it now in the deamon.json as described in your first link. Do I understand correctly that if I configure the DNS IP in the deamon.json than it's used by all docker containers / edge modules and I don't have to configure it for each module in the createOptions anymore !?
Can you elaborate a bit on why setting the DNS server should improve reconnecting after network failure.
BewaControl-ReneDivossen
on 1 Apr 2019
DNS IP in the deamon.json than it's used by all docker containers / edge modules
Yes, that is correct.
The reason it is needed seems to be related to behavior of docker containers on certain platforms or operating systems. If a module reboots when offline, sometimes default DNS is not set for it in manner that would allow it to communicate externally (without a container restart) when IoT Hub connectivity is restored. Specifying a DNS server improves robustness to the recovery process.
veyalla
on 2 Apr 2019
@veyalla - thanks for this insight. I'll have an eye on it to see if with this setting it is able to recover after the next network failure.
Any idea why we are missing data (see my second post on this issue)? We have set the "timeToLiveSecs" to 604800 (7 days) and the disk was not out of space so IotEdgeHub should be able to buffer all messages and send them when network is available again !?
BewaControl-ReneDivossen
on 2 Apr 2019
@veyalla
I provoked a short network failure and this time the edgeAgent (Version - 1.0.6.19913336 (8288bc9bd6f6e15295fea506cd3f99d7f6347a6a) ) could not reconnect afterwards.
EdgeHub and our customModules did reconnect.
Logging output of edgeAgent:
2019-04-02 07:37:11.153 +00:00 [INF] - Starting compaction of stores
2019-04-02 07:37:11.153 +00:00 [INF] - Starting compaction of store moduleState
2019-04-02 07:37:11.154 +00:00 [INF] - Starting compaction of store deploymentConfig
2019-04-02 07:37:11.154 +00:00 [INF] - Starting compaction of store default
2019-04-02 07:42:01.008 +00:00 [ERR] - Error refreshing edge agent configuration from twin.
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeoutTwin>b__64_2(Task`1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.RefreshTwinAsync() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 190
2019-04-02 07:42:02.111 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-04-02 07:42:32.163 +00:00 [WRN] - Updating reported properties failed with error Operation timeout expired. type System.TimeoutException
2019-04-02 07:42:37.209 +00:00 [WRN] - Reconcile failed because of the an exception
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 115
2019-04-02 07:42:43.755 +00:00 [INF] - Updated reported properties
Logging output of edgeHub:
2019-04-02 07:37:31.473 +00:00 [INF] - Exiting connected state
2019-04-02 07:37:31.478 +00:00 [INF] - Entering unreachable state
2019-04-02 07:37:51.534 +00:00 [INF] - Entering disconnected state
2019-04-02 07:37:51.536 +00:00 [INF] - Cloud connection for bewaPi1/$edgeHub is False
2019-04-02 07:37:51.537 +00:00 [INF] - Connection status for bewaPi1/$edgeHub changed to Disconnected
2019-04-02 07:37:51.547 +00:00 [INF] - Closing receiver in cloud proxy 631c244f-3ead-489b-af1e-baf641b69310 for bewaPi1/$edgeHub
2019-04-02 07:37:51.548 +00:00 [INF] - Closed cloud proxy 631c244f-3ead-489b-af1e-baf641b69310 for bewaPi1/$edgeHub
2019-04-02 07:37:51.548 +00:00 [INF] - Cloud connection for bewaPi1/publisher is False
2019-04-02 07:37:51.549 +00:00 [INF] - Connection status for bewaPi1/publisher changed to Disconnected
2019-04-02 07:37:51.551 +00:00 [INF] - Closing receiver in cloud proxy bcea90e9-1d1e-49a3-9bdf-4f58a62cef56 for bewaPi1/publisher
2019-04-02 07:37:51.552 +00:00 [INF] - Closed cloud proxy bcea90e9-1d1e-49a3-9bdf-4f58a62cef56 for bewaPi1/publisher
2019-04-02 07:37:51.552 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is False
2019-04-02 07:37:51.552 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to Disconnected
2019-04-02 07:37:51.553 +00:00 [INF] - Closing receiver in cloud proxy bff57581-5cd9-43e3-bd1f-8fdb1badd6a4 for bewaPi1/OpcDataToMessage
2019-04-02 07:37:51.553 +00:00 [INF] - Closed cloud proxy bff57581-5cd9-43e3-bd1f-8fdb1badd6a4 for bewaPi1/OpcDataToMessage
2019-04-02 07:37:51.553 +00:00 [INF] - Cloud connection for bewaPi1/Alarm is False
2019-04-02 07:37:51.554 +00:00 [INF] - Connection status for bewaPi1/Alarm changed to Disconnected
2019-04-02 07:37:51.556 +00:00 [INF] - Closing receiver in cloud proxy 7837831c-89ea-468a-9115-3bd12dfc7478 for bewaPi1/Alarm
2019-04-02 07:37:51.556 +00:00 [INF] - Closed cloud proxy 7837831c-89ea-468a-9115-3bd12dfc7478 for bewaPi1/Alarm
2019-04-02 07:37:51.557 +00:00 [INF] - Cloud connection for bewaPi1/OpcUa is False
2019-04-02 07:37:51.557 +00:00 [INF] - Connection status for bewaPi1/OpcUa changed to Disconnected
2019-04-02 07:37:51.578 +00:00 [INF] - Closing receiver in cloud proxy 9763a8fd-a3aa-470c-b72e-21471c8582d7 for bewaPi1/OpcUa
2019-04-02 07:37:51.579 +00:00 [INF] - Closed cloud proxy 9763a8fd-a3aa-470c-b72e-21471c8582d7 for bewaPi1/OpcUa
2019-04-02 07:37:51.579 +00:00 [INF] - Cloud connection for bewaPi1/Webserver is False
2019-04-02 07:37:51.579 +00:00 [INF] - Connection status for bewaPi1/Webserver changed to Disconnected
2019-04-02 07:37:51.587 +00:00 [INF] - Closing receiver in cloud proxy 36b6126c-00bf-4eee-a769-0c505d81e704 for bewaPi1/Webserver
2019-04-02 07:37:51.588 +00:00 [INF] - Closed cloud proxy 36b6126c-00bf-4eee-a769-0c505d81e704 for bewaPi1/Webserver
2019-04-02 07:37:54.874 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:38:08.232 +00:00 [INF] - Starting compaction of stores
2019-04-02 07:38:08.232 +00:00 [INF] - Starting compaction of store default
2019-04-02 07:38:08.234 +00:00 [INF] - Starting compaction of store twins
2019-04-02 07:38:08.355 +00:00 [INF] - Starting compaction of store messages
2019-04-02 07:38:10.938 +00:00 [INF] - Starting compaction of store DeviceScopeCache
2019-04-02 07:38:11.059 +00:00 [INF] - Starting compaction of store bewaPi1/Support/input1
2019-04-02 07:38:11.155 +00:00 [INF] - Starting compaction of store iothub
2019-04-02 07:38:11.383 +00:00 [INF] - Starting compaction of store bewaPi1/Alarm/input1
2019-04-02 07:38:11.457 +00:00 [INF] - Starting compaction of store bewaPi1/OpcDataToMessage/input1
2019-04-02 07:38:14.015 +00:00 [INF] - Starting compaction of store checkpoints
2019-04-02 07:38:14.924 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:38:15.287 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:38:15.301 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:38:19.147 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:38:19.662 +00:00 [INF] - Reauthenticating connected clients
2019-04-02 07:38:38.363 +00:00 [INF] - Starting refresh of device scope identities cache
2019-04-02 07:38:39.176 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:38:39.207 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:38:39.219 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:38:48.002 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:39:08.034 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:39:08.050 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:39:08.055 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:39:13.933 +00:00 [WRN] - Encountered an error while refreshing the device scope identities cache. Will retry the operation in some time...
2019-04-02 07:39:24.883 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:39:44.919 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:39:44.930 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:39:44.940 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:40:22.064 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:40:26.066 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:40:26.067 +00:00 [INF] - Token updated for bewaPi1/Support
2019-04-02 07:40:42.099 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:40:42.130 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:40:42.139 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:41:00.042 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint bewaPi1/Support/input1
2019-04-02 07:41:00.044 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint bewaPi1/Support/input1 and 0 messages from message store.
2019-04-02 07:41:30.051 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub
2019-04-02 07:41:31.959 +00:00 [INF] - Cleaned up 854 messages from queue for endpoint iothub and 854 messages from message store.
2019-04-02 07:41:42.142 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:42:01.963 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint bewaPi1/Alarm/input1
2019-04-02 07:42:02.330 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:42:02.335 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__278_1(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-02 07:42:02.348 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-02 07:42:02.389 +00:00 [INF] - Cleaned up 34 messages from queue for endpoint bewaPi1/Alarm/input1 and 34 messages from message store.
2019-04-02 07:42:32.392 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint bewaPi1/OpcDataToMessage/input1
2019-04-02 07:42:34.468 +00:00 [INF] - Cleaned up 960 messages from queue for endpoint bewaPi1/OpcDataToMessage/input1 and 960 messages from message store.
2019-04-02 07:43:02.344 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-02 07:43:02.917 +00:00 [INF] - Exiting disconnected state
2019-04-02 07:43:02.917 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
2019-04-02 07:43:02.917 +00:00 [INF] - Processing subscriptions for client bewaPi1/Webserver.
2019-04-02 07:43:02.918 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/Webserver via AMQP...
2019-04-02 07:43:02.937 +00:00 [INF] - Cloud connection for bewaPi1/OpcUa is True
2019-04-02 07:43:02.938 +00:00 [INF] - Connection status for bewaPi1/OpcUa changed to ConnectionEstablished
2019-04-02 07:43:02.938 +00:00 [INF] - Cloud connection for bewaPi1/Webserver is True
2019-04-02 07:43:02.938 +00:00 [INF] - Connection status for bewaPi1/Webserver changed to ConnectionEstablished
2019-04-02 07:43:02.938 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.939 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.939 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.939 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcUa via AMQP...
2019-04-02 07:43:02.939 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.940 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.941 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.941 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.942 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.943 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.944 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.944 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.945 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.945 +00:00 [INF] - Cloud connection for bewaPi1/OpcDataToMessage is True
2019-04-02 07:43:02.946 +00:00 [INF] - Connection status for bewaPi1/OpcDataToMessage changed to ConnectionEstablished
2019-04-02 07:43:02.946 +00:00 [INF] - Entering connected state
2019-04-02 07:43:02.946 +00:00 [INF] - Created cloud proxy for client bewaPi1/OpcDataToMessage via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:02.947 +00:00 [INF] - Initialized cloud proxy d5e00884-905f-436b-991c-932f6aaecd0f for bewaPi1/OpcDataToMessage
2019-04-02 07:43:02.947 +00:00 [INF] - Created cloud connection for client bewaPi1/OpcDataToMessage
2019-04-02 07:43:03.045 +00:00 [INF] - Cloud connection for bewaPi1/Webserver is True
2019-04-02 07:43:03.046 +00:00 [INF] - Connection status for bewaPi1/Webserver changed to ConnectionEstablished
2019-04-02 07:43:03.046 +00:00 [INF] - Created cloud proxy for client bewaPi1/Webserver via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.046 +00:00 [INF] - Initialized cloud proxy 166683e9-2019-4537-9e38-601417bc4e45 for bewaPi1/Webserver
2019-04-02 07:43:03.046 +00:00 [INF] - Created cloud connection for client bewaPi1/Webserver
2019-04-02 07:43:03.047 +00:00 [INF] - Processing subscription Methods for client bewaPi1/Webserver.
2019-04-02 07:43:03.054 +00:00 [INF] - Cloud connection for bewaPi1/OpcUa is True
2019-04-02 07:43:03.054 +00:00 [INF] - Connection status for bewaPi1/OpcUa changed to ConnectionEstablished
2019-04-02 07:43:03.054 +00:00 [INF] - Created cloud proxy for client bewaPi1/OpcUa via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.055 +00:00 [INF] - Initialized cloud proxy cf360c87-a5db-4c6f-8c9d-311d19896b34 for bewaPi1/OpcUa
2019-04-02 07:43:03.055 +00:00 [INF] - Created cloud connection for client bewaPi1/OpcUa
2019-04-02 07:43:03.131 +00:00 [INF] - Processing subscriptions for client bewaPi1/Alarm.
2019-04-02 07:43:03.131 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/Alarm via AMQP...
2019-04-02 07:43:03.252 +00:00 [INF] - Cloud connection for bewaPi1/Alarm is True
2019-04-02 07:43:03.252 +00:00 [INF] - Connection status for bewaPi1/Alarm changed to ConnectionEstablished
2019-04-02 07:43:03.255 +00:00 [INF] - Created cloud proxy for client bewaPi1/Alarm via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.256 +00:00 [INF] - Initialized cloud proxy e584f6d3-522b-49df-941b-70e675cbb2ec for bewaPi1/Alarm
2019-04-02 07:43:03.257 +00:00 [INF] - Created cloud connection for client bewaPi1/Alarm
2019-04-02 07:43:03.258 +00:00 [INF] - Processing subscription DesiredPropertyUpdates for client bewaPi1/Alarm.
2019-04-02 07:43:03.346 +00:00 [INF] - Processing subscription ModuleMessages for client bewaPi1/Alarm.
2019-04-02 07:43:03.347 +00:00 [INF] - Processing subscriptions for client bewaPi1/Support.
2019-04-02 07:43:03.348 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/Support via AMQP...
2019-04-02 07:43:03.491 +00:00 [INF] - Cloud connection for bewaPi1/Support is True
2019-04-02 07:43:03.491 +00:00 [INF] - Connection status for bewaPi1/Support changed to ConnectionEstablished
2019-04-02 07:43:03.492 +00:00 [INF] - Created cloud proxy for client bewaPi1/Support via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.492 +00:00 [INF] - Initialized cloud proxy 9b9ccbda-4c34-4232-9bea-5f3e47d21cef for bewaPi1/Support
2019-04-02 07:43:03.492 +00:00 [INF] - Created cloud connection for client bewaPi1/Support
2019-04-02 07:43:03.493 +00:00 [INF] - Processing subscription ModuleMessages for client bewaPi1/Support.
2019-04-02 07:43:03.493 +00:00 [INF] - Processing subscriptions for client bewaPi1/OpcUa.
2019-04-02 07:43:03.493 +00:00 [INF] - Processing subscription Methods for client bewaPi1/OpcUa.
2019-04-02 07:43:03.580 +00:00 [INF] - Processing subscription ModuleMessages for client bewaPi1/OpcUa.
2019-04-02 07:43:03.580 +00:00 [INF] - Processing subscriptions for client bewaPi1/OpcDataToMessage.
2019-04-02 07:43:03.581 +00:00 [INF] - Processing subscription DesiredPropertyUpdates for client bewaPi1/OpcDataToMessage.
2019-04-02 07:43:03.581 +00:00 [INF] - Processing subscription ModuleMessages for client bewaPi1/OpcDataToMessage.
2019-04-02 07:43:03.582 +00:00 [INF] - Processing subscriptions for client bewaPi1/$edgeHub.
2019-04-02 07:43:03.582 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/$edgeHub via AMQP...
2019-04-02 07:43:03.706 +00:00 [INF] - Cloud connection for bewaPi1/$edgeHub is True
2019-04-02 07:43:03.707 +00:00 [INF] - Connection status for bewaPi1/$edgeHub changed to ConnectionEstablished
2019-04-02 07:43:03.707 +00:00 [INF] - Created cloud proxy for client bewaPi1/$edgeHub via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.707 +00:00 [INF] - Initialized cloud proxy 44ca820f-e9b2-4975-bbac-0a3ca78e3cb2 for bewaPi1/$edgeHub
2019-04-02 07:43:03.708 +00:00 [INF] - Created cloud connection for client bewaPi1/$edgeHub
2019-04-02 07:43:03.708 +00:00 [INF] - Processing subscription DesiredPropertyUpdates for client bewaPi1/$edgeHub.
2019-04-02 07:43:03.786 +00:00 [INF] - Processing subscription Methods for client bewaPi1/$edgeHub.
2019-04-02 07:43:03.870 +00:00 [INF] - Processing subscriptions for client bewaPi1/publisher.
2019-04-02 07:43:03.871 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/publisher via AMQP...
2019-04-02 07:43:03.989 +00:00 [INF] - Cloud connection for bewaPi1/publisher is True
2019-04-02 07:43:03.989 +00:00 [INF] - Connection status for bewaPi1/publisher changed to ConnectionEstablished
2019-04-02 07:43:03.989 +00:00 [INF] - Created cloud proxy for client bewaPi1/publisher via AMQP, with client operation timeout 20 seconds.
2019-04-02 07:43:03.989 +00:00 [INF] - Initialized cloud proxy c6dfd585-4da1-4ce7-8781-c7d9755d3164 for bewaPi1/publisher
2019-04-02 07:43:03.990 +00:00 [INF] - Created cloud connection for client bewaPi1/publisher
2019-04-02 07:43:03.990 +00:00 [INF] - Processing subscription Methods for client bewaPi1/publisher.
2019-04-02 07:43:19.664 +00:00 [INF] - Reauthenticating connected clients
2019-04-02 07:48:19.666 +00:00 [INF] - Reauthenticating connected clients
2019-04-02 07:48:25.074 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:48:25.076 +00:00 [INF] - Token updated for bewaPi1/publisher
2019-04-02 07:48:26.061 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:48:26.063 +00:00 [INF] - Token updated for bewaPi1/OpcDataToMessage
2019-04-02 07:48:28.072 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:48:28.073 +00:00 [INF] - Token updated for bewaPi1/Alarm
2019-04-02 07:48:28.075 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:48:28.076 +00:00 [INF] - Token updated for bewaPi1/OpcUa
2019-04-02 07:48:35.091 +00:00 [INF] - New token received on the Cbs link
2019-04-02 07:48:35.093 +00:00 [INF] - Token updated for bewaPi1/Webserver
Get Module Info in Azure IotHub Extension in VS Code provides:
"connectionState": "Disconnected",
"connectionStateUpdatedTime": "2019-04-02T07:38:35.7239759",
"lastActivityTime": "0001-01-01T00:00:00",
The time of disconnecting seems to be correct.
Any idea why the "lastActivityTime" is not set ?
After restarting the edgeAgent with sudo iotedge restart edgeAgent it reconnected to IotHub.
BewaControl-ReneDivossen
on 2 Apr 2019
@BewaControl-ReneDivossen - there seem to be 2 issues here -
- EdgeHub lost some data collected when the edge device was disconnected.
This should not happen - we have tested this several times. I will investigate what might be going on here. But meanwhile, can you please give me some more info -
- Are you mounting a volume in EdgeHub to store the data?
- If not, can you please confirm that the EdgeHub module was _not_ removed during the repro timeframe? Say after the dns was set?
- Is this issue consistently reproducable? Do you see it in all your tests when the network goes away?
- About the EdgeAgent not connecting when the network resumes -
EdgeAgent and EdgeHub use similar mechanisms to connect to IoTHub. So again, this should not happen. Was the EdgeAgent container re-created/re-started after the dns setting was added to docker?
My guess would be that the dns setting did not apply to it somehow.
As for the lastActivityTime - it is set by IoTHub based on the D2C and C2D messages activity. Since EdgeAgent does not send any of those messages, the property is never set.
varunpuranik
on 2 Apr 2019
@varunpuranik -
Towards 1:
I use a storage folder outside of the edgeHub docker container. I'm mounting a folder to store the messages like this:
In createOptions -> HostConfig section of edgeHub I have
"Binds": [
"/iotedge/storage/:/iotedge/storage/"
]
and in the env section I have
"storageFolder": {
"value": "/iotedge/storage/"
}
I'll try to find some time to do more testing, to see if loosing data can be reproduced.
Towards 2:
After setting the dns in the deamon.json I restarted docker like this
sudo systemctl restart docker
This restarted all the modules - at least I think I remember that the uptime displayed with
sudo iotedge list
was reset for all the modules (including the edgeAgent).
What do you mean by "Has the EdgeAgent container been recreated" ? I did not remove it. Are there any other actions that would cause a recreation of the container ?
BewaControl-ReneDivossen
on 3 Apr 2019
@varunpuranik, @veyalla
The edgeAgent again did not connect anymore after network failure.
using docker inspect I realized that the DNS setting I provided in daemon.json was not transferred to the edgeAgent container. The same was true for all other modules.
My daemon.json looks like this:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "5m",
"max-file": "2"
},
"dns": ["1.1.1.1"]
}
The logging settings are visible in the docker inspect output but the dns setting is not!
I also tried
- restarting docker
- restarting iotedge
- removing all docker images and containers
all of this did not work.
Finally I added the dns setting to the createOptions ->HostConfig section of all modules and finally the dns setting was visible in the docker inspect output.
After that I provoked a network failure again but unfortunately the edgeAgent once again could not reconnect! That's when I gave up for today - this is very frustrating!
BewaControl-ReneDivossen
on 3 Apr 2019
Setting "dns" in the daemon.json will apply to all containers started by the engine. However, it won't show up in the docker inspect output. You can confirm that it worked by doing docker exec -it <module-name> cat /etc/resolv.conf
What is the test you are using to determine that edgeAgent is not connecting? Like Varun mentioned lastActivityTime is not reliable. We're looking to improve status reporting to IoT Hub in the upcoming release.
veyalla
on 3 Apr 2019
@veyalla - thanks for the info about the dns setting behaviour. This keeps me wondering why some daemon.json settings (e.g. logging) are reflected in the docker inspect output and others (e.g. dns) are not. Such inconsistencies are very confusing.
I did understand Varuns comment about the edgeAgent not setting the lastActivityTime so I'm not using that to determine if the edgeAgent is connected or not.
First I'm looking at the Modules in the Azure IotHub Extension for VSCode. After the network connection recovers, all modules except for the edgeAgent are marked green and labeled "Connected(running)". The edgeAgent module is marked blue though. If I right click on edgeAgent and select "Get Module Info" from the context menu, the "connectionState" is displayed as "Disconnected" and the "connectionStateUpdatedTime" is set to the time of the begin of the network failure.
Is there any other way to test if the the edgeAgent is connected or not?
BewaControl-ReneDivossen
on 4 Apr 2019
@veyalla, @varunpuranik
This is continuously getting more confusing:
In the edgeAgent documentation it says:
The IoT Edge agent is the other module that makes up the Azure IoT Edge runtime. It is responsible for instantiating modules, ensuring that they continue to run, and reporting the status of the modules back to IoT Hub.
If the edgeAgent is reporting the status of all modules and the status of all modules (except for the edgeAgent itself) is reported as "Connected(running)" doesn't that mean, the edgeAgent is connected as well because it was able to report that status ??? So communication from edgeAgent to cloud seems to work.
To test the communication from cloud to edgeAgent I made some change in the deployment.template and deployed to the edgeDevice again. In the logging output of edgeAgent I could see that it did perform the deployment tasks.
So it looks like the edgeAgent did in fact reconnect to the cloud but it was not able to update and report its own status correctly!
BewaControl-ReneDivossen
on 4 Apr 2019
@veyalla
I just used the recommended docker exec -it <module-name> cat /etc/resolv.conf to check if the dns setting is applied to the modules.
Unfortunately the answer is NO.
The output (for all modules) is:
nameserver 127.0.0.11
options ndots:0
If that command displays which nameserver is really used, then neither defining the dns setting in the daemon.json nor defining it via createOptions is working!
BewaControl-ReneDivossen
on 4 Apr 2019
@veyalla
In the docker documentation it says
The IP addresses passed via the --dns option is used by the embedded DNS server to forward the DNS query if embedded DNS server is unable to resolve a name resolution request from the containers. These --dns IP addresses are managed by the embedded DNS server and will not be updated in the container's /etc/resolv.conf file.
and
Note: The DNS server is always at 127.0.0.11.
BewaControl-ReneDivossen
on 4 Apr 2019
@veyalla, @varunpuranik,
can you provide any more insight on this issue?
Or do I have to wait for the 1.0.7 release and pray for it to work reliably?
BewaControl-ReneDivossen
on 8 Apr 2019
I checked last week for edgeAgent and it reflected the custom DNS address in its resolv.conf
However, this may be dependent on whether the agent was started on the default vs the container network. If the deployment has a different edgeAgent version than the config.yaml then it is put in the container network. That might explain the difference in behavior.
In the end, if “nslookup
veyalla
on 8 Apr 2019
Since the new IotHub SDK has been released by now, I decided to give it another try.
I referenced Microsoft.Azure.Devices.Client 1.20.0 in all my modules and updated edgeAgent and edgeHub to 1.0.7-rc2. Then I provoked another network failure. At first it looked like none of the modules where able to reconnect afterwards but after a few minutes all modules (including the edgeAgent) were reported Connected(running) again!
I'll keep an eye on the issue but at least it lookes like we are making progress :-)
BewaControl-ReneDivossen
on 17 Apr 2019
I tried a few more times and so far the modules always reconnected - but reconnecting takes about 15 minutes! Before that the log shows a lot of errors like this:
2019-04-17 14:19:33.405 +00:00 [INF] - Starting refresh of device scope identities cache
2019-04-17 14:19:41.401 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/OpcDataToMessage via AMQP...
2019-04-17 14:20:18.268 +00:00 [INF] - Reauthenticating connected clients
2019-04-17 14:20:41.370 +00:00 [INF] - Attempting to connect to IoT Hub for client bewaPi1/$edgeHub via AMQP...
2019-04-17 14:20:41.462 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/OpcDataToMessage
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: The operation did not complete within the allocated time 00:01:00 for object session210. ---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object session210.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpUnit.OpenAsync(TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass22_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-17 14:20:41.494 +00:00 [INF] - Error getting cloud connection for device bewaPi1/OpcDataToMessage
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: The operation did not complete within the allocated time 00:01:00 for object session210. ---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object session210.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpUnit.OpenAsync(TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass22_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-17 14:20:41.503 +00:00 [WRN] - Could not get an active Iot Hub connection for device bewaPi1/OpcDataToMessage
2019-04-17 14:21:41.400 +00:00 [WRN] - Error creating cloud connection for client bewaPi1/$edgeHub
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: The operation did not complete within the allocated time 00:01:00 for object session211. ---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object session211.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpUnit.OpenAsync(TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass22_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
2019-04-17 14:21:41.420 +00:00 [INF] - Error getting cloud connection for device bewaPi1/$edgeHub
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: The operation did not complete within the allocated time 00:01:00 for object session211. ---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object session211.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpUnit.OpenAsync(TimeSpan timeout)
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass22_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper.OpenAsync() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ModuleClientWrapper.cs:line 48
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.ConnectToIoTHub(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 125
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.CreateNewCloudProxyAsync(ITokenProvider newTokenProvider) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 97
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.Create(IIdentity identity, Action`2 connectionStatusChangedHandler, ITransportSettings[] transportSettings, IMessageConverterProvider messageConverterProvider, IClientProvider clientProvider, ICloudListener cloudListener, ITokenProvider tokenProvider, TimeSpan idleTimeout, Boolean closeOnIdleTimeout, TimeSpan operationTimeout) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 86
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.<>c__DisplayClass15_1.<<Connect>b__2>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 134
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider.Connect(IIdentity identity, Action`2 connectionStatusChangedHandler) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnectionProvider.cs:line 129
We're encountering the same issue at the moment. Whenever there is a network failure it takes a considerable amount of time before we get a connection with the cloud again. Is there any update on this problem?
corollaries
on 3 Jun 2019
The same huge delay before reconnecting also occurs in our tests. This issue is a show-stopper for a platform that is operated via the cloud! Especially when the products developed on top of it need either human interaction (you can't/shouldn't expect a user to wait 15 minutes) or are limited in storage (thus requiring a cloud connection to upload data).
@veyalla I hope that you can prioritize releasing a fix as soon as possible.
Maher4Ever
on 3 Jun 2019
👍1
Which version of IoT Edge are you using (recommend 1.0.7.1) and what is the upstream protocol (default is amqp)?
veyalla
on 3 Jun 2019
@veyalla We're using 1.0.7.1 and use MqttWs as the upstream protocol. (We tried several other upstream protocols but had the same problem with those as well)
corollaries
on 4 Jun 2019
👍1
@corollaries / @Maher4Ever - Can you please describe your setup? Specifically -
- What device (OS and H/w)
- How was the network disconnected / reconnected
- What workload (custom modules)
- Any other specific config that you used (such as MqttWs upstream protocol)
We have seen this issue before in some isolated scenarios. If you have a reproducable setup, I would like to try on that, to help us investigate.
varunpuranik
on 5 Jun 2019
@varunpuranik
OS: Ubuntu 18.04 w/ kernel 4.18.0-20-generic
HW: Lanner LEC7233
Tests
We tested the issue in two ways:
- Bringing network interface down and up by restarting
systemd-networkdafter configuring IP settings. (We stopped usingsystemd-networkdbecause of this issue and moved to NetworkManager which does not bring all interfaces down). - Taking out the ethernet cable and reconnecting it after some time.
We noticed a few things after testing this multiple times:
- We see that it takes a considerable amount of time before the IoT Hub detects the device is not online anymore.
- After reconnecting it takes around 10 minutes before the device is connected again. In this 10 minutes we also see an exception occurring:
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry. Not sure if the exception is related to this problem.
Workload
Workload is two custom IoT Edge modules written with the IoT Edge Node SDK. They use very little resources as they're mostly standby listening for changes in the module twin.
Custom configuration
We did adjust the daemon.json of docker. The default IP address is conflicting with our private network so we changed it by setting the bip to a different address. (Note that we also tested it on a public network but we still have the same issues there as well)
corollaries
on 5 Jun 2019
I just installed the runtime version 1.0.7.1 and edgeAgent + edgeHub 1.0.8-rc1
I disconnected the network cable for 30 minutes and then plugged it in again. The edgeDevice (Raspberry Pi) reconnected to IotHub in about one minute. So with the latest and greatest a first quick test was successfull.
BewaControl-ReneDivossen
on 5 Jun 2019
We ran the same tests but unfortunately this has not fixed our issue. We disconnected for 10/15 minutes and every time we reconnect it would take longer than 10 minutes with the 1.0.8-rc1 version of edgeAgent and edgeHub.
corollaries
on 9 Jun 2019
This issue is being marked as stale because it has been open for 30 days with no activity.
![github-actions[bot] picture](https://avatars.githubusercontent.com/in/15368?v=4&s=40)
github-actions[bot]
on 3 Nov 2019
This is fixed with 1.0.8 release coming out by end of this week. Please give it a try when available and re-open issue as needed.
lt72
on 18 Dec 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings
Related issues
arabold
·
3Comments
prashbv
·
6Comments
asergaz
·
4Comments
yenseaman
·
5Comments
mathiash98
·
4Comments
Most helpful comment
This is fixed with 1.0.8 release coming out by end of this week. Please give it a try when available and re-open issue as needed.