Iotedge: Modules never reconnect after updating from 1.0.x to 1.1.x

Created on 30 Apr 2021 · 17Comments · Source: Azure/iotedge

Expected Behavior

Modules reconnect to edgeHub after the new 1.1 container has started.

Current Behavior

All our modules fail and never reconnect to the edgeHub. The silent failure does not exit the container, so we never self heal.

Steps to Reproduce

Update security daemon

SSH to host
Follow instructions on how to update security daemon
When installing new versions, use command sudo apt-get install iotedge=1.1.1* libiothsm-std=1.1.1*

Manually create host directory for edgeHub

mkdir -p /var/lib/edgeHub && \ 
  sudo chown 1000 /var/lib/edgeHub && \ 
  sudo chmod /var/lib/edgeHub

Deploy new edge container versions

We have an edge deployment schema in git, and we just simply sed our versions and apply the schema. Here's a redacted version:

{
  "content": {
    "modulesContent": {
      "$edgeAgent": {
        "properties.desired": {
          "schemaVersion": "1.1",
          "runtime": {
            "type": "docker",
            "settings": {
              "minDockerVersion": "v1.25",
              "loggingOptions": "", // Defined on a module basis
              "registryCredentials": {
                "aregistry": {
                  "username": "aRegistry",
                  "password": "<A_REGISTRY_PASSWORD>",
                  "address": "aregistry.azurecr.io"
                }
              }
            }
          },
          "systemModules": {
            "edgeAgent": {
              "type": "docker",
              "settings": {
                "image": "mcr.microsoft.com/azureiotedge-agent:1.1",
                "createOptions": "{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeAgent/:/tmp/edgeAgent\"],\"LogConfig\":{\"Type\":\"json-file\",\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"}}}}"
              }
            },
            "edgeHub": {
              "type": "docker",
              "status": "running",
              "restartPolicy": "always",
              "startupOrder": 1,
              "settings": {
                "image": "mcr.microsoft.com/azureiotedge-hub:1.1",
                "createOptions": "{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeHub/:/tmp/edgeHub\"],\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]},\"LogConfig\":{\"Type\":\"json-file\",\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"}}}}"
              }
            }
          },
          "modules": {
            // 5 of our own modules
          }
        }
      },
      "$edgeHub": {
        "properties.desired": {
          "schemaVersion": "1.1",
          "routes": {
            "upstream": {
              "route": "FROM /messages/* INTO $upstream",
              "priority": 0,
              "timeToLiveSecs": 31536000
            }
          },
          "storeAndForwardConfiguration": {
            "timeToLiveSecs": 30
          }
        }
      }
    }
  }
}

This is then deployed using from bitbucket using:

az iot edge deployment create --hub-name $HUB_NAME --deployment-id edge-deployment-$BITBUCKET_BUILD_NUMBER --content deployment-manifest.json --priority 0 --target-condition "tags.environment = 'dev' AND tags.type = 'type'" --subscription $AZURE_SUBSCRIPTION_ID

Context (Environment)

Output of `iotedge check`

Click here

$ sudo iotedge check
Configuration checks
--------------------
√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: certificates - Warning
    The Edge device is using self-signed automatically-generated development certificates.
    They will expire in 23 days (at 2021-05-23 12:22:11 UTC) causing module-to-module and downstream device communication to fail on an active deployment.
    After the certs have expired, restarting the IoT Edge daemon will trigger it to generate new development certs.
    Please consider using production certificates instead. See https://aka.ms/iotedge-prod-checklist-certs for best practices.
√ production readiness: container engine - OK
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the default network can connect to IoT Hub AMQP port - OK
√ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the default network can connect to IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK

20 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.

Device Information

Host OS Ubuntu 18.04.5 LTS
Architecture amd64
Container OS Linux containers

Runtime Versions

aziot-edged 1.1.1
Edge Agent 1.1 rolling tag
Edge Hub 1.1 rolling tag
Docker/Moby
- Client 19.03.13+azure
- Server
- Engine 19.03.13+azure
- containerd 1.3.7+azure
- runc 1.0.0-rc92
- docker-init 0.18.0

Logs

our container logs

2021-04-28 09:25:53,268 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(720) - Amqp CBS session closed unexpectedly. Closing this connection...
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpLinkDetachForcedException: Server busy, please retry operation
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:38)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(AmqpsIotHubConnection.java:719)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsCbsSessionHandler.onLinkClosedUnexpectedly(AmqpsCbsSessionHandler.java:143)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsSenderLinkHandler.onLinkRemoteClose(AmqpsSenderLinkHandler.java:116)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:25:53,287 [azure-iot-sdk-ConnectionReconnectionTask:13738d06-ec74-4b6b-8ca5-ae604ed5a453] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-04-28 09:25:53,594 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:25:54,085 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:25:54,992 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:25:56,781 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:25:59,539 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-28 09:26:06,148 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)

Note: I cannot reproduce the error and cannot supply these logs again without updating another device from 1.0.x to 1.1
That said, when this occurred both edgeAgent and edgeHub started without any issues, it was only my containers that did not reconnect.

aziot-edged logs

<Paste here between the triple backticks>

edge-agent logs

<Paste here between the triple backticks>

edge-hub logs

<Paste here between the triple backticks>

Additional Information

We connect our modules like this in Java:

ModuleClient.createFromEnvironment(IotHubClientProtocol.AMQPS);

Usually the runtime is resilient against disconnects, like pulling the ethernet cord, disabling interfaces. I've never seen our modules _not_ reconnect before.

This would have self healed if the error produced made our containers exit.

customer-reported iotedge

Source

jokarl

All 17 comments

Issue I originally opened on the docs repo: https://github.com/MicrosoftDocs/azure-docs/issues/74473

jokarl on 30 Apr 2021

@jokarl Some qeustions:

Do you happen to have edge hub logs around the time of this problem? As a sanity check I would like to see if EdgeHub is actually up and running.
What version of the java Devices SDK are you using?
Why did you run these commands?

mkdir -p /var/lib/edgeHub && \ 
sudo chown 1000 /var/lib/edgeHub && \ 
sudo chmod /var/lib/edgeHub

This _might_ be an sdk issue, in which case we can open the issue in the java devices sdk repo.
https://github.com/Azure/azure-iot-sdk-java

and-rewsmith on 30 Apr 2021

Sorry no logs, I can't recreate this without doing the upgrade and I only have production machines left. But all devices I've done this on still works, so I can give a prod machine a go when I'm back at work.

I'll get back to you with version information, we're probably lagging behind there.

It's part of the steps to resolve production ready checks when running sudo iotedge check. I mount /tmp/edgeHub from the edgeHub container to this directory on the host.

jokarl on 30 Apr 2021

@jokarl Sounds good.

I'll get back to you with version information, we're probably lagging behind there.

It is possible that this is causing the error. Might be worth upgrading this version in your modules as these updates also probably have some security fixes. I'd recommend doing a full batch of testing before rolling that out for production.

If your modules restart, does that resolve the issue? If so, a possible mitigation would be to add failure logic in the case where the java devices sdk cannot connect.

I am also wondering what will happen if, instead of upgrading, the EdgeHub container is force removed via docker rm -f. If it reproduces in this way we know it is not specific to the upgrade scenario.

and-rewsmith on 30 Apr 2021

@and-rewsmith here's the rest of the info:

I can produce similar error logs in our containers by running sudo docker rm -f edgeHub, but our containers eventually reconnect (see below). I actually found a log snippet from an upgrade I performed but not from the edge runtime containers unfortunately. Note that I might not have waited long enough, snippet from upgrade scenario is only 2 minutes while I had to wait 10 for a reconnect after forcibly removing the container. The errors do look different though:
- Force remove causes: Amqp transport error occurred, closing the amqps connection com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
- Update scenario causes: Amqp CBS session closed unexpectedly. Closing this connection... com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpLinkDetachForcedException: Server busy, please retry operation
SDK versions from Microsoft:
- com.microsoft.azure.sdk.iot:iot-device-client:1.29.0
- com.microsoft.azure.sdk.iot.provisioning:provisioning-device-client:1.4.0
- com.microsoft.azure.sdk.iot.provisioning.security:x509-provider:1.1.3
It's part of the steps to resolve production ready checks when running sudo iotedge check. I mount /tmp/edgeHub from the edgeHub container to this directory on the host.

Logs

One of our containers when upgrading from 1.0.x to 1.1.x

2021-04-23 12:40:57,050 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(720) - Amqp CBS session closed unexpectedly. Closing this connection...
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpLinkDetachForcedException: Server busy, please retry operation
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:38)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(AmqpsIotHubConnection.java:719)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsCbsSessionHandler.onLinkClosedUnexpectedly(AmqpsCbsSessionHandler.java:143)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsSenderLinkHandler.onLinkRemoteClose(AmqpsSenderLinkHandler.java:116)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:40:57,063 [azure-iot-sdk-ConnectionReconnectionTask:fb2ac148-3b4e-41a0-97a7-2c2883b4e9f2] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-04-23 12:40:57,352 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:40:57,753 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:40:58,502 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:41:00,231 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:41:03,855 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-04-23 12:41:10,654 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)

All logs here are from _after_ running sudo docker rm -f edgeHub

One of our containers

2021-05-03 07:34:18,063 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpConnectionFramingErrorException: connection aborted
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:44)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:18,076 [azure-iot-sdk-ConnectionReconnectionTask:fc9cf057-2472-4046-bcff-0630f42c5be3] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-05-03 07:34:18,358 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:18,760 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:19,660 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:21,560 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:24,723 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:34:30,517 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-03 07:42:36,376 [azure-iot-sdk-ConnectionReconnectionTask:7295bf85-90ab-4fbe-84fe-d937b9024266] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - LockerClient connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-05-03 07:42:37,298 [azure-iot-sdk-ConnectionReconnectionTask:7295bf85-90ab-4fbe-84fe-d937b9024266] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.notifyConnect(65) - Notifying upstream about connected client of type LockerClient

edgeAgent

<6> 2021-05-03 00:06:53.337 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 00:06:53.337 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 00:06:53.337 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 00:06:53.339 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 00:06:53.349 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 00:06:53.373 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 00:06:53.373 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 00:07:01.744 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 00:07:01.838 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 00:07:01.846 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 00:08:04.532 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 00:08:04.532 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 00:08:04.637 +00:00 [INF] - Published metrics
<6> 2021-05-03 00:08:04.645 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 01:04:59.899 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 01:04:59.899 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 01:04:59.899 +00:00 [INF] - Published metrics
<6> 2021-05-03 01:04:59.899 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 01:06:44.860 +00:00 [INF] - Starting compaction of stores
<6> 2021-05-03 01:06:44.861 +00:00 [INF] - Starting compaction of store default
<6> 2021-05-03 01:06:44.861 +00:00 [INF] - Starting compaction of store moduleState
<6> 2021-05-03 01:06:44.861 +00:00 [INF] - Starting compaction of store Metrics
<6> 2021-05-03 01:06:44.861 +00:00 [INF] - Starting compaction of store deploymentConfig
<6> 2021-05-03 01:06:53.372 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 01:06:53.372 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 01:06:53.372 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 01:06:53.374 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 01:06:53.381 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 01:06:53.404 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 01:06:53.404 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 01:07:01.845 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 01:07:01.941 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 01:07:01.943 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 02:01:55.145 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 02:01:55.145 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 02:01:55.240 +00:00 [INF] - Published metrics
<6> 2021-05-03 02:01:55.248 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 02:06:53.408 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 02:06:53.408 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 02:06:53.408 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 02:06:53.410 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 02:06:53.416 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 02:06:53.436 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 02:06:53.436 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 02:07:01.947 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 02:07:02.036 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 02:07:02.042 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 02:58:50.500 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 02:58:50.500 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 02:58:50.599 +00:00 [INF] - Published metrics
<6> 2021-05-03 02:58:50.605 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 03:06:44.862 +00:00 [INF] - Starting compaction of stores
<6> 2021-05-03 03:06:44.862 +00:00 [INF] - Starting compaction of store default
<6> 2021-05-03 03:06:44.862 +00:00 [INF] - Starting compaction of store moduleState
<6> 2021-05-03 03:06:44.862 +00:00 [INF] - Starting compaction of store Metrics
<6> 2021-05-03 03:06:44.862 +00:00 [INF] - Starting compaction of store deploymentConfig
<6> 2021-05-03 03:06:53.436 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 03:06:53.437 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 03:06:53.437 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 03:06:53.439 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 03:06:53.446 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 03:06:53.469 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 03:06:53.469 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 03:07:02.040 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 03:07:02.126 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 03:07:02.130 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 03:55:45.853 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 03:55:45.853 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 03:55:46.001 +00:00 [INF] - Published metrics
<6> 2021-05-03 03:55:46.007 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 04:06:53.475 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 04:06:53.475 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 04:06:53.475 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 04:06:53.477 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 04:06:53.486 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 04:06:53.505 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 04:06:53.505 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 04:07:02.131 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 04:07:02.202 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 04:07:02.207 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 04:52:41.256 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 04:52:41.256 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 04:52:41.344 +00:00 [INF] - Published metrics
<6> 2021-05-03 04:52:41.348 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 05:06:44.862 +00:00 [INF] - Starting compaction of stores
<6> 2021-05-03 05:06:44.862 +00:00 [INF] - Starting compaction of store default
<6> 2021-05-03 05:06:44.862 +00:00 [INF] - Starting compaction of store moduleState
<6> 2021-05-03 05:06:44.863 +00:00 [INF] - Starting compaction of store Metrics
<6> 2021-05-03 05:06:44.863 +00:00 [INF] - Starting compaction of store deploymentConfig
<6> 2021-05-03 05:06:53.505 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 05:06:53.505 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 05:06:53.505 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 05:06:53.508 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 05:06:53.516 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 05:06:53.536 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 05:06:53.536 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 05:07:02.205 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 05:07:02.314 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 05:07:02.320 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 05:49:36.596 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 05:49:36.596 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 05:49:36.818 +00:00 [INF] - Published metrics
<6> 2021-05-03 05:49:36.826 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 06:06:53.537 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 06:06:53.537 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 06:06:53.537 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 06:06:53.539 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 06:06:53.550 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 06:06:53.572 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 06:06:53.572 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 06:07:02.322 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 06:07:02.423 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 06:07:02.430 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 06:46:32.077 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2021-05-03 06:46:32.077 +00:00 [INF] - Uploading Metrics
<6> 2021-05-03 06:46:32.174 +00:00 [INF] - Published metrics
<6> 2021-05-03 06:46:32.180 +00:00 [INF] - Successfully completed periodic operation Metrics Upload
<6> 2021-05-03 07:06:44.860 +00:00 [INF] - Starting compaction of stores
<6> 2021-05-03 07:06:44.861 +00:00 [INF] - Starting compaction of store default
<6> 2021-05-03 07:06:44.861 +00:00 [INF] - Starting compaction of store moduleState
<6> 2021-05-03 07:06:44.861 +00:00 [INF] - Starting compaction of store Metrics
<6> 2021-05-03 07:06:44.861 +00:00 [INF] - Starting compaction of store deploymentConfig
<6> 2021-05-03 07:06:53.574 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2021-05-03 07:06:53.574 +00:00 [INF] - Scraping Metrics
<6> 2021-05-03 07:06:53.575 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2021-05-03 07:06:53.576 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2021-05-03 07:06:53.582 +00:00 [INF] - Storing Metrics
<6> 2021-05-03 07:06:53.608 +00:00 [INF] - Scraped and Stored Metrics
<6> 2021-05-03 07:06:53.608 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
<6> 2021-05-03 07:07:02.428 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2021-05-03 07:07:02.503 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 07:07:02.510 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-03 07:32:33.558 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2374.
<6> 2021-05-03 07:34:22.375 +00:00 [INF] - Plan execution started for deployment 124
<6> 2021-05-03 07:34:22.376 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2021-05-03 07:34:22.376 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2021-05-03 07:34:23.660 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2021-05-03 07:34:24.696 +00:00 [INF] - Plan execution ended for deployment 124
<6> 2021-05-03 07:34:25.940 +00:00 [INF] - Updated reported properties
<6> 2021-05-03 07:34:31.188 +00:00 [INF] - Updated reported properties
<6> 2021-05-03 07:34:34.076 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2376.
<6> 2021-05-03 07:37:19.735 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2376.
<6> 2021-05-03 07:40:13.036 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 124 and reported properties version 2376.

edgeHub

<6> 2021-05-03 07:34:28.110 +00:00 [INF] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-03 07:34:28.111 +00:00 [INF] - OptimizeForPerformance=True
<6> 2021-05-03 07:34:28.111 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2021-05-03 07:34:28.112 +00:00 [INF] - Loaded server certificate with expiration date of "2021-06-07T08:09:04.0000000+00:00"
<6> 2021-05-03 07:34:28.128 +00:00 [INF] - Using Asp Net server for metrics
<6> 2021-05-03 07:34:28.224 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2021-05-03 07:34:28.224 +00:00 [INF] - Created new message store
<6> 2021-05-03 07:34:28.394 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2021-05-03 07:34:28.540 +00:00 [INF] - Created device scope identities cache
<6> 2021-05-03 07:34:28.551 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2021-05-03 07:34:28.578 +00:00 [INF] - Initialized storing twin manager
<6> 2021-05-03 07:34:28.586 +00:00 [INF] - Initializing configuration
<6> 2021-05-03 07:34:28.618 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:28.653 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2021-05-03 07:34:28.738 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub via AMQP...
<6> 2021-05-03 07:34:28.774 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:29.822 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - Set the following 24 route(s) in edge hub
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - upstream: FROM /messages/* INTO $upstream
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsCancelSessionRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/session_cancel INTO BrokeredEndpoint("/modules/lockerParcel/inputs/session_cancel")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsErrorEventToWs: FROM /messages/modules/lockerWebsocket/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsGetLayoutRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/locker_get-layout INTO BrokeredEndpoint("/modules/lockerParcel/inputs/locker_get-layout")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsLargerSlotRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_request-bigger INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_request-bigger")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsPackageRetrievedToLocker: FROM /messages/modules/lockerWebsocket/outputs/package_retrieved INTO BrokeredEndpoint("/modules/lockerParcel/inputs/package_retrieved")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsResolveRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsSlotUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsSlotLoadedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_loaded INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_loaded")
<6> 2021-05-03 07:34:30.326 +00:00 [INF] - wsOverdueRemovedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_overdue-removed INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_overdue-removed")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - wsSlotsUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_unlock")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - wsSlotsLockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_lock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_lock")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerCancelSessionResponseToWs: FROM /messages/modules/lockerParcel/outputs/session_canceled INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/session_canceled")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerConfigurationStartToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-start INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-start")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerConfigurationEndToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-end INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-end")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerConfigurationUpdateToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-update INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-update")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerErrorEventToWs: FROM /messages/modules/lockerParcel/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerGetLayoutResponseToWs: FROM /messages/modules/lockerParcel/outputs/locker_layout INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_layout")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerLargerSlotResponseToWs: FROM /messages/modules/lockerParcel/outputs/slot_response-bigger INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_response-bigger")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerResolveResponseToWs: FROM /messages/modules/lockerParcel/outputs/code_resolved INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/code_resolved")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerSlotStateChangeToWs: FROM /messages/modules/lockerParcel/outputs/slot_state-change INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_state-change")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - lockerSlotUnlockToLocker: FROM /messages/modules/lockerParcel/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - scannerErrorEventToWs: FROM /messages/modules/lockerScanner/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:30.327 +00:00 [INF] - scannerResolveRequestToLocker: FROM /messages/modules/lockerScanner/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-03 07:34:30.331 +00:00 [INF] - Updated message store TTL to 30 seconds
<6> 2021-05-03 07:34:30.332 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-03 07:34:30.375 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2021-05-03 07:34:30.376 +00:00 [INF] - Initialized edge hub configuration
<6> 2021-05-03 07:34:30.389 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2021-05-03 07:34:30.517 +00:00 [INF] - Scheduling server certificate renewal for "2021-06-07T08:06:34.0004324Z".
<6> 2021-05-03 07:34:30.520 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2021-05-03 07:34:30.526 +00:00 [INF] - Starting MQTT head
<6> 2021-05-03 07:34:30.579 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2021-05-03 07:34:30.629 +00:00 [INF] - Exiting disconnected state
<6> 2021-05-03 07:34:30.662 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2021-05-03 07:34:30.665 +00:00 [INF] - Processing subscriptions for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub on device connected to cloud.
<6> 2021-05-03 07:34:30.666 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:30.673 +00:00 [INF] - Received device connected callback
<6> 2021-05-03 07:34:30.681 +00:00 [INF] - Starting AMQP head
<6> 2021-05-03 07:34:30.689 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2021-05-03 07:34:30.809 +00:00 [INF] - Entering connected state
<6> 2021-05-03 07:34:30.811 +00:00 [INF] - Started MQTT head
<6> 2021-05-03 07:34:30.825 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub is True
<6> 2021-05-03 07:34:30.831 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub changed to ConnectionEstablished
<6> 2021-05-03 07:34:30.840 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2021-05-03 07:34:30.840 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:30.864 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-03 07:34:30.886 +00:00 [INF] - Initialized cloud proxy 616cbfd3-3e14-421a-ad3f-2529c8110692 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:30.889 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:30.906 +00:00 [INF] - Started AMQP head
<6> 2021-05-03 07:34:30.911 +00:00 [INF] - Starting HTTP head
<6> 2021-05-03 07:34:31.373 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:31.411 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:31.456 +00:00 [INF] - New token received on the Cbs link
<4> 2021-05-03 07:34:31.483 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2021-05-03 07:34:31.511 +00:00 [INF] - Started HTTP head
<6> 2021-05-03 07:34:31.829 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner in device scope authenticated locally.
<6> 2021-05-03 07:34:31.846 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2021-05-03 07:34:31.880 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.883 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner connected to edgeHub, processing existing subscriptions.
<6> 2021-05-03 07:34:31.913 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.915 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.931 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.939 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner via AMQP...
<6> 2021-05-03 07:34:31.948 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.956 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.957 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.962 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - Set the following 24 route(s) in edge hub
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - upstream: FROM /messages/* INTO $upstream
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsCancelSessionRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/session_cancel INTO BrokeredEndpoint("/modules/lockerParcel/inputs/session_cancel")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsErrorEventToWs: FROM /messages/modules/lockerWebsocket/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsGetLayoutRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/locker_get-layout INTO BrokeredEndpoint("/modules/lockerParcel/inputs/locker_get-layout")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsLargerSlotRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_request-bigger INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_request-bigger")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsPackageRetrievedToLocker: FROM /messages/modules/lockerWebsocket/outputs/package_retrieved INTO BrokeredEndpoint("/modules/lockerParcel/inputs/package_retrieved")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsResolveRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsSlotUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsSlotLoadedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_loaded INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_loaded")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsOverdueRemovedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_overdue-removed INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_overdue-removed")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsSlotsUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_unlock")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - wsSlotsLockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_lock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_lock")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - lockerCancelSessionResponseToWs: FROM /messages/modules/lockerParcel/outputs/session_canceled INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/session_canceled")
<6> 2021-05-03 07:34:31.984 +00:00 [INF] - lockerConfigurationStartToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-start INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-start")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerConfigurationEndToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-end INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-end")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerConfigurationUpdateToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-update INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-update")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerErrorEventToWs: FROM /messages/modules/lockerParcel/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerGetLayoutResponseToWs: FROM /messages/modules/lockerParcel/outputs/locker_layout INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_layout")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerLargerSlotResponseToWs: FROM /messages/modules/lockerParcel/outputs/slot_response-bigger INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_response-bigger")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerResolveResponseToWs: FROM /messages/modules/lockerParcel/outputs/code_resolved INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/code_resolved")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerSlotStateChangeToWs: FROM /messages/modules/lockerParcel/outputs/slot_state-change INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_state-change")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - lockerSlotUnlockToLocker: FROM /messages/modules/lockerParcel/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - scannerErrorEventToWs: FROM /messages/modules/lockerScanner/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - scannerResolveRequestToLocker: FROM /messages/modules/lockerScanner/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-03 07:34:31.985 +00:00 [INF] - Updated message store TTL to 30 seconds
<6> 2021-05-03 07:34:31.986 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-03 07:34:32.192 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner is True
<6> 2021-05-03 07:34:32.192 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner changed to ConnectionEstablished
<6> 2021-05-03 07:34:32.192 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner connected to cloud, processing existing subscriptions.
<6> 2021-05-03 07:34:32.192 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:32.193 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-03 07:34:32.193 +00:00 [INF] - Initialized cloud proxy 8c018488-9760-492a-835d-d579a7fd8a72 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:32.193 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:32.271 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-03 07:34:33.859 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:37.470 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser in device scope authenticated locally.
<6> 2021-05-03 07:34:37.482 +00:00 [INF] - Successfully generated identity for clientId parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser and username cnc-dev.azure-devices.net/parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser/?api-version=2019-10-01&DeviceClientType=azure-iot-device%2F1.17.1%20(node%20v10.19.0%3B%20Ubuntu%2020.04%3B%20x64)
<6> 2021-05-03 07:34:37.493 +00:00 [INF] - ClientAuthenticated, parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser, 17392044
<6> 2021-05-03 07:34:37.569 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.570 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser connected to edgeHub, processing existing subscriptions.
<6> 2021-05-03 07:34:37.577 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.579 +00:00 [INF] - Binding message channel for device Id parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.635 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, Methods for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser.
<6> 2021-05-03 07:34:37.635 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.650 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser via AMQP...
<6> 2021-05-03 07:34:37.664 +00:00 [INF] - Set subscriptions from session state for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.925 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser is True
<6> 2021-05-03 07:34:37.925 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser changed to ConnectionEstablished
<6> 2021-05-03 07:34:37.925 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser connected to cloud, processing existing subscriptions.
<6> 2021-05-03 07:34:37.925 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:37.926 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-03 07:34:37.926 +00:00 [INF] - Initialized cloud proxy 296d9efc-e848-4023-ab75-99c39c588667 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:37.926 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-03 07:34:39.303 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:34:39.803 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-03 07:34:39.813 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket in device scope authenticated locally.
<6> 2021-05-03 07:34:39.815 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.817 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket connected to edgeHub, processing existing subscriptions.
<6> 2021-05-03 07:34:39.823 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket via AMQP...
<6> 2021-05-03 07:34:39.830 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.830 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.831 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.853 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.860 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.860 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:39.862 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:40.122 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket is True
<6> 2021-05-03 07:34:40.122 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket changed to ConnectionEstablished
<6> 2021-05-03 07:34:40.122 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket connected to cloud, processing existing subscriptions.
<6> 2021-05-03 07:34:40.122 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:40.122 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-03 07:34:40.123 +00:00 [INF] - Initialized cloud proxy e43439be-eaf0-47a4-b481-47aa93adfd31 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:40.123 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:40.204 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-03 07:34:40.645 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-03 07:34:40.668 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel in device scope authenticated locally.
<6> 2021-05-03 07:34:40.672 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.673 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel connected to edgeHub, processing existing subscriptions.
<6> 2021-05-03 07:34:40.676 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel via AMQP...
<6> 2021-05-03 07:34:40.694 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.694 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.696 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.705 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.708 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.724 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.728 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.967 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel is True
<6> 2021-05-03 07:34:40.967 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel changed to ConnectionEstablished
<6> 2021-05-03 07:34:40.968 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel connected to cloud, processing existing subscriptions.
<6> 2021-05-03 07:34:40.968 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel for subscription processing, as it is currently being processed.
<6> 2021-05-03 07:34:40.969 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-03 07:34:40.969 +00:00 [INF] - Initialized cloud proxy 02e8f4da-bd90-4da0-bd58-e11f087ad2d3 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:40.969 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:41.055 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-03 07:34:44.789 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-03 07:39:30.404 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2021-05-03 07:40:04.822 +00:00 [INF] - Exiting connected state
<6> 2021-05-03 07:40:04.822 +00:00 [INF] - Entering unreachable state
<6> 2021-05-03 07:40:04.990 +00:00 [INF] - Entering connected state

jokarl on 3 May 2021

👍1

@jokarl Initially I was thinking that your upgrade scenario went like this ordering:

Running 1.0.10 daemon and 1.0.10 system modules (Edge Agent / Edge Hub)
Install newer 1.1 version of the daemon. Your custom modules then start up and connect to EdgeHub, but the 1.0.10 system modules are still running
You update the system modules via new deployment. Your custom modules then lose connection to 1.0.10 Edge Hub, and even cannot connect once 1.1 Edge Hub comes back up again.

But I believe my understanding is wrong because when you perform the deployment update in step 3, the Edge Agent container should restart with an updated version. This Edge Agent restart then should issue SIGTERM (followed by SIGKILL) for all your custom modules. Theoretically they should have a guaranteed restart _after_ the 1.1 Edge Agent starts, rather than there be a disconnection then a reconnection like the issue implies.

Does this make sense? If so, I am confused why your custom modules are dealing with a reconnection at all.

I just confirmed this upgrade scenario behavior in a test environment with four test modules (although using C# devices SDK) connected over the following protocols:

Mqtt
Mqtt Websockets
Amqp
Amqp Websockets

and-rewsmith on 5 May 2021

Hi @and-rewsmith

You are describing the exact steps I am taking to upgrade.

Your explanation makes sense and I see why it causes confusion, I'll perform another update tonight.

Sorry for the slow response time. We don't have hardware left to do the upgrade steps on. We have devices in staging and dev that have been running fine for a few weeks despite this issue.

After doing this, I will run sudo iotedge support-bundle --since 1h to get the logs from all containers.

jokarl on 5 May 2021

I performed an update according to the steps outlined in my first post, and here are the results.

I waited 20 minutes and containers did not reconnect
I restarted _one_ of our disconnected containers and it immediately worked
Containers not restarted still no connection to edgeHub
I force removed the edgeHub sudo docker rm -f edgeHub and all my containers disconnect
- Interestingly, this produced new log output in the containers that I did not forcefully restart
After forcefully removing edgeHub, I waited another 15 minutes but nothing happens. Not even our containers that I restarted manually that were previously connected
Again, sudo docker restart mycontainer and the container connects
Finally, I just restarted the entire runtime with sudo systemctl restart iotedge
All containers are now running. But I'm worried what will happen on power cycles, new edge deployments etc.

So when receiving the new 1.1.1 edgeAgent container I do not get any SIGTERM for my containers, they just simply remain on as can be seen here by running iotedge list:

NAME             STATUS           DESCRIPTION      CONFIG
edgeAgent        running          Up 43 minutes    mcr.microsoft.com/azureiotedge-agent:1.1
edgeHub          running          Up 10 minutes    mcr.microsoft.com/azureiotedge-hub:1.1
mycontainer1     running          Up 20 minutes    myregistry.azurecr.io/xxx
mycontainer2     running          Up an hour       myregistry.azurecr.io/xxx
mycontainer3     running          Up an hour       myregistry.azurecr.io/xxx
mycontainer4     running          Up an hour       myregistry.azurecr.io/xxx
mycontainer5     running          Up an hour       myregistry.azurecr.io/xxx

edgeAgent after new edge deployment

2021-05-05 18:38:45  Starting Edge Agent
2021-05-05 18:38:45  Changing ownership of storage folder: /tmp/edgeAgent to 1000
2021-05-05 18:38:45  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2021-05-05 18:38:45.779 +00:00 Edge Agent Main()
<6> 2021-05-05 18:38:46.180 +00:00 [INF] - Initializing Edge Agent.
<6> 2021-05-05 18:38:46.545 +00:00 [INF] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-05 18:38:46.546 +00:00 [INF] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2021-05-05 18:38:46.653 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2021-05-05 18:38:46.898 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:05/01/2031 20:43:58] to Root
<6> 2021-05-05 18:38:47.208 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics
<6> 2021-05-05 18:38:47.559 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2021-05-05 18:38:47.570 +00:00 [INF] - Started operation Get system resources
<6> 2021-05-05 18:38:47.574 +00:00 [INF] - Collecting metadata metrics
<6> 2021-05-05 18:38:47.762 +00:00 [INF] - Set metadata metrics: 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"linux","Architecture":"x86_64","Version":"1.1.1","Provisioning":{"Type":"manual.device_connection_string","DynamicReprovisioning":false},"ServerVersion":"20.10.2+azure","KernelVersion":"4.15.0-112-generic","OperatingSystem":"Ubuntu 18.04.5 LTS","NumCpus":4,"Virtualized":"no"}, True
<6> 2021-05-05 18:38:47.808 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2021-05-05 18:38:47.815 +00:00 [INF] - Started operation refresh twin config
<6> 2021-05-05 18:38:47.861 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2021-05-05 18:38:48.551 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2021-05-05 18:38:48.730 +00:00 [INF] - Started operation Metrics Scrape
<6> 2021-05-05 18:38:48.730 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2021-05-05 18:38:49.276 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2021-05-05 18:38:49.277 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2021-05-05 18:38:49.277 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2021-05-05 18:38:49.278 +00:00 [INF] - Registering request handler RestartModule
<6> 2021-05-05 18:38:49.477 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2021-05-05 18:38:49.721 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2021-05-05 18:38:49.846 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 12 and reported properties version 260.
<6> 2021-05-05 18:38:50.818 +00:00 [INF] - Plan execution started for deployment 12
<6> 2021-05-05 18:38:50.826 +00:00 [INF] - Executing command: "Stop module edgeHub"
<6> 2021-05-05 18:39:01.296 +00:00 [INF] - Executing command: "Remove module edgeHub"
<6> 2021-05-05 18:39:01.381 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2021-05-05 18:39:01.389 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2021-05-05 18:39:01.461 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"
<6> 2021-05-05 18:39:01.465 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)"
<3> 2021-05-05 18:39:05.739 +00:00 [ERR] - Executing command for operation ["update"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 144
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 187
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
<3> 2021-05-05 18:39:05.779 +00:00 [ERR] - Executing command for operation ["Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 144
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 187
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
<3> 2021-05-05 18:39:05.788 +00:00 [ERR] - Step failed in deployment 12, continuing execution. Failure when running command Command Group: (
  [Command Group: (
  [Prepare update module edgeHub]
  [Stop module edgeHub]
  [Update module edgeHub]
)]
  [Start module edgeHub]
). Will retry in 00s.
<6> 2021-05-05 18:39:05.790 +00:00 [INF] - Plan execution ended for deployment 12
<3> 2021-05-05 18:39:05.794 +00:00 [ERR] - Edge agent plan execution failed.
System.AggregateException: One or more errors occurred. (Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub)
 ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 144
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 187
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 87
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.<>c.<ExecuteAsync>b__7_0(List`1 f) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 115
   at Microsoft.Azure.Devices.Edge.Util.Option`1.ForEach(Action`1 action) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/Option.cs:line 141
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 116
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 162
<4> 2021-05-05 18:39:05.877 +00:00 [WRN] - Reconcile failed because of the an exception
System.AggregateException: One or more errors occurred. (Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub)
 ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 144
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 187
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 87
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.<>c.<ExecuteAsync>b__7_0(List`1 f) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 115
   at Microsoft.Azure.Devices.Edge.Util.Option`1.ForEach(Action`1 action) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/Option.cs:line 141
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 116
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 162
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 174
<6> 2021-05-05 18:39:06.209 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:39:11.477 +00:00 [INF] - Plan execution started for deployment 12
<6> 2021-05-05 18:39:11.479 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2021-05-05 18:39:11.479 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2021-05-05 18:39:12.278 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2021-05-05 18:39:13.329 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2021-05-05 18:39:13.563 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:39:18.836 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:43:05.679 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'lockerParcel' as it has been running healthy for 00:10:00.
<6> 2021-05-05 18:43:05.681 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'lockerWebsocket' as it has been running healthy for 00:10:00.
<6> 2021-05-05 18:43:05.684 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'logAnalytics' as it has been running healthy for 00:10:00.
<6> 2021-05-05 18:43:05.685 +00:00 [INF] - Plan execution started for deployment 12
<6> 2021-05-05 18:43:05.686 +00:00 [INF] - Executing command: "Saving lockerParcel to store"
<6> 2021-05-05 18:43:05.687 +00:00 [INF] - Executing command: "Saving lockerWebsocket to store"
<6> 2021-05-05 18:43:05.687 +00:00 [INF] - Executing command: "Saving logAnalytics to store"
<6> 2021-05-05 18:43:05.687 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2021-05-05 18:43:10.973 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:43:16.008 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'omboriBrowser' as it has been running healthy for 00:10:00.
<6> 2021-05-05 18:43:16.011 +00:00 [INF] - Plan execution started for deployment 12
<6> 2021-05-05 18:43:16.011 +00:00 [INF] - Executing command: "Saving omboriBrowser to store"
<6> 2021-05-05 18:43:16.012 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2021-05-05 18:43:21.279 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:43:46.452 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'lockerScanner' as it has been running healthy for 00:10:00.
<6> 2021-05-05 18:43:46.455 +00:00 [INF] - Plan execution started for deployment 12
<6> 2021-05-05 18:43:46.456 +00:00 [INF] - Executing command: "Saving lockerScanner to store"
<6> 2021-05-05 18:43:46.457 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2021-05-05 18:43:51.722 +00:00 [INF] - Updated reported properties
<6> 2021-05-05 18:49:28.865 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2021-05-05 18:49:28.869 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2021-05-05 18:49:28.938 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"edgeHub","filter":{"tail":1000,"since":"2d","until":null,"loglevel":null,"regex":null}},"encoding":1,"contentType":1}
<6> 2021-05-05 18:49:29.011 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2021-05-05 18:49:29.377 +00:00 [INF] - Received 3581 bytes of logs for edgeHub
<6> 2021-05-05 18:49:29.394 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2021-05-05 18:50:33.355 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2021-05-05 18:50:33.356 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2021-05-05 18:50:33.358 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"edgeHub","filter":{"tail":1000,"since":"2h","until":null,"loglevel":null,"regex":null}},"encoding":1,"contentType":1}
<6> 2021-05-05 18:50:33.377 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2021-05-05 18:50:33.400 +00:00 [INF] - Received 3581 bytes of logs for edgeHub
<6> 2021-05-05 18:50:33.401 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2021-05-05 18:51:19.974 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2021-05-05 18:51:19.975 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2021-05-05 18:51:19.977 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"edgeHub","filter":{"tail":null,"since":"2h","until":null,"loglevel":null,"regex":null}},"encoding":1,"contentType":1}
<6> 2021-05-05 18:51:20.002 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2021-05-05 18:51:20.035 +00:00 [INF] - Received 3581 bytes of logs for edgeHub
<6> 2021-05-05 18:51:20.036 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2021-05-05 18:56:40.268 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2021-05-05 18:56:40.269 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2021-05-05 18:56:40.270 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"edgeHub","filter":{"tail":null,"since":"3h","until":null,"loglevel":null,"regex":null}},"encoding":1,"contentType":1}
<6> 2021-05-05 18:56:40.299 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2021-05-05 18:56:40.316 +00:00 [INF] - Received 3631 bytes of logs for edgeHub
<6> 2021-05-05 18:56:40.316 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2021-05-05 18:56:52.845 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2021-05-05 18:56:52.845 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2021-05-05 18:56:52.846 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"edgeAgent","filter":{"tail":null,"since":"3h","until":null,"loglevel":null,"regex":null}},"encoding":1,"contentType":1}

edgeHub after new edge deployment

2021-05-05 18:39:13  Starting Edge Hub
2021-05-05 18:39:13.421 +00:00 Edge Hub Main()
<6> 2021-05-05 18:39:15.388 +00:00 [INF] - Installing certificates [CN=iotedged workload ca:05/01/2031 20:43:58],[CN=Test Edge Device CA:05/01/2031 20:43:58],[CN=Test Edge Owner CA:05/01/2031 20:43:58] to Root
<6> 2021-05-05 18:39:15.465 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:05/01/2031 20:43:58] to Root
<6> 2021-05-05 18:39:15.486 +00:00 [INF] - Enabling SSL protocols: Tls, Tls11, Tls12
<6> 2021-05-05 18:39:15.722 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2021-05-05 18:39:16.643 +00:00 [INF] - Created persistent store at /tmp/edgeHub
<6> 2021-05-05 18:39:16.728 +00:00 [INF] - Created device scope identities cache
<6> 2021-05-05 18:39:16.740 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2021-05-05 18:39:16.740 +00:00 [INF] - Initializing Edge Hub
<6> 2021-05-05 18:39:16.741 +00:00 [INF] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2021-05-05 18:39:16.746 +00:00 [INF] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-05 18:39:16.746 +00:00 [INF] - OptimizeForPerformance=True
<6> 2021-05-05 18:39:16.746 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2021-05-05 18:39:16.747 +00:00 [INF] - Loaded server certificate with expiration date of "2021-08-03T18:39:13.0000000+00:00"
<6> 2021-05-05 18:39:16.763 +00:00 [INF] - Using Asp Net server for metrics
<6> 2021-05-05 18:39:16.852 +00:00 [INF] - Created new message store
<6> 2021-05-05 18:39:16.859 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2021-05-05 18:39:16.995 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2021-05-05 18:39:17.054 +00:00 [INF] - Initialized storing twin manager
<6> 2021-05-05 18:39:17.080 +00:00 [INF] - Initializing configuration
<6> 2021-05-05 18:39:17.111 +00:00 [INF] - New device connection for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:17.134 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2021-05-05 18:39:17.264 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:17.828 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub via AMQP...
<4> 2021-05-05 18:39:17.836 +00:00 [WRN] - Empty edge hub configuration received. Ignoring...
<6> 2021-05-05 18:39:19.086 +00:00 [INF] - Exiting disconnected state
<6> 2021-05-05 18:39:19.116 +00:00 [INF] - Received device connected callback
<6> 2021-05-05 18:39:19.126 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2021-05-05 18:39:19.131 +00:00 [INF] - Processing subscriptions for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub on device connected to cloud.
<6> 2021-05-05 18:39:19.132 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:19.152 +00:00 [INF] - Entering connected state
<6> 2021-05-05 18:39:19.158 +00:00 [INF] - Cloud connection for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub is True
<6> 2021-05-05 18:39:19.163 +00:00 [INF] - Connection status for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub changed to ConnectionEstablished
<6> 2021-05-05 18:39:19.167 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2021-05-05 18:39:19.168 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:19.175 +00:00 [INF] - Created cloud proxy for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-05 18:39:19.185 +00:00 [INF] - Initialized cloud proxy a3a4d5f2-8652-4c36-8e53-6868b421a9fd for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:19.188 +00:00 [INF] - Created cloud connection for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:19.510 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:19.548 +00:00 [INF] - Updated reported properties for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:19.965 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2021-05-05 18:39:20.147 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2021-05-05 18:39:20.609 +00:00 [INF] - Set the following 24 route(s) in edge hub
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - upstream: FROM /messages/* INTO $upstream
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsCancelSessionRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/session_cancel INTO BrokeredEndpoint("/modules/lockerParcel/inputs/session_cancel")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsErrorEventToWs: FROM /messages/modules/lockerWebsocket/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsGetLayoutRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/locker_get-layout INTO BrokeredEndpoint("/modules/lockerParcel/inputs/locker_get-layout")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsLargerSlotRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_request-bigger INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_request-bigger")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsPackageRetrievedToLocker: FROM /messages/modules/lockerWebsocket/outputs/package_retrieved INTO BrokeredEndpoint("/modules/lockerParcel/inputs/package_retrieved")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsResolveRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-05 18:39:20.610 +00:00 [INF] - wsSlotUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - wsSlotLoadedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_loaded INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_loaded")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - wsOverdueRemovedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_overdue-removed INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_overdue-removed")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - wsSlotsUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_unlock")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - wsSlotsLockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_lock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_lock")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerCancelSessionResponseToWs: FROM /messages/modules/lockerParcel/outputs/session_canceled INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/session_canceled")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerConfigurationStartToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-start INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-start")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerConfigurationEndToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-end INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-end")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerConfigurationUpdateToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-update INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-update")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerErrorEventToWs: FROM /messages/modules/lockerParcel/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerGetLayoutResponseToWs: FROM /messages/modules/lockerParcel/outputs/locker_layout INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_layout")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerLargerSlotResponseToWs: FROM /messages/modules/lockerParcel/outputs/slot_response-bigger INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_response-bigger")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerResolveResponseToWs: FROM /messages/modules/lockerParcel/outputs/code_resolved INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/code_resolved")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerSlotStateChangeToWs: FROM /messages/modules/lockerParcel/outputs/slot_state-change INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_state-change")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - lockerSlotUnlockToLocker: FROM /messages/modules/lockerParcel/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - scannerErrorEventToWs: FROM /messages/modules/lockerScanner/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-05 18:39:20.611 +00:00 [INF] - scannerResolveRequestToLocker: FROM /messages/modules/lockerScanner/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-05 18:39:20.614 +00:00 [INF] - Updated message store TTL to 30 seconds
<6> 2021-05-05 18:39:20.615 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-05 18:39:20.628 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2021-05-05 18:39:20.629 +00:00 [INF] - Initialized edge hub configuration
<6> 2021-05-05 18:39:20.643 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2021-05-05 18:39:20.759 +00:00 [INF] - Scheduling server certificate renewal for "2021-08-03T18:36:43.0003035Z".
<6> 2021-05-05 18:39:20.762 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2021-05-05 18:39:20.767 +00:00 [INF] - Starting MQTT head
<6> 2021-05-05 18:39:20.813 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2021-05-05 18:39:20.910 +00:00 [INF] - Starting AMQP head
<6> 2021-05-05 18:39:20.929 +00:00 [INF] - Started MQTT head
<6> 2021-05-05 18:39:21.071 +00:00 [INF] - Started AMQP head
<6> 2021-05-05 18:39:21.075 +00:00 [INF] - Starting HTTP head
<4> 2021-05-05 18:39:21.470 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2021-05-05 18:39:21.485 +00:00 [INF] - Started HTTP head
<6> 2021-05-05 18:39:21.650 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser in device scope authenticated locally.
<6> 2021-05-05 18:39:21.672 +00:00 [INF] - Successfully generated identity for clientId parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser and username cnc-prod.azure-devices.net/parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser/?api-version=2019-10-01&DeviceClientType=azure-iot-device%2F1.17.1%20(node%20v10.19.0%3B%20Ubuntu%2020.04%3B%20x64)
<6> 2021-05-05 18:39:21.683 +00:00 [INF] - ClientAuthenticated, parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser, 61e61454
<6> 2021-05-05 18:39:21.732 +00:00 [INF] - New device connection for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:21.734 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser connected to edgeHub, processing existing subscriptions.
<6> 2021-05-05 18:39:21.759 +00:00 [INF] - Bind device proxy for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:21.761 +00:00 [INF] - Binding message channel for device Id parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:21.810 +00:00 [INF] - Processing subscriptions Methods for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser.
<6> 2021-05-05 18:39:21.811 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:21.843 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser via AMQP...
<6> 2021-05-05 18:39:21.854 +00:00 [INF] - Set subscriptions from session state for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:22.027 +00:00 [INF] - Cloud connection for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser is True
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Connection status for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser changed to ConnectionEstablished
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser connected to cloud, processing existing subscriptions.
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Created cloud proxy for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Initialized cloud proxy 86cef19c-6ab6-4d84-aff9-f9c318c2255d for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:22.028 +00:00 [INF] - Created cloud connection for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/omboriBrowser
<6> 2021-05-05 18:39:22.283 +00:00 [INF] - Updated reported properties for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:39:28.434 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-05 18:39:28.532 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner in device scope authenticated locally.
<6> 2021-05-05 18:39:28.546 +00:00 [INF] - New device connection for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.547 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner connected to edgeHub, processing existing subscriptions.
<6> 2021-05-05 18:39:28.548 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner via AMQP...
<6> 2021-05-05 18:39:28.550 +00:00 [INF] - Bind device proxy for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.553 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.563 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.573 +00:00 [INF] - Opened link Events for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.579 +00:00 [INF] - Opened link ModuleMessages for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.607 +00:00 [INF] - Opened link MethodReceiving for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.623 +00:00 [INF] - Opened link MethodSending for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.755 +00:00 [INF] - Cloud connection for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner is True
<6> 2021-05-05 18:39:28.755 +00:00 [INF] - Connection status for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner changed to ConnectionEstablished
<6> 2021-05-05 18:39:28.755 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner connected to cloud, processing existing subscriptions.
<6> 2021-05-05 18:39:28.755 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:28.756 +00:00 [INF] - Created cloud proxy for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-05 18:39:28.756 +00:00 [INF] - Initialized cloud proxy 97a48995-7f53-4036-8092-16fa1492fe50 for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.756 +00:00 [INF] - Created cloud connection for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:28.811 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerScanner
<6> 2021-05-05 18:39:29.605 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-05 18:39:29.620 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket in device scope authenticated locally.
<6> 2021-05-05 18:39:29.624 +00:00 [INF] - New device connection for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.627 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket connected to edgeHub, processing existing subscriptions.
<6> 2021-05-05 18:39:29.632 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket via AMQP...
<6> 2021-05-05 18:39:29.645 +00:00 [INF] - Bind device proxy for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.646 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.647 +00:00 [INF] - Opened link Events for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.667 +00:00 [INF] - Opened link ModuleMessages for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.669 +00:00 [INF] - Opened link MethodReceiving for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.670 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.672 +00:00 [INF] - Opened link MethodSending for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.842 +00:00 [INF] - Cloud connection for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket is True
<6> 2021-05-05 18:39:29.842 +00:00 [INF] - Connection status for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket changed to ConnectionEstablished
<6> 2021-05-05 18:39:29.843 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket connected to cloud, processing existing subscriptions.
<6> 2021-05-05 18:39:29.843 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:29.843 +00:00 [INF] - Created cloud proxy for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-05 18:39:29.843 +00:00 [INF] - Initialized cloud proxy d9e2ae9a-7882-4aca-8ae4-b0bad03a4efb for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.843 +00:00 [INF] - Created cloud connection for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:29.895 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerWebsocket
<6> 2021-05-05 18:39:30.351 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-05 18:39:30.363 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel in device scope authenticated locally.
<6> 2021-05-05 18:39:30.365 +00:00 [INF] - New device connection for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.365 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel connected to edgeHub, processing existing subscriptions.
<6> 2021-05-05 18:39:30.367 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel via AMQP...
<6> 2021-05-05 18:39:30.373 +00:00 [INF] - Bind device proxy for device parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.374 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.374 +00:00 [INF] - Opened link Events for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.374 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.374 +00:00 [INF] - Opened link MethodReceiving for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.375 +00:00 [INF] - Opened link ModuleMessages for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.375 +00:00 [INF] - Opened link MethodSending for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.546 +00:00 [INF] - Cloud connection for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel is True
<6> 2021-05-05 18:39:30.546 +00:00 [INF] - Connection status for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel changed to ConnectionEstablished
<6> 2021-05-05 18:39:30.546 +00:00 [INF] - Client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel connected to cloud, processing existing subscriptions.
<6> 2021-05-05 18:39:30.547 +00:00 [INF] - Skipping parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel for subscription processing, as it is currently being processed.
<6> 2021-05-05 18:39:30.547 +00:00 [INF] - Created cloud proxy for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-05 18:39:30.547 +00:00 [INF] - Initialized cloud proxy 363ecb26-73d0-4cd5-82e6-9b685e4d1eab for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.547 +00:00 [INF] - Created cloud connection for client parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:30.760 +00:00 [INF] - Processing pending subscriptions for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/lockerParcel
<6> 2021-05-05 18:39:32.692 +00:00 [INF] - Updated reported properties for parcel-b90dc38207336daa6ff37a5d5bf9e8dd/$edgeHub
<6> 2021-05-05 18:44:20.656 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2021-05-05 18:49:20.643 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2021-05-05 18:54:01.827 +00:00 [WRN] - Received request for unsupported method GetModuleLogs
<6> 2021-05-05 18:54:20.645 +00:00 [INF] - Entering periodic task to reauthenticate connected clients

Our container after new edge deployment (same version, only new edge containers)

2021-05-05 18:33:04,770 [main] INFO  se.itab.locker.parcel.Parcel.start(75) - Starting Locker
2021-05-05 18:33:05,235 [main] INFO  se.itab.locker.parcel.Parcel.start(80) - Sending control-board serial number event over serialPort
2021-05-05 18:33:05,326 [main] INFO  se.itab.locker.parcel.Parcel.start(84) - Sent control-board serial number event over serialPort
2021-05-05 18:33:08,576 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:33:08,609 [main] WARN  s.i.locker.core.azure.OpenableClient.openAndRetryOnFailure(97) - Failed to open client connection, retrying in 6990 ms
2021-05-05 18:33:16,362 [main] INFO  s.i.locker.core.iot.edge.EdgeModule.open(73) - Successfully connected module lockerParcel
2021-05-05 18:33:16,362 [main] INFO  se.itab.locker.Locker.bootstrapLockerClient(68) - Bootstrapping upstream communication
2021-05-05 18:33:16,372 [main] INFO  se.itab.locker.grpc.cert.DirUtil.hasValidSignedCertificate(63) - Checking if certificate at path /app/certs/device.pem exists
2021-05-05 18:33:16,373 [main] INFO  se.itab.locker.grpc.cert.DirUtil.hasValidSignedCertificate(67) - Device certificate exists, checking validity
2021-05-05 18:33:16,413 [main] INFO  se.itab.locker.grpc.GrpcService.performHermesAssistedBootstrap(59) - Valid device certificate exists, continuing.
2021-05-05 18:33:16,506 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(40) - Started provisioning client with endpoint global.azure-devices-provisioning.net
2021-05-05 18:33:16,517 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(45) - Waiting for status assigned...
2021-05-05 18:33:16,519 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:17,520 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:18,521 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:19,522 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:20,522 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:21,523 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:22,524 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:23,525 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:24,527 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:25,528 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:26,529 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:27,530 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 18:33:28,531 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(48) - Got IOT hub url...
2021-05-05 18:33:28,531 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(51) - Provisioned device ID: itab-parcel-b90dc38207336daa6ff37a5d5bf9e8dd
2021-05-05 18:33:29,389 [main] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.handleOpenableClientRaceCondition(53) - Got a connected event from a client of type LockerClient, but the client wasn't fully online yet. This is most likely a race condition only occurring during startup
2021-05-05 18:33:29,460 [main] INFO  se.itab.locker.core.iot.LockerClient.open(86) - Successfully opened device client
2021-05-05 18:33:29,499 [main] INFO  s.i.locker.core.iot.edge.EdgeModule.open(63) - Client already online, ignoring open request
2021-05-05 18:33:29,850 [main] INFO  s.i.l.core.state.LockerStateService.loadLockerStateFromFile(156) - Loading state from file /app/state/state.json
2021-05-05 18:33:30,016 [main] INFO  se.itab.locker.parcel.Parcel.initLocker(122) - Issuing a lock slot request for each slot in locker state
2021-05-05 18:33:30,096 [main] INFO  se.itab.locker.parcel.Parcel.start(113) - Started Locker
2021-05-05 18:33:30,096 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$0(185) - Subscribed to device twin and got status OK
2021-05-05 18:33:30,134 [azure-iot-sdk-IotHubReceiveTask] INFO  s.i.l.c.i.p.ConfigTwinPropertyCallback.handleDesiredProperty(20) - Received device twin update with desired customer configuration
2021-05-05 18:33:30,140 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$1(187) - Received callback for twin reported property config
2021-05-05 18:33:30,299 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$0(185) - Subscribed to device twin and got status OK_EMPTY
2021-05-05 18:33:31,292 [azure-iot-sdk-IotHubReceiveTask] INFO  s.i.l.c.i.m.d.UpdateStateCallback.executeMethod(47) - Received call to UPDATE_STATE
2021-05-05 18:38:50,926 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(720) - Amqp CBS session closed unexpectedly. Closing this connection...
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpLinkDetachForcedException: Server busy, please retry operation
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:38)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(AmqpsIotHubConnection.java:719)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsCbsSessionHandler.onLinkClosedUnexpectedly(AmqpsCbsSessionHandler.java:143)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsSenderLinkHandler.onLinkRemoteClose(AmqpsSenderLinkHandler.java:116)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:38:50,951 [azure-iot-sdk-ConnectionReconnectionTask:6441205e-1e0b-4044-a891-189e04c372da] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-05-05 18:38:51,192 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:38:51,673 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:38:52,483 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:38:54,079 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:38:57,676 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 18:39:20,220 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)

Our containers after sudo docker restart

2021-05-05 18:39:20,220 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:01:21,403 [main] INFO  se.itab.locker.parcel.Parcel.start(75) - Starting Locker
2021-05-05 19:01:21,855 [main] INFO  se.itab.locker.parcel.Parcel.start(80) - Sending control-board serial number event over serialPort
2021-05-05 19:01:21,885 [main] INFO  se.itab.locker.parcel.Parcel.start(84) - Sent control-board serial number event over serialPort
2021-05-05 19:01:23,861 [main] INFO  s.i.locker.core.iot.edge.EdgeModule.open(73) - Successfully connected module lockerParcel
2021-05-05 19:01:23,863 [main] INFO  se.itab.locker.Locker.bootstrapLockerClient(68) - Bootstrapping upstream communication
2021-05-05 19:01:23,868 [main] INFO  se.itab.locker.grpc.cert.DirUtil.hasValidSignedCertificate(63) - Checking if certificate at path /app/certs/device.pem exists
2021-05-05 19:01:23,870 [main] INFO  se.itab.locker.grpc.cert.DirUtil.hasValidSignedCertificate(67) - Device certificate exists, checking validity
2021-05-05 19:01:23,914 [main] INFO  se.itab.locker.grpc.GrpcService.performHermesAssistedBootstrap(59) - Valid device certificate exists, continuing.
2021-05-05 19:01:24,023 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(40) - Started provisioning client with endpoint global.azure-devices-provisioning.net
2021-05-05 19:01:24,037 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(45) - Waiting for status assigned...
2021-05-05 19:01:24,039 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:25,040 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:26,041 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:27,042 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:28,043 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:29,044 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:30,045 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:31,046 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:32,047 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:33,049 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:34,050 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:35,051 [main] INFO  s.i.l.a.DeviceProvisioningService.waitForAssigned(69) - Waiting for Provisioning Service to register
2021-05-05 19:01:36,052 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(48) - Got IOT hub url...
2021-05-05 19:01:36,054 [main] INFO  s.i.l.a.DeviceProvisioningService.provisionDevice(51) - Provisioned device ID: itab-parcel-b90dc38207336daa6ff37a5d5bf9e8dd
2021-05-05 19:01:36,954 [main] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.handleOpenableClientRaceCondition(53) - Got a connected event from a client of type LockerClient, but the client wasn't fully online yet. This is most likely a race condition only occurring during startup
2021-05-05 19:01:37,016 [main] INFO  se.itab.locker.core.iot.LockerClient.open(86) - Successfully opened device client
2021-05-05 19:01:37,058 [main] INFO  s.i.locker.core.iot.edge.EdgeModule.open(63) - Client already online, ignoring open request
2021-05-05 19:01:37,434 [main] INFO  s.i.l.core.state.LockerStateService.loadLockerStateFromFile(156) - Loading state from file /app/state/state.json
2021-05-05 19:01:37,584 [main] INFO  se.itab.locker.parcel.Parcel.initLocker(122) - Issuing a lock slot request for each slot in locker state
2021-05-05 19:01:37,653 [main] INFO  se.itab.locker.parcel.Parcel.start(113) - Started Locker
2021-05-05 19:01:37,664 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$0(185) - Subscribed to device twin and got status OK
2021-05-05 19:01:37,711 [azure-iot-sdk-IotHubReceiveTask] INFO  s.i.l.c.i.p.ConfigTwinPropertyCallback.handleDesiredProperty(20) - Received device twin update with desired customer configuration
2021-05-05 19:01:37,719 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$1(187) - Received callback for twin reported property config
2021-05-05 19:01:37,897 [azure-iot-sdk-IotHubReceiveTask] INFO  se.itab.locker.core.iot.LockerClient.lambda$startDeviceTwin$0(185) - Subscribed to device twin and got status OK_EMPTY
2021-05-05 19:01:38,835 [azure-iot-sdk-IotHubReceiveTask] INFO  s.i.l.c.i.m.d.UpdateStateCallback.executeMethod(47) - Received call to UPDATE_STATE

One of our containers that did not reconnect gets a different error message after sudo docker rm -f edgeHub (EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason: NO_NETWORK)

2021-05-05 19:03:54,650 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpConnectionFramingErrorException: connection aborted
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:44)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:03:54,665 [azure-iot-sdk-ConnectionReconnectionTask:0db2e36e-1583-4067-8981-5f759bc12778] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-05-05 19:03:54,876 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:03:55,262 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:03:55,984 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:03:57,818 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-05 19:04:01,279 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: null
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)

jokarl on 5 May 2021

👍1

@jokarl Thanks for the logs. It seems the new versions of Edge Hub and Edge Agent come up fine. I am wondering why the new Edge Agent isn't restarting your custom modules. Currently asking around internally.

and-rewsmith on 5 May 2021

We've been running 1.0.x for over a year and I have not seen any issues, perhaps it's just our SDKs that are out of date?

jokarl on 5 May 2021

@jokarl I certainly recommend updating the SDK. But your custom modules should be restarting, and if they were I feel like this problem wouldn't have happened in the first place. So my understanding is that there are two independent problems here:
1 - EdgeAgent doesn't trigger restart of custom modules
2 - Some SDK reconnection issue

and-rewsmith on 5 May 2021

Tell me if you need anything else from me

jokarl on 5 May 2021

👍1

@jokarl I have been looking into the Edge Agent code and it seems weird to me that we are seeing these exceptions for "no such container" when attempting to update Edge Hub. Edge Agent gets a list of current modules from the iotedge service, which interfaces directly with docker to know what modules are actually running. Nothing should be removing the container except a plan made by Edge Agent.

You said you waited 20 minutes before force removing the Edge Hub container, so I don't think this is caused by your commands. Any ideas on what could be removing Edge Hub container?

Also, after some more investigation I think your modules are probably running with a restartPolicy of "never" right? This will make the modules avoid the stop/start on Edge Agent 1.1 startup. But it shouldn't avoid the stop from Edge Agent 1.0.10 shutting down. So I would still think they would get restarted. I have setup a meeting for tomorrow to discuss this with some team members.

So when receiving the new 1.1.1 edgeAgent container I do not get any SIGTERM for my containers,

Just to confirm, when the old 1.0.10 Edge Agent shuts down, it does not issue SIGTERM?

As a mitigation I think you can change the restartPolicy of your custom modules to "always" if it is conducive to your workflow.

and-rewsmith on 11 May 2021

I updated the security daemon, rolled out an edge deployment with a new rolling tag 1.1 (previously rolling tag 1.0) for edge runtime containers. That made all of our custom modules disconnected from the edgeHub. To see if it would reconnect automatically, I waited 20 minutes before manually restarting one of our custom modules. A forced restart of one our custom modules and it connected to the edgeHub once again. Containers _not_ forced to restart remained disconnected.

To see if an automatic reconnect could be forced by removing the edgeHub, I ran sudo docker rm -f edgeHub after those 20 minutes had passed. No change in connection status, but our custom modules produced additional logs. So nothing is removing our edgeHub container except for the new edge deployment where I specifically want them replaced with the new version.

All our custom modules are running the restartPolicy of always. The issue here is that I have never seen our modules receive a disconnect and then never reconnect. This error does not cause our containers to crash, if they would have crashed we would have recovered from this due to the restart policy.

I am 90% sure that when we roll out new edge deployments to 1.0.10, _all_ containers get the same uptime (e.g. they all restart). I cannot verify this right now, but I could try it out later this evening, we don't have any dev 1.0.x left.

I tried to issue a new edge deployment to one of our newly updated systems running 1.1, to see what the behavior was there. By _issue new edgeployment_ I mean:

Go to IoT Hub
Go to IoT Edge
Go to IoT Edge deployments
Clone latest edge deployment
Change nothing
Click create

I deployed the _exact same versions_ as before, I just created a new deployment with the same versions. Unfortunately, new deployments also breaks everything, I've never seen this issue pre 1.1.

Before deployment

$ sudo iotedge list
NAME             STATUS           DESCRIPTION      CONFIG
edgeAgent        running          Up a week        mcr.microsoft.com/azureiotedge-agent:1.1
edgeHub          running          Up 5 days        mcr.microsoft.com/azureiotedge-hub:1.1
mycontainer1     running          Up 5 days        myregistry.azurecr.io/my-container-1:release-0.6.2
mycontainer2     running          Up 5 days        myregistry.azurecr.io/my-container-2:release-0.6.2
mycontainer3     running          Up 5 days        myregistry.azurecr.io/my-container-3:release-0.6.2
mycontainer4     running          Up a week        myregistry.azurecr.io/my-container-4:0.1
mycontainer5     running          Up a week        myregistry.azurecr.io/my-container-5:release-0.6.2

After deployment

$ sudo iotedge list
NAME             STATUS           DESCRIPTION      CONFIG
edgeAgent        running          Up a week        mcr.microsoft.com/azureiotedge-agent:1.1
edgeHub          running          Up 20 seconds    mcr.microsoft.com/azureiotedge-hub:1.1
mycontainer1     running          Up 5 days        myregistry.azurecr.io/my-container-1:release-0.6.2
mycontainer2     running          Up 5 days        myregistry.azurecr.io/my-container-2:release-0.6.2
mycontainer3     running          Up 5 days        myregistry.azurecr.io/my-container-3:release-0.6.2
mycontainer4     running          Up a week        myregistry.azurecr.io/my-container-4:0.1
mycontainer5     running          Up a week        myregistry.azurecr.io/my-container-5:release-0.6.2

And unfortunately, I see the same disconnect here and we never recover, see logs below after new deployment received on target device:

edgeAgent

<6> 2021-05-11 05:49:52.260 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2021-05-11 06:07:56.092 +00:00 [INF] - Plan execution started for deployment 131
<6> 2021-05-11 06:07:56.093 +00:00 [INF] - Executing command: "Stop module edgeHub"
<6> 2021-05-11 06:08:06.617 +00:00 [INF] - Executing command: "Remove module edgeHub"
<6> 2021-05-11 06:08:06.675 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2021-05-11 06:08:06.676 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2021-05-11 06:08:06.676 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"
<6> 2021-05-11 06:08:06.676 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)"
<3> 2021-05-11 06:08:07.142 +00:00 [ERR] - Executing command for operation ["update"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 146
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
<3> 2021-05-11 06:08:07.144 +00:00 [ERR] - Executing command for operation ["Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 146
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
<3> 2021-05-11 06:08:07.146 +00:00 [ERR] - Step failed in deployment 131, continuing execution. Failure when running command Command Group: (
  [Command Group: (
  [Prepare update module edgeHub]
  [Stop module edgeHub]
  [Update module edgeHub]
)]
  [Start module edgeHub]
). Will retry in 00s.
<6> 2021-05-11 06:08:07.146 +00:00 [INF] - Plan execution ended for deployment 131
<3> 2021-05-11 06:08:07.146 +00:00 [ERR] - Edge agent plan execution failed.
System.AggregateException: One or more errors occurred. (Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub)
 ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 146
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 87
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.<>c.<ExecuteAsync>b__7_0(List`1 f) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 115
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 116
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208
<4> 2021-05-11 06:08:07.186 +00:00 [WRN] - Reconcile failed because of the an exception
System.AggregateException: One or more errors occurred. (Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub)
 ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling stop module edgeHub: Could not stop module edgeHub
    caused by: Could not stop module edgeHub
    caused by: No such container: edgeHub, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 235
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 146
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.StopModuleAsync(String name) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 189
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 67
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 87
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.<>c.<ExecuteAsync>b__7_0(List`1 f) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 115
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 116
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208
<6> 2021-05-11 06:08:07.357 +00:00 [INF] - Updated reported properties
<6> 2021-05-11 06:08:12.588 +00:00 [INF] - Plan execution started for deployment 131
<6> 2021-05-11 06:08:12.589 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2021-05-11 06:08:12.589 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2021-05-11 06:08:13.063 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2021-05-11 06:08:14.089 +00:00 [INF] - Plan execution ended for deployment 131
<6> 2021-05-11 06:08:14.297 +00:00 [INF] - Updated reported properties
<6> 2021-05-11 06:08:19.523 +00:00 [INF] - Updated reported properties

edgeHub

2021-05-11 06:08:14  Starting Edge Hub
2021-05-11 06:08:14.186 +00:00 Edge Hub Main()
<6> 2021-05-11 06:08:15.850 +00:00 [INF] - Installing certificates [CN=iotedged workload ca:05/01/2031 20:17:41],[CN=Test Edge Device CA:05/01/2031 20:17:41],[CN=Test Edge Owner CA:05/01/2031 20:17:41] to Root
<6> 2021-05-11 06:08:15.937 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:05/01/2031 20:17:41] to Root
<6> 2021-05-11 06:08:15.958 +00:00 [INF] - Enabling SSL protocols: Tls, Tls11, Tls12
<6> 2021-05-11 06:08:16.195 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2021-05-11 06:08:17.600 +00:00 [INF] - Created persistent store at /tmp/edgeHub
<6> 2021-05-11 06:08:17.705 +00:00 [INF] - Initializing Edge Hub
<6> 2021-05-11 06:08:17.706 +00:00 [INF] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2021-05-11 06:08:17.709 +00:00 [INF] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-11 06:08:17.709 +00:00 [INF] - OptimizeForPerformance=True
<6> 2021-05-11 06:08:17.709 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2021-05-11 06:08:17.711 +00:00 [INF] - Loaded server certificate with expiration date of "2021-08-09T06:08:14.0000000+00:00"
<6> 2021-05-11 06:08:17.727 +00:00 [INF] - Using Asp Net server for metrics
<6> 2021-05-11 06:08:17.825 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2021-05-11 06:08:17.825 +00:00 [INF] - Created new message store
<6> 2021-05-11 06:08:17.984 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2021-05-11 06:08:18.136 +00:00 [INF] - Created device scope identities cache
<6> 2021-05-11 06:08:18.150 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2021-05-11 06:08:18.183 +00:00 [INF] - Initialized storing twin manager
<6> 2021-05-11 06:08:18.192 +00:00 [INF] - Initializing configuration
<6> 2021-05-11 06:08:18.226 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:18.254 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2021-05-11 06:08:18.346 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub via AMQP...
<6> 2021-05-11 06:08:18.383 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:19.452 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2021-05-11 06:08:19.978 +00:00 [INF] - Set the following 24 route(s) in edge hub
<6> 2021-05-11 06:08:19.978 +00:00 [INF] - upstream: FROM /messages/* INTO $upstream
<6> 2021-05-11 06:08:19.978 +00:00 [INF] - wsCancelSessionRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/session_cancel INTO BrokeredEndpoint("/modules/lockerParcel/inputs/session_cancel")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsErrorEventToWs: FROM /messages/modules/lockerWebsocket/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsGetLayoutRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/locker_get-layout INTO BrokeredEndpoint("/modules/lockerParcel/inputs/locker_get-layout")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsLargerSlotRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_request-bigger INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_request-bigger")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsPackageRetrievedToLocker: FROM /messages/modules/lockerWebsocket/outputs/package_retrieved INTO BrokeredEndpoint("/modules/lockerParcel/inputs/package_retrieved")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsResolveRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsSlotUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsSlotLoadedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_loaded INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_loaded")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsOverdueRemovedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_overdue-removed INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_overdue-removed")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsSlotsUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_unlock")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - wsSlotsLockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_lock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_lock")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerCancelSessionResponseToWs: FROM /messages/modules/lockerParcel/outputs/session_canceled INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/session_canceled")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerConfigurationStartToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-start INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-start")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerConfigurationEndToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-end INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-end")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerConfigurationUpdateToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-update INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-update")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerErrorEventToWs: FROM /messages/modules/lockerParcel/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerGetLayoutResponseToWs: FROM /messages/modules/lockerParcel/outputs/locker_layout INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_layout")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerLargerSlotResponseToWs: FROM /messages/modules/lockerParcel/outputs/slot_response-bigger INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_response-bigger")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerResolveResponseToWs: FROM /messages/modules/lockerParcel/outputs/code_resolved INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/code_resolved")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerSlotStateChangeToWs: FROM /messages/modules/lockerParcel/outputs/slot_state-change INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_state-change")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - lockerSlotUnlockToLocker: FROM /messages/modules/lockerParcel/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - scannerErrorEventToWs: FROM /messages/modules/lockerScanner/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:19.979 +00:00 [INF] - scannerResolveRequestToLocker: FROM /messages/modules/lockerScanner/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-11 06:08:19.982 +00:00 [INF] - Updated message store TTL to 30 seconds
<6> 2021-05-11 06:08:19.983 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-11 06:08:20.017 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2021-05-11 06:08:20.018 +00:00 [INF] - Initialized edge hub configuration
<6> 2021-05-11 06:08:20.033 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2021-05-11 06:08:20.075 +00:00 [INF] - Exiting disconnected state
<6> 2021-05-11 06:08:20.106 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2021-05-11 06:08:20.110 +00:00 [INF] - Processing subscriptions for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub on device connected to cloud.
<6> 2021-05-11 06:08:20.111 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:20.117 +00:00 [INF] - Received device connected callback
<6> 2021-05-11 06:08:20.142 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2021-05-11 06:08:20.160 +00:00 [INF] - Entering connected state
<6> 2021-05-11 06:08:20.167 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub is True
<6> 2021-05-11 06:08:20.176 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub changed to ConnectionEstablished
<6> 2021-05-11 06:08:20.182 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2021-05-11 06:08:20.182 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:20.189 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-11 06:08:20.191 +00:00 [INF] - Scheduling server certificate renewal for "2021-08-09T06:05:44.0013566Z".
<6> 2021-05-11 06:08:20.195 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2021-05-11 06:08:20.200 +00:00 [INF] - Initialized cloud proxy 90062b1a-e99f-4817-906b-dd473e45348c for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:20.202 +00:00 [INF] - Starting MQTT head
<6> 2021-05-11 06:08:20.204 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:20.296 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2021-05-11 06:08:20.420 +00:00 [INF] - Starting AMQP head
<6> 2021-05-11 06:08:20.497 +00:00 [INF] - Started MQTT head
<6> 2021-05-11 06:08:20.593 +00:00 [INF] - Started AMQP head
<6> 2021-05-11 06:08:20.597 +00:00 [INF] - Starting HTTP head
<6> 2021-05-11 06:08:20.680 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:20.693 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:21.034 +00:00 [INF] - Obtained edge hub config from module twin
<4> 2021-05-11 06:08:21.096 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2021-05-11 06:08:21.101 +00:00 [INF] - Set the following 24 route(s) in edge hub
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - upstream: FROM /messages/* INTO $upstream
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsCancelSessionRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/session_cancel INTO BrokeredEndpoint("/modules/lockerParcel/inputs/session_cancel")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsErrorEventToWs: FROM /messages/modules/lockerWebsocket/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsGetLayoutRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/locker_get-layout INTO BrokeredEndpoint("/modules/lockerParcel/inputs/locker_get-layout")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsLargerSlotRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_request-bigger INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_request-bigger")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsPackageRetrievedToLocker: FROM /messages/modules/lockerWebsocket/outputs/package_retrieved INTO BrokeredEndpoint("/modules/lockerParcel/inputs/package_retrieved")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsResolveRequestToLocker: FROM /messages/modules/lockerWebsocket/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsSlotUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsSlotLoadedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_loaded INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_loaded")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsOverdueRemovedToLocker: FROM /messages/modules/lockerWebsocket/outputs/slot_overdue-removed INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_overdue-removed")
<6> 2021-05-11 06:08:21.102 +00:00 [INF] - wsSlotsUnlockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_unlock")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - wsSlotsLockToLocker: FROM /messages/modules/lockerWebsocket/outputs/slots_lock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slots_lock")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerCancelSessionResponseToWs: FROM /messages/modules/lockerParcel/outputs/session_canceled INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/session_canceled")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerConfigurationStartToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-start INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-start")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerConfigurationEndToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-end INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-end")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerConfigurationUpdateToWs: FROM /messages/modules/lockerParcel/outputs/locker_configuration-update INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_configuration-update")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerErrorEventToWs: FROM /messages/modules/lockerParcel/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerGetLayoutResponseToWs: FROM /messages/modules/lockerParcel/outputs/locker_layout INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/locker_layout")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerLargerSlotResponseToWs: FROM /messages/modules/lockerParcel/outputs/slot_response-bigger INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_response-bigger")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerResolveResponseToWs: FROM /messages/modules/lockerParcel/outputs/code_resolved INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/code_resolved")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerSlotStateChangeToWs: FROM /messages/modules/lockerParcel/outputs/slot_state-change INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/slot_state-change")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - lockerSlotUnlockToLocker: FROM /messages/modules/lockerParcel/outputs/slot_unlock INTO BrokeredEndpoint("/modules/lockerParcel/inputs/slot_unlock")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - scannerErrorEventToWs: FROM /messages/modules/lockerScanner/outputs/system_error INTO BrokeredEndpoint("/modules/lockerWebsocket/inputs/system_error")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - scannerResolveRequestToLocker: FROM /messages/modules/lockerScanner/outputs/code_resolve INTO BrokeredEndpoint("/modules/lockerParcel/inputs/code_resolve")
<6> 2021-05-11 06:08:21.103 +00:00 [INF] - Updated message store TTL to 30 seconds
<6> 2021-05-11 06:08:21.104 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-11 06:08:21.112 +00:00 [INF] - Started HTTP head
<6> 2021-05-11 06:08:22.468 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser in device scope authenticated locally.
<6> 2021-05-11 06:08:22.495 +00:00 [INF] - Successfully generated identity for clientId parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser and username cnc-dev.azure-devices.net/parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser/?api-version=2019-10-01&DeviceClientType=azure-iot-device%2F1.17.1%20(node%20v10.19.0%3B%20Ubuntu%2020.04%3B%20x64)
<6> 2021-05-11 06:08:22.507 +00:00 [INF] - ClientAuthenticated, parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser, 56412f66
<6> 2021-05-11 06:08:22.585 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:22.587 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser connected to edgeHub, processing existing subscriptions.
<6> 2021-05-11 06:08:22.610 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:22.612 +00:00 [INF] - Binding message channel for device Id parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:22.619 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser via AMQP...
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser is True
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser changed to ConnectionEstablished
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser connected to cloud, processing existing subscriptions.
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-11 06:08:22.786 +00:00 [INF] - Initialized cloud proxy ab12d542-f313-433c-a779-2a8a32ee45ba for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:22.787 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:23.519 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:08:25.713 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-11 06:08:25.887 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel in device scope authenticated locally.
<6> 2021-05-11 06:08:25.911 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:25.912 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel connected to edgeHub, processing existing subscriptions.
<6> 2021-05-11 06:08:25.915 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel via AMQP...
<6> 2021-05-11 06:08:25.930 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:25.935 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:25.948 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-11 06:08:25.992 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner in device scope authenticated locally.
<6> 2021-05-11 06:08:26.010 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.014 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.015 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner connected to edgeHub, processing existing subscriptions.
<6> 2021-05-11 06:08:26.022 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner via AMQP...
<6> 2021-05-11 06:08:26.032 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.032 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.037 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.040 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.040 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.040 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.117 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel is True
<6> 2021-05-11 06:08:26.117 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel changed to ConnectionEstablished
<6> 2021-05-11 06:08:26.117 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel connected to cloud, processing existing subscriptions.
<6> 2021-05-11 06:08:26.122 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:26.123 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-11 06:08:26.123 +00:00 [INF] - Initialized cloud proxy 082848ad-933b-48b8-91cb-3b4fafb71c4f for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.123 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.126 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.156 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.157 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.185 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.192 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.278 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner is True
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner changed to ConnectionEstablished
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner connected to cloud, processing existing subscriptions.
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Initialized cloud proxy 20310c12-e7ff-49ff-9c1b-272465c7e9ed for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.279 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.340 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerParcel
<6> 2021-05-11 06:08:26.428 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerScanner
<6> 2021-05-11 06:08:26.468 +00:00 [INF] - New token received on the Cbs link
<6> 2021-05-11 06:08:26.494 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket in device scope authenticated locally.
<6> 2021-05-11 06:08:26.499 +00:00 [INF] - New device connection for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.500 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket connected to edgeHub, processing existing subscriptions.
<6> 2021-05-11 06:08:26.502 +00:00 [INF] - Attempting to connect to IoT Hub for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket via AMQP...
<6> 2021-05-11 06:08:26.505 +00:00 [INF] - Bind device proxy for device parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.505 +00:00 [INF] - Initialized device listener in the AMQP protocol head for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.505 +00:00 [INF] - Opened link Events for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.514 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.515 +00:00 [INF] - Opened link ModuleMessages for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.522 +00:00 [INF] - Opened link MethodSending for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.525 +00:00 [INF] - Opened link MethodReceiving for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.728 +00:00 [INF] - Cloud connection for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket is True
<6> 2021-05-11 06:08:26.729 +00:00 [INF] - Connection status for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket changed to ConnectionEstablished
<6> 2021-05-11 06:08:26.730 +00:00 [INF] - Client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket connected to cloud, processing existing subscriptions.
<6> 2021-05-11 06:08:26.735 +00:00 [INF] - Skipping parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket for subscription processing, as it is currently being processed.
<6> 2021-05-11 06:08:26.742 +00:00 [INF] - Created cloud proxy for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-11 06:08:26.744 +00:00 [INF] - Initialized cloud proxy 54f52232-991e-4527-979f-cd3c27ec1569 for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.752 +00:00 [INF] - Created cloud connection for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:26.809 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/lockerWebsocket
<6> 2021-05-11 06:08:28.205 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates, Methods for client parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser.
<6> 2021-05-11 06:08:28.205 +00:00 [INF] - Processing pending subscriptions for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:28.228 +00:00 [INF] - Set subscriptions from session state for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/omboriBrowser
<6> 2021-05-11 06:08:28.933 +00:00 [INF] - Updated reported properties for parcel-f2b63c8d8ecf94620c513aa6d11f89e9/$edgeHub
<6> 2021-05-11 06:13:20.050 +00:00 [INF] - Entering periodic task to reauthenticate connected clients

I have waited > 10 minutes for this to recover

One of our modules

2021-05-11 06:07:56,208 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(720) - Amqp CBS session closed unexpectedly. Closing this connection...
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.AmqpLinkDetachForcedException: Server busy, please retry operation
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:38)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onCBSSessionClosedUnexpectedly(AmqpsIotHubConnection.java:719)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsCbsSessionHandler.onLinkClosedUnexpectedly(AmqpsCbsSessionHandler.java:143)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsSenderLinkHandler.onLinkRemoteClose(AmqpsSenderLinkHandler.java:116)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:07:56,236 [azure-iot-sdk-ConnectionReconnectionTask:36cfd693-d9fd-45df-8918-8b9e0e1e1ac5] INFO  s.i.l.c.i.e.c.ConnectionChangeCallback.execute(36) - EdgeModule connection status changed to 'DISCONNECTED_RETRYING'. Reason:
NO_NETWORK
2021-05-11 06:07:56,512 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:07:56,953 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:07:57,681 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:07:59,022 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:08:01,656 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)
2021-05-11 06:08:15,451 [azure-iot-sdk-ReactorRunner] ERROR c.m.a.s.i.d.t.a.AmqpsIotHubConnection.onTransportError(486) - Amqp transport error occurred, closing the amqps connection
com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:835)

jokarl on 11 May 2021

👍1

@jokarl Thanks for the detailed logs. That makes it a lot easier for us to debug.

I tried to reproduce your exact upgrade scenario (i.e. mounted folders for hub/agent, java devices sdk 1.29.0, az cli deployment). I was not able to reproduce and my custom module was able to reconnect successfully. I also did not see Edge Agent fail to update the Edge Hub container. Below are the logs from my reproduction attempt in case you think I slipped up somewhere.

1.0.10 daemon and 1.0.10 system modules

azureuser@log-upload-vm2:~$ iotedge --version
iotedge 1.0.10.4

azureuser@log-upload-vm2:~$ iotedge list
NAME                STATUS           DESCRIPTION      CONFIG
edgeAgent           running          Up 2 minutes     mcr.microsoft.com/azureiotedge-agent:1.0.10
edgeHub             running          Up 2 minutes     mcr.microsoft.com/azureiotedge-hub:1.0.10
javaDeviceSdkIssue  running          Up 2 minutes     edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64

azureuser@log-upload-vm2:~$ iotedge logs -f javaDeviceSdkIssue
Start to create client
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Client created
Connection Status: Connected
sent message0
sent message1
sent message2
sent message3
sent message4
sent message5
sent message6
sent message7
sent message8
sent message9
sent message10

1.1 daemon and 1.0.10 system modules (error happens because Edge Hub not up yet)

azureuser@log-upload-vm2:~$ iotedge --version
iotedge 1.1.1

azureuser@log-upload-vm2:~$ iotedge list
NAME                STATUS           DESCRIPTION                CONFIG
edgeAgent           running          Up 29 seconds              mcr.microsoft.com/azureiotedge-agent:1.0.10
edgeHub             running          Up 25 seconds              mcr.microsoft.com/azureiotedge-hub:1.0.10
javaDeviceSdkIssue  failed           Failed (1) 22 seconds ago  edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64

azureuser@log-upload-vm2:~$ iotedge list
NAME                STATUS           DESCRIPTION      CONFIG
edgeAgent           running          Up 2 minutes     mcr.microsoft.com/azureiotedge-agent:1.0.10
edgeHub             running          Up 2 minutes     mcr.microsoft.com/azureiotedge-hub:1.0.10
javaDeviceSdkIssue  running          Up a minute      edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64

azureuser@log-upload-vm2:~$ iotedge logs javaDeviceSdkIssue -f
Start to create client
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Client created
Connection Status: Connected
sent message0
sent message1
sent message2
sent message3
sent message4
sent message5
sent message6
sent message7
sent message8
sent message9
sent message10
sent message11
sent message12
Start to create client
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Client created
java.io.IOException: Could not open the connection
    at com.microsoft.azure.sdk.iot.device.DeviceIO.open(DeviceIO.java:162)
    at com.microsoft.azure.sdk.iot.device.InternalClient.open(InternalClient.java:175)
    at com.edgemodule.App.main(App.java:77)
Caused by: com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Start to create client
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Client created
Connection Status: Connected
sent message0
sent message1
sent message2
sent message3
sent message4
sent message5
sent message6
sent message7
sent message8
sent message9
sent message10
sent message11
sent message12
sent message13
sent message14
sent message15
sent message16
sent message17
sent message18
sent message19
sent message20
sent message21
sent message22
sent message23
sent message24
sent message25
sent message26

1.1 daemon and 1.1 system modules

azureuser@log-upload-vm2:~$ iotedge --version
iotedge 1.1.1

azureuser@log-upload-vm2:~$ iotedge list
NAME                STATUS           DESCRIPTION      CONFIG
edgeAgent           running          Up 2 minutes     mcr.microsoft.com/azureiotedge-agent:1.1
edgeHub             running          Up 2 minutes     mcr.microsoft.com/azureiotedge-hub:1.1
javaDeviceSdkIssue  running          Up 7 minutes     edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64

azureuser@log-upload-vm2:~$ iotedge logs javaDeviceSdkIssue --tail 40
sent message314
sent message315
sent message316
sent message317
sent message318
sent message319
Connection Status: Retrying
sent message320
sent message321
sent message322
sent message323
sent message324
sent message325
sent message326
sent message327
sent message328
sent message329
sent message330
sent message331
sent message332
sent message333
sent message334
sent message335
sent message336
sent message337
sent message338
sent message339
sent message340
sent message341
sent message342
sent message343
Connection Status: Connected
Connection Status: Connected
sent message344
sent message345
sent message346
sent message347
sent message348
sent message349
sent message350

azureuser@log-upload-vm2:~$ iotedge logs --tail 10000 edgeAgent
2021-05-12 00:47:23  Starting Edge Agent
2021-05-12 00:47:23  Changing ownership of storage folder: /tmp/edgeAgent to 1000
2021-05-12 00:47:23  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2021-05-12 00:47:23.841 +00:00 Edge Agent Main()
<6> 2021-05-12 00:47:24.019 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Initializing Edge Agent.
<6> 2021-05-12 00:47:24.183 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-12 00:47:24.183 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2021-05-12 00:47:24.226 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<7> 2021-05-12 00:47:24.241 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to TrustBundleAsync
<7> 2021-05-12 00:47:24.262 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:24.265 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:24.269 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/trust-bundle?api-version=2019-01-30
<7> 2021-05-12 00:47:24.291 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:24.316 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for TrustBundleAsync
<6> 2021-05-12 00:47:24.333 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Installing certificates [CN=Test Edge Owner CA:07/05/2021 06:05:39] to Root
<6> 2021-05-12 00:47:24.465 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics
<6> 2021-05-12 00:47:24.619 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Updating performance metrics every 05m:00s
<6> 2021-05-12 00:47:24.623 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Started operation Get system resources
<6> 2021-05-12 00:47:24.625 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Collecting metadata metrics
<7> 2021-05-12 00:47:24.628 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2021-05-12 00:47:24.636 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:24.636 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:24.636 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2020-07-07
<7> 2021-05-12 00:47:24.670 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:24.692 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2021-05-12 00:47:24.702 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Set metadata metrics: 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"linux","Architecture":"x86_64","Version":"1.1.1","Provisioning":{"Type":"manual.device_connection_string","DynamicReprovisioning":false},"ServerVersion":"20.10.5+azure","KernelVersion":"5.4.0-1043-azure","OperatingSystem":"Ubuntu 18.04.5 LTS","NumCpus":2,"Virtualized":"yes"}, True
<6> 2021-05-12 00:47:24.719 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Metrics.DeploymentMetrics] - Started operation Checkpoint Availability
<6> 2021-05-12 00:47:24.722 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Started operation refresh twin config
<7> 2021-05-12 00:47:24.726 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Initialize twin refresh with interval '01:00:00'
<7> 2021-05-12 00:47:24.731 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Begin refreshing twin from upstream...
<7> 2021-05-12 00:47:24.736 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Getting module client to refresh the twin with retrying set to False
<6> 2021-05-12 00:47:24.757 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2021-05-12 00:47:25.224 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Modules.AgentModule] - Created persistent store at /tmp/edgeAgent
<6> 2021-05-12 00:47:25.324 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Scrape
<6> 2021-05-12 00:47:25.325 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<7> 2021-05-12 00:47:25.328 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Waiting on shutdown handler to trigger
<7> 2021-05-12 00:47:25.603 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Connected with reason Connection_Ok
<6> 2021-05-12 00:47:25.613 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<7> 2021-05-12 00:47:25.622 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Device client for edge agent created.
<7> 2021-05-12 00:47:25.628 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2021-05-12 00:47:25.628 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:25.628 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:25.628 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2020-07-07
<7> 2021-05-12 00:47:25.646 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ConfigSources.TwinConfigSource] - TwinConfigSource Created
<7> 2021-05-12 00:47:25.684 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Edge agent config backup created here - /tmp/edgeAgent/backup.json
<7> 2021-05-12 00:47:25.695 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:25.695 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2021-05-12 00:47:25.700 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadModuleLogs
<6> 2021-05-12 00:47:25.701 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler GetModuleLogs
<6> 2021-05-12 00:47:25.701 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadSupportBundle
<6> 2021-05-12 00:47:25.701 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler RestartModule
<7> 2021-05-12 00:47:25.743 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to Decrypt
<7> 2021-05-12 00:47:25.761 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:25.761 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:25.762 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/modules/%24edgeAgent/genid/637532847882281426/decrypt?api-version=2020-07-07
<6> 2021-05-12 00:47:25.776 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClient] - Initialized new module client with subscriptions enabled
<7> 2021-05-12 00:47:25.783 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Got module client to refresh the twin
<7> 2021-05-12 00:47:25.800 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:25.804 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for Decrypt
<6> 2021-05-12 00:47:25.837 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Obtained Edge agent twin from IoTHub with desired properties version 38 and reported properties version 217.
<7> 2021-05-12 00:47:25.957 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Edge agent updated deployment config from desired properties.
<7> 2021-05-12 00:47:25.957 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Updated edge agent configuration from upstream twin.
<7> 2021-05-12 00:47:25.961 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Edge agent created.
<7> 2021-05-12 00:47:25.964 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation
<7> 2021-05-12 00:47:25.979 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules
<7> 2021-05-12 00:47:25.985 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:25.985 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:25.985 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2020-07-07
<7> 2021-05-12 00:47:25.991 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:25.991 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config...
<7> 2021-05-12 00:47:26.023 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules
<7> 2021-05-12 00:47:26.071 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to Encrypt
<7> 2021-05-12 00:47:26.081 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:26.081 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:47:26.081 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/modules/%24edgeAgent/genid/637532847882281426/encrypt?api-version=2020-07-07
<7> 2021-05-12 00:47:26.107 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:26.109 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for Encrypt
<7> 2021-05-12 00:47:26.127 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Obtained edge agent config
<7> 2021-05-12 00:47:26.150 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List identities
<7> 2021-05-12 00:47:26.156 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.156 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.156 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/identities/?api-version=2020-07-07
<7> 2021-05-12 00:47:26.186 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:26.192 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List identities
<7> 2021-05-12 00:47:26.199 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Update identity for $edgeHub with generation ID 637532847882281426
<7> 2021-05-12 00:47:26.207 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.207 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.207 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/identities/%24edgeHub?api-version=2020-07-07
<7> 2021-05-12 00:47:26.251 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:26.253 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Update identity for $edgeHub with generation ID 637532847882281426
<7> 2021-05-12 00:47:26.266 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - List of desired modules is - {"edgeHub":{"status":"running","restartPolicy":"always","version":"","imagePullPolicy":"on-create","type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-hub:1.1","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeHub/:/tmp/edgeHub\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"},\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}},"edgeAgent":{"version":"","status":"running","restartPolicy":"always","imagePullPolicy":"on-create","startupOrder":0,"type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-agent:1.1","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeAgent/:/tmp/edgeAgent\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"}}"},"env":{"RuntimeLogLevel":{"value":"debug"}},"javaDeviceSdkIssue":{"status":"running","restartPolicy":"always","version":"","imagePullPolicy":"on-create","type":"docker","settings":{"image":"edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64","createOptions":"{\"HostConfig\":{\"PortBindings\":{\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}}
<7> 2021-05-12 00:47:26.303 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - List of current modules is - {"edgeHub":{"restartPolicy":"always","exitCode":0,"statusDescription":"running","lastStartTimeUtc":"2021-05-12T00:41:18.8137254","lastExitTimeUtc":"2021-05-12T00:41:00.1869456","restartCount":1,"lastRestartTimeUtc":"2021-05-12T00:41:17.9468742Z","runtimeStatus":"running","status":"running","imagePullPolicy":"on-create","type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-hub:1.0.10","imageHash":"sha256:086c8ebad9582526bd213818340850d21500539143d4deb42e373f737b4aa4c1","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeHub/:/tmp/edgeHub\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"},\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}},"edgeAgent":{"exitCode":0,"statusDescription":"running","lastStartTimeUtc":"2021-05-12T00:47:23.7569161","lastExitTimeUtc":"0001-01-01T00:00:00","runtimeStatus":"running","imagePullPolicy":"on-create","startupOrder":0,"type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-agent:1.1","imageHash":"sha256:85d5b4a5ab9460f3919d92f511e6b969525c53bec450b82b1f67aefc22f6e25b","createOptions":"{\"Labels\":{\"net.azure-devices.edge.create-options\":\"{\\\"HostConfig\\\":{\\\"Binds\\\":[\\\"/var/lib/edgeAgent/:/tmp/edgeAgent\\\"],\\\"LogConfig\\\":{\\\"Config\\\":{\\\"max-size\\\":\\\"10m\\\",\\\"max-file\\\":\\\"3\\\"},\\\"Type\\\":\\\"json-file\\\"}}\",\"net.azure-devices.edge.env\":\"{\\\"RuntimeLogLevel\\\":{\\\"value\\\":\\\"debug\\\"}}\",\"net.azure-devices.edge.owner\":\"Microsoft.Azure.Devices.Edge.Agent\"}"},"env":{"RuntimeLogLevel":{"value":"debug"}},"javaDeviceSdkIssue":{"exitCode":0,"statusDescription":"running","lastStartTimeUtc":"2021-05-12T00:42:06.0647339","lastExitTimeUtc":"2021-05-12T00:41:21.9783683","restartCount":3,"lastRestartTimeUtc":"2021-05-12T00:42:05.3882285Z","runtimeStatus":"running","version":"","status":"running","restartPolicy":"always","imagePullPolicy":"on-create","type":"docker","settings":{"image":"edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64","imageHash":"sha256:01f2bedca40729e1c74d4e9b12a454f495efcadc7d5e711ff269f62e3207bde6","createOptions":"{\"HostConfig\":{\"PortBindings\":{\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}}
<7> 2021-05-12 00:47:26.392 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - HealthRestartPlanner created Plan, with 2 command(s).
<6> 2021-05-12 00:47:26.400 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Plan execution started for deployment 38
<7> 2021-05-12 00:47:26.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Received new deployment 38
<6> 2021-05-12 00:47:26.403 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Saving edgeHub to store"
<7> 2021-05-12 00:47:26.410 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["Microsoft.Azure.Devices.Edge.Agent.Core.Commands.RemoveFromStoreCommand`1[Microsoft.Azure.Devices.Edge.Agent.Core.ModuleState]"] succeeded.
<6> 2021-05-12 00:47:26.428 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"
<6> 2021-05-12 00:47:26.431 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)"
<7> 2021-05-12 00:47:26.433 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to prepare update for module edgeHub
<7> 2021-05-12 00:47:26.446 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.446 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.446 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/edgeHub/prepareupdate?api-version=2020-07-07
<7> 2021-05-12 00:47:26.638 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NoContent
<7> 2021-05-12 00:47:26.642 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for prepare update for module edgeHub
<7> 2021-05-12 00:47:26.645 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to stop module edgeHub
<7> 2021-05-12 00:47:26.652 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.653 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:26.653 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/edgeHub/stop?api-version=2020-07-07
<7> 2021-05-12 00:47:37.540 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NoContent
<7> 2021-05-12 00:47:37.542 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for stop module edgeHub
<7> 2021-05-12 00:47:37.544 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to update module edgeHub
<7> 2021-05-12 00:47:37.556 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:37.556 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:37.556 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/edgeHub?api-version=2020-07-07
<7> 2021-05-12 00:47:37.834 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:47:37.839 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for update module edgeHub
<7> 2021-05-12 00:47:37.840 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["update"] succeeded.
<6> 2021-05-12 00:47:37.840 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Start module edgeHub"
<7> 2021-05-12 00:47:37.841 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to start module edgeHub
<7> 2021-05-12 00:47:37.848 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:37.848 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:47:37.849 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/edgeHub/start?api-version=2020-07-07
<7> 2021-05-12 00:47:38.772 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NoContent
<7> 2021-05-12 00:47:38.774 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for start module edgeHub
<7> 2021-05-12 00:47:38.775 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["start"] succeeded.
<7> 2021-05-12 00:47:38.775 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["Command Group: (\n  [Command Group: (\n  [Prepare update module edgeHub]\n  [Stop module edgeHub]\n  [Update module edgeHub]\n)]\n  [Start module edgeHub]\n)"] succeeded.
<6> 2021-05-12 00:47:38.776 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Plan execution ended for deployment 38

I want to call out that in the above logs I do see this error which looks similar to the one in your custom modules. But this error looks a bit different and results in the container failing out and restarting successfully.

azureuser@log-upload-vm2:~$ iotedge logs javaDeviceSdkIssue
Start to create client
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Client created
java.io.IOException: Could not open the connection
    at com.microsoft.azure.sdk.iot.device.DeviceIO.open(DeviceIO.java:162)
    at com.microsoft.azure.sdk.iot.device.InternalClient.open(InternalClient.java:175)
    at com.edgemodule.App.main(App.java:77)
Caused by: com.microsoft.azure.sdk.iot.device.transport.amqps.exceptions.ProtonIOException: Connection refused
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsExceptionTranslator.convertFromAmqpException(AmqpsExceptionTranslator.java:84)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection.onTransportError(AmqpsIotHubConnection.java:484)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.IotHubReactor.run(IotHubReactor.java:25)
    at com.microsoft.azure.sdk.iot.device.transport.amqps.AmqpsIotHubConnection$ReactorRunner.call(AmqpsIotHubConnection.java:1115)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I also want to point out that on this upgrade scenario, the custom modules don't restart even with a policy of always. I am not sure why, but it appears that this is not behaving like I would expect. For example when I restart edge agent you will see the below log. I am still discussing with multiple other team members regarding the custom module restart behavior.

<6> 2021-05-12 00:56:11.185 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Command Group: (\n  [Stop module javaDeviceSdkIssue]\n  [Start module javaDeviceSdkIssue]\n  [Saving javaDeviceSdkIssue to store]\n)"

Edge agent restart logs

2021-05-12 00:56:08  Starting Edge Agent
2021-05-12 00:56:08  Changing ownership of storage folder: /tmp/edgeAgent to 1000
2021-05-12 00:56:08  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2021-05-12 00:56:08.618 +00:00 Edge Agent Main()
<6> 2021-05-12 00:56:08.812 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Initializing Edge Agent.
<6> 2021-05-12 00:56:08.970 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-12 00:56:08.971 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2021-05-12 00:56:09.021 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<7> 2021-05-12 00:56:09.041 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to TrustBundleAsync
<7> 2021-05-12 00:56:09.066 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:09.071 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:09.075 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/trust-bundle?api-version=2019-01-30
<7> 2021-05-12 00:56:09.100 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:09.127 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for TrustBundleAsync
<6> 2021-05-12 00:56:09.143 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Installing certificates [CN=Test Edge Owner CA:07/05/2021 06:05:39] to Root
<6> 2021-05-12 00:56:09.289 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics
<6> 2021-05-12 00:56:09.458 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Updating performance metrics every 05m:00s
<6> 2021-05-12 00:56:09.463 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Started operation Get system resources
<6> 2021-05-12 00:56:09.465 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Collecting metadata metrics
<7> 2021-05-12 00:56:09.469 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2021-05-12 00:56:09.476 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:09.476 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:09.476 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2020-07-07
<7> 2021-05-12 00:56:09.508 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:09.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2021-05-12 00:56:09.539 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Set metadata metrics: 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"linux","Architecture":"x86_64","Version":"1.1.1","Provisioning":{"Type":"manual.device_connection_string","DynamicReprovisioning":false},"ServerVersion":"20.10.5+azure","KernelVersion":"5.4.0-1043-azure","OperatingSystem":"Ubuntu 18.04.5 LTS","NumCpus":2,"Virtualized":"yes"}, True
<6> 2021-05-12 00:56:09.556 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Metrics.DeploymentMetrics] - Started operation Checkpoint Availability
<6> 2021-05-12 00:56:09.560 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Started operation refresh twin config
<7> 2021-05-12 00:56:09.566 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Initialize twin refresh with interval '01:00:00'
<7> 2021-05-12 00:56:09.573 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Begin refreshing twin from upstream...
<7> 2021-05-12 00:56:09.577 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Getting module client to refresh the twin with retrying set to False
<6> 2021-05-12 00:56:09.595 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2021-05-12 00:56:10.061 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Modules.AgentModule] - Created persistent store at /tmp/edgeAgent
<6> 2021-05-12 00:56:10.161 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Scrape
<6> 2021-05-12 00:56:10.161 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<7> 2021-05-12 00:56:10.164 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Waiting on shutdown handler to trigger
<7> 2021-05-12 00:56:10.456 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Connected with reason Connection_Ok
<6> 2021-05-12 00:56:10.468 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<7> 2021-05-12 00:56:10.475 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2021-05-12 00:56:10.475 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:10.476 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:10.476 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2020-07-07
<7> 2021-05-12 00:56:10.481 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Device client for edge agent created.
<7> 2021-05-12 00:56:10.495 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ConfigSources.TwinConfigSource] - TwinConfigSource Created
<7> 2021-05-12 00:56:10.538 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Edge agent config backup created here - /tmp/edgeAgent/backup.json
<7> 2021-05-12 00:56:10.550 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:10.551 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2021-05-12 00:56:10.556 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadModuleLogs
<6> 2021-05-12 00:56:10.556 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler GetModuleLogs
<6> 2021-05-12 00:56:10.557 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadSupportBundle
<6> 2021-05-12 00:56:10.557 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler RestartModule
<6> 2021-05-12 00:56:10.597 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClient] - Initialized new module client with subscriptions enabled
<7> 2021-05-12 00:56:10.600 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to Decrypt
<7> 2021-05-12 00:56:10.601 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Got module client to refresh the twin
<7> 2021-05-12 00:56:10.618 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:10.618 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:10.618 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/modules/%24edgeAgent/genid/637532847882281426/decrypt?api-version=2020-07-07
<7> 2021-05-12 00:56:10.660 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:10.665 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for Decrypt
<6> 2021-05-12 00:56:10.677 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Obtained Edge agent twin from IoTHub with desired properties version 38 and reported properties version 221.
<7> 2021-05-12 00:56:10.818 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Edge agent updated deployment config from desired properties.
<7> 2021-05-12 00:56:10.818 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Updated edge agent configuration from upstream twin.
<7> 2021-05-12 00:56:10.822 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Edge agent created.
<7> 2021-05-12 00:56:10.826 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation
<7> 2021-05-12 00:56:10.840 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules
<7> 2021-05-12 00:56:10.847 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:10.847 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:10.847 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2020-07-07
<7> 2021-05-12 00:56:10.854 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:10.854 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config...
<7> 2021-05-12 00:56:10.892 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules
<7> 2021-05-12 00:56:10.943 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to Encrypt
<7> 2021-05-12 00:56:10.955 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:10.955 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2021-05-12 00:56:10.956 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/modules/%24edgeAgent/genid/637532847882281426/encrypt?api-version=2020-07-07
<7> 2021-05-12 00:56:10.988 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2021-05-12 00:56:10.992 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for Encrypt
<7> 2021-05-12 00:56:11.012 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Obtained edge agent config
<7> 2021-05-12 00:56:11.044 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - List of desired modules is - {"edgeAgent":{"version":"","status":"running","restartPolicy":"always","imagePullPolicy":"on-create","startupOrder":0,"type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-agent:1.1","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeAgent/:/tmp/edgeAgent\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"}}"},"env":{"RuntimeLogLevel":{"value":"debug"}},"edgeHub":{"status":"running","restartPolicy":"always","version":"","imagePullPolicy":"on-create","type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-hub:1.1","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeHub/:/tmp/edgeHub\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"},\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}},"javaDeviceSdkIssue":{"status":"running","restartPolicy":"always","version":"","imagePullPolicy":"on-create","type":"docker","settings":{"image":"edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64","createOptions":"{\"HostConfig\":{\"PortBindings\":{\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}}
<7> 2021-05-12 00:56:11.084 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - List of current modules is - {"edgeAgent":{"exitCode":0,"statusDescription":"running","lastStartTimeUtc":"2021-05-12T00:56:08.5313602","lastExitTimeUtc":"2021-05-12T00:56:07.5303032","runtimeStatus":"running","imagePullPolicy":"on-create","startupOrder":0,"type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-agent:1.1","imageHash":"sha256:85d5b4a5ab9460f3919d92f511e6b969525c53bec450b82b1f67aefc22f6e25b","createOptions":"{\"Labels\":{\"net.azure-devices.edge.create-options\":\"{\\\"HostConfig\\\":{\\\"Binds\\\":[\\\"/var/lib/edgeAgent/:/tmp/edgeAgent\\\"],\\\"LogConfig\\\":{\\\"Config\\\":{\\\"max-size\\\":\\\"10m\\\",\\\"max-file\\\":\\\"3\\\"},\\\"Type\\\":\\\"json-file\\\"}}\",\"net.azure-devices.edge.env\":\"{\\\"RuntimeLogLevel\\\":{\\\"value\\\":\\\"debug\\\"}}\",\"net.azure-devices.edge.owner\":\"Microsoft.Azure.Devices.Edge.Agent\"}"},"env":{"RuntimeLogLevel":{"value":"debug"}},"edgeHub":{"restartPolicy":"always","exitCode":137,"statusDescription":"exited","lastStartTimeUtc":"2021-05-12T00:47:38.749718","lastExitTimeUtc":"2021-05-12T00:56:07.6643078","restartCount":0,"lastRestartTimeUtc":"2021-05-12T00:56:07.6643078","runtimeStatus":"backoff","status":"running","imagePullPolicy":"on-create","type":"docker","settings":{"image":"mcr.microsoft.com/azureiotedge-hub:1.1","imageHash":"sha256:df894fd144fd8f15ca02493b3548e7aeb276ef4cc0f33609747eb000503ab9c6","createOptions":"{\"HostConfig\":{\"Binds\":[\"/var/lib/edgeHub/:/tmp/edgeHub\"],\"LogConfig\":{\"Config\":{\"max-size\":\"10m\",\"max-file\":\"3\"},\"Type\":\"json-file\"},\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}},"javaDeviceSdkIssue":{"exitCode":143,"statusDescription":"exited","lastStartTimeUtc":"2021-05-12T00:42:06.0647339","lastExitTimeUtc":"2021-05-12T00:55:57.5726009","restartCount":0,"lastRestartTimeUtc":"2021-05-12T00:55:57.5726009","runtimeStatus":"backoff","version":"","status":"running","restartPolicy":"always","imagePullPolicy":"on-create","type":"docker","settings":{"image":"edgebuilds.azurecr.io/andsmi/javadevicesdkissue:0.0.6-amd64","imageHash":"sha256:01f2bedca40729e1c74d4e9b12a454f495efcadc7d5e711ff269f62e3207bde6","createOptions":"{\"HostConfig\":{\"PortBindings\":{\"443/tcp\":[{\"HostPort\":\"443\"}]}}"},"env":{}}
<6> 2021-05-12 00:56:11.147 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.RestartPolicyManager] - Module 'edgeHub' scheduled to restart after 10s (06s left).
<7> 2021-05-12 00:56:11.166 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Planners.HealthRestartPlanner] - HealthRestartPlanner created Plan, with 1 command(s).
<6> 2021-05-12 00:56:11.182 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Plan execution started for deployment 38
<7> 2021-05-12 00:56:11.182 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Received new deployment 38
<6> 2021-05-12 00:56:11.185 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Command Group: (\n  [Stop module javaDeviceSdkIssue]\n  [Start module javaDeviceSdkIssue]\n  [Saving javaDeviceSdkIssue to store]\n)"
<6> 2021-05-12 00:56:11.189 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Stop module javaDeviceSdkIssue"
<7> 2021-05-12 00:56:11.192 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to stop module javaDeviceSdkIssue
<7> 2021-05-12 00:56:11.202 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:11.202 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:11.202 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/javaDeviceSdkIssue/stop?api-version=2020-07-07
<7> 2021-05-12 00:56:11.205 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NotModified
<7> 2021-05-12 00:56:11.214 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Error when getting an Http response from unix:///var/run/iotedge/mgmt.sock for stop module javaDeviceSdkIssue
HTTP Response: 



Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.GeneratedCode.SwaggerException: Not Modified

Status: 304
Response: 

   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.GeneratedCode.EdgeletHttpClient.StopModuleAsync(String api_version, String name, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/generatedCode/EdgeletHttpClient.cs:line 719
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.<>c__DisplayClass34_0.<<Execute>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 117
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 126
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 126
<7> 2021-05-12 00:56:11.244 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["stop"] succeeded.
<6> 2021-05-12 00:56:11.244 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Start module javaDeviceSdkIssue"
<7> 2021-05-12 00:56:11.246 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to start module javaDeviceSdkIssue
<7> 2021-05-12 00:56:11.254 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:11.254 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2021-05-12 00:56:11.254 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules/javaDeviceSdkIssue/start?api-version=2020-07-07
<7> 2021-05-12 00:56:12.000 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NoContent
<7> 2021-05-12 00:56:12.001 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for start module javaDeviceSdkIssue
<7> 2021-05-12 00:56:12.003 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["start"] succeeded.
<6> 2021-05-12 00:56:12.003 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command: "Saving javaDeviceSdkIssue to store"
<7> 2021-05-12 00:56:12.010 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["Microsoft.Azure.Devices.Edge.Agent.Core.Commands.AddToStoreCommand`1[Microsoft.Azure.Devices.Edge.Agent.Core.ModuleState]"] succeeded.
<7> 2021-05-12 00:56:12.010 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory] - Executing command for operation ["Command Group: (\n  [Stop module javaDeviceSdkIssue]\n  [Start module javaDeviceSdkIssue]\n  [Saving javaDeviceSdkIssue to store]\n)"] succeeded.
<6> 2021-05-12 00:56:12.012 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner] - Plan execution ended for deployment 38

Here are my recommendations / questions that will hopefully steer towards mitigation:

It appears that docker is behaving strangely in that Edge Hub container is not found. I notice that you are on an old version. Recommend updating versions as I have seen some very strange behavior on old docker versions.
It is possible switching the protocol that your custom modules communicate with will resolve the issue, though I don't think it is super likely.
In my reproduction attempt, my custom module always failed out when getting connection refused. Do you have custom logic in your module that prevents it from doing so? I think a mitigation would be to deploy new module code which will fail fast in the case where you have connection refused. Then the runtime should be able to restart your custom modules.
I notice that your stack traces look a bit different. You are using ModuleClient from the devices SDK? I see you are also using the provisioning sdk, but based on the stack I didn't think your custom modules were using it.

Let me know if you have feedback for my repro attempt. I can tweak things and retry if you spot anything is off.

and-rewsmith on 12 May 2021

Thanks for the mitigation steps @and-rewsmith, it will take some time for us to work through them.

We will try updating docker version in dev and see if it works
We were seeing issues with MQTT and recently switched to AMQPS for module communication, but unfortunately I can't remember exactly what those issues were.
We can look into failing fast on certain statuses and make our containers restart. Restarting has always solved any issues we've seen, but I'd like to see it as a last resort because the edge hub connection has always been reliable so far.
We are using both, this is a legacy aspect of our system. IIRC when we started working on this, only HSM and manual provisioning was available for edge. We needed certificate provisioning, so one module actually connects as a device. This is technical debt we will fix (sooner or later :)) and only use modules

Since you can't reproduce this, do you want to close this? I'm fairly certain we can solve this using suggestion 3.

I am still planning to get back to you with logs for how our modules behave when rolling out a new deployment with edge 1.0.10.

jokarl on 12 May 2021

Yep I will close it.

I am still planning to get back to you with logs for how our modules behave when rolling out a new deployment with edge 1.0.10.

I did some more testing and found that custom modules will restart on a new edge agent deployment only if the restartPolicy is set to always and the edge agent folder is not mounted. We are reviewing this behavior and might log a bug.

and-rewsmith on 12 May 2021

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Prometheus Regex doesn't match prometheus exposition format

alaendle · 5Comments

Error creating a device-to-cloud connection

yenseaman · 5Comments

Updating modules

nhuurnink · 6Comments

Throttling IoT Edge Upload Speeds?

ejectbutton · 4Comments

Errors running iotedge check w/proxy enabled

Lexmark-pcarey · 4Comments

Iotedge: Modules never reconnect after updating from 1.0.x to 1.1.x

Expected Behavior

Current Behavior

Steps to Reproduce

Update security daemon

Manually create host directory for edgeHub

Deploy new edge container versions

Context (Environment)

Output of iotedge check

Device Information

Runtime Versions

Logs

Additional Information

All 17 comments

Related issues

Output of `iotedge check`