Iotedge: Request: Wait for all modules to download before starting manifest

[mathiash98]

Problem:

We are using IotEdge to manage and deploy our system to vessels around the world, where quite a few of them have extremely slow internet speed. This will often lead to downtime for quite a long time when downloading updates.

Desired behavior:

When new deployment manifest is detected:

1. Download all new modules with correct version
2. When all modules are successfully downloaded: Start manifest
3. If system restarts for some reason: Run old manifest until all modules are downloaded

Scenarios:

When we deploy new software we have 2 cases:

No breaking changes between modules:

• Not really a problem as long as system is not restarted.
• If system restarts then edgeAgent will wait until the new module is downloaded and we will experience downtime for hours and some times days, due to slow internet and timeout exceptions.
  
  
  
  • Our solution to solve it faster is then to connect manually and run docker pull loop until images is downloaded.
  
  
  • Since modules are started in series, some modules will be blocked by the download of another images, since edgeAgent is waiting for the new module to be downloaded.
  
  

Breaking changes between modules:

• Same issues as above, but this time we will have modules depending on each other where one version is running old version and the other is still waiting for download. This will lead to non-working system for a long time.

Current manual solution:

Our current solution to this problem is to manually download the desired module images with docker pull before sending out deployment.

Please reach out if anything is unclear. This issue is quite important for us in continuing to use and pay for Azure IotEdge.

[sachidanand]

The same issue we are also facing ..

[varunpuranik]

@mathiash98 - Thanks for the issue and the detailed context. Your ask makes sense. I will add it to our backlog and we will consider taking it soon.
I don't have a timeline on how and when this feature will be available, yet. We will update the thread when we have something to share.
Thanks!

[myamama]

Potential Workaround (not very neat but should do the trick), Try doing it over 2 steps:

1. Step1: Pass the manifest file with new images without updating the routes and once deployment is complete (the new containers will be up and running beside the old ones however they stay orphan (no inputs or outputs) i.e old pipe is still active),
2. Step 2: Pass a new manifest excluding the old containers (Agent should delete them) and updating the routes for the new pipe including new containers. This might require some version control as outputname of modules should have same version of the module image.

[yuanzhedong]

Suffering the same issue here, our edge machines are deployed to the farms with a very poor internet connection.

[sampathweb]

@myamama - We are interested in your workaround. Can you provide a sample manifest file for the step 1 and 2. Currently, we have one manifest file with the docker image and tag. But we run into the same issue as pointed in this github issue and it takes 20 minutes to download the docker container layers and the system is offline during this entire time since the Agent doesn't start the old container in the device waiting on the download to complete.

Let us know if you can point us to some samples or ways to implement the step 1 and 2 workaround you suggested here.

[veyalla]

Please add this functionality as a new suggestion (or upvote an existing one) at https://aka.ms/iotedge-suggestion

However, feel free to continue to discuss potential strategies to help with this scenario.