Iotedge: Edge running on AKS is not updating module images with new tag and version in IotHub

[ms-amakumar]

Hey all,

We have an edge device running on AKS and our release pipeline on every deployment updates the image tag as well as module version. We are using automatic deployments. I see that the deployments are being applied on the devices, but the module image is actually not updating iot hub.

I have been using Helm chart from below
https://edgek8s.blob.core.windows.net/staging

Screenshot for reference:

Edge agent version: Version - 1.0.10-dev.35780923 (0d09dc260e49baed02df9021a0046610d081bc0a)
Edge agent image: azureiotedge/azureiotedge-agent:0.1.0-beta11
Iotedged image: azureiotedge/azureiotedge-iotedged:0.1.0-beta11

Also, Iotedged pod seems to be generating every log as error.

Expected Behavior

Tell us what should happen
Module images should be correctly updated whenever a new deployment is made with different image tag.

Current Behavior

Module images are still showing old values while the deployment is already applied to edge device.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Setup an edge device in AKS cluster
2. Create a deployment and apply it to the device.
3. Create another deployment with same priority and update the module tag to a higher version.
4. Validate image tag in Iot Hub.

Context (Environment) - PreProd

Output of iotedge check

Click here

<Paste here between the triple backticks>

Device Information

• Host OS [e.g. Ubuntu 16.04, Ubuntu 18.04, Windows IoT Core]: Ubuntu 18.04
• Architecture [e.g. amd64, arm32, arm64]: amd64
• Container OS [e.g. Linux containers, Windows containers]: Linux

Runtime Versions

• iotedged [run iotedge version]: azureiotedge/azureiotedge-iotedged:0.1.0-beta11
• Edge Agent [image tag (e.g. 1.0.0)]: azureiotedge/azureiotedge-agent:0.1.0-beta11
• Edge Hub [image tag (e.g. 1.0.0)]: 1.0.0
• Docker/Moby [run docker version]: Docker

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

iotedged logs

<Paste here between the triple backticks>

edge-agent logs

<Paste here between the triple backticks>

edge-hub logs

<Paste here between the triple backticks>

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

[darobs]

Thank you @ms-amakumar, I have started investigating.

[github-actions[bot]]

This issue is being marked as stale because it has been open for 30 days with no activity.

[alessandro-gargiulo]

Is there any update on this issue?
I ran in the same issue some days ago in Azure AKS

[veyalla]

We were able to repro and prepping a fix in the next beta release.

[veyalla]

This fix released today as part of beta13.

[alessandro-gargiulo]

I ran in this issue again after a period in which the problem seemed to be fixed.

Versions installed:
Proxy: azureiotedge/azureiotedge-proxy:0.1.0-beta13
Agent: azureiotedge/azureiotedge-agent:0.1.0-beta13

I updated image tag version by launching set-modules command of azure-cli at 11.58 +00:00
The update was recognized by edgeagent as shown in its logs below:

<6> 2021-02-25 11:58:16.018 +00:00 [INF] - Plan execution started for deployment 35
<6> 2021-02-25 11:58:16.018 +00:00 [INF] - Executing command: "Create an EdgeDeployment with modules: [Module1, edgeAgent, edgeHub, Module2, Module3, Module4]"
<6> 2021-02-25 11:58:16.089 +00:00 [INF] - Executing command: "Report EdgeDeployment status: [Success]"
<6> 2021-02-25 11:58:16.089 +00:00 [INF] - Plan execution ended for deployment 35
<6> 2021-02-25 11:58:22.601 +00:00 [INF] - Plan execution started for deployment 36
<6> 2021-02-25 11:58:22.601 +00:00 [INF] - Executing command: "Create an EdgeDeployment with modules: [Module1, edgeAgent, edgeHub, Module2, Module3, Module4]"
<6> 2021-02-25 11:58:22.789 +00:00 [INF] - Executing command: "Report EdgeDeployment status: [Success]"
<6> 2021-02-25 11:58:22.789 +00:00 [INF] - Plan execution ended for deployment 36
<6> 2021-02-25 11:58:23.189 +00:00 [INF] - Updated reported properties
<6> 2021-02-25 11:58:28.327 +00:00 [INF] - Plan execution started for deployment 36
<6> 2021-02-25 11:58:28.327 +00:00 [INF] - Executing command: "Create an EdgeDeployment with modules: [Module1, edgeAgent, edgeHub, Module2, Module3, Module4]"
<6> 2021-02-25 11:58:28.347 +00:00 [INF] - Executing command: "Report EdgeDeployment status: [Success]"
<6> 2021-02-25 11:58:28.347 +00:00 [INF] - Plan execution ended for deployment 36

But nothing seems to happen in Kubernetes neither in the reported properies of the modules

This issue seems to appear after few days of modules deployment inactivity.

[veyalla]

@alessandro-gargiulo We ran into the same issue with extended testing. We found some other paths in the code where the IoT Edge CRD watch can get dropped which is the most likely cause for this. I have a private (x64) version of edgeAgent we're currently testing: veyalla/ea:feb24.1. Could you use this in your cluster and let us know if it does better? You'll need to use the following command to install (in a new namespace):

helm install --repo https://edgek8s.blob.core.windows.net/staging edge1 edge-kubernetes \
  --namespace helloworld1 \
  --set "edgeAgent.image.repository=veyalla/ea" \
  --set "edgeAgent.image.tag=feb24.1" \
  --set "provisioning.deviceConnectionString=$connStr"

FYI @darobs

[alessandro-gargiulo]

Thank you for support @veyalla
I will try your image and let you know.

[alessandro-gargiulo]

The issue still exist with your private image.

Here the images version of my setup:

• edgeagent: veyalla/ea:feb24.1
• iotedged: azureiotedge/azureiotedge-iotedged:0.1.0-beta13
• edgehub: mcr.microsoft.com/azureiotedge-hub:1.0

[veyalla]

Thanks for trying, can you provide:

Kubernetes distribution? (AKS, K3s, OpenShift etc.)
Kubernetes version? (1.16, 1.19 etc)

[alessandro-gargiulo]

Thanks for trying, can you provide:

Kubernetes distribution? (AKS, K3s, OpenShift etc.)
Kubernetes version? (1.16, 1.19 etc)

Kubernetes distribution: AKS
Kubernetes version: 1.18.10

[veyalla]

Thanks, per our analysis so far, it seems like the failure is limited to AKS clusters with 1.18.x k8s version. I have not been able to repro this on AKS with 1.19 k8s or on-prem clusters. Our theory is that there is something specific to 1.18+AKS combination that is causing the failure.

Can you able to test with 1.19.x cluster with AKS? We haven't been able to get a repro with this combination so far.

[alessandro-gargiulo]

Ok i will try with 1.19.x version of Kubernetes on Azure.

[alessandro-gargiulo]

@veyalla following further tests in AKS with 1.19.7 version of kubernetes the issue still persists.
I've tried both the official and your edgeagent image.

[veyalla]

Thanks for the update, I think we have narrowed it down to being impacted by this AKS-specific issue:
pod watches stop sending data after a while, but remain connected in 1.18.4 · Issue #1755 · Azure/AKS.

This explains why we can only repro the issue on AKS.

As recommended in the issue, we have been testing with watch timeout set to under 4 min, and so far it is looking good. We'll
continue testing for a bit longer and provide an updated release with the fix if everything checks out. Let me know if this is blocking you and we'll get you another private image before the next beta release.

[alessandro-gargiulo]

As recommended in the issue, we have been testing with watch timeout set to under 4 min, and so far it is looking good. We'll
continue testing for a bit longer and provide an updated release with the fix if everything checks out. Let me know if this is blocking you and we'll get you another private image before the next beta release.

It would be great if you could provide me another private image before the next beta release. We are deployng continuously new image versions of our application and it's getting frustrating to install the workload again for each deployment. So your help is greatly appreciated.

[veyalla]

Here is the newest private image to use: darobs/azureiotedge-agent:20210317.7

[veyalla]

Beta14 is released with a fix for this. It looked good in our extended testing, so closing this issue. Please let us know if this doesn't work for you.