Iotedge: Edge runtime failed to start after reboot my Linux system

[pbiswas-kol]

Expected Behavior

After reboot Linux system edge runtime should run as expected

Current Behavior

IoT Edge runtime is not working .workload.sock & mgmt.sock file missing after reboot my Linux.

Steps to Reproduce

1. Reboot my RHEL 7.5
2. After Reboot iot Edge runtime is not working. Not able to find workload.sock & mgmt.sock under /var/run/iotedge

Context (Environment)

Device (Host) Operating System

RHEL 7.5

Architecture

amd64

Container Operating System

Linux container

Runtime Versions

1.0.1

iotedged

iot edge 1.0.1

Edge Agent

mcr.microsoft.com/azureiotedge-agent:1.0.1

Edge Hub

mcr.microsoft.com/azureiotedge-hub:1.0.1

Docker

Docker version 18.09.0-dev, build 021bf39d

Additional Information

Edge runtime faild to start

[Jan-Olof]

The same thing happens to us as well on RHEL 7.5. It works fine on Ubuntu, Debian and Windows.

[darobs]

Thanks, @pbiswas-kol, @Jan-Olof

I will take a look. Do you have logs from iotedged (with any PII removed) you would be willing to share?

Thanks,
Dave

[darobs]

I had 3 problems upon a reboot:

1. The container service failed to restart (I installed our Moby package). The workaround was to run

sudo systemctl start docker

1. The /var/run/iotedge directory was not present. The workaround for this was

sudo mkdir /var/run/iotedge
sudo chown iotedge.iotedge /var/run/iotedge
sudo systemctl restart iotedge

1. The socket files were created, but they had the wrong permissions. The workaround for this was

sudo chmod 660 /var/run/iotedge/mgmt.sock
sudo chmod 666 /var/run/iotedge/workload.sock

Are these the same issues you had?

I am still investigating why all this happened.

[Jan-Olof]

@darobs Yes, that pretty much sums up our problem. Great that you are able to recreate this.

By the way, we can get rid of the Moby problem by running this once:

sudo systemctl enable docker

[pbiswas-kol]

@jan-Olof Yes you are correct. We can rid of mody restart problem on reboot by just enable the docker.

@darobs yes , this the issue that we are facing.

I reinstall edge runtime and configure config.yml again after reboot. It works. I will try your workaround. When we expect a permanent solution for this issue?

[darobs]

With distributions where systemd doesn't manage the sockets, we made a strategic error in placing the socket files in /var/run/iotedge - a directory that doesn't survive a reboot.

The plan going forward is for distributions like RHEL 7.5 is to move the location of a socket file to a different directory: /var/lib/iotedge - a directory we already create at installation and survives a reboot.

Right now, to emulate what we're planning to change (NOTE - this requires you to stop and restart iotedge):

1. enable the two services

sudo systemctl enable docker iotedge

1. Create the new location for the sockets

sudo touch /var/lib/iotedge/mgmt.sock
sudo touch /var/lib/iotedge/workload.sock
sudo chown iotedge:iotedge /var/run/iotedge/mgmt.sock
sudo chown iotedge:iotedge /var/run/iotedge/workload.sock
sudo chmod 660 /var/run/iotedge/mgmt.sock
sudo chmod 666 /var/run/iotedge/workload.sock

1. stop the iotedge service and containers.

sudo docker rm -f $(sudo docker ps -aq -f "label=net.azure-devices.edge.owner=Microsoft.Azure.Devices.Edge.Agent") && sudo systemctl stop iotedge

1. edit /etc/iotedge/config.yaml "connect" and "listen" sections to:

  management_uri: "unix:///var/lib/iotedge/mgmt.sock"
  workload_uri: "unix:///var/lib/iotedge/workload.sock"

1. Finally

sudo systemctl restart iotedge

You should see a PR for this in a day or two - If I'm lucky, we can get this into the next release.

[darobs]

I have a PR out, testing it now. #271

[pbiswas-kol]

@darobs
Command you mentioned here should be like this.

sudo touch /var/lib/iotedge/mgmt.sock
sudo touch /var/lib/iotedge/workload.sock
sudo chown iotedge:iotedge /var/lib/iotedge/mgmt.sock
sudo chown iotedge:iotedge /var/lib/iotedge/workload.sock
sudo chmod 660 /var/lib/iotedge/mgmt.sock
sudo chmod 666 /var/lib/iotedge/workload.sock

So after doing all the steps "iotedge list" commands are not working

An error in the management http client occurred.
caused by: Hyper error
caused by: Connection refused (os error 111)

And "iotedge logs XXModule" commands also not working

A module runtime error occurred.

But if we run systemctl status iotedge . I can see IoT Edge runtime is running

iotedge.service - Azure IoT Edge daemon
Loaded: loaded (/usr/lib/systemd/system/iotedge.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2018-09-14 10:51:30 CEST; 12min ago
Docs: man:iotedged(8)
Main PID: 26420 (iotedged)
Tasks: 6
Memory: 3.6M
CGroup: /system.slice/iotedge.service
└─26420 /usr/bin/iotedged -c /etc/iotedge/config.yaml

Sep 14 11:03:38 xxx iotedged[26420]: 2018-09-14T09:03:38Z [INFO] - [mgmt] - - - [2018-09-14 09:03:38.301188814 UTC] "GET /modules?api-version=2018...pid(26446)
Sep 14 11:03:38 xxx iotedged[26420]: 2018-09-14T09:03:38Z [INFO] - [work] - - - [2018-09-14 09:03:38.302829177 UTC] "POST /modules/%24edgeAgent/ge...pid(26446)
Sep 14 11:03:43 xxx iotedged[26420]: 2018-09-14T09:03:43Z [INFO] - [mgmt] - - - [2018-09-14 09:03:43.326107588 UTC] "GET /modules?api-version=2018...pid(26446)
Sep 14 11:03:43 xxx iotedged[26420]: 2018-09-14T09:03:43Z [INFO] - [work] - - - [2018-09-14 09:03:43.327485648 UTC] "POST /modules/%24edgeAgent/ge...pid(26446)
Sep 14 11:03:48 xxx iotedged[26420]: 2018-09-14T09:03:48Z [INFO] - [mgmt] - - - [2018-09-14 09:03:48.349399179 UTC] "GET /modules?api-version=2018...pid(26446)
Sep 14 11:03:48 xxx iotedged[26420]: 2018-09-14T09:03:48Z [INFO] - [work] - - - [2018-09-14 09:03:48.351180321 UTC] "POST /modules/%24edgeAgent/ge...pid(26446)
Sep 14 11:03:53 xxx iotedged[26420]: 2018-09-14T09:03:53Z [INFO] - [mgmt] - - - [2018-09-14 09:03:53.376399686 UTC] "GET /modules?api-version=2018...pid(26446)
Sep 14 11:03:53 xxx iotedged[26420]: 2018-09-14T09:03:53Z [INFO] - [work] - - - [2018-09-14 09:03:53.378147913 UTC] "POST /modules/%24edgeAgent/ge...pid(26446)
Sep 14 11:03:58 xxx iotedged[26420]: 2018-09-14T09:03:58Z [INFO] - [mgmt] - - - [2018-09-14 09:03:58.400162022 UTC] "GET /modules?api-version=2018...pid(26446)
Sep 14 11:03:58 xxx iotedged[26420]: 2018-09-14T09:03:58Z [INFO] - [work] - - - [2018-09-14 09:03:58.400864174 UTC] "POST /modules/%24edgeAgent/ge...pid(26446)
Hint: Some lines were ellipsized, use -l to show in full.

[darobs]

The error you're seeing is because the default socket for the iotedge command is unix:///var/run/iotedge/mgmt.sock
Try iotedge list --host unix:///var/lib/iotedge/mgmt.sock

[myagley]

In the 1.0.2 version of the iotedged, the rpm and deb packages have been updated with the changes @darobs describes above. Please upgrade to 1.0.2. Closing this issue for now, but feel free to reopen. Thanks.