Azure-docs: How to shutdown an IoTEdge device?

[djaus2]

I have my CSharpModule running on a MinnowBoardMax with OS Win 10 IoT-Core.
If I shut the device down, when I bring it back up the modules, apart from EdgeAgent fail to start.
Is there a correct way to shut such a device down and to bring it back up?

 iotedge list
NAME             STATUS           DESCRIPTION                         CONFIG
CSharpModule     failed           Failed (3221225786) 11 minutes ago  djauscontainerreg.azurecr.io/csharpmodule:0.0.1-windows-amd64
edgeHub          failed           Failed (3221226219) 10 minutes ago  mcr.microsoft.com/azureiotedge-hub:1.0
tempSensor       failed           Failed (3221225786) 11 minutes ago  mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
edgeAgent        running          Up 2 minutes                        mcr.microsoft.com/azureiotedge-agent:1.0

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 181d5562-f8d7-9c59-85ea-09efbe397f3e
• Version Independent ID: dbedd952-a050-286e-c951-4529ec8e87b8
• Content: Tutorial create custom C# module - Azure IoT Edge
• Content Source: articles/iot-edge/tutorial-csharp-module.md
• Service: iot-edge
• GitHub Login: @kgremban
• Microsoft Alias: kgremban

[djaus2]

I find I have to completely remove the Edge and then reinstate it:

[192.168.0.26]: PS C:\iotedge> iotedge list
NAME             STATUS           DESCRIPTION      CONFIG
edgeAgent        running          Up 30 seconds    mcr.microsoft.com/azureiotedge-agent:1.0
tempSensor       running          Up 3 minutes     mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
edgeHub          running          Up 2 minutes     mcr.microsoft.com/azureiotedge-hub:1.0
CSharpModule     running          Up 2 minutes     djauscontainerreg.azurecr.io/csharpmodule:0.0.1-windows-amd64

[Alberto-Vega]

@djaus2 Thanks for the feedback! We are currently investigating and will update you shortly.

[djaus2]

Restarted my device today (powered it up). Some logs:

19/01/2019 11:21:30 PM info: edgelet_http::logging -- [mgmt] - - - [2019-01-19 12:21:30.809744900 UTC] "POST
                       /modules/CSharpModule/stop?api-version=2018-06-28 HTTP/1.1" 304 Not Modified 0 "-" "-" pid(344)
19/01/2019 11:21:30 PM info: edgelet_docker::runtime -- Starting module CSharpModule...
19/01/2019 11:21:31 PM warn: edgelet_utils::logging -- Could not start module CSharpModule
19/01/2019 11:21:31 PM warn: edgelet_utils::logging --  caused by: network
                       9ece62ae76817c220c79eb991d921564c8a5de4501da52931ebbf1d2ca8e1f6b not found
19/01/2019 11:21:31 PM info: edgelet_http::logging -- [mgmt] - - - [2019-01-19 12:21:31.406848300 UTC] "POST
                       /modules/CSharpModule/start?api-version=2018-06-28 HTTP/1.1" 404 Not Found 196 "-" "-" pid(344)
19/01/2019 11:21:31 PM info: edgelet_docker::runtime -- Stopping module edgeHub...
19/01/2019 11:21:31 PM warn: edgelet_utils::logging -- Could not stop module edgeHub
19/01/2019 11:21:31 PM warn: edgelet_utils::logging --  caused by: Target of operation already in this state
19/01/2019 11:21:31 PM info: edgelet_http::logging -- [mgmt] - - - [2019-01-19 12:21:31.454844800 UTC] "POST
                       /modules/edgeHub/stop?api-version=2018-06-28 HTTP/1.1" 304 Not Modified 0 "-" "-" pid(344)
19/01/2019 11:21:31 PM info: edgelet_docker::runtime -- Starting module edgeHub...
19/01/2019 11:21:32 PM warn: edgelet_utils::logging -- Could not start module edgeHub
19/01/2019 11:21:32 PM warn: edgelet_utils::logging --  caused by: network
                       9ece62ae76817c220c79eb991d921564c8a5de4501da52931ebbf1d2ca8e1f6b not found

[djaus2]

Any ideas on what

caused by: network
                       9ece62ae76817c220c79eb991d921564c8a5de4501da52931ebbf1d2ca8e1f6b not found

means?

[Alberto-Vega]

@djaus2 Thanks again for the feedback! It will add value to other customers to include information on how to graciously shutdown. I have assigned the issue to the content author to evaluate and update as appropriate. @kgremban

[Alberto-Vega]

@djaus2 About the error I do not know. Is the module running properly?

[kgremban]

@djaus2 I think I saw someone else encounter this same product bug. Here's what they had to say about it:

After installation, all runs fine. But after every reboot the edgeAgent does not find its network anymore and thus fails to start the modules. It looks like the “azure-iot-edge” network is recreated on every docker restart with a different id. But the edge containers do not get notified about this by the daemon. See logs attached. The id 4e9dd0ab1a4eb3b1a3a72ff26ace2994e40ace176f821d3bfc7aee8c496a2e55 was the previous id of the azure-iot-edge network. After the reboot it’s something different. I have tried a couple of times in my own setup (on an Azure VM) and have seen the issue twice as well. “Workaround” so far seems to be to stop the edge daemon, do a “docker rm edgeAgent” and restart the daemon. Daemon will create a new container with the new network id and it will run ok until next restart. The agent then can start up the other modules again. The other containers do not seem affected.

If those steps resolve your issue, then this is probably the same bug that the product team is already aware of and working on.

This does look to be a product bug, because IoT Edge is supposed to shut down and start up cleanly. There's nothing to add to the documentation about that. In the future, if you continue to encounter errors like this, you can also create issues directly on the IoT Edge github page (https://github.com/azure/iotedge/issues).

[djaus2]

Q On the VM. how did you stop the Edge Daemon. I did it using Stop-Service iotedge

PS C:\Windows\system32> iotedge list
NAME                        STATUS           DESCRIPTION                        CONFIG
SimulatedTemperatureSensor  failed           Failed (3221225786) 2 minutes ago  mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
edgeAgent                   running          Up a minute                        mcr.microsoft.com/azureiotedge-agent:1.0
edgeHub                     failed           Failed (3221226219) 2 minutes ago  mcr.microsoft.com/azureiotedge-hub:1.0
PS C:\Windows\system32> stop-service iotedge
WARNING: Waiting for service 'iotedge (iotedge)' to stop...
WARNING: Waiting for service 'iotedge (iotedge)' to stop...

I then issued docker rm edgeAgent
That failed with the message:

PS C:\Windows\system32> docker rm edgeAgent
error during connect: Delete http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/containers/edgeAgent: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
PS C:\Windows\system32

I did make a Checkpoint of the VM when running with Edge working and was able to revert the VM to that and restart it; and Edge was working OK.

[djaus2]

I find that I do have do a complete Uninstall then Install to get it working again.
When it comes up though the SimulatedTemperatureSensor module does eventually run again with further intervention and so a Startup script calling both of these actions does work. For that I use a local version of IotEdgeSecurityDaemon.ps1

[kgremban]

@djaus2 IoT Edge 1.0.5 installs moby as the default docker engine for IoT Edge when you're running windows containers on windows devices. When you run docker commands directed at IoT Edge modules, you need to specify that you're using the moby engine by including -H npipe:////./pipe/iotedge_moby_engine in your docker commands.

If you only have the moby engine on your device and don't intend to also have Docker for Windows running, then you can create an environment variable that points docker commands to the moby engine by default.

I'm still working on how and where to document this information.

[djaus2]

Yes it would be useful to include this in the documentation. As I understand it, Moby is the only option on IoT-Core (correct me if I'm wrong). Some examples would be useful.
eg Instead of docker image list
run:

docker  -H npipe:////./pipe/iotedge_moby_engine image list

Output:

PS C:\Windows\system32> docker  -H npipe:////./pipe/iotedge_moby_engine image list
REPOSITORY                                                    TAG                 IMAGE ID            CREATED             SIZE
mcr.microsoft.com/azureiotedge-hub                            1.0                 ba85a25f1d3f        5 weeks ago         477MB
mcr.microsoft.com/azureiotedge-simulated-temperature-sensor   1.0                 4cabf4bf91a5        5 weeks ago         420MB
mcr.microsoft.com/azureiotedge-agent                          1.0                 cdf3515e0b23        5 weeks ago         462MB

[djaus2]

And finally, Post above for restarted on reboot DOES work:
Firstly:

PS C:\Windows\system32> iotedge list
NAME                        STATUS           DESCRIPTION                         CONFIG
edgeHub                     failed           Failed (3221226219) 34 minutes ago  mcr.microsoft.com/azureiotedge-hub:1.0
SimulatedTemperatureSensor  failed           Failed (3221226219) 34 minutes ago  mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
edgeAgent                   running          Up 1 second

Then issue the commands in order:

Stop-Service iotedge
docker  -H npipe:////./pipe/iotedge_moby_engine rm edgeAgent
Start-Service iotedge

Then get:

PS C:\iotedge> iotedge list
NAME                        STATUS           DESCRIPTION      CONFIG
SimulatedTemperatureSensor  running          Up 41 seconds    mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
edgeAgent                   running          Up a minute      mcr.microsoft.com/azureiotedge-agent:1.0
edgeHub                     running          Up a minute      mcr.microsoft.com/azureiotedge-hub:1.0

Wala :smiley:

[kgremban]

@djaus2 The 1.0.6 version of IoT Edge was released last week, which should have resolved this issue. I've restarted all my devices and haven't been able to recreate this issue with the latest version. Please update your devices if you haven't already, and let me know if you're still experiencing this issue.

Thanks.

[djaus2]

I updated to 1.0.6

It still fails to restart properly upon reboot. NB: Using MinnowBoardMax running IoT-Core.
Sample log:

5/02/2019 9:29:00 PM info: edgelet_docker::runtime -- Stopping module edgeHub...
5/02/2019 9:29:00 PM warn: edgelet_utils::logging -- Could not stop module edgeHub
5/02/2019 9:29:00 PM warn: edgelet_utils::logging --    caused by: Target of operation already in this state            
5/02/2019 9:29:00 PM info: edgelet_http::logging -- [mgmt] - - - [2019-02-05 10:29:00.889033200 UTC] "POST
                     /modules/edgeHub/stop?api-version=2018-06-28 HTTP/1.1" 304 Not Modified 0 "-" "-" pid(3124)
5/02/2019 9:29:00 PM info: edgelet_docker::runtime -- Starting module edgeHub...
5/02/2019 9:29:01 PM warn: edgelet_utils::logging -- Could not start module edgeHub
5/02/2019 9:29:01 PM warn: edgelet_utils::logging --    caused by: network                                              
                     3cee9ad11b6793693d3fe21e6b220bb4b8e4e5511459c12ae58d6bdd3c3ed43c not found

[djaus2]

[192.168.0.26]: PS C:\iotedge> iotedge -V
iotedge 1.0.6.1 (3fa6cbef8b7fc3c55a49a622735eb1021b8a5963)

[djaus2]

My restart-rm.ps1 script does get things going again though.
As on GitHub: djaus2/AzureIoT-CoreUtils

[Alberto-Vega]

@djaus2 Was your issue resolved?

[Alberto-Vega]

@djaus2 Closing this issue since we haven't heard back from you. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

[djaus2]

No the matter wasn't resolved as the update still exhibited the problem. On vacation this week so will revisit this next week. Maybe there is a later version to try.