Machine: "Internal failure while setting the bit: open /var/lib/docker/network/files/local-kv.db: no such file or directory"

[Delpedro]

Hi,

I have used the new command export AZURE_SUBSCRIPTION_ID=
(docker-machine create --driver azure

When I SSH into the machine via docker-machine ssh

docker info, docker ps, docker images commands work fine.

But when i run docker run -itd -P tutum/hello-world, i get the following error

docker: Error response from daemon: internal failure while setting the bit: open /var/lib/docker/network/files/local-kv.db: no such file or directory.

[nathanleclaire]

cc @ahmetalpbalkan

[ahmetb]

@nathanleclaire doesn't seem like azure related if docker ps works fine? :)

[nathanleclaire]

@ahmetalpbalkan Not really enough information to say, but yes it seems more likely to be related to daemon configuration.

@Delpedro What's the output of the Docker log? Probably something like sudo journalctl -u docker on the VM.

[Delpedro]

Hi Nathan,

Please find attached results from the command you gave me.....

Ahmet, easy way out!! As I said to you on Twitter, AWS is so much less hassle with that Docker Swarm example on docker.com :P (all these issues I'm having and this is just at the start, just kidding!)

3240_Issue_DockerLogFiles.txt

[nathanleclaire]

It seems to be upstream daemon issue.

If you can sudo rm -rf /var/lib/docker (warning: destructive operation that will remove all your images and containers) and sudo systemctl restart docker, see if it will work after that.

[Delpedro]

Hi Nathan,

That worked a treat! Can you explain what those commands do please or direct me to a link that explains them please.

Thank you!
Del.

[nathanleclaire]

Sure

• sudo rm -rf /var/lib/docker -- destroy all Docker internal files (sometimes Docker gets into an invalid state and, unfortunately, this is the only way to correct it)
• sudo systemctl restart docker - restart the Docker system service (turn it off and turn it back on again)

[Delpedro]

Nathan,

I just ran another VM from docker-machine in docker toolbox in Azure and SSH'd into it - im getting a cannot connect to the Docker daemon. Is the docker daemon running on this host

I have carried out the eval "$(docker-machine env

I even ran the two commands above

Not working, does this happen regularly when using this method to spin up VMs in a cloud infrastructure from docker-machine???

[nathanleclaire]

If you are SSHed in you can invoke docker commands directly, you don't need to do the docker-machine env song and dance. If create didn't fail the daemon should be running fine.

[nathanleclaire]

That said, if the daemon isn't running, check for its status using sudo systemctl status docker

[Delpedro]

Ok cool, I will run that command. Anything in particular I'm looking out for when running this command? This would be a pain if lets say I started to test this voting app using Docker swarm https://github.com/docker/swarm/tree/master/docs/swarm_at_scale (since it has only been done on AWS and Digital Ocean - no one has done it in Azure) and I'm spinning up like 6/8 nodes/VMs and I have to run this status docker command every time!

P.S. What is there refresh rate for the export azure_subscription_id - it seems to ask to delete stale token a number of times in a very short period......a little bit frustrating and time consuming

[nathanleclaire]

Ok cool, I will run that command. Anything in particular I'm looking out for when running this command? This would be a pain if lets say I started to test this voting app using Docker swarm (since it has only been done on AWS and Digital Ocean - no one has done it in Azure) and I'm spinning up like 6/8 nodes/VMs and I have to run this status docker command every time!

You shouldn't have to. It's to check if the daemon is up. If it's up, it should be responding to requests normally, so probably there is something mis-configured in your client or in between, not in the daemon itself.

P.S. What is there refresh rate for the export azure_subscription_id - it seems to ask to delete stale token a number of times in a very short period......a little bit frustrating and time consuming

There's a bug that limits it to 12h today. It's a known issue that the Azure (upstream API) team is working on.

[Delpedro]

Ok i get you about the daemon. However, i have no control over the configuration of that VM. Its all done by Azure no? I'm just running the command docker-machine create -d azure , i could understand if it was a user command and a setup configuration command that is causing the error, if you get me.

PS Have you tried that Docker swarm voting app in Azure (https://github.com/docker/swarm/tree/master/docs/swarm_at_scale)? As soon as i fix all these small wee irritating issue that is next on my list....as i only have access to Azure and not Digital Ocean or AWS as per the instruction on docker.com....

[nathanleclaire]

Ok i get you about the daemon. However, i have no control over the configuration of that VM. Its all done by Azure no? I'm just running the command docker-machine create -d azure , i could understand if it was a user command and a setup configuration command that is causing the error, if you get me.

No, you definitely do -- after Azure creates the resources, including the VM where Docker will be run, it hands back control of the provisioning process to Machine, who sets all of that configuration (such as whether or not to use TLS, etc.) over SSH. You can configure what will happen in this step via Machine, or do your own custom management of the VM after Docker Machine is finished running.

[nathanleclaire]

I haven't tried the voting app, but given enough resources, it should work pretty much the same on all cloud providers.

EDIT: Forgot I actually have tried it. It should work smoothly.

[Delpedro]

Nathan,

Thank you for all that information!

• I'm timing the reset of the Azure sub ID.....ill let you know later when it firsts asks me to navigate to a path and remove the stale information where it then asks the user to re-generate the Azure code for Azure Login.
• After stopping my latest node / VM (created using docker-machine c/w with new driver method) and starting it again, it appears to have removed the Docker Daemon issue, I would love to know why and how this happened)
• As soon as I can sort out issue #3239 I will try the Docker Swarm Voting app (ill try the birthday versions first) before attempting Azure. Is it OK if i pop you some questions or will i direct them in the Docker:Swarm section?
• Is there examples of this on the docker.com site or GitHub to explain the following (I do not understand this fully)
  _"You can configure what will happen in this step via Machine, or do your own custom management of the VM after Docker Machine is finished running."_
• Have you monitored containers / or VMs (nodes) of a running MySQL / MariaDB or Web Server container in a VM in Azure using J-Meter / cAdvisor by any chance?

[rafaelgamezdiaz]

Thanks Nathan!!

I was trying to do: docker-compose up -d in my proyect but I got the error message:

ERROR: failed to update bridge store for object type *bridge.networkConfiguration: open /var/lib/docker/network/files/local-kv.db: no such file or directory

I use your commands:

 sudo rm -rf /var/lib/docker -- destroy all Docker internal files (sometimes Docker gets into an invalid state and, unfortunately, this is the only way to correct it)

and

sudo systemctl restart docker - restart the Docker system service (turn it off and turn it back on again)

It resolve my problem.
Thanks Very much!!