Ddev: Windows 10: corrupted CA certificate: failed to read the CA certificate: unexpected content

[rfay]

Describe the bug

It starts with a message on start like:
ddev-router failed to become ready: logOutput=, err=container exited, please use 'ddev logs -s ddev-router to find out why it failed`

That's actually an error, and it should say to use docker logs ddev-router

Then you do the docker logs ddev-router and you see:
Using the local CA at "/root/.local/share/mkcert" ✨ ERROR: failed to read the CA certificate: unexpected content

If this happens to you,

Before beginning, please ddev poweroff && ddev start to get back to the beginning and demonstrate the error.

1. Please report the results of ddev exec ls -lR /mnt/ddev-global-cache/mkcert
2. ddev exec sudo rm -rf /mnt/ddev-global-cache/mkcert && ddev poweroff && ddev start might fix it.
3. If not, docker volume rm ddev-global-cache && ddev start will delete ddev's global cache (which contains the root CAs and composer cache)
4. If not, try to fix the CA certs on the host side with mkcert -uninstall && rm -rf "$(mkcert -CAROOT)" && mkcert -install && ddev poweroff && ddev start

Please report your results here.

[maddy2101]

I managed to create a scenario for how-to-reproduce. Still I think it is an edge case, and no one in their right mind would attempt such folly.
The use case for my setup is to have a cleanly checked out and built version of the project under development, in order to deploy it to a target server.

Given a fully functional ddev based project with the name 'intranet' (as seen in .ddev/config.yaml). The project must be stopped.

mkdir -p .ddev/deploy/transfer
cd .ddev/deploy/transfer
git clone <your project> .
# this is what causes the failure - rename the project avoiding ddev API
sed -i 's/intranet/intranet-deploy/ .ddev/deploy/transfer/.ddev/config.yaml
ddev start

Still, this is working just fine. But when you have this 'inner' project active and also attempt to start the outer one, you will run into the mentioned failure.

[CreateSean]

I just ran into this issue and step 4 solved it for me. Thank you @rfay

[Martin-Ernst]

it happened to me and the output of ddev exec ls -lR /mnt/ddev-global-cache/mkcert is
Project is paused. Run 'ddev start' to start it.

However start will fail because of the failure of ddev-router.

docker volume rm ddev-global-cache helped me though

[rfay]

@maddy2101 I was unable to reproduce the error using your instructions, Windows 10, Docker desktop for Windows 2.3.0.4 (with the default WSL2 backend, not the older Hyper-V backend), running in git-bash on the Windows side.

It is pretty wild to have another project inside a project's .ddev directory though.

[janisint]

Happended to me shortly after the update from 1.14 to 1.15.3
I did all the steps described by rfay in my git bash on Windows 10. Still get that error.

Result for ddev exec ls -lR /mnt/ddev-global-cache/mkcert:
Project is paused. Run 'ddev start' to start it.

it happened to me and the output of ddev exec ls -lR /mnt/ddev-global-cache/mkcert is
Project is paused. Run 'ddev start' to start it.

However start will fail because of the failure of ddev-router.

docker volume rm ddev-global-cache helped me though

Unfortunately this doesn't work for me.
Are there any other things to try?

[rfay]

@janisint is it possible you didn't do a ddev poweroff before upgrading, as is required? You'll want to do that. Please ddev poweroff and then ddev start, then do these steps again. I edited the OP to encourage that.

[janisint]

@rfay No, I am quite sure that I did ddev poweroff before.

Luckily I found the problem.
The error happended to me after I got a 502 bad gateway error on one of my TYPO3 installations.
Apparently, the database was destroyed. Here is what fixed it for me:

1. Navigate to the project folder (within the cls)
2. Execute docker volume ls
3. Find the volume with a name like "YourProject-mariadb" (can vary)
4. Execute docker volume rm YourProject-mariadb

Warning:
This deletes your database! You cannot restore it after!!!
If you need the db, make a ddev snapshot first.

[rfay]

@janisint if you were getting a 502, that wouldn't be related to this issue, which is about a corrupted CA.

[janisint]

Okay, but all I know for sure is that the steps described in my other post solved the issue for me.

[plousia]

I had this problem and none of the suggested steps worked for me. I kept getting:
Error response from daemon: Conflict: volume is in use

What worked for me was running docker container prune and docker volume prune (from one of the answers to this StackOverflow question)

(ETA: this did blow away my local db and I had to do a pull from the remote environment, but nothing else was working)

[tobiasgraeber]

I can confirm the case like @janisint described. So https://github.com/drud/ddev/issues/2415#issuecomment-669735297 worked for me as well.

Environment: Win10, latest Docker Desktop for Windows Version. Seems like a bug/issue that made its way into the project with one of the latest updates... (was running fine all the time until a week or two ago...)

Doing a ddev poweroff && ddev start as statet in OP did not help.

Before that, the issue occures after a ddev start.

Note, the first visible error message is:

failed to copy root CA into docker volume: container run failed with exit code 1, output='cp: read error: Function not implemented cp: read error: Function not implemented '

process still running further until creating ddev-router ...done and then it echos:

Failed to start XXXXXX: ddev-router failed to become ready: logOutput=, err=container exited, please use 'ddev logs -s ddev-router to find out why it failed`

so...doing a 'ddev logs -s ddev-router outputs:

Using the local CA at "/root/.local/share/mkcert" ✨ ERROR: failed to read the CA certificate: unexpected content

---

As described by janisint, removing and recreating the docker volume works (until now) but the real underlying reason seems not to be found until now ... maybe somebody else has an idea what´s really happening here ...?

[rfay]

Ah, the famous "Function not implemented". That's a docker bug that was fixed some time ago. What version of Docker Desktop do you have @tobiasgraeber? That may be the missing link here. Current stable is 2.3.0.4. I'm pretty astonished that deleting the *_mariadb volume would affect this as you say, have no idea how that could be.

[tobiasgraeber]

docker version states: 19.03.12; API Version 1.40. I was also astonished that deleting the *_mariadb volume would affect this but it´s defenetyl the case (or doing so produces a unknown positive side-effect that leads to the solution here)

[rfay]

@tobiasgraeber I was asking for the Docker Desktop version - in "About Docker Desktop".

[tobiasgraeber]

@rfay Sorry. About Docker Desktop shows: 2.3.0.4 (46911), see https://ibb.co/HV3tFTh

[rfay]

Thanks. I'm surprised at that, I thought the fixes for that bug had made it into stable Docker Desktop for Windows. I don't imagine this is easy to recreate, but it will be interesting if

1. Other folks who have this problem see "function not implemented"
2. The edge version of Docker Desktop would solve it (but that only helps if it's recreatable)

[rfay]

I talked to the Docker Desktop team, and they say there are actually a couple of "function not implemented" fixes that haven't made it into stable yet.

I had a quick look at the git log -- I think there might be 2 relevant fixes only in edge:

1. a fix for fchown
2. a fix for deleted files which are still open
   The next stable 2.4.0.0 with these fixes is due to be released from the edge 2.3.6.0 branch in this sprint

[rfay]

Oh, one further question for you @tobiasgraeber - It's important to know whether you're using the WSL2 backend or the traditional Hyper-V backend (this is a selection on the main preferences page). Note that Docker's future is to use the WSL2 backend, regardless of whether you develop in WSL2 or run ddev in WSL2.

[tobiasgraeber]

Hyper V

[tobiasgraeber]

@rfay sounds great - thank you for your efforts!

[rfay]

Thanks. You'll want to change over to WSL2 for the back-end at some point. WSL2 is now available for 1903.1049, 1909.1049, 2004 and higher.

[rfay]

I'm glad we haven't been hearing about this for a couple of months. Maybe there was a docker fix. Closing for now.