Docker-alpine: update-ca-certificates fails

[BrianHicks]

When run under Docker, update-ca-certificates (from the package ca-certificates) fails with the line . All the information I can find seems to suggest it's a locale issue but the system profile seems to have the correct information. Any ideas?

The failure:

bash-4.3# update-ca-certificates
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

[andyshinn]

That isn't necessarily a failure. It is just saying "the ca-certificates.crt file has multiple certificates so we are not considering it to be symlinked". What is the actual problem here? Do you have an example I can try to reproduce?

[BrianHicks]

sure, CiscoCloud/nginx-consul (currently on the Docker registry as asteris/nginx-consul.) If you mount a directory with CA certs in the place mentioned in the README and run update-ca-certificates that happens. Am I misunderstanding the package instructions?

[andyshinn]

Do you have an example CA PEM or a command I can use to generate one? I'll see if I can reproduce.

To clarify: the CA is not getting added to the ca-certificates.crt file regardless of the error message?

[BrianHicks]

correct. I don't have one, but you can get one by running the security-setup script in CiscoCloud/microservices-infrastructure.

[andyshinn]

Indeed, looks like an upstream bug. The line at https://github.com/alpinelinux/aports/blob/master/main/ca-certificates/update-ca-certificates#L50 should be for key,fn in pairs(certlist) do. I'll poke upstream and try to get a patch out.

[BrianHicks]

Great, thank you!

[andyshinn]

Patch submitted: http://patchwork.alpinelinux.org/patch/104/.

[andyshinn]

Merged in https://github.com/alpinelinux/aports/commit/3faf2e032027ffa255f9a49f2f17c8e0239173ff. New package pushed to edge. Give it a try and let me know if we can close this.

[andyshinn]

This fix should also be in Alpine 3.2 as well. We just pushed a new 3.2 tag. Reopen if still an issue.

[sudoforge]

This looks to be an issue in alpine:3.3:

/ # apk add ca-certificates
OK: 28 MiB in 31 packages
/ # update-ca-certificates 
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

[andyshinn]

What specifically is the issue?

[sudoforge]

On second inspection, I don't think ca-certificates is the root of my issue, but it may be worth patching anyhow (that's obviously at your discretion).

[andyshinn]

The warning is just a warning. It doesn't affect anything.

[dmugtasimov]

FROM alpine:3.5

RUN apk add --no-cache ca-certificates && \
    update-ca-certificates

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

The message says that something is skipped. It makes an impression that user get incomplete set of root certificates. Could you, please, rephrase, remove or fix this message, so people are not getting confused about the message?

[andyshinn]

This should be brought up on http://bugs.alpinelinux.org/projects/alpine/issues. It comes from the utility in the ca-certificates package at https://git.alpinelinux.org/cgit/aports/tree/main/ca-certificates/c_rehash.c#n210. I agree, the warning could be better or ignored altogether if the file is ca-certificates.crt.

[adolphlwq]

+1

[sudoforge]

@adolphlwq, your "+1" comment doesn't really give us any new information.

If you're experiencing the same issue, have none of the above comments helped? As @andyshinn noted two full moons ago, this issue doesn't really belong here.

[pedromanoel]

@andyshinn mentioned that this is just a warning and shouldn't affect anything. However, update-ca-certificates is exiting with code 1, and I took some time to find out why my command list was failing...

[sergeyklay]

What about this guys:

FROM alpine:3.7

RUN apk update \
    && apk upgrade \
    && apk add --no-cache \
        ca-certificates \
    && update-ca-certificates 2>/dev/null || true

[danielpalstra]

I think it's great for a workaround, but the command shouldn't exit 1 on a warning. Or we should be able to ignore warnings. I have implemented this workaround in my Dockerfile, but I don't think I can really assume that everything is fine certificate wise anymore.

[Lauriy]

This is still a problem.

[butla]

This still doesn't work (produces the warning, doesn't add stuff to /etc/ssl/certs/ca-certificates.crt) with alpine 3.9:

COPY ca-cert-XXX.pem /etc/ssl/certs/
update-ca-certificates

my workaround:

COPY ca-cert-XXX.pem /etc/ssl/certs/
RUN cat ca-cert-XXX.pem >> /etc/ssl/certs/ca-certificates.crt

@andyshinn so shouldn't the issue be reopened?

[jharshman]

Just ran into this today, not able to add a CA.

[rvsrinivas]

This is still a problem

[da3mon-01]

How I can verify that my cert has been actually imported into the ca-certs?

[butla]

@da3mon-01 it should get appended to /etc/ssl/certs/ca-certificates.crt

[ibrt]

I think there are two separate issues here, stemming from the same root cause. It looks like update-ca-certificates skips any PEM file in /usr/local/share/ca-certificates if it contains more than one certificate.

If you only get a warning about ca-certificates.crt it's harmless, but if you just added a new PEM file and you want to import it (e.g. the RDS certificate chain), you will get a warning for that file, and the certificates will not be imported. Confusing.

The workaround I found is to split the new PEM file into individual files first, one per certificate:

cd "/usr/local/share/ca-certificates" && curl "https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem" | csplit -f "rds-" - '/-----BEGIN CERTIFICATE-----/' '{*}'`

[cmcarey]

Months later, still a problem. Issue should _not_ be closed.

[MattFriedman]

WARNING: nothing is wrong

The message above should be either reworded so it makes sense or removed. It's useless and people, like me, are wasting their time trying to figure out what it means.

[Aeres-u99]

Apparently, I am also experiencing the very same issue. I dont know how to approach to solve it.

root@animeVPS:~# wget https://rclone.org/install.sh 
Connecting to rclone.org (5.153.250.7:443)
wget: SSL/TLS certificate is not being validated!

root@animeVPS:~# update-ca-certificates 
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

root@animeVPS:~# apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86/APKINDEX.tar.gz
v3.5.3-45-g195c4dea45 [http://dl-cdn.alpinelinux.org/alpine/v3.5/main]
v3.5.3-44-g08ec10f083 [http://dl-cdn.alpinelinux.org/alpine/v3.5/community]
OK: 7924 distinct packages available

root@animeVPS:~# apk add ca-certificates
OK: 468 MiB in 138 packages

root@animeVPS:~# update-ca-certificates 
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

# ---- Not a server side Issue? ----

root@animeVPS:~# wget --no-check-certificate https://rclone.org/install.sh 
Connecting to rclone.org (5.153.250.7:443)
wget: error getting response: Connection reset by peer

#--- this might be unrelated but still worth taking a look? ---

root@animeVPS:~# apk add curl-dev
ERROR: unsatisfiable constraints:
  libressl-dev-2.4.4-r0:
    conflicts: openssl-dev-1.0.2q-r0[pc:libcrypto=2.4.4] openssl-dev-1.0.2q-r0[pc:libssl=2.4.4] openssl-dev-1.0.2q-r0[pc:openssl=2.4.4]
    satisfies: libssh2-dev-1.7.0-r2[libressl-dev] libssh2-dev-1.7.0-r2[pc:libcrypto] libssh2-dev-1.7.0-r2[pc:libssl] curl-dev-7.61.1-r1[libressl-dev]
  openssl-dev-1.0.2q-r0:
    conflicts: libressl-dev-2.4.4-r0[pc:libcrypto=1.0.2q] libressl-dev-2.4.4-r0[pc:libssl=1.0.2q] libressl-dev-2.4.4-r0[pc:openssl=1.0.2q]
    satisfies: world[openssl-dev] libssh2-dev-1.7.0-r2[pc:libcrypto] libssh2-dev-1.7.0-r2[pc:libssl]

[janschloss]

Problem still exists

[Tatukis]

/ # mv DigiCertSHA2SecureServerCA.crt.pem /usr/local/share/ca-certificates/DigiCertSHA2SecureServerCA.crt / # update-ca-certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
I'm running this in the image itself and get that warning, and I don't know if my certificates are updated or not.

[Lauriy]

Just don't use Alpine & Docker. Nobody cares if you shave 100 MB off an image. 2 hours of your work is a terabyte disk.

[GabLeRoux]

Message is misleading, but it's only a warning. It doesn't mean it failed to run update-ca-certificates as according to exit code, it did succeed if it continues to next step.

In case you have access to python inside your docker image and really want to make sure the certificate was correctly added, here's an example with a verification.

ARG CA_BUNDLE_SOURCE=https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
ARG CA_BUNDLE_DESTINATION=/usr/local/share/ca-certificates/rds-combined-ca-bundle.pem

ADD $CA_BUNDLE_SOURCE $CA_BUNDLE_DESTINATION
RUN update-ca-certificates
RUN python -c "x=open('$CA_BUNDLE_DESTINATION').read(); y=open('/etc/ssl/certs/ca-certificates.crt').read(); exit(0) if x in y else exit(-1)"

_Above python trick comes from https://unix.stackexchange.com/a/114882/63222 and was adapted to fail if file doesn't contain other file. There are probably much better ways out there._

At this point, it's probably easier to append the bundle directly to /etc/ssl/certs/ca-certificates.crt like this:

ARG CA_BUNDLE_SOURCE=https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
ARG CA_BUNDLE_DESTINATION=/usr/local/share/ca-certificates/rds-combined-ca-bundle.pem

ADD $CA_BUNDLE_SOURCE $CA_BUNDLE_DESTINATION
RUN cat $CA_BUNDLE_DESTINATION >> /etc/ssl/certs/ca-certificates.crt

[bbolli]

I have looked into this a bit closer.

The reason for this warning is the c_rehash program that runs at the very end of update-ca-certificates. Its purpose is to (re)generate the hash symlinks in /etc/ssl/certs. This is only possible if a file contains a single certificate (or revocation list).

When c_rehash encountersca-certificates.crt which contains the concatenation of all certificates, it rightly complains that this file cannot be hashed. But as noted above, this is indeed just a warning and the update of the certificates did work.

Source: https://git.alpinelinux.org/ca-certificates/

I suggest to close this issue.

[nikskiz]

It may be a warning, but why does the message include skipping. This is misleading as others have mentioned.

[bbolli]

but why does the message include skipping

Because the file mentioned contains more than one certificate and thus can't be hashed. This is output from c_rehash.

[franz-josef-kaiser]

The issue still persists. I get the warning both for a cert and the store file:

Reproduce:

1. Try to install cert
   shell /etc/ssl/certs # update-ca-certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping WARNING: ca-cert-<DOMAIN>.<TLD>.pem does not contain exactly one certificate or CRL: skipping

1. Checking, if the cert was added to the store: The solution by @GabLeRoux is good, but probably an overkill. There's no need for python in your container: Alpine Linux ships with comm (a reverse diff):

   comm -12 /etc/ssl/certs/ca-cert-<DOMAIN>.<TLD>.pem  /etc/ssl/certs/ca-certificates.crt
   

   The result, that you should see if the cert was added to the ca-certificates.crt-store is:

   -----BEGIN CERTIFICATE-----
   

   …or nothing. If you see a complete cert, then it tells you that the cert wasn't successfully added.
   In my case, the cert was added, still the WARNING about the single cert is there.

2. Try to fetch from a local URl:

   /etc/ssl/certs # wget -qO- http://127.0.0.1/nginx_status 
   ssl_client: <DOMAIN>.<TLD>: certificate verification failed: unable to get local issuer certificate
   wget: error getting response: Connection reset by peer
   

3. Ignore the cert for a wget call. Does not work for other connections where you do not have access to request args:
   shell /etc/ssl/certs # wget --no-check-certificate -qO- http://127.0.0.1/nginx_status Active connections: 1 server accepts handled requests 10 10 8 Reading: 0 Writing: 1 Waiting: 0

[rustyx]

@franz-josef-kaiser that is another issue, your cert ca-cert-<DOMAIN>.<TLD>.pem is skipped from rehashing because it didn't contain exactly one cert or CRL (like the message says).

ca-certificates.crt isn't really used. Instead a certificate is looked up by its hash - a symlink like fe8a2cd8.0. These symlinks are created by the program c_rehash which is invoked as part of update-ca-certificates. And that's where the warnings come from.

The warning about the bundle is really just a warning though. It's because update-ca-certificates is writing the bundle before invoking c_rehash /etc/ssl/certs/. I've submitted a PR#1 to Alpine to address this. Just tested and seems to be working fine.

~ # update-ca-certificates
~ #

[franz-josef-kaiser]

@rustyx Thanks for clarifying that (and submitting the PR to Alpine@GitLab)

Instead a certificate is looked up by its hash - a symlink like fe8a2cd8.0.