Rules_docker: Pushes to gitlab fail with SSL error.

[GauntletWizard]

Bazel version

Build label: 0.24.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar

rules_docker version

HEAD

Issue:

When attempting to run a docker push through a container_push rule to a registry.gitlab.com target, I am recieving the following error:
registry.gitlab.com/gauntletwizard/bazel-go:{BUILD_EMBED_LABEL} was resolved to registry.gitlab.com/gauntletwizard/bazel-go:foo
F0430 20:27:05.873595 4586 fast_pusher_.py:194] Error publishing registry.gitlab.com/gauntletwizard/bazel-go:foo: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
This happens in both CI and local images (Running under Gitlab CI and a local build on OSX)

This is new as of <24hrs ago; Previously, builds worked fine. Other users are seeing the same issue, though their timing was different. I've had a successful push this morning, well after that post was made.

Pushes and pulls using the docker tool are working correctly. Browsing to https://registry.gitlab.com on my local machine shows a valid SSL certificate.

Example target:

https://gitlab.com/gauntletwizard/bazel-go/blob/master/BUILD#L47

[nlopezgi]

Hi @GauntletWizard ,
Thanks for opening this issue.
So is the issue still persisting? We have not made any changes to rules_docker in the last days that I can think of, which would match with the issues you are seeing. Does the issue also occur if you pin to the latest release from a 2 weeks ago? Does the issue only occur with Bazel 0.24.1?
Thanks for the additional info.

[s-garg]

I am running into the same issue. And it is still persistent. I dug around a bit and looks like google containerregistry library started failing with the same error - even though there has been no change in any dependencies anywhere:
https://travis-ci.org/google/containerregistry/jobs/526607399
0430 20:59:08.509730 1 __main__.py:147] Error pulling and saving image registry.gitlab.com/mattmoor/test-project/image:latest: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

[nlopezgi]

Is this only occurring with registry.gitlab.com? Are you confirming did none of your dependencies (bazel, pins to rules_docker, targets) changed? If so, then the issue might be related to some change on the registry.gitlab.com side (e.g., change in how they accept certificates)? Could you confirm if my assumptions are correct, if so I'll try to dig up what might have occurred on registry.gitlab.com that is impacting how containerregistry is doing auth.

[s-garg]

My push failure was specific to gitlab. This seems to be the case for @GauntletWizard & @mattmoor (in the travis CI log) as well.

No change in dependencies for me.
Instead, you can see the error occur in travis CI tests of a library that rules_docker depends on:

• https://github.com/google/containerregistry
• https://travis-ci.org/google/containerregistry/jobs/526607399

Docker push & pull to gitlab seems to work fine. So it seems specific to https://github.com/google/containerregistry

If you can confirm that the relevant rules_docker code is using https://github.com/google/containerregistry, it might make sense to move the bug report to that project. If not anything else, it would be useful to have a way of turning off SSL cert verification.

[GauntletWizard]

I agree - it seems likely to be an issue in containerregistry. It's not new to this release; My build was using the rules_docker release from December and had the same issue. I'll open this issue with the upstream now that I've figured out how to invoke pusher directly.

[s-garg]

Is this only occurring with registry.gitlab.com? Are you confirming did none of your dependencies (bazel, pins to rules_docker, targets) changed? If so, then the issue might be related to some change on the registry.gitlab.com side (e.g., change in how they accept certificates)? Could you confirm if my assumptions are correct, if so I'll try to dig up what might have occurred on registry.gitlab.com that is impacting how containerregistry is doing auth.

Apologies... I missed out parts of your comment.
If you dig up what might have occurred with containerregistry + registry.gitlab.com, that would be awesome! Thanks!!

[nlopezgi]

thanks everyone for additional info. Just to confirm. Yes rules_docker heavily depends on https://github.com/google/containerregistry. Created https://github.com/google/containerregistry/issues/150 . Will also try to look later at any changes in registry.gitlab.com

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"