Skaffold: Kaniko + Container Structure Tests doesn't work

[kevinleturc]

Expected behavior

container-structure-tests works as expected when image is built with Kaniko

Actual behavior

container-structure-tests failed with the following log:

=====================================
====== Test file: non_root.yml ======
=====================================
=== RUN: root exist and has correct permission
--- FAIL
Error: error setting env vars: Error creating container: no such image

Information

• Skaffold version: 1.5
• Operating system: Unix / Jenkins X
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta11
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: "{{.DOCKER_REGISTRY}}/{{.ORG}}/{{.IMAGE_NAME}}:{{.VERSION}}"
  artifacts:
    - image: my-example
      context: docker/my-example
      kaniko:
        dockerfile: Dockerfile
        buildContext:
          localDir: {}
        buildArgs:
          VERSION: "{{.VERSION}}"
test:
  - image: my-example
    structureTests:
      - './docker/my-example/test/*'

Some clue: as Kaniko builds the Docker image without putting it in the local Docker daemon, and thus container-structure-tests don't find it.

Passing --pull to container-structure-tests command could do the work, but it's not working in my case due to this error:

FATA[0000] error pulling remote image docker.packages.nuxeo.com/nuxeo/nuxeo:11.x@sha256:924dde3e2eca1f41a78061d25dba6b11960dc30cc0b3694679926919788908a1: API error (500): Get https://docker.packages.nuxeo.com/v2/nuxeo/nuxeo/manifests/sha256:924dde3e2eca1f41a78061d25dba6b11960dc30cc0b3694679926919788908a1: no basic auth credentials

Which sounds like container-structure-tests doesn't handle authenticated docker registry.

Steps to reproduce the behavior

The issue can be reproduced with any Dockerfile as the issue seems to be the non-existence of image in the Docker daemon.

[tejal29]

@kevinleturc Can you please provide us which version of container-structure-tests?
This issue might belong to https://github.com/GoogleContainerTools/kaniko.

[kevinleturc]

The _container-structure-tests_ version we use is v1.8.0.

I don't know where the issue is but I wouldn't say _kaniko_ was the culprit because, in my understanding, _container-structure-tests_ seems to expect the Docker image to be in the docker daemon host if --pull option is missing. It seems that _skaffold_ doesn't give this option to _container-structure-tests_ and as kaniko doesn't use the Docker daemon to build the image, thus we're getting this error.

Why do you think it is a kaniko issue?

[dgageot]

This is related to https://github.com/GoogleContainerTools/skaffold/issues/1082

[dgageot]

Why I'm going to do is have container-structure-tests pull the images when they are built remotely (with Kaniko or GCB, for eg.) so that the tests don't fail. Ideally, we'd run the tests remotely, where the images are built, but that's a whole different problem!

[dgageot]

Ok, so it turns out that container-structure-tests doesn't really know how to pull images... It can only work on public images