Kaniko: Some log lines prefixed with "ERROR: logging before flag.Parse:"

[ahmetb]

Actual behavior

I see log lines from executor image like the following:

ERROR: logging before flag.Parse: E1204 18:56:47.163876       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
ERROR: logging before flag.Parse: E1204 18:56:47.166507       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url

Expected behavior

I come across this in a simple Knative Build with kaniko executor:

apiVersion: build.knative.dev/v1alpha1
kind: Build
metadata:
  name: example-build
spec:
  source:
    git:
      url: https://github.com/knative/docs.git
      revision: 1e779dca208d0b99221b9ce2d1b5491a60967a75
    subPath: serving/samples/helloworld-go/
  steps:
  - name: build-and-push
    image: gcr.io/kaniko-project/executor
    args:
    - "--dockerfile=/workspace/Dockerfile"
    - "--destination=gcr.io/ahmetb-samples-playground/from-kaniko:v1"

Here's the logs I see:

ERROR: logging before flag.Parse: E1204 18:56:47.163876       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
ERROR: logging before flag.Parse: E1204 18:56:47.166507       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
2018/12/04 18:56:47 No matching credentials were found, falling back on anonymous
INFO[0000] Executing 0 build triggers
INFO[0000] Unpacking rootfs as cmd RUN CGO_ENABLED=0 GOOS=linux go build -v -o helloworld requires it.
INFO[0032] Taking snapshot of full filesystem...
INFO[0037] Skipping paths under /builder/home, as it is a whitelisted directory
INFO[0037] Skipping paths under /dev, as it is a whitelisted directory
INFO[0037] Skipping paths under /kaniko, as it is a whitelisted directory
[...]

I see similar errors in issue #407.

Additional Information

• Dockerfile https://github.com/knative/docs/blob/1e779dca208d0b99221b9ce2d1b5491a60967a75/serving/samples/helloworld-go/Dockerfile
• Build Context https://github.com/knative/docs/blob/1e779dca208d0b99221b9ce2d1b5491a60967a75/serving/samples/helloworld-go/
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands): None, just use the link above + dockerfile.
• Kaniko Image (fully qualified with digest) https://console.cloud.google.com/gcr/images/kaniko-project/GLOBAL/executor@sha256:6e7b4d43158024b1945005d45b767722a7f139f5c9c572615986754feb07d94a/details?tab=info

[priyawadhwa]

Hey @ahmetb, it seems like these warnings are coming from a vendored dependency via glog. Does the build succeed in the end?

[ahmetb]

It succeeds. But the message looks confusing/irrelevant, so it probably should be addressed.

[ahmetb]

From very far away, looks like you probably need to initialize glog even though you’re not using it (because one if your deps do)? There are ways to get logrus/cobra and glog both working.

[jeff-knurek]

These erroneous error logs have finally started to annoy other developers and not just me, so I tracked down this issue.

I don't really see how initializing glog would change them _(unless this issue is just about the formatting - and if that's the case the WithAuthFromKeychain dependency is also wrong)_.

What I don't understand is why these calls to
http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
and http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url are even happening. I mean I see that it happens when trying to retrieve the remote cache layers, but the DefaultKeychain is empty, and we're not setting anything to use the google credential provider, but I don't see a way to disable this.

[ffledgling]

Running into this as well, except mine ends with a SIGSEGV.

ERROR: logging before flag.Parse: E0325 00:56:34.795093       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
ERROR: logging before flag.Parse: E0325 00:56:34.797453       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x10e1725]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/remote.Write(0x19c3bc0, 0xc4200b5fc0, 0x0, 0x0, 0x199bbc0, 0xc4203ffa80, 0x19985c0, 0xc4200d3380, 0x0, 0x0)
    /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/remote/write.go:33 +0x45
github.com/GoogleContainerTools/kaniko/pkg/executor.DoPush(0x0, 0x0, 0x24a2780, 0x0, 0x0)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/executor/push.go:100 +0x89b
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x24932c0, 0xc4203bab40, 0x0, 0x3)
    /go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/cmd/root.go:89 +0xb3
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).execute(0x24932c0, 0xc4200b4010, 0x3, 0x3, 0x24932c0, 0xc4200b4010)
    /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x24932c0, 0x0, 0x18447ea, 0x25)
    /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).Execute(0x24932c0, 0xc420511f78, 0xc4200a0058)
    /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/main.go:26 +0x2d

I'm not sure how or why this is happening. This happens when trying to push to public dockerhub registry.

[medyagh]

this is related to glog, it is a wide spread issue, we also face it https://github.com/kubernetes/minikube/issues/4866

[gothburz]

it seems to be related to the flag package as well, glog relies on the flag package.

[pedrofolim]

Hello people,

      We are also having the same problem pushing to a private nexus using kaniko and kubernetes.

  `ERROR: logging before flag.Parse: E1216 13:37:22.432101      39 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg

ERROR: logging before flag.Parse: E1216 13:37:22.434716 39 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url`

Tks

[Djabx]

Hello everyone,

We also have the same problem with kaniko (debug-v0.17.1) on gke (1.13.12-gke.25).
BTW We use the debug image because we launch it via gitlab-ci.

ERROR: logging before flag.Parse: E0218 15:17:19.926474      14 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
ERROR: logging before flag.Parse: E0218 15:17:19.932711      14 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

Thanks.

[fitex]

We also experience the same issue using GKE (1.15.9-gke.22), Kaniko (debug-v0.18.0) and Gitlab runner (12.4.1):

$ /kaniko/executor --context ./dockerfiles --dockerfile ./dockerfiles/Dockerfile --destination eu.gcr.io/<gke_id>/<image-name>:<image-tag>
...
ERROR: logging before flag.Parse: E0313 12:36:01.315276      20 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
ERROR: logging before flag.Parse: E0313 12:36:01.319595      20 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
...

[srinikitha09]

Is there any workaround to resolve this issue?