Skaffold: Trouble with pulling image vs manual pull

[sfxworks]

I am trying to build my image locally and pull it with crio on the cluster so I can dev.

Expected behavior

The image pulls and pod starts so I can start dev.

Actual behavior

Image fails to pull

Information

When trying to troubeshoot this,

kubectl run --image docker.io/quantomworks/git-to-k8s:2019-05-08_01-30-59.487_EDT@sha256:8119b054b57e8624644ba3f1db71913990c75d4965bff6866e4a3c2f39f45b9e test --port 8080

vs

kubectl run --image docker.io/quantomworks/git-to-k8s:2019-05-08_01-30-59.487_EDT nosha --port 8080

The one without the sha works. I think this may have something to do with crio as I have tested it with a docker based cluster before, but unsure.

• Skaffold version: v0.28.0
• Operating system: Windows 10
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta9
kind: Config
deploy:
  kubectl:
    manifests:
    - manifest\skaffold\pod.yml
    - manifest\skaffold\service.yml
build:
  tagPolicy:
    dateTime:
      format: "2006-01-02_15-04-05.999_MST"
      timezone: "Local"
  artifacts:
  - image: docker.io/quantomworks/git-to-k8s
    context: operator
    sync:
      'src/***/*.js': src/

Steps to reproduce the behavior

1. Pull my branch https://github.com/sfxworks/git-to-k8s/tree/skaffold
2. Configure & Build
3. skaffold dev

[sfxworks]

No, upon further investigation, that theory is wrong.
@sha isn't the cause. For some reason, I still cannot pull okay though. Maybe something to do with dirty images and dockerhub?

[priyawadhwa]

Hey @sfxworks, just to clarify, the issue is that if you build an image yourself the cluster is able to pull it, but when skaffold builds the image the cluster is not able to pull it?

[sfxworks]

Hi @priyawadhwa! Almost yeah. When I use skaffold to build it with the config set to call my local docker engine on my PC, it can build and push it just fine. Where the issue splits is after that. Skaffold cannot automatically spin up a pod to build the image using whatever it injects to my manifest, but deploying the pod.yml manually with kubectl after that build process it can pull the image just fine.

[sfxworks]

Going into the node itself, crio does seem to not like the SHA tag when I try to pull it with it vs without. Even when using grc.io.

root@a2:~# crictl pull gcr.io/allthingssfx/hook-to-k8s:v0.28.0-116-ga1172365-dirty@sha256:9154d56ca7364f5f3a005c469ab0e53ee005641beb70fcbbf6d014af1f19858b
FATA[0000] pulling image failed: rpc error: code = Unknown desc = Invalid image name "gcr.io/allthingssfx/hook-to-k8s:v0.28.0-116-ga1172365-dirty@sha256:9154d56ca7364f5f3a005c469ab0e53ee005641beb70fcbbf6d014af1f19858b", unknown transport "gcr.io/allthingssfx/hook-to-k8s"
root@a2:~# crictl pull gcr.io/allthingssfx/hook-to-k8s:v0.28.0-116-ga1172365-dirty
Image is up to date for gcr.io/allthingssfx/hook-to-k8s@sha256:9154d56ca7364f5f3a005c469ab0e53ee005641beb70fcbbf6d014af1f19858b

Following up with this, going into edit the pod after skaffold deploys it and removing the @sha suffix of the image property allows me to continue developing with skaffold.

[priyawadhwa]

@sfxworks I see, so crio doesn't support pulling an image specified by both tag and digest.

I would recommend opening up an issue in their repo and seeing if they would be willing to add this feature. It's supported by other runtimes (like docker) and I think it would be useful for other developers as well. If not, we can consider adding an option to skaffold to only specify by tag and not digest.

[dgageot]

I'm going to close this one since it's crio related

[sfxworks]

@dgageot I don't see crio allowing SHA along with tags. It would be cool if there could be an exception coded in when it detects crio as container runtine.

[maxres-fr]

@priyawadhwa and @dgageot I'd like to have this issue re-opened (I think). I'm working with OpenShift at the moment which uses cri-o. I did a little digging around and the OCI spec does state that the reference may be a "tag" (annotation) or a digest. Maybe somewhere else in the OCI spec there's something that would cover this behavior. I believe that the there's another issue opened w.r.t to k3s since it uses containerd and it doesn't work as well.

relevant bit on pulls from the spec:

GET /v2//manifests/
The name and reference parameter identify the image and are REQUIRED. The reference MAY include a tag or digest.

[chanseokoh]

This is closed as a dup of #3220. Please follow #3220.