Serving: KNative cannot create pods for nvcr.io

Created on 14 Jun 2019 · 7Comments · Source: knative/serving

In what area(s)?

Serving

What version of Knative?

v0.6.0

Expected Behavior

Expected pods to be created for the revision

Actual Behavior

Revision fails, pods are not created. Instead the configuration reports the following error:

  - lastTransitionTime: "2019-06-14T01:31:59Z"                                                                                       
    message: "Revision \"simple-string-default-cndmb\" failed with message: Unable                                   
      to fetch image \"nvcr.io/nvidia/tensorrtserver:19.05-py3\": unsupported status                                                 
      code 500; body: <html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body
      bgcolor=\"white\">\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>nginx/1.14.2</center>\r\n</body>\r\n</
html>\r\n."                                                                                                                  
    reason: RevisionFailed

Steps to Reproduce the Problem

Apply the following knative configuration:
(the key is image: nvcr.io/nvidia/tensorrtserver:19.05-py3)

apiVersion: serving.knative.dev/v1alpha1
kind: Configuration
metadata:
  name: simple-string-default
  namespace: default
spec:
  revisionTemplate:
    metadata:
      annotations:
        autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
        autoscaling.knative.dev/target: "1"
    spec:
      container:
        args:
        - --model-store=gs://rakelkar1/trt_sample
        - --allow-poll-model-repository=false
        - --allow-grpc=true
        - --allow-http=true
        - --grpc-port=9000
        - --rest-port=8080
        command:
        - trtserver
        image: nvcr.io/nvidia/tensorrtserver:19.05-py3
        name: ""
        ports:
        - containerPort: 8080
          name: h2c
        resources:
          requests:
            cpu: "1"
            memory: 2Gi
      timeoutSeconds: 300

kinbug

Source

rakelkar

All 7 comments

500 indicates a bug with nvcr rather than with knative...

vagababov on 14 Jun 2019

Ys, looks resolution can be turned off via config: https://github.com/knative/serving/blob/master/config/config-deployment.yaml

rakelkar on 14 Jun 2019

thanks for the pointer @raushan2016 :)

rakelkar on 14 Jun 2019

To workaroud the nvcri.io issue run:

kubectl edit cm config-deployment -n knative-serving

add the line:

  registriesSkippingTagResolving: nvcr.io

rakelkar on 16 Jun 2019

/reopen

This is actually a bug on our side, reopening to track updating the ggcr dependency once we fix it.

jonjohnsonjr on 17 Jun 2019

👍1

@jonjohnsonjr: Reopened this issue.

In response to this:

/reopen

This is actually a bug on our side, reopening to track updating the ggcr dependency once we fix it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.