Kfserving: not able start kfserving with s3

[tiru1930]

/kind bug

Running kfserving with kubeflow installed on minkube for s3 tensorflow storage model but awaiting at PodInitializing

kubectl get inferenceservice -n kfserving-test-2
NAME             URL                                                                           READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
effidet-s3                                                                                     False                                      83m
effidet-s3-1                                                                                   False                                      7m30s
flowers-sample   http://flowers-sample.kfserving-test-2.example.com/v1/models/flowers-sample   True    100                                109m
sklearn-iris     http://sklearn-iris.kfserving-test-2.example.com/v1/models/sklearn-iris       True    100                                17h

deployment.apps/effidet-s3-1-predictor-default-khtjq-deployment     0/0     0            0           6m45s
deployment.apps/effidet-s3-1-predictor-default-nc6kn-deployment     0/0     0            0           6m45s
replicaset.apps/effidet-s3-1-predictor-default-khtjq-deployment-55fc98844f    0         0         0       6m45s
replicaset.apps/effidet-s3-1-predictor-default-nc6kn-deployment-6dcfb7bf56    0         0         0       6m45s
revision.serving.knative.dev/effidet-s3-1-predictor-default-khtjq     effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-khtjq     1            False   PodInitializing
revision.serving.knative.dev/effidet-s3-1-predictor-default-nc6kn     effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-nc6kn     2            False   PodInitializing
configuration.serving.knative.dev/effidet-s3-1-predictor-default     effidet-s3-1-predictor-default-nc6kn                                              False   RevisionFailed
service.serving.knative.dev/effidet-s3-1-predictor-default     http://effidet-s3-1-predictor-default.kfserving-test-2.example.com     effidet-s3-1-predictor-default-nc6kn                                              False   RevisionMissing
route.serving.knative.dev/effidet-s3-1-predictor-default     http://effidet-s3-1-predictor-default.kfserving-test-2.example.com     False   RevisionMissing



md5-0283cdd3a524f839a68b7c38456186ba



      s3://bucket_name/savd_model
      /mnt/models
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 10 Jul 2020 08:04:58 +0000
      Finished:     Fri, 10 Jul 2020 08:05:00 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 10 Jul 2020 08:04:41 +0000
      Finished:     Fri, 10 Jul 2020 08:04:43 +0000
    Ready:          False
    Restart Count:  2



md5-099128e435afbc6d1538377fdc403097



apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "effidet-s3-1"
spec:
  default:
    predictor:
      serviceAccountName: sa
      tensorflow:
        storageUri: "s3://bucket/saved_model"
---
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.amazonaws.com.
    serving.kubeflow.org/s3-usehttps: "1" 
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: us-east-1
type: Opaque
stringData:
  awsAccessKeyID: xxxx=
  awsSecretAccessKey: xxx==
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
  - name: mysecret

Environment:

• Kubeflow version: v1.0.2
• Minikube version: v1.8.0
• Kubernetes version: v1.17.3
• OS (e.g. from /etc/os-release):

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.95 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.95 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

@tiru1930 Can you get the full pod yaml output and see if the S3 environment variables are injected? You can also checkout KFServing debugging guide to see why it is failing.

[tiru1930]

@yuzisun

kind: Pod
apiVersion: v1
metadata:
  name: effidet-s3-1-predictor-default-jpkhx-deployment-5f55cd679-rzf8d
  generateName: effidet-s3-1-predictor-default-jpkhx-deployment-5f55cd679-
  namespace: kfserving-test
  selfLink: >-
    /api/v1/namespaces/kfserving-test/pods/effidet-s3-1-predictor-default-jpkhx-deployment-5f55cd679-rzf8d
  uid: fa5cd736-6ac9-4618-90e0-c2ad0d590dd5
  resourceVersion: '5479250'
  creationTimestamp: '2020-07-10T09:37:52Z'
  labels:
    app: effidet-s3-1-predictor-default-jpkhx
    pod-template-hash: 5f55cd679
    serving.knative.dev/configuration: effidet-s3-1-predictor-default
    serving.knative.dev/configurationGeneration: '2'
    serving.knative.dev/revision: effidet-s3-1-predictor-default-jpkhx
    serving.knative.dev/revisionUID: 2b85c221-9ced-4403-974a-c7cde5de3fd4
    serving.knative.dev/service: effidet-s3-1-predictor-default
    serving.kubeflow.org/inferenceservice: effidet-s3-1
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/target: '1'
    internal.serving.kubeflow.org/storage-initializer-sourceuri: 's3://bucketname/eff_savd_model'
    queue.sidecar.serving.knative.dev/resourcePercentage: '0.2'
    serving.knative.dev/creator: 'system:serviceaccount:kubeflow:default'
    sidecar.istio.io/status: >-
      {"version":"5f3ae3613c7945ef767cb9fd594596bc001ff3ab915f12e4379c0cb5648d2729","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"],"imagePullSecrets":null}
    traffic.sidecar.istio.io/includeOutboundIPRanges: '*'
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: effidet-s3-1-predictor-default-jpkhx-deployment-5f55cd679
      uid: 72034545-9319-452e-8235-5782bc3a2a9b
      controller: true
      blockOwnerDeletion: true
spec:
  volumes:
    - name: knative-var-log
      emptyDir: {}
    - name: sa-token-49ddf
      secret:
        secretName: sa-token-49ddf
        defaultMode: 420
    - name: kfserving-provision-location
      emptyDir: {}
    - name: istio-envoy
      emptyDir:
        medium: Memory
    - name: istio-certs
      secret:
        secretName: istio.sa
        defaultMode: 420
        optional: true
  initContainers:
    - name: storage-initializer
      image: 'gcr.io/kfserving/storage-initializer:0.2.2'
      args:
        - 's3://bucketname/eff_savd_model'
        - /mnt/models
      env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: mysecret
              key: awsAccessKeyID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: mysecret
              key: awsSecretAccessKey
        - name: S3_USE_HTTPS
          value: '1'
        - name: S3_ENDPOINT
          value: s3.amazonaws.com.
        - name: AWS_ENDPOINT_URL
          value: 'https://s3.amazonaws.com.'
        - name: AWS_REGION
          value: us-east-1
        - name: S3_VERIFY_SSL
          value: '1'
      resources:
        limits:
          cpu: '1'
          memory: 1Gi
        requests:
          cpu: 100m
          memory: 100Mi
      volumeMounts:
        - name: kfserving-provision-location
          mountPath: /mnt/models
        - name: sa-token-49ddf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
    - name: istio-init
      image: 'docker.io/istio/proxy_init:1.1.6'
      args:
        - '-p'
        - '15001'
        - '-u'
        - '1337'
        - '-m'
        - REDIRECT
        - '-i'
        - '*'
        - '-x'
        - ''
        - '-b'
        - '8080,8022,9090,9091,8012'
        - '-d'
        - '15020'
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
      volumeMounts:
        - name: sa-token-49ddf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          add:
            - NET_ADMIN
        runAsUser: 0
        runAsNonRoot: false
  containers:
    - name: kfserving-container
      image: >-
        index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
      command:
        - /usr/bin/tensorflow_model_server
      args:
        - '--port=9000'
        - '--rest_api_port=8080'
        - '--model_name=effidet-s3-1'
        - '--model_base_path=/mnt/models'
      ports:
        - name: user-port
          containerPort: 8080
          protocol: TCP
      env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: mysecret
              key: awsAccessKeyID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: mysecret
              key: awsSecretAccessKey
        - name: S3_USE_HTTPS
          value: '1'
        - name: S3_ENDPOINT
          value: s3.amazonaws.com.
        - name: AWS_ENDPOINT_URL
          value: 'https://s3.amazonaws.com.'
        - name: AWS_REGION
          value: us-east-1
        - name: S3_VERIFY_SSL
          value: '1'
        - name: PORT
          value: '8080'
        - name: K_REVISION
          value: effidet-s3-1-predictor-default-jpkhx
        - name: K_CONFIGURATION
          value: effidet-s3-1-predictor-default
        - name: K_SERVICE
          value: effidet-s3-1-predictor-default
      resources:
        limits:
          cpu: '1'
          memory: 2Gi
        requests:
          cpu: '1'
          memory: 2Gi
      volumeMounts:
        - name: knative-var-log
          mountPath: /var/log
        - name: sa-token-49ddf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: kfserving-provision-location
          readOnly: true
          mountPath: /mnt/models
      lifecycle:
        preStop:
          httpGet:
            path: /wait-for-drain
            port: 8022
            scheme: HTTP
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      imagePullPolicy: IfNotPresent
    - name: queue-proxy
      image: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb
      ports:
        - name: http-queueadm
          containerPort: 8022
          protocol: TCP
        - name: queue-metrics
          containerPort: 9090
          protocol: TCP
        - name: http-usermetric
          containerPort: 9091
          protocol: TCP
        - name: queue-port
          containerPort: 8012
          protocol: TCP
      env:
        - name: SERVING_NAMESPACE
          value: kfserving-test
        - name: SERVING_SERVICE
          value: effidet-s3-1-predictor-default
        - name: SERVING_CONFIGURATION
          value: effidet-s3-1-predictor-default
        - name: SERVING_REVISION
          value: effidet-s3-1-predictor-default-jpkhx
        - name: QUEUE_SERVING_PORT
          value: '8012'
        - name: CONTAINER_CONCURRENCY
          value: '0'
        - name: REVISION_TIMEOUT_SECONDS
          value: '60'
        - name: SERVING_POD
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: SERVING_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: SERVING_LOGGING_CONFIG
          value: |-
            {
              "level": "info",
              "development": false,
              "outputPaths": ["stdout"],
              "errorOutputPaths": ["stderr"],
              "encoding": "json",
              "encoderConfig": {
                "timeKey": "ts",
                "levelKey": "level",
                "nameKey": "logger",
                "callerKey": "caller",
                "messageKey": "msg",
                "stacktraceKey": "stacktrace",
                "lineEnding": "",
                "levelEncoder": "",
                "timeEncoder": "iso8601",
                "durationEncoder": "",
                "callerEncoder": ""
              }
            }
        - name: SERVING_LOGGING_LEVEL
        - name: SERVING_REQUEST_LOG_TEMPLATE
        - name: SERVING_REQUEST_METRICS_BACKEND
          value: prometheus
        - name: TRACING_CONFIG_BACKEND
          value: none
        - name: TRACING_CONFIG_ZIPKIN_ENDPOINT
        - name: TRACING_CONFIG_STACKDRIVER_PROJECT_ID
        - name: TRACING_CONFIG_DEBUG
          value: 'false'
        - name: TRACING_CONFIG_SAMPLE_RATE
          value: '0.100000'
        - name: USER_PORT
          value: '8080'
        - name: SYSTEM_NAMESPACE
          value: knative-serving
        - name: METRICS_DOMAIN
          value: knative.dev/internal/serving
        - name: USER_CONTAINER_NAME
          value: kfserving-container
        - name: ENABLE_VAR_LOG_COLLECTION
          value: 'false'
        - name: VAR_LOG_VOLUME_NAME
          value: knative-var-log
        - name: INTERNAL_VOLUME_PATH
          value: /var/knative-internal
        - name: SERVING_READINESS_PROBE
          value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
        - name: ENABLE_PROFILING
          value: 'false'
        - name: SERVING_ENABLE_PROBE_REQUEST_LOG
          value: 'false'
      resources:
        limits:
          cpu: 40m
          memory: 200Mi
        requests:
          cpu: 25m
          memory: 50Mi
      volumeMounts:
        - name: sa-token-49ddf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        exec:
          command:
            - /ko-app/queue
            - '-probe-period'
            - '0'
        timeoutSeconds: 10
        periodSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        allowPrivilegeEscalation: false
    - name: istio-proxy
      image: 'docker.io/istio/proxyv2:1.1.6'
      args:
        - proxy
        - sidecar
        - '--domain'
        - $(POD_NAMESPACE).svc.cluster.local
        - '--configPath'
        - /etc/istio/proxy
        - '--binaryPath'
        - /usr/local/bin/envoy
        - '--serviceCluster'
        - effidet-s3-1-predictor-default-jpkhx.$(POD_NAMESPACE)
        - '--drainDuration'
        - 45s
        - '--parentShutdownDuration'
        - 1m0s
        - '--discoveryAddress'
        - 'istio-pilot.istio-system:15010'
        - '--zipkinAddress'
        - 'zipkin.istio-system:9411'
        - '--connectTimeout'
        - 10s
        - '--proxyAdminPort'
        - '15000'
        - '--concurrency'
        - '2'
        - '--controlPlaneAuthPolicy'
        - NONE
        - '--statusPort'
        - '15020'
        - '--applicationPorts'
        - '8080,8022,9090,9091,8012'
      ports:
        - name: http-envoy-prom
          containerPort: 15090
          protocol: TCP
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: ISTIO_META_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ISTIO_META_CONFIG_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ISTIO_META_INTERCEPTION_MODE
          value: REDIRECT
        - name: ISTIO_METAJSON_ANNOTATIONS
          value: >
            {"autoscaling.knative.dev/class":"kpa.autoscaling.knative.dev","autoscaling.knative.dev/target":"1","internal.serving.kubeflow.org/storage-initializer-sourceuri":"s3://bucketname/eff_savd_model","queue.sidecar.serving.knative.dev/resourcePercentage":"0.2","serving.knative.dev/creator":"system:serviceaccount:kubeflow:default","traffic.sidecar.istio.io/includeOutboundIPRanges":"*"}
        - name: ISTIO_METAJSON_LABELS
          value: >
            {"app":"effidet-s3-1-predictor-default-jpkhx","pod-template-hash":"5f55cd679","serving.knative.dev/configuration":"effidet-s3-1-predictor-default","serving.knative.dev/configurationGeneration":"2","serving.knative.dev/revision":"effidet-s3-1-predictor-default-jpkhx","serving.knative.dev/revisionUID":"2b85c221-9ced-4403-974a-c7cde5de3fd4","serving.knative.dev/service":"effidet-s3-1-predictor-default","serving.kubeflow.org/inferenceservice":"effidet-s3-1"}
      resources:
        limits:
          cpu: '2'
          memory: 128Mi
        requests:
          cpu: 10m
          memory: 40Mi
      volumeMounts:
        - name: istio-envoy
          mountPath: /etc/istio/proxy
        - name: istio-certs
          readOnly: true
          mountPath: /etc/certs/
        - name: sa-token-49ddf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        httpGet:
          path: /healthz/ready
          port: 15020
          scheme: HTTP
        initialDelaySeconds: 1
        timeoutSeconds: 1
        periodSeconds: 2
        successThreshold: 1
        failureThreshold: 30
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        runAsUser: 1337
        readOnlyRootFilesystem: true
  restartPolicy: Always
  terminationGracePeriodSeconds: 60
  dnsPolicy: ClusterFirst
  serviceAccountName: sa
  serviceAccount: sa
  nodeName: ip-172-31-23-49
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
status:
  phase: Pending
  conditions:
    - type: Initialized
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2020-07-10T09:37:52Z'
      reason: ContainersNotInitialized
      message: 'containers with incomplete status: [storage-initializer istio-init]'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2020-07-10T09:37:52Z'
      reason: ContainersNotReady
      message: >-
        containers with unready status: [kfserving-container queue-proxy
        istio-proxy]
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2020-07-10T09:37:52Z'
      reason: ContainersNotReady
      message: >-
        containers with unready status: [kfserving-container queue-proxy
        istio-proxy]
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-07-10T09:37:52Z'
  hostIP: 172.31.23.49
  podIP: 172.17.0.49
  podIPs:
    - ip: 172.17.0.49
  startTime: '2020-07-10T09:37:52Z'
  initContainerStatuses:
    - name: storage-initializer
      state:
        waiting:
          reason: CrashLoopBackOff
          message: >-
            back-off 5m0s restarting failed container=storage-initializer
            pod=effidet-s3-1-predictor-default-jpkhx-deployment-5f55cd679-rzf8d_kfserving-test(fa5cd736-6ac9-4618-90e0-c2ad0d590dd5)
      lastState:
        terminated:
          exitCode: 1
          reason: Error
          startedAt: '2020-07-10T09:43:44Z'
          finishedAt: '2020-07-10T09:43:46Z'
          containerID: >-
            docker://a5e3978fd7e9fa1eea8c52baa03f063d664baf3ec8575f6498de81109125aab0
      ready: false
      restartCount: 6
      image: 'gcr.io/kfserving/storage-initializer:0.2.2'
      imageID: >-
        docker-pullable://gcr.io/kfserving/storage-initializer@sha256:7a7d3cf4c5121a3e6bad0acc9e88bbdfa9c7f774d80bd64d8e35a84dcfef8890
      containerID: >-
        docker://a5e3978fd7e9fa1eea8c52baa03f063d664baf3ec8575f6498de81109125aab0
    - name: istio-init
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: 'docker.io/istio/proxy_init:1.1.6'
      imageID: ''
  containerStatuses:
    - name: istio-proxy
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: 'docker.io/istio/proxyv2:1.1.6'
      imageID: ''
      started: false
    - name: kfserving-container
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: >-
        index.docker.io/tensorflow/serving@sha256:f7e59a29cbc17a6b507751cddde37bccad4407c05ebf2c13b8e6ccb7d2e9affb
      imageID: ''
      started: false
    - name: queue-proxy
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb
      imageID: ''
      started: false
  qosClass: Burstable

kubectl get revision $(kubectl get configuration effidet-s3-1-predictor-default --output jsonpath="{.status.latestCreatedRevisionName}" -n kfserving-test)  -n kfserving-test
NAME                                   CONFIG NAME                      K8S SERVICE NAME                       GENERATION   READY   REASON
effidet-s3-1-predictor-default-jpkhx   effidet-s3-1-predictor-default   effidet-s3-1-predictor-default-jpkhx   2            False   PodInitializing

since its failing at podinitializing am not able to see any container logs

[yuzisun]

@tiru1930 I think there is an extra dot in the s3 endpoint. Also if you check the pod log right away after deploying you should be able to catch the log.

        - name: AWS_ENDPOINT_URL
          value: 'https://s3.amazonaws.com.'

[tiru1930]

@yuzisun

I have removed extra dot

[I 200710 10:22:55 initializer-entrypoint:13] Initializing, args: src_uri [s3://detect-tenant-models/eff_savd_model] dest_path[ [/mnt/models]
[I 200710 10:22:55 storage:35] Copying contents of s3://detect-tenant-models/eff_savd_model to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
    Storage._download_s3(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 71, in _download_s3
    for obj in objects:
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 943, in list_objects
    headers=headers)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 1904, in _url_open
    region = self._get_bucket_region(bucket_name)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 1833, in _get_bucket_region
    region = self._get_bucket_location(bucket_name)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 1876, in _get_bucket_location
    raise ResponseError(response, method, bucket_name).get_exception()
minio.error.ResponseError: ResponseError: code: AuthorizationHeaderMalformed, message: The authorization header is malformed; the authorization component "Credential=QUtJQVdOTEpUWTRORlpYVFpJVABVVM=/20200710/us-east-1/s3/aws4_request" is malformed., bucket_name: detect-tenant-models, object_name: None, request_id: 81831C2DD3460409, host_id: MovuDFPYdm7OGF6PISRyhVVq78srrZzLLcIj/Sm54FxygKzxkzD7UXzem4O6UTv8zojnGjj9p3abc3U=, region: 

[yuzisun]

@harshavardhana Do you have any idea about the error above?

[tiru1930]

@yuzisun @harshavardhana

Just an extra info in my env minio is not running

minio-6b88d6499f-wjnmc                                         1/2     CreateContainerConfigError   0          22s

[yuzisun]

@yuzisun @harshavardhana

Just an extra info in my env minio is not running

minio-6b88d6499f-wjnmc                                         1/2     CreateContainerConfigError   0          22s

@tiru1930 we are only using minio as client to access S3 compatible services, so this should not be a problem.

[tiru1930]

@yuzisun looks like i found the issues, I have replicated the same with minio client. its because authentication keys

ubuntu@ip-172-31-23-49:~$ kubectl get inferenceservice -n kfserving-test
NAME             URL                                                                         READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
effidet-s3-1                                                                                 False                                      3h4m
effidet-s3-3                                                                                 False                                      138m
effidet-s3-4                                                                                 False                                      135m
effidet-s3-5                                                                                 False                                      133m
effidet-s3-6                                                                                 False                                      21m
effidet-s3-7     http://effidet-s3-7.kfserving-test.example.com/v1/models/effidet-s3-7       True    100                                12m
flowers-sample   http://flowers-sample.kfserving-test.example.com/v1/models/flowers-sample   True    100                                3h5m

Now the service is ready effidet-s3-7

[tiru1930]

issue with keys

[aishwary09]

" I have replicated the same with minio client. its authentication keys" can anyone suggest what this means, in final comment when @tiru1930 found the solution