Kfserving: storage-initialiser fails to copy model from S3

[karlschriek]

Upon using a Secret and Service account that look like this:

apiVersion: v1
metadata:
  name: kfserving-aws-secret
  namespace: {namespace}
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.eu-east-1.amazonaws.com
    serving.kubeflow.org/s3-usehttps: "1"
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: eu-east-1
data:
  AWS_ACCESS_KEY_ID: {access_key_base64}
  AWS_SECRET_ACCESS_KEY: {secret_key_base64}
kind: Secret
type: Opaque
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: {namespace}
secrets:
  - name: kfserving-aws-secret  

...and an InferenceService like this:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: {model_name}
  namespace: {namespace}
spec:
  default:
    predictor:
      serviceAccountName: kfserving-sa
      tensorflow:
        storageUri: {model_base_path}    

...the model fails to serve, with the storage-initialiser pod throwing the following error:

[I 200226 14:09:42 initializer-entrypoint:13] Initializing, args: src_uri [s3://at-kubeflow-training/tf-job/ksc-ebce76a1] dest_path[ [/mnt/models]
[I 200226 14:09:42 storage:35] Copying contents of s3://at-kubeflow-training/tf-job/ksc-ebce76a1 to local
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 50, in download
    Storage._download_s3(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 65, in _download_s3
    client = Storage._create_minio_client()
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 217, in _create_minio_client
    secure=use_ssl)
  File "/usr/local/lib/python3.7/site-packages/minio/api.py", line 150, in __init__
    is_valid_endpoint(endpoint)
  File "/usr/local/lib/python3.7/site-packages/minio/helpers.py", line 301, in is_valid_endpoint
    if hostname[-1] == '.':
IndexError: string index out of range

This was using Kubeflow 1.0.0RC4, KFServing 0.2.2 and KNative 0.11.1

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.88 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[marcindulak]

https://www.kubeflow.org/docs/aws/aws-e2e/ shows an example of the kind: Secret which uses

data:
  # echo -ne "AKIAxxx" | base64
  awsAccessKeyID: QUtJQVhxxxVXVjQ=
  awsSecretAccessKey: QzR0UnxxxVNOd0NQQQ==

I've just used the original AWS variable naming successfully (at least in terms of storage-initializer not failing with IndexError: string index out of range) with minio.

Let me post a typical minio setup (https://github.com/kubeflow/kfserving/tree/master/docs/samples/s3) in case someone needs it

apiVersion: v1
kind: Secret
metadata:
  name: kfserving-minio-secret
  namespace: anonymous
  annotations:
    serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000
    serving.kubeflow.org/s3-usehttps: "0"
    serving.kubeflow.org/s3-verifyssl: "0"
    serving.kubeflow.org/s3-region: us-east-1
type: Opaque
stringData:
  awsAccessKeyID: minio
  awsSecretAccessKey: minio123
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfserving-sa
  namespace: anonymous
secrets:
  - name: kfserving-minio-secret

[yuzisun]

the default s3 secret key names expected by kfserving is configured here
https://github.com/kubeflow/kfserving/blob/master/config/default/configmap/inferenceservice.yaml#L89

[karlschriek]

Ok thanks for the input. Is it necessary to use Minio at all here? I'm not really that familiar with how Minio works exactly. On my cluster I have access directly to S3, and normally would use an endpoint like "s3.eu-east-1.amazonaws.com". Are you saying that I need to use Minio as a go-between?

Also, doesn't it make more sense to use by default the variable names AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as that is what is commonly used by applications as environment variables for AWS credentials?

[yuzisun]

Minio is not a requirement, the idea is to support any S3 compatible storage. The above naming convention is for environment variable, here we are using it for the secret data key name, but you are free to configure what you want.

[karlschriek]

Yes I know that it is for a Secret, but every other application I've seen
that uses Secrets to pass AWS credentials have used the standard naming
convention. I realise I can change it myself, I just find it strange that
you use a different convention here.

On Thu, 27 Feb 2020 at 11:33, Dan Sun [e-mail] wrote:

Minio is not a requirement, the idea is to support any S3 compatible
storage. The above naming convention is for environment variable, here we
are using it for the secret data key name, but you are free to configure
what you want.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubeflow/kfserving/issues/696?email_source=notifications&email_token=AGBE4ODQ3BR5HGQJTFDJYC3RE6JHJA5CNFSM4K4GNHZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND2P7Q#issuecomment-591898622,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGBE4OGHOYNBYBT6MFISHW3RE6JHJANCNFSM4K4GNHZQ
.

[yuzisun]

I think the name is copied from original kubeflow s3 example since we want to keep consistent, I just checked it is changed there too, so feel free to raise a PR to fix the default then. Thanks for brought this up.

[Jude188]

Try:
serving.kubeflow.org/s3-endpoint: http://s3.eu-east-1.amazonaws.com as your annotation. I had a similar issue.

The URL just needed a scheme in my case.

[theofpa]

If that helps, this configuration works smoothly for me:

```apiVersion: v1
kind: Secret
metadata:
name: mysecret
annotations:
serving.kubeflow.org/s3-endpoint: s3.eu-west-1.amazonaws.com
serving.kubeflow.org/s3-usehttps: "1"
serving.kubeflow.org/s3-verifyssl: "1"
serving.kubeflow.org/s3-region: eu-west-1
type: Opaque
data:
awsAccessKeyID: xxx=

awsSecretAccessKey: xxx==

apiVersion: v1
kind: ServiceAccount
metadata:
name: sa
secrets:

• name: mysecret

[CreatorSky]

@karlschriek what did you end up using? I am facing similar issue, tried different variations of configuration but still getting the same error.