Project: Continuous Archiving to S3 hardcoded to us-east-1 region

[soosap]

Hi, I have tried using continuously archiving my postgres instance wal-g files to s3. Unfortunately, it seems only buckets in the us-east-1 region are accepted. I am using kubedb v0.12.0. I upgraded from v0.9.0. When I was doing continuous archiving to s3 then it did work. The limitation must have been sneaked in after that. Here is the error message from the postgres pod:

ERROR: 2019/06/22 11:06:00.871972 failed to upload 'kubedb/default/registry-postgres/archive/basebackups_005/base_000000010000000000000005/tar_partitions/part_001.tar.lz4' to bucket 'xxx-backup': BucketRegionError: incorrect region, the bucket is not in 'us-east-1' region

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.60. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[iamrz1]

Hello,
Can you add a bit more details so that the issue can be reproduced? Especially your bucket configuration in the CRD yaml file can be of help. Ae you using the enpoint field?

[iamliamc]

I am having the same issue!

Using S3 clone 'spaces' by digital oceans:
https://developers.digitalocean.com/documentation/spaces/#bucket-ops

RUN

kubectl create secret -n kubedb generic db-backup-do-space \
--from-file=./AWS_ACCESS_KEY_ID \
--from-file=./AWS_SECRET_ACCESS_KEY

I am able to connect with a client fine, but when I add this to my postgres instance in kubedb I get:

→ python test_api.py
Spaces List: ['my-do-bucket']

backupSchedule:
  cronExpression: "@every 24h"
  storageSecretName: db-backup-do-space
  s3:
    bucket: my-do-bucket
    endpoint: 'ams3.digitaloceanspaces.com'

The region of my space is ams3, but it keeps putting the us-east-1 into the header.

  Phase:            Failed
  Reason:           Container, getting the bucket location: RequestError: send request failed
caused by: 
Get https://ams3.digitaloceanspaces.com/my-do-bucket?location=: net/http: invalid header field value 

"AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXX\n/20190627/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXXXXXXXXXX" for key Authorization

So it fails to find my bucket.

[iamrz1]

@iamliamc We are planning to add another field region to allow users to set appropriate regions in order to support more s3 compatible storages.
But currently, we only support S3, Azure, GCS, Swift, and minIO.

[soosap]

Hi there, yes I am using the endpoint field which I have set to s3.amazonaws.com. Also I want to note that this used to work in the past. I was able to upload wal-g files to s3 eu-central-1 region when using kubedb v0.9.0. Have you identified the problem?

Name:         registry-postgres
Namespace:    default
Labels:       app=postgres
              chart=postgres-0.1.0
              heritage=Tiller
              release=registry
Annotations:  <none>
API Version:  kubedb.com/v1alpha1
Kind:         Postgres
Metadata:
  Cluster Name:        
  Creation Timestamp:  2019-06-22T13:19:37Z
  Finalizers:
    kubedb.com
  Generation:        1
  Resource Version:  115834373
  Self Link:         /apis/kubedb.com/v1alpha1/namespaces/default/postgreses/registry-postgres
  UID:               625969c0-94f0-11e9-9133-0635d3b16004
Spec:
  Archiver:
    Storage:
      S 3:
        Bucket:             xxx-backup
        Endpoint:           s3.amazonaws.com
      Storage Secret Name:  xxx-backup-s3-secret
  Database Secret:
    Secret Name:  registry-postgres-auth
  Leader Election:
    Lease Duration Seconds:  15
    Renew Deadline Seconds:  10
    Retry Period Seconds:    2
  Pod Template:
    Controller:
    Metadata:
    Spec:
      Resources:
  Replica Service Template:
    Metadata:
    Spec:
  Replicas:  1
  Service Template:
    Metadata:
    Spec:
      Type:  ClusterIP
  Storage:
    Access Modes:
      ReadWriteOnce
    Data Source:  <nil>
    Resources:
      Requests:
        Storage:         1Gi
    Storage Class Name:  aws-ebs-gp2
  Storage Type:          Durable
  Termination Policy:    Delete
  Update Strategy:
    Type:   RollingUpdate
  Version:  11.2
Status:
  Observed Generation:  1$2803726858785808847
  Phase:                Running
Events:                 <none>

[iamrz1]

@soosap We started adding support for storages with s3 compatible apis (minIO) since kubeDB v0.11. and the endpoint field in CRD is being used exclusively to identify those compatible storages since it is not necessary to mention endpoint for actual aws storage (it gets detected automatically). Hopefully, keeping the endpoint field empty will solve your problem. But please do let us know if your problem persists and we will take a closer look at it.

[soosap]

Ok I can confirm that workaround works for me. I leave this ticket open since specifying the endpoint should ideally not lead to this error. Thanks for pointing this out.

[kapolos]

+1 for adding region. I tried to use Scaleway's s3-compatible object storage and the block is the hard-coded region:

AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'

[shivpathak]

I am using v0.12.0 kubedb version.

can we get the region feature soon.. i am also facing while trying to backup mongodb to s3 bucket created in eu-west-1 region.

 backupSchedule:
    cronExpression: "@every 1m"
    storageSecretName: mongo-backup-s3-secret
    s3:
      bucket: mongodb-bkp-bkt
      endpoint: "s3.eu-west-1.amazonaws.com"

but it's not working it throws following error -

Warning SnapshotFailed 15s KubeDB operator Failed to generate osm secret. Reason: RequestError: send request failed caused by: Get https://s3.amazonaws.com/mongodb-bkp-bkt?location=: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxx\n/20191121/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b276e978fa767bd6745e8bb6397c1cc609e879ecf83079e47dbf6387c4cd5737" for key Authorization

I can see while generating request it has hard-coded us-east-1 location.

[iamrz1]

Don't use the endpoint field for amazon s3. That will fix your problem. Endpoint will be automatically detected for actual amazon s3 storage.

Endpoint field is for s3 compatible storage services (ie. MinIO, clone-space etc) only.

And the region feature has been merged to the master branch, and will be available in the next release.

[shivpathak]

@iamrz1 Thanks for your response.

I got it fixed by following -

1 - remove the endpoint for s3
2 - create s3 secrets using below command -

echo -n "YOUR AWS ACCESS KEY" > AWS_ACCESS_KEY_ID
echo -n "YOUR AWS SECRET KEY" > AWS_SECRET_ACCESS_KEY
kubectl create secret -n kubedb generic s3-secret
--from-file=./AWS_ACCESS_KEY_ID
--from-file=./AWS_SECRET_ACCESS_KEY

Now I am able to backup using s3 secrets successfully but I got in new trap when using IAM roles instead of aws secret credentials.

I am using kube2iam to pass roles in kuebernetes. but some how it's not working -

I have posted more details about this on https://github.com/kubedb/project/issues/212#issuecomment-558000158

[shivpathak]

@all I think "Continuous Archiving to S3 hardcoded to us-east-1 region" issue should be closed.

You may refer to https://github.com/kubedb/issues/issues/568#issuecomment-558002719

to get solution of it.

[gris-gris]

I have the same problem when I use S3 storage from Scaleway:

Status:                                                                                                             │
│  Completion Time:  2020-03-05T23:33:45Z                                                                            │
│  Phase:            Failed                                                                                          │
│  Reason:           GetBucketLocation: AuthorizationHeaderMalformed: The authorization header is malformed; the regi│
│on 'us-east-1' is wrong; expecting 'fr-par'