Aws-sdk-go: Getting SignatureDoesNotMatch on keys with sub-folders

[davisford]

I recognize folders are just an abstract concept and in S3 everything is just a key + value. I've setup a top-level bucket, and created some buckets underneath. I generated an IAM user and gave the user full S3 r+w access. I added the credentials to ~/.aws/credentials and confirmed the following does work:

        var l = aws.LogDebugWithSigning | aws.LogDebugWithHTTPBody | aws.LogDebugWithRequestRetries | aws.LogDebugWithRequestErrors

        var chainVerboseErrors = true
        config := &aws.Config{
            CredentialsChainVerboseErrors: &chainVerboseErrors,
            Region:   aws.String("us-east-1"),
            LogLevel: &l,
            Logger:   aws.NewDefaultLogger(),
        }

        sess := session.New(config)
        Ω(sess).ToNot(BeNil())

        svc := s3.New(sess, config)

        params := &s3.ListObjectsInput{
            Bucket: aws.String("rootbucket"),
        }
        resp, err := svc.ListObjects(params)

This works fine, but if I change the bucket name to be Bucket: aws.String("rootbucket/childbucket"), or any combination of a bucket name that contains a path in it, it generates a SignatureDoesNotMatch error.

Does it have something to do with url escaping of the forward slash? I see this when I dial up logging:

---[ CANONICAL STRING  ]-----------------------------
GET
/rootbucket%2Fchildbucket%2F

      <*awserr.requestError | 0xc820bec1b0>: {
          awsError: {
              code: "SignatureDoesNotMatch",
              message: "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
              errs: nil,
          },
          statusCode: 403,
          requestID: "3FB8D6B1A02D295C",
      }

I can confirm that the bucket rootbucket/childbucket does indeed exist, and also contains further buckets underneath. When I list just rootbucket it returns all child buckets. Why can I not address the bucket directly?

[jasdel]

Hi @davisford thanks for contacting us. I think the issue you're seeing is with the usage of a bucket, and key. A bucket contains many objects, and all objects have a name which is its key. To reference a object you need to provide both the bucket, and the key to object.

If we have a bucket named mybucket and an object with the key my/object/namespace/log.txt, the elements my, object, namespace are not nested buckets, but portions of the object key's name. The S3 Web console will represent these parts as folders but this is a convenience method to make viewing buckets with many objects easier. You can use any form of delimiter in key names you'd like, or none at all if that makes sense in your use case. / is commonly used since it mimics how unix filesystems are delimited. Take a look at BucketBasics which goes over this topic in more detail.

With that said it sounds like you'd like to list buckets and only list a subset of the objects in the bucket. To do this checkout the ListObjectsInput.Prefix which will allow you to provide the key prefix to filter the list of objects by.

In addition I would suggest using the S3.ListObjectsPages func since it will provide pagination, which is especially helpful if you have many objects in that bucket.

[davisford]

Thanks, @jasdel for the detailed response. That makes sense.