Amplify-cli: More clarification needed on S3Object usage with GraphQL transformer...

[hisham]

* Which Category is your question related to? *
graphql-transformer

* What AWS Services are you utilizing? *
appsync graphql-transformer, s3

* Provide additional details e.g. code snippets *
In #274, it's claimed that the graphql-transformer supports s3 objects out of the box. And https://github.com/aws-samples/aws-amplify-graphql and https://github.com/aws-samples/aws-amplify-graphql/blob/4e008e9a6ae97b2cb25e00ff8496ac968ca38696/photo-client/src/Components/AddPhoto.js#L54-L59 are given as examples of how to do it in the frontend.

However the linked example (and all appsync s3 examples), also have specific code in the resolvers to transform the s3object into native dynamo db s3object types. See https://github.com/aws-samples/aws-amplify-graphql/blob/4e008e9a6ae97b2cb25e00ff8496ac968ca38696/photo-client/awsmobilejs/backend/appsync/resolver-mappings/Mutation.addPicture.request for example where you have this line in the resolver code:

#set( $attribs.file = $util.dynamodb.toS3Object($ctx.args.file.key, $ctx.args.file.bucket, $ctx.args.file.region, $ctx.args.file.version))

And this: https://github.com/aws-samples/aws-amplify-graphql/blob/4e008e9a6ae97b2cb25e00ff8496ac968ca38696/photo-client/awsmobilejs/backend/appsync/resolver-mappings/Picture.file.response

And this: https://github.com/aws-samples/aws-amplify-graphql/blob/4e008e9a6ae97b2cb25e00ff8496ac968ca38696/photo-client/awsmobilejs/backend/appsync/resolver-mappings/Picture.file.request

I do not believe the graphql-transpiler generates these s3-specific things in its resolvers, so I'm not sure if graphql-transpiler actually supports s3objects in the way that all aws appsync demos show as the way to do it. Specifically dynamodb.fromS3ObjectJson and dynamodb.toS3ObjectJson methods need to be used in the resolver code to serialize and deserialize s3 objects. And a local resolver is needed to deserialize the s3object. Does the graphql-transpiler generate code that actually does this?

[mikeparisstuff]

The $util.dynamodb.toS3Object(key, bucket, region, version) helper function results in the object being stored as a JSON serialized string in DynamoDB.

The output of this:

$util.toJson($util.dynamodb.toS3Object("key", "bucket", "region", "version"))

is

{"S": "{\"s3\": {\"key\":\"key\",\"bucket\": \"bucket\",\"region\": \"region\",\"version\":\"version\"}}"} 

The transform stores the bucket, key, version, and region as a DynamoDB map which avoids having to do input/output manipulation on the object itself. As far as I am aware, this has no impact on how the underlying Amplify/AppSync clients manage S3 under the hood and both approaches expose the same interface at the GraphQL schema level. I believe that helper was added as a utility to help people using the DynamoDB Mapper Java library easily convert the s3 links to the same format. If you have experienced something different please let me know. We can add support for that utility but I don't think its required and using it adds some complexity as we would need to add resolvers to every S3Object field just to do the conversion from the inline JSON stored in DynamoDB back to the object as it is expected at the GraphQL layer.

[hisham]

Yep thanks for the clarification and background info. I agree. I'll try to implement s3 upload/download in my app without the toS3Object/fromS3Object stuff and report back on how well that works.

[mikeparisstuff]

@hisham sounds good! It looks like you already know about this sample https://github.com/aws-samples/aws-amplify-graphql that shows how to get the URL and download a file directly from the browser. If you work out the code that takes it a step further and renders the file in an html image tag or similar it would be great to get that added to the sample as well. Let me know what you find. Thanks

[hisham]

Sure, though I am doing this in angular and only need file download functionality for now.

As for the whole 'The S3ObjectInput type will be generated for you' thing. While this is true, the S3ObjectInput type needs to have a localUri that references the local file that will be uploaded. The aws javascript sdk will then upload whatever localUri points to to S3. See https://github.com/awslabs/aws-mobile-appsync-sdk-js/issues/63#issuecomment-370175926.

But we don't want to store localUri in the S3Object in dynamodb as it's irrelevant and also can potentially contain blobs and buffers which could exceed dynamo's row limits.

The S3Object that should be stored in dynamo should be something like this:

type S3Object {
  bucket: String!
  key: String!
  region: String!
  mimeType: String
  fileName: String
}

But S3ObjectInput needs to add a localUri to this so it should be this:

input S3ObjectInput {
  localUri: String!
  bucket: String!
  key: String!
  region: String!
  mimeType: String
  fileName: String
}

So I don't recommend adding localUri to S3Object so that the graphql-transpiler can auto generate S3ObjectInput in the correct format. Instead this input should be explicitly written out in the user's graphql schema with localUri added.

[hisham]

Ok I got it working. Everything was fine except with the additional variables needed in S3ObjectInput that will not end up in S3Object. The AWS JS sdk will actually delete 'localUri' and 'mimeType' from S3ObjectInput so those values will never reach the backend.

So I think a person can have this definition in their graphql schema:

type S3Object {
  bucket: String!
  key: String!
  region: String!
  fileName: String
  localUri: String
  mimeType: String
}

And the graphql-transpiler will automatically create the right S3ObjectInput. But localUri and mimeType in S3Object in the db will always not be set as the aws js client will delete these, so I prefer to be explicit as follows in the schema.graphql:

type S3Object {
  bucket: String!
  key: String!
  region: String!
  fileName: String
}

input S3ObjectInput {
  bucket: String!
  key: String!
  region: String!
  fileName: String
  # aws js client will delete these after upload
  localUri: String
  mimeType: String
}

[hisham]

I ended up showing the image also in an <img> tag. The process roughly is:

1) Download the s3 object via the amplify js lib - this.amplifyService.storage().get...
2) Wrap the resulting s3 object.Body (which is a uint8array) into a blob like new Blob([object.Body]). You can also pass content type with: blob = new Blob([object.Body]), { type: object.ContentType })
3) Use FileReader.readAsDataURL(blob) to get the data uri
4) Set the img src to that data uri

[mikeRChambers610]

It would be great if there was an actual tutorial with the steps to achieve file upload via complex object.
Even the AWS docs seem unsure of this process...

Tutorial Part 1 : Client -> AppSync -> S3
Tutorial part 2 : S3->Appsync -> client

[michaelcuneo]

I still can't even get a single file to upload, the steps in the tutorial don't appear to work.

[kaustavghosh06]

We're actively looking at this. @yuth

[mfreeman451]

any updates? In the examples I've seen it says to use S3Object, but amplify/appsync complain that it doesnt exist unless you define the type.

[tafelito]

For me the upload works fine on JS when I use localUri prop, but the example shows that the key is composed by the name of the file and also the visibility level. The problem with that, is that the key is saved to the DB as is, so then when using Storage.get, I need to strip out the visibility first.
Without concatenating the visibility, then the uploads fails because of permissions

Is it possible to set the visibility as a different prop?