Aws-sdk-java-v2: S3 URI Parser

[millems]

1.11.x provides http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3URI.html to parse the interesting components out of a S3 URI.

A similar piece of functionality should be made available in 2.0.

In the short term, customers can still use it from 1.11.x.

[perihelion1]

Similarly, my project uses AmazonS3Client.getUrl(String bucketName, String key) to generate and save the URL of an object I've just created. Is there anything equivalent available in 2.0?

[millems]

@perihelion1 https://github.com/aws/aws-sdk-java-v2/issues/860 tracks that feature.

[ribeirux]

Any update on this?

Should we still use the AmazonS3URI from 1.11.x to get the bucket/key/version/region from a S3 URL?

Is there a way of downloading an object from S3 using the URl with v2 sdk for java?

[millems]

@ribeirux There's currently only a function to create a URL, given the bucket/key/etc in 2.x.

Parsing the URI for bucket/key/etc. is surprisingly challenging, and our functionality in 1.11.x doesn't cover all scenarios. We'd like our 2.x implementation to work for all URLs, but things are in flight with S3's URI patterns right now (see: path style deprecation, among other things).

We'd like to see the dust settle on that chaos before we commit to being able to implement this functionality.

What's the reason you need parsing of the URI? That would help us when we're designing things out.

As for downloading an object using a URI in the SDK: not really. Is there a reason the JDK's URL connection (or something similar like Apache's HTTP client) isn't sufficient? We'd like to create a way to download objects using the SDK's retry policies (see: the downloading via presigned URLs design discussion), but that's lower in our backlog than removing other 2.x migration blockers.

[mrog]

@millems I can't speak for @ribeirux , but I'll give you my use case. We have a service that saves files to S3 and includes their HTTPS URLs in the output. These URLs are meant to be consumed both by people (via web browsers on our internal network) and other services (in our data center).

Our security has been set up so that every service has its own AWS key. Services can't retrieve files from S3 without using the S3 client and authenticating with their keys. This provides better security than letting every service read from every bucket. It also means we have to extract the bucket names and keys from the URLs so we can provide them to the S3 client.

While we could write our own code to parse the URLs, having the AWS SDK do that step would be a much more maintainable solution.

[millems]

That makes sense. I don't think S3 URLs were originally designed to be reversible, but I can see how it's easier to store one thing (the URL) than both the bucket and the key for later use.

[ribeirux]

Thanks @millems for the quick reply. @mrog that's precisely my use case :)

The aws cli supports downloads using the S3 URI. It would be great to have the same logic across different sdks/tools.

[slobo]

Parsing the URI for bucket/key/etc. is surprisingly challenging, and our functionality in 1.11.x doesn't cover all scenarios

Are we talking about s3://bucket/path/to/key style URIs? Very curious to know what the caveats are, as in some projects we used a simple pregex (^s3://([^/]+)/(.+)$), and I can't think of a situation when it would fail for a valid URI...

[carlspring]

Are there any plans to prioritize this and look into it? We could really use it for our work on the s3fs-nio project.

[Tvaroh]

Parsing the URI for bucket/key/etc. is surprisingly challenging, and our functionality in 1.11.x doesn't cover all scenarios

Are we talking about s3://bucket/path/to/key style URIs? Very curious to know what the caveats are, as in some projects we used a simple pregex (^s3://([^/]+)/(.+)$), and I can't think of a situation when it would fail for a valid URI...

I use it to parse signed S3 links which look quite differently.

[LikeLifeItself]

Spring cloud aws (version 2.x.x) uses s3://blah-blah URIs in SimpleStorageProtocolResolver. I see that spring cloud in progress with SDK V2 integration and hope that they will support URIs too, but don't see any ways to build URIs by SDK V2.