Aws-cli: Unable to get JSON output from `aws s3 ls` command

[boosh]

Output from aws s3 ls is always returned as text regardless of the value of the --output parameter.

[danielgtaylor]

I can confirm that this does not work as expected. If you are looking for JSON output from a specific S3 command, you can however use the low level API:

$ aws s3api list-buckets --output json
...
$ aws s3api list-buckets --output table
...

[jamesls]

The --output parameter is not intended to work with the aws s3 commands. You'll need to use the s3api commands to get different output. Given that --output is a global argument, the only real option we have is to document this.

[bes1002t]

this is a mess. S3 is able to sync and to ls but not to output the result as JSON string. S3api is able to output in JSON strings but not able to use sync command or ls. Many different tools and non of them is able to provide this simple featureset.

[dagwieers]

@bes1002t This works for me

$ aws --output json s3api list-objects --bucket your-bucket --prefix 'some/path/'

As an alternative to:

$ aws --output json s3 ls s3://your-bucket/some/path/

However, I do not see a good reason why JSON output would not be supported by aws s3 ls.

@jamesls Would a PR to add this be accepted?

[dannystaple]

Perhaps if it's not supported, the command should output a message stating that the output format isn't supported, and at a stretch, suggest an alternative that is supported? Kind of in the way git has the "did you mean" suggestions.

[bes1002t]

@dagwiers list-objects prefix could be a solution. When I posted my rent I did not know about that command. :P

How about the performance of list-objects vs ls? I experienced some performance issues using list-objects because with large buckets it compares the prefix with EACH file in bucket, which needs a huge amount of time for many files.

[dagwieers]

@bes1002t That is why I asked if a PR would be accepted. Now it looks like this was by design and will not be improved/fixed.

[nikolai-derzhak-distillery]

Yes. On large buckets s3api takes forever. So what is solution really ? Besides using external DB to index s3 objects :)

[bes1002t]

@nikolai-derzhak-distillery it seems there is no real solution so far. I tried several ways and none of them was a good solution for large buckets. I don't understand why nobody cares about such an performance issue.