Aws-cli: Support different S3 ls output formats

[zasran]

aws s3 ls does not support the usual formats (json, text, table) and the default format is not very good for scripting, it prints out some extra irregular data in addition to names of buckets, keys and objects.

It also does not differentiate between keys and objects, if asked to list an object it treats it as a key and essentially lists it as empty directory. In comparison I find s3cmd output more useful, it lists the object itself (plus last modified time and size).

[devoncrouse]

+1

[slowbluecamera]

+1

[ilkkarinne]

+1
I was surprised not see the --output flag working for s3

[mcaramello]

+1

aws s3 ls accepts and ignore the --output option silently, which is extra source of confusion. At the very least I'd expect an error if the option is not supported.

[slowbluecamera]

Forehead _slap_. Somehow I missed the "s3api" command group 3 months ago. Found it now. Very very happy.

[danpritts]

+1, 9 months later

[jamesls]

Just to clarify the current state of things.

The aws s3 ls command emulates the ls command and has a similar sort of output.

For more programatic access, you can use the low level aws s3api list-objects command. The commands in s3api map 1-1 to the operations exposed by S3 (http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html). Using list-objects will give you all the normal output types available via --output and exposes all the parameters supported by s3 (http://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html). Some examples of using this command, along with query to demonstrate programatic processing:

First 10 key names, in JSON:

$ aws s3api list-objects --bucket bucket-name --max-items 10 --query Contents[].Key --output json
[
    "00/00/21/3a5632f9aa790e2685092b98aea5f1079d",
    "00/00/60/d8bd62e4f269c9aef69c677038f9b5c959",
    "00/00/66/4c4ce464364b5a262665b7c69a7ccdd2ca",
    "00/00/72/8292a4fc9cb568b981582455a00662e0a2",
    "00/00/7a/fd6300d520aaed196854424e969cf6833f",
    "00/00/aa/b49aa10ac7600f11f5bc951bcc2629bdf7",
    "00/00/ac/1168571edf8e5fb41b281095d9531df742",
    "00/00/b8/c895f493785ac3e4f6e83914fd0947ac91",
    "00/00/d3/15fc5da49e1dd59e2b2408009c4db64c60",
    "00/00/ee/01f441ab77a000099e5a865c3f9e5b851a"
]

Show a table of objects that are 4k in size with headers of key name, size, and last modified time:

$ aws s3api list-objects --bucket bucket-name --max-items 10 --query 'Contents[?Size == `4096`].[Key,Size,LastModified]' --output table
-------------------------------------------------------------------------------------
|                                    ListObjects                                    |
+----------------------------------------------+-------+----------------------------+
|  00/00/21/3a5632f9aa790e2685092b98aea5f1079d |  4096 |  2014-06-06T19:48:54.000Z  |
|  00/00/60/d8bd62e4f269c9aef69c677038f9b5c959 |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/66/4c4ce464364b5a262665b7c69a7ccdd2ca |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/72/8292a4fc9cb568b981582455a00662e0a2 |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/7a/fd6300d520aaed196854424e969cf6833f |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/aa/b49aa10ac7600f11f5bc951bcc2629bdf7 |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/ac/1168571edf8e5fb41b281095d9531df742 |  4096 |  2014-03-13T21:35:24.000Z  |
|  00/00/b8/c895f493785ac3e4f6e83914fd0947ac91 |  4096 |  2014-03-13T21:35:25.000Z  |
|  00/00/d3/15fc5da49e1dd59e2b2408009c4db64c60 |  4096 |  2014-03-13T21:35:25.000Z  |
|  00/00/ee/01f441ab77a000099e5a865c3f9e5b851a |  4096 |  2014-03-13T21:35:25.000Z  |
+----------------------------------------------+-------+----------------------------+

Or, if you prefer tab separated output for easier access with cut/awk/sed/grep:

$ aws s3api list-objects --bucket bucket-name --max-items 10 --query 'Contents[].[Key,Size,LastModified,Owner.DisplayName]' --output text
00/00/21/3a5632f9aa790e2685092b98aea5f1079d     4096     2014-06-06T19:48:54.000Z     ownername
00/00/60/d8bd62e4f269c9aef69c677038f9b5c959     4096     2014-03-13T21:35:24.000Z     ownername
00/00/66/4c4ce464364b5a262665b7c69a7ccdd2ca     4096     2014-03-13T21:35:24.000Z     ownername
00/00/72/8292a4fc9cb568b981582455a00662e0a2     4096     2014-03-13T21:35:24.000Z     ownername
00/00/7a/fd6300d520aaed196854424e969cf6833f     4096     2014-03-13T21:35:24.000Z     ownername
00/00/aa/b49aa10ac7600f11f5bc951bcc2629bdf7     4096     2014-03-13T21:35:24.000Z     ownername
00/00/ac/1168571edf8e5fb41b281095d9531df742     4096     2014-03-13T21:35:24.000Z     ownername
00/00/b8/c895f493785ac3e4f6e83914fd0947ac91     4096     2014-03-13T21:35:25.000Z     ownername
00/00/d3/15fc5da49e1dd59e2b2408009c4db64c60     4096     2014-03-13T21:35:25.000Z     ownername
00/00/ee/01f441ab77a000099e5a865c3f9e5b851a     4096     2014-03-13T21:35:25.000Z     ownername

Hope this helps. I do think we should be giving an error message if --output is specified in any of the s3 high level commands, with a mention to using the s3api commands. I'll file an issue for this.

[danpritts]

Adding this writeup to the official documentation would be really useful. thanks for taking the time to write it up.

A simple 'ls' (rather than the 'ls -l' emulated) is what I wanted, and it's achievable with:

aws s3api list-objects --bucket bucket-name --query 'Contents[].[Key]' --output text

[isaacegglestone]

This issue is closed but realistically here the best thing to do would be to put this information in the s3 ls command output. This tool does not support output formats use s3api instead if you need this.
Here I am some four years later still encountering this problem and finding the solution here when it could easily be added to the help for the tool.

[jarruda]

@jamesls In my opinion this issue should not have been closed just because a workaround exists. The customer request is for the ls command to support it, and the workaround is cumbersome at best.

[zasran]

@jamesls thanks for providing information about workaround but I don't think this issue should have been closed without the fix. Either providing support for different formats (good fix) or at least error message mentioning s3api (IMO bad fix but sort of acceptable) would be ok.