Aws-cli: aws s3 exclude / include commands ignored

[paulsickling]

On Windows - when using aws s3 commands (cp or mv) with exclude and/or include parameters the parameters only seem to be honoured if the command is executed from the same drive as the source parameter.

For instance if the source folder is d:\backup and the command is run anywhere from d:\ within cmd then exclude / include parameters are honoured. If the cmd drive is changed to c:\ then the exclude and include parameters appear to be ignored.

See output below

C:>aws s3 cp d:/backups/temp3/ s3://xxx/ --recursive --exclude * --include *.zip
upload: d:\backups\temp3\2.zip to s3://xxx/test/2.zip
upload: d:\backups\temp3\readme.zip to s3://xxx/readme.zip
upload: d:\backups\temp3\readme.txt to s3://xxx/readme.txt (<<< EXTRA FILE PICKED UP)

C:>d:

D:>aws s3 cp d:/backups/temp3/ s3://xxx/ --recursive --exclude * --include *.zip
upload: backups\temp3\readme.zip to s3:/xxx/readme.zip
upload: backups\temp3\2.zip to s3:/xxx/test/2.zip

I've tried various source path syntaxes including d:\backups... d://backups/... but all yield the same result.

The workaround for the time is to ensure cmd is started from the same drive as the source but I assume that isn't by design.

Love the tool in every other way!

[lashex]

the --exclude and --include commands also seem to be ignored when performing aws s3 rm. Here's an example:

$ aws s3 rm s3://bf-prv/ --include logs/access_log-2013-08* --recursive --dryrun

...and here's some of the dry run output.

(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-29-17-15-36-F60D010783904E0E
(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-29-22-16-20-B923803ECB07E887
(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-30-18-16-55-CF342DE4040EE552
(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-30-21-16-20-0CAAE0E25000837D
(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-30-22-16-54-071252853D47D631
(dryrun) delete: s3://bf-prv/logs/access_log-2013-08-31-19-16-54-F58E3C03368C3584
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-01-17-15-55-3D577D7374E3E8EE
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-01-17-16-55-85694A47CFCDE456
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-02-00-15-46-6FC907538B5F7614
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-02-02-15-46-306F3EE8FAD968A2
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-05-16-10-B6AFAE34E296649E
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-16-16-58-870D5E7612399120
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-05-16-28-800721A64131EB23
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-16-16-59-36225A9A99DC22CD
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-17-17-07-E7767A22E59B2DF5
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-18-16-49-A2F9BBEC63DBDB3C
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-19-16-48-A2C281930CA7DE99
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-22-16-17-FE0A56C5EC17DDBD
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-23-16-30-4E0CC006CC599849
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-03-23-16-36-A685601F821DCBEB
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-04-01-15-59-B67124FBD56CD47C
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-04-02-16-40-6A600E338B8307C1
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-04-04-16-29-A0CFAE575FF94F1A
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-04-06-15-43-B4569B48D81D0AA6
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-04-06-16-16-C8D541524F67F882
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-05-01-15-52-1F3EECB676D8DE2A
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-05-01-16-03-CE28DA97291F8635
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-05-01-16-30-572D5453403C1C9F
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-12-07-18-42-78B95F6C083BCF0B
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-12-07-19-35-A6DA139AE6CDFF2F
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-12-08-18-53-527DE911716E748C
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-12-07-19-02-AEBE198092C68610
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-16-18-19-10-8F8695CF562A01E2
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-16-18-19-16-0993EBD2E89EDD75
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-16-18-19-57-146C8C57C0ACFFE3
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-16-18-20-11-291FE8484CC69212
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-17-00-18-29-4646DF7630F003D7
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-17-00-19-25-51D5A20202DA89DD
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-24-18-18-00-61E67783DF74CF51
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-24-18-18-53-06DDB0DA501F559F
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-24-18-19-40-93673B7B4068D350
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-24-18-19-45-4BEA28E1FA6E717B
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-24-18-19-59-5EE0DA43DF6D60F6
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-26-19-17-36-D6F672DEA57A295C
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-26-19-18-08-CD53455E6B2AEBD0
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-26-19-19-05-D383CD948770443D
(dryrun) delete: s3://bf-prv/logs/access_log-2013-09-26-19-20-23-CC3D3451DC1957E2
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-05-05-16-23-2F9F7F02B3FF247E
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-05-05-16-33-2164276C5D881664
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-05-05-17-39-1D1478DCFE02E53E
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-17-17-12-137EC5617749C235
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-17-18-04-D28DBEB60AB8119A
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-17-19-36-0F659894EFBACFAD
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-17-21-37-942C71ABAC4BD017
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-17-22-56-7EDC7890A35234F1
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-16-17-6CE5DEB48B1CB34A
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-16-39-9CDA6D316FE79508
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-16-50-6E43A70FF0175703
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-17-38-70FC123BE5AFE3E2
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-18-59-CA218ECBCAB722E0
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-20-16-43767693F7BDA633
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-10-18-20-50-8607FE99963949D6
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-20-16-46-F93A71C6592B8715
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-20-18-01-85F004CD8D6A21B5
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-20-19-21-5643B10101B458CD
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-22-18-14-44C87A78541B11BE
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-22-18-42-42CA9FCDB249170A
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-22-19-58-4E968CD493117893
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-22-21-07-F87AC5785BAF1F8C
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-22-22-14-55B2F6433F10A771
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-23-19-20-D299DC8B9BB7AE92
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-11-23-21-30-F4D1B7C490318865
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-19-24-D6A7C4359E5170C8
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-21-01-3B3EC977260C5591
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-23-09-30B6F4E31064A40D
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-21-51-A698129F8B663D43
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-24-18-1813D72F442ACBB1
(dryrun) delete: s3://bf-prv/logs/access_log-2013-10-21-22-25-18-C71A738D0F2140F0
(dryrun) delete: s3://bf-prv/logs/ex130515.zip  
(dryrun) delete: s3://bf-prv/logs/ex130516.zip  
(dryrun) delete: s3://bf-prv/logs/ex130517.zip  
(dryrun) delete: s3://bf-prv/logs/ex130518.zip  
...

[deltheil]

The same occurs with aws s3 sync:

$ aws --version
aws-cli/1.2.6 Python/2.7.5 Darwin/12.5.0
$ tree test
test
├── a.jpg
├── b.png
└── c.jpg

0 directories, 3 files
$ aws s3 sync test s3://my-bucket/test --include "*.png" --dryrun
(dryrun) upload: test/a.jpg to s3://my-bucket/test/a.jpg
(dryrun) upload: test/b.png to s3://my-bucket/test/b.png
(dryrun) upload: test/c.jpg to s3://my-bucket/test/c.jpg

[estahn]

:+1:

[andrassy]

+1

Just bitten by this nasty bug. Please fix.

[danieljimenez]

Just got bit by this one too. It's a shame it's documented but not working. Not running Windows either.

[jamesls]

There's two issues being discussed here. The first is a windows specific issue.

The other comments here appears to be about --include filters that aren't specific to windows. Please note that if you provide --include it does _not_ mean "only include." This is documented here: http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#use-of-exclude-and-include-filters. By default, _nothing_ is excluded so a bare --include filter will not change anything. As the OP shows, the expected way to do this is --exclude "*" --include "*.jpg" --include "*.png" .... A better way of thinking of --include is "include this file even if it was previously excluded" or "don't exclude files matching this pattern". Perhaps we can update the docs to word this better and make this more clear.

[danieljimenez]

Thanks for your comment Jamesis, I actually thought that might be how it worked but I tried the following with no joy:

aws s3 sync s3://mys3logbucket/ ./ --exclude "*" --include "log2013*"

[danieljimenez]

So, shouldn't this work?

$ aws s3 sync s3://myBucket/ ./ --dryrun --exclude "cdn2012*"
(dryrun) download: s3://myBucket/cdn2012-01-01-01-29-34-9B3A2A60D8038717 to ./cdn2012-01-01-01-29-34-9B3A2A60D8038717
(dryrun) download: s3://myBucket/cdn2012-01-01-01-29-48-1C79006F07427CC0 to ./cdn2012-01-01-01-29-48-1C79006F07427CC0
(dryrun) download: s3://myBucket/cdn2012-01-01-01-30-25-102D7B26ADE615DC to ./cdn2012-01-01-01-30-25-102D7B26ADE615DC
(dryrun) download: s3://myBucket/cdn2012-01-01-01-32-35-5A35952D8E6B792E to ./cdn2012-01-01-01-32-35-5A35952D8E6B792E
(dryrun) download: s3://myBucket/cdn2012-01-01-01-33-40-A95FE8889DB4DBBF to ./cdn2012-01-01-01-33-40-A95FE8889DB4DBBF

[jamesls]

@danieljimenez I believe you're running into https://github.com/aws/aws-cli/pull/531 which should be fixed in the latest version (currently 1.2.7). There was a bug in earlier versions that required the bucket name to be included for include/exclude filters for s3 locations, which is inconsistent with how local file include/exclude works. This has now been fixed and you're example should work on the latest version.

Debug logs have also been added to help troubleshoot include/exclude issues. For example, running with the --debug option now gives this output:

$ aws s3 ls s3://bucket-name/
2013-12-11 13:54:08          0 bar.jpg
2013-12-11 13:54:08          0 bar.png
2013-12-11 13:54:08          0 foo.jpg
2013-12-11 13:54:08          0 foo.png

$ aws s3 sync s3://bucket-name/ . --dryrun --exclude 'foo*' --debug 2>&1 | fgrep 's3.filters' 
2013-12-11 13:58:59,913 - awscli.customizations.s3.filters - DEBUG - bucket-name/bar.jpg did not match exclude filter: bucket-name/foo*
2013-12-11 13:58:59,913 - awscli.customizations.s3.filters - DEBUG - =bucket-name/bar.jpg final filtered status, should_include: True
2013-12-11 13:58:59,914 - awscli.customizations.s3.filters - DEBUG - bucket-name/bar.png did not match exclude filter: bucket-name/foo*
2013-12-11 13:58:59,914 - awscli.customizations.s3.filters - DEBUG - =bucket-name/bar.png final filtered status, should_include: True
2013-12-11 13:58:59,915 - awscli.customizations.s3.filters - DEBUG - bucket-name/foo.jpg matched exclude filter: bucket-name/foo*
2013-12-11 13:58:59,915 - awscli.customizations.s3.filters - DEBUG - =bucket-name/foo.jpg final filtered status, should_include: False
2013-12-11 13:58:59,915 - awscli.customizations.s3.filters - DEBUG - bucket-name/foo.png matched exclude filter: bucket-name/foo*
2013-12-11 13:58:59,915 - awscli.customizations.s3.filters - DEBUG - =bucket-name/foo.png final filtered status, should_include: False

So in this example, we can see that the pattern "bucket-name/foo.png" matched the exclude filter "bucket-name/foo*" and should_include: False which indicates that the file is excluded (and won't be synced).

Hope that clarifies things, I'll get the docs updated to better explain this.

[jamesls]

I can confirm that the windows issue originally mentioned is still an issue. This appears to be because we are including the drive name as part of the filter on windows:

2013-12-11 22:27:17,332 - awscli.customizations.s3.filters - DEBUG - C:\Users\user1\testit\foo.zip did not match exclude filter: D:\*
2013-12-11 22:27:17,332 - awscli.customizations.s3.filters - DEBUG - C:\Users\user1\testit\foo.zip did not match include filter: D:\*.zip
2013-12-11 22:27:17,332 - awscli.customizations.s3.filters - DEBUG - =C:\Users\user1\testit\foo.zip final filtered status, should_include: True

Looking into a fix.

[jamesls]

This is now fixed in https://github.com/aws/aws-cli/pull/554 and will go out in the next release.

[kilianc]

I still experience this with rm on aws-cli/1.9.10 Python/2.7.10 Darwin/15.2.0 botocore/1.3.10

[kilianc]

aws s3 rm "$S3_BUCKET/test/" --region ${S3_REGION:=us-east-1} --recursive --exclude "$S3_BUCKET/test/test1" --dryrun

(dryrun) delete: s3://my-bucket/test/
(dryrun) delete: s3://my-bucket/test/test1/
(dryrun) delete: s3://my-bucket/test/test1/pqatf4enkhxpfyuwkstq.jpg
(dryrun) delete: s3://my-bucket/test/test2/
(dryrun) delete: s3://my-bucket/test/test2/11870654_10207611827477332_7255026580168751425_n.jpg

[jingpengw]

me too, only exclude works, include do not.

aws help --version
aws-cli/1.2.9 Python/3.4.3 Linux/4.2.0-41-generic

[1adam]

Exclude doesn't seem to be working properly when attempting to sync from local to s3:

aws-cli/1.10.67 Python/2.7.6 Linux/3.13.0-79-generic botocore/1.4.57

$ tree static/
static/
├── valid
│   └── files
│       └── file.json
└── weather
    └── json
        └── file.json

4 directories, 2 files

$ aws s3 sync ./static/ s3://my-bucket/static/ --exclude "static/weather/*"
upload: static/valid/files/file.json to s3://my-bucket/static/valid/files/file.json
upload: static/weather/json/file.json to s3://my-bucket/static/weather/json/file.json

<delete 'static' from bucket>

$ aws s3 sync ./static/ s3://my-bucket/static/ --exclude "*static/weather/*"
upload: static/valid/files/file.json to s3://my-bucket/static/valid/files/file.json
upload: static/weather/json/file.json to s3://my-bucket/static/weather/json/file.json

EDIT: The following works. Looks like it applies to subdirs of the one specified -- does not include it. This is confusing at best.

$ aws s3 sync ./static/ s3://my-bucket/static/ --exclude "*weather/*"
upload: static/valid/files/file.json to s3://my-bucket/static/valid/files/file.json

[mtocci]

@1adam thanks! This worked for me for multiple directory excludes. I'm on Win10, so it looks like this:
--exclude "*Windows10Upgrade\*"

[chriszrc]

Seems like even though the files are at the top level, if you want include/exclude to work, you need the --recursive flag:

https://stackoverflow.com/questions/43370710/s3-cli-includes-not-working