Boto3: Allow inverted filters to return all the results NOT matching specific filters

[geekpete]

Is there any way to return results NOT matching a filter currently?

[geekpete]

It seems that Django Querysets that the boto3 filtering was based on also have an Exclude method as the inverse of a filter.

This would be perfect!
When might it be implemented?

I guess it's just some set comparisons to implement this?
Fetch all results, fetch filtered results, then return the difference between those two sets?
Might need some considerations to be performant to avoid unnecessary lookups where possible, but should be easy to do.

https://docs.djangoproject.com/en/1.7/ref/models/querysets/#exclude

[geekpete]

Or would this actually be implemented in the botocore library instead?

[mbarrien]

I believe the actual implementation is done on AWS servers... boto3/botocore just pass the filter arguments to the server. Not something that can be implemented in the Python.

[geekpete]

That's fine if the AWS API doesn't actually support inverse filters, just implement them in boto3 under the hood and serve up an exclude() method to us users to compliment the existing filter() method.

Do this by:

• fetch a list of all possible items
• fetch a list of all items matching the given filter (using existing filter method)
• return the difference of those two lists, which is all items NOT matching the filter, ie, excluded by the given filter.

I did have a quick look through the source and see that there was no specific implementation for filter() anywhere in the code, so it's obviously implemented somewhere else other than boto3 and boto3 is just passing it all down to the next layer that does handle it.

So I'd understand why people might not want to muddy the code by overloading functionality in the boto3 codebase by adding extra stuff that isn't natively supported by the API, but damn it would be useful.

Now I have to add all those lines to my python code to do that exclude myself instead of just running an exclude() with a provided filter I want to exclude by.

One of the best advantages of boto3 over boto is that it reduces the amount of code I have to write to get something done and eliminates loops and fiddly stuff needed to chop the right results up once boto gets them for me. Boto3 and filters reduce so much of that pokery jiggery back to a single method call, which is awesome. But currently it only works one way and to get the inverse of a filter is still manual.

[mbarrien]

While I agree that boto3 should do some stuff, I should point out that boto2 _does_ have those filters which you think are new in boto3. I have actually had to do a hell of a lot more fiddly stuff in boto3 than boto2 because its implementations of things are incomplete. (e.g. all of Route 53, or things like filtering through spot instances in EC2).

Also keep in mind that doing the "extract all, then subtract the filtered" can get really, really screwy in the face of the API pagination implemented by AWS. Or in the face of their attempts to get this "Filter" method to work the same across all their different services, while doing this only through config files and in the face of the many different response shapes each service responds with. This is a much larger feature request than you may realize in boto; I'd argue it's actually easier for AWS to implement it server side.

That said, it would be nice!

[geekpete]

Makes sense that the API should really do it, otherwise the overhead just gets shifted somewhere else.

I didn't realise boto2 was all filter-awesome. I thought it was more of a botocore thing that boto3 inherited, but it's the API providing it so was always there I guess.

I found standardisation is a bit better for results in boto3 vs boto2, where boto2 would return different result types for one api vs another when dealing with the same resource type but via different apis.

Boto3 seems to get that lined up a bit better, but I guess lots of stuff is still just missing.

I'll close this issue for now and lobby Amazon to add more to their API ;)

Would be cool functionality, but I see by no means trivial.

Thanks.

[emersonposadas]

Please make this happen. Negative filters or 'Not tagged' Filters are really needed. Or is there any other way to do this?

[max-allan-surevine]

Can I add a "me too" here. You can definitely do it in the AWS console. I have a tag called "Scheduler" and a search for : "tag:Scheduler : Not tagged" shows instances without it. So the functionality is there somewhere.

[geekpete]

I guess someone could take boto3 and build an extra layer of "helper" functions on top of it as a new library that imports boto3 as its core.
Might be the best way forward of the boto team want to keep the underlying library clean of these "extras"