Boto3: s3 bucket filter returns too many results

[boompig]

MaxKeys parameter seems to be ignored by Bucket.objects.filter method

Code:

session = boto3.Session(profile_name=aws_profile, region_name=aws_region)
s3 = session.resource("s3")
bucket = s3.Bucket(bucket_name)
it = bucket.objects.filter(MaxKeys=10)
n = 0
for item in it:
    if n == 15:
        print("TOO MANY!!!")
        break
    print(item)
    n += 1
print(n)

Desired effect: print 10 items, then print 10
Actual effect: print 15 items, print TOO MANY, print 15

parameters to reproduce:
python version 2.7.11
boto3 version 1.1.3
platform Linux 4.1.3 AMD64

[JordonPhillips]

This is an issue with the documentation, we shouldn't be showing pagination parameters since the collections will paginate through all options. What MaxKeys does is set the number of responses to each individual list_objects request we make, but we will exhaust them all. To get a specific number, you can use .limit.

For the following example, say we have a bucket with the following objects:

bar
baz

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('foo')

len(list(bucket.objects.all()))
# 3

len(list(bucket.objects.filter(Prefix='ba')))
# 2

len(list(bucket.objects.filter(Prefix='ba').limit(1)))
# 1

len(list(bucket.objects.limit(2)))
# 2

[boompig]

This makes sense, but as you said the documentation is not correct. Also the keyword MaxKeys in this usage is not consistent with MaxKeys used in other contexts, where it does limit the number of results. In any case, thanks for clarifying this.

[selik]

@boompig Why is this closed if the documentation hasn't been fixed yet?