Boto3: Cloud Directory should accept None as NextToken

[tuukkamustonen]

Let's say your listing and index or something, and want to iterate the pages:

next = None
while True:
    resp = client.list_index(
        DirectoryArn='...',
        RangesOnIndexedValues=[...]
        IndexReference={'Selector': '..'},
        NextToken=next
    )
    next = resp.get('NextToken')
    if not next:
        break

The above doesn't work as boto doesn't accept None as NextToken value (it nags about it not being string, iirc). So I would have to:

next = None
while True:
    if next:
        resp = client.list_index(
            DirectoryArn='...',
            RangesOnIndexedValues=[...]
            IndexReference={'Selector': '..'},
            NextToken=next
        )
    else:
        resp = client.list_index(
            DirectoryArn='...',
            RangesOnIndexedValues=[...]
            IndexReference={'Selector': '..'}
            # omit it here
        )

    next = resp.get('NextToken')
    if not next:
        break

Which is pretty annoying. Workaround is to:

next = None
while True:
    params = dict(
        DirectoryArn='...',
        RangesOnIndexedValues=[...]
        IndexReference={'Selector': '..'}
    )
    if next_token:
        params['NextToken'] = next
    resp = client.list_index(**params)

    next = resp.get('NextToken')
    if not next:
        break

Which is pretty ugly, plus you lose auto-completion (which actually doesn't exist due to https://github.com/boto/boto3/issues/1055).

[stealthycoin]

Ideally this is where you would use a paginator to do this for you. It looks like there is no paginator for this operation at the moment but there is an open pull request for it here.

[tuukkamustonen]

Thanks for the pointer. Haven't used paginators - will look into it.

However, for those who want to stay at slightly lower-level, couldn't this be supported?

[boltronics]

Just hit the same problem with https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2

On an empty string:
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the ListObjectsV2 operation: The continuation token provided is incorrect

On None:
botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid type for parameter ContinuationToken, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

[boltronics]

Even setting the key prefix to None triggers an exception.
botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid type for parameter Prefix, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

[JordonPhillips]

The reason why None is that there can be a difference between not passing a value and explicitly specifying None. Unfortunately we cannot change this behavior.

As John mentioned, I would highly recommend using the paginator that got merged in to handle this for you.

[tuukkamustonen]

The reason why None is that there can be a difference between not passing a value and explicitly specifying None. Unfortunately we cannot change this behavior.

Java and JavaScript SDKs accept null as NextToken - I believe they just exclude it from the request. Why should python SDK be any different?

I would highly recommend using the paginator that got merged in to handle this for you.

Yeah, the paginators work nicely! They indeed render this ticket unnecessary.

[rkiyanchuk]

@JordonPhillips Paginators are not provided for all services in boto, so it's not a generic solution.

What is the use case for being able to tell between value not being specified and explicitly set to None? Why would the lower level code care about such difference?

It is common practice in Python to use None specifically in such situations for keyword arguments, as it helps to avoid ugly overly verbose code.

It is also very confusing that some functions allow NextToken to be set to empty string (describe_instances, describe_security_groups) while others don't (describe_flow_logs).