Boto3: [consent request] Type annotations for boto3

[vemel]

Hello @swetashre,

I added boto3-stubs that was auto-generated from boto3 source code. Is it okay to keep it public?

It is recognized by mypy as an interface module for boto3.

Regards

[vemel]

Any update?

[swetashre]

@vemel - Sorry for the late reply. boto3-stubs is not maintained by AWS. So i don't know much about it.

We have a working implementation of of boto3 type stubs for the same purpose and its located here: https://github.com/boto/botostubs. Its not production ready yet but you can see our approach at least and how it should work. It still needs some work and tests.

[vemel]

Thank you for your response!

boto3-stubs is maintained by me now.

More or less the idea is the same - I parse botocore shapes to build argument and return type annotations and I also use inspect with docstring parsing to annotate dynamic methods added by boto3 - this part is missing in botogen as I see.

What these projects have in common:

• Type annotations are extracted from botocore shapes
• Output is formatted by black for readability
• Stubs builder for custom builds

Main differences from botogen:

• TypedDict argument and return types with proper required and optional fields
• Recursive TypedDict annotations support (required for dynamodb service)
• Generated type definitions for using in user's code, e.g., to annotate a function that accepts raw AWS response or a part of it, but still have type checking
• Literal argument type annotations for IDE auto-complete and type checking
• Annotations for auto-generated boto.client(...).Exceptions structure for IDE auto-complete and type checking
• Support for methods injected by boto3
• Patches for method signatures changed by boto3 (used in s3 and ec2 service and probably some others). botocore shapes are changed by boto3 sometimes, so shapes give you incorrect method signatures.
• Links to official boto3 documentation from every class and method docstring
• Modularity, so you can install type annotations for services you use from PyPI instead of creating custom builds
• Working solituion for boto3.client and boto3.resource function overloads - basically I generate overloads only for services you have installed. Creating overloads for all services easily kills mypy and PyCharm due to high RAM usage.
• Works on Python 3.6, 3.7 and 3.8, while botogen is missing 3.6 support
• Automated builds for new boto3 versions and upload to PyPI, so end user can install them with pip or poetry for his boto3 version

I would like to help the project as my implementation works and covers all boto3 services and boto3. botocore stubs are not included, it was not one of my goals.

Let me know if you need any additional info about the project, because I guess it might be useful.

Regards, Vlad.

[vemel]

Sorry for pinging you again, @swetashre . Please let me know if there are any updates.

[drorata]

I'm joining the question... What's the status with this?

[vemel]

Still, no updates for https://github.com/boto/botostubs , while boto3-stubs is production-ready.

@swetashre let me know if we can somehow make boto3-stubs officially supported. There are few things that can be improved with the help of boto3 team:

• Proper default argument values, because passing None is not handled the same as not passing argument at all, see https://github.com/boto/botocore/issues/2075
• Some inconsistencies between streaming and non-streaming shapes (return type is always a stream while arguments accept both stream and bytes - mostly affects s3 service)
• Types for all modules other than __init__.py and session.py can be merged to boto3 source code. I annotated all boto3 modules but probably missing some attribute annotations.
• Probably boto3 can even add keyword arguments from shapes to methods instead of hiding them behind *args, **kwargs, this would make boto3 more user-friendly.

Some additional features could be to provide typing-friendly methods to avoid overloading literals which are not currently supported correctly by any IDE. To avoid breaking changes, old methods should still exist but return botocore types

s3_client = boto3.s3_client(...) # new: returns S3Client
s3_client = boto3.client("s3", ...) # old: returns botocore.BaseClient

s3_resource = boto3.s3_resource(...) # new: returns S3ServiceResource
s3_resource = boto3.resource("s3", ...) # old: returns botocore.ServiceResource

s3_client.get_bucket_exists_waiter() # new: returns BucketExistsWaiter
s3_client.get_waiter("bucket_exists") # old: returns botocore.Waiter

Regards, Vlad.

[matei-dragu]

Hey @swetashre - any updates on this issue? Radio silence for a period of almost one year is a bit concerning.