Boto3: [Feedback] API is not really pythonic

[autrilla]

While I do understand the desire to provide methods very similar (if not analogous) to the AWS CLI, I really feel like working with boto3 does not really improve on the AWS CLI. From my point of view, boto3 should not only be a wrapper, but also an adapter to make the CLI something desirable to work with from python. For instance, instead of creating an S3 client with boto3.client('s3'), hide this and use a S3Client class. Instead of managing multi-part upload with the upload id directly, it would be more 'pythonic' to have a MultiPartUpload class, and have the corresponding methods take or return an instance of this class. What is the point of having the Client do operations such as get_bucket_policy(Bucket='string')? Wouldn't it be much better (and intuitive) to have a getter for bucket_policy on the Bucket class? Also, having helper methods would help a lot. Something like Bucket.upload_file(file, path), or along those lines. The point isn't precisely the examples I have pointed out here, the point is the library is hard to work with, it is essentially the same as the CLI. Please note I do not mean any disrespect and this is just, at least from my point of view, constructive criticism.

[scriptsrc]

I love boto3, but I agree with @autrilla.

+1

[kyleknap]

@autrilla

Thank you for the feedback. We truly appreciate any feedback we can get on the project.

It is true that the Boto3 and the CLI share similar methods. This is because both of these projects are separate higher level layers to a lower level library that we maintain: botocore. Botocore is lower level in the sense that it methods are a near direct mapping to the various AWS API's . There are very few customizations added to these methods, as they are added in the higher level libraries/tools such as Boto3 and the CLI.

I do have one question. Have you had a chance to look at the resource interface of boto3?

So boto3 has two different interfaces to interact with AWS's APIs. There is the client interface that you mentioned. The client interface is pretty much the same interface used by botocore, where the methods are nearly direct mappings to the APIs. The main purpose of this interface is to allow users to make more fine-tuned requests to the APIs (i.e. you know exactly what parameters you want to a send in a request and you just want a dictionary back representing what was in the response).

The other interface, which sounds like the one you want is, the resource interface. The purpose of this interface is to make interaction more pythonic and object oriented when working with AWS. When you use resources, you are working with auto-generated classes that live onto of the client interface.

To hit some of the points you mentioned, the resource classes have getters (they are labeled as identifiers and attributes) that will actually make the underlying call for you to retrieve the necessary data. Resources also can create and return other resources to use from a method call, instead of just a dictionary. So for example an s3 Object resource object has the method initiate_multipart_upload that will return to you a s3 MultipartUpload resource object to work with a multipart upload. Then on top of these resources we have customizations that are built on top of AWS APIs that requires stringing together possiblly multiple API calls. For example, we recently added an upload_file and download_file method (note that this is only available from the client for now): https://boto3.readthedocs.org/en/latest/reference/customizations/s3.html

Please let us know if you have any follow up questions/comments/feedback/feature requests. We love getting feedback on how to improve this project.

[scriptsrc]

Awesome writeup @kyleknap. I learned a thing or two.

[kyleknap]

Closing issue. Please reopen if you have any more questions or comments.

[brianbruggeman]

So something I was disappointed with recently.

$ python

x = boto3.resource('s3')
x
s3.ServiceResource
import s3
ImportError: ...
type(x)
boto3.resources.factory.s3.ServiceResource
isinstance(s3, boto3.resources.factory.s3.ServiceResource)
AttributeError: 'module' object has no attribute 's3'

Every python API should be able to answer these questions:
"What exactly is this object I'm using?"
"Where can I find the exact code for this object?"
"How can I perform some basic operations on this object?"

[Constantin07]

@autrilla Completely agree with you!
+1

[tgolly]

type(x)
boto3.resources.factory.s3.ServiceResource
isinstance(s3, boto3.resources.factory.s3.ServiceResource)
AttributeError: 'module' object has no attribute 's3'

I'm having this exact problem, isinstance is essentially unusable. I'm not sure how I can accurately identify which type of object I'm working with.

Is there any way to import the classes created by the factory so they can be identified with isinstance()?

[robertglen]

+1, not being able to use isinstance is a significant issue

[AdilHindistan]

Adding my voice and +1. I understand that issue is probably happening b/c of the dynamic nature of producing object but isinstance should work

[fbidu]

Hello, all!

I got here while researching for more Pythonic ways to interact with AWS. Before that, I started writing a small OO oriented wrapper for ECS resources: gist. After reading @kyleknap answer, I thought that what I was doing was redundant and went looking for a Resource for ECS but found out that there isn't one.

I wonder if what I doing is something like the Resources are supposed to do and if I could help implement an ECS resource in boto3 that provides this interface

[brianroco]

This works for me:

import boto3
x = boto3.resource('ec2')
isinstance(x, boto3.resources.base.ServiceResource)
True
help(x)
Help on ec2.ServiceResource in module boto3.resources.factory object:

class ec2.ServiceResource(boto3.resources.base.ServiceResource)

| Method resolution order:
| ec2.ServiceResource
| boto3.resources.base.ServiceResource
| __builtin__.object

[brianbruggeman]

@brianroco

The problem is not that you cannot find a way to capture an isinstance. The problem is that the class is actually not boto3.resources.base.ServiceResource but in fact boto3.resources.factory.s3.ServiceResource

And while I can respect that we're using some sort of dynamic provisioning here. It's difficult to actually connect the dots unless you become quite familiar with how boto operates underneath.

Sad that this issue is considered closed.