Luigi: Update S3 client to use Boto3

[jfeala]

AWS is moving from boto to boto3 for its Python SDK. Boto3 includes high-level wrappers for simple uploads and downloads from local files with S3Transfer. Besides the simpler interface, in my own experience I've seen these advantages with Boto3

• More stable downloads for big files
• Built-in support for multipart uploads
• Encryption with KMS

Happy to contribute this when I have some time over the next few months, otherwise I thought I'd put it out there for others to pick up

[themalkolm]

Just curious. Do you have any problems with current boto implementation?

[jfeala]

Those three bullet points are the problems I have with boto. The deal-breaker for me is that you can't do encryption at rest via boto using the Key Management Service (KMS), but I also get some instability with big file downloads (occasional timeouts without restart), and it's inconvenient that multipart uploads are not included and have to be patched on (as it is implemented in the Luigi S3 client).

Boto3 solved these problems for us and so now we currently only use the Luigi S3 client for its S3Target.exists() and do the file transfers manually. We'd like to get back to using the Luigi client for its convenience and atomic transfers, once it's updated.

[vlcinsky]

There are more reasons to move to boto3

• boto is not going to be maintained - number of not resolved issues is growing
• AWS gives priority to boto3
• there are long term issues (e.g. problems to work with bucket names having dots in it)

See discussion at boto issue tracker boto3 is not a replacement of boto (yet?) #3306, which, contrary to the issue title shows, that boto3 is really the way to go.

In short using Mitch Garnaat (original boto author) words: "You Can't Go Home Again"

[brianbruggeman]

boto is no longer maintained. I'm not sure this should really be a discussion point: Luigi should move to boto3.

[burdiyan]

Can't use boto because of this issue https://github.com/boto/boto/issues/2741

Just can't talk to frankfurt buckets. So sad :( Will have to implement custom boto3 implementation, hope it won't be too tricky and I'll be able to submit the PR for this.

[erikbern]

looking forward to pull requests from @jfeala @vlcinsky @brianbruggeman @burdiyan :)

[rhaarm]

Are there any updates on moving to boto3 or should I start work on this?

[erikbern]

feel free to work on it!

[buyology]

@burdiyan — did you ever solve this for Frankfurt? Tried hacking it according to the suggestion on boto/boto#2741, got basic authentication working but got a 403 Forbidden on multipart uploads

@rhaarm — would be amazing

[burdiyan]

@buyology I made a wrapper for s3 target. Consider a custom module s3.py with the following:

from luigi import s3

import settings

CLIENT = s3.S3Client(
    host="s3.eu-central-1.amazonaws.com",
    aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
    aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)


def target(path):
    return s3.S3Target(path, client=CLIENT)

So inside a task, instead of using luigi's s3 module and it's target, I use my custom s3 module and it's target function. Or you can just provide a "provisioned" client to luigi's S3Target each time you use it.

That did a trick for me.

[tedder]

Another problem with boto, per boto/boto/issues/3579, it doesn't support IAM Roles for Tasks.. in other words, container-specific IAM roles.

[dlstadther]

@tedder Feel free to submit a PR for the update 😄

[tedder]

yeah, I know :) We just had that discussion too.

[ouanixi]

I'll try and give this a go. If anyone else is working on this please ping me. We could split the workload :)

[xelibrion]

@ouanixi how is it progressing mate?

[evanfwelch]

Hey @ouanixi and @erikbern

Is anyone working on upgrading to boto3? I'm more than happy to do this when I get some time next week but don't want to duplicate others work

[dlstadther]

@evanfwelch , @ouanixi had been working on it, but his current progress (to my knowledge) is in #2149. However I'm not sure he's had much time to work on it lately.

If you have the time, feel free to work on it and reference as much as the pre-written code as you feel is necessary. All contributions are welcome!

[atharvai]

cannot use boto because of changes to time format form s3 https://github.com/boto/boto/issues/3771

[colakong]

Hi everyone,

I made a quick implementation of luigi.contrib.s3.S3Target that uses boto3 instead of the depreciated boto module. I put here.

I didn't make a pull request for it, because:

1. I haven't run tests against it, though it worked for my use-case
2. It changes the accepted modes for S3Target.open() to 'rb' or 'wb', and I didn't include changes for a custom luigi.format.Format class to automatically encode text as utf-8 bytes if the file was opened with 'r' or 'w'. This implementation of S3Target would break a lot of pipelines if they just dropped this in as a replacement.
3. I haven't included docstrings, linted it etc

If anyone wants to clean it up and submit a PR please feel free ^^.

[eyadsibai]

This is important now after GDPR for the support of KMS encryption

[meownoid]

༼ つ ◕ ◕ ༽つ UPDATE TO BOTO3 PLS ༼ つ ◕ ◕ ༽つ

[dlstadther]

resolved by https://github.com/spotify/luigi/pull/2149