Luigi: Get ready for Python2 deprecation

[honnix]

@dlstadther @Tarrasch It's almost there. We will need to give people heads-up before dropping Python2 CI.

[dlstadther]

How should we handle versioning for this support deprecation? While day 0 will just be the removal of Python 2 testing, 2.7 will continue to work for a short time. Do we go ahead and major increase at that time?

[honnix]

I think I makes sense to do a major release on day 0 even there is nothing as major. It is a promise that we no longer support py2 and will start introducing py3 only features. I also think we should remove six and all the other try import catch thingy. Wdyt?

[drowoseque]

That sounds great
I would like to participate in it if you don't mind

[honnix]

@drowoseque Of course not.

[drowoseque]

I finally came with following idea about luigi.six

For each method in this module we

• replace one with raising an exception sth like SixUsageException
• eliminate this method usage from luigi
• passing tests guarantee that no method invocation is performing

after that we remove luigi.six

[drowoseque]

I think I'm done with erasing py2-support in luigi
any other ideas to do for that except removing six?
@honnix
@dlstadther

[Tarrasch]

Nice to see it finally happening!

[honnix]

Removing six and related docs then I think we are good. Thanks a lot for your great contribution @drowoseque . Good job!

Next step I'm thinking to release a few beta versions on that branch before merging to master (we should hold on till 2020 at least). What do you think, folks? We should also try to find a few beta testers, including Spotify as one of them.

[honnix]

We might as well stop supporting old versions of py3. I mean removing https://github.com/spotify/luigi/blob/master/setup.py#L115-L118

[drowoseque]

@honnix

Removing six and related docs then I think we are good. Thanks a lot for your great contribution @drowoseque . Good job!

Next step I'm thinking to release a few beta versions on that branch before merging to master (we should hold on till 2020 at least). What do you think, folks? We should also try to find a few beta testers, including Spotify as one of them.

we in cian.ru will gladly test this beta-version

[drowoseque]

@honnix
@dlstadther
@Tarrasch
could you please merge master into py2-deprecation or give me permissions for doing that?
thanks

[honnix]

@drowoseque I rebased from master. I don't have permission to make the change you requested either. I'm actually not sure whether it is doable to give write permission to a branch.

[honnix]

Label makes it easier to track all the PRs: https://github.com/spotify/luigi/pulls?utf8=%E2%9C%93&q=label%3A%22Python2+deprecation%22+

[honnix]

@dlstadther @Tarrasch PTAL https://github.com/spotify/luigi/releases/tag/untagged-cb779e4bce20f2ddd497

[honnix]

@drowoseque 3.0.0b1 is now available on pypi: https://pypi.org/project/luigi/3.0.0b1/

[dlstadther]

@honnix release notes lgtm. I am not a luigi user at my job anymore so i can't easily validate the beta release against representative pipelines.

[honnix]

@dlstadther No worries. Let's try to get more people onboard, somehow. :)

[honnix]

@drowoseque Have you been able to test 3.0.0b1 release? Here at Spotify we have started testing. So far luigid seems fine and we are working on the client lib bump.

[drowoseque]

@honnix we have new versions luigid running for 1.5 week and everything seems to be fine.
Bumped client side has been running in several pipelines for a week and no problems occurred.
MInd, that we in cian mostly run hive and spark tasks

[honnix]

3.0.0b2 released https://pypi.org/project/luigi/3.0.0b2/

[honnix]

I think we can give it two more weeks, then merge to master and release.

[honnix]

Now it has been more than 2 weeks. I think we can move on merging the PR and release 3.0.0. wdyt @drowoseque @dlstadther @Tarrasch ?

[drowoseque]

@honnix sounds good!

[dlstadther]

SGTM

[drowoseque]

@honnix
will you do that ?

[honnix]

Sorry this has been taking longer. I unfortunately cannot just do it now. We are having a discussion internally and hopefully we can sort it out very soon.

[Tarrasch]

@honnix Python3 for the win. :)

[honnix]

@Tarrasch @dlstadther PTAL at the release notes https://github.com/spotify/luigi/releases/tag/untagged-fe738e7cd4ef344c93fc and the PR to kick off the release https://github.com/spotify/luigi/pull/2953

[honnix]

3.0.0 has been shipped.