Airflow: Rewrite Breeze in Python

[potiuk]

This is follow up and discussion space for the discussion started in https://lists.apache.org/thread.html/ra7e0629922436607de72fa3ea124aa4336df033163f4af0ebcee7b98%40%3Cdev.airflow.apache.org%3E

While I was opposing this while we were focusing on Airflow 2.0, and there are some good reasons why initially I started Breeze in Bash, I think with the current state of Airflow 2.0 betas, with Airflow 2.0 fully based on Python 3.6 and with some "stability" and "good set of features" we have in Breeze and a good level of modularisation we achieved - it's the right time to think about a rewrite.

I did not raise this subject to add a distraction on top of what is already a lot of work for 2.0, but I think having Breeze rewritten in Python could be the "one more thing" that we could do - as a community to make 2.0 experience even better, and one that can make the community even closer.

I was thinking that Breeze is perfect to be split into separate smaller pieces, describe some assumptions that we will have for its use, and turn it into a true community effort where a lot of people will contribute and where we will be able to simplify some of the stuff, and - most importantly - make more people from the community know about how our CI and development environment works and be able to solve any problems there.

Breeze (and underlying bash libraries) are crucial, to get our CI working and I am mostly the single point of contact (and failure!) when it comes to that - I would love to not be one :) and I think with most of the core committers busy with 2.0, this is also an opportunity for more of the contributors to take their part in it (and eventually earn their rank to become committers!). For the core committers, this is an extra opportunity to learn how the system works, influence its design, and possibly simplify some parts of it - even if they will be mostly focused on 2.0.

I would like to do it well - write some assumptions in a design doc, plan the work and split it into separate issues, and lead the effort - but I would love if most of the work is done by others, who would then become familiar with the whole of it.

---

THIS IS NOT YET A DESIGN PHASE. I am merely trying to see whether there is an interest in taking part in it and helping in the project. Design discussions will follow, and proper project plan will be developed for that, I am looking for people who would like to help with that and become part of the "Breeze 2" team to design and develop it.

Anyone interested - please comment here.

---

[potiuk]

And just another comment here. I would love to have a good mixture of people participating:

• core commiters
• "fresh" commiters
• frequent contributors
• casual contributors
• users who would like to contribute

[davido912]

Raising my hands up here. Would be interested to hear and help. ✋

[phraniiac]

Hi! I would like to help with this task. ✋

[Zedmor]

I would like that! sign me up

[caddac]

I'm interested.

[yangrong688]

I would like to contribute.

[mik-laj]

I'm interested too.

[potiuk]

Cool! I think the critical mass has been reached :), I am writing a short design doc to start with :)

[judoole]

I would like to contribute if I have something to offer. I do most of the grunt work of keeping Composer and local Airflow in our company running, so I hope finally my experience will come to good open source use.

[TobKed]

I'm interested as well.

[turbaszek]

[potiuk]

OK. Seems like we are close to get 9 companions very soon !

[potiuk]

Wait It is already 9!

[potiuk]

I will prepare the initial doc over weekend :)

[leahecole]

Happy to be a tester and to help with communication + devrel-y docs things once we're closer to rollout!

[potiuk]

cool! I am pushing the providers first for beta 3 and will send the docs shortly after :)

[potiuk]

still pushing :) (but getting closer) stay tuned.