Airflow: Describe how pip and versioning interacts with the provider versions.

[potiuk]

We shoudl describe how the whole provider's split plays in the Airflow 2.0 case. This is largely following standard PIP dependency behaviour known from all python projects however there are certain scenarios/gotchas to explain to the users, so that they can know how to do certain actions:

• [x] explain how extras are supposed to be used
• [x] explain how to upgrade to latest versions of the providers
• [x] explain how to downgrade providers
• [x] explain cross-dependencies between providers
• [ ] explain how to report problems with the providers (tool to easily see provider version)

[potiuk]

cc: @vikramkoka since you have the customers that you talked to - if there are any topics to be added - feel free to comment here and see with your customers if what I proposed is answering their questions.

Everyone; I scheduled it for beta3 as this is the moment where we will be able to test all the scenarios. We wil have at least two releases in PyPI and then we will be able to make sure that whatever we explain, actually works.

In the meantime, I will add a simple placeholder in beta 1 (#11880) explaining some basics and that information about those topics above is coming.

[vikramkoka]

@potiuk
Here are the questions that I have seen come up and please let me know if I should be adding these to a document some place instead of here.

Q. When upgrading to a new Airflow version such as 2.0, but possibly 2.0.1 and beyond, is the best practice to also upgrade provider packages at the same time?

Q. I have an Airflow version (1.10.12) running and it is stable. However, because of a Cloud provider change, I would like to upgrade the provider package. If I don't need to upgrade the Airflow version anymore, how do I know that this provider version is compatible with my Airflow version?

Q. I have an older version of my provider package which we have lightly customized and is working fine with my MSSQL installation. I am upgrading my Airflow version. Do I need to upgrade my provider, or can I keep it as it is.

I hope this helps, will be happy to clarify and add more as I encounter them.

[potiuk]

Cool! Superb questions :) really helpful!

[petedejoy]

One thing I'd add is the inverse of the second question that @vikramkoka posed above- if I'd like to upgrade Airflow, how can I validate that the provider I'm installing and using in my DAG will not break? What does that experience look like both from the user perspective and from the builder/maintainer perspective?

[potiuk]

Cool. I will try to prepare small "starting point" description PR that
might be a good starting point that we will iterate over during betas.

On Mon, Nov 9, 2020 at 5:08 AM Pete DeJoy [e-mail] wrote:

One thing I'd add is the inverse of the second question that @vikramkoka
https://github.com/vikramkoka posed above- if I'd like to upgrade
Airflow, how can I validate that the provider I'm installing will not
break? What does that experience look like both from the user perspective
and from the builder/maintainer perspective?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/airflow/issues/11879#issuecomment-723742005,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAERMI23GJAXZLDG6NVYUJTSO5TMJANCNFSM4TAP2IUA
.

--

Jarek Potiuk
Polidea https://www.polidea.com/ | Principal Software Engineer

M: [phone] <[phone]>
[image: Polidea] https://www.polidea.com/

[potiuk]

Hello everyone,

I prepared the first version of the documentation update including Provider
Packages and it's dependencies. It addresses (I hope) the questions pointed
out by Vikram and Pete,

I invite everyone interested in the discussion in the PR:

https://github.com/apache/airflow/pull/12203

Suggestions on re-wording are most appreciated, I am not a native speaker,
so grammar, wording etc. are equally welcome as a concrete proposal on what
to include, what questions/answers to provide, also I look forward to
phrase it in a way that will be nice and helpful, and at the same time, it
will leave enough freedom and enough information for the customers using
Airflow to make their own decisions on how they are testing and validating
any upgrades/downgrades.

I think we should stress (and this is what I tried to show in the PR) that
Provider Packages provide the nice feature that each business /user can
incrementally update those providers as they wish, where we as community
aim to provide backwards compatibility based on the SemVer versioning
principles. We are not able to tell our users how they are going to
upgrade, but we can tell them what freedom they have.

If you think that - as a community - we should approach it differently -
happy to discuss it in the PR and any proposals there are welcome.

J.

On Mon, Nov 9, 2020 at 11:34 AM Jarek Potiuk [e-mail]
wrote:

Cool. I will try to prepare small "starting point" description PR that
might be a good starting point that we will iterate over during betas.

On Mon, Nov 9, 2020 at 5:08 AM Pete DeJoy [e-mail]
wrote:

One thing I'd add is the inverse of the second question that @vikramkoka
https://github.com/vikramkoka posed above- if I'd like to upgrade
Airflow, how can I validate that the provider I'm installing will not
break? What does that experience look like both from the user perspective
and from the builder/maintainer perspective?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/airflow/issues/11879#issuecomment-723742005,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAERMI23GJAXZLDG6NVYUJTSO5TMJANCNFSM4TAP2IUA
.

--

Jarek Potiuk
Polidea https://www.polidea.com/ | Principal Software Engineer

M: [phone] <[phone]>
[image: Polidea] https://www.polidea.com/

--

Jarek Potiuk
Polidea https://www.polidea.com/ | Principal Software Engineer

M: [phone] <[phone]>
[image: Polidea] https://www.polidea.com/