Pipelines: release: KFP 1.0.1 umbrella tracker

[jingzhang36]

To track the progress of release 1.0.1

Release instructions: https://github.com/kubeflow/pipelines/blob/master/RELEASE.md

[jingzhang36]

@chensun Do you mind cherrypicking this PR https://github.com/kubeflow/pipelines/pull/4298 into the branch release-1.0? There are some merging conflicts during the cherrypick. Therefore, we ask original author to take a look, to avoid possible incorrect merging changes. A cherrypick guide is at https://github.com/kubeflow/pipelines/blob/master/RELEASE.md#option---git-cherry-pick

[jingzhang36]

@Ark-kun Do you mind cherrypicking the following PR https://github.com/kubeflow/pipelines/pull/4320 into branch release-1.0? There are some merging conflicts during the cherrypick. Therefore, we ask original author to take a look, to avoid possible incorrect merging changes. A cherrypick guide is at https://github.com/kubeflow/pipelines/blob/master/RELEASE.md#option---git-cherry-pick

[jingzhang36]

@alexlatchford Do you mind cherrypicking your PR https://github.com/kubeflow/pipelines/pull/4218 into the release branch release-1.0 of kubeflow/pipelines repo? There are some merging conflicts during the cherrypick. Therefore, we ask original author's help to cherrypick. A cherrypick guide is at https://github.com/kubeflow/pipelines/blob/master/RELEASE.md#option---git-cherry-pick

[jingzhang36]

The remaining PRs that need to be cherrypicked is at https://github.com/kubeflow/pipelines/pulls?q=is%3Apr+label%3Acherrypick-approved+-label%3Acherrypicked+is%3Aclosed+sort%3Aupdated-asc

[Bobgy]

and FYI, if a merge conflict is too complex, we can release it next month and skip cherrypick

[jingzhang36]

@chensun Do you mind cherrypicking this PR #4298 into the branch release-1.0? There are some merging conflicts during the cherrypick. Therefore, we ask original author to take a look, to avoid possible incorrect merging changes. A cherrypick guide is at https://github.com/kubeflow/pipelines/blob/master/RELEASE.md#option---git-cherry-pick

Per offline discussion, this PR will be skipped for patch release.s

[Bobgy]

I think we only need this one: https://github.com/kubeflow/pipelines/pull/4320.
It's an important fix.

Other PRs can be waited until KFP 1.1 release

[Ark-kun]

@Ark-kun Do you mind cherrypicking the following PR #4320 into branch release-1.0? There are some merging conflicts during the cherrypick.

I've found the reason for conflicts: We've committed the https://github.com/kubeflow/pipelines/pull/4246 PR in the release branch, not master. This is a pretty bad incident. We need to be careful about the target branch for PRs.
We should probably protect the release branches and only allow cherry-picks/merges there.

[Ark-kun]

I think we only need this one: #4320.

I made a PR for it: https://github.com/kubeflow/pipelines/pull/4382

[Bobgy]

Agree with the problem, how can we enforce it?

I think github UI isn't showing strong indication where the PR goes to. Adding an extra approval process for release branches might be a way.

any other ideas?

[Ark-kun]

Maybe we can have a bot that marks or adds comments to PRs that do not go to master.

[Bobgy]

I think prow has a plugin to add a label

[jingzhang36]

The above PR fixed the setup tools error in buildCpuTrainer, but now we have a missing license error.

[Bobgy]

TODO:

• [x] add a configmap in gcp marketplace that records all supplied variables

[Bobgy]

I'll continue to drive this release.

[Bobgy]

Took a while to understand the problem.
It seems to be caused by we bumped tensorflow version in https://github.com/kubeflow/pipelines/pull/4405/files, the new tensorflow image 2.3.0 comes with pygobject installed. We'd need to install https://github.com/kubeflow/pipelines/pull/2946/files#diff-e3fa8e7715928ccd1476debc9822407bR24 for all images that need to download pygobject's source by pip install -t XXX pygobject (the command tries to compile it, so it can need additional dependencies).

However, in our case, to comply with pygobject license (LGPL), we only need to download its source code. Is there a better way that avoids pip install -t?

/cc @Ark-kun @numerology

[Bobgy]

Figured out we don't need our own scripts to comply with pypi package licenses, pip should already do that for us.
Reported in #4461

[Bobgy]

I'm a little stuck at fixing postsubmit tests: https://github.com/kubeflow/pipelines/commit/e275548996afc219918f18bab1005003effd4811

integration test is currently failing, when inspecting the test logs, I found minio server failed and restarted once (which caused many components to fail to save outputs), but I didn't find why it failed. I'm cherry picking one more PR to have a retry.

[Bobgy]

Released 1.0.1-rc.1 to verify the problems in a real deployment.
https://github.com/kubeflow/pipelines/commit/98ded41032b1ff3482290896f656c0df131ef795

[Bobgy]

xgboost sample failed with

Traceback (most recent call last):
  File "/ml/roc.py", line 29, in <module>
    from pathlib import Path
ImportError: No module named pathlib

So we need to fix component images with the fix in https://github.com/kubeflow/pipelines/pull/580

[Bobgy]

I decided to revert the offending change in this release first.

[Bobgy]

Now I'm blocked by the kubeflow_tf_serving.ipynb sample. It's failing post submit test cases.
Example error log in stackdriver

level=error msg="No Major.Minor.Patch elements found\nTo undo this simply delete directory 'tf-serving-app' and re-run ks init.\nIf the error persists, try using flag '--context' to set a different context or run ks init --help for more options"

It seems to be still using ksonnet though.

The component's base image is debian and it isn't pinned to a version, so probably something changed there: https://github.com/kubeflow/pipelines/blob/release-1.0/components/kubeflow/deployer/Dockerfile

[Bobgy]

All postsubmit tests passed! https://github.com/kubeflow/pipelines/commit/135715dbecf7a077c468bf9c3b58e45de101e991

We are good to go on the release.

[Bobgy]

/assign @PatrickXYS
Hi Patrick, can you verify KFP 1.0.1-rc.1 on AWS?
Current release-1.0 branch head is not much different from rc1, so if you can verify that, I think we can release the final 1.0.1

[PatrickXYS]

@Bobgy Thanks for reminding me, I'll go ahead to verify that. Will keep you updated

[PatrickXYS]

Hi @Bobgy I found that AWS PR is not cherry-picked into release-1.0 branch.

Create a PR for fixing it https://github.com/kubeflow/pipelines/pull/4490.

[PatrickXYS]

Confirmed that KFP 1.0.1-rc.1 works on AWS.

Can we also merge #4490, and then cut release? @Bobgy Let me know if any concerns

[Bobgy]

OK, I can continue the release

[Bobgy]

Released https://github.com/kubeflow/pipelines/releases/tag/1.0.1

[Bobgy]

There were some vulnerabilities we need to address to release it in Google Cloud marketplace.
I'll work on these in 1.0.2: https://github.com/kubeflow/pipelines/issues/4531