Pipelines: Upgrade - Best way to update pipelines when a new release is published?

[logicbomb421]

Sorry if this isn't the best place for this, but I can't seem to find any concrete info on the matter.

Currently we use Kubeflow as a whole (deployed via the GCP Kubeflow deploy integration), however we really only make use of the Pipelines piece right now.

I'd like to be able to keep up-to-date with releases in this project, but all info I'm finding on updating tells me to upgrade Kubeflow as a whole (via kfctl, which I don't believe I have since I used the cloud deployer?). This is likely the 'correct' way to update, however I'm wondering if it's possible to just update the pipeline pieces?

From digging into what's running in my cluster, it appears all I would need to do is set the image field of all ml-pipeline-* deployments to whatever version I wish to update to, is this the case? Does this need to be coordinated with the argo images in any way? Is there a more concrete way of achieving this goal?

Please let me know if more detail is needed. Thanks!

[Ark-kun]

New SDK features almost never require any cluster upgrades.
When upgrading the cluster, changing the image names is often enough, but it's not always the case. Sometimes, new services are added or roles are changing, etc. @IronPan knows the details.

[wstian]

use another namespace instead of kubeflow , so that the previous components will not be affected by the new ones.

except the following global components, which should be renamed manually:

• CRD
• ClusterRole
• ClusterRoleBindings

finally, migration needed:

• mysql
• minio
• other configuration

[rmgogogo]

It's on our radar.

[Bobgy]

Do you deploy on GCP and needs only pipelines?
If yes, you can refer to https://www.kubeflow.org/docs/pipelines/standalone-deployment-gcp/. We support upgrading to each new release.

For kubeflow, so far it only has unreleased upgrade support for kubeflow as a whole. There's no plan to support component level upgrade yet.

[jonasdebeukelaer]

Hi, I tried the image renaming approach using the below to update from 0.2.0 to 0.4.0, but am now getting no healthy upstream when trying to load /_/pipeline/.

kubectl -n kubeflow patch deployment ml-pipeline-ui -p '{"spec":{"template":{"spec":{"containers":[{"name":"ml-pipeline-ui", "image":"gcr.io/ml-pipeline/frontend:0.4.0"}]}}}}'
# and same for:
# ml-pipeline
# ml-pipeline-ml-pipeline-visualizationserver
# ml-pipeline-persistenceagent
# ml-pipeline-scheduledworkflow
# ml-pipeline-viewer-controller-deployment

This is a new kubeflow cluster I've just spun up using kfcli so happy to just recreate it if required, but I can't figure out how to get latest pipelines installed? Is it something I should do _before_ running kfcli apply?

Thanks!

[Bobgy]

That wouldn't be enough. You also need to mimic manifest changes in KFP https://github.com/kubeflow/pipelines/commits/master/manifests/kustomize between 0.2.0 and 0.4.0.

You probably need more knowledge about KFP to be able to get there.
To troubleshoot your direct root cause this time: did you try kubectl get pod kubectl logs <the-pod-that-fails> kubectl describe pod <the-pod-that-stuck> etc and find any clues?

[Bobgy]

@jonasdebeukelaer

[jonasdebeukelaer]

@Bobgy thanks for the reply.

It seems to be working now, I did the following:

1. prep a new kubeflow instance locally using https://www.kubeflow.org/docs/gke/deploy/deploy-cli/ (don't apply, but build only)
2. replace all instances of 0.2.0 into 0.4.0 in the following subfolders in kustomize (should have created a neat recursive sed command really)
   
   
   
   • ml-pipeline
   
   
   • ml-pipeline-ml-pipeline-visualizationserver
   
   
   • ml-pipeline-persistenceagent
   
   
   • ml-pipeline-ui
   
   
   • ml-pipeline-scheduledworkflow
   
   
   • ml-pipeline-viewer-controller-deployment
   
   
3. (if upgrading) sort out the two disks: https://www.kubeflow.org/docs/pipelines/upgrade/#reinstalling-kubeflow-pipelines)
4. apply the config file

EDIT:
Pod events are not able to be fetched currently (great new feature btw!). I assume I'm missing a permission for the pipelines service account 🤷 will look into it

[Bobgy]

FYI, some recent efforts to upgrade to KFP 0.2.5 in Kubeflow 1.0.2:

If you want to upgrade manually, you can refer to manifests there: https://github.com/kubeflow/manifests/issues/984

[HassanOuda]

@jonasdebeukelaer
manifests/kustomize/base/pipeline/ml-pipeline-ui-role.yaml if you check 0.4 with 0.2.5 there is a change here with events. Not sure if this is the issue.

[jonasdebeukelaer]

@HassanOuda cheers that did the job 👍

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.