Pipelines: Feature Request: upgrade from Kubeflow 1.0 to 1.1 on GCP

[Bobgy]

Kubeflow 1.1 for GCP is drastically different from Kubeflow 1.0, in-place upgrade is almost impossible.

Significant updates:

• KFP in Kubeflow 1.1 defaults to separate multi-user: https://master.kubeflow.org/docs/pipelines/multi-user/.
• Kubeflow 1.1 on GCP is deployed by blueprint, rather than kfctl.

I need some feedback about specific scenarios of your upgrade:

• Do you care about keeping existing data in the new cluster?
• How should they (existing data) be shared among with multi-user separation? Is it enough they are only accessible by SDK, but not UI?
• Are you interested in storing data in managed storages (GCS + Cloud SQL) to make future upgrades easier?
• Will you prefer managed storages as the default option?

[munagekar]

In-place upgrade is almost impossible.

It would be good to have this, in future releases.

Kubeflow 1.1 on GCP is deployed by blueprint, rather than kfctl.

Would it possible in future to deploy future version of kubeflow without the management cluster. In our organization we use terraform for infrastructure management. To test out Kubeflow 1.1 we used management cluster to spin up the cluster and then shut it down. It would nice if Management Cluster was an optional opt-in feature, instead of pre-requisite.

Do you care about keeping existing data in the new cluster?

No

How should they be shared among with multi-user separation? Is it enough they are only accessible by SDK, but not UI?

Both UI and SDK would be required. It would be confusing to have it only with SDK.

Are you interested in storing data in managed storages (GCS + Cloud SQL) to make future upgrades easier?

Usage of Managed storages seems like a good idea.

[Bobgy]

In-place upgrade is almost impossible.

It would be good to have this, in future releases.

Yes, this is already supported for future upgrades (from 1.1 to any later version).

Kubeflow 1.1 on GCP is deployed by blueprint, rather than kfctl.

Would it possible in future to deploy future version of kubeflow without the management cluster. In our organization we use terraform for infrastructure management. To test out Kubeflow 1.1 we used management cluster to spin up the cluster and then shut it down. It would nice if Management Cluster was an optional opt-in feature, instead of pre-requisite.

It's not possible now, but it depends on what https://cloud.google.com/config-connector/docs/overview and provide in the futre.
We have heard other users who prefer terraform, so it may be possible community can maintain a terraform version of required cloud resources.

How should they be shared among with multi-user separation? Is it enough they are only accessible by SDK, but not UI?

Both UI and SDK would be required. It would be confusing to have it only with SDK.

Sorry, my question was a little confusing. I was only asking for the migrated existing data. Of course both UI and SDK will be available for data with multi-user separation.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[slenky]

If someone ended up googling that issue, here is the list of SQL commands I've executed on Kubeflow mysql pod and got it working.

Use case: migrating from Kubeflow 1.0.2 to Kubeflow 1.2.0 with multi-user.

Env info: Kubeflow 1.2.0, Azure, KFP 1.0.4

CREATE DATABASE `cachedb`;

USE `cachedb`;
DROP TABLE IF EXISTS `execution_caches`;
CREATE TABLE `execution_caches` (
  `ID` bigint(20) NOT NULL AUTO_INCREMENT,
  `ExecutionCacheKey` varchar(255) NOT NULL,
  `ExecutionTemplate` longtext NOT NULL,
  `ExecutionOutput` longtext,
  `MaxCacheStaleness` bigint(20) NOT NULL,
  `StartedAtInSec` bigint(20) NOT NULL,
  `EndedAtInSec` bigint(20) NOT NULL,
  PRIMARY KEY (`ID`),
  KEY `idx_cache_key` (`ExecutionCacheKey`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

USE `mlpipeline`;

ALTER TABLE `mlpipeline`.`experiments`
ADD COLUMN `Namespace` VARCHAR(255) NOT NULL AFTER `CreatedAtInSec`,
ADD COLUMN `StorageState` VARCHAR(45) NOT NULL AFTER `Namespace`;

ALTER TABLE `mlpipeline`.`jobs`
ADD COLUMN `ServiceAccount` VARCHAR(255) NOT NULL AFTER `Namespace`;

ALTER TABLE `mlpipeline`.`run_details`
ADD COLUMN `ServiceAccount` VARCHAR(255) NOT NULL AFTER `Namespace`;

UPDATE mlpipeline.experiments SET `Namespace`='kubeflow-old';
UPDATE mlpipeline.run_details SET `Namespace`='kubeflow-old';

ALTER TABLE `mlpipeline`.`experiments`
DROP INDEX `Name` ,
ADD UNIQUE INDEX `idx_name_namespace` (`Name`, `Namespace`);

You also need to create a kubeflow-old profile and give your users permissions to that namespace then.

[Bobgy]

That's awesome, as a reference, there's a more simplified version if upgrading from later version of KFP to multi-user KFP.

https://gist.github.com/chensun/a57844eacdae6565bb88501a47647bac

[Bobgy]

We do not plan to provide one-click/one-script upgrade if user request is low. From what I heard, most users ended up deploying a new Kubeflow cluster, because KFP is not the only one that needs complex upgrade. Therefore, I'm going to close the issue.