Screwdriver: Ability to use multiple scms

[catto]

It would be nice if we could use multiple scms such as GitHub.com and GitHub Enterprise.
Do you have any plan to achieve it?

[d2lam]

@catto This should be already working. To use GHE, you need to add SCM_GITHUB_GHE_HOST environment variable. Does this documentation help: http://screwdriver.cd/getting-started/plugins/#source-control. You can also write another scm plugin, and change SCM_PLUGIN to use it.

We need additional work to do checking out from GHE correctly.

[stjohnjohnson]

@d2lam I think @catto is looking for the ability to support multiple SCMs at one time.

[catto]

Ah, yes, I meant to say about the ability to use the scms at one time like below

scms:
    - github.com:
      plugin: github
      oauthClientId: githubclientid
      oauthClientSecret: githubclientsecret
      secret: githubsecret
    - ghe:
      plugin: github
      oauthClientId: gheclientid
      oauthClientSecret: gheclientsecret
      gheHost: github.mysite.org
      secret: ghesecret

[stjohnjohnson]

Right now our recommendation is going to be to run multiple clusters. However, we'd be open to hear any ideas you have around how to make this kind of change (as it would touch all parts of SD).

[catto]

@stjohnjohnson Thank you, I got it. I'll think about it and let you know if I find the way how to achieve it.

[catto]

I had thought about this feature, and I came up with one idea.
It uses an authentication context and switches the context completely between multiple scms.

The abstract is as follows.

• Cluster admin configures multiple scms to their cluster, like this.
• Screwdriver shows multiple login buttons for each scm in the login page.
• When an authentication is completed, Screwdriver issues a session cookie for a user with an auth context.
  
  
  
  • This can be "github.com:catto" or "my-ghe:catto"
  
  
  • It is better to show a user name with an auth context in the header.
  
  
• If a user wants to use other scm, the user need to re-login to the desired scm.

This method would require major changes to the user model, but it seems that other ones don't need to do so much.

[stjohnjohnson]

I like this @catto. How do we want to detect which scm_url works for which scm module?

[catto]

@stjohnjohnson How about this Façade-like design?

• Implement and expose a function such as canHandle which can determine if it can handle the specified scm_url or hostname, in each scm module. Or, implement a detector directly into scm-base.
  
  
  
  • Every functions in scm-base has scmUri, checkoutUrl or hostname param, so we can switch the module using hostname.
  
  
• scm-base consumes scm config array from around here, and instanciate modules using the config.
• Each function in scm-base get the desired module by calling canHandle in each module or the detector, and calls the concrete function in it.

[catto]

I talked with @stjohnjohnson about this, and we decided to implement it.
Please let me know if you have any concerns.

Rules

• Cluster admins can specify two or more scm modules at the same time
• Users can login to specified scm
• Pipelines and users should be discriminable by a scm context
• User can handle their pipeline if scmContext matches

First pass

Change scm modules, api and ui to support multiple scm config and module at the same time.
We would support both current scm and new scms config. Use "default" context for backward compatibility if a name for scm config is not set.
User can see only the first scm via UI at this time and can use Screwdriver as they have so far.

The new scms config format is as below:

scms:
    - name: my-git
      plugin: github
      config: 
        oauthClientId: foo
        oauthClientSecret: bar
        ...
    - name: super-gitlab
      plugin: gitlab
      config:
        ...

Changes:

• [x] data-schema
• [x] models
• [x] scm modules (create scm-router like executor-router?)
• [x] api
• [x] ui
• [x] cucumber feature file

Second pass

Change UI to provide multiple scm environment to users. This might include an adjustment of the
UI design.
Such as

• Show username with scmContext in the header
  
  
  
  • e.g. my-git:catto, github:catto
  
  
• Show multiple login buttons for each scm
• Show scmContext for each search result of pipelines
  
  
  
  • e.g. my-git:catto/models, github:screwdriver-cd/models
  
  

[nkatzman]

Would love to have this feature 👍 👍

[s-yoshika]

Details for first pass

We dug the details for the first pass.

data-schema and model

We'll add scmContext for the schema of user and pipeline.
scmContext represents the scm which user or pipeline are belonging.

scm modules

We'll create scm-router (like executor-router) which deregates the scm modules and selects appropriate module in every builds.
To select a correct scm-module, scm-router call canHandleUrl function in each scm module.
canHandleUrl can determine if the scm module can handle the specified scm_url or hostname by itself.

And also every scm modules should return the itself context name.
The context name represents the hostname of the server which the scm service is hosted.
Because of this, scm modules should also have getContextName function.

Summary:

• Create new scm-module: scm-router module
• Add canHandleUrl(scmUrl), getContextName(scmUrl) and getDisplayName(scmContext) funcitons to all scm modules

API

authorization

The users want to select scm context when they log-in. To do this, we have to change the API path of login.

Before: /auth/login/{web?}
After: /auth/login/{scmContext?}/{web?}

{scmContext} parameter specifies the scm by which user want to log-in.

handling models

The user and pipeline models will have scmContext as new field, so API should handle the models correctly.

settings

Also cluster settings will change. To support multiple SCMs, we add new setting scms like bellow.

scms:
    - displayName: my-git
      plugin: github
      config: 
        oauthClientId: foo
        oauthClientSecret: bar
        ...
    - displayName: super-gitlab
      plugin: gitlab
      config:
        ...

scms is array of scm settings, and each setting must have displayName.
displayName is like the nickname for the scm and is displayed as username(e.g. my-git:catto) and pipeline name(e.g. my-git:catto/models) in UI.

DB migration plan

This is a breaking change, because the existing users have no scmContext, but the new users from this upgrade have scmContext.It is difficult to ensure the compatibility between both users, so we need to migrate the database appropriately.

Currently, screwdriver-datastore-sequelize seems not to add a new column to existing tables automatically (#633) .Due to this, we have to add scmContext column to users table, and pipelines table manually.

ALTER TABLE `users` ADD COLUMN `scmContext` VARCHAR(128) NOT NULL DEFAULT “<the scm hostname>”;
ALTER TABLE `pipelines` ADD COLUMN `scmContext` VARCHAR(128) NOT NULL DEFAULT “<the scm hostname>”;

If DEFAULT attribute is set to column, the existing rows are also set with the default value.

---

We'll update this if something is changed.
Please let me know if there is anything to worry about.

[s-yoshika]

This is a breaking change, because the existing users have no scmContext, but the new users from this upgrade have scmContext.It is difficult to ensure the compatibility between both users, so we need to migrate the database appropriately.

Currently, screwdriver-datastore-sequelize seems not to add a new column to existing tables automatically (#633) .Due to this, we have to add scmContext column to users table, and pipelines table manually.

ALTER TABLE `users` ADD COLUMN `scmContext` VARCHAR(128) NOT NULL DEFAULT “<the scm hostname>”;
ALTER TABLE `pipelines` ADD COLUMN `scmContext` VARCHAR(128) NOT NULL DEFAULT “<the scm hostname>”;

If DEFAULT attribute is set to column, the existing rows are also set with the default value.

[s-yoshika]

SUMMARY 7/12/17

API

Users want to select a scm to log-in, so the API such as /auth/contexts should return the list of scmContext and displayName.
The response schema of this API:

[
  {"context1" : "displayName1" },
  {"context2" : "displayName2" },
   ...
]

scm modules

On the other hands, scm-router handles multiple scm modules internally and each scm module handles single scm. All modules inherit from scm-base, so it is better to use the same function signature.
Because of these, we have decided to change the function name from getScmContext to getScmContexts and implement it in form of "() => [strings]".
router.getScmContexts may return ["githubContext", bitbucketContext"] and github.getScmContext may return ["githubContext"]

The payload in webhook is different for each scm. We'll add canHandleWebhook ((headers, payload) => boolean) function to each scm module so that scm-router can determine which scm module to use.

settings

We move each displayName settings into scms[].config. so latest scms setting is like bellow.

scms:
    - plugin: github
      config: 
        displayName: github.com
    - plugin: github
      config:
        displayName: super-github

[d2lam]

@s-yoshika I merged the PR for data-schema and modified the tables to have the scmContext and default value to github.com for users and pipelines (beta & prod):

ALTER TABLE "users" ADD COLUMN scmContext VARCHAR(128) NOT NULL DEFAULT 'github.com'

The API build to pull in the new schema is failing right now. I will take a look tomorrow morning.

[catto]

The build failed because of new version of data-schema. It needs new parameter schema for model functions.
Let's revert the changes for data-schema, develop components using current (multiple scm supported) data-schema, and finally re-revert data-schema.

[catto]

Summary 2017/07/25

We tried to integrate it and checked if it works or not. Most features work well except the login route.
Hapi doesn't allow registering duplicated strategy, so we are going to change the implement of the login route as below.

• /auth/login/{web?} just redirects to /auth/login/$defaultContext/{web?} with no auth.
• /auth/login/{$scmContext}/{web?} handles oauth as is.

We also found a blocker around hapi auth strategy: https://github.com/hapijs/bell/issues/59
I'll set cookie key to all oauth strategy configs and test it again.

Please wait a little longer until everything goes well.

[wahapo]

In the settings, it is easier to not support both scm and scms, but is it necessary to support both?
https://github.com/screwdriver-cd/scm-router/pull/1#discussion_r129924453

[catto]

I've tested the changes locally, and basic features work well. I haven't tested all the APIs such as collections.
But I found an issue that UI shows an error with a message like "Assertion Failed: You cannot update the id index of an InternalModel once set. Attempted to update 14." when I start a new build.
API returns "201 Created" and the build starts normally though. I'm investigating why the error shown.

The support of both scm and scms is an initial idea of the first pass which keeps compatibility: https://github.com/screwdriver-cd/screwdriver/issues/365#issuecomment-311012335

I think there is no problem that we support only scms if we can change the scm config when upgrade.

[wahapo]

ok, I removed support of scm in scm-router.

[d2lam]

Update August 10, 2017
Changes are rolled out to production. We tried plugging them in and it works so far. However, we still need to add UI change in order to start using multiple SCMs (ability to login to multiple SCMs). Currently, nothing will break since we are only using 1 SCM.

[tkyi]

For a better user experience (until we work on refining the user model), there are a couple more things that could be done with multiple SCMs in the UI.

• [ ] Account notification
  When a user hasn't visited Screwdriver in a while, they may have forgotten which account they are logged into. They should see a notification telling them which SCM they are logged into. There should be an Accounts link. When clicked, it will display a menu dropdown for the user's Profile (covered in next bullet).

  • [x] (_Edit_)
    Forgot to include that the Collections and Search pages should show which SCM account the pipeline belongs to as well.

• [x] SCM accounts info (on user dropdown)
  When a user clicks on Accounts link from the above notification or clicks on the dropdown Profile icon from the menu, all possible SCM accounts should be displayed. The account the user is logged into should have an "Active" tag. If the user clicks on a different SCM, it should log the user out of their current account and into the chosen SCM.

• [x] Repo SCM info (on pipeline page)
  It's hard to tell from a pipeline page in the Screwdriver UI which SCM the pipeline/repository belongs to. Since a user cannot interact with a pipeline from an SCM they are not currently logged into, it would be helpful to display the SCM information somewhere for the pipeline. This information should be displayed next to the branch name at the top of the page.
  Note: this includes the mock for the new pipeline page, you can ignore everything except the SCM info.

[tkyi]

Blog post: http://blog.screwdriver.cd/post/167244393257/using-multiple-scms-in-a-single-screwdrivercd
Documentation: https://docs.screwdriver.cd/cluster-management/configure-api under SCM_SETTINGS