Screwdriver: SCM Branch-specific jobs

[yuichi10]

Context

The issue of #723 was closed but SCM Branch-specific jobs workflow is not implemented yet.
The workflow will make ease to use git flow.

Objective

Allow to specify branch to commit trigger.

Related links

Documentation: https://github.com/screwdriver-cd/screwdriver/blob/master/design/workflow.md#scm-branch-specific-jobs
Related to: #723

[yuichi10]

I have split up the work into these tasks for now:

• [x] add function to get branch list from scm api.
• [x] search pipelines which you need to trigger.
• [x] fix regex TRIGGER to allow regex characters at data-schema.
• [x] use edge.src as a regex and check the trigger does match the regex or not at workflow-parser.
• [x] if startFrom is ~commit, search jobs by trigger of ~commit and ~commit:{branch name} at models.
• [x] udpate workflow documentation

2018/06/12

• [x] change the clone branch

2018/09/25
The remaining tasks are

• [x] Fix #1241
• [ ] Implement feature test

[yuichi10]

To get pipelines which you start jobs, I will add function to scm repositories which get branches from scms api.

The flow is as below

1. get branches by the new function
2. create scmUris by using the branch list
3. search pipelines from the scmUris and you get the pipelines which is related to the repository you using
4. search these pipelines which have the trigger to start by using eventFactory.getJobsFromTrigger.
5. Then you will get the pipelines which we have to start jobs.

[jithine]

@yuichi10 What's the use case for branch specific workflows ? Can't user specify branch while creating pipeline ?

[yuichi10]

Yes, user can create specify branch pipeline. But, if user can manage multi branch by one pipeline, it makes easy to manage project by git-flow.
Also user can build by each feat* branch without create new pipeline.

[yuichi10]

I want to verify the specifications about SCM Branch-specific jobs.

Assuming the project which have two pipelines as below.

When developer push to staging branch, the red jobs will triggered. (I think this is no problem)

And now when developer click start button on staging pipeline page, the red jobs will triggered now.

I want to make sure the behavior, when developer click start button.

1. When developer click start button at staging pipeline, ~commit and ~commit:staging are triggered.
2. When developer click start button at staging pipeline, it does not affect other pipelines (e.g. master pipeline's job are not triggered).

What do you think the behavior is best, when developer click the start button on the pipeline page?

[minz1027]

This behavior seems reasonable to me @jithin1987 ^^

[tkyi]

@yuichi10 Your proposal sounds reasonable to me.
If the developer clicks start button on the master pipeline page, it would start both ~commit and ~commit:/^.*$/, right?

So the rules would be:

1. If the user makes a commit to <branch>, all triggers that match ~commit:<branch> will be triggered, and ~commit if the pipeline is configured to <branch>.
2. If the user clicks the Start button on a pipeline, ~commit and all ~commit:<branch> that match that pipeline's <branch> will start.

[Filbird]

It seems to be that (2) makes the most intuitive sense. From a developer's perspective, when I specify ~commit:staging on a job, main, in pipeline:master, I expect main to run only when a new commit is made to the staging branch.

In (1) there is also an implicit assumption that main should be run whenever the staging branch pipeline runs (even if there is no new commit). In this case remote requires seems more appropriate.

requires:[sd@stagingPipelineId:stagingJobName]

[jithine]

@yuichi10

I want to make sure the behavior, when developer click start button.

1. When developer click start button at staging pipeline, ~commit and ~commit:staging are triggered.
2. When developer click start button at staging pipeline, it does not affect other pipelines (e.g. > master pipeline's job are not triggered).

👍 Sounds good to me.
When manually clicking Start button in UI it should affect jobs only in the current pipeline user is interacting with.

[yuichi10]

Thank you for your comments!

tkyi's explain is easy to understand!

1. If the user makes a commit to , all triggers that match ~commit: will be triggered, and ~commit if the pipeline is configured to .
2. If the user clicks the Start button on a pipeline, ~commit and all ~commit: that match that pipeline's will start.

I agree with Filbird's opinion. If developer want to run whenever not pipeline's job, they need to use remote requires.

Thank you guys again!
I implement SCM Branch-specific jobs by this policy!

[minz1027]

@yuichi10 Hi Yuichi, I noticed that we're only adding the branch trigger for ~commit, not ~pr. That means if I have ~commit:staging and I opened a PR against staging then nothing will happen right? Is it the expected behavior? Can you elaborate more on the use case for this feature? Thanks!!

[yuichi10]

@minz1027 Hi minz, thank you for your notice.
Yes now if you open PR for staging branch, nothing happen, even there is ~commit:staging trigger.
I am going to consider the use case of this feature.

[yuichi10]

@minz1027 To me, ~pr for specific branch is also necessary.

How about to use same way as ~commit specific branch.
If user want to trigger specific branch at pr, set ~pr:branchName or ~pr:/^branch.*$/ to requests.

I think we can develop commit and pr functions separately, thus after this commit specific branch is finished, develop ~pr function.

[yuichi10]

I found a bug of my implementation.

When restart a job, Screwdriver try to build in the same way as previous, and get screwdriver.yaml from the sha.
When you restart specific branch job, the sha is not belong to the pipeline branch, so if the branch have different screwdriver.yaml, the pipeline's workflow are changed.

It will be not good to refer other branch screwdriver.yaml.
But I think to fix this problem, have event model take the pipeline branch sha, and this way require Screwdriver some changes.

How do you think to resolve this problem? Are there any good way to fix this problem?

[yuichi10]

Maybe I can fix this problem by using configPipelineSha.

[tkyi]

Hm so if I understand correctly, you are saying:
If you have pipeline configured to branch master and a job configured to branch foo, the issue is when you restart the foo branch job, it uses foo screwdriver.yaml file instead of master screwdriver.yaml?

[yuichi10]

Yes! So I think need to preserve the branch master sha by using configPipelineSha.
Sorry for late fixing. We will fix this problem from next week

[catto]

@jithin1987 ~pr:branch triggers next jobs if it is finished. Currently this behavior confuses users since ~pr doesn't. I suggest we should unify them with letting ~pr:branch not to trigger next jobs. What do you think?

[jithine]

@catto yes, we should have the same behavior. We do get some asks about relaxing this restriction, however in my opinion it should be based on a configuration. For now, lets prevent the further job triggers.

[marian-r]

@jithin1987 @catto having it based on SD configuration would mean letting PR jobs to trigger next jobs instead of preventing it...as outlined in #1010 .

[jithine]

having it based on SD configuration would mean letting PR jobs to trigger next jobs instead of preventing it...as outlined in #1010 .

yes, doing it behind SD config will solve it.

[marian-r]

@jithin1987 cool, is that feature on the horizon?