Pipeline: Replace Pipeline linear execution with graph based execution

[tejal29]

Expected Behavior

Tasks in pipeline can run in parallel if:

• The inputs on which they run are available (via from field)
• The previous tasks (which should be indicated via runAfter - not yet in API, see description of runAfter in design doc (_note doc visible to members of [email protected]_)

Actual Behavior

Right now, all Tasks execute linearly in the order they are defined in in the Pipeline

Steps to Reproduce the Problem

Parallel case:

1. Create Pipeline with 2 tasks which do not depend on each other
2. Create a PipelineRun for this task
3. Make sure they are executed in parallel

from case:

1. Create Pipeline with 2 tasks , one which needs a resource from another, and declare them in the opposite order they should run in (i.e. put the Task the resource is from last)
2. Create a PipelineRun for this task
3. Make sure they are executed in the right order

runAfter case

1. Create Pipeline with 2 tasks , one which is supposed to runAfter another and declare them in the opposite order they should run in (i.e. put the Task that should berunAfter last)
2. Create a PipelineRun for this task
3. Make sure they are executed in the right order

And combining these cases!

Additional Info

[bobcatfish]

@tejal29 could you add a bit more detail about what the expected behaviour is after this task is implemented when you DO need tasks to execute linearly? e.g. Task A needs to execute before Task B (I'm assuming this is covered by the functionality @pivotal-nader-ziada added in #160?)

[tejal29]

@bobcatfish, in #160, we did not get rid of linear execution. I had a chat with @pivotal-nader-ziada and we decided to remove the logic in another PR.

160, does address task dependency by waiting on Task A to complete if Task B is depending on it. will add more on that in description.

[bobcatfish]

@bobcatfish, in #160, we did not get rid of linear execution

I just want to make sure that when we implement this task, we still _support_ linear execution. This task as it's currently worded sounds like we will only be able to run tasks in parallel, which would be a step backward from where we are now.

We don't want to get _rid_ of linear execution, we want to support more use cases

[nader-ziada]

The plan is to use passedConstraint to set dependency (order) of tasks, if there is no dependency, why would you want them to run in sequence?

[tejal29]

+1 to what @pivotal-nader-ziada said. It respects passedConstraint to determine if a task can run.

[bobcatfish]

The plan is to use passedConstraint to set dependency (order) of tasks, if there is no dependency, why would you want them to run in sequence?

Sometimes you want to run Tasks in sequence, and you would do this with passedConstraints.

I'm just saying that saying we are "removing" linear execution is very confusing if you don't already know intimately how the controller is implemented. But this is a moot point since @tejal29 is the one implementing it and knows what this means :)

[nader-ziada]

@bobcatfish if the tasks are not related at all, is there a need to run them in sequence? if yes, we might need a task to identify that in the yaml

[nader-ziada]

concourse has an optional serial flag to indicate you want the tasks in the pipeline to run in sequence, maybe we can add something similar to the pipeline type?

[bobcatfish]

I think we are talking about 2 different things:

1. Being able to express the order your Tasks execute in via the order they appear in a Pipeline
2. Having the ability to express a sequence of Tasks that execute in a desired order

I'm not saying we need (1), as far as I'm concerned this is just a syntax thing, not really a feature.

After this Task, we'll still have (2), which is the only thing I'm saying is important. I want to add features, not remove them.

[tejal29]

@bobcatfish i think you got confused by the title "Remove linear Execution". I will change it to "Remove Default Linear Pipeline Execution"

[bobcatfish]

Thanks @tejal29 I think that clears it up! I'm going to move some of this discussion into #137 re. passedConstraints in general.

[bobcatfish]

The initial implementation of the DAG is available in https://github.com/knative/build-pipeline/blob/master/pkg/reconciler/v1alpha1/pipeline/resources/dag.go - the next step will be to update the Pipeline to actually use this.

[bobcatfish]

I haven't gotten to this in ages so I'm taking myself off of it - the one thing I did tho was create an end to end test for the functionality which I'll try to push in a branch if I ever get around to it.

If someone reading this wants to work on it, ping me if you'd like to see the end to end test (np if you want to start fresh).

Note that in #320 we talked about also adding runAfter functionality in addition to providedBy, we might want to implement that here too.

[vdemeester]

@bobcatfish :wave: #always-interested

[bobcatfish]

hahaha o really @vdemeester , v exciting!!!! ❤️

[bobcatfish]

Updated description of Task to include runAfter from #320 :D

[bobcatfish]

Okay @vdemeester my test is at https://github.com/bobcatfish/build-pipeline/commit/39f7664d97b470dde8c7451398ca4f161df3c245 but it's pretty rough and there are a few caveats :D

(ps should we assign this one to you? 😜 )

[vdemeester]

/assign

[afrittoli]

It might make sense to limit the maximum number of parallel tasks to be executed at any point in time for one pipeline.