Pipeline: Conditional build of subproject within a monorepo depending on modified files/directories

[acim]

Use cases:

1. You have monorepo containing backend and frontend for your application. You probably want to test and build just frontend if no files inside backend directory are modified and opposite.
2. You have repo with ton of Ansible roles. You probably want to test just modified ones, not all.
3. You modified just readme.md and don't want to test/build everything.

Jenkins has something called changeset.

/kind feature

[sbwsg]

Could you elaborate a bit more here - would you want a Trigger to only fire a Pipeline on the condition that a specific subproject's files changed? A Task in a Pipeline to be executed only if a subproject had changed? Or maybe just a Step inside a Task to be run conditionally?

This can be hacked together today with some bash but it's ugly:

# inside a task
  steps:
    - name: test-subproject
      image: my-test-image-with-git
      script: |
        cd my_monorepo_checkout
        # count number of changed files in subproject
        subproject_changes=$(git log --format= -n 1 --name-only | egrep "^subproject_subdir" | wc -l)
        if [[ $subproject_changes -ne 0 ]] ; then
          # run tests!
        fi

Interesting feature request - would love to hear more about what stage in the life of a pipeline you'd like to see this kind of check happen.

[acim]

Well, my use case is the first one. I have Golang backend and Svelte frontend in separate repositories, but I would actually like to merge them together in one monorepo because they are tight related, they should be deployed togher using the same Helm chart, etc. So, I would like to find out if something has changed in directory frontend and in this case build just Svelte application image, push it to registry and release. Same for the backend.

But I have seen similar discussion related to drone.io and Jenkins and there are people having above mentioned problems like having plenty of Ansible roles which takes a long time to test all of them, but should really test just the modified ones.

Or for example Helm charts, it is huge monorepo with hundred or so charts, so it would be useful to find out really what subproject to build.

I know this is possible to do with shell script but like you said, it is ugly. So it would be nice if you can find out some nice and easy way to achieve this with Tekton as I really think it is going to replace most of the other CI/CD's.

I am not in rush with this, but maybe you put this into long term planning and make us all happy with such a feature in some of the new releases. Thank you in advance.

[dibyom]

I think Conditional Tasks in a pipeline should work for this...you can have multiple task(s) e.g. build golang, build frontend etc. Each task can have a condition that determines if it should be run. The condition can e.g. look into the git resource/PR resource and see if go files were modified etc.
(I think @pritidesai had an example for this -- conditional builds depending on the type of source files)

tasks:
- name: build-go
  conditions:
  - conditionRef: "commit-contains-go-files"
- name: build-frontend
  conditions:
  - conditionRef: "commit-contains-js-files"

[pritidesai]

Those conditions are defined here: https://github.com/pritidesai/openwhisk-build/blob/master/tasks/detect-runtimes.yaml

[acim]

Great, thank you all.

[shiyu-dong]

@sbwsg @dibyom Just want to add some details for a similar use case. We would like to know if Tekton can support "dynamic" pipelines and pipeline hierarchies.

For example, with above monorepo example that @acim mentioned but with hundreds of services, is something below possible in one pipeline?

1. Calculate changed services for a given commit. The result service list is dynamic.
2. For each of the changed service, run another "child" pipeline that run a fixed DAG of tasks in parallel. All "child" pipelines also run in parallel.
3. "Parent" pipeline finish as successful until all child pipelines finish, or as failed if any of the child pipelines fails.

[acim]

Yes, there can be quite a huge complexity behing this and it would be really nice that Tekton handles this because otherwise we still have to deal with a prehistory like Jenkins.

[dibyom]

/kind question

[dibyom]

Interesting use case.
Right now you'd have to write a custom task that can parse the changed services and trigger each of your child pipelines and wait for them to finish (probably using tkn or kubectl)

1684 might be of interest in the near future especially the bit around failureStrategies that will allow parallel branches of a pipeline to keep running even if one of the branches fails.

[dibyom]

https://github.com/tektoncd/pipeline/issues/50 might be somewhat related too

[shiyu-dong]

Right now you'd have to write a custom task that can parse the changed services and trigger each of your child pipelines and wait for them to finish (probably using tkn or kubectl)

Thanks @dibyom! For the "wait for them to finish" part, should I periodically poll the child pipelines statuses in the task itself using tkn or kubectl?

Is Cloud Event Resource relevant here? Can a cloud event signal back to a Task?

[bobcatfish]

@shiyu-dong your use cases sound awfully similar to the CI that @afrittoli put together for us for Tekton itself: https://github.com/tektoncd/plumbing/blob/master/tekton/ci/README.md#tekton-ci-with-tekton (some more details in the PR https://github.com/tektoncd/plumbing/pull/254)

[dibyom]

For the "wait for them to finish" part, should I periodically poll the child pipelines statuses in the task itself using tkn or kubectl?

Yeah that could be one option. Another thing we were considering is allowing a pipeline to trigger multiple other sub-pipelines.

If you do not need the parent pipeline to wait for finishing, using the CloudEvents output resource would be the way to go for now!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.