Copilot-cli: Monorepo support in pipeline manifest

[mikelhamer]

Looking at the docs here I don't see anything that talks about only listening for changes to a certain path / service. What if I want service specific pipelines in a multi service app? Is that possible?

https://aws.github.io/copilot-cli/docs/manifest/pipeline/

[iamhopaul123]

Hello @mikelhamer. In Copilot we allow an application to have difference local workspaces, and the pipeline uses all the services in the current workspace. So if you just want some specific services in a pipeline, you could just place them in a separate workspace and run pipeline init/update. For example, you have service A, B, and C in the same app and want service A to be in a separate pipeline. The directory will look like

|-Workspace1
|--copilot
|---serviceB
|---serviceC
|-Workspace2
|--copilot
|---serviceA

Then you would need to run app init twice those two workspaces to add them into the same app. And then pipeline init/update twice to create two pipelines. Let me know if you have any question!

[mikelhamer]

Thanks!

[mikelhamer]

@iamhopaul123 I'm finaly at a point where I'd like to implement multiple pipelines for my app but I'm a bit confused now.

My app currently looks like this, with the pipeline listening on the main branch and deploying every service whenever any change is made

|-apps/
|---service1
|---service2
|-copilot/
|---service1
|---service2
|---.workspace
|---buildspec.yml
|---pipeline.yml

but when I try running copilot pipeline init or copilot app init inside any of the subdirectories I get this response from the CLI, but nothing changes. No copilot files are created or updated.

Your workspace is registered to application myapp.
√ Created the infrastructure to manage services and jobs under application myapp.

√ The directory copilot will hold service manifests for application myapp.

Recommended follow-up actions:
- Run `copilot init` to add a new service or job to your application.

[mikelhamer]

Another thing I'm confused about is that even if I had multiple pipelines set up, wouldnt they all trigger whenever a change is made if they all point to the same branch? What I need is the ability to only trigger a build if changes are made to the corresponding directory in my monorepo

[efekarakus]

Hi @mikelhamer !

In the monorepo setup above:

├──apps/
│  ├──service1
│  └──service2
└──copilot/
   ├──service1/
   ├──service2/
   ├── .workspace
   ├──buildspec.yml
   └──pipeline.yml

When you run copilot pipeline init and copilot pipeline update you will get only 1 pipeline that will deploy to your environments.

wouldnt they all trigger whenever a change is made if they all point to the same branch?

That is correct. On any pushed commit upstream, the pipeline will be triggered and all your services will be updated 😐
A separate repo per service does get around this limitation.

Unfortunately, I can't think of a way of only updating a specific service with a monorepo setup. I wonder if there can be specific commands that we can take in the generated buildspec.yml to limit which services are going to be updated.

[huanjani]

Could you house each service on a different branch in your repo, then, following https://github.com/aws/copilot-cli/discussions/1925, create a pipeline for each branch?

[mikelhamer]

@efekarakus thanks for the advice. I will look into possibility of modifying buildspec.yml, but I'm thinking seperate repos might be the way to go. A little unfortunate, but if that's what it takes to keep them isolated then I may have to do it.

[mikelhamer]

@huanjani I hadn't thought of that, hmmm. Based on our git branching strategy I don't think this would work for our team as we are using a GitHub flow as described here. https://guides.github.com/introduction/flow/ . Wondering what the main or default branch would be in this case if each service were it's own branch.

[huanjani]

You could have the main branch as is, but add service-specific branches (that still have the code for the other services) that are where you push changes to the services (and trigger the corresponding pipelines). Then merge those branches into main when appropriate.

Would that work?

[mikelhamer]

It might! I will give this some thought.