Copilot-cli: enable multiple pipelines--same repo source, diff branches

[huanjani]

If we were to change the pipeline naming convention and include the branch name in the pipeline name, we should be able to enable multiple pipelines. We could add a selector so the user can choose the pipeline they want to update/show/status.

[joshgoodson]

This could be useful. I'm finding that if we could just use copilot pipeline init and specify the manifest or buildspec yaml files that could help deploy multiple pipelines for a single source repo. I would have interest in this type of functionality.

[huanjani]

Hi @joshgoodson! Thanks for the feedback, and apologies for that internal ref. Please see https://github.com/aws/copilot-cli/discussions/1925. That is a workaround to have different pipelines for different branches, but you can even create multiple pipelines from the same branch if you want-- the key is changing the pipeline name in copilot/pipeline.yml-- the different name will build a different CloudFormation stack and different CodePipeline pipeline.

This GitHub Issue is for a for a longer-term, more baked-in solution; we had designed Copilot pipeline functionality for single pipelines with multiple deployment stages (with optional manual approvals) but we're hearing requests for multiple pipelines.

Thanks again!

[oqrusk]

HI, I really need this feature. What I want do is to apply gitlab-flow(https://docs.gitlab.com/ee/topics/gitlab_flow.html#environment-branches-with-gitlab-flow) to my project.
However, #1925 workaround is not enough for me because single pipeline.yml will be conflict when I merge/make pull request from main branch to production(release) branch.

[huanjani]

Hello @oqrusk!

Thanks for your feedback! I believe .gitattributes (https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#Merge-Strategies) will help you do what you'd like. Also see the first answer here. Let me know how that works! I'll update #1925.

[oqrusk]

Hi @huanjani , thanks for your reply.
I've tried your suggestion, .gitattributes way. Unfortunately it didn't fit for me. merge=ours strategy works for only local client. I'm using Github/CodeCommit Pull-Request to merge branches. I think both of them don't have this feature.

What I want to do is Git-lab flow which is cascading merge flow like below.
master(deploy to staging-env) -> pre-prod (deploy to pre-production env) -> production (deploy to production env)
In this case, PR's auto-merge can't ignore pipeline.yml conflicts.

To avoid this problem, I'm thinking about making pipeline.yml for each env=branch. This way needs to modify one line on buildspec.yml which is getting stage name. Is it OK to change buildspec.yml manually?

[mvn-sonduong-dn]

Hi @huanjani cc @oqrusk ,
Same as the way of oqrusk, I tried use solution .gitattributes but it's not working. I have seen a discuss at here relate to this case, then:
"The root cause of your problem is that you're trying to do something that the designers of Git never considered to be a valid use case. The idea of having some files present in one branch but not in another related branch, independent of the actual merging of those branches, is not an idiomatic use of Git..."
So I believe that no ways for idea: can exclude files that do not merge when merge from develop branch to -> main branch (This case is file: pipeline.yml)
I am looking forward to the multi-pipeline feature. When does this feature be release?
Very thanks!

[huanjani]

@oqrusk I'm so sorry I missed your question until now! It is okay to make changes to your buildspec, yes. But you could have the same buildspec, which reads each different pipeline.yml file, pushed to each branch repo.

@mvn-sonduong-dn Yes, the most legit way of maintaining this workaround is to manually fix the merge conflicts in order to merge the branches, then revert the pipeline.yml (with the different pipeline name) after the merge in order to keep the pipelines separate.
Sorry, we don't have a date for the release of this feature; we have a lot of features in the works. It's good to know that this is a priority for our users. Please 👍🏼 the initial comment in this issue--we try to respond to customer demand!

Thank you both!

[mvn-sonduong-dn]

@huanjani Thank you for your very fast reply. I'll try with your solution.
Have a nice day ^^!

[enm10k]

Hi @huanjani,
I'm interested in this feature.
I have a use case as follows. Is this covered by your proposal?

I have two environments of my copilot application in different AWS accounts and I'd like to create independent pipelines in those AWS accounts to meet some security guidelines.

By the way, the workaround #1925 looks good to me.
I'll give it a try.

[huanjani]

Hello @enm10k!

Yes, the proposal should work for your use case-- one pipeline deploying to one account's environment and another pipeline deploying to the other account's environment.

Let us know how it goes! 😊