Amplify-cli: Repeated merge conflicts due to multi-env workflow

[tknuth]

Hello,

I am using amplify with multiple environments and a CI pipeline, and I am facing a specific problem. I am not sure whether this is due to a lack of understanding of the deployment process?

I have two environments, dev (development branch) and prod (master branch).

My workflow is as follows:

• on dev/development work on frontend/backend, push resources using amplify publish
• commit changes
• git checkout master && amplify env checkout prod
• git merge development
• amplify status shows me that I changed stuff and the backend will be updated
• amplify publish

This is all fine, the problem starts now:

When published on prod, there are file changes. The diff shows that the changes are only related to a key "Code" in the lambda functions' cloudformation templates in the backend folder:

/amplify/backend/function/function_name/function_name-cloudformation-template.json

"Code": {
-   "S3Bucket": "app_name-20190716141290-deployment",
+  "S3Bucket": "app_name-20190716180513-deployment",
     "S3Key": "amplify-builds/function_name-ru14rdtn1erne110ne-build.zip"
}

In the docs, it is said that these files should be committed. However, when committing such changes, I repeatedly get a merge conflict when merging new changes from development because then master has already diverged.

I thought about storing the backend folder in an S3 bucket for dev and prod each to avoid this versioning problem, but I am not sure whether that's intended.

What am I doing wrong?

[kaustavghosh06]

@tknuth Which version of the CLI are you using? We release a fix for this and the fix is a part of the latest version of the CLI.

[tknuth]

@kaustavghosh06 Thank you for your quick reply. I am using 1.8.2. I updated my CLI this morning but the described behaviour still occurs. I believe that there exists a similar problem, and a corresponding ticket, which refers to a problem with the S3 key, not the bucket name, and that was part of the fix — as I understood it. Could it be that this are two separate issues?

[tknuth]

Addition: I found this ticket https://github.com/aws-amplify/amplify-cli/issues/1666. I am not facing the problem with the changing bucket names, so this might be the one that has been fixed. However, there are also differing S3 keys, and that is the problem I describe here, even with CLI version 1.8.2

[kaustavghosh06]

@tknuth Okay, I see. We could probably pass in the bucketName as ref from your parent clodformation stack to avoid this naming collision. I'll add this to our backlog as an enahncement.

[tknuth]

@kaustavghosh06 I'm wondering why others do not report this problem, though? Isn't this inherent to the way the multi-env workflow is designed at the moment?

I solved this temporarily by removing the cloudformation templates from git and syncing them as a kind of "backend" via a S3 bucket when making changes, which seems to work fine.

[davekiss]

I've also encountered this issue, multi-env seems to be incompatible with Git at the moment

[plaa]

This is a major pain point for us as well. We use personal Amplify environments for development, and whenever anyone (accidentally) commits one of these files, it causes merge conflicts for everyone else.

Besides merge conflicts, it pollutes the git status with changed files. This is especially irritating when editing the template files, because you need to be very careful to check what files / lines you have actually edited and committing only those.

The way we've decided to handle it is:

1. When developing, we never commit those changed lines
2. When publishing to production we commit them to the separate env/prod branch

Currently our staging environment is also in the master branch, meaning that we don't commit changes to it. This is a bit unfortunate since staging is then different from production. We could branch both env/staging and env/prod from master, but the structure suggested in the docs (master --> env/staging --> env/prod) would again cause merge conflicts.

I think it's a mistake that Amplify modifies these files at all, since it makes multi-env workflow very painful.

[mrducky4]

I am also having this problem. Is it safe to simply not commit the changes to the lambda function cloudformation-template.json file? We use prod, test, and dev environments. We use amplify push to deploy, and we also use amplify console linked with bitbucket to deploy.

[plaa]

To my understanding it's safe not to commit those files, but it will cause excess resources (which are already up-to-date) to be deployed after a file revert

[AlessandroAnnini]

Any news about this? managing multiple envs and multiple git branches is awful when you have a lot of resources!

[lon9man]

AWS, hello, any news?? terrible thing!

i spent a lot of time to understand where is the problem, but even now can only assume.
i have several lambda functions and several environments. on merge i see the same conflists like a topic starter.

[rvanbael]

Same issue here, if I understand correctly, we cannot use properly Amplify with Lambda in a multi env. I call that a bug, not a request for enhancement.
I tried to update to last CLI version. It does not patch the issue.

Do we have any other alternative to gitignore cloudformation-template.json files? I don't like that, because these files are configuration files, to tune lambda function settings. And to play with git at each new release is simply not an option for real life applications.

[RossWilliams]

@rvanbael these values changing should not prevent pushing resources, and changes to these lines can be safely ignored. When pushing a resource, these templates get updated with correct s3 bucket keys, see code here.

If you want to keep git history clean, your team could decide not to commit changes these lines, but any changes should not have a negative effect.

[kaustavghosh06]

@tknuth @RossWilliams hit the nail - it shouldn't be affecting your push or any deployments - but we're looking into ways and tracking as to how we can avoid this altogether.
cc @UnleashedMind

[plaa]

@RossWilliams @kaustavghosh06 If those changes aren't supposed to be committed, then Amplify should not make the changes. Amplify could make a temporary copy of the file and update that instead of editing the git version.

The whole issue is that Amplify is making changes into source files, which then manually need to be avoided. It's a pain especially when editing the CF files (see my original comment).

Also _not_ committing the files has the negative impact of having them be deployed every time even though nothing has changed.

[chantlong]

Any updates on the best practice for handling this?

I don't want my GIT dev branch and prod branch to only have a difference on these S3 Keys.

Is everyone just gitignoring their amplify/backend/function/some-lambda/lambda-cloudformation-template.json ?

[RossWilliams]

My current practice is to not approve a PR containing key changes.

I don’t gitignore cloud formation files, every one of these files has a number of customisations I need for each lambda.

[chantlong]

@RossWilliams So let's say you keep the s3 keys in dev branch as it is.
And you merge dev into prod and then you amplify checkout to prod.
When you do amplify status, it's gonna show it needs to update the resources bc it's referencing the s3 keys from dev right?

I would just amplify push to get amplify status to chill out and get the new keys. (I can't just amplify pull because I will not be 100% sure what actually changed in the backend)
Are you saying you wouldn't commit the newly generated cloud formation json to the prod branch after the amplify push in this scenario?

[RossWilliams]

@chantlong This sounds correct, status is going to show function updates, and pushes are going to take a bit longer. I don't commit the new keys in the way you describe mainly because our automated pipelines and policies don't support it. If you have a production branch that you can commit to directly, your method does sound good.

[chantlong]

pushes are going to take a bit longer

That is the tradeoff I guess which I'm trying to get used to.
Even if you don't have any changes in the backend functions, it's safer to push.
I wonder if there are any AWS charges incurred on each push.
Looking below, it seems to be free?
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html

[RossWilliams]

If you use Amplify console to push, each minute is charged while waiting for CloudFormation to finish