Amplify-cli: What the **** is going on in AWS land?

[houmark]

Describe the bug
I don't really know where to start, and yes, this should probably be 2-3 bug reports and yes those bug reports should probably be in 2-3 different repositories, but I really cannot find the energy to consider all that.

I have a fairly simple Amplify setup. It has two environments, dev connected to a develop Git branch and prod connected to master.

I have @searchable on one of my models. It's been working okay. Actually, now that I think of it, I started using it because the normal list query would randomly miss some dynamodb items, without any particular reason or error at all. Anyways, that was a side story we won't get into today. I started using @searchable since I was implementing filters that the normal list resolvers would not handle anyways.

I tried at some point to figure out how the nextToken worked, but it seemed to be broken, so I dropped that idea.

Knowing the (maybe too) fast pace of releases, I usually check for package updates daily or every other day and update. In the past, that has bitten my * because the CLI broke, but it was never worse than I could revert to the second last version and be back in action.

3-4 days ago I updated all AWS packages to the latest version. I was working on new stuff and noticed that without any changes in my end, the searchApplications query broke. The one using ElasticSearch. nextToken was now expected to be a String, and it was an Int. I did some amplify codegen and now there was some changes both to $nextToken and $sortDirection. Since AWS is basically not releasing any information about these changes, one is left to scout around to find bug reports and PR's to figure out what happened and what solved it and from that figure out what to do in my end.

I ended up accepting the codegen suggested changes, but the sort argument was now broken. I was having a pretty basic sort on createdAt to desc but this would now create another error. I can't even keep track of all the errors, but I guess it would be helpful, so if anyone would want to get into debugging this, I would be happy to try to recreate the errors and post them here. I ended up removing the sort argument and sort client-side as a hack (the usual "wait for someone else to report this in a few days" or do it yourself and hope for a quick fix applies here).

Moving on, and this is where it gets freaking **, all the above happened in my dev environment. I finished my features and wanted to push the changes to my testing/staging environment in the cloud. So I did. Using the Amplify Console. Result. This error:

{ ValidationError: 1 validation error detected: Value '2019/08/17/[$LATEST]aa8b9f0ca2bd453ab1e1d6b1b2b2a56d' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*
                                 at Request.extractError (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/protocol/query.js:50:29)
                                 at Request.callListeners (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
                                 at Request.emit (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
                                 at Request.emit (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:683:14)
                                 at Request.transition (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:22:10)
                                 at AcceptorStateMachine.runTo (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:14:12)
                                 at /root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:26:10
                                 at Request.<anonymous> (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:38:9)
                                 at Request.<anonymous> (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:685:12)
                                 at Request.callListeners (/root/.nvm/versions/node/v10.16.0/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
                                 message:
                                 '1 validation error detected: Value \'2019/08/17/[$LATEST]aa8b9f0ca2bd453ab1e1d6b1b2b2a56d\' at \'stackName\' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*',
                                 code: 'ValidationError',
                                 time: 2019-08-19T07:10:17.685Z,
                                 requestId: '660ab2f4-c250-11e9-bba5-6d24919267f6',
                                 statusCode: 400,
                                 retryable: false,
                                 retryDelay: 55.434459637885006 }

So that's a new one. I have done some 120+ deployments for this environment more or less without issues. Today not so much. I had a sneak in the only file that has stackName and it all looks fine and dandy. I have not done any manual modification to anything in the recent days. My app is in production and I have not had to do any major stuff.

Anyways, since I was not able to deploy to my testing for unknown reasons and with no obvious solution (and unfortunately no bug reports from anyone else hitting the same with a workaround) I decided to deploy my app to production. Hey, it should work, my local dev is working smooth, and I need this live so... let's go!

Not so much... searchApplications is now broken. Complaining that nextToken should be an Int. But hey, I just changed that to be a String because you complained about that the other day? Now you want to be an Int again?

I did amplify codegen and I kid you not. Every time I ran it, it results in different changes. Like really, wtf? Am I hitting some drunk AWS server that just gives me a random result back? What's going on here? In the end, it wanted me to revert everything it wanted to change a few days ago. I am lost. My LIVE platform is down and I end up changing:

export const searchApplications = `query SearchApplications(
  $filter: SearchableApplicationFilterInput
  $sort: SearchableApplicationSortInput
  $limit: Int
-  $nextToken: String
+  $nextToken: Int
) {

I deploy again to LIVE and this time it actually works. Great. ** is rolling again. Now let's get testing working again.

Swap back to my dev environment. Same story with amplify codegen. But it now appears to me that my dev environment has fallen in love with the idea of using String as nextToken while prod wants it to be Int. Something happened in the cloud I guess, but that also means that I have inconsistencies in my environments — and no clear way of fixing it. If I fix dev, then prod breaks and vice versa.

Back to Amplify Console. After having given it some time to think about things and trying another failed deployment, I decided to delete the connection to the development branch. What a reboot can't handle, can normally be fixed by a delete and re-create, right? Nope... not so much... Not on this one. I lost my entire deployment history and my last working testing app, but I still get the same error. Oh hang on, once in a while I would get another error around the same time (but I did not save that one, unfortunately) but the result is the same. I cannot deploy. I guess I should feel lucky because at least it's testing that's broken and not production.

This "bug report" is the result of many similar nights since I entered the Amplify world. I have hit so many issues and things that told me to "walk away" but for some reason, I stuck around. Right now I honestly cannot think of why.

Team AWS / Amplify; You need to revamp and rethink your structure and how you release things. For a guy sitting here using your products and develop using them, it seems like you are a bunch of developers locked into each your own box. It's like you have no project managers, that there's no policy about documentation with each release or even a changelog to explain breaking features.

You know, people use your stuff, right? Some of us depend on what you do is stable, because if it's not, then it breaks our stuff, and there's little we can do.

I could go on and list 20+ things that are either broken, halfway done, not done, not started and should have been here from the beginning, but all I want to know is.... can you please try to up your game and coordinate breaking changes, and communicate them? You have no road map, you have no upcoming features. It seems completely random what comes out and it seems like you are doing not much more than killing fires.

Wanna hear about all those critical features that are nowhere to be found? Okay, now that I am writing the longest Github bug report ever, here we go:

• There's no way to count items in a normal graphql query. dynamodb seems to be the limitation here, but this is something EVERYONE does in their app. ElasticSearch supports it's so a searchable query could do it, buuuut it's not implemented in the resolver.
• Multi auth is still not happening
• Cognito is half baked, no way to update attributes in the UI, and no "non-hacky" way to list users using API so one can build your own user listing
• graphql queries does not cancel and cannot, even though axios has that features and is used to run this stuff. Someone made a PR, but it got stuck due to imho request to keep axios separated from this feature. This leads to extra UI work to make sure you don't get old query results back after later ones has been sent off.
• Lambda functions inside Amplify does not support layers. This prevents code share between functions and creates redundant code

I could go on here, but what I listed above should not be something I dream about, it should be something that's already done.

For me as a developer in 2019, I can choose from similar platforms from major players. My choice is based on features, stability, how fast I can develop on the platform, cost etc. But my time is also important and when I end up sitting a long * night to fix ** that you could have not released buggy if somebody would have reviewed it, I don't feel very productive and I once again start wondering if I did a huge mistake by choosing your platform.

When deployments break out of the blue, ** is getting real. This may be an isolated case for me, but your lack of handling errors when they happen is really an indicator of a too fast pace and too little focus on quality and stability. The error you display to me in my case above, does not make me any more smart about what went wrong at all. And I can't just sit around and wait for someone in your end to respond to a bug report like this one. I need my app deployed. I have deadlines you know?

Really. Get a grip AWS devs. You are all over the place and yet, nowhere. You hammer out PR's and then they sit around for weeks or months (multi auth comes to mind again). Put a road map out, stick to it, add time for critical bug fixes. It's not that hard!

To Reproduce
Simply use AWS Amplify and you will get hit sooner than later.

Expected behavior
That AWS starts respecting the developers using their platform by having some integrity, do more testing, put together a road map and communicate breaking changes better by doing what's called a changelog on every release.

[azatoth]

This also happens to me, and I'm uncertain how to proceed from here.

[kaustavghosh06]

Hey @houmark,
Sorry to hear that you are having issues with deploying your app via the Amplify CLI/Console. I’ll try helping you out so that you’re unblocked, and address your concerns.

1. Error on Amplify console and inconsistencies between codegen generated by the CLI vs codegen generated by the Console.
This issue is due to the different versions of the CLI running locally and on the Console. Currently, the Console is deploying via v1.7.0 vs the CLI which you’re using to deploy locally using v1.12.0. To use the latest version of the CLI in the console, you can have a setting in the console build configs to use the latest version of the CLI: https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#setup-live-updates. We’ve also been tracking the Amplify Console issue which you’d commented on - https://github.com/aws-amplify/amplify-console/issues/62 and the Console team has released a new build with the fix for it (all regions except Dublin at the moment).

That being said, the root-cause of the issue was that the Console pins versions of the CLI which led to the version mismatch between your local version (v1.12.0) and image version. The Amplify Console service updates the version on a monthly cadence and pins CLI versions to ensure customers running older versions aren't broken. Customers that want to run the latest can use the live updates feature (https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#setup-live-updates). Having said that, we have now updated the image version to match the latest CLI release.

2. CHANGELOG
If you take a look at our repo, we’re have a monorepo in the CLI and every package is deployed separately to NPM and each of the packages has its own Changelog. So for example, you can find the @searchable change in this Changelog - https://github.com/aws-amplify/amplify-cli/blob/master/packages/graphql-elasticsearch-transformer/CHANGELOG.md. In addition, the Changelog for console can be found here: https://github.com/aws-amplify/amplify-console/blob/master/CHANGELOG.md. For overall CLI Changelog, you can refer to https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-cli/CHANGELOG.md
Curious to learn more to improve the visibility of this based on your feedback Would the top level GitHub README be a better place to reflect this so that it’s easier for customers to find Changelogs for individual packages? Any alternate way that you think we can make this better?

3. Elasticsearch sorting issue
This issue is tied to a reported bug - https://github.com/aws-amplify/amplify-cli/issues/800 which we fixed with https://github.com/aws-amplify/amplify-cli/pull/1775 but, unfortunately, it introduced a bug in the transformer.
The root cause was while doing this PR, we missed out on the edge case for the auto-generated fields (by the resolver) for createdAt and updatedAt fields. This was not covered as a part of the integration tests in the above PR. Right now, we're working working on the end to end integration test for this use case before we publish a new version of the CLI to NPM.
The intention of the PR wasn’t to release a breaking change, but it in fact was a bug which got released and we’re working on providing a fix for it. The PR for it is - https://github.com/aws-amplify/amplify-cli/pull/2090.

For the features you'd mentioned, I can share the updates:

Multi auth:
We realize this has taken a while. The PR is ready for it and we're doing our final testing + integrating it with the amplify mock/testing feature before we release it. We're aiming end of this week for the release.

Cognito is half baked, no way to update attributes in the UI, and no "non-hacky" way to list users using API so one can build your own user listing
Unfortunately, the immutability of the sign-in/sign-up attributes from Cognito is a service limitation. As a result, there is no immediate plan to push a fix for this.
For the list-users API from Cognito, we're actively working on spinning up a REST API endpoint to list cognito users which you can easily call from your client app. We're aiming for end of August to release this feature.

Lambda functions inside Amplify does not support layers. This prevents code share between functions and creates redundant code
This is a feature request in our repo and we are going through the feature request bucket based on the count of customers asks. This is high in the list as you can see if you sort it by count and we are working on all the issues right above this one so far.

We understand that these issues have consumed a lot of time at your end and want to make the developer experience smoother for you and everyone else based on the feedback. We have taken steps towards mitigating the issues you reported from happening again to ensure we raise the quality bar with every release.

[houmark]

Thanks @kaustavghosh06 — I will respond more elaborate to your feedback later, but right now my main concern is Elasticsearch and the fact that my dev environment wants to run Elasticsearch in one way (in relation to nextToken) and my prod environment wants it opposite. There's no way I can fix this I believe. I think an amplify push changed the configuration in the cloud for my dev environment and that did not happen in prod.

I don't know how to solve that.

In relation to your changelogs. I have been following those (mainly the CLI changelog) and while they indicate a change (though in this case the Elasticsearch was not mentioned there at all), nowhere in those I see breaking mentioned or any simple explanation of what's expected of developers (and these changes were in fact very breaking for me). And you do not seem to respect semver in any way, so these updates just sneak in and break stuff. It feels dangerous to update anything because it feels like ** and if I don't test every single feature in my app, I don't know if that's the case.

You provide a platform and repositories that are fairly low-level and dependant upon by many apps and developers. That puts you in a spot where you should have a higher standard because ** can really break for a lot of people out there when you just let breaking changes slip into regular releases. I respect mono repositories, but you could also make some built script that merges all changes to ONE changelog because, for the end-users of your reps, it's almost impossible to keep up with all changelogs. And then have some kind of policy that a merge commit of any feature needs a text that explains the feature and the same goes for breaking changes.

[kaustavghosh06]

@houmark In my response above, I've mentioned about the root cause for the issue - it's tied to the inconsistency between your dev environment which was deployed via the CLI locally with version 1.12.0 and your prod environment deployed via Amplify console was using an older version of the CLI - 1.7.0. The Console is now updated to use the latest version of the CLI, and if you re-deploy your app (the prod environment) right now using the Amplify console, you should see the same behavior as your dev environment, which should solve your dev and prod inconsistency issue.
We made the change in the behavior of nextToken since it was a bug in the first place - #800 and our intention wasn't to make it a breaking change, but the change to fix this bug - #1775, introduced another bug which you've mentioned above (which made it seem like a breaking change) - tied to sorting on the "createdAt" and "updatedAt" fields which we've submitted a fix for in #2090 which wasn't caught in our integration tests. We constantly work on improving our integration test coverage and have added a bunch of tests lately to prevent this sort of behavior and breaking customers. You can check out our various integration tests out here- https://github.com/aws-amplify/amplify-cli/tree/master/packages/graphql-transformers-e2e-tests , https://github.com/aws-amplify/amplify-cli/tree/master/packages/amplify-e2e-tests & https://github.com/aws-amplify/amplify-cli/tree/integtest/packages/amplify-ui-tests

For the changelog piece, your suggestion is great and we'll accumulate all the changelogs to one top-level Changelog for easier discovery.

Please let me know if you're still blocked on this issue after re-deploying your prod environment via the Amplify console.

[houmark]

So I can confirm that I was now able to deploy dev again and it seems to be working all around, including nextToken as a String value.

Looking forward to sorting with createdAt also being resolved. Thanks for your help so far @kaustavghosh06!

[jkeys-ecg-nmsu]

@kaustavghosh06 maybe Amplify could publish a list of Known Limitations and any other gotchas that might exist, so customers can more accurately gauge ahead of time whether Amplify will work for them.

The Sumerian service team already does this.

[RossWilliams]

@kaustavghosh06 Will the upgrade cause build errors for users still on 1.7.x ? What is the plan here? The open bug in #1581 will cause a failure for some of these users who were previously relying on 1.7.

[alexladerman]

I concur!

[ambientlight]

lol, that was a long post, @houmark, out of curiosity, how long did you evaluate before jumping to ship with Amplify back then?

[kaustavghosh06]

Closing this issue since majority of the issues mentioned in the thread are resolved. If you have additional specific feature requests, please open a GitHub issue.