Amplify-js: correctClockSkew for API

[eduardogspereira]

Which Category is your question related to?

AWS Amplify API.

What AWS Services are you utilizing?

Cognito and API Gateway.

Provide additional details e.g. code snippets

Hi,

I notice that on aws-sdk there is an option for correctClockSkew (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/APIGateway.html), that is better explained here https://github.com/aws/aws-sdk-js/issues/1174#issuecomment-253600875 .

Is this or a similar option exists on AWS Amplify? The only approach for me achieve it is similar to here https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/APIGateway.html ?

My main goal on this is that when some machine has a clock that for some reason is not correctly synchronized, then, s/he stills be able to make a request.

I was getting this error with machines where the time was not correctly set:

GET :: "Signature expired: 20181029T181641Z is now earlier than 20181029T191126Z (20181029T191626Z - 5 min.)"

Thanks for the help and sorry for the few details.

[mhelgeson]

Having a similar problem: users with clocks that are not set correctly are able to authenticate with cognito (amplify/auth) but requests to api gateway are rejected (amplify/api).

Is there a way to correct the clock skew through some configuration setting?

[jayair]

I'm having this issue as well. Is there a fix for this? Is it being worked on?

AWS SDK correctClockSkew setting does not work anymore. And this issue is incredibly easy to replicate. Just set your system clock ahead by more than 5 minutes all API requests will fail.

[willdady]

Same here. Using Amplify in React Native contacting API Gateway we get several of these errors reported daily in Sentry.

Signature not yet current: 20190115T080011Z is still later than 20190115T075748Z (20190115T075248Z + 5 min.)

What can be done to fix this? On version 1.1.10 if that helps.

[jayair]

Hey guys is this being worked on? We keep getting this error.

This was an issue that the AWS SDK has a fix for. But we aren't able to use the setting through Amplify.

[taschik]

Hi,

we are constantly facing this issue to that a lot of our mobile clients run out of sync in time at some point. Is there any fix that can be provided? This happens with "aws-amplify": "^1.1.11" under react-native 0.59.4 but we were seing beasically since aws-amplify 1.0.X and older RN versions too.
Is there anything that can be done to fix this?

[gabrielenosso]

Same problem here... still no ways to fix the issue?

[harrycollin]

Same here, anything new here?

[joelfsreis]

I'm also having the exact same issue, any news?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kevinsperrine]

This isn't stale. FWIW, I've tried globally setting this via AWS.config.update({ correctClockSkew: true });, but that still fails with both Pinpoint updateEndpoint requests and GraphQL requests via appsync.

[jayair]

Yeah this needs to be addressed.

[kevinsperrine]

There are multiple places this needs to be addressed. For REST API calls, it needs to account for the systemClockSkew in the Signer.sign function (https://github.com/aws-amplify/amplify-js/blob/master/packages/core/src/Signer.ts#L266). I've been able to get PinPoint and AppSync (using aws-appsync) to retry by using a custom onError handler for Apollo, setting AWS.config.update({correctClockSkew: true}) and a few patch-packages.

export const clientInstance = () => {
    const onErrorLink = onError(errorResponse => {
        const {
          graphQLErrors,
          networkError,
          forward,
          operation
        } = errorResponse;

        if (graphQLErrors) {
          graphQLErrors.map(({ message, locations, path }) =>
            console.log(
              `[GraphQL error]: Message: ${message}, Location: ${locations}, Path: ${path}`
            )
          );
        }

        if (networkError) {
          // @ts-ignore
          const { response } = networkError;
          const dateHeader =
            response.headers.map.date || response.headers.map.Date;
          if (dateHeader) {
            const serverTime = Date.parse(dateHeader);
            const isSkewed =
              Math.abs(new Date().getTime() - serverTime) >= 300000;
            if (isSkewed) {
              AWS.config.update({
                correctClockSkew: true,
                systemClockOffset: serverTime - new Date().getTime()
              });

              return forward(operation);
            }
          }
        }
      }
    );

    const appSyncLink = createAppSyncLink({
      auth: {
        credentials: () => Auth.currentCredentials(),
        type: AUTH_TYPE.AWS_IAM
      },
      region: AmplifyConfig.AppSync.aws_appsync_region,
      url: AmplifyConfig.AppSync.aws_appsync_graphqlEndpoint
    });

    const link = ApolloLink.from([onErrorLink, appSyncLink]);

    client = new AWSAppSyncClient(
      {
        auth: {
          credentials: () => Auth.currentCredentials(),
          type: AUTH_TYPE.AWS_IAM
        },
        region: AmplifyConfig.AppSync.aws_appsync_region,
        url: AmplifyConfig.AppSync.aws_appsync_graphqlEndpoint
      },
      {
        connectToDevTools: __DEV__ ? true : false,
        link
      }
    );

  return client;
};

@aws-amplify+core+1.0.27.patch:

diff --git a/node_modules/@aws-amplify/core/lib/Signer.js b/node_modules/@aws-amplify/core/lib/Signer.js
index 8c40eb2..f4a2942 100644
--- a/node_modules/@aws-amplify/core/lib/Signer.js
+++ b/node_modules/@aws-amplify/core/lib/Signer.js
@@ -31,7 +31,7 @@ var __rest = (this && this.__rest) || function (s, e) {
 Object.defineProperty(exports, "__esModule", { value: true });
 var Logger_1 = require("./Logger");
 var Facet_1 = require("./Facet");
-var logger = new Logger_1.ConsoleLogger('Signer'), url = require('url'), crypto = Facet_1.AWS['util'].crypto;
+var logger = new Logger_1.ConsoleLogger('Signer'), url = require('url'), crypto = Facet_1.AWS['util'].crypto, config = Facet_1.AWS.config;
 var DEFAULT_ALGORITHM = 'AWS4-HMAC-SHA256';
 var IOT_SERVICE_NAME = 'iotdevicegateway';
 var encrypt = function (key, src, encoding) {
@@ -254,8 +254,9 @@ service_info: {
 var sign = function (request, access_info, service_info) {
     if (service_info === void 0) { service_info = null; }
     request.headers = request.headers || {};
     // datetime string and date string
-    var dt = new Date(), dt_str = dt.toISOString().replace(/[:\-]|\.\d{3}/g, ''), d_str = dt_str.substr(0, 8);
+    var dt = new Date(Date.now() + config.systemClockOffset), dt_str = dt.toISOString().replace(/[:\-]|\.\d{3}/g, ''), d_str = dt_str.substr(0, 8);
     var url_info = url.parse(request.url);
     request.headers['host'] = url_info.host;
     request.headers['x-amz-date'] = dt_str;

aws-appsync+1.8.0.patch:

diff --git a/node_modules/aws-appsync/lib/link/signer/signer.js b/node_modules/aws-appsync/lib/link/signer/signer.js
index a060642..2c1df67 100644
--- a/node_modules/aws-appsync/lib/link/signer/signer.js
+++ b/node_modules/aws-appsync/lib/link/signer/signer.js
@@ -8,7 +8,7 @@ or in the "license" file accompanying this file. This file is distributed on an
 See the License for the specific language governing permissions and limitations under the License.
 */
 global.Buffer = global.Buffer || require('buffer').Buffer; // Required for aws sigv4 signing
-var url = require('url'), crypto = require('aws-sdk/global').util.crypto;
+var url = require('url'), crypto = require('aws-sdk/global').util.crypto, config = require('aws-sdk/global').config;
 var encrypt = function (key, src, encoding) {
     if (encoding === void 0) { encoding = ''; }
     return crypto.lib.createHmac('sha256', key).update(src, 'utf8').digest(encoding);
@@ -197,7 +197,7 @@ var sign = function (request, access_info, service_info) {
     if (service_info === void 0) { service_info = null; }
     request.headers = request.headers || {};
     // datetime string and date string
-    var dt = new Date(), dt_str = dt.toISOString().replace(/[:-]|\.\d{3}/g, ''), d_str = dt_str.substr(0, 8), algorithm = 'AWS4-HMAC-SHA256';
+    var dt = new Date(Date.now() + config.systemClockOffset), dt_str = dt.toISOString().replace(/[:-]|\.\d{3}/g, ''), d_str = dt_str.substr(0, 8), algorithm = 'AWS4-HMAC-SHA256';
     var url_info = url.parse(request.url);
     request.headers['host'] = url_info.host;
     request.headers['x-amz-date'] = dt_str;

[cyberfox1]

Does this apply to normal XHR calls made through Amplify? I.e. does correctClockSkew:true fix the problem?

[fcasals]

Same behavior here.
Using Amplify 1.1.32 and React
Thanks in advance

[zigzhang]

Same behavior as well.
Using React and

    "@aws-amplify/api": "^1.0.36",
    "@aws-amplify/auth": "^1.2.25",
    "@aws-amplify/core": "^1.0.28",

[neskimoo]

Having the same issue here with Amplify 1.1.36

[kgpasta]

Any update on this issue? Also seeing this pretty frequently on my RN app

[jayair]

Yeah once again. This is a very easy to reproduce issue. I'm not sure why it's not even being looked at...

[willdady]

@powerful23 can we get an update on this? This has been a major issue for over a year now. I can attribute several 1 star app store reviews to this bug. 😕

[dorsegal]

So I have just googled my problem and got here.
Same happens here with:
aws-amplify: ^1.2.4
react: 16.6.3

[dorsegal]

for all who want a workaround, we used @thiagohirata fix.
you can see how he managed to solve it here: https://github.com/aws-amplify/amplify-js/pull/4251

[kgpasta]

@dorsegal thanks for the example, unfortunately that doesn't work for federated auth, only user pools 😞

[ericclemmons]

I was able to reproduce this within a sample application:

1. Update my Date/Time to 2 hours ahead.
2. Refresh my app (being logged in wasn't necessary)

3. My AlbumList component makes this query:

   API.graphql({
         authMode: owner ? 'AMAZON_COGNITO_USER_POOLS' : 'AWS_IAM',
         query: listAlbums,
         variables: {
           nextToken: pageToken
         }
       })
         .then(payload => ...)
         })
         .catch(payload => {
           throw new Error(payload.errors[0].message);
         });
   

4. I get a 403 error:

   {
     "errors" : [ {
       "errorType" : "InvalidSignatureException",
       "message" : "Signature not yet current: 20200207T211846Z is still later than 20200207T192350Z (20200207T191850Z + 5 min.)"
     } ]
   }
   

Note that setting AWS.config.correctClockSkew = true didn't fix it:

import AWS from 'aws-sdk';
import { Auth, Hub } from 'aws-amplify';

AWS.config.correctClockSkew = true;
...

[tommedema]

I am able to reproduce this when using Storage.put. I changed my clock to 4 hours forward, then called Storage.put, and it failed to upload:

Uncaught (in promise) RequestTimeTooSkewed: The difference between the request time and the current time is too large.

This is affecting our users in production at usebubbles.com. It would be great if we can get this resolved asap.

I think Amplify should enable correctClockSkew by default.

Until then, we need a workaround quickly.

[cyberfox1]

I don't think anyone here understands what the issue/problem is. #4251 is a joke, setting an explicit offset when you don't know what the actual offset between any given client and the server will be? You have to be kidding me (apparently he wasn't).

The actual problem is for any given client/server there may be a clock offset. If that offset is too great, then sigv4 signed requests will fail validation on the server.

The actual solution - which seems beyond the capacity of people here to even comprehend - is for the client to inspect headers returned by the server in a HEAD request that tells the client what time the server currently thinks it is. The client then compares that to the time it believes is correct, and if they are different by some margin, uses that to generate an offset in the time used to sign the request. That way the server will not reject the request as invalid due to the time being too far out.

The aws-sdk already implements the above. This bug was always about implementing the same thing in Amplify. The fact we are still trying to get people to understand the underlying issue shows how far away we are from an actual solution.

[ericclemmons]

The actual solution [ snip ] is for the client to inspect headers returned by the server in a HEAD request that tells the client what time the server currently thinks it is. The client then compares that to the time it believes is correct, and if they are different by some margin, uses that to generate an offset in the time used to sign the request. That way the server will not reject the request as invalid due to the time being too far out.

We're in agreement. #4251 (via #4844) is the building block to handle this automatically.

I apologize for the delay, as we've been getting a major release out the door before addressing this.

I'm tracking this issue (among others) in the Auth Clockdrift milestone so we can resolve them all.

[tommedema]

@ericclemmons good to hear. Since this is open since 2018, is there a short term workaround for organizations that cannot wait for this to be resolved?

[ericclemmons]

@tommedema Not that I can recommend. There were reports that you could basically "retry" with the corrected clock skew (after the first one failed) here:

https://github.com/aws-amplify/amplify-js/pull/4251#issuecomment-568915659

In my testing, errors were thrown (due to clock skew of course), but the offset wasn't available to customers via this error for setting offset manually.

@sammartinez Let's bring this into tomorrow's grooming session.

[tommedema]

@ericclemmons is there a way to get the response headers of e.g. a failed Storage.put call in a catch statement? That way we could programmatically retry with a systemClockOffset defined as a workaround.

EDIT: we are working around this for Storage.put using a custom lambda that returns the server time. This is super ugly, but because this is such a critical issue it beats waiting for another couple of months:

https://gist.github.com/tommedema/13b5840ac3b1003178dfab8a0aa6bc6d

[tommedema]

Any update on this? This is now more important since Amplify 3 does not support AWS.config, we can no longer manually work around this using:

   AWS.config.update({
        systemClockOffset
      })

[jayair]

Anybody?

[ericclemmons]

@jayair @tommedema Can you share scenarios affected by clock skew?

We're now using https://github.com/aws/aws-sdk-js-v3, which automatically handles clock skew for several of our API calls: https://github.com/aws/aws-sdk-js-v3/pull/459

When reproducing https://github.com/aws-amplify/amplify-js/issues/2014#issuecomment-606802386 with aws-amplify@latest, Storage.put and GraphQLAPI.graphql both succeeded. (I authenticated with federated auth)

[yansokalau]

@jayair @tommedema @ericclemmons hi guys, just wanted to share how I've dealt with the issue with the help of https://github.com/aws-amplify/amplify-js/pull/4844

1. Updated Amplify till 2.2.7
2. Applied correctClockSkew after configure: Amplify.configure(environment.amplify); AWS.config.correctClockSkew = true;
3. Before app initialization executing the currentUserCredentials().then((cred) => { DateUtils.setClockOffset(cred['cognito']['config']['systemClockOffset']); }); where DateUtils: import { DateUtils } from '@aws-amplify/core';

It took 2 months to first of all detect the issue (those random 403 errors in logs didn't tell a lot), then spend a day reading all the related topics on github and grabbing the information. So hopefully these instructions help you. @ericclemmons please let us know if smth wrong with the solution. You mentioned that issue has to be solved automatically without any actions required, is that correct? For sure it doesn't work without correctClockSkew=true.
Thanks to guys who dig into the issue and helped with the fix!

P.S. We use Cognito Auth for API calls signing only.

[hisham]

I'm also still seeing this error when clock is let's say 10min wrong, while calls directly to cognito go through fine. I'm going to try @yansokalau's solution.

[houmark]

I've also done attempts to this on a public REST API endpoint and it still returned errors. Having so many other issues with Amplify, I quickly moved on to something else, as documentation here is really weak and I did not want to enter the rabbit hole of one day of trying stuff out.

Would be great with more examples.

[hisham]

I'm investigating and I think I know what my issue is. Since this error happens in the API Gateway level (not my express server lambda), my browser fails with a CORS error and the API Gateway response is ignored. DateUtils.isClockSkewError checks the headers to determine if the error is due to clock skew. And so the amplify auto fix clock skew code is not run.

Not sure what the solution is yet. You can read this for more details: https://serverless-stack.com/chapters/handle-api-gateway-cors-errors.html

[hisham]

I'm still trying to solve this issue. I do see the response from API Gateway, but it seems to come from Cloudfront and the x-amzn-errortype is InvalidSignatureException while DateUtils.isClockSkewed wants "BadRequestException":

https://github.com/aws-amplify/amplify-js/blob/main/packages/core/src/Util/DateUtils.ts#L52

So this might be the issue?

[hisham]

@ericclemmons looking at https://github.com/aws/aws-sdk-js-v3/blob/47770fa493c3405f193069cd18319882529ff484/packages/service-error-classification/src/constants.ts#L8, we have a variety of possible options for clock skew error codes:

export const CLOCK_SKEW_ERROR_CODES = [
  "AuthFailure",
  "InvalidSignatureException",
  "RequestExpired",
  "RequestInTheFuture",
  "RequestTimeTooSkewed",
  "SignatureDoesNotMatch",
];

while in https://github.com/aws-amplify/amplify-js/blob/main/packages/core/src/Util/DateUtils.ts#L52 you only check for BadRequestException which does not seem to be an option from above even:

    isClockSkewError(error: any) {
        if (!error.response || !error.response.headers) {
            return false;
        }

        const { headers } = error.response;

        return Boolean(
            headers['x-amzn-errortype'] === 'BadRequestException' &&
                (headers.date || headers.Date)
        );
    },

So this might be the issue?

[hisham]

I went ahead and filed https://github.com/aws-amplify/amplify-js/issues/6494 so that amplify team takes a second look at this. Seems like DateUtils' isClockSkewError is not handling all possible error types related to clock skew.

[hisham]

Ok I experimented some more, this one liner fixed my issue:

DateUtils.setClockOffset(user.signInUserSession.clockDrift * 1000 * -1);

The key was multiplication by -1 to set the correct clock offset from the clockDrift we get from cognito.

[KevinToala]

How can I get CLOCK_OFFSET for users who have not logged? I use cognito hosted ui with amplify to login with facebook and google.

Auth.currentUserCredentials() can't return cognito.config.systemClockOffset, return accessKeyId, sessionToken, secretAccessKey, identityId, authenticated properties :/