Google-api-nodejs-client: Service account based auth credentials being cached improperly

[nikmartin]

Environment details

• OS: Google App Engine
• Node.js version: 10
• npm version:?
• googleapis version: 36.0.0

Steps to reproduce

1. create a new GoogleApis instance and try to get a client using different service account credentials multiple times in different parallel calls

I have a node.js app engine app function:

app.post('/myroutehandler/sometask', async (req, res) => {

  const storeID = Buffer.from(req.body.message.data, 'base64').toString('utf-8');
  let pageToken = '';
  if (req.body.message.attributes) {
    pageToken = req.body.message.attributes.pageToken;
  }

  const pageSize = 250;
  console.log('StoreID:', storeID);
  const config: StoreConfig = await getConfigById(storeID);

const g = new GoogleApis({});
  g.auth
    .getClient({
      credentials: {
        client_email: config.content_api.credentials.client_email,
        private_key: config.content_api.credentials.private_key.replace(/\\n/g, '\n').trim(),
      },
      scopes: ['https://www.googleapis.com/auth/content'],
    })
    .then(cli => {
      const content = g.content({ version: 'v2', auth: cli });

const qryParam: any = { merchantId: config.content_api.merchant_id, maxResults: pageSize, auth: cli };
      if (pageToken) {
        qryParam.pageToken = pageToken;
      }
      content.productstatuses
        .list(qryParam)
        .then(errorList => {
          const nextPageToken = errorList.data.nextPageToken;

          const prmAry = [];
          const upd = fsdb.batch();
          errorList.data.resources.forEach(prod => {
            if (
              prod.itemLevelIssues &&
              prod.itemLevelIssues.some(issue => issue.code === 'policy_violation')
            ) {
              const pid = prod.productId;
              console.log('VIOLATION: ', pid);
              const parsedPid = pid.split(':');
              const issueRef = feedIssuesRef.doc(parsedPid[3]);
              upd.set(issueRef, { pid: parsedPid[3] });
              prmAry.push(deleteFeedItem(pid));
            }
          });
          Promise.all(prmAry)
            .then(results => {
              // console.log('feed updated!');
              upd.commit().then(() => {
                // console.log('issues updated!');
                if (nextPageToken) {

                  const pubsub = new PubSub({ projectId: 'my-project-id' });
                  const dataBuffer = Buffer.from(storeID);
                  const customAttributes = { pageToken: nextPageToken };

                  pubsub
                    .topic('my-special-topic')
                    .publisher()
                    .publish(dataBuffer, customAttributes, (err, messageID) => {
                      res.status(200).end();
                    });
                } else {
                  console.log('no more tokens');
                  res.status(200).end();
                }
              });
            })
            .catch(err => {
              console.log('error in feed update! ', err);
              upd.commit().then(() => {
                console.log('issues updated anyway!');
                res.status(200).end();
              });
            });
        })
        .catch(error => {
          console.error(error.code);
          console.error(error.errors);
          return res.status(200).end();
        });
    });
});

This function takes a config id as a query parameter, looks up the service account info, and attempts to connect to the google content api to check for feed issues. when this function is called within the same instance for two different service accounts, all but the first will throw permission errors "'User cannot access account nnnnnnn'. These all are verified to work when called independently in their own app engine instance, but when called back to back or simultaneously, the credentials are being cached somewhere, even though we are creating a new instance of the GoogleApis object.

[klassicd]

This is related to https://github.com/googleapis/google-auth-library-nodejs/issues/390

As mentioned in the thread, you can instantiate a new GoogleAuth() object each request. Both libraries can be used together

import { google } from 'googleapis';
import { GoogleAuth } from 'google-auth-library'; 
const auth = new GoogleAuth({
  credentials: {/* Credentials Here */},
  scopes: ['https://www.googleapis.com/auth/calendar']
});
const client = await auth.getClient();
const calendarClient = google.calendar({
  version: 'v3',
  auth: client
});

[nikmartin]

@klassicd I changed my above function to this:

const auth = new GoogleAuth({
    credentials: {
      client_email: config.content_api.credentials.client_email,
      private_key: config.content_api.credentials.private_key.replace(/\\n/g, '\n').trim(),
    },
    scopes: ['https://www.googleapis.com/auth/content'],
  });
  const cli = await auth.getClient();

  const g = new GoogleApis();

  const content = g.content({ version: 'v2', auth: cli });

And it still seems to be caching if run from multiple service accounts inside the same app engine instance. I still get 1 successful auth, and failed auth on each subsequent connection to a different service account. The error returned from the content api is reason: 'auth/account_access_denied', { domain: 'content.ContentErrorDomain'}

[nikmartin]

I may have discovered my issue, if i pass in the client auth object with every request, not just when the API object is created, it seems to behaving much better. Otherwise each command will also use some cached auth client which may be the wrong one. For instance

content.products
      .delete({ merchantId: config.content_api.merchant_id, productId: pid })
      .then(_ => {
        console.log(`Deleted MC Feed: ${pid}`);
      })
      .catch(err => {
        console.log(`NOT DELETING ${pid}`);
        console.log(err.code);
        console.error(err.errors);
      });

becomes

content.products
      .delete({ merchantId: config.content_api.merchant_id, productId: pid, auth: cli })
      .then(_ => {
        console.log(`Deleted MC Feed: ${pid}`);
      })
      .catch(err => {
        console.log(`NOT DELETING ${pid}`);
        console.log(err.code);
        console.error(err.errors);
      });

And all seems to be good now.

[filecage]

@nikmartin I can not confirm that behaviour. The auth does not change when you change the way of passing it to the client as the global context and the per-call parameters are simply being merged by Object.assign shortly after the invocation of an API's method (that happens here: https://github.com/googleapis/nodejs-googleapis-common/blob/master/src/apirequest.ts#L78-L83)

As I've mentioned in my comment of googleapis/google-auth-library-nodejs#390 this is due to the singleton usage of the GoogleAuth class here and here.

The easiest fix of the whole issue would be to remove the google.auth singleton completely and create all future auths by using GoogleAuth in userland like the workaround that has been posted.

It's a code change from:

async createAuth(options) {
    return await google.auth.getClient(options);
}

to

async createAuth(options) {
    return await (new GoogleAuth(options)).getClient();
}

Have you tried investigating why this didn't fix your problem?

[chscott]

@nikmartin Thanks a ton for posting your solution about sending the auth object with each request. I lost an entire day yesterday to this issue. I think the official samples need to be cleaned up to illustrate this technique, as it breaks with any trivial amount of concurrency (two overlapping calls, in my case).

[bcoe]

@nikmartin @filecage :wave: this and a couple related issues motivated me to refactor how our auth works a bit. While getClient() still works (and should no longer have side effects), I recommend that for calling an API in multiple contexts (with different credentials) you instead use this approach:

const auth = new google.auth.GoogleAuth({
  // Scopes can be specified either as an array or as a single, space-delimited string.
  scopes: ['https://www.googleapis.com/auth/compute']
});
const authClient = await auth.getClient();

[filecage]

@bcoe seems to be a very sane way of interfacing the auth client. Thanks a lot!