Firebase-functions: Cannot read property 'name' of undefined

[iouhammi]

Hello,

I start getting this error on my cloud functions logs interface:

TypeError: Cannot read property 'name' of undefined
    at Firestore.snapshot_ (/user_code/node_modules/firebase-admin/node_modules/@google-cloud/firestore/build/src/index.js:422:127)
    at beforeSnapshotConstructor (/user_code/node_modules/firebase-functions/lib/providers/firestore.js:140:30)
    at changeConstructor (/user_code/node_modules/firebase-functions/lib/providers/firestore.js:144:49)
    at cloudFunctionNewSignature (/user_code/node_modules/firebase-functions/lib/cloud-functions.js:104:28)
    at cloudFunction (/user_code/node_modules/firebase-functions/lib/cloud-functions.js:135:20)
    at /var/tmp/worker/worker.js:768:24
    at process._tickDomainCallback (internal/process/next_tick.js:135:7)

This is happening sometimes but frequently. I set a console log on the top of my function, but it doesn't seem to be logged, so the function does not even started.

Environment:

"dependencies": {
    "cors": "^2.8.5",
    "express": "^4.16.4",
    "firebase-admin": "~6.0.0",
    "firebase-functions": "^2.1.0",
    "lodash": "^4.17.11"
  },

This issue begins in the same time as the issue mentioned here 357. But I cannot confirm it's related to. Maybe both bugs are due to a change made by Firebase.

[maxnucci]

This is also affecting us. Along with #357

[rmacfie]

Same for us.

And as @iouhammi wrote, this error occurs before the functions enters our code, which makes it much harder to find a work-around.

[wsh]

Hey, we're so sorry about this; just so you know, it's a known issue and we're rolling back now to fix it. Thanks for your patience.

[iouhammi]

The issue seams to be solved on my side, as soon as the incident was solved by firebase.

[joshdifabio]

Hey, we're so sorry about this; just so you know, it's a known issue and we're rolling back now to fix it. Thanks for your patience.

Firestore is an amazing product, but this incident is extremely worrying given the explanation on the incident page:

An upgrade of our infrastructure did not support a feature we thought was no longer in use

How was this able to happen? Are you guys not testing the whole stack before you deploy to production? I would appreciate some reassurance here, as would others I'm sure.

Furthermore, I hope you appreciate the severity of this incident. Applications which use Firestore and depend on at-least-once delivery of events will now be in a corrupted state. Actual downtime would have been much less damaging in many cases.

[rmacfie]

I agree with @joshdifabio. The optics here are quite bad.

1. Rolling something out on a Friday afternoon

2. Not monitoring what must have been a significant increase in errors for a specific trigger type right after the rollout

3. The very vague and almost nonchalant explanation.

As mentioned, due to the mechanics of the triggers this has likely caused a lot of corrupted state and nasty errors that will creep up in the coming days or weeks, for your customers.

It would be very appreciated with a bit of transparency on what happened and what you're doing to prevent similar problems in the future.

[thechenky]

Hi all, confirming that this was an issue with a rollout on our side, which we have now rolled back. I understand your frustration and apologize about the incident. We will be investigating into the reasons why this occurred. Internal bug reference: 121064658

[errorhandler]

I'm still seeing this behaviour on a function which has retry on failure enabled. Is such behaviour still expected?

[joshdifabio]

I'm still seeing this behaviour on a function which has retry on failure enabled. Is such behaviour still expected?

I got the same. I had to disable retries for each function temporarily, which was extremely time consuming and is obviously not really safe to do in production.

[thechenky]

@errorhandler @joshdifabio thanks for the update, I'm checking in on this.

[swftvsn]

Same here, we're failing catasthropically, there are over 30 000 failures this morning due to this broken context parameter parsing!

[swftvsn]

And our data is flying all over the place, there are now multiple orders going to "undefined" due to this!
It'll take weeks to parse all this ffs.

[swftvsn]

Hmm, sorry for the bad lingo in previous message - to elaborate more, I'm seeing #357 still happening

[swftvsn]

Also there's no mention in the status page about this service distruption - how much does not get logged to that page at all? This does not build trust.

[inlined]

@swftvsn New events should have the parameters back. if you have retries enabled and the old events causes your code to crash, then you will have the backlog of bad events for 7d. Fortunately, there are workarounds you can add to your code to handle the events _without_ turning off retries. One such method is on StackOverflow

Another is to parse the params yourself. Here's a snippet to fix the context.params object for any affected Firestore functions. A lesser-reported issue is that context.resource was reverted to a string instead of an object with string fields name and service; that should be equally easy to normalize too.

function guardIssue358(fn, context) {
  if (typeof context.params === 'undefined') {
    context.params = {};
  }
  if (Object.keys(context.params).length != 0) {
    return;
  }
  // slightly modified version of cloud_functions.ts:_makeParams
  const WILDCARD_REGEX = new RegExp('{[^/{}]*}', 'g');
  let triggerResource = fn.__trigger.eventTrigger.resource;
  let wildcards = triggerResource.match(WILDCARD_REGEX);
  if (wildcards) {
    let triggerResourceParts = triggerResource.split('/');
    let eventResourceParts = context.resource.split('/');
    for(wildcard of wildcards) {
      let wildcardNoBraces = wildcard.slice(1, -1);
      let position = triggerResourceParts.indexOf(wildcard);
      context.params[wildcardNoBraces] = eventResourceParts[position];
    }
  }

// Usage:
exports.myFirestoreFunc = functions.firestore.document('some/{path}/with/{params}').onCreate((data, context) => {
  // This should be your first line until the backlog is cleared.
  guardIssue358(exports.myFirestoreFunc, context);
  // leave the rest of your code unchanged ...
});

[swftvsn]

@inlined The problem is that we see this, quite constantly, on new data also. I think we need to start migrating our whole codebase at this point to use the work-around.

[swftvsn]

@inlined scrap that, no, we are NOT seeing this on new data.

[swftvsn]

This is OK for us also, so for me it is ok to close this ticket.

[thechenky]

Thanks for everyone's patience, going to close this out. As always, if you see any more problems, please feel free to open up another issue.

[joshdifabio]

Buried.

[mtzfactory]

Hi, is still happening...

[thechenky]

@mtzfactory please file a new issue if you're seeing issues and fill out all the required fields.

[javiermuniz]

For anyone still getting this, the solution in #528 fixed it for me.

[ollydixon]

Happening to me this morning. The client's app is now crippled. It's firing this error before we get into the function. To be clear this is triggers that it's happening on, specifically functions.firestore.document