Eleventy: RangeError: Maximum call stack size exceeded

[robdodson]

I think I have a somewhat gnarly 11ty bug but I feel a little sheepish about putting it on GitHub because it's specific to how we are doing things (very weirdly) in our project.

I have a few levels of nested loops to zero in on a post (related to #399). When I've found the post I'm calling a shortcode. But now I'm getting a Maximum call size exceeded error.

If I remove one doc it works, so it's definitely a thing where the call stack is just building and building and one extra doc puts it over the edge.

I tried to create a reduced test case in this branch:
https://github.com/GoogleChrome/web.dev/tree/broken-audits

This is where things are going bad:
https://github.com/GoogleChrome/web.dev/blob/broken-audits/src/site/_includes/path.njk#L45

The loops are iterating over the slugs in this data file:
https://github.com/GoogleChrome/web.dev/blob/broken-audits/src/site/content/en/lighthouse-performance/lighthouse-performance.11tydata.js

I added one called 'foo' which pushes my machine over the limit. Remove that and things compile fine.

I added a debugger statement to the shortcode in question.
https://github.com/GoogleChrome/web.dev/blob/broken-audits/src/site/_includes/components/AuthorInfo.js#L8

If you do npm run debug and then open chrome://inspect you should be able to poke around. Hit play a few times and the call stack will keep growing. It seems like it keeps adding async iterators in nunjucks but I'm not very familiar with that's happening there.

[robdodson]

One data point worth mentioning, if I swap this shortcode out for a macro that does the same thing it doesn't hit the call stack limit. I'm not sure if that's because something is fundamentally different, or if the macro is just marginally fewer function calls and therefore just deferring the problem till the number of docs grows more.

[zachleat]

Hmm, this is a tough one Rob. It looks like this might be a fundamental limitation with Nunjucks, maybe? It looks like they’ve had a few one off fixes trying to mitigate the problem, dunno if you perused their issue tracker or not: https://github.com/mozilla/nunjucks/issues?utf8=%E2%9C%93&q=is%3Aissue+Maximum+call+stack+size+exceeded

What are you trying to do in that path.njk template? You may have a few paths forward here, if any of them interest you:

• Can you refactor those loops so you’re not 3-deep in nested loops there?
• Can you move that loop logic into your own JavaScript function to manage that complexity outside of Nunjucks?
• Can you refactor this template to use Liquid or a raw JavaScript template instead 😈?
• Or use a macro as you suggested?

Although now that I’ve typed all that, it does fail on the {% AuthorInfo {author: authors['robdodson']} %} line and not on the loops, so it could very well be a limitation with Eleventy shortcodes. I’ll dig a bit deeper.

[robdodson]

Can you refactor those loops so you’re not 3-deep in nested loops there?
Can you move that loop logic into your own JavaScript function to manage that complexity outside of Nunjucks?

Yep I think I found a nicer way (I hope) of doing it by memoizing collection.all and turning it into a lookup table. Since slugs are unique on our site (or they will be soon) then I can just grab the item by its slug. Detailed here: https://github.com/11ty/eleventy/issues/399#issuecomment-484692942

Although now that I’ve typed all that, it does fail on the {% AuthorInfo {author: authors['robdodson']} %} line and not on the loops, so it could very well be a limitation with Eleventy shortcodes. I’ll dig a bit deeper.

This was my concern. If you follow the debug steps I mentioned you can start to see the call stack in Chrome slowly grow. I wasn't sure if maybe it had to do with how eleventy was triggering the shortcodes.

It looks like they’ve had a few one off fixes trying to mitigate the problem, dunno if you perused their issue tracker or not

I'll do some digging over there. My main concern was if I was doing something fundamentally wrong and someday I'll hit a certain number of posts and the whole thing will fall over. This started to occur at around 98 posts I think, which seems way too low to be an actual limit and made me suspect there might be a bug somewhere.

[zachleat]

Hmm, well it looks like this is likely a problem with Nunjucks CallExtensionAsync for custom tags. Switching these to use the Sync variant runs the web.dev build correctly.

Looking at the code here, it doesn’t even seem necessary to use the async method in our shortcode implementation as we don’t take advantage of async features there. However, this does have possible future implications for #429. But we can cross that bridge when we come to that feature.

https://mozilla.github.io/nunjucks/api.html#custom-tags

I’m checking in the fix for this and did test it successfully against your broken-audits repo but if you could point your npm install to this repo (master branch) and test the upcoming 0.8.3 to verify, I’d really appreciate it!

[robdodson]

yep the latest commit fixes it for me :)

Thanks for taking the time to look into this!

[zachleat]

I should be clear, I did not conclusively discover whether or not this was a fix or just a mitigation. But if it is just a mitigation hopefully it buys enough time until Nunjucks can have a look at this on their end.

[robdodson]

Understood. Yeah I'm also planning to switch over to using the lookup I mentioned in #399 to avoid doing the nested loops. For the use case we're trying to solve for, the loops felt inefficient anyway.

So hopefully avoiding all the nesting will help us avoid the footgun.

[fdintino]

I only just now came across this old bug report. I have been working on nunjucks v4, which replaces the hacky loop logic with generators (or the polyfill equivalent, where native support isn't available), which seems to me the only way to fix all these looping bugs, make async a first-class citizen in nunjucks, and to allow things like loop recursion and control tags (e.g. breaks and continues). It also allows me to pattern the template execution logic more closely to jinja2, which is far more battle-tested than what we currently have.

Unfortunately for nunjucks (though fortunately for me) I'm about to be a first-time father, so I've had a few distractions that have kept me from my work. I can't 100% promise that I'll finish sometime soon, but if I half promise, perhaps that will give me the motivation I need to get it done.