Vercel: logs prematurely rate limited

[brandonmp]

Per the web dashboard, I'm using 134kb / 1 GB of logs

after deploying a project, though, now logs <deployment> is printing:

03/02 08:17 PM  Logs from deployments of your account are rate limited. Please upgrade your plan.
03/02 08:18 PM  Logs from deployments of your account are rate limited. Please upgrade your plan.

is this suppressing log content?

[rauchg]

@brandonmp We are about to deploy a large infrastructure improvement to the logging engine. You'll see drastically increased logging throughput and superior scalability.

This is only rate limiting _frequency_, not total amount. The higher the plan, the lower the rate limit.

[rauchg]

I'll keep this open until we deploy that improvement. cc @igorklopov

[brandonmp]

thanks @rauchg . I didn't know logging frequency was a constraint. is the limit on a per-instance basis?

so those messages mean my deployment was printing to stdout at too high a rate?

&& if I see this message, does this mean logged output above the rate limit has been discarded? if not, at what point would I start losing data?

[den1k]

@rauchg we're hitting this with every deployment, because we're printing a few hundred lines. Doesn't seem like a reasonable limit and our overall log usage is tiny:

Our builds are getting stuck and keep printing
Deploying image over and over again. Sometimes the 4th of 5th goes through after wiping previous ones and re-deploying. What's going on?

[dmtrKovalenko]

The same for me, actually impossible to deploy 😕

[shawninder]

I am also getting this issue. Dashboard is showing 28KB / 100MB logs usage, and my deployment is failing:
```04/17 11:02 PM (30m)
npm install
04/17 11:02 PM (30m)
✓ Using "package-lock.json"
04/17 11:02 PM (30m)
⧗ Installing 32 main dependencies…
04/17 11:02 PM (30m)
npm run build
04/17 11:02 PM (30m)

[email protected] build /home/nowuser/src
next build
04/17 11:02 PM (30m)

Failed to build
04/17 11:02 PM (30m)
Logs from deployments of your account are rate limited. The reason is "writes from sublime-music-oeotvdzzdu.now.sh". Please upgrade your plan.```

[Kikobeats]

same error, actually my app doesn't print error

[shawninder]

I found an unfortunate "solution" for this (at least in my case): I solved my problem by deploying my app to Heroku...

This also didn't work but I got much better error logs (some instead of none) and was thus able to solve my problem (in the end I was requiring a file that wasn't committed to GitHub, super easy to fix with logging). Once I got it working on Heroku, it started working on now as well. Problem solved.

For this particular problem, cloning the repo afresh and trying it out would also have identified the problem as well, but I can imagine other cases where this wouldn't work, like cases where environment variables exist locally but not on the now server, for example. In these cases, it would also be nice to see all the error logs from the server.

In the end it all comes down to getting the logs you need to identify and solve the problem. It seems you can make an app that logs absolutely nothing and still busts the log rate limit because of the logs generated by errors, and this is made worse by the fact that you can't even see the logs that tell you what went wrong.

@rauchg I'd be curious to try this thing again once you finish this "large infrastructure improvement to the logging engine" you mention. I could then confirm that the improvement solves my problem, or not. Think you could poke us here when it's done? Or link to the issue which I could then star and follow?

[develomark]

I am also having this problem. I cannot deploy and am getting this message: Logs from deployments of your account are rate limited.

I've got about 0.1% logs of my allocation.

So many problems deploying.

[brandonmp]

@develomark suboptimal workaround for sure, but when I deployed my project using pino as a logging solution with pino-syslog piping the logs to Papertrail, i was able to skip this error

of course, it skips it b/c pino-syslog swallows your stdout content and forwards it _all_ to papertrail, but in a pinch, it did the trick

[develomark]

Thanks @brandonmp I'll give that a try.

[ntziolis]

@rauchg Any update on the improvement? We keep hitting this on EVERY deployment and our CI completely breaks down as this causes the jobs to fail at some point

[Peakdrum]

I have problem on this as well I tried
now rm <my-project-name> --safe
so that all my version is gone but the problem still appeared

[cristoforo]

Same issue on every deployment.
And this is not even listed as a limitation in the pricing/plan page!

[styfle]

Everyone should click the thumbs up icon 👍 in the first post (I only see 2 votes right now). The more votes, then this issue will rise to the top and maybe get some visibility.

[aboutlo]

I'm forced to send the logs to cloudwatch... 🤦‍♂️
This is a serious issue, we can't debug

[develomark]

This is killing me. I have barely any usage and I cannot see the error that is causing my deploy to fail.

[rhysdavies1994]

Like @rauchg said, the rate-limit has do to with the FREQUENCY of logs, not the TOTAL AMOUNT of logs. e.g 5 logs a second.

I did notice with logs that are over multiple lines (any javascript object), each line has its own timestamp. So is each line counting as 1 log request. e.g 20 lines = 20 log requests? is this true?

So @rauchg what is the rate-limit for each tier?

[ntziolis]

Whatever the rate limit, its obviously the wrong metric /threshold being used if PAYING customers run into it continuously.

[boyhagemann]

Does anyone even know what these rate limits are? Seems very arbitrary to me when this limit is reached.

[jsdario]

Paying customers should be advised of such limit _before_ payment. Other limits are reasonably specified before the contract takes place.

Don't really know which is the recommended way to avoid it.

[edcarstens]

I'm now getting this too and it is impossible to continue using Now to deploy. I'll have to look for some other way to test and debug my game app. The messages refer to old deployments that I have removed.

[yieme]

As a paying customer, using a fraction of the log limit the service became unusable because we couldn't see error messages. Switched some instances to use external logging services, but a pain for a platform that has developer experience as a value. It would have been one thing for a particular offending instance that was over some limit to be blocked, but all instances being blinded is awful.

[bitfede]

Also having this issue, really killing my development efforts.

I only used 2MB/100MB and I get the same logs warnings.. Also not a single of mention of "rate limits" in the pricing or info pages.

What an unpleasant surprise... Also nobody of their team is answering this thread so if any dev finds himself stuck without help, here are some alternatives to deploy your app and continue working on your project:

https://nanobox.io/

https://openode.io/ (Super cheap node hosting service)

https://www.digitalocean.com/ (Affordable VPS used with tools like nanobox.io)

https://www.digitalocean.com/products/one-click-apps/node-js/

https://glitch.com (Service built by fog creek software, trello people) for launching and hosting MVPs

https://serverless.com/framework/

https://cloud.google.com/appengine/

https://stdlib.com (Similar to serverless)

https://surge.sh/ static publishing for front end devs

https://aws.amazon.com/elasticbeanstalk/ (For more advanced use...I think)

Hope somebody from the NOW staff wakes up from this big sleep and starts helping us out instead of just asking for money.

[joecohens]

Hi everyone,

I’m really sorry for the delayed response on this issue. We are in middle of refactoring the log pipeline that will also increase the rate limit. It’s important to note that this issue only happens if your app logs too much information at a very frequent rate.

We will get back to you as soon as this issue addressed.

Thank you very much for you patience.

[revskill10]

I couldn't deploy any app now, even i have no problem with log.
I can't trust the service in production if a breaking change could cause this without any announcement.

[ghost]

I got this issue when doing a second attempt at deploying the official Docker image for PostgreSQL (the first time the build failed).

[tomredman]

Same challenge here... stopped me in my tracks. ¯\_(ツ)_/¯

What exactly _is_ the rate limit? I'm just trying to work through a few Dockerfile issues and now I'm a bit stuck.

[leo]

Thank you so much for the effort you have put into this issue! 😊

However, since we completely re-worked the entire codebase of Now CLI and this issue does not apply to the new code anymore, I'm closing it now.

You can read more about our biggest release yet here: https://zeit.co/blog/now-2