Vercel: Not seeing logs in zeit 2.0

Created on 30 Nov 2018  Β·  29Comments  Β·  Source: vercel/vercel

Not much to say. Here's the public url: https://zeit.co/morgs32/stackshirts/ekns24hcx/logs

And here is my now.json:

{
  "version": 2,
  "public": true,
  "builds": [
    { "src": "/admin/package.json", "use": "@now/node-server" }
  ]
}

If I have devtools open while the app is "initializing", then I see the website polling for data from within _error.js

image

bug

Most helpful comment

(TLDR at the end)

Hey everyone,

In the past few months you have been experiencing poor reliability of Now 2.0 deployment logs – many logs were missing and, quite often, slow to load.

I'm more than happy to share that, as of last week, we have deployed very significant improvements to how we collect and store logs,

The problem

The reliability were issues due to a few limits we've encountered on the cloud providers we use for Lambda functions – we found that the way they buffer logs from Lambda functions doesn't scale to the volume of logs we were processing. That would cause two major issues:

  1. Logs would be dropped from the buffers and never delivered to our pipeline – that would cause logs to not be collected by our pipeline (a.k.a. "missing logs")
  2. The buffers would behave inconsistently: sometimes they would have buffer large quantities of logs (which is ideal) and sometimes they would buffer very small quantities of logs, which would cause large load on our pipeline. This would cause logs to be slow to load (and, in some cases, intermittently unavailable)

The solution

After close collaboration to said cloud providers, we have determined that the best course of action was to completely bypass their solutions for Lambda functions logs streaming and buffering, and develop our own: they do acknowledge that changes are needed in order to enable use cases like ours, but such changes aren't high priority for them.

With that in mind, we are now collecting the logs for Lambda function executions at the same time we collect the responses – at execution time. We then insert the logs into a highly performant data stream processor which reliably buffers the logs as per our configuration. In another words:

  1. Logs will no longer be dropped somewhere in the pipeline: they are collected and persisted right at execution time, excluding the possibility that we would invoke a function and not persist the resulting logs
  2. The buffers placed between the stdout/stderr of the Lambda functions and our long-term log storage are now behaving reliably and consistently, removing the unpredictable extra load from our long-term storage – accessing logs is now consistently snappy.

Future improvements

  1. We are working on short-circuiting the logs pipeline for real time streaming: if you are looking at the logs on the ZEIT Dashboard or using now logs -f in the CLI, we'll make that more real-time by cutting some corners in the pipeline.
  2. Log Drains: you'll be able to specify log drains for deployments under a project, so that you can bring your own log processing solution (Splunk, Papertrail, and friends). More news on that coming soon!

TLDR

We have made significant improvements to our log pipeline to ensure that logs are no longer missing and are always quick to access. In the coming months we will also improve real-time access and add Log Drains so that you can bring your own log processing solution.


I'm reopening the issue for visibility. Everyone is more than welcome to post any issues, questions, etc – I'll be around for anything related to logs :)

All 29 comments

Same here. All my logs are empty. Can't see why my app is failing (!)

https://zeit.co/ecl/next-js-testapp/68m3xpj1o

Experiencing same issue!

Same issue !!
now logs xxx.now.sh also returns nothing

I can't see anything either

FWIW, I was experiencing this yesterday. I believe it may be due to the 1,000 line limit on the free plan. I upgraded to the unlimited plan (and entered my credit card info) and I'm not experiencing the issue today.

https://zeit.co/pricing

This is super annoying. Most of the time (like 80%) I got 404 when my build fails and I can't even check the logs to see what happened (except for the CLI)

I have the paid tier and this is a consistent issue. It's a shame because I was trying to sell in Now at work : (

Same, happens every time.

We have made very significant upgrades and improvements to the logging infrastructure. You should be seeing build and runtime logs with no issues.

We will close this issue once the work on this area completes. Thank you for your patience!

looking forward to it!

@ekkis are you still not seeing logs?

I'm still using the GRU hack but I do see them now. a little slow but that's different. I meant I'm looking forward to the significant improvements in the platform

After redeployed my app (b), I'm facing the same issue. I do get build logs, but no application logs.

When I update the alias to the previous deployment (a) logging is working again. Switching again to the new deployment (b) does still not log any application logs.

a) octolinker-90txf3g39.now.sh (prints build and application logs)
b) octolinker-b1s50owwu.now.sh (just build logs)

@stefanbuck are you using Now 2.0 or 1.0?

2.0

We're on it. I'm so sorry about this, please expect news _very_ soon.

Hi, is this still broken?

It seemed like it was working again for a while. I saw some runtime logs from earlier today, and now I can only see the Build logs again.

Any updates on this? I have no logs displaying.

@juliangruber is there an ETA on a resolution?

@jrschumacher I'm not maintaining this any more. Sorry!

@rauchg is there an ETA on a resolution?

We fixed the issue and you should have logs by now... Did you redeploy @jrschumacher ?

@paulogdm still having the same issue of no runtime logs, just build logs.

@gtothesquare Did you redeploy? This is supposed to be 100% fixed now.

Closing. Let me know if you still face any issues.

@paulogdm yes i redeployed and when i posted the comment it didn't work. What I had to do (maybe it was coincidence) was delete the project wait a bit till the project was actually deleted and redeploy. Can confirm it works.

(TLDR at the end)

Hey everyone,

In the past few months you have been experiencing poor reliability of Now 2.0 deployment logs – many logs were missing and, quite often, slow to load.

I'm more than happy to share that, as of last week, we have deployed very significant improvements to how we collect and store logs,

The problem

The reliability were issues due to a few limits we've encountered on the cloud providers we use for Lambda functions – we found that the way they buffer logs from Lambda functions doesn't scale to the volume of logs we were processing. That would cause two major issues:

  1. Logs would be dropped from the buffers and never delivered to our pipeline – that would cause logs to not be collected by our pipeline (a.k.a. "missing logs")
  2. The buffers would behave inconsistently: sometimes they would have buffer large quantities of logs (which is ideal) and sometimes they would buffer very small quantities of logs, which would cause large load on our pipeline. This would cause logs to be slow to load (and, in some cases, intermittently unavailable)

The solution

After close collaboration to said cloud providers, we have determined that the best course of action was to completely bypass their solutions for Lambda functions logs streaming and buffering, and develop our own: they do acknowledge that changes are needed in order to enable use cases like ours, but such changes aren't high priority for them.

With that in mind, we are now collecting the logs for Lambda function executions at the same time we collect the responses – at execution time. We then insert the logs into a highly performant data stream processor which reliably buffers the logs as per our configuration. In another words:

  1. Logs will no longer be dropped somewhere in the pipeline: they are collected and persisted right at execution time, excluding the possibility that we would invoke a function and not persist the resulting logs
  2. The buffers placed between the stdout/stderr of the Lambda functions and our long-term log storage are now behaving reliably and consistently, removing the unpredictable extra load from our long-term storage – accessing logs is now consistently snappy.

Future improvements

  1. We are working on short-circuiting the logs pipeline for real time streaming: if you are looking at the logs on the ZEIT Dashboard or using now logs -f in the CLI, we'll make that more real-time by cutting some corners in the pipeline.
  2. Log Drains: you'll be able to specify log drains for deployments under a project, so that you can bring your own log processing solution (Splunk, Papertrail, and friends). More news on that coming soon!

TLDR

We have made significant improvements to our log pipeline to ensure that logs are no longer missing and are always quick to access. In the coming months we will also improve real-time access and add Log Drains so that you can bring your own log processing solution.


I'm reopening the issue for visibility. Everyone is more than welcome to post any issues, questions, etc – I'll be around for anything related to logs :)

Closing due to age, and because logs on Now 2.0 are once again working.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

olivierlesnicki picture olivierlesnicki  Β·  36Comments

gadicc picture gadicc  Β·  48Comments

CaptainChemist picture CaptainChemist  Β·  40Comments

jberglinds picture jberglinds  Β·  59Comments

rgwozdz picture rgwozdz  Β·  40Comments