Build: Improve Error DX

[DavidWells]

There have been some comments around error boundaries & general feedback to devs when errors occur.

Here is an example of a plugin with broken code.

module.exports = {
    name: 'netlify-plugin-one',
    onInit: () => {
      console.log(thing.what) // undefined ref
    },
}

or

module.exports = {
    name: 'netlify-plugin-one',
    onInit: () => {
      throw new Error('http://www.nooooooooooooooo.com/')
    },
}

Currently errors materialize like so:

Are there ways we can improve upon this?

[jlengstorf]

stoked to be talking about this!

I have a lot of different things that I think might be worth discussion, so let me know if we need to break these out into sub-discussions

at a high level, the main issues I see are:

• logs are hard to read and full of unhelpful output
• failures look the same no matter where they came from
• there is no indication in the UI that the error wasn't Netlify's fault
• there is no way to recover from build plugin errors

logs are hard to read and full of unhelpful output

the logs are made of about 95% junk in terms of useful information — we could leave out almost all of it and I'd have the same idea of what's going on

because we output so much junk, when a build fails and I go to the log and click the "jump to bottom" arrow, I don't see the actual error; I see the Build script returned non-zero exit code: 1

someone who is familiar with how Netlify works and has all the context that "oh, no, don't look at the last error, scroll up and look for the error before that" will figure this out, but new customers may not get that far — they might just contact support and/or churn because "Netlify isn't stable"

which leads to the next point:

failures look the same no matter where they came from

right now, when you look at the app, a failure just looks like a failure — there's no way to tell _why_ the thing failed without digging into the logs

this means that a build plugin makes our UI look the same way as a Netlify build error — how would our customers know not to ask support when they have a half dozen failures in a row because some plugin changed in the repo and they weren't aware?

an idea to improve it

if we use our existing badge markup, we could potentially just add another clarifier to failed builds:

this would require build plugins to capture errors and communicate them out of the build

there is no indication in the UI that the error wasn't Netlify's fault

a failed build should add some kind of big blinking marquee at the top of the build log — not just in the log — that something went wrong

right now the visual language at the top of the build log is so similar that I actually thought it was the same output between successful/failed builds until I looked just now

the build should capture errors — _especially_ if those errors come from build plugins — and expose them in a big-ass red box so people know exactly what went wrong:

there is no way to recover from build plugin errors

this may not be feasible, but in a perfect world we'd be running build plugins in a way that if they failed we could just say, "okay, this build plugin's busted, don't run any more of its lifecycles and let's keep rolling with this build"

I'm happy to be convinced otherwise, but by default I would think we'd want build plugins to fail in a recoverable way, meaning we don't actually fail a build if the plugin errors out — we'd just expose warnings and ship the site _without_ the build plugin

for plugins like build time speedup plugins, this maintains great DX vs. costing even more time (i.e. I can ignore the failed builds until I have room to breathe and dig in vs. needing to drop everything because the plugin broke and our site won't build and there's something critical we need to deploy)

if a plugin should _definitely_ fail the build if it errors out — for compliance checks or perf budgets, for example — they would explicitly opt in to fail builds on error with a setting (ideally we have a utility that they call so they can conditionally full bail, kind of like ESLint warnings vs. errors)

in very simplified pseudo-code, the logic might look like this:

let failedPlugins = [];
let pluginErrors = [];
buildPlugins.forEach(plugin => {
  // bail on future lifecycles if the plugin has already thrown an error
  if (failedPlugins.find(p => p.plugin === plugin.name)) return;

  try {
    plugin[currentLifecycle]()
  } catch (error) {
    if (plugin.config.failBuildOnError) {
      // only kill the build if the plugin _explicitly_ says it should die on failure
      noMrBuildIExpectYouToDie({ cause: plugin.name, error: error.message })
    }

    // otherwise we just keep records and share those after the build succeeds
    failedPlugins.push({ plugin: plugin.name, lifecycle: currentLifeCycle })
    pluginErrors.push({ plugin: plugin.name, error: error.message })
  }
})

---

this is A Lot™, so let me know if you want to dig into these separately or get on a call to discuss

[ehmicky]

Thanks for spending time on that feedback. Let's improve this! :)

Original message from David

Do you have some ideas on how we could improve it?

Logs are hard to read and full of unhelpful output

I am also a fan of removing unnecessary output where possible. The screenshot you are showing are logs from the buildbot, not @netlify/build, so that discussion would need to be moved there. Is there something from @netlify/build logs in particular that could be simplified or removed?

Failures look the same no matter where they came from

Very good point. Specifying some kind of error type in the UI would be great.

There is no indication in the UI that the error wasn't Netlify's fault

I agree a more visible error UI would be a good idea.

There is no way to recover from build plugin errors

I agree that we need a way for plugins to not make the build fail on errors.

Now we might want to be careful when doing so:

• If a plugin fails, there's a chance that the resulting build would be erroneous. Making the build succeed might lead to deploying to production a Site that's invalid.
• Making the build fail makes it visible to the users that something is wrong with some of their plugins. Otherwise they might overlook it.

Based on this, I would personally think that recoverability in plugins should be an opt-in not an opt-out.

We could make this an option in the plugin API, so that plugins that are known to be recoverable (such as performance plugins) can declare it. onError, onSuccess and onEnd event handlers can be used to do some logic on errors.

We could also allow users to override that with a boolean option in the netlify.yml plugins property.

[lesliecdubs]

Chiming in to +1 these suggestions! I think it's imperative we more clearly surface these errors both in the logs and the UI to avoid user confusion and churn, like @jlengstorf said:

new customers may not get that far — they might just contact support and/or churn because "Netlify isn't stable"

Could we get this work prioritized as part of the #project-build-plugins-ui scope?

Once we know how this is going to work on the backend, it'd be super helpful to get an issue opened in https://github.com/netlify/netlify-react-ui/issues so we can send the UI work through our typical process: design can take these suggestions and formalize a solution, then we can send it into frontend implementation (@drewm will lead that work), Copy Club, etc. 🙂

[jlengstorf]

thanks, @ehmicky and @lesliecdubs!

If a plugin fails, there's a chance that the resulting build would be erroneous. Making the build succeed might lead to deploying to production a Site that's invalid.

in my mind this needs to be solved — if a build plugin fails, we should roll back and continue the build as if the plugin wasn't installed. that could be brute-forced by restarting the deploy and removing the plugin config, or (ideally; maybe in the future) treating builds as immutable objects or something so that the build plugin aren't _mutating_ the build, but rather creating a modified copy that we can throw away if there are errors

Making the build fail makes it visible to the users that something is wrong with some of their plugins. Otherwise they might overlook it.

I would argue that between this and upstream failures requiring developers to drop everything and make code changes because deploys stopped working, overlooking failures is the better outcome

the UI should make it apparent that something has gone wrong, and ideally we'd be able to send out an error notification email and/or digest to make sure people see the errors

[DavidWells]

I love the level of detail in https://github.com/netlify/build/issues/711#issuecomment-575853008 fantastic. I think the feedback loop from the buildbot & the UI could be greatly improved.

There are some tricky cases to deal with here... Mainly around if a plugin fails, why did it fail, & should we continue or exit the build.

Below are a couple ways things can fail

1. Environment failures

Eg. The underlying machine can die. Git can be corrupted, incorrect node versions are specified, etc.

Not much we can do about this but report back in UI what happened, where it happened, how to fix & (if possible) automatically restart a fresh build.

2. Syntax errors & bad code

Some instances where build plugins are malformed / syntax errors should be reported back to the user. In some cases, we might be able to recover, warn the user, & disable plugins.

However, If these plugins are mission critical to the application, the entire build should likely fail. Example: A netlify CMS plugin that injects the /admin contents & routes.

We would need some sort of mechanism to determine if a failure should exit the build or ignore it. With the wide variety of errors that can occur, this might be tricky (or impossible).

3. Purposely thrown errors

Some plugins might throw errors on purpose to stop the build. E.g. noOp if files haven't changed plugins. There is discussion around how builds can exit gracefully here https://github.com/netlify/build/issues/241

In this scenario, the failure is on purpose & the plugin is doing it's job. Removing this plugin would not be the opposite of what a user expects.

---

Automatically removing plugins or ignoring errors might result in sites being published in a failed state (pages missing, redirects missing, functions missing etc)

---

In all of these cases, the feedback loop should be very very clear. What happened, what line of the build it happened on, (if known issue) how to fix, (if possible) automatically fix, (if possible) automatically retry the build.

@lesliecdubs I'd love to get more robust error messages into the build list UI.

Another area of "build feedback" is how plugins can output messages back into the netlify UI. That is defined in here https://github.com/netlify/build/issues/161

[jlengstorf]

However, If these plugins are mission critical to the application, the entire build should likely fail. Example: A netlify CMS plugin that injects the /admin contents & routes.

We would need some sort of mechanism to determine if a failure should exit the build or ignore it. With the wide variety of errors that can occur, this might be tricky (or impossible).

agreed — I buried this toward the end of my original comment, but I think build plugins should have to _opt in_ to kill the build

if a plugin should definitely fail the build if it errors out — for compliance checks or perf budgets, for example — they would explicitly opt in to fail builds on error with a setting (ideally we have a utility that they call so they can conditionally full bail, kind of like ESLint warnings vs. errors)

having an explicit way to fail builds also overcomes the next point:

Some plugins might throw errors on purpose to stop the build

instead of something arbitrary that may or may not be build-related, e.g.

if (somethingWentWrong) {
  throw new Error('oh noes!');
}

we would have a more structured error using whatever falls out of #161, e.g.

if (somethingWentWrong) {
  utils.build.failWithError('oh noes!');
}

this would presumably allow us to capture those errors in a way that's easier to surface in the UI (or at the very least allow us to make that improvement under the hood of the failWithError util later on with no user-facing API changes)

[ehmicky]

I have merged some ideas from this issue to #735.

The following issues might be of interest as well:

• allowing ignoring plugin errors: #191
• retry plugins on errors: #192

[DavidWells]

It might be possible as MVP to use output contained in logs to surface these more visible errors to users. https://github.com/netlify/netlify-react-ui/issues/3792. This would make it faster to get out the door without waiting for API updates on the backend

[DavidWells]

Updates on this here https://github.com/netlify/build/pull/773#issue-375627252

Future UI work still needed

[erquhart]

This issue has a lot of great feedback, we can use it as a parent for plugins errors in the UI. Raw mechanisms on the build side are in place via #735.

[erquhart]

Summary of a conversation w/ @rybit and @verythorough on handling unexpected failures from plugins:

• we can use Git to sandbox plugin operations by checking out a new branch for each plugin lifecycle hook and merging it back in only if it succeeds
• plugins that fail unexpectedly may result in unstable code, so hard to say if it _should_ continue
• we can get around this by making it optional for plugins to fail the build or not
• we could also give users override control
• @ehmicky mentioned the last three points above as well

Proposing the following:

• don't do any sandboxing or recovery from failures
• plugins can be enabled/disabled in the UI - this can be used to quickly get around plugin problems and continue with builds
• if a plugin is certain that it's failure should not result in a failed build, it can wrap it's own code in try/catch blocks

@jlengstorf especially interested in your thoughts here.

[ehmicky]

don't do any sandboxing or recovery from failures

Does this include not doing the Git sandboxing suggested above? I think that this Git sandboxing could create many issues:

• When builds are run through Netlify CLI, users might have uncommitted changes, be in the middle of a rebase, etc.
• This would prevent plugins running in parallel in the future.
• The sandboxing would be only partial: it would allow rolling back filesystem changes in the repository but not outside of it (which do happen) nor .gitignore'd files, nor network side effects (e.g. HTTP requests that perform actions remotely).

Personally, I do think plugin authors have enough context to know whether their plugin should fail or not on behalf of users. Plugins do many things (I/O, logs, etc.) and making the build fail is just one of the things they might explicitly want to do. For example, a testing plugin like Cypress knows for sure that if Cypress tests fail, the build should fail. Letting plugin authors decide on this would also prevent having to add the option for users to override it.

Sending PRs to plugin authors, I actually started suggested using soft failures (utils.build.failPlugin()) on some errors and got a feedback from them that they wanted to enforce the build failing instead. See this issue from @bahmutov and the related PR.

[ehmicky]

Follow-up from our team discussion: we don't want to sandbox and we are letting plugin authors decide whether to make the build fail or not, so all is good! :)

[jlengstorf]

@erquhart some thoughts:

don't do any sandboxing or recovery from failures

this is concerning because the problem here isn't that someone builds a plugin and then knows how to fix it if it breaks; it's more that someone will click to add a plugin from our UI, things will be fine, then the plugin ships an update that breaks the build and our user will have no idea why or how to fix it

plugins can be enabled/disabled in the UI - this can be used to quickly get around plugin problems and continue with builds

how are we surfacing problems? if people are going to need to parse their build log to determine where problems came from, this seems like a half-measure that will make it _appear_ to be easier to use Netlify, but ultimately add a layer of indirection that ends up letting people start fast, then hit a brick wall that they have no idea how to get around

I think if the plugins are shown in the UI along with a "working/not working" indicator, that totally works, but if the plugins are installed in one place and errors are buried in logs somewhere else and we don't have a solid recovery option in the buildbot we're going to end up with a lot of confused/sad/frustrated users

if a plugin is certain that it's failure should not result in a failed build, it can wrap it's own code in try/catch blocks

this is great, but why not just wrap the entire plugin in a try/catch inside the buildbot so we can force plugins to opt into failure?

it seems to me that a better default behavior would be:

• your builds work
• you'd like to add something to your build
• you find a plugin and install it
• that plugin doesn't work with your site
• Netlify notices and prints the error
• your build works like it did before you installed the plugin

that way the worst-case scenario is no change, as opposed to the worst-case scenario being broken builds

---

my major concern in all of this is that we're exacerbating confusion that already exists: stuff _might_ happen in our UI, or it _might_ happen in the repo, or it _might_ be handled in both places, and the only way you really know is by parsing a verbose, hard-to-follow server log

I think we could ship without this and the world won't end, but in my opinion it's tech debt that gets progressively more expensive to fix with each thing layered on top of it

(and to make sure that this isn't just me swoop-and-pooping: I 💜 you all and I think this suite of utils is dope and I can't wait to see what gets built on top of them)

[erquhart]

@jlengstorf you ruined a flawlessly executed swoop-and-poop by saying nice things at the end! 😂

Clarifications

My proposal was only addressing the question of builds continuing when plugins unexpectedly fail, the rest of your error communication recommendations have been converted to tickets for implementation.

Here are the ways plugins can fail, and the expected/proposed result:

• the plugin itself catches the failure and signals that the build should still proceed
  
  
  
  • Build proceeds, and is marked with something akin to your "recoverable build plugin errors" messaging
  
  
  • Deploy page header has whatever custom error message was passed back by the plugin through utils.build.failPlugin()
  
  
• the plugin itself catches the failure and signals that the build should not proceed
  
  
  
  • Build fails, and is marked with something akin to your "unrecoverable build plugin errors" messaging
  
  
  • Deploy page header has whatever custom error message was passed back by the plugin through utils.build.failBuild()
  
  
• Plugin fails, but the plugin does not catch the failure
  
  
  
  • Build fails, and is marked with something akin to your "unrecoverable build plugin errors" messaging
  
  
  • Deploy page header has default messaging for plugin failure
  
  

We should probably provide a stack trace in the deploy header as well, but still finalizing details on that.

Remaining issue: should unexpected plugin failures fail the build?

The problem with allowing builds to continue by default is that a plugin can do _anything_ to the cloned repo, including putting it in an unstable state. If we allow builds to continue after an unexpected plugin failure, we don't know what incomplete actions took place and what the impact on the resulting build will be. We're planning on wrapping plugins in try/catch as you would expect.

The only thing we seem to differ on is whether to make build failure opt-in or opt-out when a plugin fails (throws an uncaught error) unexpectedly. I'll distill the specific reasoning for proposing opt-out to help the conversation stay focused:

• messaging makes it clear when plugins cause a failure, and which plugin caused the failure
• enable/disable provides a quick way to get your builds going again
• utils.build.failPlugin allows plugin authors to explicitly signal that the build can continue safely even if their plugin fails unexpectedly - we can also shout about this in the plugin authoring docs to encourage it when possible

There was also plugin author feedback that they'd prefer failure-by-default for unexpected errors, but I don't recall who said that. @verythorough may know.

[jlengstorf]

all makes sense, and I'm on board. A couple wishlist items that I’d _really_ like to see in place to avoid frustration/confusion later on:

1. Make it one-click to remove and rebuild when a plugin fails

• messaging makes it clear when plugins cause a failure, and which plugin caused the failure
• enable/disable provides a quick way to get your builds going again

this makes sense, and I'll make a request since it sounds like we're already building all of this functionality:

could we make the uncaught error message say something like this?

There was an unexpected error in the plugin
`netlify-plugin-dont-touch-my-garbage`. As a
precaution, the build was stopped and nothing
was deployed.

BUTTON: **Disable this plugin and rebuild.**

clicking the button would both turn off the build plugin and restart the deploy

2. Log unexpected failures so we can identity broken plugins/opportunities to coach plugin devs

utils.build.failPlugin allows plugin authors to explicitly signal that the build can continue safely even if their plugin fails unexpectedly - we can also shout about this in the plugin authoring docs to encourage it when possible

can we make sure we're logging the failures in an analyzable way? this would help us flag and remove plugins that are outright broken, and gives us potential to provide actionable feedback to developers (your plugin has X unexpected failures with the message "Y" — consider using utils.build.failPlugin to avoid these failures)

[erquhart]

100% agreed on both insights, thank you for this.

[marisamorby]

Also see issue #1080 for additional information

[ehmicky]

I believe this can now be closed?

[verythorough]

I think as a discussion issue, it makes sense to close this now. If there are remaining smaller tasks within the comments that haven't been addressed, we should file new, focused issues for them.