Yup: Performance issues for backend use

[JaneJeon]

Hi.

I love the API of yup, and I love it SO much that I'm trying to build an isomorphic application that uses the same yup schema validation in both the frontend and the backend 🚀.

However, at the moment it's not really viable simply due to the fact that it's about one order of magnitude slower than it should be, considering what is actually running in the source code ("joi-lite"). I really don't expect ajv levels of "extreme" performance since yup is not compiling schemas, but I'd expect to at least be somewhere around joi's level.

First, the setup: https://github.com/JaneJeon/validator-benchmark (I forked https://github.com/icebob/validator-benchmark, updated the dependencies, added strict mode to the yup schema, and properly ran the async validator - I tested synchronous version, and it was literally 2x slower xP)

And, the results (run node .):
https://i.imgur.com/AuQLnTD.png

At first, I thought it was because the email/url regex yup was using was ungodly (no offense), so I thought maybe using simplified regex (like here: https://github.com/icebob/fastest-validator/blob/master/lib/rules/email.js) might help.

It didn't.

I changed the regex in both lib/string.js, es/string.js in node_modules/ and it didn't affect the performance in any noticeable way.

I messed around with sync vs. async, different ways of running the validation, etc.

Then I realized there's babel's crufty code all over the place, in both es/ and lib/. I dug around and found that babel-compiled code often has performance issues, especially regarding ES6 syntaxes.

I suspect that the default babel compilation is what's slowing down your codebase significantly since, as far as I can tell, you're only using babel for ESM syntax.

Is there any way you could directly expose the ESM code or at least use a much more efficient compiler/babel setting? People running this in node/backend just need to convert ESM -> CJS. I feel like babel config is key here.

I'd really love to use this in my backend and bring it to production! This is the last thing that's preventing me.

Thanks

[JaneJeon]

Update: after hours of bashing my head against the wall, I hacked both yup (which was a pain since ESM doesn't support import foo from './blah' and import { getter } from 'property-expr' syntaxes) and validator-benchmark (and its dependent library, benchmarkify) to use strictly ESM. As in, to run ESM directly with my node@14.

Curiously, it seems to have a _lower_ performance, at around 11.5k rps. You can try it on your own: https://github.com/JaneJeon/validator-benchmark/tree/esm (you need to npm i for the benchmarkify/ and yup/ folders).

The only difference is running babel-compiled js vs. running modified ESM source directly, so I'm not sure why it would be literally 3x slower 🤷‍♀️

I may have to just take an L on this one and accept defeat (and go back to ajv with much horror)...

[jquense]

hi there! Thanks for taking the time here to look through this! Quickly looking at the benchmarking (and without a lot of context) my hunch would that the the largest difference is due to sync vs async API usage. You mentioned that you messed around with using both, but I struggle to think of how the benchmarking lib could normalize that variable. Promises are by definition necessarily slower than straight sync calls, there would be no way the async API could compete with sync alternatives in raw speed. I would also expect that isValidSync would give much better results.

I suspect that the default babel compilation is what's slowing down your codebase significantly since, as far as I can tell, you're only using babel for ESM syntax.

I would be very surprised if that is the case, babel is already using the loose (e.g. faster) setting, but i do see that there is a lot of compilation happening that doesn't need to for Node. I can put together a far more minimal babel setup if you want to test it and confirm either way.

Some general questions:

• Do you know if the benchmarks are actually valid or apples to apples? I don't have a ton experience doing microbenchmarks but they are easy to get wrong in super non-obvious ways so worth reality checking
• Do you have any data to say that if these are accurate, this is _too_ slow for normal use? On the face of it even as the slowest option here seems fine in terms of request overhead (6μs vs 12μs per validation on my mbp), but again i don't have your context.
• Any idea how we could profile yup further here to figure out waht is slow and optimize it?

[jquense]

I did try and switch to isValidSync and it's confusing to me why that makes performance _worse_ but what do I know!

[JaneJeon]

@jquense I already tried it and looked into it, it's because synchronous-promise is NOT meant to be used in production!! Even the library author says so

[jquense]

The author is aware that I do use it tho. The warning is not bc it's bad or slow, but because it's usually not a good idea to have sync promises. The use case here is valid tho :P

[JaneJeon]

Well, somehow it slows down the performance by 2x 🤷‍♀️ I looked into the source code and it LOOKS innocent but hey, what do I know? xP

The benchmarks are, for the most part, apples to apples. There are some parts where it could be improved (e.g. joi is not tested with strict mode), but all of them test the same schema, in the same ways (compile up front if possible, then just run the actual schema checking code multiple times).

I have no idea how to profile further xP this is my first time profiling something that doesn't involve I/O or network calls >.<

In regards to "should we care about performance at all" point, 1. As I've said previously, even disregarding the "is it practical" point, it's just not right that it is a factor of magnitude slower joi. Something must be going on here, and I want to make sure yup can be as good as it should be. 2. As the slowest, it takes 27μs per validation and caps out at 36k rps. My main concern here is that since validation runs on _every single request_, it could quickly block up the event loop. For context, fastify is able to reach 77k+ req/s, and it's _with_ the ajv schema validation running on _every request_.

In general, I just don't want the schema validator to be the bottleneck in my application 🙃

[JaneJeon]

Another comment in regards to your sync vs. async point: I think cramming asynchronous API into synchronous one just doesn't make sense. Schema validations more often than not do not need to be async, and yet because _some_ people might cram in async calls in their schema validations, the _whole stack_ of yup needs to be async and brings the performance down with it.

For example, https://github.com/tensult/role-acl/ manages to implement synchronous and asynchronous APIs _without_ cramming async into sync or using something like synchronous-promise

[jquense]

I think cramming asynchronous API into synchronous one just doesn't make sense.

it's actually the other way around yup, crams a synchronous api into an async one. Originally there was no sync validate(). I also understand the sentiment, but async validation is a _core_ principle and feature of yup, I wrote it specifically b/c i needed first class async validation support and most libraries (at the time) were terrible at it.

In any case, this did prompt me to try and swap out the ugly sync-promise hack for a proper compatibility layer using callbacks. (https://github.com/jquense/yup/pull/1019) and it did bring the benchmark up past the async version, but not significantly, so the core performance issues are unlikely to be related to the promises. My guess is there is some simple thing that is the core problem but not sure how to find it

[jquense]

it's just not right that it is a factor of magnitude slower

totally, in agreement there! Nothing inherently in the approach here should require that

[JaneJeon]

@jquense

so the core performance issues are unlikely to be related to the promises. My guess is there is some simple thing that is the core problem but not sure how to find it

Oh man, I'm not even sure where to get started with that...

[JaneJeon]

Wait, shiny flame graph: https://github.com/clinicjs/node-clinic-flame-demo

Let me see if I can get something out of this (btw I'll be looking forward to your callback-based PR regardless of the performance considerations! I think it's just a better way of doing things)

[JaneJeon]

So here's the flame graph: https://i.imgur.com/jvV0Ujg.png

Now I don't claim to be an expert, but one thing that pops out to me (other than a lot of time being spent on lib/mixed.js, but that's mainly just running the actual validations I think) is that lib/util/createValidation.js is run over and over and over again.

[hdoupe]

Hi all, I use yup for client side validation but have tinkered some with using it on the backend and have enjoyed following this thread today.

Does node have a line-by-line profiling tool? Python has line_profiler which tells you how much cumulative time (over several hits) is spent in each line of code and how much time each line of code takes on average:

[JaneJeon]

@hdoupe hey, I linked a profiler above. I'm not as familiar with this tool but it (the flame graph tool) shows me what line is taking up how much time

[hdoupe]

Thanks for your reply @JaneJeon. That makes sense. I couldn't quite see from the pictures.

Also, thanks to you and @jquense for looking into this!

[JaneJeon]

@hdoupe I just really really want this to work out for backend, because it's such a wonderful tool imho. I'm not an expert with profiling landscape in nodeland other than the clinic tool I linked above, and so far the flame graph has been the one tool that has been actually somewhat useful, but I'd still love your take on it.

As far as I can tell, the performance regressions are because of the sheer number of call stacks and anonymous functions and what have you (whereas something like fastest-validator compiles to literal code that does if (blah) then foo;), but there are other troubling signs such as having to "create" validation every time you want to validate something... (see my comment above w/ the flame graph for supporting evidence)?

Yeah, I think the performance issue is because of the sheer degree of metaprogramming involved.

Update: I'm profiling with the ESM version that I wrote (since it has clearer call stacks due to not being babel compiled), and most of the time is spent on runValidations, which I guess is obvious. The only question now is why is runValidations taking so long?

Nothing in the node.js code section in the flame graph jumps out to me as obvious offenders (after all, it just tells me runValidations is taking up all the time). But once I include V8, I get _a lot_ of Abort.ExtraWide. In fact, it's taking up a good 75% of the runtime.

Other than the double .runValidations(), another thing that the flame graph points out to me is https://github.com/jquense/yup/blob/master/src/object.js#L144. Maybe the spread operator has a negative impact?

[jquense]

So taking a bit of an audit today, there is plenty crufty yup internals that do very little with a lot of indirection. A _very_ quick hack and slash of some of that cruft is seems to be yielding good results. My current guess jives with what you're seeing:

• a lot of extra functions calls, growing the stack larger than needed
• a lot of shallow object cloning.

The first is fairly addressable, I'm less sure about about the second. Aggressive object spreads are a guard against race conditions in the async code paths. I need to clean up the code here so I'm not as afraid to change stuff :P

[jquense]

I do what to hedge a bit against Joi. Yup has a more flexible API which makes it better (imo) at custom validation. The cost tho is short of something like ajv, it's harder to fine tune the hot paths bc they are implemented in a more generalized way

[JaneJeon]

Yeah that's absolutely not a problem. There's lots that Yup gives, but it obviously could use some improvements.

I'm actually really thankful that you took the time to clean up the codebase, because I don't know where to even start.
Is there a PR/branch that I can follow for your "code audit"/follow-ups to this? @jquense

[jquense]

Not yet, I'll put something up as soon as I think it will be mergable

[jquense]

see: https://github.com/jquense/yup/pull/1022

[matiaslgh]

Hey guys! It seems that that PR improved a lot the performance right?
I also want to use the same schema validation in the server and the browser and that's why I'm really interested in this improvement. Do you know when will be the release date that includes this improvement?
Thank you!

[jquense]

gonna try and get something out soon

[Timer]

[email protected]:

Suite: Simple object
✔ validator.js                418,958 rps
✔ validate.js                 203,703 rps
✔ validatorjs                 141,268 rps
✔ joi                         110,188 rps
✔ ajv                       4,465,245 rps
✔ mschema                     518,408 rps
✔ fastest-validator         5,018,584 rps
✔ yup*                         32,623 rps
   validator.js            -91.65%        (418,958 rps)   (avg: 2μs)
   validate.js             -95.94%        (203,703 rps)   (avg: 4μs)
   validatorjs             -97.19%        (141,268 rps)   (avg: 7μs)
   joi                      -97.8%        (110,188 rps)   (avg: 9μs)
   ajv                     -11.03%      (4,465,245 rps)   (avg: 223ns)
   mschema                 -89.67%        (518,408 rps)   (avg: 1μs)
   fastest-validator            0%      (5,018,584 rps)   (avg: 199ns)
   yup*                    -99.35%         (32,623 rps)   (avg: 30μs)
-----------------------------------------------------------------------

---

[email protected]:

Suite: Simple object
✔ validator.js                361,989 rps
✔ validate.js                 186,671 rps
✔ validatorjs                 134,968 rps
✔ joi                         116,485 rps
✔ ajv                       4,701,316 rps
✔ mschema                     521,154 rps
✔ fastest-validator         4,776,743 rps
✔ yup*                         66,885 rps

   validator.js            -92.42%        (361,989 rps)   (avg: 2μs)
   validate.js             -96.09%        (186,671 rps)   (avg: 5μs)
   validatorjs             -97.17%        (134,968 rps)   (avg: 7μs)
   joi                     -97.56%        (116,485 rps)   (avg: 8μs)
   ajv                      -1.58%      (4,701,316 rps)   (avg: 212ns)
   mschema                 -89.09%        (521,154 rps)   (avg: 1μs)
   fastest-validator            0%      (4,776,743 rps)   (avg: 209ns)
   yup*                     -98.6%         (66,885 rps)   (avg: 14μs)
-----------------------------------------------------------------------

[jquense]

not sure what i'm supposed to do with a context-less benchmark @Timer

Gonna guess that you are just running the benchmark from above with the newest version. It's not an apples to apples comparision, run it while using isValidSync and you get

✔ validator.js                562,183 rps
✔ validate.js                 290,012 rps
✔ validatorjs                 200,192 rps
✔ joi                         168,420 rps
✔ ajv                       7,452,251 rps
✔ mschema                     725,544 rps
✔ fastest-validator         7,541,043 rps
✔ yup                         104,305 rps

   validator.js            -92.55%        (562,183 rps)   (avg: 1μs)
   validate.js             -96.15%        (290,012 rps)   (avg: 3μs)
   validatorjs             -97.35%        (200,192 rps)   (avg: 4μs)
   joi                     -97.77%        (168,420 rps)   (avg: 5μs)
   ajv                      -1.18%      (7,452,251 rps)   (avg: 134ns)
   mschema                 -90.38%        (725,544 rps)   (avg: 1μs)
   fastest-validator            0%      (7,541,043 rps)   (avg: 132ns)
   yup                     -98.62%        (104,305 rps)   (avg: 9μs)
-----------------------------------------------------------------------

This is pretty close to joi, and i'm happy enough with the result. If anyone wants to save off more time please feel free to send a PR

[Timer]

I was just supplying it for context. Not asking for any action to be taken. :-)