Joi: Support for async valdations

[igorescobar]

This is more a feature request then a issue.

I've read quite a few issues here about users trying to accomplish external api calls as part of their validation process so I'm just "formalising" the request here.

So to give some context... the issue is with the Joi.extend API. The method rules.validate expect me to return something immediately.

Would be fantastic if it could handle a Promise as a return and still respect the method chain.

Thanks and let me know if I can provide more insights about this.

[WesTyler]

Async validation support is on the roadmap, but it is a very large piece of work. So rest assured it is coming, but probably not soon.

[akazakou]

@WesTyler, how it's goin? Can we expect async validators at 2018 year?

[rossanthony]

@WesTyler Any update on this? Is there anything we can help out with in terms of testing etc?

[tojacob]

We still expect this :)

[devinivy]

I'm not a maintainer of this module, but I would like to take a moment to reflect on the complexities of designing this feature. Suppose you turn Joi.validate() into an async function. Now every function in every call stack that includes a call to Joi.validate() needs to become async. That is a massive breaking change with net negative performance characteristics, and all to cater to what is currently a fairly niche case. So do you go forth with that plan? If not, what should the API look like? Does Joi.validate() behave differently (sync vs. async) depending on the joi extensions at play, or do you introduce a new, additional async API? Will joi extensions need to mark themselves as async? What are the effects of this on hapi itself? Are any of these options straightforward for users? How major of a rewrite of joi internals will any of these options be?

In some ways I understand the desire for async validations, but it's important to note that it's not easy to design and keep everyone happy—let alone to code and document. If this feature matters to you, rather than applying time pressure on the maintainers of this module, I imagine that coding-up a proof of concept or writing-up some ideas for how an async validations API might look would be a healthy way to move the conversation forward.

[akazakou]

@devinivy what about implement async functionality in Joi.validate() and add some configuration option as static property to Joi class, that allows switching between sync/async behaviors?

[PabloReszczynski]

To stay backwards compatible, you could make a new function Joi.validateAsync(), like the standard in a lot of libraries that have implemented async functionality.

[Marsup]

OK, I feel that I need to provide a status update on this before it goes too far.

I have a local prototype of this feature that I finished recently, without any API change, so sync + callback + async on the good old validate, but don't get your hopes up.

As @devinivy mentioned, the complexity of this is crazy high, it took me probably several weeks of work to get to this point (hard to count when it's an hour here and there), the code quality IMHO took a really big hit, and last but not least, on sync scenarios, it performs 250% slower on average. This is to me as I think it is to you, unacceptable. Now I'm allowing myself some time to think about what could be done differently or profile it to see if I missed something obvious. If this is inconclusive, it will leave me no choice but to introduce this as a breaking change and stop supporting sync validations completely. If you feel that you can achieve the same goal that I had with a minimal loss of performance, be my guest, the code is open, but I haven't seen anyone even try in the last couple of years.

Now let's keep in mind that I did all this entirely on my free time, all the paid time (and more) being spent on triaging and fixing smaller issues. So if you want to help on this one, the best way is probably to either succeed where I failed, or to help on the other issues so that I don't have to.

[igorescobar]

@Marsup I understand that changing the entire API to be async might be incredibly complex but have you thought about only implementing this on the: `https://github.com/hapijs/joi/blob/v13.3.0/API.md#extendextension

You could think of following the same implementation that mocha does to handle tests with async validations.

For example inside of the extend method you expect this validate(params, value, state, options){}. You could somehow see how many arguments were informed to it and the validate would behave differently if I inform another argument to the chain.

So it would look like this:

validate(params, value, state, options, done){
  doSomeAsyncThing().then(() => {
    // manipulate things
    done()
  })
}

So, in that case, you would be waiting for this done callback to be called. So instead of changing the entire API, you could maybe just change the extend behaviour.

PS: sorry if my suggestion isn't really "doable". I'm not really considering how things work under the hood.
PS: Thanks a lot for all the work you put into this project. <3

[Marsup]

The problem is not the extension system, it's the ripple effect. Making extensions async is the easiest part, but then all the validation calls must be ready to deal with that, which is exactly what I did.

It might be interesting for me to know how concerned people are with keeping sync validations, now that the language has the tools to manipulate this more easily. Would it be a big challenge for any of you to move away from sync validations ?

[WesTyler]

We are not attached to sync validations in our codebase. Async-only might even force us to do some cleanup in our code base. :)

[devinivy]

I wouldn't say that I'm attached to sync validations, but I am not a fan of async-ifying functions that run synchronously in the typical case. It affects entire call stacks, and is likely to require changes to the interfaces of _other libraries/modules_ that utilize joi, e.g. for validation of their function options. I don't know the specifics here, but I imagine that this might affect performance outside of joi itself, again due to other calls in the stack needing to become async.

[tojacob]

I have solved my problem with asynchronous validation, I have done it outside of Joi. I used Joi in a normal way and added a bit more code for my asynchronous validation.

After my experience and reading these last entries, I think that adding asynchronous capacity to Joi is not really necessary, since, as they say, the side effects could be negative in the ecosystem.

I want to thank everyone who took the time to perform the relevant tests to try to implement it.

From my point of view, I consider that Joi must remain synchronous and this issue should be closed for future reference :)

[WesTyler]

this usage of hypothetical Joi.validateAsync() is as useless

Agreed, but that's not the use case that has driven the investigation into async validation. :)
One of the main reasons people commonly request async validation is so that they can do database or api lookups inside of their validation rules (enforce id uniqueness or something).

[akazakou]

@WesTyler I believe that the main idea in logic separation in projects. When peoples want to move all validation to Joi functionality sometimes the required an async operation to check database or files or other staff. They want to keep all validation logic in one place. But if Joi does not allow it, they should split validation logic and store at a different part of the project.

[humanchimp]

it performs 250% slower on average

did you consider adding a .promise() method or similar as an explicit opt-in so that validating schemas not containing subschemas derived from .promise() could take a faster path?

Edit: i thought though this idea a bit more and thought i would offer this (albeit quick) sketch to improve upon what i suggested above.

const schema = Joi.promise().then(Joi.boolean().valid(true))

const usernameOk = isUserNameAvailable(); // Not synchronous; we have to check the server to find out...

// The existing interface still works and isn't degraded performance-wise
schema.validate(usernameOk); // Only capable of validating that this the argument is a promise

// A new interface pops into existence to accommodate async requirements 
await schema.validateAsync(usernameOk); // Takes longer, but is capable of validating inside the promise

// It works rather like Joi.array() and can nest inside other schemas:
const userSchema = Joi.object({
  usernameOk: Joi.promise().then(Joi.boolean().valid(true)),
  cc: Joi.string().creditCard() // gratuitous extension, irrelevant other than demonstating context
});

userSchema.validate(user) // works, but is not capable of validating that the username is available
await userSchema.validateAsync(user) // takes longer because of the network, but is capable of validating inside the promise

Please forgive mistakes or typos. I hope this communicates the idea

[OctaneInteractive]

Perhaps a bit late with this, but here's a simple fix for the lack of _async-await_ support:

router.post('/search', async (req, res, next) => {
    let isValid = false
    const data = req.body;
    const schema = Joi.object().keys({
        userID: Joi.number().required(),
        search: Joi.string().required(),
        paginateValue: Joi.number()
    })
    Joi.validate(data, schema, (err, value) => {
        if (err) {
            res.status(422).json({
                status: 'error',
                message: 'Invalid request data',
                data: data
            });
        } else {
            isValid = true
        }
    });
    if ( isValid ) {
        const searchAssets = await controllerSearch.searchAssets(data);
        res.json(searchAssets)
    }
})

[hueniverse]

I'm closing this. If one day it magically appears, people will be pleasantly surprised.