Joi: RFC: Joi plugins/extensions/custom functions

[Marsup]

See https://gist.github.com/Marsup/14597d0c8eaa10c4addb for latest version of the RFC.

[totherik]

This looks like a good start, but one major issue is that it's using what is effectively a global in an unsafe way, potentially leading to naming collisions and unexpected behavior.

For example, imagine if module A and module B both depend on the same version of Joi and both declare a type Foo. The module will be deduped and they'll both reference the same instance of the Joi module, meaning they now compete for the same name, Foo, in the same namespace. This isn't a terribly unrealistic scenario and the collision would be very difficult to debug.

[Marsup]

As written, any attempt to override a type will result in an error, I don't think this should be a problem to debug.

[totherik]

Ah, ok, I missed that part.

If I _depend_ on A and B and an error is thrown, I have no recourse or ability to remedy the situation as I don't own those dependencies and those authors could not have anticipated being I the same dependency graph as each other.

This would require module authors to define globally unique (as in unique from every other package in npm) types.

Also, as a module author I can say that having to come up with and define globally unique names via trial and error is undesirable .

[Marsup]

I'm not sure it's preferable to locally load all extensions. Uniqueness will be partially guaranteed by npm naming, for other cases I don't see how we can solve that.

[totherik]

Uniqueness will be partially guaranteed by npm naming

Can you clarify?

Also, I understand I'm the worst kind of commenter. I identify a problem but, unfortunately I don't have a solution. (Need to think on it more.) Sorry about that.

I can say that at my company where we would need to define types for validation in things like service wrappers, this mechanism would not work and we would not be able to use it due to the high likelihood of name reuse.

[Marsup]

If people publish types as modules named joi-type, that should avoid many conflicts, but that doesn't fix the internal ones. So you'd rather require all extensions each time in all your files?

[AdriVanHoudt]

1. I really like the spec and how it opens up Joi for custom stuff
2. Maybe add some docs for the handlebars in language where you reference parameters?
3. Will Joi itself now use this system? Because you can basically rebuild Joi with this right?

[Marsup]

I don't believe the templating language is relevant for this proposal but yes it'll have to be documented. Joi need to maintain a bunch of core features because of parameters validation.

[rlidwka]

1. Maybe rename Joi.addType to Joi.extend for consistency reasons?
2. What the difference between convert function Joi.addType and validate function in Joi.<type>.extend? As far as I see, both can change value, and both can throw on error.
3. I don't see a way to access arguments passed to a type. In your example, Joi.user(options), options seem to be inaccessible.

[Marsup]

1. Yeah why not, based on the options given, it should work. The Joi object is an Any currently though, you'd be extending it, I'll have to see if it's feasible.
2. The convert has a chance to do something before anything else does, this is a post-base check if you will. I've thought of this because of an issue about mongodb's ObjectId: if you need to coerce your value to a specific type (in this case new ObjectId(value)) for any other extension method to work, you have no place to put it.
3. There are no options passed to a type, what do you mean ? The options used to build a type or a method are not the concern of the user, hence not exposed, should it be ?

[myndzi]

Really excited about this, and bonus: asynchronicity!

I think maybe you misunderstand the namespace problem brought up by totherik. He's not talking about uniqueness as an NPM module, he's talking about uniqueness in the Joi namespace. I agree, this is a problem. It can be solved fairly readily, however, by allowing Joi to be instantiated rather than using it as a singleton, e.g. var Joi = require('joi')(options?);

It would then be up to the developer to share the Joi instance around, but this is trivial in Node.js since you can just create a simple wrapper that also defines your extensions:

var Joi = require('joi')();
Joi.extend(etc...)
module.exports = Joi;

You can drop the requirement for the () by making it instead a method of Joi itself, like Joi.giveMeANewDefaultInstance() (can't think of a good name!)

Joi has long been my favored validation library, but I've wound up using it almost nowhere because of the two things this proposal will address. Happy happy joy joy!

[myndzi]

Re: the question about cloning/freezing or trusting the developer, I'm in favor of the latter. Solid enforcement appeals to me, but one of the most useful use cases for Joi is in things like HTTP requests, where the performance difference can be solidly beneficial. A suitable compromise is to allow the user to specify an option about whether to freeze the objects or not, and this option could be used when running tests or development, for example, but not in production.

[totherik]

Thanks for clarifying my comments @myndzi. (Weekend away with the fam so trying to stay offline :) )

So with the pursuit of types-as-npm-modules, is it reasonable to expect that a single application/module may need to depend on incompatible versions of the same npm module type definition? If so, npm can't be used to manage those dependencies. (I'm thinking major revisions of types that aren't required to be backward compatible according to semver.)

I don't know all use-cases, but I have to imagine this could be an issue.

[Marsup]

I understood what he meant, npm is indeed a "trick", not a solution, but it might avoid some basic conflicts.
Having multiple versions of your models is indeed a problem (joi or not), but isn't it better to have an error thrown to tell you about this mistake rather than having 2 separate parts of your code supposedly manipulating the same thing and have different versions of it ?
If it is an npm module, it's probably wiser to have a single version in your dependency tree, and if it's part of your code, maybe you'll have to make this explicit, like rename type user into v1user and create a v2user if you create a new version of your API. What do you think ?

I kind of agree on the freezing/cloning/vanilla problem, if you really want to mess up your schemas, that's your responsibility, can't protect users from everything.

[myndzi]

I don't see how the name of the NPM module affects the name of the thing extending Joi in any way, really. Validation is useful for lots of things, and the same name can mean different things in different contexts. Namespacing is a troublesome solution, but extending separate instances seems like a clean way around it...

I'm not sure you do understand it, since throwing an error is out of the question. It's not a mistake so much as a circumstance that can arise as a result of two separate modules using Joi for validation. It's not about describing a model and manipulating it from two angles, it's about the fact that node.js modules are inherently singletons, so you run into namespace problems when you use multiple distinct modules that utilize joi for validation...

[myndzi]

Let's take your mongo example as a base. It extends Joi with a type called 'user'. Say you add in something like Passport, which has also utilized Joi for validation and has a 'user' type of its own with a different meaning. You, as the developer, can do nothing about the fact that you're using two modules that want the 'user' namespace, but only one is allowed to have it, unless you do evil things with the path structure which leads into undefined behavior. It's not an error that the mongo module is defining 'user', and it's not an error that passport is defining 'user'. Throwing doesn't solve it, and the developer has no tools to solve it either. Allowing the modules to utilize their own individual instance of Joi, however, cleans this kind of scenario up nicely :)

[AdriVanHoudt]

@myndzi This can easily be solved by the authors of the modules by choosing more unique names like passport-user but it is indeed a concern that you may run into duplicates which leads to not being able to combine certain modules if the authors don't want to change their names

[Marsup]

I perfectly understood the problem thanks. I'm trying to find alternatives to that, because what you're proposing doesn't make sense to me. If you inspect your API, you will find several validations with the same type but not the same underlying concept, that bothers me.

I have not made this proposal just to offer my own version of custom functions, which would be much simpler if you look at previous PRs about it. Joi is also built for documentation through code, I intend to keep it that way.

Now we could introduce an optional namespace property that would force you to do Joi.myModule.myType but I'm not even sure a single level will please everyone. I can't think of any way to have what I want without some kind of uniqueness.

[AdriVanHoudt]

Any thoughts on alias support? Joi.alias(<type>, <otherType>) e.g. Joi.alias('valid', 'only')

[Marsup]

What do you think if addType just returned a new type and let you attach it to Joi if you really want it ?
Like Joi.user = Joi.createType({ ... }) (createType makes more sense considering the behavior).

[AdriVanHoudt]

So a module would create a Joi type and export it, so you can add specific types to your custom Joi?

Something like this?

var mongoUser = require('joi-mongo-user');
var passportUser = require('joi-passport').user;
var Joi = require('joi');

// either this
Joi.user = mongoUser;
Joi.passportUser = passportUser;

// or this
Joi.user = passportUser;
Joi.mongoUser = mongoUser;

module.exports = Joi;

joi-mongo-user

module.exports = Joi.createType({...});

joi-passport

module.exports = {
    user: Joi.createType({...}),
    otherKey: Joi.createType({...})
}

I like it.

Although I still like the idea of Joi.object().isUser() since you can clearly see the base type.

[Marsup]

I've updated the proposal, see the diff in https://gist.github.com/Marsup/14597d0c8eaa10c4addb/revisions.
Does it address your concerns @myndzi and @totherik ?

[AdriVanHoudt]

So Joi.extend() will not extend the singleton Joi but return a 'new version' of it right? Whereas the Joi.type.extend() can do both(taken from the examples)? As for the rest I like it!

[nlf]

:+1: I like the idea of returning a new singleton rather than mutating the core, makes things much more pluggable and keeps up with the concept of immutability within joi. Good idea!

[AdriVanHoudt]

I agree with @nlf but some examples look otherwise, that's why i mentioned it

[myndzi]

If I understand this correctly, it's similar to what I had in mind but more fine-grained. Where my suggestion would group all extended types (by a single module) under a single instance of Joi local to the module that required it, this proposal would instead create multiple individual instances of Joi, one for each type you create. This works acceptably, but doesn't make as much organizational sense to me. Am I missing something?

In other words, I am thinking of something like this:

- Joi-Passport
  - user
  - oauthprovider (or whatever)
- Joi-mongo
  - objectid

whereas my understanding of the new proposal is more like this:

- Joi-Passport-User
  - user
- Joi-Passport-Oauthprovider
  - oauthprovider (or whatever)
- Joi-mongo
  - objectid

[Marsup]

You can just as well organize your own composition of types :

var JoiPassportUser = require('joi-passport-user');
var JoiPassportOauthprovider = require('joi-passport-oauthprovider');
var JoiMongo = require('joi-mongo');

module.exports = Joi.extend([
  { name: 'passportUser', base: JoiPassportUser },
  { name: 'passportOauthProvider', base: JoiPassportOauthprovider },
  { name: 'mongo', base: JoiMongo }
]);

You can do that in a single pass like this (which will probably be more efficient), or with several extends.

Maybe what's missing is the ability to include a Joi instance into another one, but that might prove to be a challenge.

[myndzi]

var user = Joi.string().min(1).required().extend({ ... }) feels a little clunky, I think this is why:

There are two fundamental concepts at work here, a "type" and a "rule". While it makes sense to provide a way to avoid repeating work, it doesn't quite make sense to define a new type ("user") as an extension of an existing base type, or as the result of some specified rules. What you're wanting to do here, I think, is avoid repeating the work of validating the value as a string, but this feels more at home in the validation function itself, something like:

var user = Joi.createType({
    validate: Joi.string().min(1).required(), // a joi schema or a function?
    coerce: function (value) {
         // coercion failure is also a validation failure
        return Users.findOne({ id: value });
    }
});

Coercion applies to types but not rules, and rules are specific to a given type; if we are creating a validation for a User object, it is liable to be something that needs coercing from something like an ID, and also something for which the rules for 'string' no longer make any sense to apply to.

I am currently imagining something like this:

var hasRole = Joi.createRule({
    description: 'check if a user has a role',
    parameters: {
        role: Joi.string().required(),
        db: Joi.object().required()
    },
    language: {
        database: '!!failed to verify the user because of a database error: {{err}}',
        role: '!!user {{user}} does not match role {{role}}'
    },
    validate: function (user) {
        // db stuff, i don't know mongo well
    }
});

var user = Joi.createType({
    validate: Joi.string().min(1).required(), // a joi schema or a function?
    coerce: function (value) { return Users.findOne({ id: value }); },
    rules: {
        hasRole: hasRole // it gets named here, not by a 'name' key
    }
});

module.exports = Joi.extend({
    user: user
});

In createType, 'validate' would be a validation applied _before_ coercion; it would have little meaning after coercion -- the purpose of coercion, here, is to put the value into a form the rules can be applied to, whereas the purpose of validation is to ensure that the value is acceptable for the base type to begin with. Putting 'rule's as an object in createType allows you to inherit rules if you should so desire, for example:

var SuperString = Joi.createType({
    validate: ...,
    coerce: ...,
    rules: Joi.string.extend({
        myCoolValidator: function () { ... }
    })
});

module.exports = Joi.extend({
    string: SuperString
});

The object syntax also supports 'extend' well both semantically and implementation-wise

[myndzi]

I think we're converging on the same ideas ;)

I honestly see very little use for exporting anything other than a whole-and-completely-usable Joi object, but reusability is always helpful. To me, if I need to perform some kind of custom validation, it's for the purpose of argument-checking a function I am also writing. I'm having a hard time coming up with a scenario where I'd want to export a Joi schema rather than the function it's intended to do work for. Thoughts?

[AdriVanHoudt]

I like the ideas in your proposal @myndzi
The benefits for exporting only the rule/type is then you can make 1 place where you wire them all up without having to extend every previous Joi object from the previous module

[myndzi]

What I mean is that Joi isn't likely to be layered in that fashion. I can't think of a case where I'd have Joi layered three deep as you suggest. If I require a module that uses Joi, I'm not requiring it to gain access to new Joi validation types -- I'm requiring it to gain access to whatever that module is supposed to do. If it uses Joi to validate things, great! But those are things I am not responsible for validating, they are up to this module I'm using. So why would I want this module's copy of Joi at all, let alone want to extend it?

[Marsup]

Looks like we're almost back to my draft with base type and convert. :)

I don't really see the benefit of createType if extend does the same thing, it's just an unnecessary intermediate object while you could use the POJO with extend directly.

You seem to have multiple forms of extend (on Joi and on type), care to explain the difference ?
If the 2nd form returns a type as I guess it is, I don't think it's gonna be easy to just put that into rules and merge it with the validate part.
Your last example would be better written like this :

var SuperString = Joi.createType({
    validate: Joi.string(),
    coerce: ...,
    rules: {
        myCoolValidator: function () { ... }
    }
});

I'm trying to minimize the number of apis you'll have to use, this seems a bit more complicated than I'd want it to, but yes I think we're converging.

[myndzi]

Well, the point of having a separate function is mostly so you don't have to write one huge monster function with nested objects many ways deep. It also gives you points to break things up into files; if the actual product was as straightforward as I wrote, I probably wouldn't have broken it up either.

Extend in every case should be interpreted to mean "return a new object that's a copy of this object, and assign these keys to it". You're right, my example is poorly written; what it really was meant to represent was Joi.string.rules.extend(), as in, "give me a copy of this type's rules" -- but the "rules" component seemed superfluous. The main point was to separate the rules from the type -- what I'm interested is in the rules that can be applied to a string; what I'm not interested is making the underlying value into a string or requiring it to be. Clean separation is what I was going for.

I do prefer the original form of your proposal, felt it made a clearer distinction between a type and a rule, and that it's a worthwhile distinction. All it needed was the 'extend' idea, a way of instantiating a sub-copy of Joi instead of affecting the global singleton.

Edit: Re: 'validate' vs 'base type', I think that 'base type' implies inheritance in a place that inheritance doesn't belong, but I was trying to avoid the necessity of something like:

function (value) {
    var errors = Joi.validate(Joi.string().min(1), value);
    if (errors) { throw errors; }
    // other stuff
}

... so then I came back to the idea of supplying a Joi schema to validate against as an optional alternative. The equivalence totally flew right past me ;)

Coercion vs conversion is just another choice according to intent; coercion implies "same value but in another format" whereas conversion doesn't carry that connotation as strongly.

[raisch]

As an aside, I just published joi-extender as an experiment in adding new 'top-level' types to joi. While it's clearly marked as EXPERIMENTAL, I'm using it in several projects and only mention it here for purposes of discussion.

[tsunammis]

@Marsup Do you have a branche with your current WIP about this RFC ?

[Marsup]

@tsunammis nope, the RFC is not final yet, I won't start implementing before that, this is a big refactoring ahead and I don't want to go back and forth.

[tsunammis]

@Marsup indeed, you have right to fix the RFC before implementing it ;-)

[fhemberger]

Just a short remark/question, as I just stumbled upon this issue:

Joi operates on the existing JavaScript data types (string, object, array, boolean, etc). I guess for the majority of cases you don't want to add new data types but instead add further checking patterns/functions to the existing data types, hence createType is a bit misleading.

Hapi offers the server.decorate() method, so how about adapting this for Joi as well? For Example:

Joi.decorate('string', 'usZipCode', function (value) {
    // perform check
});

I think it would be more expressive that way.

[AdriVanHoudt]

@fhemberger how would you cover the use case of decorating something like string().min(0)?

[fhemberger]

@AdriVanHoudt Hmm, I don't know. I would only extend the data types to add new assertions, and chain them with existing methods, e.g.: Joi.string().length(5).usZipCode().

That makes the validation chain more explicit than adding an arbitrary number of custom functions, which could wrap an arbitrary number of other Joi functions each.

[AdriVanHoudt]

true but you want to prevent stuff like: usZipCode() requires a string with 5 chars so you add a test in the rule for that but when you do .length(5).usZipCode() you're checking twice because I don't know whether usZipCode() checks for the length or not

[fhemberger]

But the same is true for nested custom functions. You have to look inside as well to see what they check.

[AdriVanHoudt]

hmm true, a case can be made that the custom rule should be smart enough to check (in this example) that the string is 5 long. The question is then do you want to extend string() or string().length(5)? since the later is easier for e dev to make custom rules

[myndzi]

A zip code, while basically a string, is a different "kind" than a string. It's its own entity that has its own rules. Ideally, for me, you would make a US Zip Code "type". This type could be coerced from a string or number and would throw if the input value is not a valid US Zip Code (this includes length). Rules, distinct from types, are additional constraints beyond the "kind" of thing; for example, a Zip code type might have a rule that restricts it to be within a certain state or set of states.

The distinction between "type" and "rule" is important here; "types" come with inherent validations and "rules" apply additional constraints to reduce the value from "anything that's a valid type" to "only certain valid types"

The type itself would probably have to inherit from any(), since string or number based rules don't have meaning against a zip code.

[AdriVanHoudt]

The distinction is indeed important but I do not agree with you. A zipcode is a String so it should use String() as the base just like email() and ip() for example

[myndzi]

A zipcode might be stored as a string, but it is not a "string" in the sense of the rules that apply to strings. Consider the String class rules:

Insensitive: meaningless to a zip code
Min: meaningless to a zip code
Max: meaningless to a zip code
Length: meaningless to a zip code
Regex: nominally meaningless to a zip code
Alphanum: meaningless to a zip code
Token: meaningless to a zip code
Hex: meaningless to a zip code
Uppercase: meaningless to a zip code
Lowercase: meaningless to a zip code
Trim: meaningless to a zip code
CreditCard, email, ip, url, guid, hostname: Meaningless to a zip code; should probably be their own types rather than rules against String

The only reason you would inherit the type from a String is to gain the above rules. Since none of them apply, it makes very little sense to inherit from String. In fact, if we were going by the current canon, a zipCode method would instead be a rule applied to String, not a type of its own; however, this would preclude the ability to target a zipCode type with specific rules if such was your desire, and I'd prefer instead to see the pseudo-types above become their own types instead: They also have potential for rules that target their types specifically:

CreditCard types could have a rule that matches based on provider types
IP types could have a rule that matches based on IP class, or type (local/non-routable, reserved, etc.)
Hostnames could have a rule that matches based on whether they are a root public suffix

etc...

P.S. - You can note that the same question that arose above about string length + zip code rules presently exists already in the case of the pseudo-types mentioned. You could apply .length(X).guid() or .alphanum().ip() and so on, or even create completely invalid validations such as .alphanum().hostname() - this serves only to illustrate that while your thinking matches the current approach, it is worth considering changing. Should not all rules on a given type be compatible with each other? (excluding deliberately obtuse usage of the rule arguments, such as .min(5).max(4))

[AdriVanHoudt]

The way Joi works is that the types are the base types of JavaScript. If you represent a zipcode in Js it's probably a string (or number here in Belgium). So it makes sense to say that a zipcode should be a string that represents a zipcode right? Saying that length and trim etc. are meaningless is not correct. For example a zipcode in Belgium is always 4 numbers and if you represent it with a string why not be sure and trim the string? If you want specific 'rules' on rules like ip you have the ability to pass options into the rule. For example you can specify if an ip should be ipv4 or ipv6

[myndzi]

Presumably, as part of being its own type, the zipcode would be coerced; this process would include trimming the string if desired. Our original example was a specific kind of zip code, for which the length was an inherent property; this is the context of my example. A US zip code is inherently 5 digits (leaving aside the extended zip code for the moment). The discussion was about whether this inherent property should be implicit or explicit.

Should you want to validate the length of a zip code, it also makes more sense that you would implicitly validate the length by _explicitly_ specifying the locality of the zip code. That is, you wouldn't use ZipCode().length(4) but ZipCode().country('Belgium') or some such.

Your point about Joi's current base types being 1:1 with Javascript's is noted; I think that a more deliberate and useful organization is possible, which is why I commented about it.

[AdriVanHoudt]

Just thought of this. If you really want to you can just extent any() with usZipCode(). This allows you to do just Joi.usZipCode() and solves your problem.

[myndzi]

That is indeed what I suggested originally :)

[kfitzgerald]

Greetings,

I've been creeping on this thread since February, and perhaps zip-code is a controversial use-case for this overall functionality, since on the surface, it's seemingly a relatively easy case to work around using regex. However, @myndzi hinted at the real value of what this functionality would provide. The context or implicit value would need to be validated. The string itself is useless unless the zip code is an actual registered zipcode, or contextually checking the country to which the code should must belong.

My particular use-case involves validating identifiers, specifically 12-byte base58 or base62 encoded strings with a 2-character prefix (think mongodb identifiers made smaller and less ugly). Sure, Joi.string().alphanum() works fine as a preliminary check if the string is potentially valid, but the real need is to actually strip the prefix, decode the identifier and check if the total byte count is actually valid. Since the various encodings could result in inconsistent string lengths, this would require a custom validation mechanism, applicable to this thread.

Base58-encoded identifier: PR2c4Exzy7X6cikEDmp   <-- this is what gets validated
ObjectId: 55054501fdce2c7e5f58e56f
Decoded integer: 26312596343935070124784543087

Whether the end result looks something like Joi.objectId().base58().prefix(2) or Joi.string().alphanum().objectId().base58().prefix(2), doesn't particularly matter to me.

[raisch]

I agree with those who are calling for the addition of new "intrinsic"
types to Joi.

The zip-code use case is a good one and while I understand the interest to
keep a one-to-one correspondence between Joi and JavaScript intrinsics, I
believe that, for Joi to fulfill its stated purpose (to be a useful data
validation suite), it should provide an API flexible enough to both
adequately and succinctly represent its users' needs.

Another example of a use case is validation of Designated Marketing Areas
(DMA) which are represented by strings representing integer values between
500 and 799. The list of valid DMAs is sparse, in that not all values
within the range are legal.

While it is indeed possible to validate a DMA using joi.string().regex(),
doing so requires a large and very specialized regex, which if not
constructed properly, will fail to validate correctly.

I can envision many such use cases and so strongly believe that validation
of such specialized forms should have specialized validators, if only to
allow developers to write concise, understandable code.

Rob Raisch, Internet Handyman

On Tue, Apr 14, 2015 at 11:38 AM, Kevin M Fitzgerald <
[email protected]> wrote:

Greetings,

I've been creeping on this thread since February, and perhaps zip-code is
a controversial use-case for this overall functionality, since on the
surface, it's seemingly a relatively easy case to work around using regex.
However, @myndzi https://github.com/myndzi hinted at the real value of
what this functionality would provide. The context or implicit value would
need to be validated. The string itself is useless unless the zip code is
an actual registered zipcode, or contextually checking the country to which
the code should must belong.

My particular use-case involves validating identifiers, specifically
12-byte base58 or base62 encoded strings with a 2-character prefix (think
mongodb identifiers made smaller and less ugly). Sure,
Joi.string().alphanum() works fine as a preliminary check if the string
is potentially valid, but the real need is to actually strip the prefix,
decode the identifier and check if the total byte count is actually valid.
Since the various encodings could result in inconsistent string lengths,
this would require a custom validation mechanism, applicable to this thread.

Base58-encoded identifier: PR2c4Exzy7X6cikEDmp <-- this is what gets validated
ObjectId: 55054501fdce2c7e5f58e56f
Decoded integer: 26312596343935070124784543087

Whether the end result looks something like
Joi.objectId().base58().prefix(2) or
Joi.string().alphanum().objectId().base58().prefix(2), doesn't
particularly matter to me.

—
Reply to this email directly or view it on GitHub
https://github.com/hapijs/joi/issues/577#issuecomment-92915700.

[myndzi]

A more general way to put it is this:

Input format -> representative format -> presentation format

I'm a firm subscriber to this flow for data; all calculation, storage, etc. should be done in some normalized, canonical representative format. There's probably a term for this approach but I'm unfamiliar with it. The core difference between a zip code as a type and a zip code as a rule is that one stores the meaningful information in its representative format and one stores it in its input format. (Nevermind that in this case they both are likely to be the same thing ;)

Particularly for validation, and especially in the case of web applications, where coercion is commonly needed when dealing with query string parameters and form submissions, it is both useful and important to convert the input data as soon as possible and keep it in its most useful format for as long as possible. You want to be applying rules to the normalized/representative form of some meaningful piece of data, not the transport/input form.

[gergoerdosi]

A Promise : this is the only acceptable way to handle asynchronicity.

What's the reason for this? I'm just wondering because hapi modules (under the hapijs organization) don't use promises. Why callbacks won't be supported?

[Marsup]

I want to avoid parsing the function declaration.

[gergoerdosi]

What needs to be parsed?

[Marsup]

The fact that you need to be async.

[Marsup]

This got me thinking, since a promise cannot be aborted easily, I might have to know upfront that it's asynchronous, leaving me no choice but to have a callback, well that sucks...

[myndzi]

I don't quite follow. If you want to abort a promise, you can just throw an error? You still must return it, so all you really need to do is check whether the return value is an object with a function 'then' property. Still, I much favor support for callbacks -- and I write all my async code with promises -- simply because it's kind of the baseline requirement for Node code.

Declaring up front when you define it that your validation function is async isn't too onerous, and beats the alternatives...

[DavidTPate]

@Marsup When would you need to abort a promise? Wouldn't it just be a rejection instead?

As for canceling promises Bluebird provides a way to do it, but for things that are truly async they can't really be cancelled. This functionality is still in draft form for the Promises A+ spec (there's a few different draft proposals right now).

[Marsup]

I'd need to abort if called in the synchronous mode, that would cause unnecessary work. Aborting wouldn't even be enough unless the promise starts on next tick.

[myndzi]

I don't really think it's Joi's responsibility to ensure the user doesn't do something stupid like call an asynchronous function synchronously. Just let it fly. Possibly print a warning that a promise was returned from a validation function but validate was called synchronously.

[DavidTPate]

What about removing the ability to do validation in a strictly synchronous manner? If you wanted to support callbacks or promises for calls to Joi.validate() then when there is a callback you could use it, otherwise you could return a promise from Joi.validate() if there isn't a callback.

It's difficult to handle things that can be both async and sync in the same interface and I would expect it to cause additional need for help since calls to Joi.validate() could be made in both a sync and async manner.

[raisch]

Agreed. It's not the library's responsibility to enforce/dictate caller behavior. Garbage in, garbage out.

Perhaps async could be supported using two new chainables:

joi.asPromise() - returns a promise that resolves with result.value or rejects with result.error.

joi.onComplete(cb) - calls cb(result.error, result.value) when the current validation is complete.

asPromise() can either appear as the final chainable or if it occurs (possibly multiple times) within a chain, sets a flag that is used to mutate the final result.

onComplete() can appear any number of times in a chain and is invoked in-situ allowing for multiple callbacks based on the currently validated result.

/rr

Rob Raisch, Internet Handyman

On Apr 22, 2015, at 19:11, Kris Reeves [email protected] wrote:

I don't really think it's Joi's responsibility to ensure the user doesn't do something stupid like call an asynchronous function synchronously. Just let it fly. Possibly print a warning that a promise was returned from a validation function but validate was called synchronously.

—
Reply to this email directly or view it on GitHub.

[raisch]

Actually and on further thought, I believe onComplete() makes sense as a chainable but asPromise should be an option to joi.validate()

/rr

Rob Raisch, Internet Handyman

On Apr 23, 2015, at 11:06, Rob Raisch [email protected] wrote:

Agreed. It's not the library's responsibility to enforce/dictate caller behavior. Garbage in, garbage out.

Perhaps async could be supported using two new chainables:

joi.asPromise() - returns a promise that resolves with result.value or rejects with result.error.

joi.onComplete(cb) - calls cb(result.error, result.value) when the current validation is complete.

asPromise() can either appear as the final chainable or if it occurs (possibly multiple times) within a chain, sets a flag that is used to mutate the final result.

onComplete() can appear any number of times in a chain and is invoked in-situ allowing for multiple callbacks based on the currently validated result.

/rr

Rob Raisch, Internet Handyman

On Apr 22, 2015, at 19:11, Kris Reeves [email protected] wrote:

I don't really think it's Joi's responsibility to ensure the user doesn't do something stupid like call an asynchronous function synchronously. Just let it fly. Possibly print a warning that a promise was returned from a validation function but validate was called synchronously.

—
Reply to this email directly or view it on GitHub.

[Marsup]

What about removing the ability to do validation in a strictly synchronous manner?

Never gonna happen.

[gergoerdosi]

@Marsup: When are you planning to start the work on this? So many people ask for this feature. I think we should start working on the implementation even if we don't get it right the first time. We can always make refinements in later versions. We need this feature too, so I'm happy to help.

[Marsup]

There are clear alternatives to the lack of this feature, at least in hapi, so I don't want to rush that feature. Sorry but I'm really swamped at work at the moment and can only get so much done on my free time. I've tried to integrate the last few discussions to the spec but it's still an unfinished draft, if you have time you can help out finalizing it.

[gergoerdosi]

No problem, I can totally understand it. What is an alternative to an async validation on the payload for a given route? Server methods?

[Marsup]

All the validations support declaring a function. In the meantime I'd still use joi schemas to validate basic models features in that function, then do the async things.

Something like :

validate: {
  payload: function (value, options, next) {
    var r = Joi.validate(value, schema, options);
    if (r.errors) {
      return next(r.errors);
    }
    doSomethingAsync(r.value, next); // re-use the joi mutated r.value for type casts
  }
}

[gergoerdosi]

Got it, thanks! We can live with that for now.

[Marsup]

OK, it took some time but I've updated the RFC (see 1st post) with something that I think will be simpler and more powerful, hopefully I got it right this time.
Comments ?

[nlf]

this RFC looks good to me, i feel like you've covered all the use cases i can think of. my only concern would be making sure the behavior of assert with async validators isn't ambiguous.

[simon-p-r]

Looks good, can base be a object or just a key and value?

[Marsup]

@nlf it's mentioned that any attempt to use async with sync functions (assert & validate w/o cb) will throw.
@simon-p-r a joi object necessarily, any being the most naked base you can have, why ?

[simon-p-r]

I have polymorphic json schemas at present which use async routines as part of validation that I want to migrate to joi.

[marshall007]

@Marsup this looks really great!

Just to clarify, is the order of operations base -> pre -> rules or pre -> base -> rules? Based on the user example, it appears to be the former. Will there be a way to run custom type conversions before the value is validated against the base schema?

[Marsup]

Your guess is correct. What would be the use case to go before the base ?

[marshall007]

@Marsup if, for example, you wanted to implement a custom string to Date coercion, it would allow you to convert the string to a Date yourself before the base Joi.date() validator fails.

[Marsup]

Joi.date() should be fixed rather than adding a hook for that, unless you can think of other use cases.

[marshall007]

The real use case I had in mind is a bit more involved than just a string conversion. I was hoping to convert objects that look like this: {'$reql_type$': 'TIME', epoch_time: 1376075362.662, timezone: '+00:00'} into Date objects and vice versa.

[Marsup]

So you basically need joi.date() functions on a joi.object() base ?

[marshall007]

@Marsup pretty much, yea. The value I'm validating might already be a Date though, in which case we don't need to convert first.

[nlf]

for that use case couldn't you just use the pre to convert the object to a date and call the standard date validator in your validate method? i don't think it's worth the added confusion of having two places to preprocess your input

[Marsup]

Then the base could be an alternative with 2 possibilities. I'll agree that complicates it a bit but it's the best way to auto-document it.

[AdriVanHoudt]

Any progress on this? I actually encountered a use case where this would be very handy for us

[Marsup]

Nope, still twisting my mind about that double inheritance, I don't see it working in any possible way.

[AdriVanHoudt]

double inheritance?

[jfhbrook]

A few things:

1) I need this and it really hurts that this hasn't gone anywhere.
2) I would need to define multiple custom types and the extend syntax as proposed doesn't seem to meet this use case.
3) Double inheritance?

[Marsup]

1) I know, me too, until some company hires me to do just that, I have a paying job and a life, so, sorry...
2) There can be multiple types obviously, that's the point
3) The case described up there, where a property can be either a date or an object representing a date

[jfhbrook]

Walmart doesn't sponsor dev anymore? Bummer.

[jfhbrook]

Also: What would specifying 2 new types in an extend call look like?

[Marsup]

You specify the type on each extend, so 2 calls, I know it's lacking complex examples but I thought this one was clear.

[jfhbrook]

So given that .extend returns a wholly new instance, that seems odd. Less odd, of course, if that modifies the singleton. edit: Joi.extend(myFirstType).extend(mySecondType) isn't the worst thing though.

[Marsup]

It's keeping the immutable philosophy of joi, that way you're guaranteed that your version of joi cannot suffer from any side-effect of a 3rd party library that would also be using joi.

[jfhbrook]

I'm with it in terms of returning a new instance, I think that's by far the safest way to go. It just seems weird to create 2 instances of it when you only need one.

[Marsup]

There would be a single instance if you called extend with an array, that's documented.

[amcdnl]

@Marsup is there any updates on this?

[Marsup]

I have a local proof of concept with sync extensions but no unit tests. Async will take a lot more time I think, maybe I could beta release what I have already if there's interest.

[amcdnl]

Ya, would love to give it for a spin ( as others I'm sure ).

[MarkHerhold]

@Marsup I would love a beta release of your sync version! I have an
immediate need for it. :)

On Wed, Jan 20, 2016, 11:51 AM Austin [email protected] wrote:

Ya, would love to give it for a spin ( as others I'm sure ).

—
Reply to this email directly or view it on GitHub
https://github.com/hapijs/joi/issues/577#issuecomment-173269576.

[Marsup]

Don't hold your breath, it'll probably happen in February considering my workload, I'll probably have an intermediary v8 before that.

[cur3n4]

Very keen in trying the beta

[capaj]

@Marsup can't wait for it. It would help veery much on my current project. Let us know when you get back to it.

[Marsup]

So, I've pushed a preliminary version. It's very much incomplete, things will crash, it's not documented at all apart from tests, it's probably not usable in any self-documentation tool, ... well, don't use it in production :)

The API is probably compliant with what I had in mind in the spec but it will eventually change without warning if it feels better, so I do not promise any stability either. Oh and no async yet, sorry yall.

I'm still banging my head at how to late-extend any so that all types can benefit new rules but if it becomes too complex I'll have to let you order your extensions properly.

If you want to review, contribute code, tests or documentation, feel free.

[jfhbrook]

Word, I'm pumped. I'll try to kick the tires on it when I have a spare moment.

[Marsup]

Added a commit fully testing the inputs passed to extend, that might help. Schema is not complete yet but hopefully that will help transition what you do to new versions as I add more commits.

[Marsup]

OK it's probably feature complete now, I've added descriptions and support for automatic references resolution. Feedback welcome...

[AdriVanHoudt]

played with it a little bit and looks very nice, only problem I encountered was trying to name a rule '5', you cannot access it then :P not sure if this needs to be dealt with. Also tried using Joi inside the validate func and works, just wanted to go inception on it. Not tried with super complex stuff though.

[Marsup]

Names are intentionally not validated as javascript supports any string as a key, you'll just have to access it through square brackets as usual.

[Marsup]

Renamed parameters to params, sorry, it was tedious to type the full name and felt more consistent with hapi naming this way. Also added a setup function to add stuff (like flags) to your objects, that's something I overlooked in the specs and felt really necessary.

[Marsup]

If anyone is still interested it's published on tag 9-beta, still waiting for feedbacks...

[MarkHerhold]

I'll look at it next week! :)

On Thu, May 5, 2016, 4:26 PM Nicolas Morel [email protected] wrote:

If anyone is still interested it's published on tag 9-beta, still waiting
for feedbacks...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/hapijs/joi/issues/577#issuecomment-217268066

[kamronbatman]

I'll be ready to test it in a week or two. Sorry for the delay.

[svestka]

From documentation I would expect, that when pre method cast value, validation would be run against this casted value. Imho this code should NOT throw an error:

let joi = require('joi')
let joiCustom = joi.extend({
  base: joi.boolean(),
  name: 'alwaysTrue',
  pre(value, state, options) {
    return true
  }
})
joiCustom.alwaysTrue().validate('whatever')

[Marsup]

@peterslivka have any real use case ? It feels like your base should be Any.

[svestka]

Yes, with Any base it works, but you loose boolean validation. Real use case is may be localization, e.g.:

let joi = require('joi')
let joiCustom = joi.extend({
  base: joi.boolean(),
  name: 'booleanLocalized',
  pre(value, state, options) {
    if(value==='yes' || value==='ja' || value==='ano') return true;
    if(value==='no' || value==='nein' || value==='nie') return false;
    return null;
  }
})
joiCustom.booleanLocalized().validate('ano') // 'ano' should be true, but it fails validation

[Marsup]

So you don't trust your own code to return the type you want ? That doesn't seem like a valid use case to me. I feel like this is the kind of rule that should be checked in your tests, not in joi each time a validation runs.

[svestka]

Current boolean validator also covert "yes" or "on" string to true value. For me this is the very same use case. My point is not to build full-feature validator covered by tests, but to extend existing. This is not possible because if value is not valid by base validator, it will never reach preprocess function.

[MarkHerhold]

I really like this but I don't see a way to extend everything in one go. Maybe this was by design?

If I extend Joi.any(), I would like to be able to do:

• customJoi.number().myCustomRule()
• customJoi.string().myCustomRule()
• customJoi.any().myCustomRule(), etc.

without defining multiple custom Joi objects for each type via extend().

[Marsup]

@peterslivka ok, I was trying to minimize the number of hooks but you might have a point.
@MarkHerhold I don't yet have an idea about how to do it properly without making you recreate all the types, the problem is any is already the base of all joi types, so extending it doesn't impact others, which makes sense as far as inheritance goes, but it's not what I would want either.

[nfcampos]

@Marsup one possibility would be cloning any on any call to extend, irrespective of which base the extensions have, and then recreate _all_ the internal joi types inheriting from the cloned any. That would be a completely independent clone of the original Joi object with all types inheriting from this new clone of any. I believe this way an extension with base any would correctly appear on all other types. Obvious downside is this takes up more memory. (edit: on re-reading your comment this is probably what you meant by 'making you re-create all the types')

[Marsup]

@peterslivka another beta out (9.0.0-4) with a new coerce hook.

[MarkHerhold]

@Marsup The coerce hook is very useful (and necessary!). Great addition.

[Marsup]

Just broke the previous contract in 9.0.0-8 because it seems people do weird things with Joi, you now always have to return the value in extension functions.

[MarkHerhold]

I converted my plugin/extension to use the values instead of null and everything still works great. :+1: Easy change.

[arthurvi]

Are there any plans to make async validation possible in the validate function? I saw some older discussions about this subject in the repo but don't know what the plans are for the near future?
cc @Marsup

[Marsup]

It's planned. Can't promise it's "near" future.

[lukaserat]

Is there any possible approach for async?

[kamronbatman]

I vote for Bluebird.

[rafaelcorreiapoli]

Any updates on this ?

[bodinsamuel]

Hi everyone, any news ? :)

[mehulmpt]

Is there any update on adding async validation?

[bodinsamuel]

This issue seems abandoned, so I made a little side project to handle async validation with a syntax pretty much similar to Joi, if you are interested: https://github.com/bodinsamuel/altheia

[Marsup]

It's not abandoned, you're just all commenting on an issue that's been closed forever and everyone moved on to another, namely #1194.