Joi: Recursive schema (tree-like) validation

[Marsup]

Hello,

I'm wondering what's the best way to accomplish recursive schema validation.
Let's say I have an object that contains a property "children" holding an array of objects of the same kind.
The only way I see is to create the 1st layer of validation to get a joi object without "children", then call .keys again on it like schema = schema.keys({ children: Joi.array().includes(schema) }) but that only gets me to validate the 1st level, not recursively.
Any tips ?

[Marsup]

@hueniverse you mentioned in #154 it was possible, would you mind making a short snippet for that ?

[hueniverse]

That particular example was not full recursion. I don't see how this can be done without special support for it.

[krisb]

I would like to be able to handle a recursive schema too.

@hueniverse - I'd be happy to put a PR together if you have some guidance on how you would like this done.

[Marsup]

Hey @hueniverse, I've been thinking a lot about this one, it seems that the only thing keeping it from working in the current state is that both object.keys and array.includes/excludes are making clones, if current objects were modified it would work. Would you be open to new alternative methods to these that would do that ?

[hueniverse]

@Marsup what would that look like? Show me a code example of how you would use it.

[Marsup]

@hueniverse Considering the OP example :

var schema = Joi.object({
  name: Joi.string()
});
schema.alterKeys({ children: Joi.array().includes(schema) });

It's the same API but without the inside clone, no need to grab the return value either since schema is modified.
Same way for arrays.

[markstewart]

I was in need of this feature myself and have implemented something for discussion in the pull request above.

Example:

var tree = Joi.object({
    name: Joi.string(),
    children: Joi.array().includes(Joi.recurse())
}).recursive();

It does maintain the immutable property by creating a marker recurse object that gets replaced with the type marked with recursive() at validation time.

[Marsup]

This might prove easier to describe having a concrete type.
You're missing recursive arrays though.

[markstewart]

@Marsup Can also use with recursive array types so for example:

var array = Joi.array().includes(Joi.number(), Joi.recurse()).recursive();
array.validate([[1, 2, 3, [3, [4, 5]]]], function (err, value) {});

I tried to stay away from mutating the schema objects. Not sure of the implications but it seemed an important property to maintain.

This could also be solved by introducing named types so maybe something like any.typeName(string) and Joi.namedType(string) to reference the previously marked type instead of recurse()?

[Marsup]

You're missing tests on it then :)
I thought it would be doable with existing Joi.ref but I don't have time to investigate right now.
Will it still work with 2 recursive calls ? (ie. recursive array inside a recursive object)

[markstewart]

@Marsup My proposal won't work with some nesting scenarios since it doesn't allow a way to reference the outer type from within the inner one. Will need to try something different to support that.

[martinheidegger]

Just figured out that this is very important to one of my tasks. +1 need it too, right now badly.

[Marsup]

I need it as well but I'm still searching for a decent API to represent that.

[Lotes]

I think you need a kind of "defs" section like in SVG - a region of predefined blocks for later reuse. Using a lazy lookup function you could realize the recursive aspect.

var defs = Joi.definitions(function() {
  //where "this" is the "definitions" object
  return {
    //name: type pairs
    tree: Joi.object.keys({
      value: Joi.string(),
      children: Joi.array().items(this.lookup('tree')) 
      //lazy lookup call (returning a function that will be evaluated on validation)
    }),
    anotherType: Joi.number(),
    //...
  };
});
var treeSchema = defs.lookup('tree');

Is that doable?

[DavidTPate]

Depending on the use case, I could see a representation similar to JSON schema's "additionalPropteries" or "patternProperties" working.

I think the best solution though would be to be able to reference another schema without making a clone of it.

[tjmehta]

Ran across this problem today. Would love to send a PR if anyone has implementation ideas.

[Marsup]

Quick poll, what does everyone think about a Joi.lazy primitive that would be used as follow :

const schema = Joi.object({
  children: Joi.array().items(Joi.lazy(() => schema))
}) 

With probably an additional string argument or something for description.

[Eirik81]

I also need to validate recursive objects. I think the Joi.lazy primitive seems like a good solution.

[AdriVanHoudt]

just an idea: Joi.lookup()?

[Marsup]

With the same behavior or something else entirely ?

[AdriVanHoudt]

same behaviour, just another name that popped in my head

[myndzi]

I was just looking into this to, amusingly, validate the output of schema.describe such that all items in the schema have defaults. I was looking for something like Joi.ref() but rather than referring to a _value_ it would refer to a _schema_. I'm not sure how I feel about lazy where the schema is specified by some callback function. There's already support for resolving dependencies for .ref, but in this case since it would be recursive, I'm not sure that works. It seems to me that referencing the schema to recurse by its string key is more in keeping with the Joi api as is, and simpler, too... though the syntax would have to be expanded to allow support for root references. Maybe something like this:

const schema = Joi.object().keys({
  children: Joi.object().pattern(/./, Joi.recurse('^'))
});

Or,

const schema = Joi.object().keys({
  foo: Joi.string(),
  bar: Joi.object().pattern(/./, Joi.recurse('^bar')
});

[BobCaSUaL]

Searching for recursive validation i just fall in this topic, then playing with the node shell i find that solution

var obj = Joi.object().keys({b: Joi.any()})
obj = obj.keys({a: obj}) //add the reference extending the base object
obj = obj.keys({a: obj}) //add recursion

//try it
obj.validate({ a: { a: { b: 3 }, b: 4 }, b: 3 }, (err, result) => {if(err) throw err; console.log(result);})
/* prints "{ a: { a: { b: 3 }, b: 4 }, b: 3 }" */
obj.validate({ a: { a: { b: 3 }, b: 4, c: 5 }, b: 3 }, (err, result) => {if(err) throw err; console.log(result);})
//throws ValidationError: child "a" fails because ["c" is not allowed]

despite this solution I like the proposal for

const schema = Joi.object().keys({
  foo: Joi.string(),
  bar: Joi.object().pattern(/./, Joi.recurse('^bar')
});

[myndzi]

By my understanding from looking into it the other day, this will only actually work to the depth that you repeat line 3. That is, it's not recursing, it's just explicitly specifying another level of depth each time you do obj.keys({ a: obj }). In your example, this fails:

obj.validate({ a: { a: { a: { b: 3 }, b: 3 }, b: 3 }, b: 3 })

[MarkHerhold]

Does anyone have a hack solution to this or a PR in-progress?

I think one solution would be to recurse the tree yourself, validating direct children and propagating validation errors up the call stack. Another would be to use another validator for the time being (like ajv which seems to have solved this). 😞

[jdarling]

Joi.custom(function) seems like it might be useful. Though the name may not be so good.

Basic idea:

const validateTaskOrSubTask = (value)=>{
  if(value instanceof Task){
    return Joi.validate(value, Task);
  }
  if(Array.isArray(value)){
    return value.map(validateTaskOrSubTask);
  }
  return Boom.invalid('Unknown or invalid value'); // Yeah, I don't know what this should really look like...
};

const Task = Joi.object().keys({
  type: Joi.string(),
});

const Job = Joi.object().keys({
  tasks: Joi.custom(validateTaskOrSubTask)
});

Excuse my typos, but its a proposal so hopefully they are allowed :D

[Eirik81]

I'm using hapi-swagger 6.1.0 and joi 9.0.0. When using Joi.lazy for response validation loading the /docs webpage fails with this message:

500 : {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"} 
http://localhost:9999/swagger.json

In the console I get this error:

160714/104936.718, [error], message: Uncaught error: Cannot read property 'type' of undefined stack: TypeError: Uncaught error: Cannot read property 'type' of undefined
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:144:24)
    at Object.internals.parseArray (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:396:40)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:181:30)
    at joiObj.forEach (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:93:46)
    at Array.forEach (native)
    at Object.properties.parseProperties (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:84:16)
    at Object.properties.toParameters (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:40:36)
    at Object.definitions.appendJoi (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/definitions.js:36:41)
    at Object.internals.parseObject (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:333:74)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:176:30)
Debug: internal, implementation, error 
    TypeError: Uncaught error: Cannot read property 'type' of undefined
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:144:24)
    at Object.internals.parseArray (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:396:40)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:181:30)
    at joiObj.forEach (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:93:46)
    at Array.forEach (native)
    at Object.properties.parseProperties (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:84:16)
    at Object.properties.toParameters (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:40:36)
    at Object.definitions.appendJoi (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/definitions.js:36:41)
    at Object.internals.parseObject (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:333:74)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:176:30)

[Marsup]

And it's clearly an error in hapi-swagger, how does this concern me ?

[Eirik81]

I posted it here since the error only occurs when using Joi.lazy, so I thought there might be a compability issue between hapi-swagger and the joi.lazy method. But I'll post the issue at the hapi-swagger project as well.

[lock[bot]]

This thread has been automatically locked due to inactivity. Please open a new issue for related bugs or questions following the new issue template instructions.