Renovate: Use automated validation of configuration option values (e.g. JSON Schema)

[rarkins]

We should use a validation library to explicitly define and later verify configs (especially user-provided ones in renovate.json): Example: https://validatejs.org/ (but need to evaluate)

Example field: automerge (values are 'none', 'patch', 'minor', 'any')

Question: should validations be written in javascript, or do any of these libraries support a string/json approach of definitions? Some of our config objects are overloaded (e.g. can be string or array) so Javascript might be easier for that.

Such definitions should be incorporated into our existing definitions file: https://github.com/singapore/renovate/blob/master/lib/config/definitions.js

If the config fails to parse then it should produce an error that is presented to the user (e.g. like https://github.com/singapore/renovate/blob/beb37bc1283a9cadd38f863110be15a55ab7026a/lib/workers/package-file/index.js#L22-L28 does)

[ikatyang]

Another choice: JSON Schema

• Looks like: tsconfig.json for example
• Validator Libraries
• Editor Support: validate + autocomplete
  
  
  
  • VSCode screenshots
  
  

The JSON Schema file can be put in this repo for validating in service, and can also sync to their Store once in a while for editor support.

[rarkins]

@ikatyang thanks for the feedback. So in that case the existing definitions.js file would be converted to to a JSON file? i.e. this applies to the entire definition, not just the validation part?

[rarkins]

It looks like we could define our own definition of renovate.json but I'm not sure how we could apply it to:

1. config.js (used by self-hosted/administrators - deliberately JS not JSON so they can pull values from other locations)
2. renovate config embedded inside a package.json

I like the concept but have concerns if the validation only covers 1/3 of the methods we support for configuration.

[tunnckoCore]

you may want to try map-config by jonschlinkert or doowb. or it was named like that, but has pretty good flexibility to verify and sanitize json.

typing from mobile

[ikatyang]

@rarkins It just needs an input (object) and a JSON Schema definition (object) to validate, the JSON file format is for editor support in their Store, otherwise an object is enough.

For the editor support, config.js and package.json seems not available, but for validation they are all available.

[tunnckoCore]

Oh, my mistake (jon has too much packages - 1300+ ;d and i remember almost all of them ;d)... That was what I was talking about in my previous comment is map-schema - definitely give it a try. It has pretty good support.

[rarkins]

Proposed solution: JSON Schema

[felixfbecker]

It would be great to simply have a JSON schema (registered in JSON schema store) to get autocompletion and validation in editors

[rarkins]

@felixfbecker I need to get my head around JSON schema, but it should be possible.

FYI in the meantime:

• Any config you add to renovate.json in onboarding is validated
• If you ever want to modify your existing config, create a PR that includes "Renovate" or "renovate" in its title, e.g. "Reconfigure renovate". Renovate will then validate any updated config file in its next run and add a status check pass/failed and a message if failed

[rarkins]

https://glebbahmutov.com/blog/json-schema-for-the-win/

[felixfbecker]

I hacked together a schema generated from definitions.js. Unfortunately, nested properties are not typed (i.e. they are just type: object or type: array without the nested items). It's quite hard for me to find out what properties are allowed everywhere. @rarkins maybe this can serve as a starting point for you? https://github.com/SchemaStore/schemastore/pull/489

[rarkins]

Thanks @felixfbecker - that is definitely very helpful and I'll see if I can improve it. Most properties are allowed in most locations, although they don't always make sense.

[felixfbecker]

In that case, they would probably just reference the root object with $ref

[felixfbecker]

I did another PR to improve the schema that made all nested properties and list types work properly. I would consider it near perfect.

Would be great if you could do a PR whenever the config schema is changed

[felixfbecker]

Btw, I would recommend moving away from config presets that only set a single property - we can't validate the names of presets or give autocompletion for it, but we can easily for single properties.

[rarkins]

@felixfbecker that's really great - thank you! What's the best way to maintain it moving forward?

1. If enough of it is autogenerated, can we have a script that generates it from config/definitions.js with some scripted "massaging" to finish it?
2. If it's too much done manually, could that be addressed by having better metadata in config/definitions.js?
3. Or, could we commit the final file and apply some basic sanity checking during tests to at least make sure that all options in our config definitions are present in the JSON schema?

[rarkins]

I think we should also document how to use it, e.g. with VS Code, etc.

[rarkins]

Btw, I would recommend moving away from config presets that only set a single property - we can't validate the names of presets or give autocompletion for it, but we can easily for single properties.

I'll need to think about this more. One of the reasons for favouring presets was so that the config could be "self describing", however that aim hasn't been fully realised either. Presets will remain as an essential role regardless - hopefully increasing in importance to save people the need to manually tinker with code as often.

[felixfbecker]

But how is ":semanticCommitType(chore)" any more descriptive than "semanticCommitType": "chore"? Or ":pinDigests" more descriptive than "pinDigests": true?

Not doubting they are useful for bundling multiple properties of course.

[felixfbecker]

What's the best way to maintain it moving forward?

Would it be possible to make the JSON schema the source of truth? It's possible to generate documentation from JSON schema (actually doing that at my company - e.g. https://about.sourcegraph.com/docs/config/site/) and there are lots of validation libraries for JSON schema. SchemaStore has ways to link externally hosted schemas afaik. If more metadata is added to definitions.js I feel like we would just end up with a custom clone of JSON schema.

[rarkins]

If the JSON schema repository is the single source of truth then it means we can't automatically validate PRs to make sure it matches with our config definitions. And whenever you don't automate something, it usually falls behind.

My preference is that the JSON schema generation is done via a script, ideally wholly based off config/definitions.js but OK to have some manual massages. Only then can we be sure that it is correct/in sync, and be able to catch PRs that change definitions without updating the schema.

Even if the schema is best done by hand, I'd prefer it to be checked in so that we can at least validate that all options from config definitions are present.

[felixfbecker]

I think it is possible for the schema to be in this repository, and schemastore just pointing to it for autodiscovery.

My proposal was to get rid of definitions.js entirely and change the code to read the JSON schema instead, if that is possible.

[felixfbecker]

I think we should also document how to use it, e.g. with VS Code, etc.

In VS Code it will just work without any setup, since JSON support is integrated and it talks to SchemaStore to get the right schema.

[felixfbecker]

One last thing, it would be nice if Renovate ignored the $schema property:

[rarkins]

@felixfbecker that should hopefully be fixed by this commit: https://github.com/renovatebot/renovate/commit/b3fb4e3

Will be live in the app shortly

[rarkins]

What's needed is a script to read in lib/config/definitions.js and output a JSON schema file like exists in https://github.com/SchemaStore/schemastore/blob/master/src/schemas/json/renovate.json

We could then host it within the repo so that it's always up to date with master. We could potentially run the build check and make sure git status doesn't return the schema file to make sure all PRs keep it up to date too.

[felixfbecker]

I wonder, why not get rid of definitions.js and use JSON schema? Why use a custom thing when there is a standard, with lots of available tooling?

[rarkins]

We need the flexibility to store a lot of additional fields that I assume aren't welcome in a JSON schema.

[felixfbecker]

JSON schema is extensible, it shouldn't care about additional fields I think

[renovate-bot]

:tada: This issue has been resolved in version 14.56.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: