The below represents current state of planning for presets. This implementation is inspired by eslint's approach to sharable configs and airbnb is used in many examples due to how well-known their eslint config is - they are not yet involved in Renovate!
Support a "presets" array at any level of a configuration. e.g. from a high level { "presets": ["airbnb/webapp-pinned"] } down to a low level like package rules: { "packageRules": [{ "presets": ["allAngular"], "groupName": "angular", "automerge": "none" }] }" where "allAngular" is a partial packageRule with packageNames and/or packagePatterns.
In other words, presets will essentially just be embeddable configuration that can be used both as a "complete" configuration (like eslint-config-x) or for simple config reuse for the lowest-level things like grouping together all "jest"-related packages, or all test packages, etc.
e.g. a config might look like:
"presets": ["automergeMinor", "doNotAutomergeAngular"],
"packageRules": [{
"packageNames: ["angular-animate"],
"automerge": "patch"
]}
It would then "cascade" from presets left to right followed by any custom config, e.g. the above would essentially mean "automerge all minor upgrades except Angular, but automerge patches of angular-animate".
To prevent repetition, it's necessary that presets themselves can include other presets. For example there might be a preset "pinDevDependencies" that sets pinVersions to true inside devDependencies. There might be another one called "pinNoDependenies" that simply sets pinVersions to false at the top level. Then there can be a third preset like this:
"pinOnlyDevDependencies": {
"presets": ["pinNoDependencies", "pinDevependencies"]
}
The algorithm we write to "resolve" presets into config would need to be smart enough to follow nesting up to some certain safe limit but to warn/abort if circular nesting is found.
It would be idea if configurations can be "self describing" with each preset having a human-readable one-sentence description, such a "Group together angular packages and do not automerge". This way Renovate configurations can be distilled into dot points describing what it will do without the need to understand the syntax. I'm still trying to work out how this would work though in cases of cascading and nesting. The above "pinOnlyDevDependencies" might contain this, for example:
"description": "Leave ranges as-is for dependencies, optionalDependencies and peerDependencies but pin devDependencies",
Maybe we have the concept of "descriptionOverride" for presets which defaults to true, i.e. a preset that embeds other presets will usually suppress the description of embedded presets but it can be set to false and will result in an array of descriptions.
Presets would be identified as scope/presetName e.g. a preset named "webLibrary" inside package renovate-presets-airbnb would be referenced as `airbnb/webLibrary".
We could also support scoped npm pages so an alternative would be to reference "@airbnb/webLibrary" which would resolve to the preset "webLibrary" within package "@airbnb/renovate-presets".
Question: is there any downside to this dual namespace? I am tempted to limit it to scoped namespace only because then only airbnb could publish @airbnb/webLibrary, whereas anybody could typosquat on renovate-presets-airbnb. Considering that publishing scoped packages to npm is free, do we lose anything by restricting to scoped namespaces only?
Suggest to use prefix like github:airbnb to imply the repository of github.com/airbnb/renovate-presets or github:airbnb/renovate-presets-new if the name were github.com/airbnb/renovate-presets-new. This way others like gitlab and bitbucket can be supported too.
If you manage many repositories, then reducing common config down to 1 or more preset names will be really nice, but wouldn't zero config be even nicer? I propose the following:
If a repository a/b (a = owner, b = repository name) is enabled without presets (e.g. empty renovate config), and there exists an npm package @a/renovate-presets and this includes a preset named default, then this will be used automatically. This means zero config when adding new repositories to renovate.
Downside of this is that the account 'a' on GitHub or GitLab is not guaranteed to be the same user/company as account 'a' on npm, so if account 'a' on GitHub happens to not operate 'a' on npm, and 'a' on npm has published default presets... then 'a' on GitHub must configure a preset array to override this. Is this small chance/pain worth it to benefit the majority?
Alternatively, we could remove the implicit use of these rules and instead check for them during onboarding and propose "presets": ["@a/default"] in the onboarding config. This means you still need a renovate config in renovate.json or package.json files.
I have already registered renovate-presets-default on npm and propose using it as a "special" set of presets. The primary goal of this is that the majority of user-created presets are contributed to this package for easier discoverability by the community + encouraging preset reuse. We can probably also add a little syntactic sugar by making this the only non-namespaced set of presets. e.g. if there is a preset "webapp" with default recommended rules for webapps then it could be referenced in configs and other presets simply by as "webapp" and not as "default/webapp" or "@singapore/webapp" if we were enforcing scoped namespaces.
Although there is some theoretical downside to having a "benevolent" monopoly on this namespace, I think the benefits (configuration simplicity and discoverability) are worth it. And of course, nobody is forced to use these defaults and nor does its existence cause any downside to scoped packages.
Unlike eslint configs/plugins, Renovate shareable configs will be executed on "server" side, and as such the security risk is much different. Because Renovate configs themselves are JSON, it makes most sense to restrict presets to be shared in JSON format only too (i.e. not JS).
If we are already using npm namespace for presets then the easiest location for them is within a renovate-preset-definitions field of the package's package.json. This enables Renovate to simply query https://registry.npmjs.com/<package> to get not just the latest package.json but also all versions prior too.
Hopefully the majority of users can express their Renovate configuration in one or few presets. Therefore, instead of requiring package.json > renovate > presets should we simplify to a renovate-presets array within package.json instead? This can potentially be confusing with the suggestion above for naming preset-definitions though.
One downside: if a user starts with presets but then finds they want to override something with "regular" config then they'll need to move their renovate-presets into a renovate field anyway.
If in future it is preferable to support preset definitions in JSON files within the repo, then this should also be possible. e.g. instead of a renovate-presets there might be a renovate-presets-embedded field that indicates that the full package must be installed to find the JSON files within. This will involve more complicated resolution logic from Renovate but seems manageable. The structure would presumably be a directory of preset names with .json extension.
Initially we can use "unversioned" (latest) versions of preset packages, but should consider a future case where supporting versions is preferred. Unlike eslint we don't really need to install the presets with npm, but it's still probably easiest to support versioning using package.json versions. i.e. someone who wishes their presets to be versioned can have a pinned or range versions of preset packages in devDependencies while someone who is happy with "latest" does not need to add them at all.
We want a way for users - both new and old - to discover existing presets and not "reinvent the wheel". Ideally we could have a repo/website that gathers all valid preset packages and makes them searchable by name/descriptions. In the meantime a quick and dirty "awesome renovate" repo could perhaps be maintained.
In theory, allowing parameterisation of presets would also reduce repetition, or at least make them more powerful. eslint configs allow parameters but I haven't thought of a compelling use case yet for Renovate presets.
presets or preset in package name?package.json ?package.jsonrenovate-presetsdepDependencies (optional)renovate-presets-default already).Then in no particular order, as demanded:
Thanks for @ikatyang for prompting me in #646 to write down my plans for shareable configs. For those of you "power" users (e.g. @hbetts, @charlike, @vvo, @jbaxleyiii, @luftywiranda13) I'm sure you've already experienced the feeling that you wish you didn't have to repeat/duplicate config so much. Additionally, while I know that the low-level "power" of Renovate syntax is appreciated by most, we would all benefit from our configs being more understandable/descriptive.
This proposal for "presets" intends to solve these problems and more. If you have the time then please provide your feedback here and help me tick off all the todo's before I get to work on it.
@rarkins Wow, this is gonna be awesome!
But what about if we migrate this repo to be a lerna monorepo then we also create some pre-defined good practices configs here inside the package directory?
So it's gonna be easier to new user and also for someone who wanna make their own configs can see some examples here
@luftywiranda13 thanks for the feedback. My description is long but the final result should hopefully be short and sweet!
Re: lerna, its creator wants it to go away in favour of yarn workspaces, so I would be unlikely to use lerna.
But I think your question is more generally: why have a separate repository for presets?
The main reasons why would be:
the presets one might have a LOT of commits/issues/comments and would make this repo perhaps too "noisy". I've already been told that this one has too many releases :)
could be useful to have separate maintainers for the presets one to reduce workload
Plus, they should be pretty independent. Keeping it separate means much less chance to accidentally "break" non-default ones if we "dogfood" presets for defaults.
hmm make sense,
But I think renovate should provide some good predefined presets.
Because:
Actually making the default presets internal to this repo is probably a good first step before spinning them out to a separate package
It's my intention that Renovate definitely uses these "default" presets rather than raw config and ideally covers the majority of use cases
Awesome! :tada: Do you know what time you'll need to implement the basic things? More specifically what i care is would it be possible to be in the near week or two?
Presets consist of name and description
Absolutely agree! And think that it should be required to have a description, presets without description will be ignored.
i.e. a preset that embeds other presets will usually suppress the description of embedded presets
:+1:
Use npm as primary namespace for presets
Agree and not agree. It should not be so restrictive. To use a preset without a scope it would be just
{
"presets": ["tunnckocore"]
}
_(renovate-presets- is auto assumed like in eslint)_
and if someone want to be scope it would just include the scope and preset still will be loaded correctly:
{
"presets": ["@charlike/tunnckocore"]
}
_(this one can be confusing, but it just should mean load the renovate-presets-tunnckocore from @charlike scope, so the package name in npm will be @charlike/renovate-presets-tunnckocore)_
Btw, I kinda don't like that "s" in presets ;d In config field it should be presets, but the package should be renovate-preset-foobar.
So i propose to change the prefix to renovate-preset-* instead of renovate-presets-*
Support implicit presets per-account
I'm against that. I understand it and it would be good feature, but may feel like some magic: if renovate is activated on repo and no renovate.json or no field in package.json, it will auto-magically decide that it should resolve some config (yea, it won't be just _some_ of course it will be registered package by the user, but still) for that repo? Not only that, but may just add needless general complexity.
I think it is enough easy to set config that points to your preset or preset of your company.
So, _"Autodetect default preset per-account?"_ - no. Feels too magical to me.
Presets as JSON
Probably and yaml, there someones that like its more clean approach. I prefer .js only because the support for comments, but i'm using json everywhere and always.
Preset definitions within package.json
and
Shorthand configuration in package.json
Separate fields? No. What's the benefit over just "renovate": { "presets": ["foo", "bar"] }, not make sense to me.
Preset versioning
Yep :+1: Great.
Thanks for the great feedback, @charlike.
Awesome! :tada: Do you know what time you'll need to implement the basic things? More specifically what i care is would it be possible to be in the near week or two?
For full support.. (e.g. customisable presets via npm) maybe within 2 weeks. But I think I have to see how the "internal" ones work first to see if there are any unforeseen problems.
Absolutely agree! And think that it should be required to have a description, presets without description will be ignored.
I'm not sure if we can/should necessarily "ignore" them, but maybe we have other ways to highlight the problem, e.g. linting, inclusion in the index, etc.
To use a preset without a scope it would be just
{
"presets": ["tunnckocore"]
}
_(
renovate-presets-is auto assumed like in eslint)_
As you have left off any preset name, then we'd need to imply it (e.g. default) here.
This approach non-scoped is of course technically possible, but I still have doubts if it's a good idea to have "two" npm namespaces for this - i.e. scoped and unscoped. For example do you think you'd feel the same way if someone already registered renovate-presets-tunnckocore before you could? I wonder if eslint would go back and time and support only scoped presets.. if npm scopes were around when eslint plugins/configs were created.
and if someone want to be scope it would just include the scope and preset still will be loaded correctly:
{
"presets": ["@charlike/tunnckocore"]
}
_(this one can be confusing, but it just should mean load the
renovate-presets-tunnckocorefrom@charlikescope, so the package name in npm will be @charlike/renovate-presets-tunnckocore)_
That's not how eslint works (I think) and I don't like the confusing nature of it. They say "when using scoped modules it is not possible to omit the eslint-config- prefix. "
Btw, I kinda don't like that "s" in presets ;d In config field it should be
presets, but the package should berenovate-preset-foobar.
So i propose to change the prefix torenovate-preset-*instead ofrenovate-presets-*
This is where the Renovate and eslint approach is a little different. It's my intention that each package includes multiple presets, hence the plural presets in the package name. We can consider this to be undecided (I will add it to the list) but I'm leaning towards presets not preset.
Support implicit presets per-account
I'm against that.
I was thinking this could be useful for prolific package authors and you seemed to have the most repositories I've seen so far. So if you give it a thumbs down, then probably most others won't need it either.
Presets as JSON
Probably and yaml, there someones that like its more clean approach. I prefer
.jsonly because the support for comments, but i'm using json everywhere and always.
Each preset will need to embed JSON so I'm not sure how practical it would be to support yaml. It can come later if proven practical and necessary.
Shorthand configuration in package.json
Separate fields? No. What's the benefit over just
"renovate": { "presets": ["foo", "bar"] }, not make sense to me.
Sure, we kill that idea to keep it simpler/consistent.
As you have left off any preset name, then we'd need to imply it (e.g. default) here.
It would be just collection of other presets (probably). Actually, no, it would be just my current config :D
That's not how eslint works (I think) and I don't like the confusing nature of it.
Mm don't know. I'm almost agree to support only scoped configs, but it may be limiting for some because it is "a bit harder" to publish scoped package ;d I mean, it may be hard for some new users. I hate them, but yea, we all one day were newcomers :D From time to time i'm thinking about them, that's why i'm refreshing my CONTRIBUTING.md too (just side note: you should read dwyl/contributing and opensource.guide they both are awesome!).
For example do you think you'd feel the same way if someone already registered renovate-presets-tunnckocore before you could?
That's one of the main reasons why so many like scopes, but i don't. It still exist the ability to some to register the unscoped package, then what? Again two packages, again room for mistakes and typos.
As you said it for ESLint, yes, it's true for me too. If there was that feature when i started... Now is very hard to switch all of my packages. Yea of course, it probably won't be all, but still. Actually it's not hard, it's a matter of a click, but i'm scared of the wave of things that will follow after that click.
In last 2-3 weeks, everyday, I'm thinking to make the switch and click the button, but still can't consider.
This is where the Renovate and eslint approach is a little different. It's my intention that each package includes multiple presets, hence the plural presets in the package name.
Yea i know that, felt it. But eslint has _multiple_ things (in most times multiple rules) in one config, but it is not named eslint-configs-. But probably that's why because the word config isn't real and is short for configuration? Don't know, sue me, english is not my native, so this question my look... ;d Yea.
Babel follow the singular way too - all things: plugins, transforms, presets. The names of the packages are singular, but the field name is plural.
Browserify follow that convention too with its transforms.
I was thinking this could be useful for prolific package authors and you seemed to have the most repositories I've seen so far. So if you give it a thumbs down, then probably most others won't need it either.
Hehe, yea. I think when NPM is the core behind that concept and is our source of truth, then that's enough, because everyone uses it and has access to it no matter github or gitlab. Tools such not mess in this things. Of course they can give some features, but adds and forces maintainer of the tool to think. It not worth for me, don't know. I probably will use it if it is implemented, but it still will feel me magical :D
Each preset will need to embed JSON so I'm not sure how practical it would be to support yaml. It can come later if proven practical and necessary.
I don't see what's the problem. The app will translate the yaml to JSON in any way, so it won't be a problem to merge both objects. Don't know. I'm not hard about that.
Btw, one more thing about scoped force. To ask you as semantic-release user. Is it a problem to it to publish scoped packages? And how should/can be configured? That's one more reason why i don't want to click the button to swithc my npm profile to org scope.
I've never needed to publish a scoped package but is it complicated? Doesn't sound like it based on the docs.
I've never needed to publish a scoped package but is it complicated?
Me too. But it's not mentioned in the docs and didn't found any info either. And it's sure that it need some configuration i believe. Hm. Wait, it probably reads the global/local npmrc, so you should give it there. hm
edit: yea, not so hard thing. one more thing is that you should update the semantic-release npm script to add --access public
To publish a public scoped package, you must specify --access public with the initial publication.
-- npm docs
{
"scripts": {
"semantic-release": "semantic-release pre && npm publish --access public && semantic-release post"
}
}
From https://docs.npmjs.com/misc/scope
To publish a public scoped package, you must specify --access public with the initial publication. This will publish the package and set access to public as if you had run npm access public after publishing.
someone who writes a lot of libraries will likely have identical configs for each
All my personal open source projects (about 8 of them) have identical configurations, so I definitely understand this point.
This implementation is inspired by eslint's approach to sharable configs and airbnb is used in many examples due to how well-known their eslint config is
It also aligns well with other tooling, such as Babel.
In other words, presets will essentially just be embeddable configuration that can be used both as a "complete" configuration (like eslint-config-x) or for simple config reuse for the lowest-level things like grouping together all "jest"-related packages
Yes! I was actually thinking of Babel, and needing to update Babel packages in lock-step with one another as a single pull request. Jest, though, is another great example. It would be awesome if package collections, like Jest and Babel, published renovate configuration packages that would ensure consumers of those packages received pull requests that made sense.
Presets may override each other and also be override by configuration
Seems good to me.
Presets would be identified as scope/presetName e.g. a preset named "webLibrary" inside package renovate-presets-airbnb would be referenced as `airbnb/webLibrary".
I would actually lean towards each package containing a single preset, instead of a collection of presets. In my opinion, it's less confusing, and easier to on-board with, if the preset name matches the package name (minus the renovate-presets part of the name).
I am tempted to limit it to scoped namespace only because then only airbnb could publish @airbnb/webLibrary, whereas anybody could typosquat on renovate-presets-airbnb. Considering that publishing scoped packages to npm is free, do we lose anything by restricting to scoped namespaces only?
An interesting proposal.
Support version control platforms later
Personally I'd love to see npm and yarn remove support for source control platforms as package hosts. That feature has, in my experience, led to poor practices, such as not using a specific semantic version, or version range, and an up-time guarantee only as good as the worst SCM referenced by any package in your dependency tree.
If a repository a/b (a = owner, b = repository name) is enabled without presets (e.g. empty renovate config), and there exists an npm package @a/renovate-presets and this includes a preset named default, then this will be used automatically. This means zero config when adding new repositories to renovate.
Downside of this is that the account 'a' on GitHub or GitLab is not guaranteed to be the same user/company as account 'a' on npm, so if account 'a' on GitHub happens to not operate 'a' on npm, and 'a' on npm has published default presets... then 'a' on GitHub must configure a preset array to override this. Is this small chance/pain worth it to benefit the majority?
After reading the first paragraph, the second paragraph was exactly what I was thinking (and what was worrying me).
I'm against that. I understand it and it would be good feature, but may feel like some magic: if renovate is activated on repo and no renovate.json or no field in package.json, it will auto-magically decide that it should resolve some config (yea, it won't be just some of course it will be registered package by the user, but still) for that repo? Not only that, but may just add needless general complexity.
I kind of agree with @charlike. It's too magical.
I prefer explicit configuration. If you want behavior X, you must say you want X. If you see behavior Y in your project, then you know exactly where to go (X is specified, but not behaving like what X said it would. Navigate to project X and inspect.)
Special namespace for default Renovate configurations
I have already registered renovate-presets-default on npm and propose using it as a "special" set of presets. The primary goal of this is that the majority of user-created presets are contributed to this package for easier discoverability by the community + encouraging preset reuse.
Another suggestion would be building on top of the npm registry by either linking to a pre-defined search query, like Babel does, or creating, and hosting, a list of all presets, like Yeoman does.
In the later case it would be easier to provide search functionality that allows user to search for presets that perhaps include references to Angular (for grouping), or to filer out all configurations that pin.
Absolutely agree! And think that it should be required to have a description, presets without description will be ignored.
I'm not sure if we can/should necessarily "ignore" them, but maybe we have other ways to highlight the problem, e.g. linting, inclusion in the index, etc.
Maybe a eslint-plugin-renovate that validates your renovate configuration in your project, ensuring all options are valid, no duplicated options, no configuration that leads to no effect (such as if I managed to set pin to true and false for all dependencies, causing renovate to be unable to do anything) etc.
That's not how eslint works (I think) and I don't like the confusing nature of it. They say "when using scoped modules it is not possible to omit the eslint-config- prefix. "
Here's a list of all the lookup rules used by ESLint -
https://github.com/eslint/eslint/issues/3123#issuecomment-123752646
And you are correct. You can't leave off eslint-config when using a scope.
It would be awesome if package collections, like Jest and Babel, published renovate configuration packages that would ensure consumers of those packages received pull requests that made sense.
I'm not confident this will happen, but also don't think it's necessary - I think "the community" can plug the gap. e.g. eslint plugins for React or Angular aren't necessarily published by the maintainers of those packages.
I would actually lean towards each package containing a single preset
It seems like this is an issue we need to expand on.
If we have one preset per package, then this will greatly discourage modular/composable presets in favour of large/monolithic presets. It's my goal that even the "full" presets (such as your own) would probably be something like:
..instead of your duplicating/rewriting the entire "webLibrary" or other presets yourself. But if we force each small preset to be in its own file (e.g. one file just for a regex that identifies all angular packages that need to be grouped together) then that overhead will discourage modular presets.
Therefore I'm firmly in the opinion that some packages - such as the default presets one - need to contain many presets inside. The question is: do we need to distinguish between these modules presets/rules (e.g. a regex for matching angular, or a rule which achieves "pin everything except dependencies") versus full "configs" (e.g. your personal open source config for your 8 repos).
Originally my idea had been for three different types - preset groups, preset rules, and preset configs.. then I realised that these are all basically just arbitrary config objects so we may as well call them all "presets" and keep it simple.
One more point: although eslint defaults to a single preset per package, it fully supports having multiple configs per package.
Work on internal presets logic is mostly done (See https://github.com/singapore/renovate/pull/658)
As a result, onboarded configs would now have a renovate configuration like this:
{
"presets":["library"]
}
or
{
"presets":["app"]
}
The more of these rules/presets I write, the more I'm liking the eslint idea of "extends" rather than "presets". Certainly from a preset author point of view, "extends" seems most logical. For an end user whose renovate config may be no more than a single extends/presets.. "preset" maybe makes most sense for single and "presets" for multiple. Then again eslint has raised awareness of "extends" fairly well so using it for every case is possible.
Also possible: naming as renovate-config-x instead of renovate-presets-x
Also considering:
I'm for that with both hands. Following eslint style would only benefit. It is well known and when you just say to users or end users "config and presets are just like in eslint" they'll get it immediately. And yes, renovate and its config more feels and seems like eslint, not like babel and browserify. Because it gives exactly "per rule" (here rule is each separate option) configuration flexibility.
best way to allow intelligent
example for the third case? i cant think of
Re description grouping:
Option 1:
Option 2:
This is only relevant for preset authors and not end users - end users should always use "extends".
I don't think option 2 is semantically correct because we are extending in ALL cases and this is only referring to how we autogenerate descriptions
Example of inheriting: full configs will usually just be a list of other configs and although you will use description to describe the preset simply, you would want the users to see the list of child descriptions.
Example of overriding: Define a package group "eslint". Define another for "stylelint". Now add a parent group "all linters" that extends both of those. The user should just see "All lint packages" as description and not 2 or 3 separate descriptions.
An example of combining would be for the renovate package itself:
Package rules and groups are related.
Example:
Ie although we want to apply an automerge to all linters, this doesn't mean we want to group them all into one.
It's challenging to name these package name and pattern presets because the most obvious name for them is something like angularGroup or groupAngular. But maybe we're just defining them and not actually grouping them from a Renovate perspective.
Right now I'm using the convention "allX" eg "allEslint", "allAngularJs", "allLinters" etc. and "groupX for any rule I add to group those, eg groupEslint
Naming of package patterns mad groups could be assisted by clever naming of files. Eg renovate-config-groups would have the prefix groups/x such as groups/eslint
Ultimately though there's still a naming confusion between package list definitions and the Renovate concept of "grouping" - ie applying same rule to many packages vs grouping packages together PR-wise. Renaming these concepts independently of presets would be useful but I haven't thought of anything good except "package lists".
Most helpful comment
Thanks for @ikatyang for prompting me in #646 to write down my plans for shareable configs. For those of you "power" users (e.g. @hbetts, @charlike, @vvo, @jbaxleyiii, @luftywiranda13) I'm sure you've already experienced the feeling that you wish you didn't have to repeat/duplicate config so much. Additionally, while I know that the low-level "power" of Renovate syntax is appreciated by most, we would all benefit from our configs being more understandable/descriptive.
This proposal for "presets" intends to solve these problems and more. If you have the time then please provide your feedback here and help me tick off all the todo's before I get to work on it.