Conan: Define Conan hooks management

[danimtb]

Following the feature of Conan Plugins #3555 we should define the following points:

• Installation of plugins: conan config install with custom plugins parameter or normal?
• Merge strategy for plugins with same name or same subfolders.
• Automatic activation/desactivation of plugins in conan.conf?
• What should be the strategy to for plugin versioning?
• Should we think about a way to detect compatibility of plugins in respect to the Conan client version?

[uilianries]

Hi everyone!

Conan Plugins looks an interesting feature at first look, but could be more complex than usual. Years ago, before to use Docker, I had Vagrant + Ansible to create my C++ environment. Vagrant provides a good support for plugins where is possible to install directly by a simple command:

vagrant plugin install <plugin_name>

But looking from Vagrant perspective now is necessary to review the plugin and maintain the plugin manager. Conan only supports official recipes on Conan center now, but following the same path for plugins, the community will require official plugins also.

On other hand we have pytest project, an awesome unit test python library that support dozen of plugins and doesn't have any plugin manager. Each plugin is a pip package that consumes pytest API. Do you want code coverage + pytest?

pip install pytest-cov

Conan could provide a good API, including events (pre_download, pre_export...), but install plugins using Conan is not a good idea IMO. We already have pip to install any python stuff. Conan could manages it plugins by a config file but all stuff related to install, uninstall, update, version could be solved by pip.

[memsharded]

I liked the idea of being able to support plugins that could be managed directly from git repos, more exactly, they could be just part of the conan config install and put them in a repo and get them installed together with the settings, profiles, etc. Also switching plugins on and off, seems a bit more convenient if they are listed in conan.conf, instead of found in the python installation (would almost enforce using a python virtualenv to maintain things clean, which is not done by every conan user)

Though I also understand the reasons, and why using python packages would be a good idea. Maybe supporting both? We will discuss this issue for 1.9, not to implement everything, but at least to have some idea how we want to manage them.

[uilianries]

just a question using config. It needs a full git path to install. Is there any idea to include a plugin center?

[memsharded]

To be considered: is it possible to write plugins with other kind of extensions? Example: the optimized shutil.copytree that @sztomi was proposing.

[uilianries]

It could be .zip or any other file compressed format. :thinking:

[sztomi]

In my opinion the installation and version management of plugins via conan itself adds possibly unnecessary complexity. In the end, conan only wants to be able to import the modules where the plugins are implemented. It's already possible to manage python packages via conan if one wants to; it's also already possible to import code from the repo; and it's just a matter of setting the right PYTHONPATH if people use another means of installing the modules (like pip or conda). So I would concentrate on just loading and running the plugins around the extension points.

To me, a more interesting question is how to tell conan what to load. I would prefer a repo-level config because I feel that is more portable than something in global configuration. In this case, I think this is the same for both OSS and non-OSS. Think about a contributor cloning a repo that contains a conanfile that depends on a plugin.

As for the shutil.copytree thing: I wouldn't mind having a different kind of plugin that applies a monkey-patch.

class CopytreePlugin(MonkeyPatchPlugin):
  function = shutil.copytree

  def __call__(self, *args, **kwargs):
    pass

The difficulty of doing this without upstream support is that we can't reliably apply a global monkeypatch in conan code.

[memsharded]

I would prefer a repo-level config because I feel that is more portable than something in global configuration. In this case, I think this is the same for both OSS and non-OSS. Think about a contributor cloning a repo that contains a conanfile that depends on a plugin.

I'd say this is a more general issue, like being possible to define some remotes directly in the repo, which has been a demanded issue. Could be that the solution is something like the hierarchical system-user-project .gitconfig approach.

[sztomi]

Could be that the solution is something like the hierarchical system-user-project .gitconfig approach

I think that would be a worthwhile addition. Lots of the work we did on top of conan was about making stuff self-contained (except for the cache which we want shared normally).

[danimtb]

Update on this: Finally we have considered this issue mostly referring to the renamed "plugins" to "hooks". We understand having real python plugins for Conan could make a lot of sense for more internal things such as the shutil.copytree suggestion above. In the other hand, hooks would be still useful to perform checks or basically additional tasks related to conan events/commands and they should be also considered part of the Conan client configuration

So, considering hooks management, my first approach on this issue is the following:

• Copy always files under _~/.conan/hooks_ folder if present in a conan config install avoiding git related stuff (in case you have hooks as submodules in a conan config git repo).
• Do not mess with a "merge" solution as hooks might be complex, overwrite files if present (dangerous, consider a --force or similar).
• Avoid automatic activation of hooks when they are installed in the conan config install command, they should be already activated in _conan.conf_ if user wants them activated by default.
• Best strategy to "version" hooks would be to have them separated from the config and cloning them directly in the conan hook folder ~/.conan/hooks, that way users are able to update freely or checkout new versions in there.

Hooks will have a separate folder for each of them with the same naming of the python file entry point like: ~/.conan/hooks/attribute_checker/attribute_checker.py However there might be folders with shared code between hooks in some other directories (that is why we cannot "activate" hooks automatically for every directory copied under "hooks" folder).

[sztomi]

Thanks for the update, renaming to hooks makes sense. Was the hierarchical "system-user-project" approach considered looking forward? We really try to keep everything self-contained in our projects so we try to not rely on more global things. It's possible that a certain hook makes sense for one project but not the other. Furthermore, for larger monorepos like ours, it would even make sense to have one more layer: system-user-project-subdirectory (the latter being right next to the conanfile being processed).