Conan: [question] Usage of python-requires in a generator

[fredericbloch]

Hey guys!

how would you use python-requires in a generator?

The reason I ask is, that I have e.g. a very simple use case, which is a custom python-logger-implementation. And I want to use the same implementation for multiple packages. Using the legacy-python-requires worked like a charm. But since we moved to v1.28.2, I want to use the non-legacy python-requires. This still works good on ConanFile-classes - but I have no idea how to do this on a generator - since the conan-generator-class does not have an attribute python_requires nor python_requires_extend. Can you please give me a hint where I'd have to add it? Adding a python_requires to the ConanFile-class of the generator-package does not work as far as I tried it.

• [x] I've read the CONTRIBUTING guide.

[fredericbloch]

This is an example system, using the legacy-python-requires:

• conan_generator_pyreq.zip
• consumer-pkg -> requires the pyreq and the generator
• generator -> requires the pyreq
• conan install my_consumer/0.1@local/testing -e LOGLEVEL=DEBUG -g my_generator

Explanation:

• pyreq is the implementation of a python-logger-class
• When consumer command conan-install gets executed, the consumer can change the Log-Level of the messages on the conan install. This might seem useless, but it is very helpful every now and then.
• The conan-install is executed with a "-g" argument to call a generator as well to execute logic on the consumer-client.
• The pyreq-logger reads the conan-env to set the log-level for its logging-class.

[solvingj]

This is indeed a good question, and a complex scenario that may currently have issues (I'm not sure). We're currently trying to figure out the best way to bring together python_requires, custom generators, and the -g flag. There are also some other related initialization process considerations which may make this issue even more complicated than it looks. Will keep you posted.

[fredericbloch]

Thanks for the info that you'll be looking into it.
As for now, would you @solvingj still have any idea, how I could work around this until a clean solution is found?

[memsharded]

Hi @blockontherocks

Yes, we were working yesterday in a working solution, seems doable, @solvingj will be updating later with the code changes.

[solvingj]

conan_generator_pyreq_new.zip

This zip file contains an implementation which might check all your boxes. Please have a look and let us know what you think.

[fredericbloch]

Thanks for the quick solution!

I am not quite sure whether I might be doing something you did not intend to - but when I do a conan install on these - this happens:

conan install my_consumer/0.1@local/testing -g MyGenerator -e LOGLEVEL=DEBUG
<...>
ERROR: Invalid generator 'MyGenerator'. Available types: txt, gcc, compiler_args, cmake, cmake_multi, cmake_paths, cmake_find_package, cmake_find_package_multi, qmake, qbs, scons, visual_studio, msbuild, visual_studio_multi, visual_studio_legacy, xcode, ycm, virtualenv, virtualenv_python, virtualbuildenv, virtualrunenv, boost-build, pkg_config, json, b2, premake, make, deploy, markdown

Since you removed the class to be derived from the Generator-class, I don't see, how it would be available as a generator on the cmdline. Does this work on your end?

[solvingj]

I'm super sorry, I failed to mention that part. Since the beginning of custom generators, support for passing at the CLI via -g was never added. It is still not supported with any use of python_requires. We really want to support his case, and I'm actively working on trying to define a scope and strategy for that.

However, last year, the following pull-request was accepted which enabled custom generators via the CLI in a new way:
https://github.com/conan-io/conan/pull/7527

It is experimentally supported/documented here:
https://docs.conan.io/en/latest/howtos/custom_generators.html#storing-generators-in-the-conan-local-cache

Please have a look and let me know if this strategy works for you.

[fredericbloch]

I see. Well in this very use-case I don't think it will work with that. So I'd might have to switch to implementing this differently, and change my implementation in generators to normal conan-helper-packages with strict method execution.

Just to tell the story here: What I did was to do similar stuff like the deploy-generator for a specific tool which needs a huge amount of files inside its' own root, delivering those files as conan-packages and creating the file-structur necessary for that tool to be runable. A generator was quite handy for this, because I never had to call methods within the recipe and could decide whether to call it or not on an install. For obvious reasons I wanted to use the logger-class for these generators as well as for any other package. The new implementation will enable the CLI-call, but as you said - not the python_requires.

If you'd think I can provide any more input for your scope/strategy, I'd be happy to help.
Thanks for the support and having a look at it! Looking forward to the new implementation!

[solvingj]

You could try the method of defining a new conanfile.txt or conanfile.py which exists solely for the purpose of the deployment. In either, you can specify all the dependencies of your application/tool, and then you can then use the imports to specify a list of paths to copy, including their source package, source path, and destination path. Then, you can run conan import . and it will use the local directory as the "base path" for the copy destinations.

[fredericbloch]

Well I see what you mean. In our use-case this is not just a simple copy, but generating some structures with symlinks as well (a nightmare tbh). The logic for that is quite some lines of code. Using import will not do the job for that. Implementing the "script logic" into a package_info() of an helper package will do the job though. It will also fix the problem with pyreqs in generators.

To show what I mean:

• XYZ-pkg
• |- A-pkg
• |- B-pkg
• | |- tool-pkg (only repackaged tool itself)
• | |- helper-pkg (only deploy-logic)
• | |- resource-pkg (only files)
• |- C-pkg
• |-

We need to install the conan-system-pkg on a client, by a given full command e.g. as:

conan install XYZ-pkg/<version>@<user>/<channel> -u -if . -e LOGLEVEL=DEBUG -g virtualenv -g virtualrunenv -e SPECIAL_DEPLOY_VAR=var_1

Previously we would add another generator to do the magic which the helper-pkg does here for the B-pkg. The good thing about this is, that I only need to know the command - nothing else. And this can be provided easily for a client that has been configured to the according remotes.

I can't handle this using a conanfile.txt, since it does not let me define any environment variables. These might be used for the exactly same package, but a different active config-file for a tool down the line. So my deployment does not work the way I want it to. If I'd have to use a conanfile.py, I'd have to manage that as well as the original system conanfile. And I'd still have to do special commands, because I can't enable conan to generator the virtualenv on default on an consumer-install (as far as I know). Also I don't see how I could insert special conan-environment-variables using the conanfile.py either.

I really want to point out that this is an highly unusual and specific use-case with loads of stuff you guys did not intend it to do. But it works like a charm and we are able to do deployments that way and package a whole lot of stuff in a CI, deploy it to machines and be up and running with a single conan-install command - which is amazing.

As far as the issue/question goes I think we'll wait for the next generator-implementations to come and switch to those, if they are available. But for know we might move to helper-pkgs instead of generators, since it does not seems we can work around this and do not want to use the long-gone-legacy-python-requires anymore.

Feel free to ask me about this if you're interested in what we do or why we do this. The question can be closed though from my perspective. Thank you so much for your time and the insights!

[solvingj]

Sorry, I meant to use the imports() method of conanfile.py here, I should have provided link:

https://docs.conan.io/en/latest/reference/conanfile/methods.html#imports

You should be able to perform the same arbitrarily complex python logic you were performing already.

• IMO, your case is not highly unusual or specific. Most enterprise conan use-cases end with complex deployment scenarios.
• Yes, you would have to manage another conanfile.py, which seems tedious and unnecessary
• I had the same feeling in the original issue that inspired deploy generator and deploy functions,
  https://github.com/conan-io/conan/issues/3280

It seems like you have an idea about how to achieve your goal with reasonable compromises around the current features. We will continue to pursue -g generator support for custom generators, so if/when you find that is available and it does not solve your use case in a better way, please open another issue to let us know.