Conan: exports_sources not having effect when conan source is used

[bschindler]

Conan Version 1.0.2, Ubuntu LTS 16.04.

I have a package which specifies a single file for export_sources. When creating the package using conan create, everything is fine. If I however run conan source . --source-folder tmp/source as proposed in the documentation, the files specified in export_sources are not copied.

Am I doing something wrong or is this a bug?

[memsharded]

This is intended behavior.

Locally, we realized it was extremely confusing to have 2 copies of the source code. Typically the one you get from version control, and the one that would be copied by conan. And the worst thing is that the build would be done against the copied sources, which are out of version control, so if opening an IDE (which is one of the main goals of the local flow) and editing the source code to make fixes, it would be modifiying the copy that is not under version control.

So locally, exports_sources are not copied. But the local flow can refer directly to the original source folder in the build, export-pkg, package commands with the source-folder argument which works by default in many cases. As always, it is a trade-off, but it seems to not affect badly to reproducibility, while avoiding many annoying things in the local flow.

[bschindler]

It is a bit irritating that conan source followed by conan build behaves different than conan create. I have the following case:
I'm creating a recipe for a library which is just shipped in terms of source files, no build system included. I intend to create a package which contains a static lib with those source files. So what I did is have an exports_sources for a custom made cmakelists.txt and a sources command which downloads the sources. Calling cmake in the source folder should then trigger my custom cmakelists which builds the library. Works with conan create but not with conan source.
What is the recommended way of dealing with this situation as I am clearly doing something I shouldn't be doing?

[memsharded]

It is a bit irritating that conan source followed by conan build behaves different than conan create

Yes, we know it. But we learnt also from experience that it is way more irritating losing code changes because they were done in the copied sources and not those in version control.

I think with an adequate layout, this can be made to work in both scenarios, I think I would try to reference the source code from the CMakeLists.txt in the parent folder of the code, instead of being it inside it. Do you have such recipe in a public repo I could have a look? Thanks!

[bschindler]

conanfile.py.txt
And on the same level, there is a CMakeLists.txt as mentioned in the conanfile.py
CMakeLists.txt

[memsharded]

I am having a look. I see you are building a multi-config package. Is it a requirement for your project? I suggest going with the single-config package if possible. Please tell me, I could propose some changes.

[memsharded]

I see a few points for improvements in the recipe. Is this recipe going to be public/OSS? If so, please consider put it in a github repo, and share with me, I am willing to collaborate.

[bschindler]

Even though the project using the recipe is using cmake, most of the people of the team develop on windows under Visual Studio. Without the multi-config support, the team would have rejected conan as a whole, so yes, its a requirement.

Since all our recipes are multi-config, we have not yet really made them public so far. Yesterday however, I got in touch with the bincrafters community. If multi-config can be made part of their packages/package template even if just through an option, we'd be very willing to upstream all our changes as it would also ease our life of upgrading packages. Currently, we are forking a lot of the package recipes just for multi-config which is a sad state to be in. Since the multi-config requirement is not going to go away (there is no suitable alternative currently) we have to find a way to make multi-config part of the "official" packages.

[memsharded]

Can you please ellaborate a bit further about the reasons why the team would have rejected conan? If you want we can talk in private (we are also in the CppLang slack team, #conan channel, or just write us at [email protected]). Because if it is for the experience while using the IDE, the cmake_multi and visual_studio_multi generators achieve the same experience, without needing the packages to be multi-config.

Since the multi-config requirement is not going to go away (there is no suitable alternative currently) we have to find a way to make multi-config part of the "official" packages.

I guess the best alternative would be IDE plugin for conan. Not there yet, but in our roadmap. With it you wouldn't need to fork all packages, and single-config packages will be automatically switched from the IDE.

Regarding inclusion in conan-center, I would say that the default for conan-center should be single-config packages, though maybe we might need to think about it.

[memsharded]

Also I am investigating a side bug that aroused while checking your layout. Might (or not) be the cause of it. I will try to come up with a solution for the layout, independently of the multi-config.

[bschindler]

cmake_multi gets kinda close, but doesn't match the bill. I've already explained the reasons in a very lengthy answer here: https://github.com/conan-io/conan/issues/917#issuecomment-276008768

Long story short, conan_multi breaks usage of FindXXX scripts as a single FindXXX script is unable to locate libraries from two different conan packages. And as soon you require a package where the FindXXX modules is what you have to use, cmake_multi is not an option anymore.

Our requirements/evaluation table are simple:

Requirement|single-build package/cmake|single-build package/cmake_multi|multi-build package/cmake
------------ | -------------|--------------|--------------
No additional interaction required when switching build modes. ||✅|✅
Switching build mode does not trigger cmake to run/change include flags ||✅|✅
FindXXX scripts can be used for selected packages (we prefer to not use them, but its not possible at times) | | |✅

You can quickly see that just relying on single build packages is not as straight forward.

[memsharded]

Ok, understood, it is an unfortunate accumulation of requirements and limitations, with a void intersection for single-config packages.
Seems the best way to go would be the IDE plugin, to sync conan when switching build-modes.
It is a pity that it isn't there yet, hopefully we will see in a few months.

I have submitted a potential fix for the side bug (I think indeed it can bite you to run the local flow), I will check about the layout tomorrow. Thanks for your detailed feedback!

[bschindler]

Care to explain in which way a IDE plugin would help? How does it avoid the rebuild problem if in the end conan is used together in conjunction with an IDE?

[memsharded]

Sure. I think it could be somehow similar to what is done with CLion here: https://blog.jetbrains.com/clion/2017/07/webinar-recording-developing-cc-projects-with-clion-ide-and-conan-cc-package-manager/

The IDE manages the 2 configurations, debug/release and builds for each of them, in different folders, caching the temporaries, so switching in the IDE from debug<->release is instantaneous.

Conan can install the debug dependency graph in one place (lets say, generating a conanbuildinfo.cmake in a "debug" folder), and a release dependency graph in a different "release" folder. They would be single-config packages. This is summarized in this section: http://docs.conan.io/en/latest/using_packages/workflows.html#multi-configuration

When using cmake, CLion manages to create two different build folders, the "cmake-build-debug", and the "cmake-build-release", so conan can install in each one the matching dependency graph, and switching is seamless and no need to rebuild anything.

I think that with some plugin in VS, something similar could be done. Are you using VS 2017 or older version? Maybe it is worth investigating further into the new VS 2017 cmake features.

[solvingj]

@bschindler : What @memsharded points out is what I was getting at also. If you share with us more detail about how your users do file-open and how the project is arranged on disk, I think it's very likely that we can show you how to make it seamless with the current single-package. It's designed specifically to be eliminate frequent recompilation you're describing with single-packages.

[bschindler]

@memsharded / @solvingj : I see. So it seems CLion works like qtcreator which means it creates N build folders where cmake is triggered for all N times as well. Then single packages work just fine of course. But on visual studio, you execute cmake once and you edit the solution using this one single build folder. For working with a second build folder, you have to open a second instance of Visual Studio which breaks the standard Visual Studio workflow (that's what you are suggesting in http://docs.conan.io/en/latest/using_packages/workflows.html#multi-configuration correct?)

[memsharded]

Actually, VS 2017 with the CMake support, if you open a folder with a CMakeLists.txt file in it, it will use different folders for each configuration. I just tried:

C:/Users/memsharded/CMakeBuilds/85d69b89-0a41-713a-b225-ec320101051f/build/x86-Debug
C:/Users/memsharded/CMakeBuilds/85d69b89-0a41-713a-b225-ec320101051f/build/x86-Release
... also for 64 bits

So doing a conan install in each of those folders is enough to have automatic switch in the IDE, with single-config packages and no rebuilds.
Right now, it can be done with some tricks at the beginning, but once you do it, you are fine for developing, forgetting about conan. And I am pretty positive that this can be implemented in a plugin/addon.

[bschindler]

Okay, I was not aware of this. We are still using Visual Studio 2015 so this is not available to us (so that's why... :-) ). We will be updating to 2017 rather soon, so we can give this a go. We have conan triggered from cmake automatically (for which you also have an example somewhere) which would take care of triggering it on a per-build basis as well. If this integration proofs to work reliably enough, there is naturally no reason to stick to multi-packages.

[memsharded]

Ok, lets work in that direction then :)

So, I have been trying the layout, and once the subtle bugs that appeared are fixed in the PR #2331, I am able to do:

$ conan source .
$ mkdir build && cd build
$ conan install ..
$ conan build ..

We are most likely going to release a patch 1.0.3 version soon including this bug fix, so lets revisit this layout once it is released.

I have added the issue: https://github.com/conan-io/cmake-conan/issues/64, to the cmake-conan wrapper, lets see if there is something that could be done there, and if not possible, then we would need to plugin.
Thanks for all your feedback!

[weatherhead99]

I think I have a similar issue in one of my packages, which has an unfortunate interaction with the deprecation of conanfile_directory:
I have a source tree which I get from downloading a release via FTP in the source() method. However, it requires patches (from a different place) to build on windows using MSVC. Note I cannot simply check out entirely different sources for the windows build, because the windows port is not in sync with the other builds, and hence it would have to be a different package version. However, the patchset does allow it to be build correctly same version.

My first approach, was to add the .patch file to exports_sources, however this doesn't allow me to build the package using "conan source" command, since at that point the patch file isn't included. I then thought to add this to the source() method:
if self.develop: self.copy(self.conanfile_directory+"/patchfile.patch", self.source_folder)

except now I can't do that because it's deprecated. What is the workflow for patch files etc, this must be something that others will need.

THanks :-)

[solvingj]

Try adding the file to exports instead of exports_sources

[memsharded]

There is something weird there. If you check the code, everything from the "export" folder and the "exports_sources" folder is copied to the "source" folder before calling the source() method. So when you call that method, the patch file should be accessible in self.source_folder directory. Could you please double check it @weatherhead99 ?

[weatherhead99]

@memsharded I am looking through the code, and can't see where this is supposed to happen.
Indeed, I have written a simple file, with just patch file in exports_sources, but also a source() method that fetches the upstream original source code.

Calling "conan source . -sf src" puts the original source code in src/ but doesn't put the patch file in there

the api function source(), eventually calls config_source_local() which doesn't do anything to the exports_sources...

[memsharded]

Oh, sorry, my total mistake, I was talking about the cache workflow (conan create), not about the local flow which this issue was referring to. I am going to have another look.

[memsharded]

@weatherhead99 I am adding this test and it seems to work both in create and local-flow:

from conans import ConanFile
from conans.tools import load
import os
class Pkg(ConanFile):
    exports_sources = "*"
    def source(self):
         patch = os.path.join(self.source_folder, "mypatch")
         self.output.info("PATCH: %s" % load(patch))

I think no need to check self.develop, the source code should be always patched when building from source, right?
Please tell me if the above works, or I am missing something else. Thanks!

[weatherhead99]

This doesn't work for me, maybe something I'm doing wrong. conanfile.py has the following contents:

from conans import ConanFile, CMake, tools
import os

class TestexportConan(ConanFile):
    exports_sources = ["patch.patch"]

    def source(self):
        patch = os.path.join(self.source_folder, "patch.patch")
        self.output.info("PATCH: %s" % tools.load(patch))

(patch.patch exists next to conanfile.py)
I am running:
conan source . -sf src

and get this exception:

ERROR: PROJECT: Error in source() method, line 9
        self.output.info("PATCH: %s" % tools.load(patch))
        FileNotFoundError: [Errno 2] No such file or directory: '/home/danw/Software/conan_test_export_sources/src/patch.patch'

conan version 1.0.0, but can try later with 1.0.2 if you think it likely to fix it

[memsharded]

@bschindler If you want to follow current efforts regarding VS plugin: https://github.com/conan-io/conan/issues/2353

[weatherhead99]

@memsharded I just tried this on latest 1.1.0-dev and both my example file and the one you posted earlier in the thread fail in local packaging.

Tracing through the calls, "conan source" leads to source() in command.py, which calls ConanAPIV1.source(), which calls ConanManager.source(), which calls conans.client.source.config_source_local, which doesn't do anything related to exports_sources...

[memsharded]

The above works with the command:

$ conan source .

Without specifying the -sf src

I am not sure, maybe it could be a misunderstanding of the layout and relative paths. Take the following:

from conans import ConanFile, CMake, tools
import os

class TestexportConan(ConanFile):
    exports_sources = ["src/*", "patch.patch"]

    def source(self):
        patch = os.path.join(self.source_folder, "patch.patch")
        self.output.info("PATCH: %s" % tools.load(patch))
        header = os.path.join(self.source_folder, "src/hello.h")
        self.output.info("HEADER: %s" % tools.load(header))

For the layout:

- conanfile.py
- patch.patch
- src/hello.h

Works well both with:

$ conan source .
$ conan create . Patch/0.1@user/testing

Please tell me if this helps.

[weatherhead99]

James,
yes that works, but in that case the patch file doesn't need to be copied anywhere.. I was thinking the intent of having the -sf option, is so that I can check out a clean set of sources somewhere else than where I'm writing the package file. If that isn't the intent, I think it at least needs to be clearer in the docs.

[memsharded]

Yes, you are right.

The following fetches external code, but only allows to do it following the given layout:

class TestexportConan(ConanFile):
    exports_sources = ["patch.patch"]

    def source(self):
        self.run("git clone https://github.com/memsharded/hello.git")
        patch = os.path.join(self.source_folder, "patch.patch")
        self.output.info("PATCH: %s" % tools.load(patch))
        header = os.path.join(self.source_folder, "hello/hello.h")
        self.output.info("HEADER: %s" % tools.load(header))

A solution could be to define a self.export_folder or self.recipe_folder, so it could be used in the local flow to differentiate. However, I am not sure which is simpler or most intuitive for users, because I find it quite conveniento to do just the $ conan source . and not having to deal with another extra self.xxx_folder. I will bring the issue to the team.

[weatherhead99]

ok, I think we do agree on how this is working etc.
Where I don't like it is that it is the case that the local package flow and create flow are different, and even more subtle, the difference is only noticed in local package flow only is different if we do "out of recipe" build, i.e. with -sf flag (which is much preferred to me personally, I like to not have whatever upstream source files cluttering up the recipe directory).

IMO this either needs to be consistent or it needs to be very heavily pointed out in the docs. I think it's not too hard of an edge case to stumble into.

[lasote]

Hi, I faced a similar issue with a recipe that has source() method but is "exports_sourcing" a root CMakeLists.txt.
I agree with @memsharded that you can work with the local methods in the directory where the checkout code lives, but I agree with @weatherhead99 that it can be dirty.
My opinion is, as always, can support both without much noise in Conan?

I prepared a proposition here to debate: https://github.com/conan-io/conan/compare/develop...lasote:feature/exports_optional_sources?expand=1

Am I missing some corner case? @memsharded ? The implementation is pretty straight forward.

[gklingler]

Hi! I'm having the same problem when applying a patch in the source() method - I agree that it's quite confusing that the local flow does not generically work for such a package... I ended up here.

[memsharded]

Hi @gklingler

Yes, we are aware of the issue, as explained above, it was a trade-off decission regarding not copying sources out of source control, which was the most common headache for a majority of users. We are having a look and will try to come up with some improvements for 1.2. Thanks for the feedback.

[memsharded]

We have added in https://github.com/conan-io/conan/pull/2624 the execution of exports for the conan source command as default, not necessary to opt-in with some extra argument. I think it doesn't break and is indeed reasonable. Testing and feedback very welcome.

This PR https://github.com/conan-io/conan/pull/2624 is only about the exports, it doesn't do a full source copy to the build folder, which I think it is a different feature (even not a feature, as a normal shell copy will do exactly this, and it is not something that can be done by the conan source command). In any case, if someone still thinks that copying the source code locally from the source to the build folder should be a feature, I suggest opening a dedicated issue to discuss it.

Many thanks!

[liberforce]

I've been bitten by this one too. I have patches to apply and using conan source . --source-folder=tmp/source doesn't copy them to tmp/source. So when I try to apply them in the source() method, tools.patch() fails. So when you say:

We have added in #2624 the execution of exports for the conan source command as default, not necessary to opt-in with some extra argument.

Do you mean exports or exports + exports_sources? If I want to export my patches to apply them in the source() method, where should I export them? The documentation says exports_sources is where it should be, so with your last changes, will it work?

[memsharded]

Yes, yes it should work, both exports (exports + exports_sources) are done.
It would be great if you could run from sources and give the branch a spin to try it before it is released. Thanks!

[liberforce]

Tested right now the develop branch. With exports_sources, conan source . -sf=tmp/source works as expected, as well as the whole flow up to export-pkg.

Good job guys :+1:

[memsharded]

Excellent!! Many thanks for testing :)

[memsharded]

This can be closed, it is already labeled "ready for release", it will be in next 1.2