Conan: [scm] making scm_folder.txt optimisation usable for export + install

[niosHD]

I am currently modernizing my conan recipes and replace my old scm implementation with the upstream version. While playing around with the new recipe, I discovered that the scm_folder.txt optimization did not work for some of my build scripts. In particular, since #4218, the optimization only triggers for conan create and not when using conan export followed by conan install. (By the way, the tip at the end of the documentation should probably be updated to make it clear that only create is currently supported by this optimization.)

Using the separated export and install commands is useful in larger monorepo builds where I can first export all the conan sub packages before building (i.e., via conan install --build=outdated ...) the root package. Conan figures out the correct build order and builds all packages as needed.

While I absolutely agree that #4218 had to be fixed, I am now wondering if it is possible to also support the export+install flow, in addition to the create only flow, in a sane and correct way.

Current Logic

Currently the work flow seems to be that, as soon as either the revision or url is captured by scm, scm_folder.txt is written by both, the export functionality of the export and create commands. However, the information in scm_folder.txt is actually only used during the package installation step of the create command.

Alternative Idea

Instead of writing the current path into the scm_folder.txt we could directly copy the content of the directory into the source folder of the package during the export step (i.e., triggered either as part of the export or create command). Subsequent install commands can directly use this source and do not have to clone the repository again. Similarly, in the workflow that only uses the create command, the source is already present and no additional download is required.

In summary, when using the create command, exactly the same amount of work (i.e., one copy into the source folder) would be performed. The only difference is that we perform this copy in the export instead of the install step of the command.

What do you think, would this be approach be feasible? @jgsogo (since you just recently worked with this code) did I miss important aspects?

[jgsogo]

Hi @niosHD!

First of all, thanks for your comprehension, that was a behavior we should fix ASAP and, as an optimization, now we can lean back and think about how to enable it again safely.

So your idea is to copy the sources in the export command (could take time for big libraries) when the scm_folder.txt file is created and then use them (instead of calling the SCM clone) in the create and in the install --build commands.

Right now I cannot imagine any scenario where this can lead to an error and the logic looks like simple. I will ask for some feedback before adding it to the queue, maybe someone finds a counter-example.

[lasote]

I think it sounds good. Let's confirm with the rest of the team. Would be nice to get rid of the ugly scm_folder.txt

[niosHD]

... (could take time for big libraries) ...

That is true and also the only issue I am currently aware of. However, the only usage pattern that should be affected by this change is when recipes are exported without intent of building them subsequently (e.g., to upload them to a remote). The question is how common such a pattern is. Maybe a command line flag should be added to select if capturing the source in the export step is desired or maybe even a global option to disable/enable the whole optimization?

Would be nice to get rid of the ugly scm_folder.txt

I totally agree.

[jgsogo]

Working on this (not as straightforward as I thought at first), I came up with something more: there is no reason to do this copy-during-export only for SCM with auto, if we are going to do it, we should do it always (except for explicit SCM with URL and commit, where we should do checkout).

So, proposal:

• Add argument flag --export-sources to command export:
  
  
  
  • export --export-sources=False (default): will behave as it does now: it won't copy sources to the cache, moreover, it is deleting them unless keep-sources=True is provided or recipe is exactly the same.
  
  
  • export --export-sources=True: will copy sources from local workspace (or clone the SCM-repo if not auto). It will leave a _sentinel_ in the cache to know that those sources correspond to the exported recipe.
  
  

• Command create or install --build will use existing sources if the _sentinel_ is found, otherwise, we maintain current behavior: command create will copy for SCM-auto and will checkout for install --build.

[Implementation detail] From this new perspective, we would be adding to the export command a new functionality (exporting the sources too) and there is no more _"scm_folder optimization"_, it is not needed anymore.

[lasote]

Merged. To be released at 1.20