Conda-build: how to tell conda-build not to copy the whole working directory

[stas00]

Actual Behavior

When running conda-build it first copies the whole working directory (!)

Expected Behavior

I'm not sure why you need to copy the whole thing, but if you insist, how can I tell it to skip some directories? I'm unable to find any documentation on that.

The reason I need this is that some sub-directories contain thousands of large files that have nothing to do with packaging. Plus the .git folder is pretty big! It takes really long time to copy, making the build very very slow, not talking about creating an unnecessary load for the disk.

If there is no support for this feature, perhaps the interface can be copied from pytest configuration?

[tool:pytest]
collect_ignore = ['setup.py']
norecursedirs = .* build builds conda conda-dist data dist docs examples

Thanks.

Steps to Reproduce

conda-build ./conda/ -c pytorch -c fastai/label/main --output-folder conda-dist --no-remove-work-dir
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Copying /mnt/disc1/fast.ai-1/br/fastai/master to /home/stas/anaconda3/envs/pytorch-dev/conda-bld/fastai_1539401711882/work

Output of conda info

     active environment : pytorch-dev
    active env location : /home/stas/anaconda3/envs/pytorch-dev
            shell level : 1
       user config file : /home/stas/.condarc
 populated config files :
          conda version : 4.5.11
    conda-build version : 3.0.27
         python version : 3.6.3.final.0
       base environment : /home/stas/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/stas/anaconda3/pkgs
                          /home/stas/.conda/pkgs
       envs directories : /home/stas/anaconda3/envs
                          /home/stas/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.11 requests/2.18.4 CPython/3.6.3 Linux/4.15.0-36-generic ubuntu/18.04 glibc/2.27
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[stas00]

And as discovered by https://github.com/conda/conda-build/issues/3215 conda info is broken and reports incorrect versions.

$ conda-build --version
conda-build 3.15.1

[stas00]

And there is a related problem where the same build starts to resolve symlinks and copy what they link to (like symlinks to TB-large datasets) and conda-build was running for hours copying them. It has been happening to my colleague and I wasn't able to reproduce it. He is on ubuntu-16, I'm on ubuntu-18, but perhaps you'd have some insight on this.

We both upgraded to conda-build 3.16.1

[msarahan]

Consider a PR. You are seeing documented (https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#source-from-a-local-path), well-established behavior. You may want to use a different source type, such as git_url, with a relative path.

[stas00]

Thank you, @msarahan. I was just hoping that it's be possible to configure to skip some heavy folders.

But yes, git_url would work, thank you for that suggestion. Except the release manager will have to be very careful to commit any changes before attempting to make the package, which is doable, but not as foolproof as working with the current source.

If adding support to skipping dirs is not an option, then this issue can be closed.

[msarahan]

It is an option, but we're not going to implement it for you. I'd take a PR that implements this.

[stas00]

Any pointers for a starting point, perhaps a similar existing option that I could use as an example?

I guess the simplest initial approach would be to just have a command line option --skip-files="dira dirb filea fileb" and support glob. Then, if successful, to add a yaml config option. Is my direction valid?

[msarahan]

It may be worthwhile to put this into meta.yaml instead of as a CLI option. You would add a new key to the source section. Maybe something like "path_ignore_globs."

check these places:
https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L406
https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L489
This is where you need to do the filtering somehow:
https://github.com/conda/conda-build/blob/master/conda_build/source.py#L655

copy_into does not currently have any logic for filtering:
https://github.com/conda/conda-build/blob/master/conda_build/utils.py#L426

Adding that capability is probably a good thing. rec_glob may be helpful here:
https://github.com/conda/conda-build/blob/master/conda_build/utils.py#L731

and maybe filter_files:
https://github.com/conda/conda-build/blob/master/conda_build/utils.py#L1207

[stas00]

Thank you for the guidance, @msarahan

I will also need --keep-old-work to work to be able to work on this, as it will be essential to be able to test quickly whether my efforts are correct. And it isn't working for me with the latest conda-build https://github.com/conda/conda-build/issues/3215#issuecomment-430478036

[stas00]

@msarahan, I have been contemplating alternative approaches - and what came to me when recently fixing up setup.py sdist exclusions and inclusions, is that we already have that mechanism in setup.py, why not tap into it in a similar fashion to how load_setup_py_data is used in the yml template (that is it allows the user to tap into setup.py's data and re-use it rather then maintaining to sets of configurations. If there is MANIFEST.IN, then it's already configured for sdist, we just need to make conda-build use that feature if it's told to. Do you think the needs of sdist and conda-build can diverge and the same set of files will not be sufficient?

[stas00]

and I thought more about the idea in the previous comment and solved it without any change needed to conda-build.

drums please...

conda/meta.yaml:
{% set data = load_setup_py_data(setup_file='../setup.py', from_recipe_dir=True) %}
source:
  url: ../dist/{{ data.name }}-{{ data.version }}.tar.gz

and then:

python setup.py sdist
conda-build ...

It's simple and works exactly as intended, since I can control what to copy and what not from the setup.py side, which has that supported already.

What do you think?

I'd be happy to submit a doc PR to document this, not sure where it should go. Perhaps the doc you mentioned originally where it says that the whole source folder will be copied?
https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#source-from-a-local-path