Conda-build: Updating from 3.16.2 to 3.16.3 refuses to build

[stefdoerr]

It seems like something broke with the latest version causing all our travis builds to fail. I downgraded conda-build and it builds fine. I tried googling the error but could not find any information and it's quite obscurely formulated.
Maybe there is something wrong in our build script? https://github.com/Acellera/htmd/tree/master/package/htmd

Actual Behavior

(travis-env) [xxx@xxx Thu16:55 htmd] export OSNAME=LINUX; conda build --python 3.6 package/htmd --no-include-recipe
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from config.variant
INFO:conda_build.variants:Adding in variants from config.variant
/home/sdoerr/miniconda3/lib/python3.7/site-packages/conda_build/environ.py:419: UserWarning: The environment variable 'OSNAME' is being passed through with value LINUX.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
Couldn't extract raw recipe text for htmd output

Expected Behavior

It should build :)

Steps to Reproduce

Roughly this, although you would need to set up an env with the deps, so if you want the actual reproduction steps tell me and I can post them here.

git clone https://github.com/Acellera/htmd
cd htmd
conda build --python 3.6 package/htmd --no-include-recipe

Output of conda info

     active environment : travis-env
    active env location : /home/sdoerr/miniconda3/envs/travis-env
            shell level : 1
       user config file : /home/sdoerr/.condarc
 populated config files : /home/sdoerr/.condarc
          conda version : 4.5.11
    conda-build version : 3.16.2
         python version : 3.7.1.final.0
       base environment : /home/sdoerr/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/sdoerr/miniconda3/pkgs
                          /home/sdoerr/.conda/pkgs
       envs directories : /home/sdoerr/miniconda3/envs
                          /home/sdoerr/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.11 requests/2.20.1 CPython/3.7.1 Linux/3.10.0-862.3.2.el7.x86_64 centos/7 glibc/2.17
                UID:GID : 3041:3000
             netrc file : None
           offline mode : False

Thanks!

[j3mdamas]

Really minor, but one thing that I notice is that with newer versions of conda-build, logs are duplicated (as can see above)

[nehaljwani]

The change creating a problem for this recipe is part of commit: bae55a91edcf9eb1124c8ca0003a73434e2327f3 . It adds a hard requirement that self.extract_package_and_build_text() must not be a NoneType.

The regex in question here is (^.*?)(?=^requirements:|^test:|^extra:|^about:|^outputs:|\Z)

How it fails: https://regex101.com/r/JQchEt/1

I can suggest the following workaround: Move the about section to the bottom of meta.yaml and then the regex will match: https://regex101.com/r/JQchEt/2

diff --git a/package/htmd/meta.yaml b/package/htmd/meta.yaml
index eb30a2fc5..43423e777 100644
--- a/package/htmd/meta.yaml
+++ b/package/htmd/meta.yaml
@@ -1,8 +1,3 @@
-about:
-  home: https://www.acellera.com
-  license: "HTMD Software License version 1.1"
-  license_file: HTMD_LICENCE.txt
-
 package:
   name: htmd
   version: {{ environ.get('BUILD_VERSION', 0 ) }}
@@ -34,3 +29,7 @@ requirements:
     - python 
     - htmd-deps =={{ environ.get('BUILD_VERSION') }}

+about:
+  home: https://www.acellera.com
+  license: "HTMD Software License version 1.1"
+  license_file: HTMD_LICENCE.txt

[j3mdamas]

Thanks a lot for the input! Is it known that the meta.yaml contents had to
have particular order? I don't think I saw it in the documentation.

We can make that change.
João

On Sat, Nov 24, 2018, 08:58 Nehal J Wani <[email protected] wrote:

The change creating a problem for this recipe is part of commit: bae55a9
https://github.com/conda/conda-build/commit/bae55a91edcf9eb1124c8ca0003a73434e2327f3
. It adds a hard requirement that self.extract_package_and_build_text()
must not be a NoneType.

The regex in question here is
(^.*?)(?=^requirements:|^test:|^extra:|^about:|^outputs:|\Z)

How it fails: https://regex101.com/r/JQchEt/1

I can suggest the following workaround: Move the about section to the
bottom of meta.yaml and then the regex will match:
https://regex101.com/r/JQchEt/2

diff --git a/package/htmd/meta.yaml b/package/htmd/meta.yaml
index eb30a2fc5..43423e777 100644--- a/package/htmd/meta.yaml+++ b/package/htmd/meta.yaml@@ -1,8 +1,3 @@-about:- home: https://www.acellera.com- license: "HTMD Software License version 1.1"- license_file: HTMD_LICENCE.txt-
package:
name: htmd
version: {{ environ.get('BUILD_VERSION', 0 ) }}@@ -34,3 +29,7 @@ requirements:
- python
- htmd-deps =={{ environ.get('BUILD_VERSION') }}
+about:+ home: https://www.acellera.com+ license: "HTMD Software License version 1.1"+ license_file: HTMD_LICENCE.txt

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda/conda-build/issues/3267#issuecomment-441350721,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKTzQpEl1-CFi5E4DeUVr8pjNgUrrZ8Lks5uyPw2gaJpZM4Yvncd
.

[nehaljwani]

I was under the same impression as you ... till today.

I am not saying what I proposed is the actual fix. I just wanted you to not pin the version of CB on your CI runs and unblock you :-)

[j3mdamas]

We really appreciate it, but I was just wondering if this was a bug (the
yaml file has to have those elements but not in that particular order) or
the expected behaviour.

On Sat, Nov 24, 2018, 14:16 Nehal J Wani <[email protected] wrote:

I was under the same impression as you ... till today.

I am not saying what I proposed is the actual fix. I just wanted you to
not pin the version of CB on your CI runs and unblock you :-)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda/conda-build/issues/3267#issuecomment-441367296,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKTzQs0FrB4SeJZRajLUgeMbsxIUsbW_ks5uyUa_gaJpZM4Yvncd
.

[msarahan]

While strict order hasn't been an explicit requirement of conda recipes ever, conda-forge does enforce an order with their linter, and I think it's reasonable to ask for an order, for features like the one here. Without the order, I'm not sure it would be possible to do what the recipe is doing here. Managing the recipe in terms of both what is parseable and what is not directly parseable is very difficult.

Sorry, I think this falls into "won't fix," and we need to document that the ordering of recipe sections is not completely arbitrary.

[j3mdamas]

Yes, documentation would be sufficient. Thanks for the prompt replies. I'm testing @nehaljwani's suggestion and I'll message here how it goes.

[j3mdamas]

Thanks for the help. For us (@stefdoerr and I), this can be closed.

[bergtholdt]

I had the same error but for a different reason. To keep with indentation levels (I use 4 spaces) my recipes actually contain extra white space after the hyphen in the outputs section, e.g.:

outputs:
- name: mymodulek
script: install.sh
...

Which is perfectly legal yaml syntax. However in metadata.py the output_re regexp looks for single space only. This can be fixed by introducing a + after the second \s in the regexp output_re:

output_re = re.compile(r"^\s+-\s+(?:name|type):.+?(?=^\w|\Z|^\s+-\s+(?:name|type))",
                       flags=re.M | re.S)

I'll a pull request for this.

[bergtholdt]

Unfortunately there are more occurrences where the regexp captures only single spaces (just search "\sname" within conda_build). I changed them all to "\s+name", however the regexp in metadata.py line 1641 will then not compile, since the lookbehind pattern has no longer fixed length:

f = r'(^requirements:.*?|(?<=-\s+name:\s%s\s).*?requirements:.*?)(?=^\s*-\s+name|^\s*test:|^\s*script:|^\s*extra:|^\s*about:|^outputs:|\Z)' % self.name()  # NOQA

I'll try to fix that, too, but this looks more involved.

[msarahan]

Yep, that's more complicated... =(

Sorry, this may just be a crummy quirk that we have to live with. It's probably not worth the time to reinvent how to figure out what a recipe's unparsed content is, just so that we can have more flexibility in recipe ordering and whitespace.

We definitely need better error handling and linting, though. This order and whitespace stuff should be easy to warn about and possibly sanitize prior to feeding it into the rendering stuff.

[bergtholdt]

ok, was simple enough. Just had to use a simpler lookbehind pattern: (?<=\sname:\s%s\s)