Conan: wrong md5 result for export

[blackliner]

On my system I do get a different RREF when running conan export compared to what our CI does.

CI:

$ conan export conan/recipes/zeromq/all/ 4.3.3-win@
Exporting package recipe
zeromq/4.3.3-win exports: File 'conandata.yml' found. Exporting it...
zeromq/4.3.3-win exports: Copied 1 '.yml' file: conandata.yml
zeromq/4.3.3-win exports_sources: Copied 1 '.txt' file: CMakeLists.txt
zeromq/4.3.3-win: The stored package has not changed
zeromq/4.3.3-win: Using the exported files summary hash as the recipe revision: e025e41363dcd4b490418d06224d5533
zeromq/4.3.3-win: Exported revision: e025e41363dcd4b490418d06224d5533

My machine:

$ conan export conan/recipes/zeromq/all/ 4.3.3-win@
Exporting package recipe
zeromq/4.3.3-win exports: File 'conandata.yml' found. Exporting it...
zeromq/4.3.3-win exports: Copied 1 '.yml' file: conandata.yml
zeromq/4.3.3-win exports_sources: Copied 1 '.txt' file: CMakeLists.txt
zeromq/4.3.3-win: The stored package has not changed
zeromq/4.3.3-win: Using the exported files summary hash as the recipe revision: 72778c1dd07e4d8976aa4cd2aee084e3
zeromq/4.3.3-win: Exported revision: 72778c1dd07e4d8976aa4cd2aee084e3

On CI, the conanmanifest.txt looks like this:

1613651222
conandata.yml: 0d768f680a5e682de512422da2c25303
conanfile.py: 7b6de2daa28836def00c57e07515612d
export_source/CMakeLists.txt: a7384d1eb57761e50fe52fd0d6bb97a8

on my machine:

1613641780
conandata.yml: 7754e2e98f18114fb783b276051d70fe
conanfile.py: 7b6de2daa28836def00c57e07515612d
export_source/CMakeLists.txt: a7384d1eb57761e50fe52fd0d6bb97a8

So it looks like the md5 for conandata.yml is calculated wrong on my machine. Doing it manually gives the CI version of the md5: md5sum conan/recipes/zeromq/all/conandata.yml

0d768f680a5e682de512422da2c25303  conan/recipes/zeromq/all/conandata.yml

Environment Details (include every applicable attribute)

• Operating System+version: WSL2 Ubuntu18.04
• Compiler+version: gcc 7.4
• Conan version: 1.33.0
• Python version: 3.6.9

maybe related stuff?

different algorithm may be used:
https://github.com/conan-io/conan/blob/e04135f92dcc9d339a728324b8a588c8ae167bb4/conans/util/files.py#L144
https://github.com/conan-io/conan/blob/e04135f92dcc9d339a728324b8a588c8ae167bb4/conans/util/files.py#L162

[KerstinKeller]

Both are on the same OS? Something in my head just screams EOL convention. We set all conanfile.py / conandata.yml to Linux EOL conventions, even on Windows, because they caused the creation of different packages.
But might be something different in your case.

[jgsogo]

We use this file in conan-center-index to ensure the same line-endings: https://github.com/conan-io/conan-center-index/blob/master/.gitattributes... but it looks like your issue is different because you get different results using python and using command-line program.

This is totally unexpected. I'm running some search on the Internet and there are people writing about problems with WSL2 and Virtual Machine Platform, they turn them off to fix it... you can try it, but... https://askubuntu.com/a/1252679

[blackliner]

Both are on the same OS? Something in my head just screams EOL convention. We set all conanfile.py / conandata.yml to Linux EOL conventions, even on Windows, because they caused the creation of different packages.
But might be something different in your case.

yes, its the exact same file, as you can see from the manual md5 calculation. Its only conan that comes up with a different value

@jgsogo yea, disabling the VMP will also kill WSL2 :-/

I cannot reproduce the issue with this python script 🤷

#!/usr/bin/env python3
import argparse
import hashlib
from pathlib import Path


def generic_algorithm_sum(file_path, algorithm_name):

    with open(file_path, "rb") as fh:
        try:
            m = hashlib.new(algorithm_name)
        except ValueError:  # FIPS error https://github.com/conan-io/conan/issues/7800
            m = hashlib.new(algorithm_name, usedforsecurity=False)
        while True:
            data = fh.read(8192)
            if not data:
                break
            m.update(data)
        return m.hexdigest()


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Build and install LumPDK")
    parser.add_argument("FILENAME", type=Path)
    args = parser.parse_args()

    print(generic_algorithm_sum(args.FILENAME.resolve(), "md5"))

[blackliner]

Update:
I was able to get one step further: The file in my repo has the correct hash, but the one that gets copied to the local conan export directory has the wrong one:

fberchtold@W10-RIG:~/luminar/LumPDK$ md5sum /home/fberchtold/.conan/data/zeromq/4.3.3-win/_/_/export/conandata.yml
7754e2e98f18114fb783b276051d70fe  /home/fberchtold/.conan/data/zeromq/4.3.3-win/_/_/export/conandata.yml
fberchtold@W10-RIG:~/luminar/LumPDK$ md5sum conan/recipes/zeromq/all/conandata.yml
0d768f680a5e682de512422da2c25303  conan/recipes/zeromq/all/conandata.yml

So it seems conan is cutting away entries in my conandata.yml ???

fberchtold@W10-RIG:~/luminar/LumPDK$ cat /home/fberchtold/.conan/data/zeromq/4.3.3-win/_/_/export/conandata.yml
sources:
  4.3.3-win:
    sha256: 617c52df00be35b925264079d867860fac8b0490b7ecb9dad2e7f57af138e1df
    url: https://github.com/zeromq/libzmq/archive/8d34332ff2301607df0fc9971a2fbe903c0feb7c.tar.gz
fberchtold@W10-RIG:~/luminar/LumPDK$ cat conan/recipes/zeromq/all/conandata.yml
sources:
  4.3.3-win:
    url: https://github.com/zeromq/libzmq/archive/8d34332ff2301607df0fc9971a2fbe903c0feb7c.tar.gz
    sha256: 617c52df00be35b925264079d867860fac8b0490b7ecb9dad2e7f57af138e1df
  4.3.3-qnx:
    url: https://github.com/luminartech/libzmq/archive/v4.3.3-qnx.tar.gz
    sha256: 4ba9427335a0b8e8d6ef268f69ff09db3772e6247bf0876b6e1321f6a038f01d

[jgsogo]

Yes, conan-center-index (not Conan) uses some hooks, one of them removes from conandata.yml everything not related to the version exported.

[blackliner]

oh my god, thanks, so I need to ger rid of them...

[jgsogo]

You can install the same hooks we are using https://github.com/conan-io/conan-center-index/blob/253121597d92bdaf844a31c53681a09e396fe89e/docs/how_to_add_packages.md#test-the-recipe-locally

[blackliner]

I used them, maybe they were outdated 🤷

[memsharded]

Hi @blackliner

So it seems this was not a Conan issue, but the ConanCenter hook that was trimming the conandata.yml, correct?
Could we close this issue? Thanks!

[blackliner]

yes