Or-tools: WHEEL file size mismatches for ortools-7.0.6546-cp36-cp36m-manylinux1_x86_64.whl

[georgi-yanchev]

(.venv) root@5e15c7a622d4:~/project/aaa/ortools-7.0.6546.dist-info# ls -la WHEEL 
-rw-r--r-- 1 root root 109 Mar 25 14:55 WHEEL
(.venv) root@5e15c7a622d4:~/project/aaa/ortools-7.0.6546.dist-info# cat RECORD | fgrep WHEEL
ortools-7.0.6546.dist-info/WHEEL,sha256=opqqECzkr2EiaHB1f1oMkivXbaoa7jbdA_NbjmFJL1Y,112

Note that file WHEEL is of size 109, but in RECORD it's stated to be 112 bytes.

[Mizux]

Is it an issue for you ? As i said in the release note we provide a "manylinux1" package while in reality it is a "manylinux10" so we modify the package meta using sed which would explain the size diff.

ref: https://github.com/google/or-tools/blob/39f44709bba203f5ff3bc18fab8098739f189a6d/tools/docker/build-manylinux1.sh#L186-L195

[Mizux]

note: Most distro have a glibc >= 2.12 so it's OK IMHO...

[georgi-yanchev]

I'm not sure I understand your comment about glibc.

My project has a bunch of dependencies and or-tools is one of them. I'm packaging a PEX ( https://github.com/pantsbuild/pex ) executable and this file is the only one rejected by PEX build. That's how to reproduce the error:

root@91e8620481ff:~# pip install pex==1.6.6  
root@91e8620481ff:~# pip wheel ortools==7.0.6546
root@91e8620481ff:~# pex -o ort.pex ortools-7.0.6546-cp36-cp36m-manylinux1_x86_64.whl 
Traceback (most recent call last):
  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/pex/bin/pex.py", line 675, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/usr/local/lib/python3.6/dist-packages/pex/bin/pex.py", line 608, in build_pex
    pex_builder.add_distribution(resolved_dist.distribution)
  File "/usr/local/lib/python3.6/dist-packages/pex/pex_builder.py", line 336, in add_distribution
    dist_hash = self._add_dist_zip(dist.location, dist_name)
  File "/usr/local/lib/python3.6/dist-packages/pex/pex_builder.py", line 301, in _add_dist_zip
    wf.install(overrides=self._get_installer_paths(whltmp), force=True)
  File "/usr/local/lib/python3.6/dist-packages/pex/vendor/_vendored/wheel/wheel/install.py", line 289, in install
    if self.parsed_wheel_info['Root-Is-Purelib'] == 'true':
  File "/usr/local/lib/python3.6/dist-packages/pex/vendor/_vendored/wheel/wheel/install.py", line 67, in __get__
    val = self.wrapped(inst)
  File "/usr/local/lib/python3.6/dist-packages/pex/vendor/_vendored/wheel/wheel/install.py", line 259, in parsed_wheel_info
    return read_pkg_info_bytes(self.zipfile.read(self.wheelinfo_name))
  File "/usr/lib/python3.6/zipfile.py", line 1338, in read
    return fp.read()
  File "/usr/lib/python3.6/zipfile.py", line 858, in read
    buf += self._read1(self.MAX_N)
  File "/usr/lib/python3.6/zipfile.py", line 962, in _read1
    self._update_crc(data)
  File "/usr/local/lib/python3.6/dist-packages/pex/vendor/_vendored/wheel/wheel/install.py", line 493, in _update_crc
    raise BadWheelFile("Bad hash for file %r" % ef.name)
pex.vendor._vendored.wheel.wheel.install.BadWheelFile: Bad hash for file 'ortools-7.0.6546.dist-info/WHEEL'

[Mizux]

manylinux1 defined by PEP513 require GLIBC <= 2.5 while abseil-cpp (OR-Tools dependency) require GLIBC >= 2.12 -> we can't technically provide a manylinux1 package.

So we used a manylinux2010 (PEP 571) proposal docker container to build a manylinux2010 package.
Unfortunately, while we can generate a manylinux2010 more or less, only few distro have a pip package up to date to support the manylinux2010 platform tag (as well as pypi.org didn't support it until recently) BUT since most distro have recent glibc (hey we are in 2019) so in practice providing a fake manylinux1 package which as been compiled against a recent glibc (2.12) just work...

Unfortunately pex seems to be a little bit agressive on its check, what you can do since pex seems to be a pure python lib is to hack the /usr/local/lib/python3.6/dist-packages/pex/vendor/_vendored/wheel/wheel/install.py", line 493 so pex don't care anymore of this....

note: official manylinux2010 docker container has just been released last week (yup 9 years late ;) )
see https://github.com/pypa/manylinux/issues/179 to follow the progress more specifically https://github.com/pypa/manylinux/pull/279 provide the manylinux2010 docker image on April 10th 2019...
bonus: manylinux1 container doesn't work on recent distro for years since it uses a deprecated vsyscall see https://github.com/pypa/manylinux/pull/158

[georgi-yanchev]

Got it about libc and manylinux2010. It makes sense.

In my build script I've added an exception for or-tools and it's repacked to make the hashes match, so that's not a big issue for me. I was just wondering if you want to patch RECORD, just like you patched WHEEL and have a package with matching hash sums.

[Mizux]

For sure, i'm not against this idea, just need to see if i can take time to do it.
I mean hack our manylinux.sh script to patch the RECORD and why not try to pex the package file to verify everything is good.

[georgi-yanchev]

I created a PR for this: https://github.com/google/or-tools/pull/1222

[lperron]

Thanks.
I will let Corentin review it and merge it.
Laurent Perron | Operations Research | [email protected] | (33) 1 42 68 53
00

Le ven. 26 avr. 2019 à 19:47, Georgi Yanchev [e-mail] a
écrit :

I created a PR for this: #1222
https://github.com/google/or-tools/pull/1222

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/or-tools/issues/1218#issuecomment-487143040,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUPL3LFQYY6JGHZFSPQL4TPSM523ANCNFSM4HIO5OPQ
.