Cibuildwheel: Dirty working tree for pp36-pypy36_pp73-win32.whl

[hameerabbasi]

Hello. All other wheels produced by CIBuildWheel have a clean working tree, but the pp36-pypy36_pp73-win32.whl has a dirty working tree. This causes versioneer to mess up the version (since it's based on Git) and therefore causes issues uploading to PyPI.

A temporary fix is to use CIBW_BEFORE_BUILD="git clean -xdf && git reset --hard HEAD"
This doesn't produce any macOS/Windows wheels.

[Czaki]

@hameerabbasi Could you provide link to example configuration, or even log for build? and which version of cibuildwheel?

Something similar was already fixed https://github.com/joerick/cibuildwheel/pull/418.

[hameerabbasi]

Hi, it's the latest version. You can see the specific config here: https://github.com/Quansight-Labs/uarray/tree/5a27e2412a980ba5967c627688bff55db2d2480a

Specifically, azure-pipelines.yml and the ci/ directory.

The CI logs are also public and can be seen here.

[Czaki]

Hmm. It looks rather like problem with python 3.9.
Could you run build with git status in CIBW_BEFORE_BUILD to see which files pollute build directory?
Could you run build without cp39 to check if it produce problem?

[hameerabbasi]

Here's the PR If you want to follow along: Quansight-Labs/uarray#250
Here's the log with git status: https://dev.azure.com/Quansight-Labs/uarray/_build/results?buildId=805&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=168f295b-0553-5364-35f7-923225ecd8b3

[Czaki]

So maybe add wheelhouse to your ,gitignore file will solve this?
I do not understand why in linked run all files has dirt tag and in first one only pypy?

[hameerabbasi]

So maybe add wheelhouse to your ,gitignore file will solve this?
I do not understand why in linked run all files has dirt tag and in first one only pypy?

What do you mean, only PyPy has the dirty tag. The dirty tag is only added if a tracked file is modified. Since wheelhouse/ isn't tracked, it won't add the dirty tag at all.

[hameerabbasi]

Here's a log with wheelhouse/ in .gitignore and CIBW_SKIP=cp39-*. https://dev.azure.com/Quansight-Labs/uarray/_build/results?buildId=806&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3

Still with the dirty tag.

[Czaki]

My bad. check to fast and confuse dirty tag with dev tag.
Could You add CIBW_BEFORE_TEST: "git status" to your build? Maybe this will show us the problematic files.

[Czaki]

@YannickJadoul I create PR with additional information https://github.com/Quansight-Labs/uarray/pull/251 And github status show nothing. You have much more experience wit pypy. For me its looks like some bug in pypy. Could you take a look at this?

[YannickJadoul]

@hameerabbasi & @Czaki, I'm definitely interested to have a look, but I do not have a lot of time, this weekend (and probably next week). You do have a workaround already, so there's not a big rush, or is there?

[Czaki]

I do not see any workaround.

[YannickJadoul]

A temporary fix is to use CIBW_BEFORE_BUILD="git clean -xdf && git reset --hard HEAD"

Oh, I had read this before. But I hadn't seen it was scratched out by now :-(

[Czaki]

@hameerabbasi pleas see this:

    Changes not staged for commit:
      (use "git add <file>..." to update what will be committed)
      (use "git restore <file>..." to discard changes in working directory)
        typechange: .conda/environment.yml
        typechange: binder/environment.yml

Full log here: https://dev.azure.com/Quansight-Labs/uarray/_build/results?buildId=812&view=results

it looks like source of problem is that pypy fir windows change symlink to regular file. As You could check in https://github.com/Quansight-Labs/uarray/pull/251 after change symlinks to regular files everything pass. So this is workaround which you could use until patch shows. I'm not sure if it is fixable in cibuildwheel or you need to report it to pypy project.

Other workaround is to omit windows pypy build.

Review method which I use in https://github.com/Quansight-Labs/uarray/pull/251 to identify problem. Such methods allow to provide better bug report.

[hameerabbasi]

Thanks for taking the time to debug this. I'll file an issue with PyPy.

[Czaki]

@hameerabbasi You may try first newer version of pypy. For example using this version of cibuildwheel: https://github.com/joerick/cibuildwheel/pull/430

[YannickJadoul]

@hameerabbasi Did you end up filing an issue with PyPy? If so, would you mind posting a link here, so we can follow the outcome?

[mattip]

Link to PyPy issue https://foss.heptapod.net/pypy/pypy/-/issues/3325

[YannickJadoul]

Thanks, @mattip! :-)

[mattip]

if anyone can think of a minimal reproducer for this failure, that would be very helpful. It probably is some flag to some function in os or shutil, but since PyPy does not run CI tests with admin privileges, symlinks are not tested.

[YannickJadoul]

I haven't really followed this issue well, but there seems to be a problem during the invocation of pip wheel (demonstrated here by @Czaki).

@hameerabbasi & @Czaki, did you pin down the issue better where exactly this happens?

[henryiii]

By the way, I think I'm seeing the same thing with boost-histogram. I get a dirty directory for 32bit & 64bit CPython 2.7, and all three flavors of 32-bit PyPy, only on Windows. I haven't figured out what the dirty file is, and it's clean after cibuildwheel runs.

[henryiii]

Found the issue by inserting a git status call in my setup.py:

        typechange: docs/notebooks/BoostHistogramHandsOn.ipynb
        typechange: docs/notebooks/PerformanceComparison.ipynb
        typechange: docs/notebooks/SimpleExample.ipynb
        typechange: docs/notebooks/ThreadedFills.ipynb
        typechange: docs/notebooks/aghast.ipynb
        typechange: docs/notebooks/xarray.ipynb

When copying to the build directory, symlinks are not handled correctly (on Windows Python 2 and PyPy), causing git to report a "dirty" directory due to the "type change". Example of one of the files:

lrwxr-xr-x   1 user  grp   43 Jun  3 16:18 BoostHistogramHandsOn.ipynb@ -> ../../notebooks/BoostHistogramHandsOn.ipynb

[mattip]

What command is used to copy to the build directory with a symlink?

[hameerabbasi]

I tried debugging this locally, but I couldn't find where it happens (couldn't reproduce). I'll try some more today.

[YannickJadoul]

That'd be great; thanks, @hameerabbasi! :-)

[henryiii]

Sticking a subprocess.call(["git", "status"], stdout=sys.stdout, stderr=sys.stderr) in setup.py is how I was able to debug the issue with the symlinks, maybe that would help?

And, @mattip, I think (but am not sure) the copy happens in pip or wheel, maybe part of pep517.