Weasyprint: Tests should pass on Windows

[liZe]

I've open a branch dedicated for tests on Windows: travis-windows. Thanks to the amazing work of @Tontyna for #587, only 3 tests don't pass:

• Acid2
• Flex column wrap reverse
• Condensed fonts

The third one is easy to understand: the fonts included in this Windows flavor (Windows Server, version 1803) probably don't include a condensed variant, we should try to update SANS_FONT. The first one can be complicated. The second one is a mystery.

[liZe]

Recent versions of Windows server don't include Arial Narrow according to the official list.

[liZe]

Oh, and we get random crashes :cry:…

[liZe]

Installing DejaVu seems to fix the 3 tests, but the build hangs. I'll try another build.

[Tontyna]

Can we know which script or batch or shell or whatever doesnt stop? choco? powershell? msys? py? The PATH doesn't count as a secret, doesnt it?

[Tontyna]

BTW: I'm used to see semicolons as path delimiters on Windows. Though it seems to work, export "PATH=$PATH:C:\tools\msys64\mingw64\bin" looks strange to me...

[Tontyna]

I recollect that it's extremely important to terminate the initial msys shell by killing it via the red cross close button of the window, cf. the warning in the log of choco install msys2. There's no sign of termination, looks like choco starts the system upgrade right away.

[liZe]

The PATH doesn't count as a _secret_, doesnt it?

It doesn't, but as we have company-wide secret variables, I can try to disable this.

BTW: I'm used to see semicolons as path delimiters on Windows. Though it seems to work, export "PATH=$PATH:C:\tools\msys64\mingw64\bin" looks strange to me...

I can change that.

Can we know which script or batch or shell or whatever doesnt stop? choco? powershell? msys? py?

No idea. What's strange is that builds used to finish.

[Tontyna]

Had a brief glance at the travis builds and got the impression that the "No output has been received in the last 10m0s" happens since build 1183 aka "Try to install msys2 with chocolatey"

[liZe]

It doesn't, but as we have company-wide secret variables, I can try to disable this.

It doesn't work.

Had a brief glance at the travis builds and got the impression that the "No output has been received in the last 10m0s" happens since build 1183 aka "Try to install msys2 with chocolatey"

Yes, that's what I've found too. No idea why.

[Tontyna]

See Updating MSYS2 unattended is broken

[Tontyna]

Looks like `"/NoUpdate"`` doesnt prevent the system upgrade.

BTW: WeasyPrint doesn't require MSYS2 -- it's just a convenient (?) way to retrieve the latest DLLs and accessories. Is there a way to bypass MSYS and just unpack zipped GTK3 packages?

[liZe]

Looks like `"/NoUpdate"`` doesnt prevent the system upgrade.

Yes, and that's strange.

BTW: WeasyPrint doesn't require MSYS2 -- it's just a convenient (?) way to retrieve the latest DLLs and accessories. Is there a way to bypass MSYS and just unpack zipped GTK3 packages?

I'd like to stay as close as possible to one of the documented ways to install WeasyPrint. As the GTK+ installer seems to be an interactive wizard, it's probably more simple to use MSYS2.

I'll try to download MSYS2 instead of installing it with chocolatey, that's what I used to do by the way.

[Tontyna]

OMG, msys really wants this system upgrade...

[liZe]

OMG, msys really wants this system upgrade...

Well, I've done this manually as upgrading is the "right" way to do according to the documentation. Using --noconfirm --ask 20 should work. If it doesn't, I'll remove the update :angel:.

[liZe]

Oh. Segmentation fault + the build hangs.

[Tontyna]

I see...

Maybe staying with choco install msys2 and issuing a taskkill /im C:\tools\msys64\msys2.exe afterwards could do the trick?

[Tontyna]

... or whatever task needs to be killed ...

[Tontyna]

The task to kill is probably C:\tools\msys64\usr\bin\bash.exe

[liZe]

Wow. https://travis-ci.org/Kozea/WeasyPrint/builds/504603069

[Tontyna]

:tada: :+1: :tada: :+1: :tada: :+1:

Will --ask 20 stay with us?

[liZe]

Less than 9 minutes is much better than 15~20 minutes. I think that we can forget the system upgrade.

I now have to clean this and include the change in master.

Will --ask 20 stay with us?

I don't think it's needed, I'll try to remove it.

Thanks a lot for your hard work, I sincerely didn't think that tests would ever pass on Windows. Having now tests passing on CI is unbelievable. I just wanted to clean branches during the weekend and didn't want to remove travis-window before one last attempt…

[Tontyna]

I'd like to stay as close as possible to one of the documented ways to install WeasyPrint.

The (slightly modified) commands from the travis-for-windows yml could become the third way to install WeasyPrint -- at least for people having 7z, the wget part can be done with powershell.

[liZe]

The (slightly modified) commands from the travis-for-windows yml could become the third way to install WeasyPrint -- at least for people having 7z, the wget part can be done with powershell.

You're right. I'm trying to clean the commands.

Using chocolatey to install Python + MSYS2 is not a bad idea, don't you think? There's even a one-liner available to install it from cmd.exe or powershell.

I've allowed features because of frequent segmentation faults. Do you get them too on your system?

Last question: I can't launch tests with py setup.py test because Python doesn't find setuptools (and other modules I think). Do you have an idea why?

[Tontyna]

Using chocolatey to install Python + MSYS2 is not a bad idea

If chocolatey was a standard/widespread tool. Me myself refused to install it, forgot why I didn't like it :grimacing:

Until now I experienced no segfaults, but I'm launching test via setup.py.

The answer to your last question is "Yes", to provide a solution I'll have to setup an environment that's similar to the one on travis -- will do that this evening, must leave for office now...

[Tontyna]

Some things I'm pondering on:

• The msys installation still prompts for "kill the shell"

• What (and where) is py?

• By instaling gtk3 via msys2, we install another Python, including a python3.exe in the C:\msys64\mingw64\bin folder we inject into the PATH. But AFAIK no pip.exe and no python.exe.

• I don't know how choco does its job en detail, but in the installation log it tells us:

  Environment Vars (like PATH) have changed. Close/reopen your shell to see the changes (or in powershell/cmd.exe just type refreshenv).

  Since the pip commands work this might be no problem. And once the Python executable is found the PATH doesnt matter (besides for finding our beloved Cairo etc.)

• On Windows, there is a python.exe in the Python3 root, so env: PYTHON=python is worth a try.

• Nevertheless: Both Pythons, whether msys or choco, updated or not updated, come with setuptools in their site-packages folder. Finding or not finding a module is a question of being in Python's sys.path, special effort is required to exclude the site-packages from the sys.path -- so I'm asking again: What is py?

Won't be able to create a local environment like the one on Travis to reproduce the failures. The only means to find out what Travis is suffering from is putting investigative commands into travis.yml.

[liZe]

• What (and where) is py?

I don't know. The official Python documentation uses py too, and it seems to be normal for everyone.

• By instaling gtk3 via msys2, we install another Python, including a python3.exe in the C:\msys64\mingw64\bin folder we inject into the PATH. But AFAIK no pip.exe and no python.exe.

Good to know.

• Since the pip commands work this might be no problem. And once the Python executable is found the PATH doesnt matter

I agree.

• On Windows, there is a python.exe in the Python3 root, so env: PYTHON=python is worth a try.

I've already tried, but I think that it launches a Python (2?) version with no pip. I should try python3.

• Nevertheless: Both Pythons, whether msys or choco, updated or not updated, come with setuptools in their site-packages folder. Finding or not finding a module is a question of being in Python's sys.path, special effort is required to exclude the site-packages from the sys.path -- so I'm asking again: What is py?

👽 (X-Files music)

Won't be able to create a local environment like the one on Travis to reproduce the failures. The only means to find out what Travis is suffering from is putting investigative commands into travis.yml.

I'll try to answer these questions soon.

[Tontyna]

We definitely need some commands that show the PATH resp. the location of our executables: a WHERE py, a WHERE pip-- extension not required.
When we know where pip is, we'll know where the Python executable is...

And I'd like to run a Python script echoing the sys.path.

[Tontyna]

I'd expected that python3 launches the MSYS Python, but obviously it doesn't...

:sheep: :sheep: :sheep:

[Tontyna]

Ok, back to py.
I'm keen to see the output of the print-the-sys.path-script

[Tontyna]

...drum rolls...

[Tontyna]

OMG! The PATH isnt a Windows path at all! No drive, not a single backslash -- phew, that's what you wanted! What shell is this?

[liZe]

phew, that's what you wanted!

:smile:

What shell is this?

It's Git-for-Windows' bash.

I really don't understand how py -m pip install --upgrade setuptools works (and pip even uses setuptools, so setuptools is available even before installing) and py setup.py complains about setuptools missing.

[liZe]

Isn't C:\Windows\py.exe strange? Shouldn't it be in C:\Python37?

[Tontyna]

It's Git-for-Windows' bash.

Ah! This shell never is and never will be able to do the job! Forgot the datails, but Google will reveal them.

Isn't C:\Windowspy.exe strange?

py.exe in C:\Windows is an abomination! Is that choco!? Is that Python3.7? On my system there is no py.exe at all...

But at least the sys.path looks right, site-packages included. Though, it doesnt help, thats probably cause of the Git-bash.

[Tontyna]

Maybe an env: PYTHON=C:\Python37\python is an option, maybe it works within the git-bash. Can we leave the git-bash-shell? Is the whole Travis job run in that shell? And if yes: Why?
At least I finally understand where the dollars in the log are stemming from :grimacing:

[liZe]

Maybe an env: PYTHON=C:\Python37\python is an option, maybe it works within the git-bash.

Next try.

Can we leave the git-bash-shell?

I don't think so, because…

Is the whole Travis job run in that shell?

Yes.

And if yes: Why?

Ask Travis :wink:.

[Tontyna]

Maybe an env: PYTHON=C:\Python37python is an option

Not really -- we already asked py for its sys.path and the sys.path was the one we believe is the one where choco installed everything, wasnt it?

[liZe]

Python 2.7.9 (default, May 12 2018, 10:00:46)

Sigh…

[liZe]

Not really -- we already asked py for its sys.path and the sys.path was the one we believe is the one where choco installed everything, wasnt it?

You're right. What a mess.

[Tontyna]

Meanwhile look at the PATH:

/bin
/usr/bin
/c/tools/ruby25/bin
/c/Windows/system32
/c/Windows
/c/Windows/System32/Wbem
/c/Windows/System32/WindowsPowerShell/v1.0
/c/Windows/System32/OpenSSH
/c/ProgramData/GooGet
/c/Program Files/Google/Compute Engine/metadata_scripts
/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin
/c/Program Files/Google/Compute Engine/sysprep
/c/Program Files/Docker
/c/ProgramData/chocolatey/bin
/c/Program Files/CMake/bin
/c/Program Files/Git/cmd
/c/Program Files/LLVM/bin
/c/Program Files/dotnet
/c/Users/travis/AppData/Local/Microsoft/WindowsApps
/c/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin
;C\msys64\mingw64\bin

splitted for clarification. HA! Thats not the path we want. That's why the py in C:\Windows is required. Wondering why our MSYS GTK is found or is there another one in /c/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin? Seems like you can remove the backslashes in the travis.yml.

Question: WHERE pip

[Tontyna]

BTW: Our mingw64-path should be inserted at the beginning not appended, if at all

[liZe]

Setting env: PYTHON=/c/Python37/python seems to work.

HA! Thats not the path we want.

OMG.

Seems like you can remove the backslashes in the travis.yml.

I don't know whether I'm sad or not.

BTW: Our mingw64-path should be inserted at the beginning not appended, if at all

You're right, looks like it's just useless.

[Tontyna]

Conclusion: Dont use py py py py!

[Tontyna]

Nah! Conclusion is: :+1: It works! :+1:

[Tontyna]

Next issue: frequent segfaults. I'm quite sure we shouldnt ignore pacman's advice

###################################################################
#                                                                 #
#                                                                 #
#                   C   A   U   T   I   O   N                     #
#                                                                 #
#                  This is first start of MSYS2.                  #
#       You MUST restart shell to apply necessary actions.        #
#                                                                 #
#                                                                 #
###################################################################

If possible there were not only CAPITALS but RED & BOLD CAPITALS

[Tontyna]

Oh no, only looked like it worked :sob:

[Tontyna]

Still something wrong with the modules? -- can't remember that py -m pytest tried to install any eggs. Really strange.

WHERE pip?

[Tontyna]

What's the reason you want to run setup.py?

[liZe]

Oh no, only looked like it worked

It's strange. Maybe changing the PATH was a bad idea.

can't remember that py -m pytest tried to install any eggs. Really strange.

It doesn't, but python setup.py test does.

What's the reason you want to run setup.py?

Using setup.py gives the possibility to use the same command everywhere. Developers can test WeasyPrint without having to use virtual environments or having to install testing tools (pytest and co.) on their system. Moreover, it tests the installation of dependencies without pip (only using the modules included in Python), as many tools do (my Linux distribution for example).

[Tontyna]

Please somebody tell me why WHERE pip fails!?

[Tontyna]

Please somebody tell me why WHERE pip fails!?

Oh stupid me, there isn't a pip.exe! py -m pip executes the Python module in site-packages/pip

[liZe]

Oh no, only looked like it worked

Did you see it work too once?

[Tontyna]

Did you see it work too once?

No. Not really. Was an imagination, probably wishful thinking

[Tontyna]

Another oddity: The "language: C++" in our windows builds? I recall that I read something somewhere about virtual environments that Travis provides for "language: python". Maybe, maybe, maybe?

[Tontyna]

Besides the question why pytest was able to find/install cairocffi etc. but setuptools organizes eggs -- am I right that e.g. cairocffi via pip comes with precompiled binaries (wheel) but the eggs need a compiler?

[liZe]

No. Not really. Was an imagination, probably wishful thinking

Then I'm really, really tired…

[Tontyna]

No more ideas, too. Maybe tomorrow...

[liZe]

Another oddity: The "language: C++" in our windows builds? I recall that I read something somewhere about virtual environments that Travis provides for "language: python". Maybe, maybe, maybe?

It's not supported for Windows. At least MacOS images provide a "generic" language, but Windows forces to pick one at random.

Besides the question why pytest was able to find/install cairocffi etc. but setuptools organizes eggs -- am I right that e.g. cairocffi via pip comes with precompiled binaries (wheel) but the eggs need a compiler?

Pytest is not able to find dependencies, I had to call python -m pip install . to install them before calling pytest.

When setup.py is called, it doesn't use pip to install dependencies, it uses the old easy_install / distutils mix based on eggs (built from sources during installation), a kind of deprecated package format. Why? Because Pip (and even setuptools) isn't included in the standard library.

Pip is able to install wheels, another (more recent) package format where everything is supposed to be compiled. BUT. As cairocffi needs to be linked to cairo at install time, providing a wheel for cairocffi is not a good idea, that's why only sources are provided.

Why does this fail on Windows? We'll find out tomorrow. Have a good night! :moon: :sheep:

[Tontyna]

Did you see it work too once?

I did! I did! In build 1223: Try something else, where PYTHON=/c/Python37/python was introduced. The verbose "$PYTHON -v setup.py test" failed (reason for failing not logged), but The command "$PYTHON setup.py test" exited with 0.

In the next commit "Finally fix the script part" you removed the verbose command and the non-verbose "$PYTHON setup.py test" fails, reason seems to be: Access is denied, yepp, that's the error we get in all subsequent commits:

error: [WinError 5] Access is denied: 'C:\\Users\\travis\\AppData\\Local\\Temp\\easy_install-j_bsvg3m\\cairocffi-1.0.2\\.eggs\\cffi-1.12.2-py3.7-win-amd64.egg\\_cffi_backend.cp37-win_amd64.pyd'

Get the impression that in "Try something else" the (first, dummy) verbose command prepared/unpacked/installed fine until cairocffi collapsed due to missing access rights.
The second call to setup.py had cairocffi (and the requirements for cairocffi and the right to access them) in place and was able to continue with the (unproblematic) installation of the remaining requirements.

I recall having the same phenomenon in a MSYS shell -- was unable to install all requirements for WeasyPrint in one go/one Python session, packages that require pycs (pycparser, cairocffi) interrupted, but could be installed manually/step by step/by repeating.

[liZe]

I did! I did!

Of course WE DID!

Get the impression that in "Try something else" the (first, dummy) verbose command prepared/unpacked/installed fine until cairocffi collapsed due to missing access rights.
The second call to setup.py had cairocffi (and the requirements for cairocffi and the right to access them) in place and was able to continue with the (unproblematic) installation of the remaining requirements.

I agree.

I've read somewhere that upgrading cffi may help. Let's try.

[Tontyna]

WOW! :tada:

[Tontyna]

Being an inquisitive being I'd like to know:
Is there a build that logs the output of $PYTHON -c "import sys;print(sys.path)"where $PYTHON=py?
I'm wondering why py was able to pip install/update everything, including setuptools, but afterwards unable to find setuptools.
Actually I assume that using py and not /c/Python37/python is the recommended $PYTHON.
Yes, yes, it works now ... but ...

[liZe]

I've read somewhere that upgrading cffi may help. Let's try.

For the record: https://www.reddit.com/r/Python/comments/3bt1k3/pip_install_access_denied_on_windows/

WOW!

:tada:

Is there a build that logs the output of $PYTHON -c "import sys;print(sys.path)"where $PYTHON=py?

https://travis-ci.org/Kozea/WeasyPrint/jobs/504928359

I'm wondering why py was able to pip install/update everything, including setuptools, but afterwards unable to find setuptools.

I'm wondering too…

[Tontyna]

Really strange:

• py -m <module> ... uses the Python3 installed by chocolatey in /C/Python37
• py -c "<commands>" ditto takes that Python3
• py <scriptfile.py> decides to use the minimalist Python2 from chocolatey's MSYS-department

Don't know what a command line switch got to do witch such a weird behavior, but maybe if we decided to follow choco's advice and execute refreshenv ?

[liZe]

but maybe if we decided to follow choco's advice and execute refreshenv ?

I've tried, but it said that the command was unknown. Having bash instead of cmd or powershell makes things harder sometimes.

[liZe]

Until now I experienced no segfaults, but I'm launching test via setup.py.

It actually seems to solve the problem. Why? Another mystery.

[Tontyna]

And mysteries aren't meant to be explored, but exist to be humbly contemplated aka wait & see.

[Tontyna]

However. Voting for a backslash-free, bashy PATH:

  - if [[ "$TRAVIS_OS_NAME" == "windows" ]]; then export "PATH=$PATH:/c/msys64/mingw64/bin"; fi

We can append our part as long as there is no OSError: dlopen() failed to load a library

[Tontyna]

I got it! I mean: I got what py.exe is and why it refused to find setuptools.
It's a tool to launch Python with the ability to "correctly select the most appropriate version of Python", says the doc. It's not on my system because I dont like launchers, especially launchers that go into C:\Windows, so I unchecked the corresponding checkmark during Python installation. And completely forgot about it.

I'm a notorious Windows user, but often I was longing for shebangs. Tonight, with py.exe my wish came true. The first line in setup.py looks like this:

#!/usr/bin/env python

No reason to curse, no reason to cry, no reason to complain. We got what we ordered. See PEP 397

Of course it's still a mystery to me how that shebang empowered py to detect "the most appropriate version of Python" in C:/ProgramData/chocolatey/lib/mingw/tools/install/mingw64, but obviously it did.

[liZe]

No reason to curse, no reason to cry, no reason to complain. We got what we ordered. See PEP 397

Oh :roll_eyes:. I would never have imagined something like this. When you understand that py.exe is a launcher and not an interpreter, everything makes sense. I couldn't help thinking that it's a really strange solution … until I realized that on my linux distribution (Gentoo), the linux executable is just a launcher choosing one of the versions of Python installed on the system. We may look very different, but we're all the same.

[Tontyna]

:bomb: segfault in Job 1233.5

We expected that, didnt we?

[liZe]

We expected that, didnt we?

At least it solves a mystery.

[Tontyna]

The advice given in docs.travis-ci is:

Fix the problem by clearing the cache or removing the cache key from your .travis.yml (you can add it back in a subsequent commit).

Any idea how to do that?

[liZe]

Any idea how to do that?

It seems that cache has to be set up manually and that we don't use it.

To be honest, I'm not that surprised to see WeasyPrint tests crash as I'm not sure that we do what we have to do concerning the C interfaces. I've already seen WeasyPrint crash on Linux on multithreaded environments. Cairo and Pango are said to be thread-safe, but I'm not sure that the way we use them is very safe. As Pytest shares imports and some configuration stuff through the whole testing suite, it's easy to meet race conditions creating random segmentation faults.

[Tontyna]

So let's forget about it? Until in real life #167 strikes again?

[liZe]

So let's forget about it? Until in real life #167 strikes again?

Exactly. If segmentation faults are too frequent, I'll mark Windows tests as not mandatory (as they used to be).