jq 1.5 win64 fails on UNC input files / jq 1.6 fails with UNC module paths

[KevinRodgers]

Copying the file to a local drive allows it to be processed, but otherwise I get a "jq.exe has stopped working" error (popup window) on Windows 7 Enterprise, Service Pack 1.

[nicowilliams]

What version of jq were you using? I get "jq: error: Could not open file foobar: No such file or directory" with jq 1.5 when I run jq . \\foo\bar.

[KevinRodgers]

Hi Nico,

jq --version reports "jq-1.5"

When I Debug the process, it reports "An unhandled win32 exception occurred in jq.exe [16052]." And then Visual Studio reports "Unhandled exception 0x0040bdd4 in jq.exe: 0xC0000005: Access violation reading location 0xffffffffffffffff."

[jankatins]

Could be https://github.com/stedolan/jq/issues/1072 too?

[nicowilliams]

Yes. @KevinRodgers try the fix for #1072 please.

[nicowilliams]

I believe the fix to #1072 was it.

[jankatins]

@nicowilliams: "believe"? :-) Could you try the jq from here: https://ci.appveyor.com/project/JanSchulz/jq/build/job/x22fq19dlr3tu1t5/artifacts

[nicowilliams]

@JanSchulz Yes, it's a polite way of saying I want the OP's confirmation. Thanks to your appveyor.yml I also have been using it, and my jq artifacts are here: https://ci.appveyor.com/api/buildjobs/s4co10ss1gt547kj/artifacts/jq.exe

I've asked @stedolan to enable appveyor for stedolan/jq's master branch. Travis-CI builds are already required for the master branch.

[jankatins]

@nicowilliams Oups, sorry my bad, I read that as if you were the OP :-)

[nicowilliams]

NP :)

[KevinRodgers]

Would love to, as soon as a binary (win64) is available.
Thanks!

[nicowilliams]

@KevinRodgers You can grab a jq.exe from here https://ci.appveyor.com/api/buildjobs/btulxkol0ck94psd/artifacts/jq.exe (64-bit) and here https://ci.appveyor.com/api/buildjobs/xk4n66aqf38gx957/artifacts/jq.exe (32-bit).

To find these from the https://github.com/stedolan/jq page just follow the link through the Windows CI badge on the README to https://ci.appveyor.com/project/stedolan/jq, then grab the artifacts from the latest build (or, if it is pending/running, from the history tab on that same page. Pick a build, then look for and click on the artifacts tab on the upper right-hand side of the page, and grab the jq.exe from that page.

[sschuberth]

I was having the same issue (but also for non-UNC Unicode paths) and I can confirm that https://ci.appveyor.com/api/buildjobs/btulxkol0ck94psd/artifacts/jq.exe (identifying itself as version 1.5rc2-150-g3f5feb4-dirty) fixes the issue. Please consider making a release with this fix as it's more or less a blocker on Windows.

[nicowilliams]

Thanks. I'll probably not get around to doing a release for a bit longer yet for lack of time.

[sschuberth]

FYI, my work-around this annoying issue is this wrapper jq.cmd file:

@echo off

rem jq 1.5 has a bug in handling Unicode paths, so change to the input file's path and only use its name.
rem See https://github.com/stedolan/jq/issues/1066 for details.

pushd %~dp1
"%~dp0\jq.exe" . "%~nx1" > "%2"
popd

[bit2shift]

jq-1.6 still has problems with Unicode paths when dealing with modules.
The culprit seems to be PathIsRelativeA() in linker.c.

It should use MultiByteToWideChar() to convert back to UTF-16 and then pass the result to PathIsRelativeW().

[nicowilliams]

Thanks for the report @bit2shift!

[bit2shift]

@nicowilliams ok, so I've been debugging jq with the help of gdb and eclipse, and it turns out the problem isn't PathIsRelativeA(), although that might cause some undefined behaviour within.

I have reasons to believe it's coming from the use of the archaic msvcrt.dll in MSYS2's mingw-w64 build of jq:

• the open() call in jv_load_file() from jv_file.c (when passing -f <unicode path>) fails with -1 even though the file exists (gdb tells me _O_U8TEXT is an invalid parameter being passed to _setmode() in main.c).

I've managed to build jq without -municode using MSYS2's non-mingw GCC and Unicode paths are now working with modules.
This uses the MSYS2 CRT (MSYS-2.0.DLL) which is POSIX-compliant as far as I know.

[nicowilliams]

@bit2shift would it be possible to send us a PR?

[bit2shift]

@nicowilliams I'll see what I can do.

Honestly, not only do the Windows' tidbits need a refactor, the whole shebang could use some tidying up because of conditional macros sprinkling all over the code base.

Assuming jq isn't meant to be built using MSVC, we could opt out of the MinGW shenanigans and stick to existing Windows implementations of the POSIX API: Cygwin, MSYS2, etc.

[nicowilliams]

@bit2shift ohh, I get it. Hmm, yeah, this is a big change. Can you show me roughly what you did to build it with MSYS2?

[bit2shift]

@nicowilliams it's quite simple actually.
Remove this section from Makefile.am:

if WIN32
AM_CFLAGS += -municode
endif

And then build as normally inside the MSYSTEM=MSYS MSYS2 shell.

EDIT: the typical _WIN32/WIN32 macros aren't defined when building through MSYS2's toolchain, resulting in this being the only WinAPI function being used in the final executable.

[bit2shift]

@nicowilliams It's implied you need to install MSYS2's GCC via pacman -S gcc.

[nicowilliams]

Ah, OK. Well, our Appveyor config uses MSYS to build jq, though it also has MinGW in the path.

A fix will look like this:

• teach configure.ac and Makefile.am that we have two Windows build types: cross-compilation with MinGW, and MSYS

• when building with MSYS, leave out the -municode compiler flag

For the dlopen branch I'll need to figure out how to call native WIN32 APIs when building with MSYS.

[bit2shift]

About dlopen, you needn't to worry, MSYS-2.0.DLL has got you covered.

[nicowilliams]

Ah, ok, perfect. Thanks @bit2shift!

[bit2shift]

Quick reanalysis of the open() problem in jv_load_file() from jv_file.c.
Considering we already have a wrapper for fopen() that redirects to _wfopen() in util.c, I'm going to add one of the same kind for open() that redirects to _wopen().

Additionally, I'll be fixing two existential buffer overruns in the two calls to MultiByteToWideChar() in the fopen() wrapper.
@cup added sizeof(wchar_t) * in the wrong place (the code he used as basis was buggy to begin with), since the last parameter of MultiByteToWideChar() aka cchWideChar is:

Size, in characters, of the buffer indicated by lpWideCharStr. If this value is 0, the function returns the required buffer size, in characters, including any terminating null character, and makes no use of the lpWideCharStr buffer.

[ghost]

@bit2shift in my defense - this went unchecked for 4 years.

[bit2shift]

@cup I guess most people weren't curious enough to try jq with 𝓯𝓾𝓷𝓴𝔂 paths.

[sschuberth]

Just FYI, I wanted to point out that Git for Windows' compat directory has a bunch of fopen wrappers for different platforms (MinGW, MSYS/Cygwin, MSVC) and these are extremely well tested. Maybe it makes sense to stick to compiling the Windows version of jq with MinGW (not MSYS, so no dependency on MSYS-2.0.DLL) and look at Git for Windows' mingw_fopen?

[bit2shift]

@sschuberth oooof.
You do realise that MSYS-2.0.DLL is the CRT used by native MSYS2 programs? It's an omnipresent system dependency in MSYS2.
If you're going to say "we shouldn't rely on MSYS-2.0.DLL" then you must also say "we shouldn't rely on MSVCRT.DLL".

And no, it doesn't make sense to stick with MinGW when MSVCRT.DLL is quintessentially obsolete by today's standards. MSYS2's CRT (MSYS-2.0.DLL) is POSIX-compliant, which takes away all the necessary UTF-8 -> UTF-16 cruft required to work with MSVCRT.DLL.

Making jq compilable using MSYS2's native toolchain only requires a simple tweak in Automake's config as stated above: https://github.com/stedolan/jq/issues/1066#issuecomment-493630243.

PS: Do not conflate MSYS(1) with MSYS2. The latter is in active development while the former is defunct.

[sschuberth]

You do realise that MSYS-2.0.DLL is the CRT used by native MSYS2 programs?

Yes, although in my view the mention of "MSYS-2.0.DLL" and "native" to not really go hand in hand. MSYS-2.0.DLL corresponds to cygwin1.dll (with MSYS2 still basically being a fork of Cygwin), and cygwin1.dll basically emulates a POSIX API on Windows, including a terribly slow implementation of fork (of course Cygwin's docs just say "a little slow", but it's really very slow). With all that emulation going on under the hood I really shudder to call that "native".

then you must also say "we shouldn't rely on MSVCRT.DLL".

No, because of course depending on a runtime basically is fine, but not on one that emulates POSIX underneath, at least if you care about performance.

MSVCRT.DLL is quintessentially obsolete by today's standards.

That's an interesting statement. Would you have any source for that?

Making jq compilable using MSYS2's native toolchain only requires a simple tweak

I understand that. MSYS2 / Cygwin is for the "lazy" who want to port POSIX programs to Windows with little effort. But if you care about performance and reducing dependencies, you should go "really native", IMO.

Anyway, I was just trying to be helpful by pointing out previous work in that area. Feel free to just ignore my hint.

[bit2shift]

this is just flat wrong - not everyone has an MSYS development environment - so
having a JQ that uses only MSVCRT.DLL and not MSYS DLL would be a huge benefit
for those people.

@cup ok, I might have phrased that in an awkward way. What I meant was having not 1 but 2 builds of jq available, considering the current MinGW release has long standing issues with Unicode.
Understandably, it's all about choices.

Also know your audience: @​sschuberth is one of the top developers on both git
and git for windows:

Which means he's biased towards wanting to use those wrappers. Just like I'm biased towards wanting to have a separate build that doesn't use MSVCRT.DLL.

---

Yes, although in my view the mention of "MSYS-2.0.DLL" and "native" to not really go hand in hand. MSYS-2.0.DLL corresponds to cygwin1.dll (with MSYS2 still basically being a fork of Cygwin), and cygwin1.dll basically emulates a POSIX API on Windows, including a terribly slow implementation of fork (of course Cygwin's docs just say "a little slow", but it's really _very_ slow). With all that emulation going on under the hood I really shudder to call that "native".

@sschuberth I get the performance trade-off, although I would point out that jq doesn't use or have a need for using fork().

No, because of course depending on a runtime basically is fine, but not on one that emulates POSIX underneath, at least if you care about performance.

It's all about choices. Performance vs conformance.

That's an interesting statement. Would you have any source for that?

MSVCRT.DLL was the C runtime that came with Visual C++ 4.2 up until VC++ 6.0. This was back in Windows 95/98. Maintaining it was dubbed a nightmare.
Microsoft later decided to name each newer version appropriately (msvcr###.dll) with each newer release of Visual C++. Here's an article about it written by Raymond Chen.

I understand that. MSYS2 / Cygwin is for the "lazy" who want to port POSIX programs to Windows with little effort. But if you care about performance and reducing dependencies, you should go "really native", IMO.

It's more convenient, I'd say. But then again, choose MinGW if you want performance, choose MSYS2 if you want POSIX conformance.

[nicowilliams]

So, for jq static linking is a very nice way to deliver binaries to users, which means we'll want to keep using MinGW for Windows releases.

That said, I don't like static linking at all.

I like the idea of having appropriate open() and fopen() wrapped for the various windows build systems. What's the license on Microsoft's wrappers?

[bit2shift]

-(...) some function from the [-newer-] version of Windows API. (...)
+(...) some function from the {+older+} version of Windows API. (...)

-If you can point to some functions from [-newer-] Windows API that JQ uses or should use, (...)
+If you can point to some functions from {+older+} Windows API that JQ uses or should use, (...)

@cup your comment only makes sense if you replace newer with older, because that's the root cause of most problems jq has been having on Windows with the implicit use, by MinGW, of a CRT that's barely C89-compliant (http://www.mingw.org/wiki/c99).
It would be nice if MinGW could link to the newer UCRT by default.

@nicowilliams as far as I can tell, simple open() -> _wopen() wrappers are free game.

[sschuberth]

What's the license on Microsoft's wrappers?

Git for Windows is not a Microsoft project, it's a community project, and as such is GPL v2-licensed (it just happens to be so that the main developer hired at Microsoft some time ago). Much of the code has been upstreamed to core Git already anyway.

[bit2shift]

@​bit2shift unless you can point to a specific example, im not convinced. even Golang, Rust and Nim all still use MSVCRT.DLL, so you would need a pretty compelling reason why JQ needs the newer version.

@cup If you look at the internals of those, I guarantee you'll find UTF-8 wrappers for C functions that would otherwise work properly on *nix OSes when fed UTF-8 inputs.
Although I'm fairly certain the Golang runtime only uses the WinAPI. Third-party Go libraries might use it, but what would be the point if Go's standard library is much more comprehensive?

[ghost]

@bit2shift in my eyes you havent made a single cogent argument.

Can you please explain, in clear terms, what benefit would come from newer
Visual C++? It sounds like you are saying that maybe the newer version has some
functions that handle UTF-8 more gracefully, but youve yet to point to a single
function by name or a single link in support of that argument.

If you want to enact change it helps if you make a proper argument with
evidence, else it comes off as you wanting change just because.

[bit2shift]

Can you please explain, in clear terms, what benefit would come from newer
Visual C++?

@cup less bugs and better C89/C99/C11 conformance. In retrospect, you could've stepped forward and shown me where _Golang_, _Rust_ and _Nim_ use MSVCRT.DLL.

But you're probably going to gloss over what I've just said by replying with something along the lines of "404 Argument Not Found", therefore I'm ceasing this conversation.

[nicowilliams]

There's no need to have this argument.

We can support all three Windows platforms: native, mingw, and msys2. As well as cross-compilation using mingw. The reason to support mingw is static linking (see below). The reason for the other two is to make it possible to build libjq.dlls that can be used in msys2 apps and native Windows apps.

For building statically linked executables downloads in the github releases we must use mingw.

[nicowilliams]

BTW, i hate that static linking lacks reasonable ELF-like semantics. And it's nuts that every DLL is generally linked statically with a C runtime -- that's not how we do it on ELF platforms.

But at least for the jq executable, static linking is very convenient!

[bit2shift]

@cup Where's the msvcrt.dll you said Golang was using? Because I'm only seeing kernel32.dll as the sole dependency.
And it's not about specifically supporting a Cygwin/MSYS2 build.

It's about tweaking jq's build configuration to omit the addition of -municode to the compilation flags when building under MSYS2!

@nicowilliams has already stated how this would be done in a comment above 5 days ago.

[ghost]

@bit2shift no, heres what you actually said:

I have reasons to believe it's coming from the use of the archaic msvcrt.dll
in MSYS2's mingw-w64 build of jq:

campaigning against it without evidence.

Assuming jq isn't meant to be built using MSVC, we could opt out of the MinGW
shenanigans and stick to existing Windows implementations of the POSIX API:
Cygwin, MSYS2, etc.

here campaigning against any native, again without any justification.

teach configure.ac and Makefile.am that we have two Windows build types:
cross-compilation with MinGW, and MSYS

Ignoring the proper triplets, which are:

• x86_64-w64-mingw32
• x86_64-pc-cygwin

all the while ignoring that Cygwin and MSYS2 already distribute JQ.

when building with MSYS, leave out the -municode compiler flag

what does that have to do with this thread? This thread is about UNC paths, not
Unicode issues.

@cup I guess most people weren't curious enough to try jq with 𝓯𝓾𝓷𝓴𝔂 paths.

failing even a basic search of past issues. I could go on but I wont. Youre just
a clear example of someone who spews nonsense and derails threads without a
shred of link or evidence to back up what you are saying/demanding. So Ill give
your last response the respect it deserves: none.

[bit2shift]

@cup oh... so, you like to cherry-pick stuff out of my comments to use for engaging in FUD against me? Here's the first quote in full:

@​nicowilliams ok, so I've been debugging jq with the help of gdb and eclipse, and it turns out the problem isn't PathIsRelativeA(), although that might cause some undefined behaviour within.

I have reasons to believe it's coming from the use of the archaic msvcrt.dll in MSYS2's mingw-w64 build of jq:

• the open() call in jv_load_file() from jv_file.c (when passing -f <unicode path>) fails with -1 even though the file exists (gdb tells me _O_U8TEXT is an invalid parameter being passed to _setmode() in main.c).

You chose, out of your own will, to ignore the contents of that bullet list. That's the "evidence" you're supposedly looking for, but you're probably not too savvy to understand those tidbits.
Ok, next quote:

@​nicowilliams I'll see what I can do.

Honestly, not only do the Windows' tidbits need a refactor, the whole shebang could use some tidying up because of conditional macros sprinkling all over the code base.

Assuming jq isn't meant to be built using MSVC, we could opt out of the MinGW shenanigans and stick to existing Windows implementations of the POSIX API: Cygwin, MSYS2, etc.

Again, you chose to ignore the second paragraph which sets the context for the one you chose to cherry-pick.
The rest of what you quoted was between me and @​nicowilliams, not you.

But I'll address the Cygwin/MSYS2 distribution of jq:

• Cygwin has a POSIX-compliant build of jq
• MSYS2 has a MinGW build of jq

A POSIX-compliant build of jq on MSYS2 is desirable.

Fuck off, will you?

You're gaslighting me because you got butthurt from me calling you out on the buggy code you used 4 years ago.

[nicowilliams]

At some point I'll catch up with all of this on a proper computer rather than a smartphone. In the meantime i would prefer if the rhetoric could cool down.

[ghost]

Fuck off, will you?

@bit2shift I have reported you to GitHub admins. Also @nicowilliams I would recommend blocking him from this repository, as it seems he cannot defend his points with sources or links, and can only result to foul language when challenged.

[bit2shift]

@cup you reported me for _foul language_?

@nicowilliams this is my last comment on this issue.

[nicowilliams]

I'm traveling, which is why I'm still only checking this via smartphone.

All participants in this thread should stop participating in it for a few days. Please let it cool. Thanks.