Git: Consider bundling JQ

[SteveALee]

It's excellent you bundle cURL. Thanks.

As many APIs return JSON would you perhaps also bundle jq which makes it easy to explore JSON from bash?

https://stedolan.github.io/jq/download/

Thanks

[dscho]

@SteveALee what a wonderful opportunity for you to start contributing patches!

It will require these steps:

Install Git for Windows' SDK

Get it from here: https://gitforwindows.org/#download-sdk

Add a package definition for jq

The easiest way would probably to follow this example: https://github.com/git-for-windows/MINGW-packages/tree/master/mingw-w64-connect, but add it to the https://github.com/git-for-windows/build-extra/ repository (in a new mingw-w64-jq directory).

To do that, just call sdk cd build-extra, then mkdir mingw-w64-jq && cd mingw-w64-jq, then copy-editing the PKGBUILD file of the mingw-w64-connect project.

Verify that everything works by building it using

MINGW_INSTALLS=mingw64 makepkg-mingw

and then installing it via

pacman -U <generated-pkg.tar.xz>

Then test it. If it does not work, try to fix the bugs, lather, rinse, repeat.

Open a PR at https://github.com/git-for-windows/build-extra/

Of course, we will want to benefit from this, and to do so, follow the example of wintoast in the make-file-list.sh script in build-extra.

Looking forward to reviewing your PR!

[rimrul]

There is already a JQ package. https://github.com/git-for-windows/MINGW-packages/tree/master/mingw-w64-jq The Git for Windows manifest is at version 1.5, the MSYS2 manifest is at version 1.6.

[SteveALee]

@rimrul What does that mean in terms of what needs to be done for it to be installed for use in Git 4 Windows bash?

I used to use MSYS +MINGW but MSYS2 seems very different.

[rimrul]

Basically the same things that dscho pointed out above, except writing your own PKGBUILD file. Testing that it works, potentially fixing bugs, including it in the installer and creating a PR.

[SteveALee]

I'm stuck wit ha missing run time dep.

I took the extra step of checking out master for build-extras but it made no difference.

 MINGW_INSTALLS=mingw64 makepkg-mingw
==> Making package: mingw-w64-jq 1.5-4 (14 Mar 2019 21:15:33)
==> Checking runtime dependencies...
==> Missing dependencies:
  -> mingw-w64-x86_64-oniguruma
==> Checking buildtime dependencies...
==> ERROR: Could not resolve all dependencies.

I obviously need to do somehting else. perhaps mkpkg-mingw -s?

I can't figure out where that dep is coming from.

[rimrul]

https://github.com/stedolan/jq/blob/36c250a03dfde80d6667c1c578e7e488b8d9caab/src/builtin.c

That's where they use oniguruma.

[SteveALee]

The -s option built but the tests fail. So assuming -s is safe to use they use valgrind for the tests. Is it reasonable to expect get valgrind working or should I just disable tests?

Ill switch up from 1.5 to 1.6 once I run sha256sum in wsl

[dscho]

Is it reasonable to expect get valgrind working or should I just disable tests?

No, valgrind only works on Linux and macOS, AFAICT.

But since there is a PKGBUILD definition in MINGW-packages, isn't there an official build by the MSYS2 project? If so, you could simply install it via pacman -S mingw-w64-x86_64-jq.

Having said that, I am a bit disappointed to see that dependencies are required. I thought the deal was that jq is a small, stand-alone project without dependencies. At least that is what their website claims:

jq is written in C and has no runtime dependencies [...]

BTW the 64-bit Windows download they offer weighs in with 3.4MB. I am not really enthusiastic about the idea to force yet more megabytes onto Git for Windows users who for the most part would probably not benefit from having JQ on their machines.

After all, it is not required to have JQ to keep Git for Windows working. Not at all, I would say...

Sure, we bundle curl.exe, but that is more for historical reasons, combined with the fact that we already have to bundle libcurl-4.0.dll to support fetching and pushing via HTTPS. So bundling curl.exe on top just adds a tiny penalty, and some of our scripts (e.g. git update-git-for-windows) depend on it.

On the other hand, JQ brings no benefit to Git whatsoever...

[SteveALee]

But since there is a PKGBUILD definition in MINGW-packages, isn't there an official build by the MSYS2 project? If so, you could simply install it via pacman -S mingw-w64-x86_64-jq.

Ah - I didn't realise that. That's just fine. I bet other user of Git4Win don't realise that either. I suggest it's worth mentioning on the website. I assume we have to use YOUR list of packages to see what's available

Having said that, I am a bit disappointed to see that dependencies are required.

I agree!

On the other hand, JQ brings no benefit to Git whatsoever...

So bundling curl.exe on top just adds a tiny penalty, and some of our scripts (e.g. git update-git-for-windows) depend on it.

+1

Thanks for briefly entertaining the idea :)
I learnt more about G4W runtime - eg git + MinGw + MSYS2.

[SteveALee]

oh rats pacman is not deployed when we install. I guess you meant in the SDK only :(

[dscho]

oh rats pacman is not deployed when we install. I guess you meant in the SDK only :(

Right, pacman is not installed with Git for Windows, as it is really not necessary for Git to work!

And yes, the SDK caters to contributors for the Git project, as it is historically super hard to contribute on Windows.

But here is an idea: why don't you work on a minimal Git for Windows-flavored MSYS2? I.e. a subset of Git for Windows' SDK that comes with Git preinstalled, and with pacman preinstalled, but not with gcc and friends?

Essentially, you'd modify make-file-list.sh to give you the full list of files included in the packages.

Or you'd write a new script that generates the list of files to include, based on the output of pactree and pacman -Ql <package> (with a couple of strategic top-level packages such as mingw-w64-x86_64-git, bash and openssh as starters), and also including /var/lib/pacman/ (filtered to include only the subdirectories indicates by pactree's package list)?

This script could be contributed to the build-extra repository, next to make-file-list.sh, and I could wire it up with an Azure Pipelines to generate, say, a 7-Zip-based installer just like the portable Git, whenever our SDK updates. Of course, this installer would be published as artifact of that pipeline.

We would then have the opportunity to link to those artifacts from Git for Windows' home page, and interested users could download and install this, and update with pacman -Syu and install whatever other MSYS2/MINGW packages they want with, say, pacman -Sy mingw-w64-x86_64-jq.

That way, you could totally get exactly what you want. You just have to put in a little effort, and I will help you and also put in a little effort to help you (and other users with similar wishes) to get what you want.

Deal?

[SteveALee]

Let me digest that - I think l I get what you are saying and it's IS very interesting as an extensible bash environment :)

I won't reopen this issue but might open a new one.

Thanks for the idea and offer!

[SteveALee]

Yep, deal :)

I forked this repo but that’s probably not needed I guess. Plus, I created an issue there to keep your project repo clean.

However I’ll perfectly happy to work here.

[rimrul]

You'll need the fork to create a PR. Thanks for working on this.

[SteveALee]

Good point, I had a Q on my issue and tried to tag you. Just trying to be clear if my plan is to add a script so the existing SDK can also build an new custom installer? Rather than create a new SDK release which is how I 1st read your comments above.