Meson: Project local wrap patches?

[rocha]

Is it possible to include the patch for a wrap dependency in the same project were the dependency is requested? What if the patch only adds a meson.build file?

Sometimes it is easier to create a meson build files for dependencies that do not use meson without having to publish it somewhere. This makes builds less dependent on external services like wrapdb, github or others.

An example of this functionality can be found in the bazel project, where it is possible to declare a 3rd party dependency and include the build file in the same project:

http://www.bazel.io/docs/tutorial/cpp.html#including-external-libraries

[jpakkane]

There are two different ways:

• branch the upstream git repo, add build definitions and tell Meson to get wrap with git instead of file downloads
• create a tarball with upstream + build and put it somewhere Meson uses urllib so you can use file:// urls to get to it

If you just want to use a project you have somewhere you can also just symlink the dir. For example to use subproject libbob symlink like this:

<source root>/subprojects/libbob -> /path/to/libbob

You don't need a wrap file in this case, if the libbob subdir exists Meson will use that directly.

[rocha]

Thanks @jpakkane.

What I would prefer is not to fork or publish a modified 3rd party dependency on the web. Instead, I would like to point to the original source of the 3rd party dependency and include its build file inside the project that requires de dependency. In other words, imagine that I have no way to publish my own version of the dependency or the patch on the web.

[jpakkane]

It does not need to be on the web. It can be on your own machine, just use a file url in your wrap file. Like this:

[wrap-file]
<original package information here>

patch_url = file:///home/username/stuff/mypatch.zip
<patch filename and hash here>

[tp-m]

Out of curiosity, can one specify relative paths? (I believe file:// URIs are always absolute)

[jpakkane]

That depends on how urllib handles them. But they would be relative to the directory the user launches Meson in, which can be anything so it is not reliable.

[MatthewScholefield]

I also see a strong need for this. Most of the time, I only need to insert one or two files. I'd love to be able to perhaps have a folder next to the wrap called libname_wrap or something (and then specify patch='libname_wrap') that would be inserted into the downloaded library.

[flaub]

Without the ability to specify relative paths, it's hard to share the project with a team of developers.

Ideally, the 'patch' could be specified as a shadow directory that is checked into the same repository. The subproject could be downloaded into the build directory and then the shadow directory gets applied/merged into/onto this directory. From there, meson would pick up any meson.build files it finds.

[jpakkane]

it's hard to share the project with a team of developers

For team sharing you almost always want to either:

• put your dependency project in Git with all the bells and whistles and use wrap_git instead of wrap_file
• have a local "wrap db" which houses your sources + patches, the only thing you need is an HTTP server in your intranet

We try to keep the patching and forking setup at a minimum. There are many tools that do it a lot better than we ever could, people should use those instead.

[flaub]

Do you have a tool for generating and managing these patches? I'm OK with using an internal HTTP server. But AFAIK, the dev cycle looks like:

1. Add .wrap file for desired dependency
2. Run ninja to get the dependency to be downloaded (is there a way to do this step independently?)
3. Add a meson.build file or any other files needed (like config.h files, etc) to the downloaded subproject directory.
4. Create a patch .zip file by hand (error prone) by copying new/modified files into another directory and running zip, tar, etc.
5. Compute hash and upload file to some HTTP server
6. Add patch specifications to the .wrap file
7. Delete the downloaded subproject directory and run ninja again to ensure that everything now works.

Maybe I'm doing this all wrong?

If the patch was a shadow directory, it seems like this process would be simplified quite a bit. Also, updating to a new version which may require changes to the patch would be a lot simpler if I could just edit a meson.build file that was checked into my source tree.

[jpakkane]

You don't want to create build definitions for a dependendy in a subproject. The build definition should stand on its own. Thus you want to create a fresh project with your source checkout, write the build definitions there and zip them up. This should only contain meson.build files and possibly a config.h.meson. I have done many project conversions and have not needed to change source code AFAICR. If you are wrapping an open source project, please submit your work to wrapdb so other people can use (and improve) it as well.

If you just want to hack your own thing with changes then you can do the following:

• download the source tarball
• create a new git repo with the contents of the tarball
• add build definitions (which are identical to the ones above)
• once done host the git repo in your own hosting platform (or Github, or wherever you want)
• use it as a subproject by writing a wrap file that uses a git checkout rather than plain files

Some people also get subprojects as Git submodules which is similar to the one mentioned above. I'm personally not a fan of submodules but if they work out for your project setup, go for it.

[flaub]

I suppose I don't need to use wrap files since it doesn't suit my needs. As long as I have something that gets the subprojects directory setup, then I can do the shadow thing I'm proposing by myself. I thought perhaps other users might find this helpful. I can submit a PR with a prototype if you're still interested.

[rib]

The idea of local patches sounds good to me too (i.e. not a file: url specific to my system, but a relative path to a .patch file to be applied after unpacking the source.)

In my case I'm wrapping some upstream projects (e.g. dlib, flann, pcl, opencv) that need no other changes besides the addition of a minimal meson.build file but some of them are fairly complex projects where I only care to build some minimal subset for my needs. In that case it wouldn't seem appropriate to publish something in wrapdb since these meson.builds are currently tailored to one project (otherwise it's a huge effort to create a fully fleshed out meson.build for these projects up front that's suited for all use cases).

Hosting these meson.build files externally for the sake of having a url currently seems a bit overkill.

[rib]

Just a late follow up to mention that the workaround I went with for our project was to create wrap files like:

[wrap-file]
directory = flann-1.9.1

source_url = https://github.com/mariusmuja/flann/archive/1.9.1.tar.gz
source_filename = 1.9.1.tar.gz
source_hash = b23b5f4e71139faa3bcb39e6bbcc76967fbaf308c4ee9d4f5bfbeceaa76cc5d3

patch_url = https://fake.url.to.please.meson.but.file.in.packagecache/
patch_filename = flann-wrap-1.9.1.tar
patch_hash = ab55865b80ac5819b8a31c318155186ab04e3fee622d0573f5dd4ff42d90b71d

_(note the fake url)_

and commit uncompressed patch tarballs with our patches to the subprojects/packagecache based on the knowledge that that's where meson looks first for downloaded patch files before fetching from the url. It's of course not ideal to rely on an internal detail like that but for some subprojects it really suits us better to have the patches version controlled locally.

Besides it not always making sense to create or use shareable wrapdb patches I also feel a bit more confident that I could check out older versions of our project and have the build work with it being less dependent on remote resources.

[textshell]

I think this is mostly implemented by #4327. Can this be closed now?

[rocha]

@textshell Excelent! I haven't tried it yet, but looks very useful. Maybe close this once #4327 is merged? I am not sure what is the issues policy for this project.

[textshell]

It's already merged. Feel free to close this issue :)

[rocha]

Fixed by #4327