Meson: Allow configure_file, custom_target and generator output file to subdirectories under builddir

[jasonszang]

I've encountered two cases that would require configurating / generating file into subdirectories.

1. Project configuration header that would be included from other headers in the same project

Say I have a library project named foo and the project layout is like this:

include/
    foo/
        bar/
            foobar/
                foobar.h # includes config.h
            bar.h        # includes config.h
        foo.h            # includes config.h
        config.h.in # configured by meson
src/
    foo/
        foo.cpp
...

We would want our client code use #include <foo/config.h> to include our config.h since we don't install config.h directly to includedir. But for our own code this would be impossible since generated config.h is directly under <builddir>, not <builddir/foo>.

Of course, for foo.cpp the work-around is easy: just #include <config.h>. But it becomes complicated we need to include config.h from bar.h and foobar.h.

In fact, there is no way to correctly include config.h that both work in project directory and after installation. If we want bar.h and foobar.h to be correct after installation we must #include <foo/config.h> or use relative path in them. Including <foo/config.h> won't compile, and although relative path happens to work for bar.h, it won't for foobar.h. Neither will it work if we reside in a subdir. Using #include <config.h> allows us to compile, but bar.h and foobar.h breaks up after installation.

The correctly way to configure and include config.h should be just generate it under builddir/foo and #include <foo/config.h> everywhere, just as if it were in the position of config.h.in. Currently meson does not allow this, and forces a flat include directory, or hacking into builddir.

1. Protobuf generation

This one is simpler. If we have some protos like

protos/
    foo/
        foo.proto
    bar.proto
````
in which `bar.proto` imports `foo.proto`.
~~it would be nice if we could use one generator for both protos and and get under build directory something like this:~~
For the compiled protos to work the directory structure of generated sources **must** mirror the protos, because bar.pb.h will `#include "foo/foo.pb.h"`. The generated sources must look like this:

protos/
foo/
foo.pb.h
foo.pb.cc
bar.pb.h
bar.pb.cc
````
Althouth this would also require something like @DIRNAME@ substitution if we are to pass generated sources to build targets properly, but without supporting generating source trees we cannot use protos with subdirectories. This also prevents wrapping of existing projects that use protos this way, which is quite a common practice.

PS: I suggest putting every generated file from configure_file() and custom_target() (including ones in subdirs) into one directory tree named something like <builddir>/sources@gen, add it to include directory, and allow outputing to subdirectories under it. This should avoid potential name clashes and give developers control over generated folder layout.

[rom1v]

I think I have a similar case: https://stackoverflow.com/questions/46729488/how-to-compile-aidl-files-using-meson

I need these source files:

• src/com/rom1v/example/Service.aidl
• src/another/package/Other.aidl

to be compiled under:

• builddir/gen/com/rom1v/example/Service.java
• builddir/gen/another/package/Other.java

From the java compiler point of view, this is an error for a java file not be in a directory matching the package name.

[liugang]

2548 is possible way to fix this type of issue.

if one project call configure_file() with install_dir, install_headers(), and so on. copy these header files to a include_intermediates_dir, and all compile target include this intermediate directory.

[jasonszang]

@nirbheek
Would you please be kind enough to explain to me the considerations behind the design choice of Output must not contain a path segment. in source generation targets like custom_target and why are generating code into subdirectories of build dir considered harmful?
I am trying to figure out a way to properly compile protobuf proto files with custom_target (by properly I mean get meson to figure out the dependency relations correctly), which I would like to use in a small production project which for some reason does not fit in our in-house build system. I acturally found a way that work with proto files that has no subdirectories (all protos must live in a single folder), but proto subtrees like the one I wrote at the top can not be built by meson. Then I commented out these two lines in build.py

    if '/' in i:
        raise InvalidArguments('Output must not contain a path segment.')

And it worked surprisingly well. But I'm afraid by doing this I may be breaking something else inside meson. Could you please give me some explanation about this limitation and let me know your thought about its removal (or replacement with something like must not contain .. component if escaping is the problem)? With this limitation proto trees with subdirectories seems never could be properly built by meson.

[rib]

Just to mention that I recently (https://github.com/mesonbuild/meson/issues/3023) also found it a little awkward that configure_file()'s output filename couldn't include a subdirectory while I wanted to generate a .java file based on some build time config options. The javac compiler and Meson's jar() require source code to be laid out according to the namespace of the Java package.

In my case it's possible to work around the limitation by creating a meson.build under src/java/com/impossible/glimpse/meson.build where I use configure_file() which results in a file like <builddir>/src/java/com/impossible/glimpse/<output file>

I wonder if the subdirectory were limited to matching the subdirectory of the input filename that might appease some of the general concern around having random control over the build directory layout. This would still allow meson control to change the top level structure of the build directory but it would have to preserve the subdirectories requested for an output file.

E.g. if I want to generate a GlimpseConfig.java from GlimpseConfig.java.in in the package com.impossible.glimpse with a file under src/java/com/impossible/glimpse/GlimpseConfig.java.in then ideally I could do:

configure_file(input: 'com/impossible/glimpse/GlimpseConfig.java.in',
        output: 'com/impossible/glimpse/GlimpseConfig.java',
        configuration: conf_data)

And in the build directory meson can create:
<build>/any/meson/private/top/level/com/impossible/glimpse/GlimpseConfig.java

The top level layout doesn't really matter (at least in my case) so long as the java compiler + ninja backends knows what it is.

So it might not imply spaghetti access to arbitrary build directories, just some control to build a sub-heirachy within the build directory.

[NickeZ]

Also ran into this problem with custom_target and doxygen. The call to doxygen will produce the html files needed and the latex sources needed. Doxygen also produces a Makefile that helps you run pdflatex the right amount of times and in the right order.

I would like to write something like this:

cdata = configuration_data()
cdata.set('TOP_SRCDIR', meson.source_root())
cdata.set('TOP_BUILDDIR', meson.build_root())
cdata.set('VERSION', meson.project_version())

doxyfile = configure_file(
    input: 'Doxyfile.in',
    output: 'Doxyfile',
    configuration: cdata,
    install: false
)

html_target = custom_target(
    'htmldocs',
    build_by_default: false,
    input: doxyfile,
    output: ['html/index.html', 'latex/Makefile'],
    command: [doxygen, doxyfile],
)

pdf_target = custom_target(
    'pdfdocs',
    build_by_default: false,
    input: 'latex/Makefile',
    output: 'latex/refman.pdf',
    command: [make, '-C', '@OUTDIR@/latex'],
)

docs/meson.build:15:0: ERROR: Output must not contain a path segment.

[nussbrot]

I also ran into this problem and currently don't know how to work around it.
I'm evaluating meson and am trying to figure out if i can use it to build a FPGA project.
One of the problems is IP-generation.

This usually works like this: You generate something like a IP-variation file, which is some kind of XML format, good for versioning and then you run a IP-generator on this file and get a bunch of output files.

Example output files for generating the IP cms_sys_pll:

cms_sys_pll.cmp
cms_sys_pll.csv
cms_sys_pll.html
cms_sys_pll.spd
cms_sys_pll.xml
cms_sys_pll_generation.rpt
simulation\aldec\rivierapro_setup.tcl
simulation\cadence\cds.lib
simulation\cadence\cds_libs\pll_0.cds.lib
simulation\cadence\hdl.var
simulation\cadence\ncsim_setup.sh
simulation\cms_sys_pll.sip
simulation\cms_sys_pll.v
simulation\mentor\msim_setup.tcl
simulation\submodules\cms_sys_pll_pll_0.vo
simulation\synopsys\vcs\vcs_setup.sh
simulation\synopsys\vcsmx\synopsys_sim.setup
simulation\synopsys\vcsmx\vcsmx_setup.sh
synthesis\cms_sys_pll.debuginfo
synthesis\cms_sys_pll.qip
synthesis\cms_sys_pll.vhd
synthesis\submodules\cms_sys_pll_pll_0.qip
synthesis\submodules\cms_sys_pll_pll_0.v

So i tried to put this into a cutom target like so:

qsys = find_program('qsys-generate')
python = import('python').find_installation('python')

files = run_command(
  python, 'cat.py', files('ip/cms_sys_pll.outputs'),
).stdout().strip().split('\n')

custom_target('cms_sys_pll',
  build_by_default: true,
  output: files,
  input: 'ip/cms_sys_pll/cms_sys_pll.qsys',
  command: [
      qsys,
      '--output-directory=@OUTDIR@/cms_sys_pll',
      '--synthesis=VHDL',
      '--simulation=VERILOG',
      '@INPUT@']
)

And of course i ran into the error:

Output 'simulation\\aldec\\rivierapro_setup.tcl' must not contain a path segment.

Do you have any suggestions to work around this issue?

[jasonszang]

Well personally I decided to give up meson and went back to the good old CMake because of this one. And I find modern style CMake comfortable enough I stopped looking for another build system generator.
And unfortunately I am not aware of any universal work-around beside removing this limitation from meson.

[nussbrot]

Hi @jasonszang, thanks for your answer. I see this issue was first posted by you. If you don't mind i would like to ask you if you have experience with CMake and let's say "custom toolchains"?
But it is really off-topic and this is probably not the right place to ask you that.

[jpakkane]

The question here is does the file really need to be in a cm_sys_pll subdirectory (some tools are nasty in this way). If no, then write it to @OUTDIR@. If yes, then you need to have the custom target declaration in the cm_sys_pll subdirectory (and also have just --outdir=@OUTDIR@. In Meson all outputs go to the build directory that corresponds to the source directory they are declared in. This simplifies debugging build problems, since you always know that files in build directory X must come from the corresponding source dir X.

[nussbrot]

Yeah, you're right, that is a mistake in my build file. I can give the tool any directory, so --outdir=@OUTDIR@ works fine. But i can not control the subfolders like synthesis\... or simulation\... that are created in @OUTDIR@.
So i can not add the resulting files as outputs since stuff like output: 'synthesis\cms_sys_pll.qip' does not work.

[ptsneves]

@jasonszang I have the same frustration. One way to work around it is that you create a sub meson.buil in the subdir and do the remaining thing there. Then subdir the newly created meson.build. Not optimal but i think it works

[myfreeweb]

gir-to-d also generates subdirectories, like

somelib/c/types.d
somelib/c/functions.d
somelib/Obj1.d
somelib/Thingy.d

Why is this ridiculous limitation still present? >_<

UPD: seems like there are workarounds in gir-to-d for this

[rulatir]

One way to work around it is that you create a sub meson.buil in the subdir and do the remaining thing there. Then subdir the newly created meson.build. Not optimal but i think it works

@ptsneves, how does it work for you? It most certainly doesn't work for me. Assuming my project is in foo, then I create foo/images/openresty/meson.build and attach the directory with subdir('images/openresty') in the main meson.build file. Now when I write the command to build the openresty image, there is not a single @ VARIABLE@ that contains the exact string 'images/openresty' that I could use to construct the output file path that the build tool must receive. The build tool is a wrapper that cds into the source directory and executes the remaining arguments as a command, and that is necessary because of course Meson is so opinionated as to disallow setting the working directory for the command, so I must hack around that. Then again the command that the cd-wrapper receives is supposed to be a pipeline (an alternative to which is writing ANOTHER wrapper script, which I want to avoid at all cost because at this point the build configuration and script wrappers combined would become bigger than the Dockerfile they are supposed to build from). It must be a pipeline because it must do two things: build the image and then pull the image .Id from docker inspect and save it in the surrogate output file (because of course Meson has no idea about outputs that don't materialize as files).

Yes, this is becoming a rant. It would seem like Meson is just limitations, quirks and opinionatedness without offering "strength" as a reward for accommodating those. Doing hashes instead of timestamps and correct graph-theoretic minimal incremental builds are about the only two "cool" things that Meson offers, and both are becoming standard in the build system world.

[ptsneves]

@rulatir What about you maintaining the variable for the subdir yourself. You do _subdir(var_with_path)__ and then use the __var_with_path__ as you see fit. Does this not work for you?

[rulatir]

@rulatir What about you maintaining the variable for the subdir yourself. You do _subdir(var_with_path)__ and then use the var_with_path as you see fit. Does this not work for you?

"Why don't you do it yourself" is the ultimate non-answer to every feature request. I mean why even use software? Just compute in your head.

[ptsneves]

@rulatir I understand your point as a user of the software in general. As a developer and user of open source software I have to say that we are in a "tough luck" situation. The thing is that in open source software there are 2 ways to move forward with a feature: To convince somebody or to submit a patch/pull request yourself. In a closed source software you basically only have to convince, but the convincing can be pretty hard to do if there is no business case.

It seems the convincing another person to fix/do this feature failed, but the pull request approach has not failed from what i see in the thread. Even if it is not accepted you can patch your meson to have this feature if you want. It is not ideal but it is yet another workaround. I have been in this last situation a lot of times, but it is damn better well than just not being able to do anything about it.

[mensinda]

@jpakkane are there any design decisions for this restriction or is this only an implementational detail? If not, I would see no reason to not allow this behavior (with tests of course).

[rulatir]

@jpakkane

This simplifies debugging build problems

When you are forced to refactor your build process inside out to accommodate a build system with artificial restrictions, you will inevitably make mistakes, and thus introduce problems.

[dankegel]

generator() has a preserve_path_from option which may help.

custom_target() doesn't, which is the subject of https://github.com/mesonbuild/meson/issues/6418

[xclaesse]

Note that @nirbheek already made a patch that does this, in an abandoned PR: https://github.com/mesonbuild/meson/pull/2617/commits/7c15837bd25d37672bfddcb6ef9402792b3d3c23.

[stapelberg]

I’m running into the same problem, and the following workaround (creating a stamp file like with autotools) worked for me:

anyevent_i3 = custom_target(
    'anyevent-i3',
    # Should be AnyEvent-I3/blib/lib/AnyEvent/I3.pm,
    # but see https://github.com/mesonbuild/meson/issues/2320
    output: 'AnyEvent-I3.stamp',
    command: [
        'sh',
        '-c',
        'cp -r @0@/AnyEvent-I3 . && cd AnyEvent-I3 && perl Makefile.PL && make && touch ../AnyEvent-I3.stamp'.format(meson.current_source_dir()),
    ],
)

test(
    'complete-run',
    complete_run,
    depends: [
        anyevent_i3,
    ],
)

[zzyyxxww]

This is the second "our way or the highway" thing I see in meson, really not enjoying it. This limitation is annoying and from reading this thread I don't see a real reason for it.