Rules_go: How to access generated files in go_binary

[Globegitter]

I am quite stuck here now, I am using grpc gateway succesfully, but want to add the inbuilt swagger support and quite stuck now on how to do that.

I am using the rule grpc_gateway_swagger_compile from https://github.com/pubref/rules_protobuf/ which generates a file *.swagger.json (which can then be found in the bazel genfiles folder). How can I now make a go_binary (or library) now able to access this json files? If I add it to data the rule just seems to be ignored. I can build my server, but do not have access to the json file. If I try to add it as a dep it errors out with No attribute '_go_context_data' in attr. Make sure you declared a rule attribute with this name.

How is it supposed to work, for a go_library or go_binary to depend on any other rule that generates a file that the code needs at runtime?

[ianthehat]

If you add it to data, it will end up in the runfiles if you bazel run the binary, not if you build it.
This is not really anything to do with the go rules, but I am happy to help out anyway. What is it you tried that did not work, and what were you hoping it would do?

[Globegitter]

@ianthehat thanks for that. I am using bazel run but I will set up a quick example repo when I am back at work tomorrow to demonstrate.

[Globegitter]

@ianthehat Here is the repo: https://github.com/Globegitter/bazel-grpc-gateway-data-example

You see a proto file in the proto folder that is used to generate swagger definitions (a json file). Which after building I can find in bazel-genfiles. Then in the gateway folder both the go library as well as the binary depend on that swagger-generating rule via the data attr. But I can not see the json file in the runfiles folder. If I manually copy the generated file and use export_files I can get it working.

[achew22]

@Globegitter, have you considered using the go_embed_data rule with the output of the grpc_gateway_swagger_compile as the src? That would give you a .go file with a variable that contained the swagger that was generated and compile it into the binary instead of forcing you to construct a runfiles path during deployment? That's exactly what I do in my project for this. I also serve my index.html in the gateway this way rather than doing a runfiles directory. It's great to have a statically linked go binary that is 100% self contained and needs nothing from the FS!

[Globegitter]

@achew22 Ahh that is a good pointer, I did see the go_embed_data, but was not sure if that was a use-case for it. I will give it a try, thanks for the pointer. Would go_embed_data also work for embedding multiple files? E.g. if I wanted to embed a index.html as well as the swagger.

[achew22]

I think it can take a list of targets, but I haven't played with that part of it. You're going to have to try it :wink:. Try creating a go_embed_data rule and then bazel building it. You can then go find it in one of the output files (usually bazel prints the path out) if you wanna see what it generated. I use that trick a lot to see what is generated in rules that I don't understand fully.

[jayconrod]

@Globegitter go_embed_data does work with multiple files. See documentation.

There are basically two modes you can use. If you provide the file to be embedded through the src attribute, the generated file will have a []byte or string variable with the content of that file. You can also embed multiple files with the srcs attribute. In that case, the generated file will have a map[string][]byte or map[string]string variable where the keys are file names and the values are contents.

[Globegitter]

@achew22 @jayconrod Thanks I got it working with go_embed_data. That does make the direct solution to my answer far less important, but is there any reason my posted example should not work?

But if go_embed_data can be used for all use-cases that one could use the data dependency for than I am also happy for this to be closed. And it would now be more a question for completeness sake.

[achew22]

My guess would be that you tried to open the file using the wrong path. Bazel puts files in directories in strange ways. There is a helper that figures out where the files are for you at https://github.com/bazelbuild/rules_go/blob/master/go/tools/bazel/bazel_test.go

[ianthehat]

How did you check whether the file was present?
Looking at the code you were expecting the file to just be in the same directory, but it would probably have had a path on the front.
The best thing to do is look at the .runfiles_manifest file to see what got put where when the binary was run.

[Globegitter]

@ianthehat sorry for the late reply I could work around the issue but now running into it again - a go server wants to depend on a generated css file - I depend on it the following way:

go_library(
    name = "my_library",
    data = [
        ":static_assets",
        "//assets/styles:main",
    ]
)

go_binary(
    name = "my_binary,
    embed =  [":my_library"],
)

Then I look in bazel-bin/my_binary/linux_amd64_stripped/my_binary.runfiles/__main__/ - I can see the static assets but not the generated ones.

Edit: Aahh, if I depend on that generated file directly in the go_binary then it shows up. It seems to me more intuitive to depend on that in the library, but is that maybe expected behaviour?

[jayconrod]

Closing old issues.

You should be able to depend on the data file in go_library (any library in your transitive dependencies), and it will be available in go_binary. I just verified this works.

For go_binary, you can access files in the same workspace using a path relative from the workspace root. For go_test, by default tests run in the directory where they were declared, so a different relative path may be needed. That can be overridden with rundir (which accepts a path relative to the workspace root).

Note that runfiles are not symlinked on Windows. On Windows, you need to to read the runfiles manifest to get absolute paths to data files. The package github.com/bazelbuild/rules_go/go/tools/bazel may help with that.