Jsonnet: STATIC ERROR: std.jsonnet:1:1: Unexpected end of file.

[nnmrts]

Hey! I tried using the binary version of jsonnet. However, whatever I do, it gives me back this error:

STATIC ERROR: std.jsonnet:1:1: Unexpected end of file.

I read on the jsonnet webpage here, that std or stdlib is kind of a utility functions object for jsonnet itself. The error message suggests that the file is empty. Unfortunately we don't know really, which file is meant, it could be the std.jsonnet or the std.jsonnet.h file. At first I thought it's the first one, but that was full with 42kb of data. However, the problem is, there is an error, as I just discovered. A specific unexpected token prevents the file from being built and so the std.jsonnet.h file is empty in the end.

Will create a pull request fixing this issue and reference it here. Sorry for even submitting this issue now that I already discovered what the actual issue is. Nevertheless, I think it will bring more clarity to my pull request.

[sbarzowski]

Thanks for reporting the issue. Could you provide some more details?

• What version of jsonnet are you using to get the error (jsonnet --version). You mentioned _binary version_, but where it's from?
• The specific command that leads to the error (steps to reproduce)
• Can you try reproducing the issue on current master?

The : is there for a reason - it's a slice (like in Python). This syntax is supported only in relatively new versions of jsonnet (and not recognized by some 3rd party tools like e.g. vscode extension).

Here is some more context in case you are interested:
The standard library is embedded in the final executable or library. This means that std.jsonnet is used only in the build process, to build std.jsonnet.h - which is basically an array of bytes from std.jsonnet (https://github.com/google/jsonnet/blob/master/core/desugarer.cpp#L74). When some code is executed, the standard library is taken from memory (i.e. the array I mentioned earlier) and attached to that code (https://github.com/google/jsonnet/blob/master/core/desugarer.cpp#L884), with some additional magic for builtins etc.

[nnmrts]

Hey! Sorry for providing so few information. These are the commands and versions and all of that, sorted by their time of installation:

Versions:

Windows: 10.0.15063
Powershell: 5.1.15063.608
Python: 2.7.14:84471935ed
Visual Studio Community 2017: 15.3.5
chocolatey: 0.10.8
Bazel: 0.6.0
jsonnet: 0.9.5

Commands:

all executed in one session in the local jsonnet repo

set BAZEL_SH=C:\msys64\usr\bin\bash.exe
pacman -Syuu git curl zip unzip
set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_112
set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC

bazel build -c opt //cmd:jsonnet

jsonnet "somefilename.jsonnet" -o "someotherfilename.json"

The thing is, I don't really understand C++ and things involved with it, so I kinda brute forced the available build options to get jsonnet to work on a Windows machine. So some things probably were downloaded or installed already half way through when they threw an error. I currently can't reproduce this issue with another, "clean" Windows 10 machine, because I simple don't have another and didn't want to reset all of the requirements I installed on my current machine, I hope that's understandable 😄. I try to provide you with the infromation relevant to what I think finally built the jsonnet.exe for command line:

I already had Python 2.7 and Python 3.6, but I needed msys2 packages and the Java JDK 8 to get Bazel to work. So I downloaded and installed them successfully. Then I downloaded and installed Bazel with chocolatey, the first few times it wasn't working because some requirements weren't installed, then finally I got it. Then I cloned the jsonnet repository to my desktop and tried to build it with Bazel. This wasn't working several times because my pc needed some Windows build tools, SDK, C++ stuff and Visual Studio 2017. Installing these things was a little depressing and took so crazy long, but after that, the bazel build command for jsonnet didn't show errors anymore. Yay! So I thought jsonnet finally works now. Tried it out, didn't work. It just showed me the error STATIC ERROR: std.jsonnet:1:1: Unexpected end of file.

So I looked in the repository for this file name and found two files, the std.jsonnet and the std.jsonnet.h. The std.jsonnet.h was empty, the std.jsonnet wasn't. So I thought: "Yeah, the .h file probably gets built from the std.jsonnet file when running the bazel build command. So I opened the .jsonnet file in Visual Studio Code and the jsonnet syntax extension for it showed me an error immediately: Unexpected token: ':'

So removed the colon, entered the bazel build command again. And yay, now everything was working fine.

---

So to finally answer your three questions 😆:

What version of jsonnet are you using to get the error (jsonnet --version). You mentioned binary version, but where it's from?

Jsonnet commandline interpreter v0.9.5 - and I think currently the jsonnet command just executes the jsonnet.exe located in jsonnet/bazel-bin/cmd. However, it works globally because I set it as an environment variable.

The specific command that leads to the error (steps to reproduce)

jsonnet "somefilename.jsonnet" -o "someotherfilename.json" - The jsonnet file was definitely a valid, simple file. I even pasted it into the online demo and got it working.

Can you try reproducing the issue on current master?

As I said, I cloned the repository directly, and that was two days ago, so my local version of the repo is identical to the current master branch here, which was updated four days ago the last time. And yes, I retried this whole process (not the whole process, but you know, cloned the "clean" repo again and executed the build command). The final jsonnet file compile command still only works if I remove the colon from std.jsonnet and build jsonnet again.

---

Thank you for reading! I hope this is somehow helpful. 🌷

[sbarzowski]

Thanks for the detailed explanation. You may be one of the first people to use it on Windows, so you're exploring some uncharted territory. Anyway, we want to provide proper experience on Windows too, so that's very helpful!

I doubt that it's the colon that made the difference (we don't lex or parse it at all at this point, we just convert it to a comma separated list of byte values, using a bunch of piped commands https://github.com/google/jsonnet/blob/master/stdlib/BUILD#L20). My working theory is that it's something else that your editor did (newlines? modification time?). In particular I suspect that if you added the colon back it would still work.

I don't have any Windows machine available, but I'll try to get one soon to get to the bottom of this.

[nnmrts]

Wow. Yes. You're right. :D

Just cloned the repository to a new test folder, and it works, untouched...so probably the colon wasn't even significant at all...

smh 😆

why

why

why

_why

😢

---

So I guess this has something to do with corrupted c++ or bazel or visual studio installations? Meh. I will never be able to reproduce this issue.

But now that I actually know how to build jsonnet on Windows, I would be the perfect person to write the docs for it, haha! Yeah...no. I'm not a native english speaker and couldn't keep it simple and short. But I tried to make a short list of steps, maybe that's useful in the future:

1. Download and install Python.
2. Download and install msys2, open it, run pacman -Syuu git curl zip unzip and close it after it's finished.
3. Download and install Visual Studio 2017 with the "Visual C++" package, including "Common Tools for Visual C++" and "Microsoft Foundation Classes for C++" packages.
4. Download and install the Windows SDK.
5. Install chocolatey according to the instructions here.
6. Close any open shell and open it again.
7. Run set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC.
8. Run choco install bazel.
9. Download and install Git, run refreshenv and then run git clone https://github.com/google/jsonnet.git or just download this repository and unzip it.
10. Run cd path/to/local/copy/of/repo and then bazel build -c opt //cmd:jsonnet.
    

    ---

    
    Should work on every system >= Windows 7.

[sbarzowski]

Ok, I got a Windows machine. So far I was able to build jsonnet using g++ and make (installed using pacman/msys2). The jsonnet command works and tests for the command pass. The tests for the library don't work out of the box, though.

cc -g -O3 -Wall -Wextra -pedantic -std=c99 -fPIC -Iinclude  core/libjsonnet_test_snippet.c -L. -ljsonnet -o libjsonnet_test_snippet
/usr/lib/gcc/x86_64-pc-msys/6.3.0/../../../../x86_64-pc-msys/bin/ld: cannot find -ljsonnet
collect2: error: ld returned 1 exit status
make: *** [Makefile:146: libjsonnet_test_snippet] Error 1

I haven't tried VS or bazel yet.

[sparkprime]

If I had to guess, I would say that this rule is not working properly on Windows:

genrule(
    name = "gen-std-jsonnet-h",
    srcs = ["std.jsonnet"],
    outs = ["std.jsonnet.h"],
    cmd = "((od -v -Anone -t u1 $< | tr \" \" \"\n\" | grep -v \"^$$\" " +
          "| tr \"\n\" \",\" ) && echo \"0\") > $@; " +
          "echo >> $@",
)

[sparkprime]

The std.jsonnet file is "compiled" into a C file where it is represented as an array of bytes. This makes it easy to link it into the Jsonnet binary. So std.jsonnet.h should not be empty, it should be a very long file of comma separated decimal byte numbers (all on one line) and it should be in bazel-genfiles/stdlib/std.jsonnet.h

[sparkprime]

Did this get resolved? If not I'm inclined to close it as we have no other reports like it. Thanks

[sbarzowski]

It worked for me and @nnmrts in the end, so it can be closed IMO.