Esy: Windows Support: 'test-e2e' blockers

[bryphe]

I've been working through some of the blockers for getting our test-e2e tests passing / green on Windows. After #270 is fixed, we'll be up to 5 tests green on Windows, which is an improvement 😄

However, the next class of blockers is a more fundamental issue - it's the fact that we tend to verify our tests by dropping a bash script and then running it, ie:

    "build": [
      [
        "bash",
        "-c",
        "echo '#!/bin/bash\necho #{self.name}' > #{self.target_dir / self.name}"
      ],
      "chmod +x $cur__target_dir/$cur__name"
    ],
    "install": [
      "cp $cur__target_dir/$cur__name $cur__bin/$cur__name"
    ]

And then the test validates that script outputs successfully, like esy x no-deps should print out no-deps.

The problem, though, is that __native Windows does not support running bash scripts__. So if we try and run esy x no-deps on windows, we'd get:

esy: error, exiting...
     Exec format error

The exec format error is Windows saying it doesn't know how to execute it - which makes sense, as it has no support for running a bash file.

So what we really need is a way to have a cross-platform way to specify our build scripts (ideally). I can see a few options:

__1)__ Simply have a different suite of test cases for Windows and Linux/Mac systems. On Windows, we'd have a similiar test, but it would look something like this:

    "build": [
      [
        "bash",
        "-c",
        "echo 'echo #{self.name}' > #{self.target_dir / self.name}.bat"
      ],
      "chmod +x $cur__target_dir/$cur__name" <-- this isn't really needed on Windows
    ],
    "install": [
      "cp $cur__target_dir/$cur__name.bat $cur__bin/$cur__name.bat"
    ]

And then run esy x no-deps.bat

__2)__ Use the same suite of test cases, but use our conditional expressions to pick between creating a bash / bat file. This means we have a 'common' set of test cases, but each test gets significantly more complex to handle either running on Windows or Linux/Mac. I'm not sure if this is really feasible, but perhaps there is a reasonable approach to this that someone knows.

__3)__ Use a common executable format. One way to accomplish this would be to bring in ocaml and compile a quick script, like echo print_endline("#{self.name}")' > test.ml and then use ocamlopt to build an executable. On each platform, we'd get a native executable we could validate. The nice thing about this is we could use a consistent test suite with all platforms - the downside is that ocaml takes a long time to build for people running the tests!

(cc @ulrikstrid - I know you've been doing a lot of work on the e2e-tests! 😄 )

[bryphe]

A fourth option is - perhaps we should run all x commands in our esy-bash wrapper? Right now,we only use that for the esy-build-package, but perhaps it makes sense for us to use that for the x commands, too.

[ulrikstrid]

Maybe we can vendor one OCaml that can be used for all e2e tests by copying in the init. This would also speed up the release test significantly.

[andreypopp]

I think the approach with ocaml is the way to go. Right now release test case already builds one and it ups tests runtime from 4min to 8min (which is bearable for integration tests, I think).

What we need to do, I think, is to move ocaml build out of test cases to a setup phase and build it only once for all test cases. We have two options:

1. As all test cases have isolated build stores inside /tmp then we can relocate ocaml to it using esy export-build / import-build commands. The downside is that import is still not free and takes some time for such large builds as ocaml.
2. Make all test cases using single temp store so we can build ocaml once into it. That lowers isolation between stores but I don't think having isolated stores is so critical.

Thoughts?

[bryphe]

Cool, I like the idea of using OCaml the best out of all the options! One other benefit of using the ocaml compiler directly in the e2e tests is that I can write some tests that directly exercise some of the windows blockers - like #242, #273 - would be great to have those exercised and validated on AppVeyor.

As all test cases have isolated build stores inside /tmp then we can relocate ocaml to it using esy export-build / import-build commands.

I actually hadn't used these commands yet... it seems like it could save me a lot of time locally (I soak up a ton of time building OCaml locally).

I'd be OK with option 2 for now, as it is simple, and if we found that we required the store isolation, we could always switch to option 1.

[ulrikstrid]

If this haven't started when I'm done with the node conversion I could do this. But I would need some pointers for how to start it.