Esy: Windows Master Issue

[jordwalke]

Let's maintain a list of all the things we need to change in order to get windows esy support off the ground. Let's also track the things we need in order to get an entirely pure windows install setup:

Goal

• esy supports windows out of the box without having to install anything but esy
• One click install and build of any opam dependencies (so long as those dependencies don't contain unixisms):
• Pure native builds of binaries/executables. (Linking performed with mingw).
• Ability to compile C stubs with bundled gcc.
• Stretch goal: Ability to optionally perform linking/compiling-c-stubs with msvc.
• Big Stretch goal: Ability to build opam packages that have unixisms in them.

Blockers To Preliminary Support:

• Port esy release wrappers to something cross platform.
• Get self contained npm install -g esy package that includes everything esy install needs on windows to fetch packages and pull from git etc.
• Perform esy build steps for packages some prebuilt/prebundled windows gcc native compiler. (so long as those packages don't have unixisms).

Completely Eliminating Cygwin:
As a temporary solution, we might embed an unattended cygwin install to get some helpful utilities like curl/rsync etc. This is the list of things we need to do to completely eliminate any use of cygwin:

• Cross plat replacement for curl
• Cross plat replacement for rsync
• Cross plat replacement for git
• Cross plat replacement for tar
• Cross plat replacement for patch

Notes
For some of the operations that create or import npm releases for packages, it may be suitable to rely on node to do some of the releases, as long as the startup time is not important: Justification:

• If you are creating/consuming an npm release, we know the user already has npm installed.
• If it's not performance critical operation then the node startup time isn't as big of a deal.

[bryphe]

Current Plan

__DONE__ Part 1: esy running on cygwin

• [x] Bootstrap esy install - can a separate, standalone version of esy-install suffice?
• [x] Bootstrap esy build - can we use a bytecode version of esyBuildPackage on a specific plan?

__Goals: __

• [x] esy commands should all work on Cygwin. Test with a canonical set of packages.

Part 2: esy native executables running on Windows (w/ cygwin behind the scenes)

• [x] Add silent, unattended install of cygwin with relevant packages. Good reference here and here

esy install

• [x] esy-install.js - Incorporate esy-cygwin to enable tar -xzf call on Windows
• [x] esy-install.js - Incorporate esy-cygwin to enable patch call on Windows

__Goals:__

• [x] esy install works on Windows

__OPEN ISSUE__: Will we need to point the OPAM path to the windows fork? Will come out in esy build if we have failures that are addressed in the Windows OPAM repo.

esy build:

• [x] Fix ShellParamExpansion escape issue - #232
• [x] Task.ml - fix path character there - #232
• [x] Fix rsync invocation when relocating sources - #233
• [x] __PATH LIMIT:__ Windows has a 260 character limit on paths, hit by jbuilder (large nested path while building). For now, working around by setting padding low. #238
• [x] esy-ocaml/ocaml: OCAMLLIB & OCAML_TOPLEVEL_PATH need to point to $ocaml_install/lib instead of $ocaml_install/lib/ocaml - #226 , esy-ocaml/ocaml#3
• [x] esy-ocaml/ocaml: CAML_LD_LIBRARY_PATH needs ; separators instead of : - #226 , esy-ocaml/ocaml#3
• [ ] Implement Windows sandbox strategy
  
  
  
  • [ ] Allow passing sandbox file, like in the Mac strategy - requires update to esy-bash. (In progress - #239 )
  
  
• [ ] Windows strategy for resolveCommand #247
• [x] Fix esy-bash argument escaping (failing esy-installer step) bryphe/esy-bash#5
• [x] ocaml configuration steps are specific to Linux + Mac. Need to investigate porting over setup strategy from fdopen's repo to esy-ocaml/ocaml. esy-ocaml/ocaml#1

CI Support:

• [x] Initial AppVeyor seutp
• [x] Initial environment setup (Cygwin)
• [ ] esy-solve-cudf fails on postinstall: andreypopp/esy-solve-cudf#4
• [x] Validate esy install for esy-reason-project
• [x] Validate esy build for esy-reason-project
• [ ] Incorporate existing test suite for Windows. Will we need to port our bash tests? Once this is addressed, it will give us visibility into future work.
  
  
  
  • [ ] e2e #241
  
  
  • [ ] opam
  
  

NPM Publish:

• [x] __BLOCKER:__ Need a cross-platform plan for publish: esy/esy#220

__Goals:__

• [x] npm install -g @bryphe/esy-windows should work on Windows to provide esy build and esy install

Part 3: Publish to npm

• [x] Set up platform-release equivalent target for Windows, to create architecture specific bundles (x86, x64)
• [x] Update our release bundler to incorporate windows dependencies

Part 4: esyi running instead of JS-based strategy

• [x] esyi - fix yaml dependency - #214
• [x] andreypopp/esy-solve-cudf#1 - postinstall fails on cygwin / windows
• [ ] andreypopp/esy-solve-cudf#3 - mccs fails to install on cygwin / windows # #
• [ ] Set up cygwin CI on esy-solve-cudf, publish cygwin-friendly NPM package

My windows-specific investigation branch is here: https://github.com/bryphe/esy/commits/windows-bootstrap

Part 5: Additional esy command support

Still unknowns here - need testing to evaluate blockers.

__Sandbox introspection__

• [ ] esy ls-libs
• [x] esy ls-builds
• [x] esy build-env
• [x] esy command-env

__Command execution:__

• [x] esy build <anycommand>
• [x] esy x <any command>

__Environment-specific shells:__
__TBD:__ Assuming build environment shell would open in cygwin-bash on Windows, for now?

• [ ] esy build-shell <path-to-package>
• [ ] esy shell

__Other:__

• [ ] esy add
• [ ] esy release
• [ ] esy config
• [ ] esy import-opam
• [ ] esy install-cache
• [ ] esy export-dependencies
• [ ] esy export-build - #286
• [ ] esy import-build

Longer term:

• [ ] Common build strategy cross-platform - we can leverage the Makefile with esy-bash once there is an 'npm-installable'esy`
• [ ] Revisit 'slash normalization' (\ -> /) when we have test coverage
• [ ] Investigate if we can make initial Cygwin install more silent (no popup at all)
• [ ] Investigate removing bash dependency and just calling the Cygwin bundled tools directly.

[jordwalke]

Can this be updated in any way with recent findings? I'm thinking an emphasis on esyi instead of esy install, maybe something else? I'm catching up now.

[bryphe]

@jordwalke - sure thing! I just updated it. @andreypopp was able to unblock me on esy-install.js - so I went ahead with that for now to investigate esy build. I'll revisit esyi shortly - my hope is I can get a set of tests working on AppVeyor with esy install (via esy-install.js) andesy build... would be great to have a test story in place when we swap inesyi`!

My hope is that I can have esy install and esy build working from a Windows command prompt by EOW for a simple project, and then we can start porting over PRs to esy. One issue we'll have to consider as we start porting in the Windows-specific fixes esy/esy#220 - how we want to handle the cross-platform artifacts.

In addition, if we can leverage the existing automated tests on AppVeyor, that will help highlight additional commands that will need to be addressed for Windows.

[bryphe]

esy build goals / status for windows

| Project | Status | Comments |
| -- | -- | -- |
| esy-minimal-ocamlbuild-project | WORKS * | esy-ocaml/ocaml#6aacc055 |
| esy-minimal-jbuilder-project | WORKS *| esy-ocaml/ocaml#6aacc055 |
| esy-minimal-merlin-project | WORKS *| esy-ocaml/ocaml#6aacc055 |
| esy-reason-project | BLOCKED | Blocked by #295, requires esy-ocaml/ocaml#6aacc055 |
| esy-solve-cudf | BLOCKED | Blocked by #302 |
| esy | BLOCKED | Blocked by #295 |
| fastpack | *??| ?? |
| graphql_ppx | *?? | ?? |

[bryphe]

We've made great progress on the Windows front! esy can be installed via npm on Windows, and can build projects like hello-reason and hello-ocaml that already have a lockfile.

I created a new issue to track the next steps for Windows support: #428