Fsharp: Repo alignment - How to enable xplat workflow for easy contribution

[7sharp9]

@dsyme says:

We would like to make it possible to contribute to core components that underlie the OSX and Linux experiences for F# by contributing to this repo (rather than the historical way of routing these requests into the fsharp/fsharp and fsharp/FSharp.Compiler.Service repos. We can call this "repo alignment".

Separately, people have discussed splitting this repo - putting the "Visual F# IDE Tools" (vsintegration and setup directories) in a separate repo, and perhaps putting "FSharp.Core" in a separate directory too. For now we will assume that this will be done eventually

Here's a list of things we can reasonably unify:

• [x] Mono friendly fixes and hacks from fsharp/fsharp
• [ ] Mono friendly testing from fsharp/fsharp (or replacement)
• [x] "make install" or equivalent from fsharp/fsharp
• [x] local builds of nuget packages for FSharp.Core nuget (from fsharp/fsharp)
• [ ] local builds of nuget packages for FSharp.Compiler.Tools nuget (from fsharp/fsharp)
• [x] code additions in Fsharp.Compiler.Service (notably some fixes and Exprs.fs) (from fsharp/FSharp.Compiler.Service)
• [x] local buidls of FSharp.Compiler.Service DLL (from fsharp/FSharp.Compiler.Service)
• [x] some extra testing (from fsharp/FSharp.Compiler.Service)
• [x] local builds of nuget packages for FSharp.Compiler.Service
• [ ] type provider authoring SDK from type provider starter pack
• [ ] compiling FSharp.Compiler.Service as a Fable component, a highly valuable long term addition to the F# ecosystem that we don't want to bit-rot
• [x] make sure documentation in this repo is not skewed towards Visual Studio and Windows. There should be a strong path where neither Visual Studio nor Windows are assumed or even mentioned. OSX and Linux contributors should feel "first class"
• [x] rename this repo to "fsharp" (even if it continues to hold the setup and vsintegrationdirectories). fsharp/fsharp could then be a mirror of this repo
• [x] Move tests\service* tests into FSharp.Compiler.Unittests.dll instead of VS-only tests since they don't need any VS bits to run
• [ ] enable running FSharp.Compiler.Unittests.dll on Mono under Jenkins
• [ ] enable running FSharp.Tests.FSharpSuite.DrivingCoreCLR on Mono under Jenkins
• [x] make sure the documentation on this repo is appropriate for FSharp.Compiler.Service and fsc.exe contributors on Mono/Linux/macOS
• [x] make sure the testing in this repo is appropriate for contributors on Mono/Linux/macOS
• [x] remove or minimize the use of CMD scripting such as build.cmd

Various problem with this plan:

• This repo becomes a logjam containing everything related to both F# core engineering and the Visual F# IDE Tools.
• The Windows-specific nature ("feel") of this repo will still hang around while it continues to contain the Windows-specific Visual F# IDE Tools
• The "make install" business currently uses a bunch of makefiles, and adds things like policy files for Mono's unification of FSharp.Core. It's not at all clear we want to pollute this repo with that
• Supporting the Fable bootstrap of Fsharp.Compiler.Service adds a very new and different dimension to this repo that may blow the minds of contributors, and leave them unable to debug issues with Fable.
• Do the new binaries and packages get signed, and if so using which strong names

Aside: this does not mean that Microsoft Visual F# Tools team become publishers of any of the alternative packagings of F# - and indeed the final "tagging" and "publishing" of these components may still be done in other repos - see this description for the actual tagging and release repos, including those used by Xamarin. This is normal. But contributions would flow into this repo.

---

@7sharp9 writes:

As a starting point I want to help with the type provider type input testing and implementation, as well as many other features but doing so means porting code to and from FCS and also building and installing the compiler into osx/mono.

• I can now build and run from this repo with the resource issue: #720 I think
• To get tooling support I will need to build visualfsharp and use it in place of FCS in ionide/xs/
• I also need to be able to install the compiler with a script, Im assuming theres one described in the readme somewhere.

So I would need two build target outputs for this repo, one for an FCS compatible build, and one for a compiler build, and a script to install into mono.

[vasily-kirichenko]

I'm +1000 for this. Enabling @7sharp9 to contribute worth any effort needed to make FSharp.LanguageService.Compiler usable from Ionide/XS (this means to make the API optionally public I guess?)

[cartermp]

Agreed, this is important.

[7sharp9]

At the moment Im trying to figure out how to replace the compiler installed in mono with the one just built in release, I cant see a script for that in this repo.

[7sharp9]

I added some notes here on trying to get that fork running the TP sample: https://github.com/colinbull/visualfsharp/commit/c295f8a264efeef1faed305cebbd2dbfb666ce8c

[dsyme]

@7sharp9 @vasily-kirichenko @KevinRansom @cartermp @brettfo I've added a lengthy description of all the work I know of to be done in the area of "repo unification" up above in the issue description, plus some additional comments

@KevinRansom @cartermp @brettfo You guys are the repo owners - if you don't want these things coming in to this repo, then you should say :)

[KevinRansom]

We are eager for the repos to be aligned. And are willing to make the necessary changes to make that happen. So long as our product pipeline is not broken by any changes we are good.

[cartermp]

Agreed with what @KevinRansom said. Let's do it.

[dsyme]

@KevinRansom Thanks, that's great to know.

p.s. That said, I fear you may have a heart attack (joke!! I hope not!! :) ) when you see a PR proposing to integrate the machine-wide policy files for FSharp.Core installed as part of Mono... The Mono packagings of F# have always used those to unify FSharp.Core to the latest version - for better or worse.... And I assume there is code that depends on that. We can perhaps reassess that, but probably best not to do that as part of unification.

So I suggest we put those under an explicit mono directory :) And in general aim to keep each of the above items fairly separable.

[dsyme]

@vasily-kirichenko

...make FSharp.LanguageService.Compiler usable from Ionide/XS ...

Let's aim instead for building FSharp.Compiler.Service in this repo.

For now let's assume FSharp.LanguageService.Compiler remains a special version of this component used only by the Visual F# Tools. This is because historically there was huge pressure from Visual Studio engineering to make FSharp.LanguageService.Compiler smaller - trimming out unnecessary parts. This is one reason I didn't integrate Exprs.fs - which is unused by FLSC. Size minimization was to reduce overall address space usage in the 32-bit devenv.exe Visual Studio process. It's possible this issue will come back to haunt us again one day, so for now just assume that the Visual F# Tools need their own special build of this component.

[smoothdeveloper]

Disclaimer: not directly related to mono support in this repository.

Can we move the current scripts under scripts/misc and plan for a comprehensive layout for future build scripts under scripts/build?

My feeling is that we should start splitting the behemoth build.cmd into smaller independant scripts, each put in scripts/build, and have the top level build.cmd handling the main parts (setting env variables for the build conditionals, calling other .cmd, and doing the main build).

This way when we start introducing FAKE script, we can focus on a smaller set of behaviour to replicate.

individual scripts I could see:

• dotnet core stuff
• vssdk stuff
• ngen stuff
• testing stuff

I know this will make it a bit harder (but how much harder? not so much) to deal with small changes, but I don't see how the community can put effort in a big bang PR to change the whole thing at once, that will need long lead time for you guys to review / assess, while changing each small script or the top level one, one by one is definitely doable.

[dsyme]

@smoothdeveloper Let's keep this thread on topic please, about repo unification

[dsyme]

@7sharp9 I've updated the issue description with a list of all the problems that will arise because of repo unification.

Specifically, if we do this unification we should go into it with our eyes open: it's important that you be aware of the inevitable problems listed above (e.g. the mess that is the Windows-specific build.cmd, or the fact that the Visual F# IDE Tools are part of this repo with all that brings in biasing it towards Windows...) . We can fix these things, but it will require work.

The point is - don't assume unification is some kind of panacea. It will bring new problems of a different kind.

[KevinRansom]

Mostly we need to be able to do mono build and testing as part of normal dev work.

[smoothdeveloper]

@dsyme, sorry but I think it is relevant.

For example people using windows (hence build.cmd) might also want to loopback on getting FCS built from this repository, hence I think we should make the effort of making any addition to build scripts (which are not straight ports from fsharp/fsharp or FCS repositories at least) in smaller scripts, and possibly .fsx already if we don't want another round of .sh / .cmd duplication.

Am I making any sense?

[dsyme]

@smoothdeveloper Yes, it makes sense. But whether the scripts are large or small, how they are named, which directories they go in etc. is a detail we will sort out along the way.

[dsyme]

Mostly we need to be able to do mono build and testing as part of normal dev work.

Yes, that's crucial.

@7sharp9's criteria is to be able to build and dev the core components that flow into Xamarin Studio (Visual Studio for Mac) using this repo (rather than making contributions here and then flowing them into the FCS repo). And in general to have a first class OSX contribution story for this repo.

BTW I think I need to get myself a Mac.... My kids destroyed the last one :)

[vasily-kirichenko]

so for now just assume that the Visual F# Tools need their own special build of this component.

It makes contributing by non-windows people, like @7sharp9 much more harder. There're a very few of us who contribute to FCS and cutting off #2 in this list is just unforgivable.

[dsyme]

It makes contributing by non-windows people, like @7sharp9 much more harder. There're a very few of us who contribute to FCS and cutting off #2 in this list is just unforgivable.

I thiink I was misunderstood. I was meaning adding something like this in this repo:

build fcs   -- builds the FSharp.Compiler.Service.dll
build fcs test  -- builds and tests the FSharp.Compiler.Service 
build fcs nuget -- builds the FSharp.Compiler.Service and makes the nuget

This:

build vs   -- builds the FSharp.LanguageService.Compiler and other VS stuff

would build the separate FSharp.LanguageService.Compiler DLL from the same code for use by VS, obeying whatever extra requirements Microsoft demand (make stuff internal, minimize size or whatever)

So on OSX Dave's workflow would simply be repeated:

1. develop
2. build.sh fcs nuget
3. Build new F# tools for Xamarin (or Visual Studio for Mac or Ionide or whatever)

and a feed of FSharp.Compiler.Service nuget would be available from AppVeyor CI of this repo

[vasily-kirichenko]

Ah. I see. That's great. But we will end up building almost all the sources three times (FSharp.Compiler, FSharp.LanguageService.Compiler and FCS). However, if we, who use VS, won't have to build FCS in our workflow (not having it in VisualFSharp solution, basically), that's OK.

[7sharp9]

Why does there have to be 3 compilers built, is FSharp.LanguageService.Compiler the Windows / VS equivalent of FCS?

[7sharp9]

The non windows workflow would also need a compiler built and installed as well as FCS, unless there is a way of unifying both of those, or specifically decoupling the FCS parts as a facade over the compiler.

[smoothdeveloper]

Is FSharp.LanguageService.Compiler the Windows / VS equivalent of FCS?

yes

Why does there have to be 3 compilers built

• fsharp.compiler (referenced by fsc and fsi)
• fsharp.languageservice.compiler for VS, it might trim down stuff with ifdef from FCS (due to memory constraint in devenv.exe)
• fsharp.compiler.service

We don't need the latest one when working in VS (as @vasily-kirichenko says) and indeed it would help if the second one was just adding to the compiler itself instead of compiling again (I wonder if the reason is also related to internals?).

[7sharp9]

@dsyme I added this to the top too:

If this is the defacto fsharp compiler then a tag a versioned build need to be integrated into mono

[dsyme]

Why does there have to be 3 compilers built

We can maybe reduce it to two or one - perhaps almost immediately after unification or even during it. It depends in part if Microsoft continue to want internal symbols only in the VS bits they ship. If they are ok with dropping this requirement - and are ok with slightly larger overall binary size - then I reckon we can just use one FSharp.Compiler.Service.dll and nothing else.

[KevinRansom]

Or ...

We can move vsintegration into a separate repo. If Iwe were to create a new repo … github.com\microsoftfsharp we could move the cross platform compiler there. Limit visualfsharptools to the vsintegration and windows packaging that might make a useful factoring.

There would be internal work we have to do to support that … but perhaps it now makes sense and will at least simplify some of the build and test complexity.

Kevin

[7sharp9]

I think we all realise that any major change to code organisation is not going to happen in a reasonable time.

To me the first few steps could be:

• Get the necessary extra files in here from FCS and #if deffing them so a build could be generated for FCS. This would satisfy retiring FCS quite quickly.

• The next part is getting an install script for mono, I must admit I find it hard to follow the install points in the make file in fsharp/fsharp

• I noticed in the samples that the built compiler is used rather than the installed one, this is very handy rather than having to replace the compiler, unfortunately it doesn't work on osx yet. See: https://github.com/colinbull/visualfsharp/commit/c295f8a264efeef1faed305cebbd2dbfb666ce8c#commitcomment-21764715

[smoothdeveloper]

We might want to stick to one repository for now, and be very conscious about the build scripts (split), the single code base for all is the faster turnaround, given we put attention for each step to be optimized (don't rebuild stuff, don't build 3 compilers, etc.).

But well, I'm just a random peon as they say, you guys are the one doing real work...

[7sharp9]

So Im going to take a crack at merging some bit of FCS over here and see where the #ifs go.

#if FCS
...

?

[smoothdeveloper]

@7sharp9 I'd go with bit more verbose: #if FSHARP_COMPILER_SERVICE.

[cloudRoutine]

originally wrote this here

a suggestion on how we could make the alignment easier in the future - most places where #if COMPILER is used seem like types that it would be more useful to have the compiler service expose. So if we got rid of COMPILER as one of the defines for FCS add || COMPILER_SERVICE to the 10 or so places where having #if COMPILER defined was actually useful and change all of the

type (* internal *) foo = ...

to

#if COMPILER
type internal foo 
#else
type foo
#endif 
    (args ) =

it should cut out a bit of merge conflict resolution busywork

---

the COMPILER_SERVICE directive is already being used in a few places, so it makes sense to stick with that.

[dsyme]

@KevinRansom @Pilchie @brettfo Historically Microsoft-built DLLs that are private to Visual Studio have only had internal accesiblity on their APIs. That's one reason that forces us to have two separate DLLs - one for FSharp.Compiler.Service (public API) and one for FSharp.LanguageService.Compiler.dll (internal API, shipped with VS).

Do you know if that needs to continue to be the case? Adding the #if is a PITA. Could we just have MIcrosoft ship a private DLL with "public" accessibility?

[dsyme]

@7sharp9 I started some work towards supporting an installation process for Mono builds here https://github.com/fsharp/fsharp/pull/730/

It's hard to avoid just bringing the basic Makefile across, since we know it actually works on various linux distros, homebrew etc. But I'll try to avoid bringing across all the files, and put everything in a "mono" directory

[dsyme]

@7sharp9 @vasily-kirichenko The first of the many work items here has been completed - we now have a make install path for Mono users. Please try it out when you get the chance.

I will now work on build fcs for building the FSharp.Compiler.Service DLL in this repo.

[dsyme]

@7sharp9 The FSharp.Compiler.Service parts of this work are underway here: https://github.com/Microsoft/visualfsharp/pull/2951

[dsyme]

@cartermp Note that an unfortunate side-effect of bringing FCS development into this repo would be bringing in 114 issues from that repo (though we could continue to track them in the other repo for a while while we trimmed them down). I'll try to close out many issues there beforehand.

That's one reason I've been going through the VFT issue list today/yesterday - to trim out some old issues and make a bit of room :)

[cartermp]

@dsyme All good to me. More than happy to triage them so we can really know where we're at.

[7sharp9]

Ive closed some of mine that are no longer relevant or Ive forgotten
context, I think Im one of the biggest culprits :-)

On 29 April 2017 at 17:26, Phillip Carter [e-mail] wrote:

@dsyme https://github.com/dsyme All good to me. More than happy to
triage them so we can really know where we're at.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/visualfsharp/issues/2854#issuecomment-298178859,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAj7ylv34xQe8S2Qqixl7x6Lh61e4EWNks5r02S8gaJpZM4M9A3D
.

[dsyme]

@7sharp9 @vasily-kirichenko Thank you so much for closing the swathes of old bugs, it's really important to keep things ship-shape

[dsyme]

Just to mention that I'm working on a nasty PR to bring the FCS changes back into Microsoft/visualfsharp, see https://github.com/Microsoft/visualfsharp/pull/2951

There are a bunch of complications with this - there are subtle differences between FSharp.Compiler.Service.dll (public API), FSharp.Compiler.dll (internal to fsc.exe and fsi.exe) and FSharp.LanguageService.Compiler.dll (internal to VS, some things trimmed out) but it's getting there.

[7sharp9]

Just checking in / resurrecting this thread slightly.

Im just reading the build build on OS X and was wondering if there is a way of building without using mono, e.g. with only a netstandard installation.

A lot of people are getting interested in F# because of x-platform stability from netstandard and are probably not as enthralled by having to install "mono". I know its possible to build FCS using only net standard so this should also be possible here.

[brettfo]

I'm currently working on converting our projects over to use the new Microsoft.NET.Sdk which means you'd be able to dotnet build FSharp.Core.fsproj, etc., but due to a complicated mix of our projects, I'm first converting the Visual Studio specific projects in #4317. While that work doesn't directly make building with only a dotnet CLI install possible, it's a step I have to take before I can re-look at converting everything under src\fsharp to the new SDK project format.

[dsyme]

A lot of people are getting interested in F# because of x-platform stability from netstandard and are probably not as enthralled by having to install "mono". I know its possible to build FCS using only net standard so this should also be possible here.

Right now .NET Framework/Mono is still required to do any major testing, tooling work etc. in this repo

BuildFromSource should work without .NET Framework/Mono. But there's no testing.

If it helps Mono has been rock solid for our purposes in this repo lately. I haven't reported a bugzilla bug in about a year I think, and the only issues we've had have been xbiuld --> msbuild (which is about convergence) And performance is good on my Mac.

[cartermp]

From #3015

@dsyme suggestions renaming:

FSharp.Editor --> VisualFSharp.Editor
FSharp.LanguageService --> VisualFSharp.LanguageService

[7sharp9]

Where are we with this, are we near to killing fsharp/fsharp?

[cartermp]

This has progressed a lot, though not precisely as the checklist implies:

• [x] Source build is now compliant with .NET Core source build
• [x] Repo is 100% migrated to the .NET SDK with individual pieces buildable piecemeal
• [x] Building itself is now simplified and can be tweaked based on the solution you want to build against and CI names now make sense
• [x] README is now simplified and there is a quickstart for newcomers

Additionally:

• #6549 outlines simplifying the name and dropping the "visual" part of this
• FST-1008 tracks FCS being fully built and published from this repository (after it's migrated to dotnet/fsharp)

I'm going to write another RFC for archiving fsharp/fsharp, though I do see that as the last effort in this process.

[cartermp]

I think we can reasonably close this out. FCS is now incorporated into the root of the repo and builds as a package, with release notes and docs underway. All the mono stuff is now irrelevant since we _yote_ mono out of this repo and are unifying on the .NET SDK. The only remaining piece is emitting Fable stuff, but frankly I think Fable has been getting along fine enough without us ruining their day. Not that there aren't issues, but we've had multiple folks contribute compiler features without ever using Windows, so I consider that a success.