Fsharp: Move sigdata and optdata into FSharp.Core

[praeclarum]

Currently, FSharp.Core's sigdata and optdata are not stored in FSharp.Core. Instead they are put in parallel files to core.

I would like to suggest moving those files into FSharp.Core.dll.

Past Rationale

DLL size on small profile devices was the only justification. But old compilers require them separate
@dsyme

Current file sizes on iOS are:

1526272 Dec  1 08:07 FSharp.Core.dll
 326778 Dec  1 08:07 FSharp.Core.dll.mdb
 429831 Dec  1 08:07 FSharp.Core.optdata
 556780 Dec  1 08:07 FSharp.Core.sigdata
 704019 Dec  1 08:07 FSharp.Core.xml

That puts this data around 1 MB in size.

Reason for Change

To make using FSharp.Core more reliable.

Currently we have to

1. Remember to bring these two extra files along with the assembly
2. Work to make sure we don't use the wrong metadata

I don't believe the perceived benefit of smaller mobile apps outweighs the mistakes made when using FSharp.Core.

Thank you for considering this!

[KevinRansom]

I have considered this very issue several times. It’s not actually hard to do and has a benefit, that we wouldn’t really need to search the filesystem for them, just pull them out of the fsharp.core being used to compile.

However, FSharp.Core.dll serves a dual purpose:

1. Is it serves as a source for metadata describing the core library for the use of the compiler.
2. It is the runtime dll containing the types used by dependent assemblies.

The .sigdata and .optdata files are compile time only files and provide no value in run time environments, therefore, embedding them in FSharp.Core.dll would make the size of the deployment bigger by approximately 1 MB. Since the release build FSHarp.Core.dll is only approx. 1.5 MB, you can see that is quite a deployment penalty.

The more interesting question though is why it would make F# more reliable … in a correct FSharp compiler deployment, .sigdata and .optdata should be deployed alongside the FSharp.Core.dll to be used for compilation purposes. If not then it is a deployment bug that needs fixing.

Now …. In reality … the F# compiler is often uses with hand-rolled, hand tweaked builds, where the build relies on an unfortunate compiler behavior of searching around in various well known locales, including in the past the GAC which I believe we have finally got rid of the other well know locales are still around and will need to be elided when we get a decent design for that.

I hope this clarifies for you why they are not embedded.

Kevin
From: Frank A. Krueger [mailto:[email protected]]
Sent: Tuesday, December 6, 2016 12:31 PM
To: Microsoft/visualfsharp [e-mail]
Subject: [Microsoft/visualfsharp] Move sigdata and optdata into FSharp.Core (#1953)

Currently, FSharp.Core's sigdata and optdata are not stored in FSharp.Core. Instead they are put in parallel files to core.

I would like to suggest moving those files into FSharp.Core.dll.

Past Rationale

DLL size on small profile devices was the only justification. But old compilers require them separate
@dsymehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fdsyme%2Fstatus%2F806229188818698240&data=02%7C01%7CKevin.Ransom%40microsoft.com%7C8aa4bca129fc4c58925708d41e16cc2a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636166530641521622&sdata=HLw5uuiyJdJDkenqQZuCU5gfYr9qOH0vLuExmFq2FbQ%3D&reserved=0

Current file sizes on iOS are:

1526272 Dec 1 08:07 FSharp.Core.dll

326778 Dec 1 08:07 FSharp.Core.dll.mdb

429831 Dec 1 08:07 FSharp.Core.optdata

556780 Dec 1 08:07 FSharp.Core.sigdata

704019 Dec 1 08:07 FSharp.Core.xml

That puts this data around 1 MB in size.

Reason for Change

To make using FSharp.Core more reliable.

Currently we have to

1. Remember to bring these two extra files along with the assembly
2. Work to make sure we don't use the wrong metadata

I don't believe the perceived benefit of smaller mobile apps outweighs the mistakes made when using FSharp.Core.

Thank you for considering this!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fvisualfsharp%2Fissues%2F1953&data=02%7C01%7CKevin.Ransom%40microsoft.com%7C8aa4bca129fc4c58925708d41e16cc2a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636166530641521622&sdata=cVUGGp9bGg%2Frdhls0tCMo7zo0kjFdx%2B3RWpMAJnj%2Fls%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAE76FjNYq1_mtuv2By6b6lNMRtE7rNX9ks5rFcYGgaJpZM4LF2HW&data=02%7C01%7CKevin.Ransom%40microsoft.com%7C8aa4bca129fc4c58925708d41e16cc2a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636166530641521622&sdata=GfkK5bJqx3FGIt%2BNtmgJS1kn9ld%2FFr7lwqnCvC9pi5U%3D&reserved=0.

[ReedCopsey]

The .sigdata and .optdata files are compile time only files and provide no value in run time environments,

@KevinRansom That's not necessarily true. They aren't only used at compile time, especially in some environments.

These files also get used by FSI, which has also lead to many deployment issues when using FSI (along with issues for FAKE and other tools, etc)

For example, having this separate is problematic for Azure Functions (see https://github.com/Azure/azure-webjobs-sdk-script/issues/970 ), causes issues for things using the compiler services, has caused issues for the Intellifactory guys ( https://bitbucket.org/IntelliFactory/cloudsharper/issues/52/fsharpcoresigdata-and-optdata-not-found-on ), and others I can't recall off the top of my head.

[forki]

Yes I think it's mostly working in classic fsc / VS scenario. But in other
compiler as a service scenarios it can be a problem

[KevinRansom]

When are they used in runtime environments Reed?

[7sharp9]

Ive heard this discussion a few times and the justification for splitting them out doesn't seem to make sense unless you are using a 33k modem, it only makes for a fragmented deployment.

[KevinRansom]

@7sharp9. I expect that will eventually move over to using reference assemblies for compilation similarly to C# and the BCL. And so deploying the assembly used for compilation would result in a non-operational program.

[rojepp]

@KevinRansom What are those files used for?

[7sharp9]

Im still none the wiser on why they have to be external.

[KevinRansom]

@rojepp.

I believe:
.optdata contains data that improves cross module inlining optimization scenarios and
.sigdata contains F# compiler specific metadata, descriptions of the types and functions that is lost by the compilation into dotnet assemblies.

@7sharp9:
I guess deploying and loading into memory a 2.5 MB FSharp.Core.dll rather than a 1.5 MB FSharp.Core.dll is consider a waste of time and memory, when they are only useful in compilation scenarios. But they don't have to be external ... they just are .... for the reasons already enumerated.

[ReedCopsey]

When are they used in runtime environments Reed?

@KevinRansom Yes. Anything doing "compiler as a service" scenarios often need to have those files, since they're effectively doing compilation at runtime.

There are a surprisingly large number of places where that becomes an issue, given some of the very cool (though unusual) ways F# gets used in the wild. Simple google and StackOverflow searches will turn up lots of people running into issues because they're separate.

While I get _why_ they were separated out, the rationale seems to be based on decisions that aren't nearly as relevant today. The extra 1mb of deployment is a much smaller issue (even though it has a broader reach in terms of users) than the pain that arises when not having those is a problem, and that pain does actually hit a significant number of users (even if it's a small minority), which is why I'd vote to bundle them together.

[KevinRansom]

I’m thinking that the “Compiler” in that indicates that it is a compilation scenario there Reed? And really and truly the compiler is also doing a compilation at runtime …

The point is that when referencing fsharp.core.dll … just referencing whatever fsharp.core.dll the compiler is able to find might not be the most deterministic compilation environment. If you deploy fsharp.core.dll for compilation purposes deploy sigdata and optdata.

If your compilation fails with the message that it can’t load those files then check that you have told your compiler … or service where to find the fsharp.core.dll that you want to use.

Now an argument that may make sense to me is for the compiler service to not fail if it can’t find those, and turn off the advanced scenarios.

That way it is pay for play … you lose cross module in-lining if you don’t deploy the right files, but you can use any old fsharp.core.dll because you didn’t really care.

Oh the discussions are relevant today … you should see the hoops we jump through to reduce working set in the VS IDE. And regardless if we move to AOT compilation then there won’t even be an fsharp.core.dll deployed with the app.

Kevin

[ReedCopsey]

@KevinRansom I guess I'd flip this around and word this another way - Why should FSharp.Core be any different than _every other F# library_, where this is normally embedded?

[rojepp]

@ReedCopsey Especially given that the assumption 'Only used by the compiler' clearly doesn't hold. We've all seen this from time to time, when we'd rather focus on the business logic. +1 for embedding these.

[7sharp9]

It's normally in the assembly resource data

[0x53A]

This would obviously not work with existing compilers because they couldn't read it, but what about compressing the optdata/sigdata before embedding it?

The combined uncompressed filesize is ~960KB. (I tested embedding it, and it seems .net assemblies don't compress resources, an empty console app with both files as embedded resource was also 967kb)

If I put them in a zip file, the zip file is 246kb. This was just using the windows zip implementation, maybe there is an even better compression algorithm.

Also, "compilation time" is a bit of a murky term with all of the scripting and hacking done by F# libs. If I compile a standalone app, I better know what I compile against, but if I compile in a scripting environment, I usually want to compile against exactly the version of FSharp.Core I have currently loaded (whatever that is, after bindign redirects and what not), because I might want to interact with the script from the host.

[KevinRansom]

@roejepp it is not an assumption. The compiler service is the compiler. If you see this error then it is because your build is is referencing a version of FSharp.Core that it happened to find, so you have a genuine compilation error. In many ways this reminds of the versioning issues a few years ago caused by Type Providers and incompletely specified builds.

We can add those files back into faharp.core and it will allow sloppy deployments and builds to complete more often. But the underlying problem will still exist. If you see this error it is because the necessary files were not deployed.

It seems like that is the right thing to do, why do you think it is not?

[dsyme]

We should just merge these into FSharp.Core (compressed). We will need to work out an interim plan to give a decent back story for previous compilers.

Every other F# dll has them merged in as resources (though uncompressed, which we need to consider). The small amount of space saved on deployments is outweighed by the regular runtime failures we're seeing in tools that use dynamic F# compilation or execution of some kind.

Azure Functions and Azure Notebooks are two examples of recent new deployments of hosted F# compilation and execution. We had a production bug in Azure Functions when these files couldn't be found for the AF command -line tools (AF defaults to a specific Fsharp.Core).

I sympathize with the goals of smaller binaries. But 250k for one DLL is not wirth the pain. Deterministic compilation is also important but once you explicitly reference a specific FSharp.Core it should just work. We're not doing anyone favours by making a complex world even more complex (and buggy) in an attempt to save a relatively small amount of space.

[dsyme]

"If you see this error then it is because your build is is referencing a version of FSharp.Core that it happened to find"

Sort of. You also see this error when you (or a tool you're using) explicitly references a specific, deterministic FSharp.Core, and the files just happen to be missing. This happens all the time when packaging tools that use hosted compilation or code analysis.

It's true it also happens for other slightly more sloppy compilation/analysis tools that go searching for an FSharp.Core to reference. There are many such tools. But it's not surprising that these tools go searching for an FSharp.Core since they need one. If they find one and decide to reference it, in a deterministic way, then things should just work.

An alternative would be have all tools always find FSHarp.Core via an FSharp.Core nuget package. That's fine for compilation/analysis tools that understand nuget - but quite a few don't and the churn in msbuild/nuget/project.json doesn't make it an attractive dependency. In any case making tools download a large nuget package is not a great solution, since it's even worse for tool correctness (the nuget feeds consistently go down) and performance (the download is large).

[dsyme]

Done