Fable: Discussion: Fable dependencies packaging

[et1975]

It seems DLL + js are the implicit goal of future fable package distribution.
Current direction does not address:

• Transitive dependencies potentially introducing copies of the types (via included F# sources) .
• Given the significant mismatch in runtime type information - inline/generics constraints.

The proposed workaround for passing type information via implicit type array arg, introduces significant departure from the core F# functionality, adds cognitive load and essentially forks the language.
I'd like to raise the question of alternative packaging of dependencies, primary motivation being the same fundamental F# features as when targeting the .NET runtime. Presently, distributing packages as F# sources "almost" works. AFAIK, the only downside/overhead is the maintenance of the ordered list of sources by the consumer, which seems like it could be addressed via some additional metadata+tooling.

I might be missing something, but it looks to me that if distribution via sources works for Rust, Swift, Elm, JS, etc. we should be able to do that for Fable as well.

[alfonsogarciacaro]

We could work on that direction instead, but I think there are advantages and disadvantages for each approach. There're a few things that concern me about source distribution (besides the inconvenience of referencing many files).

• Compilation can be slow when you have many files. For example, at the beginning I tried to distribute Fable.Core bindings by source, but this made compilation twice as slow.
• The current solution for resolving generics by inlining also adds cognitive load, as this is not necessary in _standard_ F#.
• What worries me the most is having to deal with different settings in fableconfig. I'm not sure how it works in Rust or Swift, but because in Fable you can modify compilation via plugins (either Fable or Babel plugins) it may happen that a library needs certain plugins that are not compatible with the consumer project. As all the F# sources should be compiled together to allow inlining, I'm not sure yet how we could apply the different configurations.

Besides being able to inline code to be able to access generic information, is there any advantage you can think of about distributing source files over dlls? About the transitive dependency issue, hopefully this should be rare when distributing dlls gets more common (and it can also happen with sources if npm duplicates the packages).

[davidtme]

I quite like the distribution model of .fs files.

A few time I’ve had to go in and correct a few import files - then moving them out of node_modules. With PIXI on I had to edit a lot of it (PIXI has been renamed to pixi.js in v4). As we are shadowing ever-changing JavaScript files not being able to do this would be a great loss. Correcting a dll would be very difficult, checking out repos, building etc.

Didn’t funscript use dlls? I've never used it but how what that when you found a problem?

_A side note: Was there talk of trying to point fable at fable to output a pure JavaScript version (I seem to remember at the moment it couldn’t be done because of msbuild)? Would distributing dll stop that?_

[et1975]

When you say:

inlining also adds cognitive load, as this is not necessary in standard F#.

I have to disagree, it might be an unusual requirement, but only for the narrow case involving runtime type information while the code ultimately retains the portability - I can still compile with F# compiler (w/o fable) and it would still work. Not only bringing in code that targets .NET to run on fable would be much easier if inlineing was the price to pay, but a fable lib can be ported to run on .NET, all it would take is some namespace/imports shimming.

Compilation can be slow when you have many files.

I think compilation can be slow even w/o too may files, that's why I love the -w feature. But speaking of Fable.Core, seeing how it's not implemented in F#, I don't see a problem if it keeps shipping as a DLL.

Not to dismiss the rest of the concerns, but can you elaborate on:

What worries me the most is having to deal with different settings in fableconfig.

I'm under impression that leaving the fableconfig in the hands of the final consumer would actually simplify things a lot. And:

and it can also happen with sources if npm duplicates the packages.

Btw, I'm not suggesting that npm would remain the tool of choice, or that it would be npm alone (perhaps npm could be used under the hood, or post-install scripts could do fable's bidding?).

[alfonsogarciacaro]

This is a very interesting discussion. You both have made strong points :+1:

A mixed approach for distribution is going to be ver messy, so we need to agree on the best way for all. Let's try to listen some more opinions: Summon the king of distribution, @forki! 😄

About the different configs, I'm thinking on the scenario where the lib author uses a tricky Babel plugin for whatever reason, and it relies on that for the code to work properly. Of course if we were to distribute sources directly, I guess lib authors would avoid that and it'll help standardisation.

[Pauan]

@alfonsogarciacaro Using Babel _at all_ is very tricky right now, even with "DLL + Compiled JS". I don't think the trickiness is related to whether we use a DLL or not.

If we want to support Babel (and we do), then we will probably need Fable to use Babel internally, so that Fable can automatically compile the ES6 files and place the ES5 output into the outDir folder (and automatically adjust any references to use the ES5 version).

Whatever solution we agree on, I think it's important that a consumer does not need to know about the internal structure of a library. When I import a Fable library, I shouldn't need to carefully order the library's files in my .fsproj, and I certainly shouldn't need to carefully order the library's dependencies or sub-dependencies!

That's one major benefit of the .dll approach: I can just add a single Reference into my .fsproj file and everything Just Works. I don't need to know how the library is implemented, or what its file order is, or what sub-dependencies it has.

(Also using a .dll gives good tool support)

[et1975]

it's important that a consumer does not need to know about the internal structure of a library.

Agreed. For fsx scripts it could be trivially addressed with index.fsx distributed with the lib that loads the rest, for the projects some external action needs to take place. Maybe @Krzysztof-Cieslak has any ideas in that area?

[alfonsogarciacaro]

BTW, Fable will soon allow compilation of multiple projects at once (#500), which could make referencing sources easier. But you will still need to send the project files in the proper order.

[alfonsogarciacaro]

@forki Is there a way (or could it be done) for a project that's supposed to be referenced as git dependency to indicate its own git dependencies to Paket?

[alfonsogarciacaro]

Another question, are project references guaranteed to appear in the same order as in packet.dependencies?

[mastoj]

Personally I like the simplicity of fsx-files at the moment, but I realize that there might be some issues with that approach when working on a larger project. For smaller project I really think the option to just use fsx-files and git dependencies should be there.

[alfonsogarciacaro]

[et1975]

Hate to be "that guy", but I think github dependencies have one seriously weak aspect - fixed source. I think modern package manager needs to be able to take private dependencies and the repository needs to be able to mirror the sources. All of Elm development suffered last Friday as result, Go has the same weakness.

[et1975]

Here's a proposal...

Package format/manager:
npm

Package metadata:
fableconfig.json/exports section - list of F# sources that export something.

Proposed Fable CLI extensions:
fable install/update/uninstall package

Proposed Fable implementation:
Forward to npm for management of packages (as content) + call paket for post install management using package's fableconfig.json/exports. If there's no fableconfig/exports, consider the work is done.

Transitive dependencies: ???
I think Elm doesn't support those, maybe we could do the same. I don't know enough to propose the course of action.

This would make the packages already published work on day one. And lets you install pure JS npm packages with the same tool.

[tpetricek]

I did not read the whole thread - but I think Fable should reuse existing ecosystems as much as possible. What do JavaScript people when they want to have "github dependencies"?

(A lot of the power of Fable comes from the fact that it plays well with the JS ecosystem - as flawed as it may be - so I think trying to build too many custom things would not be good.)

[alfonsogarciacaro]

My current state of mind is that I'm convinced by @et1975 in that it's better to distribute sources and compile everything at once (this may be because he has strong arguments or because I'm a weak character :wink:) so currently I'm removing all the tricks used to allow dll/project references in Fable and force the new model on the user.

My take is that first we make sure Fable allows this (for example, I've to solve how to put sources from different places in outDir in a convenient way that avoids name conflicts) in an agnostic manner (sources can come from npm, github, a local dir...) even if it's a bit more work to the users (download all the necessary packages, using the IDE to add project references and setting the proper order in fableconfig). Then we can think of tooling to automate the process :+1:

[Pauan]

@et1975 I'm okay with a special fable CLI which shells out to npm, but I would suggest using yarn rather than npm, especially if we can use Yarn's flat mode.

That Elm page you linked to says this about dependencies:

Dependencies: The initial release does not permit dependencies except on the standard library. That simplifies my task for now, getting this release out earlier. The first batch of libraries won't have other dependencies anyway. (Thank you Mads for this idea!) Lifting this restriction is the highest priority for this project, and the ultimate goal is to allow multiple versions of a library in the same project (trying to avoid cabal hell).

We definitely need transitive dependencies.

---

@tpetricek

What do JavaScript people when they want to have "github dependencies"?

They just use npm + package.json, which has decent (but imperfect) built-in support for Git repositories:

{
  "dependencies": {
    "foo": "https://github.com/foo/foo.git",
    "bar": "bar/bar"
  }
}

• foo is a direct Git URL.
• bar is a GitHub author/package path.

You can read more here.

Git repositories have some caveats, though. They're not as good as using the npm registry.

---

@alfonsogarciacaro I'm okay with distributing only Fable source code and doing all compilation in the application, but my concerns about fsx and tooling still stands. :confused:

So it seems we have some hard choices to make: how much do we want compatibility with Visual Studio? How much do we want compatibility with Ionide? How much do we want a clean and easy user experience (especially for people coming from JavaScript to Fable)?

I definitely don't like .fsproj and .dll, but they have the advantage that they Just Work, especially with existing tools.

So, what's the plan? Try to push for fstoml and abandon Visual Studio support? Use custom Fable metadata which doesn't work in any tool? Try to push for fsx despite its problems with load order (which especially confuses JS developers)?

[et1975]

I'm not aware of any issues with VSCode+Ionide or VS with distribution as sources.
DLLs are actually worse at the moment, at least for me Go To Definition does nothing in VSCode.

In terms of metadata - post install step could integrate the package sources into the consumer project, VS/VSCode don't need to know anything special, so I don't really see a downside.

[forki]

Yes go to definition was always nice.

Am 25.10.2016 8:50 nachm. schrieb "Eugene Tolmachev" <
[email protected]>:

I'm not aware of any issues with VSCode+Ionide or VS with distribution as
sources.
DLLs are actually worse at the moment, at least for me Go To Definition
does nothing in VSCode.

In terms of metadata - post install step could integrate the package
sources into the consumer project, VS/VSCode don't need to know anything
special, so I don't really see a downside.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fable-compiler/Fable/issues/498#issuecomment-256137383,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNMH5TazSFxZUEkroVMZ--QZ2j6u7ks5q3k9egaJpZM4KeyzT
.

[Pauan]

@et1975 What do you mean about "integrate the package sources into the consumer project"? Do you mean that a tool would parse the library's fsproj, read out the <Compile Include>, then automatically add them to the consumer's fsproj in the right order?

[et1975]

Yes, given the package's fableconfig.json/exports provides the ordered list of F# sources we could

parse the library's fsproj, read out the , then automatically add them to the consumer's fsproj in the right order

This is where transitive dependencies would come in as well, I guess. So... parse package's package.json/devDependencies as well, consolidate it with consumer's list of dependencies then run it through solver to determine the correct package order first.

@forki is doing something like that in paket already... can we fork @forki? His clone could probably do it in a lazy afternoon :)

[Pauan]

@et1975 I see, I'm okay with that kind of workflow :+1: Just keep in mind that not everybody uses VS Code or Visual Studio (I personally use Atom), so this would need to have good support from the Fable CLI.

Assuming that the support is solid, we can then treat fsproj as an auto-generated file which users don't manually touch at all. Instead, they would edit fableconfig.json and then run fable install (or whatever) to generate the fsproj

[Pauan]

If we're going to push for metadata in fableconfig.json, we can even consider removing package.json: package dependencies would be specified in fableconfig.json. This makes things more convenient: we have a single place where we can list Paket, npm, bower, and Git dependencies.

[tpetricek]

Just for the record, I use a mix of Atom and Visual Studio and would be quite sad if they stopped working with Fable.

[et1975]

@tpetricek I don't see any reason for them to stop working. We are just talking about automating some stuff that already happens today.

[et1975]

Since performance is not in scope for FSAC, I think the question source-based distribution can be closed.