Scala-native: CBT tooling support

[RomanHargrave]

I have a few projects I have been wanting to do, and I would like to use SN for them. They aren't urgent, so chances are that I will wait for some level of SN maturity. But nonetheless, it would be nice to have good tooling for SN.

For instance, C is pretty easy to write builds for due to it's simple and well documented tooling. Due to the modular nature of SN, tooling more like D would have to be used, but nonetheless, there is still a single go-to utility that can locate a default runtime library, build object files, and link libraries and executables.

Currently, SN requires invoking the scala compiler with a path to the plugin, and the build automation support is just not there (I don't really like SBT all that much).

[densh]

This is a great discussion topic @RomanHargrave, thanks for bringing this up.

Our current tooling strategy is based on sbt. We're working on a plugin that should make it extremely easy to build/run and publish Scala Native code. This build tool was chosen as it's most familiar to existing Scala developers. I understand your gripes with sbt, it has a bit of a learning curve, even if you're just using the basic features. We'll do our best to document the ways we expect people to build Scala Native with sbt on our side.

An alternative build tool support is borderline out-of-scope at the moment but I would gladly help anyone exploring this space. I'm leaving this ticket open for now to gauge interest on the topic.

[RomanHargrave]

If an SBT plugin is in the pipeline, that may well serve as good enough documentation for alternative build methods.

My dislike for SBT stems mainly from the issues that plagued older versions (e.g. newlines, etc, didn't behave like scala even though it is scala). Nonetheless, what would be needed for implementing other systems is documentation on how the compiler is invoked and what parameters must be fed to it. That's the most important part.

[frgomes]

SBT is complex internally and difficult to understand even if you are not interested on its internals. Despite SBT wosks really well and its very flexible, I cannot stop thinking that its over-engineered.

Recently I've discovered CBT, which embraces the lemma that "a task is a def". Looks pretty simple and flexible enough. I will try CBT on my http://github.com/frgomes/poc-scala-native . I've also asked in CBT's gitter about the idea of porting CBT to SN. No feedback yet... maybe I even try that...

The point is: I find SBT needlessly complex. Being so dependent on macros, it diverges from future advancements in the language and basically anchors itself on Scala 2.10, which obviously depends on the JVM, which is exactly what we intend to get rid of with SN.

[Milyardo]

Being so dependent on macros, it diverges from future advancements in the language and basically anchors itself on Scala 2.10

SBT isn't anchored on Scala 2.10, SBT 1.0 is already on Scala 2.11, will probably be scala 2.12 depending on which reaches maturity first. SBT 0.13 is anchored on scala 2.10 because of binary compatibility with plugins.

which obviously depends on the JVM,

Marcos aren't JVM dependent and already work in SN for free.

[RomanHargrave]

I do have to agree with @frgomes, though.

It would be nice to have a language that is independent of the JVM.

[xeno-by]

@frgomes As @Milyardo rightfully pointed out, macros work in all three backends for Scala - Scala JVM, Scala.js and Scala Native. That's because they expand in the compiler frontend that is shared between all those backends.

Also, there are no plans to abandon macros in future versions of the language. There are too many popular libraries that use macros for that to happen. See Martin Odersky's presentation at recent ScalaDays (https://twitter.com/odersky/status/730033553434628096) for details.

[frgomes]

@xeno-by When I saw the other day the presentation you mention, I understood from slide 30 that part of the current support for macros in 2.11 would be dropped. Remember that breaking things was never a blocking factor for evolution of the language, and it will not be different in 2.12.

[xeno-by]

@frgomes There are no plans to drop support for macros in 2.12. We haven't planned anything like that, and, even if something comes up now, 2.12 is already in a feature freeze mode anyway.

What Martin meant is that Dotty (slide 26), the new experimental compiler for a Scala-like language, isn't going to support current macros.

Instead (slide 50), we want to implement similar functionality in a somewhat different way. The API is going to change, so some migration effort will be needed, indeed. However, this new functionality is planned to be close enough to existing macros. This suggests that most existing macros are likely to be migratable, and after migration the same code should work both in Scala 2.x and Dotty. This is all in future tense, so we'll see how it goes, but the currently planned design looks pretty convincing.

[cvogt]

FYI: Scala Native support for CBT is being tracked here now: https://github.com/cvogt/cbt/issues/129

[RomanHargrave]

@cvogt great!

[densh]

Both having CBT support for building SN code and being built by SN are reasonable goals we can target. It's also actionable, so this ticket will stay to track progress on that front.

[RomanHargrave]

That's great @densh

I am still curious about the possibility of getting some form of cc style command.

[frgomes]

Both scala-native from command line and cbt from command line would be great alternatives for development of system software (instead of cc + make) and also for developing quick ad-hoc tools (instead of python). Scala end-to-end, all over the place.

[RomanHargrave]

@frgomes I had more thoughts today WRT a more unixy interface to the SN tooling, which prompted me to look in to the architecture.

Currently, SN is "implemented", as far I can tell, as a compiler plugin for the official scala compiler that emits LLIR and then compiles the IR.

Steps that would need to be taken to separate from the plugin approach to a free-standing approach, or at least allow for a (preferably self-hosting) free-standing approach would basically involve writing, from the ground up, a lexer for scala.

Ultimately, I would like to look in to building a clang-like LLVM frontend that understands "Scala". This means that, eventually, there would be tooling which is inline with the tooling for other native-first systems languages, such as C, D, and Rust. This would also allow to remove the level of build complication that arises by requiring that the whole library of dependencies for a unit be provided to the compiler at the time of compilation, as linking is typically a separate stage when dealing with systems development (unless you're using go, lol). Obviously, this is beyond the purview of scala-native.

[sjrd]

would basically involve writing, from the ground up, a lexer for scala.

You need much, much more than a lexer. You need to reimplement most of the Scala compiler. And most of that will do exactly the same thing a scalac. This is simply an insane amount of duplicated work for no good reason, compared to the compiler plugin approach.

What you suggest is completely unreasonable. We are talking about man-decades of work, here.

[RomanHargrave]

When I wrote that last night, I intended to add a list of things, but I suppose that I forgot. The better solution would be to get SN to a point where the plugin can compile the compiler.

It probably can at this point, I haven't been following SN that much lately.

[densh]

Starting from 0.3.7 we offer portable API for 3-rd party build tools. SN support is already merged into Scalor Plugin for Maven, Bloop and there is an outgoing PR to Mill build tool. I believe we've addressed anything that can be done on our side.