Fs2: Road to 0.10

[mpilquist]

Hi all,

I want to update everyone on the plan for FS2 1.0. This release will be focused on two things: performance improvements of the main interpreter and ecosystem integration.

Performance Improvements

We've started significant performance work on the main interpreter, StreamCore. In essence, we'll be replacing StreamCore with a new algebra and interpreter which are significantly simpler and faster. Some features of the new interpreter include:

• Stream[F, A] is implemented as a Pull[F, A, Unit], which is implemented with a Free-like monad
• Optimized such that "pure segments" don't bounce through the interpreter loop, resulting in much higher performance
• The implementation of uncons and unconsAsync are significantly simpler than their counterparts in 0.9.

Overall, the API of 1.0 will be mostly the same as 0.9, besides the things noted in the "Ecosystem Integration" section. There will be some minor API changes to some of the more "advanced" APIs, but the main types (e.g. Stream, Pull, Pipe, Pipe2, and Sink) will all remain mostly the same API-wise.

The performance work is happening on the topic/fast branch. As of now, there's a temporary fs2.fast package that contains a prototype of the new interpreter. Eventually, this will get folded back in to the fs2 package, replacing StreamCore.

Ecosystem Integration

Since we started development on FS2 0.9, the FP ecosystem has changed significantly. As a result of the maturation of cats and the release of cats-effect, FS2 1.0 will remove its custom type classes in favor of the type classes defined in these libraries. Specifically:

• remove fs2.util.Functor in favor of cats.Functor
• remove fs2.util.Applicative in favor of cats.Applicative
• remove fs2.util.Monad in favor of cats.Monad
• remove fs2.util.Traverse in favor of cats.Traverse
• remove fs2.util.Catchable in favor of cats.MonadError[F, Throwable]
• remove fs2.util.Suspendable in favor of cats.effect.Sync
• remove fs2.util.Effect in favor of cats.effect.Effect
• rename fs2.util.Async to fs2.util.Concurrent and have it extend cats.effect.Effect

The cats-effect library and its type class hierarchy was largely inspired by fs2.util and I participated in the development of cats-effect. I consider it a strict improvement over the concepts incubated in fs2.util.

Upgrade from 0.9 to 1.0 will be relatively painless (nothing at all like the scalaz-stream to FS2 upgrade). Further, we plan for 1.0 to be API stable for a long time -- at least 1 year if not more.

FS2 will remain polymorphic in the effect type throughout the library. As such, you'll remain free to bring your favorite effect type to the table -- e.g., cats.effect.IO or Monix's Task or scalaz.concurrent.Task.

Scalaz support for FS2 will continue to be provided by the fs2-scalaz interop library. This will include an fs2.util.Concurrent[scalaz.concurrent.Task] instance. The only change to Scalaz users is that you'll have cats-core and cats-effect on the classpath.

If you're interested in following the work in progress, take a look here. I'll have a PR up in a few days.

Cheers,
Michael

[djspiewak]

HYPE HYPE HYPE HYPE HYPE HYPE

[wedens]

Does it imply replacing fs2.Task with cats.effect.IO?

[mpilquist]

@wedens We're not sure yet. The current fs2.Task is basically the same as cats.effect.IO and if it stays that way, then I'm in favor of removing it. If we instead differentiate it somehow such that it provides meaningful utility, then it may stick around.

[pchlupacek]

@mpilquist Just to make clear what we are committing here to if I understand this correctly :

• fs2 will need to maintain binary compatibility builds with every binary version of cats - starting with cats 0.9 likely ?
• fs2 will "own" IO type and will be responsible for maintaining it, including binary releases like fs2 for cats

Anything else ?

[mpilquist]

@pchlupacek Right, we'll have an fs2 release for every cats release. Note that cats-core 1.0 is imminent and will have long term binary compatibility. It's release frequency will slow, similar to Scalaz 7.x. Cats-effect will also have long term binary compatibility once cats-core 1.0 is ready. Both of those 1.0 releases will happen before FS2 1.0.

The FS2 contributors will become contributors on cats-effect.

[BardurArantsson]

I'm assuming that this will mean that fs2 will an unavoidable dependency on cats-core (and cats-effect, I suppose)? It wouldn't even be possible to "shade" it away or similar?

One of the reasons I, personally, am very interested in fs2 was that it had no dependency on either cats or scalaz. These dependencies have been very problematic for my company in the past -- not so much due to incompatibilities per se, but just the fact that the diamond dependency problem is very much still a problem if you're building bigger systems that have to depend on lots of Scala libraries. (Further compounded by scala.js when that was first unleashed upon the world, but it seems the ecosystem has mostly recovered from all of the forking that went on there.)

Of course, if it's becoming an huge maintenance burden, then this is what must happen. I just wanted to offer a counter-argument which holds a lot of weight for me.

[mpilquist]

@BardurArantsson Yeah, it's an unavoidable dependency on cats-core and cats-effect. Note that cats-core 1.0 is imminent (in FS2 timescales :wink:), which should help avoid dependency conflicts due to diamond dependencies. cats-core 1.0 should have a stable ABI for a long time period.

We aren't shading cats-core & cats-effect b/c doing so would mean we'd lose first class interoperability with the larger ecosystem. For example, if we shaded, doobie & http4s would need a shim layer between the shaded fs2 version of cats and cats proper.

[djspiewak]

@BardurArantsson I (and other people working on cats/cats-effect) are very, very sensitive to the problem you reference. I've been working on large projects in the scalaz ecosystem for almost 7 years now, and it has definitely been a nightmare at times.

Compatibility is hard, and eventually things will break, but we're hoping to be able to do a little better than scalaz on this front by keeping things more modular. While cats-core is certainly not as modular as it was originally hoped to be, it is still an order of magnitude smaller than scalaz-core, and critically all of the non-core pieces are split out into separate repos. This allows us to version things independently. For example, if there's a necessary but incompatible fix to cats.effect.IO, that doesn't change anything in cats-core, and it would be able to remain on a single version. The same thing goes for projects like cats-mtl (monad transformer stack), dogs (data structures like IList) and so on.

Additionally, as a community, I think we've learned some things in the past 5-10 years about where compatibility tends to go south in Scala and how to design around it. Tooling is better (MiMa is not perfect, but still worlds better than what we used to have), and design conventions and idioms have warped around the quirks of the language and longer term compatibility constraints.

Finally, Scala 2.12 helps a lot. The new trait encoding makes it considerably easier to maintain binary compatibility. It's not perfect, but we're finally getting to the point where Scala is not vastly worse than Java in this regard, which is nice. That doesn't help if you're stuck on 2.10 or 2.11, and cats-core has to preserve compatibility with at least 2.11 for quite a long time to come, but it means the future only gets brighter.

So, tl;dr, I'm incredibly sensitive to exactly the problems you point out. I think we all are. Because we've all been there. We wouldn't be doing this and taking on the burden of this kind of upstream dependency if we didn't think the benefits strongly outweighed the (very significant!) downsides.

[mpilquist]

New release plan:

• 0.10, to be released imminently, will be the 0.9 design with the custom type classes replaced with cats-core and cats-effect. fs2.Task will be removed.
• 0.11 will have the performance improvements and API simplifications. There will be milestone and release candidates released throughout the summer with a final release targeted for September.

[pchlupacek]

@mpilquist I don't want to complicate things here, but my understanding was we agreed on variant B that was set as 0.10 will be what you refer to be 0.11. I am just bringing this up, so I believe is better to have one significant change than two, i.e. first replace task code with IO and then again replace combinators with fs2/fast, but likely do both at once. I may be wrong there, but certainly for our codebase we will likely skip 0.10 alltogether.

[pchlupacek]

@mpilquist I though about this more and I believe it will be shame if we won't publish anymore fixes to 0.9.x series. I also think that keep 0.9.x and 0.10.x and 0.11.x with backported fixes is overkill. I really would like just to keep one release back where we do just maintenance and then one main release.

[mpilquist]

@pchlupacek Reviewing the gitter log, I see the confusion. Everyone voted for plan B but then for some reason, I summarized the decision as if plan A won. Let's see if anyone really wants a 0.10 now as described above. I don't mind doing one but I prefer a single release that includes topic/fast to ease maintenance. If there isn't a lot of demand for a 0.10 now, then I agree, let's go back to a single release that includes the topic/fast work.

[pchlupacek]

@mpilquist thanks. you will run the poll :-) ?

[mpilquist]

@pchlupacek Yep https://github.com/functional-streams-for-scala/fs2/issues/878