Fs2: RFC: Ideas for improving FS2 documentation

[mpilquist]

I've seen a few comments recently in various places about FS2 and the lack of documentation. I'd like to remedy such concerns. There was a brief chat in Gitter today where we brainstormed various ways to improve the documentation but I'd like to capture ideas here.

We currently have:

• the guide
• Limited / terse ScalaDoc for all public members
• Some screencasts that are already out of date

We've already talked about adding sbt-doctest based examples to many of the combinators and I think we should continue to pursue that. Some other ideas:

• More annotated example applications
• Cookbook style patterns -- e.g., how to put together certain combinators
• Searchable Q&A
• Microsite? https://47deg.github.io/sbt-microsites/

What other ideas do folks have? What kind of documentation is most valuable when learning a library like FS2? What things were particularly tricky when learning FS2? Any thoughts on small, focused articles that describe a certain pattern or example application as opposed to some huge, monolithic book-like reference? What other Scala open source projects have exemplary documentation?

/cc @adelbertc @dwhitney

[tpolecat]

The best thing for me is always small standalone _working_ examples (with all imports, etc.) which is where tut came from. I think your Guide is great, so if you were to split it up into independent chapters and make a micro-site that would probably be a good starting point. At least it would give people a specific place to go, a specific thing to contribute to, and a specific thing to complain about.

[adelbertc]

Echo what @tpolecat said - best tutorials I've found are little bits on how to use various pieces: Stream, Handle, Chunk, Queue, Signal, etc. I really liked the Argonaut guide where they show snippets of common use cases that folks can glance at and get an idea of how to use it.

[pchiusano]

Just as a general remark, I'd love it if we can have docs 'teach people to fish' rather than just give them the answer to the immediate question they have. I think the big thing missing from the documentation is conveying how one can assemble more complex functions. Telling them how a particular function can be written doesn't necessarily generalize to other cases.

I think the 'teach people to fish' version of the docs would probably take the form of a series of progressively more interesting examples _and exercises_, introducing all the primitives of the library and then various concepts useful for composing more complex functions. Maybe not many people would want to invest the time in going through this (at least not right away), though I think it could be awesome for those that did. It would also be something we could gently point people towards in various places.

That is, there is a place for both sorts of docs - a person gets their immediate questions answered (possibly without developing a deep understanding), but then also gets pointed toward the docs / training which takes more work to get through but also empowers them to answer more of their own questions.

[dnrusakov]

IMHO, the current guide doesn't cover the following important topics:

(1) comparison with other competitors (Akka Streams, Monix);
(2) it's unclear what is the correct way to use FS2 with external sources like RabbitMQ (proper acknowledgements required), databases,...
(3) how to do streaming IO properly with FS2 in general;

It would be great to have those topics fully covered in the official guide.

[weihsiu]

More screencasts! I loved your screencasts and the way you explained things were very clear. If you think they are out of date, do a new one with all the updated information will be really nice.

[piotrga]

As much as I appreciate that people should be aware of the chunks and free monad bits to be able to use the framework consciously I believe that it would be useful to show what the library is capable of without revealing the nitty gritty.

All guides I've seen so far always mention the underlying abstractions which might cloud the picture for people just wanting to understand what problem does the library solve.
After saying the above I realised that I am missing the most the answer to the question "why does this library rock? What problem is it going to solve for me." It is quite an investment to learn it so I need to be sure that it is worth it.

[dwhitney]

Hey I'm a bit late to the party, but here is some spaghetti on the wall:

I totally agree with @tpolecat and disagree with @pchiusano. I like worked out examples with all of the imports included that avoid using symbols in favor of method names where possible. For example the current docs use >> instead of .flatMap in the "Statefully Transforming Streams" section. I don't have these symbols memorized and have to go look them up at which point I've lost the thread. It's minor, but when I'm learning something, I prefer the "teacher" expect I know next to nothing.

The reason I disagree with @pchiusano is realistically I'm never going to do the exercises. It's also frustrating when I have a question, and I see an exercise describing the problem I'm trying to solve, but no solution is linked. I learn best by solving "real" problems (problems I need to solve rather than problems I could solve for fun). wrt "teach a man to fish" -- I've been using fs2 for a few months and I understand it WAY better than I did when I started. I've done none of the exercises to get where I am. Would I have gotten there faster if I had done the exercises? Maybe, but while I was doing the exercises, I'd have had something else fall off my plate. And as I said, if I'm being realistic, I'm simply not going to do the exercises, and I think more people are like me than unlike me in that regard. Why not embrace that fact rather than fight it?

Random thoughts:

• once I realized Pipe was a type alias, it made so much more sense to me. I use type aliases in my own code, but sometimes they obscure rather than illuminate. I could be totally in the minority on this, and I wouldn't remove the type alias, but maybe in some examples it wouldn't be a bad idea to show it both with and without the alias.
• I'd take some bread and butter examples from Akka Streams and show how to achieve the same thing with fs2. .mapAsync and .mapAsyncUnordered would be good examples IMHO. Those two functions were 90% of what I used akka streams for and I don't think I'm in the minority.
• I've toyed around a bit with the io lib, and had unacceptable performance from it (like 10x slower than oldshool java io). I'm sure I'm doing something wrong and haven't taken the time to figure out what, but my code doesn't seems much different than the Readme. Putting together some examples that show poor performance and then how to boost the performance with complimenting explanations would be great. I suspect if I used better chunking I'd get a boost, but haven't looked much further.
• I think this is a tricky one, but also one of the cool things about fs2! I rarely use Task until I get to "the end of the world" these days. I think the current docs were good at pointing out that the effect type is arbitrary but I was a bit lost on when/why I would make my own. I use typically now just pass a F[_] around and eventually it'll either become a Task for a cats.free.Free. It may be cool to show a worked out example of this, but this is maybe a bit more advanced. That said, I think this is a major component in _why_ anyone would want to use fs2 -- great abstraction! As a side note, it was kind of a PITA to make the typeclass implementations, and I know I need to go look at them again because the error state in my impl just throws an exception. It would be great to have some impls that other people are using in the fs2-cats project.

I love this project and it has made my code so much better. I'd like to help where I can, especially when I feel more confident with my understanding. Thanks for all of the effort you guys have put in!

[pchiusano]

Re: @dwhitney on exercises, I don't think it needs to be either/or. :) Not everybody is going to want to do exercises or will have time to do so. But IMO well-designed exercises are a really efficient way of learning, you can learn the general principles very quickly, and I think it could be nice to have them for people willing / able to put in the time. Like @dwhitney maybe if such exercises existed with answers and explanations, and you cleared out some time to work through the exercises, that would have _saved you time overall_ compared to how you've been learning so far. :) Or at least it may have complemented the ways you've been learning the library.

It's sort of like: if a person asks the same sorts of questions over and over, it pretty quickly becomes more efficient to just have them understand the general principle. So how do you teach the general principle? You do it with short, simple, examples that focus just on the 'essence' of that principle, and as little else as possible. And how do you confirm that the principle has been understood? With an exercise!

I think part of the problem is sometimes that the user isn't (yet) bought into understanding the general principles. Like the just wants their immediate question answered, and the docs are not doing that, so the docs and the user are 'talking past' each other. :)

I guess that's why I'm saying there's a place for both. The docs (and us in the Gitter room) don't need to be pushy. The docs can both answer questions economically (hopefully addressing immediate problem user wants to solve), while also _linking to docs that introduce the general principles_. Of course the docs can't cover every possible concrete use case, but they can probably cover more than they do now, especially stuff that comes up more frequently.

This all sounds like a ton of work. :\ Honestly, I was happy that we got any sort of guide together at all, it was a major step up from before. :)

[ismagin]

Add some example of Free/runFree with streams. I clearly understand the concept of Free, ~>, Coproduct and so on, but I still have no idea how to use that in fs2, just _any_ example would be great.
Probably I'm not alone.

[jvliwanag]

One thing I've been searching for is a walkthrough of the building blocks of fs2. I've followed along the explanation of streams over the FP in Scala book, which resembles the old scalaz-stream. Some of the changes that went to fs2 were alluded to in @djspiewak 's scalaworld 2015 talk. But I haven't found any recent material that does a deep dive into fs2 internals.

[ShaneDelmore]

I spent a few hours attempting to put something together in FS2 streams tonight, after having read the guide and watched the first couple FS2 videos, together with a very helpful few folks on Gitter. For some background, I did a similar experiment about a month back with Swave and had great success, I found it easy to translate my existing scala code to Swave's streaming version. With FS2, I had very little success. @mpilquist asked me to provide feedback so this isn't meant as any sort of knock on the library or complaint, just my user experience.

So...my two cents. Once you get beyond the initial early adopters I think the next wave will come from people who already have a solution in mind, but want to see how they would translate it to FS2. For that reason examples of how to accomplish this is a known framework, and what it would look like in FS2 would be VERY helpful. I especially like what Swave did by leveraging the fact that most Scala devs are very comfortable with the Scala collections api. http://swave.io/talks/2016/scala.io/index.html#/

They provide examples of what you would do in plain old Scala collections, then show how to translate it. A couple of other things they did that made it easier for me to get going were to annotate code samples with graphs of the flow (as well as being able to produce the graph from code) and addressing common concerns early. For example, Swave in their ScalaDays talk gave examples of how to handle retries early on (put them back in the incoming stream by joining the failed output back up to the original input).

I was also surprised that the whole stream failed with one uncaught exception in a Task. For ease of use it would be wonderful if there was something I could mix in that would redirect failed rows on exception, not blow up the whole stream. I understand that I should never have exceptions, but if I do I can't think of many situations where the behavior I want on a thousand item stream when a single item fails (say to parse for example) is to have the whole stream die.

I wasn't able to tell from docs or examples if the recommended pattern is to restart the stream, or make sure my Tasks can never contain exceptions, or follow every Task with a recover maybe.

[abuggin]

Even though I personally like them, screencasts (apparently) don't age well when apis change and are more difficult to update than plaintext.