Roslyn: Generators should be able to add source files originating from a StringBuilder

[jasonmalinowski]

Right now the only way you add source is with a SourceText. Unfortunately if you want to use a StringBuilder to build up your generated file, and then want to pass that, your only out-of-the-box way to do it is to realize the StringBuilder to a string. That's pretty icky if the string is huge.

See where I worked around this: https://github.com/dotnet/roslyn/pull/47252#discussion_r480506021

[AArnott]

I don't want to use a StringBuilder either. I have a CompilationUnit that I'd like to just "write out" to the SourceText but can't figure out how to do it given the API.

[chsienki]

@AArnott This mostly comes back to the parsing problem. Roslyn has some strong internal assumptions that anything it's working with is well formed (at least from the perspective that it was successfully parsed, it can of course contain error tokens etc).

In order to accept arbitrary syntax, we have to validate that somehow. The only real way to do that is to parse it, so even if you passed us the syntax, we'd just turn it into a string and re-parse it ourselves anyway.

I do wonder if we should accept syntax, so that under the hood we can 'stream' convert it to text and re-parse it, rather than making it the responsibility of the user to do so.

[jasonmalinowski]

@chsienki That might be good at least because that way we can do it "efficiently" as opposed to letting the generator author do it less efficiently. :smile:

[chsienki]

@jasonmalinowski Yeah that's exactly my thinking.

[jasonmalinowski]

I also wonder if the parsing overhead ever becomes expensive, then we can attempt to do some well-formedness validation. I know that's not easy to do (read: seriously I don't have a good idea either, good luck), but at least we'd be able to change it rather than suffer for the fact it was embedded in the generator itself.

[chsienki]

@jasonmalinowski Yep, I think it basically gives us more flexibility under the hood to make changes in the future, while also potentially making it easier for users. Win-win.

I'll add it to the proposal backlog and schedule some time to get a proper proposal written and reviewed.

[AArnott]

By far the very slowest part of my code generation is the (non-cancelable!) call to NormalizeWhitespace(), so if I could just pass my CompilationUnit to Roslyn and let Roslyn not only decide how/whether to re-parse it but also decide how to _format_ it with whitespace so that it complies with .editorconfig or whatever, that would be great.

BTW, It's very likely that my syntax tree isn't exactly what the C# parser would produce, particularly around the tokens that trivia is attached to. So I have no problem with you re-parsing it. In my last comment I wasn't exactly asking for you to take the CompilationUnit itself but rather than you give me a way to efficiently write it out to a SourceText without allocating huge strings so that I can then send you the SourceText to parse from.

[YairHalberstadt]

Implementation at https://github.com/dotnet/roslyn/pull/49689