Rls: RLS is not ready for 1.0

[jntrnr]

Seeing the release candidate post for RLS was a good reminder to "kick the tires" and see where things are. There are some concerning things that came up as a result.

The RLS 1.0 Release Candidate post mentions "1.0 for the RLS is a somewhat arbitrary milestone." Respectfully, this is a mistake. Having a 1.0 announcement is something that you can only do once and has its own set of user expectations. The amount of disappointment for folks around this announcement is pretty clear. It feels like one of those times where's it's important to take a step back and take stock of the objective experience of where the RLS is.

As it stands right now, the RLS works _sometimes_ for _some_ projects. This has steadily improved over the last year, and I think the team should be applauded for the efforts to continually improve the RLS. While it's improved, looking at where things are today it's not 1.0-ready.

Some examples:

• Completions are not up to user expectations and are missing important patterns like https://github.com/rust-lang-nursery/rls/issues/588 or https://github.com/rust-lang-nursery/rls/issues/904.
• Relatedly, there are also stability issues around completions(eg https://github.com/rust-lang-nursery/rls/issues/842 or https://github.com/rust-lang-nursery/rls/issues/833 or https://github.com/rust-lang-nursery/rls/issues/835)
• Renaming has issues, enough so that it can't be used reliably (eg https://github.com/rust-lang-nursery/rls/issues/760 or https://github.com/rust-lang-nursery/rls/issues/467)

Not to nitpick the announcement, but one quote that stuck out to me was "Because (racer) is syntactic there are many instances where it is incomplete or incorrect. However, we believe it is useful." Again, respectfully, it's up to the users to say if the completions are useful rather than the team saying so.

We're not there yet, but we do have users who are willing to try out the RLS and helping it continue to improve. For the RLS, it's probably best to leverage those users and get to the point where they give it the thumbs up. From there, you have your release candidate.

The rustc and cargo tools set a high bar for how tools should work if they're official tools. Other tools, like rustfmt have progressed well towards that high bar. That said, judging from the feedback, the RLS isn't quite there, yet.

My humble suggestions for a 1.0 candidate:

• Survey the users about RLS completions. Improve until this hits a strongly positive value (80%?) saying they're happy with the completions.
• Clear through the P1 and common pattern issues in the issue tracker. Users are going to hit these, and they're going to ask why you made a 1.0 release without fixing them.
• Make sure you have a full regression suite ready. Once a completion/refactor/codenav works for the user, don't break them with the next version. You may already have this, but it's worth mentioning.
• Match user expectations in terms of project size and complexity. Does the RLS work for people who want to use it? You mention "small and medium" sized projects, but it's not clear what this means. It may mean different things to different users.

In short - I think the RLS can get there. And there's plenty of folks who can help test and guide towards hitting that bar. We're just not there, yet.

[nrc]

I expanded on the announcement in another in another blog post: https://www.ncameron.org/blog/more-on-the-rls-and-a-1-0-release/

tl;dr we don't have to go 1.0 yet if we're not ready, but I would like to indicate that the RLS is ready to be used by a wider audience and I'm not sure about the best way to do that.

"1.0 for the RLS is a somewhat arbitrary milestone." Respectfully, this is a mistake

By "arbitrary" here I mean that there is no backwards compatibility guarantee to make and no hard measure of correctness at which to make a cut-off. That is, we have to choose when to do a 1.0 based on a general assessment of the quality.

it's up to the users to say if the completions are useful

I have had plenty of feedback that it is useful. I think the question is "is it useful enough". A lot of the 1.0rc feedback suggests not. There is a wide spectrum of useful from, 'more than zero suggestions' to perfect suggestions. I think it is a pretty hard task to pick a point there where it is 'good enough' and for users, that's primarily a matter of expectations.

... RLS completions. Improve until ...

There is not a lot (though more than nothing) that can be done in the short term. This would probably mean accepting no RLS 1.0 for several years. That might be fine, and we could encourage more people to use it without formally being 1.0. But I want to clarify the time scale here.

Clear through the P1 and common pattern issues in the issue tracker.

We've done this. That is what the 1.0 milestone was for.

Make sure you have a full regression suite ready.

We could definitely improve here, but we have regression testing for common actions such as code completion and types/docs on hover.

Match user expectations in terms of project size and complexity.

I'm not really sure what this means. There is such a wide range of user expectations about what size of project 'should' work and what acceptable performance means. As you say, It may mean different things to different users. It's also very dependent on the user's hardware and their tolerance for battery use/fan noise. In other words, I agree performance is important, but I don't see how to make that into a measurable milestone towards a release.

[jntrnr]

tl;dr we don't have to go 1.0 yet if we're not ready, but I would like to indicate that the RLS is ready to be used by a wider audience and I'm not sure about the best way to do that.

Doing a release announcement of new features, new accuracy, etc, and showing how it works and why it's awesome is a great way of getting eyes on the project. Especially if you want a "try it again, even if you tried it before" crowd. It doesn't need a label like "1.0 RC" to get attention if it's interesting, has good info/screenshots/etc.

I was also wondering about current RLS usage, so I took a peek at the current survey for this one. It's definitely getting used. I'm seeing 2 out of 3 Rust users are currently using (or have used) it. (with the caveat that the survey is still in progress and won't fully complete for another couple weeks).

With that many users you should be able to feel it on the repo. Are you not getting enough feedback from users?

By "arbitrary" here I mean that there is no backwards compatibility guarantee to make and no hard measure of correctness at which to make a cut-off. That is, we have to choose when to do a 1.0 based on a general assessment of the quality.

Why is there no backward compatibility guarantee? There are a few that you could potentially make that I think would align with expectations from the users.

• The correctness of completions, codenav, refactoring, etc will not regress past the point delivered in the 1.0
• For consumers of the RLS, the API will not change after this point unless the version number also is bumped a major version (semver for API, basically)

If you're not offering either guarantee, I think there's a pretty good case here that it's not quite time for a 1.0.

I have had plenty of feedback that it is useful. I think the question is "is it useful enough". A lot of the 1.0rc feedback suggests not.

To see why, something that might be helpful here is to try out other language servers, if you're not already. Try Java in Eclipse, or C# in VS (or VS Code), or TypeScript in VSCode. These are the kinds of experiences people are going to assume a complete LS will have. They have a few features in common:

• They're pretty fast
• Generally they're "never wrong". If a completion list comes back, it's correct. Refactoring will work.
• They rarely, if ever, crash

The RLS has made progress on the last bullet, but the first two are where folks coming from the other languages are going to feel the most lack.

Am I suggesting we hold off a 1.0 until the compiler can drive the completions? I don't see a harm in it. Not being 1.0 isn't what's holding it back right now. Rather, it's not finished, yet, and users are responding to the experiences they're having with it.

Clear through the P1 and common pattern issues in the issue tracker.

We've done this. That is what the 1.0 milestone was for.

There isn't a rush to get 1.0 out the door, at least there shouldn't be. You don't need to put an RC out that still has P1 issues. To try to fix them before the 1.0 final would mean landing the release hot. Instead, you could opt to fix all the major issues you know of and then ask people to try it. That way you're saying "this is what it says on the box, a release candidate. Any of these might be the real 1.0"

Match user expectations in terms of project size and complexity.

I'm not really sure what this means.

There are some good data sources you can pull from to calibrate the team with the user expectations.

• Interact with users on the forums (which you're doing). You can get a good "temperature reading" that way of how on or off the features are from the expectation
• Work through the data from the survey team. This isn't ready yet, but when the survey closes in a few weeks we can take a look at the raw data.
• An RFC. Not sure if you're doing this -- and it may be a bit odd to RFC the release -- but since it's contentious it might give you more user feedback.

You can also close the loop with the users who are giving feedback. If they report that completions aren't good enough, you have someone you can potentially reach out to and say "hey, we've done a ton of work recently on completions and we'd love to get your feedback"

[Pzixel]

I read the whole articel and here are my 0.02$

What does 1.0 mean for the RLS?

There is no backwards compatibility guarantee for the RLS, and no obvious quality cut-off.

1.0 should defenitely give such a guarantee. If it doesn't, it shouldn't be 1.0.

I have mainly been thinking in terms of stability for 1.0 - it is essential that for nearly all users, crashes and incorrect results are rare. I believe that we have achieved that (or that we can achieve that by the end of the 'release candidate' phase).

I think, it's the most discutable bullet in the article. I think the opposite: I encounter myself with tons of errors or invalid completions, as well as people I know. I don't think that all them could be fixed earlier than 2019. And I also have a strong feeling against "Well, we maybe will fix these issues by the end of RC".

in particular, the current situation can give the impression that Rust is 'not ready'

There is no shame to face the truth, it is what it is

---

For example, I do believe that rustfmt actually reached the 1.0 RC bar. It just works, it provides great experience. But RLS doesn't. There is no point in announcing it 1.0 just to announce it. If you need more feedback - just ask for it.

Making RLS in on its current stage an 1.0 version would deliver nothing but disappointment.

[norru]

My opinion:

RLS is promising but there are too many open issues on:

• code completion is hit and miss, but mostly miss
• definition/documentation lookup is not fully implemented and, lately, subject to regression problems
• configuration - since .rls.toml is no longer read, configurability depends on how up-to-date the host IDE is. This means that any new features may not be available until the client catches up.
• there is too much development focus on That One IDE™ - which in some cases determined loss of functionality for other IDEs

In order of importance, I would personally prioritize in this order:

• stability (nothing to complain about!)
• reliable and complete symbol definition lookup
• configuration
• code completion

TL;DR: It's definitely nowhere near the "1.0" badge.

[nrc]

I wrote a bit more on this: https://www.ncameron.org/blog/more-on-rls-version-numbering/ tl;dr - we're at some kind of stability milestone which is worth advertising, but we'll avoid using the 1.0 label.

[jntrnr]

@nrc - "By starting at 1.31, we're deliberately avoiding the 1.0 label."

The version 1.31 is after 1.0. This implies that the release satisfies the need of a 1.0 label, in the same way that Rust hit 1.0 and has earned the trust of the community to pass from that milestone to the 1.31 release.

The RLS still hasn't won that level of trust from the user, as you mention in your blog post. I'm afraid that while your intention to message some utility is good, you're perhaps missing the larger picture that people want the RLS to continue to work towards earning the trust of the community.

"I believe the 1.x version number is appropriate in that context - if you want to build an IDE, then the RLS is stable enough to use as a library. However, it is lacking some user-facing completeness and so an IDE built using the RLS should probably not use the 1.0 number."

What the user uses will say that it has passed 1.0, if I'm understanding the post correctly. If that's the case, the phrase "it is lacking some user-facing completeness and so an IDE built using the RLS should probably not use the 1.0 number" should apply to any use of the RLS that the user would encounter.

I hope you reconsider this direction. Feedback from the latest survey has put the RLS's incompleteness as one of the top concerns. Judging from the survey responses, I suspect that users will not generally care if the RLS is API-stable enough to consume, but rather will care about the experience of using it in an IDE. Taking those concerns seriously, as I'm sure the team does, should be reflected in the messaging and versioning.

[Pzixel]

FYI I often see "Don't use RLS"/"It's nearly dead"/"Wait until Kladov finishes rust-analyzer" and so on recently. It's not even close to be 1.0. IntelliJ-Rust has no near plans to go 1.0 and still be much more stable and providing much better UX.

It has trust level "still better than notepad with no highlighting", but nothing more.

I appreciate work has been done on it, it has a great team. But AFAIK it requires massive refactoring and it has so much legacy that it's easier to rewrite from scratch (hello mentioned rust-analyzer), that means it would contain bugs unfixed for years, and their amount will increase amongside with rustc evolution.

[norru]

still be much more stable and providing much better UX

@Pzixel Not in my experience. I'm much happier with the "live" analysis (correct compiler errors as-you-type) of RLS than I was with IntelliJ's. I hope I don't have to step back to loss of features should the community decide to adopt that approach. I could use RLS to be more stable though :)

[nrc]

The version 1.31 is after 1.0. This implies that the release satisfies the need of a 1.0 label, in the same way that Rust hit 1.0 and has earned the trust of the community to pass from that milestone to the 1.31 release.

The intention is to deliberately avoid 1.0 so as not to mark it as a significant event. And to avoid the usual 'mission accomplished' celebration which goes with it. However, a good number of people do think it is worth indicating that stability has improved significantly.

you're perhaps missing the larger picture that people want the RLS to continue to work towards earning the trust of the community.

I don't think anything about this precludes that.

What the user uses will say that it has passed 1.0, if I'm understanding the post correctly.

No, the user will basically never see anything about the RLS. The VSCode extension will be version 0.5, and we won't move to 1.0 until we reach the kind of quality bar you have in mind.

I hope you reconsider this direction. ...

To be clear this is very much an 'insiders' blog post. We're stepping back from using any of the usual 1.0 language for the RLS and I hope the user-facing version number in the extensions gives the right message about expectations for the whole IDE experience, while the non-user facing RLS version number gives the right message about stability and so forth.

[Pzixel]

@norru I have the opposite experience. However, I'm probably the last person on the planet wanting subtle plugin for proprietare IDE to be the only de-facto tool for rust developing. I'm really hoping that problem will be solved.

But you can't solve problem if you deny its very existence. I'm also looking toward language server (as I'm really happy using Roslyn and its api on day-to-day basis), but in it's very sad to see its current state is considered "it has several problems, but it's fine overall".

[jntrnr]

The intention is to deliberately avoid 1.0 so as not to mark it as a significant event.

I think you may be focused on the exact version "1.0", rather than the messaging of "1.0 or later". Giving the RLS the version number that falls after 1.0 does not avoid 1.0 and its significance.

I'll let it lie, but even for insiders, I believe the best plan of action is to maintain pre-1.0 versioning.

[norru]

"it has several problems, but it's fine overall".

Of course it's "not fine" - needs a lot of improvement, but saying "I hope it to become as good as IntelliJ" (hope it rather gets better) would be at least as misleading :)

[Pzixel]

@norru It's just my observation, I don't pretend it's the only truth :)

My position that there is:

RLS + VSCode - free and debuggable, but unstable and lack a lot of features
IDEA + Plugin - free and stable, but has no debugger so unusable in some cases
CLion + Plugin - stable and has a debugger, but is not free

So it's currently a classic 2 out of 3 kind of things, so I hope situation improves.

[MSxDOS]

IDEA + Plugin - free and stable, but has no debugger so unusable in some cases
CLion + Plugin - stable and has a debugger, but is not free

Subjectively, they're also quite slow and resource-hungry even after a few JVM tweaks I applied, but I still prefer them over VSC + RLS due to much better code completion and proper syntax highlighting. Regex-based highlighting in VSC is a joke and AFAIK, RLS lacks one completely.

[norru]

CLion + Plugin - stable and has a debugger, but is not free

Price is not a problem if it's worth paying for. I have a (paid) licence of CLion and yet I prefer Corrosion because of the weird inspection system in CLion.

Summarizing: my priorities for an IDE at the moment, and in the context of this discussion, are:

Deal breaking:

• responsive editor with acceptable syntax highlighting
• must have light or customizable theme as I get eye strain from dark background
• reliable compiler messages and error reporting (as you type/on save)
• functional debugger with variable inspections
• reliable code navigation (go to definition)
• receptive developers (issues are not ignored)
• multiplatform Windows/Linux

Nice to have:

• customizable screen layout (actually this is kinda deal-breaking but I could tolerate)
• reliable doc hover/autocompletion
• open source (so I can fix showstopper bugs if dev too busy)
• free of charge

None of the IDEs I know has all of this - Corrosion is the best match so far. A lot of the goodness I'm striving for depends on RLS.

[Pzixel]

@norru

Price is not a problem if it's worth paying for. I have a (paid) licence of CLion and yet I prefer Corrosion because of the weird inspection system in CLion.

I'm happy for you, but if you are say a Java/Python/... developer and you heard about wonderful Rust you won't be able to spend even a buck to buy an IDE just to play. Not to say about students etc who just don't have any money.

And I don't think I have to explain that language server is a must have for any modern language. Plugins won't help.

[norru]

@Pzixel

I'm happy for you, but if you are say a Java/Python/... developer and you heard about wonderful Rust you won't be able to spend even a buck to buy an IDE just to play. Not to say about students etc who just don't have any money.

Oh, I absolutely agree :) While I can afford the (small) licence fee if I need it for work or training I perfectly understand that not everybody would be willing/capable of shelling out, so I do welcome free for the community all the time! On top of that, since OSS is making me save money on licences (because it replaces a commercial program), I am perfectly happy to pass some of that saving on in the form of donations to the developers of the projects I like.

Also I agree that Language Servers are the way forward (unified backend to serve multiple IDE frontends = homogeneous behaviour + less duplication of effort).

Besides, Eclipse Corrosion leverages the LSP4E project (generic support for LSP) and the RLS (LSP backend to Rust) and it's supposed to be a "thin" layer on top of the language server so the more features get implemented at that level, the better the plugin will become for free +@mickaelistria.