Solargraph: invalid byte sequence in US-ASCII

[JeffLuckett]

Environment Info:
MacOS High Sierra
VS Code: 1.22.1 (1.22.1)
Solargraph Gem: 0.18.1
Ruby Solargraph Plugin: 0.14.1
Ruby: 2.4.1p111 (Large-ish Rails app)

Let me know if you need more info, or if you have any suggestions.

[castwide]

Is there a particular file that gives you this error? The most likely cause is a literal string with escaped characters, e.g., "\xcf".

I'm not able to reproduce this error in Windows. I'll try on MacOS later.

[JeffLuckett]

@castwide - I'm guessing that we've got something like that _somewhere_ in the code base, but unfortunately the error message just isn't specific about where the error occurred.

The project is ~250K lines of code in ~2K .rb files, not including gems, etc...

[castwide]

No problem. Even in the unlikely event that I can't reproduce the error, at least I can add the name of the offending file to the message.

[TheTharin]

But is there a way to ignore it in the future? I mean I can't just go and add #encoding to any file in the project solargraph finds offending.

[castwide]

Yes, I think I can make the parser ignore it. There are already similar character encoding issues that it's able to resolve. I'll still make it report the file name in the event of an edge case it can't handle.

[castwide]

Reproduced on MacOS. I'm looking into solutions.

[castwide]

A fix is pushed to the master branch. The parser should never fail. Even if the file isn't valid Ruby code, Solargraph should add it to the workspace and report the problems in diagnostics. Character encoding should be a non-issue. It's possible for Ruby code to contain string literals with invalid characters, but Solargraph shouldn't care as long as it doesn't try to write back to the file, which it never does.

I'll update the gem in the next day or so.

[castwide]

Version 0.18.3 is published.

[JeffLuckett]

Thanks @castwide, instead of the original error, I now receive [NoMethodError] undefined method 'node' for nil:NilClass

Would you prefer I open a new issue on this?

[castwide]

@JeffLuckett Are you still getting that error in the current version (0.19.1)?

[JeffLuckett]

@castwide - no, now I get:

However, I've only got about 2K files in the repo that are .rb files.

=> git ls-files | grep .rb | wc -l
    1981

[castwide]

Do you have a .solargraph.yml file? If so, does the include section select files that do not have the .rb extension?

[JeffLuckett]

There is no .solargraph.yml file in my project root, or in my home dir.

[castwide]

Is the project root your open folder in VSCode? Solargraph treats whatever folder you open as the workspace.

Does the project have cache or vendor/cache directories, or something similar that contains bundled gems? If so, you might need to create a .solargraph.yml and add them to the exclude section. You can generate a default config by running solargraph config from the command line or selecting Create a Solargraph config file from the VSCode command palette.

Also, this command should give you a more accurate idea of what gets added to the workspace:

find -name "*.rb" -not -path "./spec/*" -not -path "./test/*" | wc -l

The default is to include all ruby files except the spec and test folders.

[JeffLuckett]

I did solargraph config and then added - vendor/**/* to the excludes. That seems to have made it happy.

Thanks for sticking with me on this :D

[castwide]

Glad you got it working. This made me think of two more items for the to-do list.

• Add vendor to the list of folders excluded by default.
• Implement a solargraph subcommand called something like ls-workspace that lists all the files that would be mapped based on the current configuration.

[JeffLuckett]

This issue is resolved, thanks again ... closing.

[AeroCross]

Hi! I was wondering if I found an edge case.

I'm using Solargraph's 0.21 version and I am still getting this error. This is a _huge_ codebase though (about 9k Ruby files). I've reduced the count to less than 5000 to make the server work, but I still struggle to get it working, since I'm finding a lot of non-ASCII characters in form of comments, tests, etc.

After looking for a way to check what non ASCII characters are present in my codebase, I got the following:

$ ag --ruby "[\x80-\xFF]" . --stats-only
3382 matches
149 files contained matches
9451 files searched
19659019 bytes searched
0.363241 seconds

The stuff I found are characters like ’, ™, ö, etc. In other codebase I work on (which doesn't have non-ASCII characters), this doesn't occur.

These are contained in Ruby files, so I can't just get rid of them (we're specifically _testing_ for issues with those characters, so I can't change the code to remove that).

I also notice that autocompletion still works, but it works in a very barebones way — no inline documentation, no differentiation between methods, constants, variables, etc., no symbol information for Ruby files, etc.).

I am using VS Code, if this helps, on Mac OS X 10.12.6.

$ code --version
1.23.1
d0182c3417d225529c6d5ad24b7572815d0de9ac
x64

$ solargraph --version
0.21.0

[747]

Similar error. I started to get "Invalid byte sequence in Windows-31J" after some recent update.

[Error - 14:29:36] Server initialization failed.
  Message: [ArgumentError] invalid byte sequence in Windows-31J
  Code: -32603 

> solargraph -v
0.21.1
> code --version
1.23.1
d0182c3417d225529c6d5ad24b7572815d0de9ac
x64
> [System.Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.16299.0 Microsoft Windows NT 10.0.16299.0

[castwide]

@747 I started #63 to track character encoding issues.

[tekknovator]

Hi!

63 has been closed but the issues persist. I work with scripts generating some bash with osascript calls (among other stuff) wich requires "¬" characters at some line endings. Unfortunately there is no work around that.

So solargraph keeps failing with
"Failed to start Solargraph: Error: [ArgumentError] invalid byte sequence in UTF-8"
I might just add encode or switch back to rubyLocate. Solargraph is (feels) faster though, would be nice to have.
Any ideas anyone?

[castwide]

@tekknovator Can you provide a minimal example that triggers the error? I've found at least one bug in how certain UTF-8 characters get handled (including "¬"), but it doesn't make the server fail to start.

[kkneutgen]

Problem seems to appear again with version 0.31.0 of solargraph gem.

I downgraded to version 0.30.2 and solargraph-server starts inside atom.

[AlanWarren]

Problem seems to appear again with version 0.31.0 of solargraph gem.

I downgraded to version 0.30.2 and solargraph-server starts inside atom.

Thanks, I experienced this with a rails project today using solargraph-0.31.3. Downgrading to 0.30.2 fixed the issue.

Is there anyway to view output of which files solargraph is parsing when it hits the invalid byte sequence and crashes?

[castwide]

@AlanWarren I'm working on an update that will report which file triggered the error. Hopefully it will also be able to skip offending files and parse the rest of the workspace.

Can anyone who's experiencing this error let me know if there's a backtrace or any additional information in the console log? (In VS Code, Help -> Toggle Developer Tools; in Atom, View -> Developer -> Toggle Developer Tools.)

[kkneutgen]

@castwide I just sent the console-output via email to you.

[castwide]

Thanks, @kkneutgen. That was exactly what I needed. I was able to trace the problem to a regular expression match that choked on invalid character encoding.

To reproduce the error, I created a file that contained UTF-8 (e.g., '¬'), converted its contents to ANSI with an external tool, and ran the file through the Solargraph source mapper.

I think I'll be able to have a fix for this in the next minor release.

[castwide]

The master branch has an update that might resolve this issue. It fixes the problem in the variation of the error I reproduced yesterday.

[castwide]

One more change: exceptions during the mapping process will emit warnings that identify the file where the problem occurred.

[castwide]

Gem v0.32.0 includes the latest fix for invalid byte sequences and the new mapper exception handling.

[kkneutgen]

Great, thank you. No more problems with strange encodings here.

[tekknovator]

Works here as well now. Sorry for not responding earlier. I used the characters inside of a heredoc.

[sent-hil]

I'm getting the same error in version 0.37.2. Only version I can get to work is 0.31.3.

~/w/f/bevel:develop$ code --version
1.40.0
86405ea23e3937316009fc27c9361deee66ffbf5
x64

ag --ruby "[\x80-\xFF]" . --stats-only
265 matches
11 files contained matches
1519 files searched
2217378 bytes searched
0.041453 seconds

ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin18]

[castwide]

@sent-hil Can you give me a simple example of a file that causes the error, the character set it uses, and a sequence of steps to cause it if it doesn't happen on startup? I haven't been able to reproduce it.

[apsoto]

I see invalid byte sequence in UTF-8 (ArgumentError) with the following text

# DÌaz

Example:

$ solargraph --version
0.39.17
[/tmp/foo git:() ]
$ ls
fails.rb*
[/tmp/foo git:() ]
$ cat fails.rb
# D�az

[/tmp/foo git:() ]
$ solargraph scan
Traceback (most recent call last):
    15: from /Users/apsoto/.rvm/gems/ruby-2.7.0/bin/solargraph:5:in `<main>'
    14: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
    13: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
    12: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
    11: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
    10: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/shell.rb:172:in `scan'
     9: from /Users/apsoto/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
     8: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/shell.rb:173:in `block in scan'
     7: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:170:in `load'
     6: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:67:in `catalog'
     5: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:67:in `each'
     4: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:85:in `block in catalog'
     3: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map.rb:156:in `map'
     2: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:50:in `map'
     1: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:27:in `map'
/Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:201:in `process_comment_directives': invalid byte sequence in UTF-8 (ArgumentError)