Truffleruby: [!] There was an error parsing `Gemfile`: cannot instance_eval() a String with binary encoding, with no magic encoding comment and containing a non-US-ASCII character: \xC3. Bundler cannot continue.

[guizmaii]

Hi,

I was trying TruffleRuby for the first time and encountered a problem (bug?) really fast.

I installed the GraalVM / TruffleRuby 1.0.0-rc5 on a up to date MacOS, installed bundler and finally ran bundle install on my Ruby project.
Here's the answer I received:

[!] There was an error parsing `Gemfile`: cannot instance_eval() a String with binary encoding, with no magic encoding comment and containing a non-US-ASCII character: \xC3. Bundler cannot continue.

 #  from /Gemfile:1
 #  -------------------------------------------
 >  source 'https://rubygems.org'
 #  
 #  -------------------------------------------

Any idea how to fix this ?

[guizmaii]

Ok I've found the source of the bug:

# Easy google map intégration for rails

with a french é

Do you consider this a bug or should I close this issue ?

FYI, If I switch to Ruby 2.4.4 (rvm use 2.4.4) and launch bundle install on the Gemfile containing the é, bundler starts correctly.

[eregon]

Hello,
Thank you for the report.

Could you tell me which Bundler version you used? (bundle --version after rvm use truffleruby)
I think Bundler 1.16.2 has a bug with the encoding used to read a Gemfile, which might be the cause of this bug.
Bundler 1.16.3 might work better.

Could you also share your Gemfile or the part around the comment with the é?

[eregon]

Do you consider this a bug or should I close this issue ?

It's definitely a bug, we should behave like MRI.
But the issue might be in upstream Bundler.

[eregon]

This seems a variant of https://github.com/bundler/bundler/issues/6598 but with non-ASCII characters in the Gemfile and not a .gemspec. I have a local fix for Bundler to avoid this issue.

As a workaround, you can add this line at the top of the Gemfile:

# encoding: UTF-8

[eregon]

@guizmaii I opened an issue and a PR to Bundler (see links above).

MRI still works in this case, and TruffleRuby doesn't but I believe fixing the problem In Bundler by giving a proper encoding is better here. The reason of the exception in TruffleRuby is TruffleRuby needs to convert to a Java String for debugging support and that is unreliable if we cannot know the encoding. And so the debugger wouldn't know how to display this é because it might not be always UTF-8 but could be anything as the String has the binary encoding.

Not having a proper encoding is problematic anyway:
If a Gemfile is read with the binary encoding as in the latest Bundler release, then the é would only display correctly in the terminal if the terminal also has a UTF-8 encoding. If not, it will be misinterpreted by the terminal and look like garbled text.
Similarly, any String containing non-US-ASCII characters in the Gemfile would have the same issue and only works if luckily the environment is setup with UTF-8 (which is the common case, but not the only possibility).

[guizmaii]

Hi @eregon,

Here's the info you asked:

➜  ~ ruby --version
truffleruby 1.0.0-rc5, like ruby 2.4.4, GraalVM CE Native [x86_64-darwin]
➜  ~ bundle --version
Bundler version 1.16.3

Could you also share your Gemfile or the part around the comment with the é?

No, I can't. I already shared the maximum I can. Sorry.
If you have specific questions, just ask. I'll evaluate if I can give you the info or not. ;)

Not having a proper encoding is problematic anyway:
If a Gemfile is read with the binary encoding as in the latest Bundler release, then the é would only display correctly in the terminal if the terminal also has a UTF-8 encoding. If not, it will be misinterpreted by the terminal and look like garbled text.
Similarly, any String containing non-US-ASCII characters in the Gemfile would have the same issue and only works if luckily the environment is setup with UTF-8 (which is the common case, but not the only possibility).

Thanks for the detailed answer !

[eregon]

Thanks for info.

No, I can't. I already shared the maximum I can. Sorry.
If you have specific questions, just ask. I'll evaluate if I can give you the info or not. ;)

No worry, I could reproduce with a Gemfile with the comment with é alone, so that's enough.

I think we just need to wait for Bundler to merge my PR and make a new release now.

[guizmaii]

Isn't it strange to fix bundler when we know that the bug is in Truffle ?
Aren't you afraid to have this problem somewhere else ?

Noob questions, sorry.

[nirvdrum]

I think @eregon is saying that there are issues with both projects. The issue with Bundler can manifest itself with MRI if your system's locale isn't set to UTF-8. These sorts of environmental encoding issues can be very tricky to debug so it's nice that we have an easy failing case here that we could contribute back upstream.

We'll see what we can do on the TruffleRuby side to address the issue so existing versions of Bundler work.

[guizmaii]

Ok thanks :)

[eregon]

@guizmaii Bundler 1.16.4 was just released and I confirm it works now for a Gemfile with:

# Easy google map intégration for rails
gem 'rake'

So just updating bundler should solve it: gem update bundler.

We'll fix the underlying issue as @nirvdrum said, but that's considerably more involved than a simple gem update bundler, so it might take some time.

[nirvdrum]

Unfortunately, other things got in the way and I haven't addressed the core problem here. Since it wasn't presenting itself in Bundler any longer though, I think we can close this issue out. I'm tracking the eval encoding issue in a separate issue in our internal tracker.

Please comment or re-open if you think I've closed this prematurely.