Ace: Cannot make ACE detect or use unix newlines

[docwhat]

Hi!

I use GitLab and when using ACE to edit files in git, it keeps saving with windows-style newlines.

This makes the git history nearly useless as well as breaking things that don't know what do with the \r character (e.g. Augeas). See gitlabhq/gitlabhq#3982 for the history.

I know it is ACE but I don't know _why_ it is happening.

I've tried (via the console in chrome) things like:

• editor.session.setNewlineMode('unix')
• editor.session.getDocument().setNewlineMode('unix')

... but it still sends it back to GitLab with \r\n as the line ending.

I've read through the API docs and googled a bunch but cannot find a solution to this problem.

Is this a bug in ACE or is gitlab misusing ACE?

Ciao!

[nightwing]

Hi

I think it's highly unlikely that this is an issue with ace.
Can you test it with following code?

var session = editor.session
session.setNewLineMode("unix"); // same as session.setOption("newLineMode", "unix");
JSON.stringify(ace.session.doc.getNewLineCharacter()); // == '"\n"'
session.doc.getValue().indexOf("\r") == -1

also it would be helpful if you could give an url to example page where i could try to reproduce the problem

[docwhat]

Alas, our gitlab is private.

I don't know what else it could be because it goes out of the server without \r but has them when it comes back.

When I get home I'll see about running your test code.

[docwhat]

Results of your code:

session = editor.session
// => undefined
session.setNewLineMode("unix")
// => undefined
JSON.stringify(session.doc.getNewLineCharacter())
// => ""\n""
session.doc.getValue().indexOf("\r") == -1
// => true

And when I save it, it is received by Rails as containing \r\n for EOL.

I'm still poking at this, but it is driving me nuts.

[nightwing]

look for the code which uploads file
either it replaces \n with \r\n or browser does that automatically.

[docwhat]

Here's the code that puts the text into a hidden field for uploading:

https://github.com/gitlabhq/gitlabhq/blob/5-4-stable/app/views/edit_tree/show.html.haml#L42-L45

It isn't doing anything to change the line endings. Why would a browser do this change?

[docwhat]

Okay, I created an example gist with a php file that demonstrates the error.

https://gist.github.com/docwhat/5992954#file-ace-windows-newlines-php

Hopefully this will help us figure out what is going wrong.

Ciao!

[nightwing]

It would be better to create a test with regular textarea, since we've seen it's not caused by ace.
There are many ways in which text on windows can be converted to '\r\n', that's why git have autocrlf option.
The right solution here is to check on the server if the file already contains windows line endings, if not convert it to linux format.

[docwhat]

There are many ways in which text on windows can be converted to '\r\n', that's why git have autocrlf option.

But windows isn't involved in any way -- The server is CentOS6 and the client is Chrome on OS X.

I'll try changing my example to using a hidden textarea instead and see if that helps.

[docwhat]

Nope. Running my test page on OS X with Chrome on OS X I still get the \r characters on submit.

I sort-of agree it isn't Ace's problem, since it makes it to the textarea without \r being added. But something weird is going on here and if we can figure it out, it'll make a good FAQ item for Ace so others don't have this problem.

[docwhat]

Okay, so I know what's going on and I feel stupid for not knowing this:

When you POST a form, it uses the MIME type application/x-www-form-urlencoded. According to the W3 Form Content Types a browser should always encode newlines as \r\n.

So that's where the \r\n is coming in. I guess I never noticed before.

The solution is to have the server application do the "Right Thing" with the data that comes back, which might
include replacing \r\n with \n.

Thanks, guys. Closing this issue. Hopefully anyone with a similar problem will find this issue helpful.