Wire: Editor: Add detection of mixed line-endings (EOL), and allow users to save&open files with different EOL

[CaptainPRICE]

Garry's Mod supports all the three major operating system platforms:

• Windows
• Linux / *nix
• Macintosh (Mac) / OS X

And all of the three systems uses a different sequence of characters for the end of a line (EOL), line-ending (_some call it "line-breaking"_)[¹]:

| Operating System | Line-Ending | ASCII | ASCII hex |
| :-: | :-: | :-: | :-: |
| Windows | \r\n CRLF | 13 10 | 0D 0A |
| Linux & OS X | \n LF | 10 | 0A |
| Old Mac OS | \r CR | 13 | 0D |

To find out on what operating system the game is running, see system library. There are three functions, self-explanatory (note: call it once):
system.IsWindows
system.IsLinux
system.IsOSX

See also jit.os (not a function), it contains the target operating system name...

So, the task-list:

• [ ] Allow users to change/convert line-ending for the opened file(s), there should also be an option to perform this from the file hierarchy, via both folder and file (right-click) context menu. There should be the 4th option here, "Dominant", which should be smart enough to automatically detect the most dominant line-ending. There should also be two sub-options to "Dominant", only if this is performed from a right-click context menu of the folder: "Global", it finds out the most dominant line-endings of all of the files inside of a selected folder/directory, and 2nd option: "Local", it finds the most dominant line-endings per file.
• [ ] Add line-ending indicator for the opened file, and allow it to be enabled or disabled via Editor options.
• [ ] Line-ending indicator should have the 4th display option, "Mixed". When the opened file has mixed line-endings, that's what the indicator should say..
• [ ] Editor should default to operating system EOL. So, files that has been saved on old Macintosh should have CR line-endings, Linux & OSX should have LF line-endings, and Windows OS should have CRLF line-endings, respectively.

I hope I have explained that clearly. 💤

---

_If no Wire dev will implement this, I will do it myself._

[Divran]

Are you sure this is actually a problem? I don't own a mac so I can't test it on there. I could test it on linux in my VM but my VM is slow af, so I would rather not. Thing is, we have not received a single complaint from anyone using mac or linux about any issue even related to this. In your post, you don't mention testing the issue to see if it actually exists. When reporting bugs, testing that the bug actually exists first is pretty standard stuff.

I believe \r counts as whitespace (I know \n does) in Lua patterns so it should be ignored by E2s compiler just fine.
The E2 editor also strips out all \r in the file when the code is retrieved from the editor and sent to the E2 chip: https://github.com/wiremod/wire/blob/master/lua/wire/client/text_editor/texteditor.lua#L345-L347
It doesn't appear to strip out the \r in the file when loading the file, though (only when spawning the E2 chip), so that could possibly cause issues when viewing the code in the E2 editor, just after opening a file. But it would have to be tested to confirm.

The only issue I've seen related to using E2 on different operating systems was when someone using mac changed the font of the editor to a font that didn't exist on their computer, which is unrelated to this.

[jfmherokiller]

I have a mac and line endings are not a problem at all

[oldmud0]

Uh, no. Keep things simple.

[CaptainPRICE]

Are you sure this is actually a problem?

Well, let's find out... I'll perform throughout testing tomorrow (on all systems), I am 😴 atm. (I'll edit this comment, and provide my results then.)
In the meantime, people, please provide your test results below. But, first make sure you're doing this on the server which is running the latest version of wire of course (clone the master branch from GitHub).

Test Case 1.

1. Create three new text files, each one with different line-endings.
eol-windows.txt:

@name EOL-Windows\r\n
2nd line on Windows\r\n
3rd line on Windows\r\n
4th line on Windows\r\n

eol-linux-osx.txt:

@name EOL-Linux and OS X\n
2nd line on Linux and OS X\n
3rd line on Linux and OS X\n
4th line on Linux and OS X\n

eol-old-mac.txt:

@name EOL-Old Macintosh\r
2nd line on old Macintosh\r
3rd line on old Macintosh\r
4th line on old Macintosh\r

2. Save those text files into ./Steam/steamapps/common/GarrysMod/garrysmod/data/expression2 folder (create expression2 folder manually, if it does not exists).
3. Have friend(s) on the server you are on (allow the prop-protection on the both sides, if any is installed). Also, make sure to note-down the operating system of your friend.
4. Get/Select Wire Expression 2 tool, press Update button below file list/hierarchy, while holding down spawn menu key (by default Q), and
4.1. Search for the text files you have saved previously (in the file list/hierarchy).
6. Select (left-click) a file which has Windows line-endings (eol-windows.txt), and then left-click with toolgun to spawn E2 chip.
7. Now, have your friend download the E2 chip you have spawned, by right-clicking it. Then ask your friend, if all of the lines are displayed correctly in the Editor (would be nice to have screenshots).
8. Tell your friend to save a file (either by clicking a save button, or pressing CTRL/⌘+S while the editor is open).
9. Examine the EOL sequence of the file, and then make sure that it was not changed, simply by comparing the two.

Repeat the steps 4.1. - 9., but on the step 6., instead of uploading eol-windows.txt file again, use: eol-linux-osx.txt and eol-old-mac.txt files. Finally, report your results here.

---

My results on the first test case (click on the thick for a screenshot):

On Windows

• Windows line-endings [ [✖️](https://cloud.githubusercontent.com/assets/9789070/19621831/37771458-989b-11e6-87d4-c2a19ec5765d.png) ]
• Linux / OSX line-endings [ [✔](https://cloud.githubusercontent.com/assets/9789070/19621835/4a8b67a6-989b-11e6-86cf-ec9b95f9ea56.png) ]
• Old Macintosh line-endings [ [✖️](https://cloud.githubusercontent.com/assets/9789070/19621836/59fd7544-989b-11e6-8313-debc545b6352.png) ]

On Linux

• Windows line-endings [ ✖️ ]
• Linux / OSX line-endings [ ✔ ]
• Old Macintosh line-endings [ ✖️ ]

On Macintosh OS X

• Windows line-endings [ ✖️ ]
• Linux / OSX line-endings [ ✔ ]
• Old Macintosh line-endings [ ✖️ ]

The results are obvious at this point. It always uses \n (Line Feed) character. So, there you go, @Divran & @TomyLobo, a problem has been already presented, and it is just as I have thought. For example, if you would open eol-windows.txt text file on Windows OS (via Notepad), you will see this:

Because, once the Editor saves the file, it will have LF line-endings no matter what OS.

[CaptainPRICE]

I have a mac and line endings are not a problem at all

Can you try/test the first case?
There is no line-ending indicator, editor doesn't let you save files with different line-endings, and we don't know how does editor handle mixed line-endings, etc. I believe it misbehaves at least at some..
Any reason why did you downvote this todo?

[TomyLobo]

As all unixes, mac os x uses \n
mac os 9 and below used \r

So, can you correct that in your OP? :)

Also, I agree with @Divran here in that you should present a problem, not a solution without a problem.

[AbigailBuccaneer]

Old Mac isn't relevant. We should certainly use different line endings on Windows and Unix. I'd prefer to keep it simple - we should use native line endings (do we at the moment?), and maybe it's worth converting files to consistent native line endings (which I assume we don't), but I don't feel it's useful to offer settings for non-native endings. KISS and all that.

[CaptainPRICE]

but I don't feel it's useful to offer settings for non-native endings

Nah. I ain't keeping this one simple, perhaps assign me as I am nearly done (or don't if you want it simple).
I have some plans about the Editor layout, that I will try to do soon.

P.S. I have updated my comment above.
Also, cut/copy to clipboard always uses CRLF line-endings, this just so wrong...

[Divran]

By far the easiest solution would be to just remove all \r when the file is opened, and not only when the file is retrieved from the editor (which it does currently, see my other reply above). This means that as soon as a file is opened in the E2 editor, all \r are gone, and if they save the file again, it will overwrite their file with all \r gone (unlike the current behavior where you must spawn the E2 into the world, then right click it, and then save the code, in order to clear all \r from your saved file).

Then, if a user wants to edit the E2 outside gmod, it would be up to them to use an editor that can handle \n in any os. Pretty much all editors nowadays handle this easily.

This is the easiest solution, since you only need to change one single line of code in the E2 editor. Whether it's the best solution is another question.

[AbigailBuccaneer]

Can we take an informal poll?

• React to this comment with :smile: if you think we should keep it simple and save with consistent \n line endings (but support loading files with any line endings).
• React with :tada: if you think we should keep it simple and save with consistent native line endings (but support loading files with any line endings).
• React with :-1: if you don't think either of these two routes are appropriate and you'd like something else (eg. an interface for changing the current file's line endings)

[Divran]

I voted 😄 but I'm fine with 🎉 as well. I think 👎 is overkill; overly complicated.

For reference, here's how the most popular text editors manage this (keep in mind I only googled for a minute or less for each editor because I'm short on time right now)

• 🎉 Sublime text drops all \r when it opens the file, and then saves with OS native line endings. Sublime text has many, many plugins which can do anything you can imagine with line endings (including plugins that display the current line endings, and converter plugins).
• 🎉 A quick search on google for Atom editor seems to indicate it also converts and saves with OS native line endings. I haven't found anything that shows how it loads files though. Atom also has many plugins.
• 👎 I couldn't find as much solid info for Notepad++ as for the others, but it appears that it detects which form of line endings you use for each opened file (it then uses the detected form if you add new lines to the file), and then it won't auto convert at all when you save the file. It saves the file with whatever line endings it had when opened. It does however have many built in options for displaying the current line ending type and it also has a built in converter. I believe Notepad++ also has options to override this behavior, to force a certain form of line breaks, always. Notepad++ most likely has plugins for this as well, but it already has most of it natively so you don't really need plugins for this.

Seems like 🎉 is the most popular option, I guess?

[Nebual]

This is absurd, OS specific line endings are a horrible relic of software history, like endianness. Users should not be allowed to change line endings or realize they exist at all, as E2 is a whitespace irrelevant language, and there's zero benefit to using os native endings. I strongly vote for only using \n for saving, and then converting anything on load to using just \n.

:D (or however you type that smile)

[Codingale]

I think it's not needed, basic external editors such as notepad in windows break yeah, but not good editors such as NotePad++, let's be honest who uses notepad for actual coding now?

[CaptainPRICE]

Even Windows Notepad handles it better than Wire Text Editor, because it doesn't force \n EOL. It is just that display is wrong, but EOL characters (\r and \n) are still present. See my comment here.

Notes for later:
New files: Default to native/OS line endings (or fallback to user preference/settings).
Existing files: Never normalize without prompting the user. Detect EOL (smart)! Mixed should have higher priority over dominant detection.
On paste: Use the same approach/detection as with existing files!