Kakoune: kak always writes files with a newline at the end

[shachaf]

$ printf test > file
$ xxd file
00000000: 7465 7374                                test
$ kak -e 'wq' file
$ xxd file
00000000: 7465 7374 0a                             test.

Opening a file and writing immediately shouldn't modify the file.

Vim keeps track of whether a file ended with a newline on opening with an "endofline" flag. Kakoune could presumably do the same thing.

[mawww]

Kakoune is a code editor, not an arbitrary data editor. Adding an option to control that would be relatively simple, but I think it would go a bit against Kakoune's principles. Unix text files are expected to end with an end of line character, its usually an error when they dont.

The motivating example is not strong enough to convince me we need to support that. Kakoune does not guarantee that opening a file and writing it should be a no-op (should it touch the file modification date ? should it change the inode number as suggested in another issue ? I dont think Kakoune should make any promises about any of those).

That said, you could argue that BOM and end of line format support are already precedents in your requested direction, which is true, but I added them because I had to work on source files where those needed to be preserved. I am not sure we have real source code where the last line needs not to end with an EOL.

[shachaf]

There's plenty of source code that doesn't end with n, though. I'm not sure which editors do that -- and they probably shouldn't -- but I run into it regularly. The files should probably be fixed but that's a separate issue.

More generally, even though this isn't primarily a code feature, I'd like to be confident that if I open a file and make no changes, or only make changes in a specific area, that the rest of the file doesn't get changed unexpectedly. Also note that writing empty files isn't supported right now (though looking at vim's handling for empty files, it seems like a special case anyway).

It's also a useful feature of vim that you can edit or examine binary data with :%!xxd and :%!xxd -r, though this is obviously not the primary use case.

This seems like a small feature -- and the only change necessary? -- to be able to open and modify an arbitrary file without causing unexpected changes.

[tmccombs]

fwiw, editorconfig does have an option insert_final_newlinewhich enforces either always ending with newlines _or not_.

[lenormf]

Related #3669. Seems like once the choices have been made, we'll have to document them.

[JJK96]

I often have changed files in git where the only change is a newline at the end as a result of this behaviour.
I would prefer to change this behaviour as suggested by this issue.

[lenormf]

If the premise that Kakoune is a code editor and is used to edit text files is respected, then the behaviour is consistent with what is expected by the standard:

A text file, under unix, consists of a series of lines, each of which ends with a newline character (n). A file that is not empty and does not end with a newline is therefore not a text file.

I don't know the reason why other editors wouldn't terminate all text files with a newline character, or why users wouldn't want that standard behaviour to be applied to source code, but if Git is showing \ No newline at end of file, then it's the contents of these files you need to fix.

[shachaf]

Kakoune can make whatever decision it wants on this feature, but these arguments don't seem very relevant. A text editor is a tool that I use to deal with (mostly) text files, but I need to deal with text files as they are, not as a standard wants them to be. They might not be Unix-related files (e.g. Kakoune deals with rn fine), so Unix standards don't get that much of a say here -- the question is how it should deal with files that occur in practice. In practice there are source code files that don't end in n everywhere, so "if you edit these files you must modify the last byte" is a strong stance to be taking.

A quick search shows noeol source files in cpython, CoreCLR, gdb, gtk, LLVM, SDL, vscode, etc. repositories. There are also many language grammars that allow files without n, whatever POSIX might say.

(I should also note that, if Kakoune keeps not supporting this, it just means using vim a bit more often for me. My preference is to be able to use one tool rather than two.)

[lenormf]

Your "the argument is not relevant" can be replied with the exact same dismissal.

If you don't want a trailing newline character in files edited with Kakoune, remove it before committing. Following whatever inconsistency other editors put on their users is not a relevant argument.

[mawww]

It is still unclear to me why you'd need to preserve the missing end-of-line of a file, and it looks like a can of worms to me.

Say we modify the last line while editing, is it then fair game to add the end-of-line character ? expected ? How about if we add an empty line at the end of the buffer, shall we drop that last line because the preceeding line did not contain an end-of-line at load time ? I dont think there is any good, consistent answer to those questions, so I am inclined to let the core of Kakoune do the simple thing.

I also suspect this is not that hard to implemement using hooks, something like (untested)

hook global BufOpenFile .* %{evaluate-commands %sh{ 
   if [ $(tail -c1 $kak_hook_param | wc -l) = "0" ]; then
       echo 'hook buffer BufWritePost .* %{ nop %sh{ truncate -1 $kak_hook_param } }'
   fi
}

To be honest, the pragmatic part of me says to just merge the PR and add that feature, but its quite conflicted with my minimalist part...

Could you maybe expand on what use cases you have for Kakoune where preserving that missing newline is really important ?

[occivink]

I was also confused about what kakoune _should_ do, but I think ultimately the "correct" behavior is to simply keep the style that the file had at opening, regardless of any changes. It's consistent and predictable.
Anything more elaborate also runs the risk of being difficult to understand.
It is still possible to implement fancier behavior with BufWritePre hooks which could change newline_at_eof, based on whether the last line is empty or not, ...

I personally don't really need this behavior, but the following arguments convinced me:

• the precedent is set by eolformat and BOM
• it's noninvasive, easily implemented and follows the same scheme as the other two options
• it's the only piece missing in making kakoune leave arbitrary files 'untouched' when doing open-write-quit
• it's part of editorconfig

[lenormf]

What should happen if a user with newline_at_eof enabled opens a new empty file? Should the editor have a main selection set over a character that doesn't exist in the file? Should the editor now open the door to zero-length selections?

There's no practical use to spending time answering those questions, which are asked solely because “other editors implement the feature, therefore Kakoune should too”.

[occivink]

Should the editor have a main selection set over a character that doesn't exist in the file?

Yes? It's not like you see a 100% faithful representation of the file content anyway: the BOM byte is stripped if it exists, and \r\n is turned into \n. What exactly is the problem in doing so?

[lenormf]

You can't have nothing selected. That's why always adding a newline character to a file keeps the paradigm consistent, there's always something selected.

[occivink]

The editing paradigm is completely independent from how files are loaded and saved on disk, I don't think it's relevant to the discussion. The newline at the end will still always be there when editing, it just might not be saved in the file depending on an option.
If you test out the PR you'll see that it doesn't change anything to the editing.

[lenormf]

The newline at the end will still always be there when editing, it just might not be saved in the file depending on an option.

conflicts with your previous statement:

the "correct" behavior is to simply keep the style that the file had at opening. It's consistent and predictable.

[occivink]

Ok, I thought it was obvious that this whole discussion was about how kakoune reads and writes files, but I suppose not.

[tmccombs]

It is still unclear to me why you'd need to preserve the missing end-of-line of a file,

• it should be possible to save an empty file
• sometimes I want to edit a file with a single line of text that I don't want to have a newline because then I would have to trim the newline before using with other tools
• you might not want to have the added newline in your git diff (or other vcs)
• a project might require files not to have trailing newlines, and you have to follow that.
• plugins that allow editing encrypted or compressed files have to remove the trailing newline (although this could be better handled another way)

[shachaf]

As far as empty files go, I'd suggest compatibility with vim, which saves a 0-byte file if you write an empty buffer (even though its internal representation is an array of lines, like Kakoune). This makes sense because n is a line terminator, and an empty file has no lines to terminate. Other editors I tested (nano, gedit, vscode, vis) have the same behavior, so Kakoune is the odd one out here in being unable to create empty files. This seems like good behavior regardless of whether the noeol patch is merged.

[tmccombs]

Kakoune is a code editor, not an arbitrary data editor.

Is the implication here that if you want to edit text files that aren't code you need to use a different editor?

Unix text files are expected to end with an end of line character, its usually an error when they dont.

What do you mean by error? It might technically not meet the posix definition of a text file, but most unix tools (grep, sed, awk, sort, etc.) handle missing newlines at the end of files fine.

[mawww]

Is the implication here that if you want to edit text files that aren't code you need to use a different editor?

For text files, no, as written in the design document, Kakoune should be good at editing general text as a consequence of its focus on editing code. for arbitrary data files, yeah, Kakoune might work well enough for some tasks, but editing non-text files is not really its focus.

What do you mean by error? It might technically not meet the posix definition of a text file, but most unix tools (grep, sed, awk, sort, etc.) handle missing newlines at the end of files fine.

error might not be the best word, but text files missing the last newline are often treated as-if they ended up with a final new-line, and when they are not it leads to surprising result, like wc -l returning 0 for a file containing a single line with no final new-line. I dont know of any case where adding that missing newline when writing the file breaks things.

I still miss a clear motivating use case for this feature, you posted some examples of things that could be desirable, but thats not really what I am looking for, I'd like to hear about a real world case where not having this option has been making your work harder or prevented you from using Kakoune for that task.

For example, I cant quite see why Kakoune should be able to create empty files that contain no bytes, touch already does a good job at that. I wonder if adding a final new-line would be problematic when editing say a self-extracting shell script (shell-script followed by some tar.gz data), this could be a reasonably strong case of that feature.

I undestand it might sound like I am giving you a hard time for a pretty small feature, but Kakoune C++ codebase is already far too complex for my taste and seeing how hard it is to remove features, I am trying hard to make sure new ones are properly motivated.

[caksoylar]

I'd like to hear about a real world case where not having this option has been making your work harder or prevented you from using Kakoune for that task.

Here is an example that might not be very convincing since it is just an extra truncate call (although that might not be available in all systems). This is a light plugin for reading and writing gzipped files, where it would be handy to have the option to not have the buffer contain the EOL: https://github.com/caksoylar/dotfiles/blob/master/kak/autoload/gzip.kak#L30 (Also would be nice to be able to manually set the modifiable flag.)