Neovim: clipboard=unnamed: block copy/paste

[ghost]

I recently recompiled from source and block based editing and pasting is entirely screwy.

For example with set ve=block and in ^V mode

This  []
Is    [] 
A     []
Column[]

Puting your cursor in that column then hitting I then adding a comma inserts the comma only on the first line. However doing an append adds commas to every line.

Pasting is also screwed up, when selecting a block and pasting it in another space neovim adds new lines instead of pasting blockwise. So if we selected the first column in the above text and tried to paste it as second column we get this.

This    This
Is
A
Column
Is
A
Column

Instead of

This   This
Is     Is
A       A
ColumnColumn

Prepaste in block mode reformats the block into something entirely grotesque by adding spaces like so

This 



Is
A
Column

And doesn't even paste the buffer contents.

[ghost]

This only happens when you have :set clipboard+=unnamed.

[fmoralesc]

I can confirm.

This is becaue the current method for handling the clipboard is naive, and it uses the default text clipboard target, instead of what vanilla vim does, which is to encode into the "_VIM_TEXT" and "_VIMENC_TEXT" targets what sort of selection was passed, either block or normal. In my tests, in _VIMENC_TEXT a selection made in block mode is appended with \x02utf-8\x00, while a normal visual selection is appended with \x00utf-8\x00. Neither xclip nor xsel support special targets (and I doubt pbcopy does), so the long term solution would be to implement more featureful platform-specific clipboard handlers, and give nvim a way to handle this extended clipboard data format.

To play with this:

from gi.repository import Gtk, Gdk

c = Gtk.Clipboard.get(Gdk.SELECTION_CLIPBOARD)
d = [c.wait_for_contents(Gdk.Atom.intern(target), False) \
                                     for target in ["_VIM_TEXT", "_VIMENC_TEXT"]]
print(d)

[justinmk]

Nice investigating @fmoralesc.

In my tests, in _VIMENC_TEXT a selection made in block mode is appended with \x02utf-8\x00, while a normal visual selection is appended with \x00utf-8\x00. Neither xclip nor xsel support special targets

I remember seeing this in the source, and was worried about it. To make sure I understand, the "non special" clipboard text that is shared with other processes, is _not_ appended with the special marker, correct? (Otherwise it would corrupt the text for non-Vim processes.)

We could store an internal flag that remembers the selection type, so that at least the most annoying and visible case would be fixed (yanking and pasting using the same nvim process). We could even get fancy and broadcast that information to other nvim processes (https://github.com/neovim/neovim/pull/1302). That wouldn't fix the problem if all nvim processes are shutdown, nor for non-local processes. But it would be a fairly low-cost band-aid.

[fmoralesc]

@justinmk I wrote a clipboard manager once, I know my clipboards ;)

To make sure I understand, the "non special" clipboard text that is shared with other processes, is not appended with the special marker, correct?

Yes. The X clipboard contents are actually a list of "targets", which contain the real data (this is actually requested from the sources if possible, or if the sources haven't actually requested to store it in the server, which is why in ancient times clipboard contents were lost once applications stopped). Most programs provide at least a plain text target (which is what xclip and xsel use), but it is the clients responsibility to process it (the data can be arbitrary). So, even if vim uses a custom target, it the client doesn't process it, it doesn't matter. Some applications use conventional methods to extend the clipboard functionality; for example, when copying files, GNOME actually sets a target with a list of URIs. I created a patch to KeepasX to set a flag that indicated that some text in the clipboard was a password, so the clipboard manager could delete it after some time.

I _think_ we already use an internal flag for when the unnamed register is not used, we should look into that.

Perhaps building our own clipboard setter and getter with support for vim's _VIMENC_TEXT convention wouldn't be a bad idea; that would provide us with clipboard-compatibility (ja!) with vanilla vim.

(I didn't know about #1302, that might be helpful for rewriting macros/editexisting.vim, which I use all the time in gvim).

[bfredl]

Perhaps building our own clipboard setter and getter with support for vim's _VIMENC_TEXT convention wouldn't be a bad idea; that would provide us with clipboard-compatibility (ja!) with vanilla vim.

That would probably be the best :+1: In the meanwhile, we could run xsel (not sure of pbcopy) in nofork mode as a job and be able to detect when pasting, if this instance still "owns" the clipboard (and then use an internal representation with correct regtype), so block yank/paste at least work locally.

[fmoralesc]

I wrote a prototype.

It uses both gtk2 and gtk3, because for some reason the gtk3 bindings don't have the ability to set arbitrary contents in the clipboard (at least, I couldn't make it work). Anyways, I couldn't make it copy text :/ (it does copy, but then it doesn't store the data); the copy operation should probably be embedded in a Gtk main loop.

[fmoralesc]

Update: I managed to make it copy and save text to the clipboard, but I still can't make it set the _VIMENC_TEXT target. As-is, this would be feature complete replacement for xsel, so perhaps I (or someone else) should just take the plunge and port it to C. The Gtk clipboard implementation is good to have, because it should already support the wayland clipboard. I'm not sure other toolkits do this yet (probably yes).

[fmoralesc]

I tried to write the thing using PySide, and setting the _VIMENC_TEXT target works... but then it won't store the clipboard.

#!/usr/bin/env python3

import sys
import PySide.QtCore
import PySide.QtGui

def set(data):
    a = PySide.QtGui.QApplication(sys.argv)
    c = a.clipboard()
    mimedata = PySide.QtCore.QMimeData()
    mimedata.setText(data)
    mimedata.setData("_VIMENC_TEXT", data)
    c.setMimeData(mimedata, mode=c.Clipboard)
    timer = PySide.QtCore.QTimer()
    timer.start(500)
    timer.timeout.connect(lambda: a.exit())
    a.exec_()

if not sys.stdin.isatty():
    data = sys.stdin.read()
    if data not in ("", None):
        set(data)

[fmoralesc]

The latest iteration of the clipboard handler is at https://gist.github.com/771eefee45eefd3d1a2b. It is now written in vala (because I didn't want to try with C). I still don't manage to make it store the clipboard, but everything else is OK. It would be great if someone else could take a look at this, I'm working on other stuff this weekend.

EDIT: it also needs a method to encode the string in _VIMENC_TEXT's format.

[bfredl]

I had problems with store before when I tried to write a pygtk3 provider for an earlier iteration of the provider infrastructure. Something to try instead could be to fork to the background (or run it as a nvim job) and run the gtk eventloop (until an owner-changed event when someone else takes over the clipboard)

I also tried running xcopy/xsel in nofork mode as a job earlier today (to be able to know that this instance still owns the clipboard) but it seems I can't send the neccessary EOF to the stdin of a job (needed for xcopy to start accepting readers)

[fmoralesc]

Yes, probably a demon would work fine. Still, xsel manages to set the clipboard, I think it would be good to take a look at what it does.

[justinmk]

Dependency on python or gtk, or linking to xorg, is really something we should avoid. A gtk or python tool is good to have as an option, which is why it is a goal to make it easy for users (or plugins) to swap in alternative providers. But we we still need to provide an out-of-the-box half-solution for users who don't have gtk or python.

We can fix the most salient problem (for the current nvim process) by tracking the clipboard type internally.

An almost-complete solution could be to have a global user data key-value store: https://github.com/neovim/neovim/issues/1180. Then, the only remaining problem AFAICT is pasting from clipboard into a remote (ssh) nvim.

[justinmk]

We could ship the gtk tool with neovim, and have the provider select that if gtk is available.

[fmoralesc]

We could ship the gtk tool with neovim, and have the provider select that if gtk is available.

@justinmk That's what I was thinking of. I made the tool take very similar commandline arguments to what xsel and xclip have for that purpose. It's also the reason why I ported the utility to vala, which allows native compilation. The current binary is pretty small on x68_64 at 21k (smaller than xsel, actually).

[fmoralesc]

If we port this utility to xcb, the tool itself would only have X as a runtime dependency. But first, we have to build something that works :p

[fmoralesc]

Also, neovim needs a way to detect the _VIMENC_TEXT target, because currently it treats all clipboards the same (just tried pasting a text block from gvim into nvim).

[fmoralesc]

Disregard the last comment; this is responsibility of the clipboard provider. How can it implement block paste, though? Should it recompose the lines to be affected?

[justinmk]

How can it implement block paste, though? Should it recompose the lines to be affected?

s:try_cmd() in the provider should return the type (c, l, or b, per :help setreg()). Also I just noticed that s:clipboard.set() accepts a regtype parameter, but not using it yet.

For handling the regtype, we should be able to re-use the existing logic for other non-clipboard registers to "recompose" the text. We also need to make sure that setreg() and getreg() are handling the regtype correctly for clipboard (I believe they are, @bfredl wrote tests for this).

[fmoralesc]

I just wrote a function that pastes the + register in a blockwise manner, feel free to improve it:

function! BlockPaste()
    let c = split(getreg('+'), '\n', 1)
    let lenght =  len(c)
    let start = line('.')
    let end = start + lenght -1
    let col = col('.')
    let op_lines = range(start, end)
    let idx = 0
    for line in op_lines
        let line_orig = getline(line)
        let head = line_orig[:col-1]
        exe "let tail = line_orig[".col.":]"
        if len(head) < col
            let spaces = col - len(head)
            let head = head.repeat(' ', spaces)
        endif
        let new_line = head . c[idx] . tail
        call setline(line, new_line)
        let idx += 1
    endfor
endfunction

I imagine we could add a check on s:clipboard.get to call something like this in the case a flag has been set to indicate the register is in block mode.

[bfredl]

the clipboard provider can return [ [ _lines_ ], 'b'] (or any regtype) to indicate a blockwise paste. Not sure what you mean with "recomposing" ?

[fmoralesc]

I meant, to actually apply the blockwise paste (like my BlockPaste function does). But I didn't see this was already handled elsewhere.

[bfredl]

Right now, It is _almost_ the case that clipboard.get should return [lines, regtype] where these two are the arguments to clipboard.set, for a "picture perfect" copy/paste. The provider infrastructure will add an extra newline for a linewise set that won't get removed if get actively specifies the regtype to l. But if we both support the "text" and "vim" clipboard format, it is perhaps clearer if all this logic is handled in the vimscript provider (so that "almost" is removed)

[justinmk]

There should be logic in the C source to do that (str_to_reg ?). But all the provider needs to do is return the type from s:clipboard.get() and let the nvim core paste it appropriately, like any other register. s:clipboard.set() just needs to store the regtype in a script-local s:regtype variable, in case it is not using a target-capable clip tool.

[fmoralesc]

I'm taking a look at do_put() in src/nvim/ops.c and it seems like nvim the unnamed register logic removed is not that extensive. Perhaps it could be implemented usint the provider. At the start of do_put, this was removed:

#ifdef FEAT_CLIPBOARD
    /* Adjust register name for "unnamed" in 'clipboard'. */
    adjust_clip_reg(&regname);
    (void)may_get_selection(regname);
#endif

Now, adjust_clip_reg is this:

/*
 * Adjust the register name pointed to with "rp" for the clipboard being
 * used always and the clipboard being available.
 */
    void
adjust_clip_reg(rp)
    int     *rp;
{
    /* If no reg. specified, and "unnamed" or "unnamedplus" is in 'clipboard',
     * use '*' or '+' reg, respectively. "unnamedplus" prevails. */
    if (*rp == 0 && (clip_unnamed != 0 || clip_unnamed_saved != 0))
    {
    if (clip_unnamed != 0)
        *rp = ((clip_unnamed & CLIP_UNNAMED_PLUS) && clip_plus.available)
                                  ? '+' : '*';
    else
        *rp = ((clip_unnamed_saved & CLIP_UNNAMED_PLUS) && clip_plus.available)
                                  ? '+' : '*';
    }
    if (!clip_star.available && *rp == '*')
    *rp = 0;
    if (!clip_plus.available && *rp == '+')
    *rp = 0;
}

(notice it doesn't seem to link X related functions). may_get_selection() is:

/*
 * When "regname" is a clipboard register, obtain the selection.  If it's not
 * available return zero, otherwise return "regname".
 */
    int
may_get_selection(regname)
    int regname;
{
    if (regname == '*')
    {
    if (!clip_star.available)
        regname = 0;
    else
        clip_get_selection(&clip_star);
    }
    else if (regname == '+')
    {
    if (!clip_plus.available)
        regname = 0;
    else
        clip_get_selection(&clip_plus);
    }
    return regname;
}

Perhaps we could replace clip_get_selection with a call to the provider functions.

[bfredl]

@fmoralesc not sure if it Is related to what you're considering or not, but note #1814 will change the unnamed logic somewhat

[fmoralesc]

I was just checking if neovim hadn't removed code which could be useful here.

[fmoralesc]

I modified the clipboard provider so it uses vim-clip if available. This manages to detect if the register is in block mode. The relevant commit is https://github.com/fmoralesc/neovim/commit/cab2430c78aa86127cee8c6ed235f5f3fff568f9. I'm not clear about what to do to indicate neovim the register type, so it pastes in block mode.

A x86_64 version of vim-clip s available here, for testing.

EDIT: As of https://github.com/fmoralesc/neovim/commit/6810ebe3f825017b32e60dec577afbe6fdac2ecd (it was really simple, in the end) block paste works with vim-clip.

[fmoralesc]

OK, I've updated my patches and block paste works both within neovim and from vim. Yanking to vim is still unsupported.

The provider changes are at my vim-clipboard branch and the lastest version of vim-clip is available here. I think the changes to the clipboard provider should work with xsel et al, but I haven't tested that through.

[bfredl]

@fmoralesc Good work! :+1:
I think, the detection of regtype still can be improved somewhat (in the vim format case, the header should also be used to to distinguish between char- and line-mode pasting, autodetect is an heuristic specifically made for the plaintext format)
When I get time, I will try making yanking to vim format work also.

[fmoralesc]

@bfredl, for yanking to vim only thing needed would be to fix vim-clip so it sets the targets correctly, and then replace xsel et all with it. I couldn't make that work, but I didn't give it that much time. I found xsel daemonizes itself to own the clipboard, so that is what vim-clip should do too (as it is, it will wait until someone else owns the clipboard).

[fmoralesc]

I've managed to make vim-clip set all the targets correctly and the data gets stored. Only problem now is the clipboard data won't be registered in other applications.

[bfredl]

You mean non-vim apps cannot paste the data copied with vim-clip ?

[fmoralesc]

Yes, that is the problem. I think something is missing in the way vim-clip puts data in the clipboard. vim-clip itself can recover it (so copying and pasting within neovim and copying from vim works), but other programs can't:

$ echo "hello" | vim-clip -s -c
$ vim-clip -g -c
hello
$ xsel -o -b
$

[fmoralesc]

I opened an issue at xsel's repo, let's hope we get a response.

[bfredl]

I am looking into on solving this using some improvements to job control and some modifications to xsel.

[iperry]

Sorry to poke, it's very hard to keep track of what's going on re: clipboard. Is this still actively being worked on? I've had to disable clipboard=unnamedplus or clipboard=unnamed because block operations don't work as they did on vim.

[bfredl]

@iperry Block editing should work locally even if clipboard=unnamed/plus on X11 (but not on OSX yet). I will rebase #2077 soon to make it work across instances on X11, ( but someone developing on OSX need to chime in to fix it on OSX )

[wjakob]

Hi -- I can confirm this as well. After switching from VIM to NVIM, block pasting inserts unwanted newlines.

I can only get back the expected behavior by removing set clipboard=unnamed from .vimrc.

[dphase]

Has anyone come up with a workaround for this on OS X? I didn't realize how often I use block pasting until I stumbled across this issue.

[alok]

I'm also desperate for a fix. I use block-pasting a lot and I use the unnamed clipboard even more.

[bfredl]

I hope to find time to work on #4523 again soon... Until then, you could try my nvim-miniyank plugin that try to fix this issue (at least in the most common usage patterns).
You could add to your vimrc:

Plug 'bfredl/nvim-miniyank'
map p <Plug>(miniyank-autoput)
map P <Plug>(miniyank-autoPut)

and try to use ^Vy and p normally with clipboard=unnamed (both within and between instances)
I recently wrote this part of the code, so there could be bugs, but I would be happy for any feedback.

[alok]

@bfredl That fixed my problem. I only really use the default register, so I switched your suggestion to use the startput maps. Thanks!

[dumblob]

I'm sorry I know it's a rude question, but I'm stumbling across this issue for quite some time under Windows and would like to ask whether there is anyone working on the support for register types in clipboard or at least on the support for Windows for the nvim-miniyank plugin.

[bfredl]

It is not rude. Fixing windows support for nvim-miniyank should not be too hard, I think we just need to find a good default path. It should really be a "runtime" path, but if there isn't any we can use a "cache" path.

[dvcrn]

I would like to chime in. Is there a possible fix on the horizon? I'm using miniyank now as well but this really should get fixed in core.

I didn't even know that the behavior in neovim wasn't normal until a couple days ago

[janlazo]

Is https://github.com/vim/vim/tree/v8.0.0522 relevant?

[thecodewarrior]

This was kind of mentioned before, but I think a simple—and honestly pretty sturdy—solution would be to maintain the value of the internal register and only overwrite it when the system clipboard has changed. That way you could copy/paste within vim without issue, and if you copied something outside of vim, the next time you try to use the clipboard in vim it would overwrite its internal register (and the selection type) with the new data. It would probably be good to have the "annotated clipboard" as part of some shared state, so you can copy/paste between processes, but that seems relatively simple.

Yes it would be more ideal to be able to use the system clipboard, but I really don't see much of a benefit, and the cost seems to have been four years and counting of having broken copy/paste. I may not understand something under the hood that makes this difficult, I may not be aware of things going on in the background, but this seems like a large and pretty simple to solve issue.

[justinmk]

@thecodewarrior clipboard works just fine except for the very specific case of <c-v>. Calling this "large" and "broken" is an exaggeration.

Nothing is blocking it except lack human attention. Lots of humans leaving comments here, we need less of that and more actual work :)