Client: Git LFS - cannot push files on windows

[kubalobo]

Hi!
I'm testing the new functionality from v5.1.0 - git LFS support.
I do everything what's said in doc (https://keybase.io/docs/git/index).
Everything is working for normal repo but when I try to push the LFS connected file I got an error:

git push
Initializing Keybase... done.
Syncing with Keybase... done.
Uploading LFS objects: 100% (1/1), 1.2 GB | 0 B/s, done
invalid character 'U' looking for beginning of value
error: failed to push some refs to 'keybase://team/xxx/Repo2'

Propably Uploading LFS objects part isn't done because it not take any time to upload file.

Is it a bug or maybe am I doing something wrong?

I'm on Windows 10.
My log id: 2dc714afb17faf0439897d1c

Thanks!

[ThePhxRises]

I'm having this same problem, looks to me like the invalid character 'U' part is some failure looking for an encryption key or something like that.

[strib]

What shell/command prompt are you both using on Windows?

Also, what exactly does this command show when you run it within your local checkout:

git config --get lfs.customtransfer.keybase-lfs.args

Thanks for trying out this new feature!

[ThePhxRises]

I was using git bash (which I believe is MINGw64)
that command returns
\"lfs origin keybase://team/[MYTEAMNAME]/[MYGITNAME]\"
What's in brackets is indeed correct and where I'm trying to send the files LFS uploads.

[strib]

Ah hrm. Maybe you don't need the quotes on git bash. Can you edit your .git/config file to remove both \" from that option and see if it helps?

[strib]

(I tested this in the normal command prompt I think.)

[ThePhxRises]

In the config file the "s have three slashes before them on either side.
\\\"lfs origin keybase://team/[MYTEAMNAME]/[MYGITNAME]\\\"
How much of this should be removed?

[strib]

Oh sorry. Both sets of \\\" should be removed for the test.

[ThePhxRises]

No luck, still the invalid character 'U' error.

[strib]

Hmm strange. The git config command above now shows it without any backslashes or quotes, right?

Also, what does this return:

git config --get lfs.customtransfer.keybase-lfs.path

I wonder if that contains backslashes or something else that the git shell can't handle.

[ThePhxRises]

Yeah the command from earlier now shows without slashes or quotes
git-remote-keybase
The new command outputs the above.

[strib]

Ok thanks. I am stumped for now, this kind of thing is hard to debug. Maybe you put the \\\"s back and try it under regular command prompt, until I get the chance to test it myself? I don't often get time on a Windows machine.

[kubalobo]

I tried it already with git bash, PowerShell and normal CMD - the error is the same everywhere.

[ThePhxRises]

Oh right, I tried GitHub Desktop as well, same error there.

[strib]

Thanks all. I just got some testing in on a Windows device, and you'll never guess this one simple trick to getting it to work...

Instead of completely removing each \\\" from the .git/config file, you can change them into \". That solved it for me.

I swear I tested the current code in CMD before and got it to work, so I'll have to go back and try to figure out how to make it work automatically in the next release. Sorry for the inconvenience. Let me know if the above helps.

[ThePhxRises]

That seems to have done it, LFS files are uploading now. Thanks a ton.

[kubalobo]

Yes that helps.
But I have another problem. :(
I have big (~5GB) repo and after uploading push hasn't finished.

I'm waiting already few hours but it seems stuck after:
Syncing with Keybase... done.

@ThePhxRises didn't you have such a problem?

[strib]

Uploads are pretty slow. But if you do a keybase log send I can take a look and see what it's doing.

[kubalobo]

I leave it for the night and it hasn't finished. :(
Here is my log id: 280bee8fce552c2d2fe21d1c

Thank you for your support!

[ThePhxRises]

I did around a 3.5gb push the other day, which took around an hour and a half-ish. Completed without issue

[strib]

@kubalobo thanks. In your logs I see a bit of weirdness with the KBFS service being restarted occasionally during the middle of the transfer (not sure why), and also possibly some pressure based on the size of the disk. But unfortunately, the git logs themselves don't contain much info. keybase log send only sends back a portion of the logs, and I guess the interesting parts of the git operation must have taken place before this portion of the logs.

If you're willing, could you please zip up all the *.git.* files in %APPDATA%\Local\Keybase and share them with me in our keybase://private/kubalobo,strib folder? That way I can get more info to see what the problem was. Thanks!

[kubalobo]

I put everything there.
Thank you!

[strib]

Thanks, that was helpful! I think we probably need to re-think the way we upload huge collections of LFS files written all at once. We're trying to sync them all at once, instead of in small batches, and I think that's causing stress and maybe hanging on your system. I'll make an internal ticket to address.

In the meantime, if possible, I recommend only pushing small batches of files if you can break them up that way. Sorry for the bad experience, and thanks for the report!

[kubalobo]

Hi!
I just confirm that separating my files for independent commits and pushes result with no errors so it is a workaround.
Thank you again!

[rjurney]

Thanks all. I just got some testing in on a Windows device, and you'll never guess this one simple trick to getting it to work...

Instead of completely removing each \\\" from the .git/config file, you can change them into \". That solved it for me.

I swear I tested the current code in CMD before and got it to work, so I'll have to go back and try to figure out how to make it work automatically in the next release. Sorry for the inconvenience. Let me know if the above helps.

This fixes the problem for me on Mac OS X with Keybase Version 5.1.1-20191211223249+15bbb94c23, git-lfs/2.9.2 and git version 2.21.1. This Keybase version is current as of Jan 25, 2020.

[arcticicestudio]

From the feedback in this issue I guess this problem is not related to Windows based OS only.
I can confirm that the following additional escaping in the .git/config file of my repository also fixed the „invalid character 'U' looking for beginning of value” error on my Linux system.

[lfs "customtransfer.keybase-lfs"]
  path = git-remote-keybase
  args = \"lfs origin keybase://private/arcticicestudio/repository\"

I'm running Arch Linux kernel 5.4.1-arch1-1 x86_64 and the Keybase versions Client: 5.1.1-20191211223501+15bbb94c23 and Service: 5.1.1-20191211223501+15bbb94c23.

[rjurney]

Is this something keybase can fix? It seems a trivial bug to fix.