Client: Having a git repo inside the keybase file system private folder was not a good idea

[johanbove]

Love the idea of Keybase File System! What did not work so well was having a git repository to keep a history of my personal "journal" file. Too many updates by git internal manipulations made Keybase File System go a little nuts on my machine. So I'm posting this as a word of advice and warning; do not keep a git repository in a private Keybase Filesystem folder. At least not on Windows.

Keybase version: 1.0.17-20160822152935+692af12

[maxtaco]

We'll take a look, can you do a keybase log send?

Thank you!

Cc: @taruti

[johanbove]

Of course. My log id: 3a749a5ed7218c87d6d1091c
Not sure if anything was sent though since I got this error message when typing "yes" in the "Continue sending logs to keybase.io?" dialog:

- WARNING error opening log "C:\\Users\\Johan\\AppData\\Roaming\\Keybase\\keybase.start.log": open C:\Users\Johan\AppData\Roaming\Keybase\keybase.start.log: The system cannot find the file specified.

[daniel-noland]

I can second this from Linux land (4.7.2-1-ARCH). I will make a clean log report from a fresh VM when I get home.

I suspect that @johanbove has the correct reason. There may be some hacks to get around this problem using something like archivemount in the short term.

keybase version:
Client: 1.0.17-20160906160030+1f66ba7
Service: 1.0.17-20160906160030+1f66ba7

[taruti]

Looking at this. @songgao probably too.

[strib]

Thanks, we got them @johanbove. Could you be a little more specific about what you mean by "go nuts"?

We have put some work into making git repos work, especially in macOS/Linux, though we haven't completed our testing yet so we haven't advertised it yet. I don't think we've tested it on Windows, and in any case our latest related changes aren't yet in that Windows build, but it would be helpful to get more context about what exactly went wrong. Thanks!

(P.S.: Note that https://keybase.io/docs/kbfs/understanding_kbfs already warns against using git:

In particular, nothing that does file locking (like git) should be used from multiple devices yet, and appending to a single shared file (e.g., a log file) from multiple devices should be avoided.

)

[johanbove]

Thanks for the quick replies! :rocket: With "go nuts" I mean: I could clearly see Keybase crunching the signing and the encrypting every time I made a change to a file and updated the repo as I received a lot of Windows 10 notifications every time Keybase detected a changed file in the git repo. I was also using Visual Studio Code for editing my markdown file, which has a nice built-in Git GUI, but the whole process lagged behind to what Visual Studio Code was expecting and it all just did not work anymore.

[strib]

Ah @johanbove, thanks. So you didn't see any corruption or anything like that, just general slowness? That's probably expected. We will be improving write performance in the near future by not requiring network access on most writes; that said, git tends to open a lot of files in "exclusive" mode that might negate most of those optimizations (though not sure what it does on Windows exactly). We'll be continuously looking into it though, since it's an important use case we want to support. Thanks for the feedback!

@daniel-noland: do you have similar performance complaints? Or did you encounter corruption or other errors?

[johanbove]

You're welcome. I did now see that I lost the file in the /johanbove/private folder. No biggy, as I'm fully aware of the current status of the application. :smile: But it now seems that the update process did a full reset of the folder. I would have lost data now.

My log id: 049479607743d36ea79add1c

I'm using git version 2.8.0.windows.1 on Windows 10 Version 1607 (OS Build 14393.105)

[strib]

@johanbove: Hrm, you definitely shouldn't lose data from your folder. At worst I can imagine that when you checked out from the repo in KBFS, it didn't check out a file you thought you had committed. Can you be more specific about which file you lost, and where exactly it's missing from? (If it's sensitive, you can encrypt it just for me or put it in /keybase/private/johanbove@github,strib@github.)

[johanbove]

The file was called "journal.md"; it was only a text file containing a couple of lines of super private thoughts 😄 I had a copy on my work mac, but when I logged in this morning, I unfortunately forgot to turn off KBFS and it synchronized the "private" folder and removed the file. Now I'm not sure what exactly happened to the file, but I can try to reproduce. I think Visual Studio Code's git implementation might have swallowed it.

[strib]

It seems from the log that you (or some program on your behalf) purposely removed that file:

2016-09-07T22:38:37.658700 - [DEBU kbfs fs.go:483] 18c => File Cleanup [tags:DID=IDuaB_n6nRGucKNYTr6yKw]
2016-09-07T22:38:37.659195 - [DEBU kbfs file.go:55] 18d Cleanup {{{1} journal.md 0x12583080 0x122b32a8 0x122b21a8 {}}} [tags:DID=IDuaB_n6nRGucKNYTr6yKw]
2016-09-07T22:38:37.659195 - [DEBU kbfs file.go:57] 18e Removing file in cleanup journal.md [tags:DID=IDuaB_n6nRGucKNYTr6yKw]
2016-09-07T22:38:37.659195 - [DEBU kbfs(FBO 0346c42f) folder_branch_ops.go:2659] 18f RemoveEntry 0x122262d0 journal.md [tags:DID=IDuaB_n6nRGucKNYTr6yKw]2016-09-

It also looks to me like the file was not in git at all, it was just stored as /keybase/private/johanbove/journal.md. Is that right? It's unclear to me if the file disappearing has anything to do with the git issues discussed above. Can you elaborate a little please? Can you think of anything you did on your Windows machine that might have caused the flie to be deleted?

Regarding the file still being available on your work mac: KBFS doesn't store local copies of the file on each machine. They are only stored once, in the "cloud", without using any of your disk space. So once it's deleted from KBFS, it will be gone from all of your machines. We will at some point be releasing a way to access the history of your folder, from which you will be able to recover deleted files, but we haven't enabled that feature yet publicly.

[johanbove]

Well, I did delete the .git folder from my /johanbove/private folder; I am now thinking I might have deleted the journal.md file in the same swift action... I looked into the "Recycle Bin", but the file wasn't there, so I assumed KBFS gobbled it up somehow.
A "history feature" would most definitely be a welcome addition to the tool.

[strib]

From the log, it looks to me like the git repo was removed around 21:59 local time on the 7th, while hello.md and journal.md were removed around 22:38 local time (~40 min later). If it helps refresh your memory, about 30s after that some keybase log files were removed as well.

[johanbove]

Hmm, that's some seriously detailed logs you got there :smile: It's weird. I most definitely did not want to remove that journal.md file on purpose. I was playing around with setting up a shared folder by creating the folders in the Windows command line. I initially used md johanbove,max; I soon realized this wasn't working as expected as I needed to create the folder using the command md "johanbove,max". Perhaps the whole folder was recreated in this action?

[strib]

Yes, with a product this early, detailed logs are essential!

Hmm, interesting theory about md but I can't find any evidence that that command would delete things inside an existing directory. (KBFS definitely doesn't let the folder be re-created, so it would have to be md itself doing the deleting.) I do see a lot of accessed to a shared directory, both before and after the deletes, but it looks like you might have misspelled max's twitter handle (it's "maxtaco", not "maxtaxo").

Looking more into your log -- a few minutes before those files were deleted, it looks like KBFS crashed (and then restart):

2016-09-07T22:35:07.865294 - [DEBU kbfs folderlist.go:70] 25db FL Lookup []string{} public=false upper=false [tags:DID=Y7jGWZhGQSPlw0hiZqONmw]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x2c pc=0x5c3b7d]

goroutine 83 [running, locked to thread]:
panic(0xf046e0, 0x121be030)
    C:/Go/src/runtime/panic.go:481 +0x326
github.com/keybase/kbfs/dokan.kbfsLibdokanGetFileInformation(0x1a706bc, 0x3582fcd0, 0x3582fc88, 0x0)
    C:/Jenkins/workspace/gui_wix/src/github.com/keybase/kbfs/dokan/cgo.go:194 +0x22d
github.com/keybase/kbfs/dokan._cgoexpwrap_bfb0e0901e40_kbfsLibdokanGetFileInformation(0x1a706bc, 0x3582fcd0, 0x3582fc88, 0x1aab710)
    ??:0 +0x31

^ @zanderz @taruti is this a known issue that's been fixed in a more recent release?

Not sure what the crash has to do with the files getting deleted, but it seems a little suspicious...

[daniel-noland]

@strib I finally got a moment to spin a fresh vm.
I have noticed both significant performance issues and good old "buggy" behavior. For example:

[dnoland@keybase-test dnoland]$ git clone https://github.com/daniel-noland/RatSpaces.git
Cloning into 'RatSpaces'...
remote: Counting objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
error: unable to write sha1 filename /home/dnoland/keybase/public/dnoland/RatSpaces/.git/objects/c8/3533140e8efe8625c621e543eb24f69dfeecb1: Interrupted system call
fatal: failed to write object
fatal: unpack-objects failed

Can't tell you why that happened. As you are free to check, that repo is quite trivial. Short commit history and very few files.

I can't say if the performance issue is a bug in keybase per se, but I can tell you the relative time for cloning to my public dir:

Cloning to my home dir:

zsh% time git clone https://github.com/daniel-noland/RatSpaces
Cloning into 'RatSpaces'...
remote: Counting objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
Unpacking objects: 100% (32/32), done.
Checking connectivity... done.
0.02s user 0.02s system 9% cpu 0.474 total

Cloning to my keybase public dir:

[dnoland@keybase-test dnoland]$ time git clone https://github.com/daniel-noland/RatSpaces.git                                                      
Cloning into 'RatSpaces'...
remote: Counting objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
error: unable to create temporary file: Interrupted system call
fatal: failed to write object
fatal: unpack-objects failed

real    0m34.315s
user    0m0.010s
sys     0m0.007s

I made two attempts to clone this way.
Log id 1: 95deccc9ed04a3ca1750381c
Log id 2: fa6bff6791a0e42bf24c0d1c

$ keybase version
Client:  1.0.17-20160908193355+c2873a5
Service: 1.0.17-20160908193355+c2873a5

If it is helpful I can do more tests, e.g. copy existing git repo directly to public or private dir. Run commits or merges. Let me know what is helpful :smile:

[strib]

Thanks for testing @daniel-noland! As in the other bug, your log sends didn't include the KBFS logs sadly.

But yesterday we did encounter a similar clone bug internally. @songgao do you think it could be the same thing as the one you're investigating?

As for performance -- yeah, it's going to be slow. That will hopefully be fixed down the road, but for now I'm more concerned with the "interrupted system call" issues...

[daniel-noland]

@strib,

Hmmm, I must have gone off the reservation with respect to my mounting procedure. I am known to be a bit thick from time to time. If just posting the log file here is a good call let me know. Otherwise I can spin a fresh VM and use a less confused procedure. This bug seems quite repeatable, so that should be very easy.

As for the performance issues, yeah that seems to be the case. I only even mention it because I thought it may help debug the other problem. I know the system is new and problems are to be expected. That said, I would much rather see correctable speed issues now than impossible to maintain moon-logic code later :+1:. Premature optimization and all that.

Keep up the good works friends :smile:.

[daniel-noland]

Ok. Following the resolution of #4228, I gave this another try.

Log id: 2c1b7b1fcc0837e1e12e021c

Hopefully this helps. I would love to have this feature!

[soulofmischief]

I was able to successfully clone both a public GitHub repository and a private keybase repository into a public git keybase folder using git clone <path> /keybase/public/<user>/<repo>.

I was able to clone using the --bare command and then push and pull from the repository under the account owning the public folder, and pull from other accounts.

@daniel-noland if you're still interested in this, give it another shot.

Is keybase team interested in developing this workflow into a gui-supported feature as an alternative to other hosted git solutions?