Client: Upload files to KBFS without filling local encryption queue

[501st-alpha1]

I copied a bunch of files to KBFS, and I started getting the following error:

Out of temporary space
Keybase is using 104% of its temporary write space (1.7GB), and writes will fail until the data syncs to the remote server.

(The percentage and size have varied slightly, but the percentage is always over 100%.)

So while I haven't read any documentation to this effect, I conclude from this that Keybase caches my files locally, returns that the file was written successfully, and then encrypts and uploads it in the background.

This is fine and expected for files that I am directly interacting with, but if I have a script that copies a large number of files, I would like the ability to wait for each file to actually be encrypted and uploaded before continuing to the next one, so that e.g. if I need to use my computer for something else, I can abort the script and be able to use my full CPU in a reasonable amount of time. (Also, more importantly, if I'm tracking which files are uploaded based on whether the write succeeds, I'd like to be able to rely on that to mean they have actually been uploaded, rather than sitting in a queue on my local disk, waiting to be uploaded at a later time.)

Does such a feature exist in the Keybase client currently? If not, is there anything that would prevent adding such a feature? An alternative I'd be happy with is a command that returns whether or not there are any files in the queue waiting to be encrypted and uploaded.

Shouldn't make much of a difference, but just in case: I'm running Keybase on Debian 8 Jessie. Output of keybase version:

Client:  3.0.0-20190205202117+6394d03e6c
Service: 3.0.0-20190205202117+6394d03e6c

[strib]

You're right on how things work -- we write locally for performance reasons.

You can definitely find out if there's un-uploaded stuff. In a top-level folder (for example, /keybase/private/you) you can read the special file .kbfs_status (like cat /keybase/private/you/.kbfs_status and look for DirtyPaths and UnflushedBytes. if those are empty and 0 respectively, everything is uploaded.

Also, it's not really supported, but you can turn that feature off for a given top-level folder with echo 1 > /keybase/private/you/.kbfs_disable_journal. You have to do this after each restart of KBFS. There is still some in-memory caching, but you can wait for flushes by calling fsync like you normally would on a file system. I don't really recommend it, but I hope it helps.

[501st-alpha1]

Awesome, looks like I can parse .kbfs_status to do what I want, thanks!

To confirm, by "top-level folder" you mean a folder such as /keybase/private/me, and not anything further down? Or could I do echo 1 > /keybase/private/me/path/to/some/dir/.kbfs_disable_journal and have it work for just that folder? Additionally, are there any data-integrity risks to disabling that feature?

[strib]

Yeah, just the setting is for everything under /keybase/private/me, we're not able to be more granular that that.

I don't know of any specific risks; just the vague sense that we don't test that mode as much, or at all, internally.

[strib]

Oh to clear one thing up from your description that's wrong though -- we always encrypt before writing files to the local disk. We never write unencrypted file data to the disk.

[501st-alpha1]

Thanks for the info!