Client: Questions about how `UnflushedPaths` in `.kbfs_status` file works; Keybase not uploading files

[501st-alpha1]

Following up on #17141, I have a couple questions about how the .kbfs_status file works.

I've been using this command to see how many files are in the upload queue:

cat /keybase/private/<me>/.kbfs_status | jq '.Journal.UnflushedPaths | length'

I noticed the length was remaining constant per machine, so I started looking at the array itself. When I removed the length, I noticed a couple things about the UnflushedPaths array.

First, I see that the last element is "...", and this is true even before piping to jq. I assume this means that there are more paths in the queue that are not shown (probably to save memory). Is this correct?

Second, I see that at least the first path (jq '.Journal.UnflushedPaths[0]') changes regularly. Is the UnflushedPaths array a FIFO queue then? (This would mean that files are still being uploaded, which is a good sign, since I was originally worried it was stuck on certain files.)

I also noticed that .Journal.UnflushedBytes is remaining constant, which doesn't seem to fit with my above assumptions.

[501st-alpha1]

Just fixed an issue that was causing intermittent connectivity dropouts on my network, and within 10 minutes or so the UnflushedBytes number started going down, so maybe that was related. (Maybe failed uploads were just being moved to the end of the queue?)

[strib]

First, I see that the last element is "...", and this is true even before piping to jq. I assume this means that there are more paths in the queue that are not shown (probably to save memory). Is this correct?

Correct. We max out around 1000 paths I think.

Second, I see that at least the first path (jq '.Journal.UnflushedPaths[0]') changes regularly. Is the UnflushedPaths array a FIFO queue then? (This would mean that files are still being uploaded, which is a good sign, since I was originally worried it was stuck on certain files.)

I don't think we make any effort to sort the paths, so probably they remain unsorted due to the particulars of Go's implementation of maps.

Just fixed an issue that was causing intermittent connectivity dropouts on my network, and within 10 minutes or so the UnflushedBytes number started going down, so maybe that was related. (Maybe failed uploads were just being moved to the end of the queue?)

If KBFS wasn't able to flush any files out of the network, it makes sense that the number of paths and bytes weren't going down. Hopefully it all flushes successfully now!

If you continue to see problems, you can always do a keybase log send and we can debug it further.

[501st-alpha1]

Okay, thanks for the info.

Since I have too many paths I'll be watching UnflushedBytes for now. Apparently I was mistaken before, it actually went up a little bit. :grimacing:

How long should it take for uploads to happen, assuming a decent network connection (which I have)?

I'll keep an eye on it and will report back if it doesn't go down. (And let me know if you'd rather I open a new issue for debugging that.)

[strib]

I can't give an estimate -- it completely depends on your network, disk, and CPU conditions. Note that you can get KBFS's estimate with this:

cat /keybase/private/strib/.kbfs_status | jq '.Journal.EndEstimate'

But if it's not going down at all, something else is wrong and we should check it out with a log. This issue is fine.

[501st-alpha1]

Yeah, it hasn't gone down at all since yesterday. Submitted logs, id is 52a6c5a557d5e67a0e3a6f1c.

[strib]

@501st-alpha1 You appear to be out of disk space completely on the local drive that hosts your home directory. Would it be possible to clear up some space on that and then restart Keybase? It's failing while trying to write some little tiny files needed to make progress. (At least that's the failure covered by the logs -- it's possible another failure will come into view once that one is fixed.)

[501st-alpha1]

Ah okay, that makes sense; I have been running low on space on that drive recently.

Do you know about how much free space I should need? I have ~700M free right now, and have just restarted Keybase, but it still hasn't gone down yet.

[strib]

From what I saw you shouldn't need that much, but it's hard to say. Please do another keybase log send and post the log ID here so I can see why it's stuck now.

[501st-alpha1]

Log ID is ac4da5e5e6a64ee06d9bda1c. Restarted Keybase after getting free space up to 1.6G.

[strib]

Ok I see. It seems like your device hit some sort of conflict when trying to merge with the server, and it hit many errors in a row trying to resolve it (maybe due to the low disk space?), and now it's refusing to try again.

It would be great to see what the errors were, they should be listed in your /keybase/private/501st_alpha1/.kbfs_status, if you can share that with me it might be useful.

If you want, you can move your local changes aside with this command:

keybase fs clear-conflicts /keybase/private/501st_alpha1

That will revert you to the server view of the folder under the usual path /keybase/private/501st_alpha1, but you can still access your changes under a different path. Then you can manually copy any data over from the different path to the usual path, and when you're done you can use keybase fs finish-resolving-conflicts when you're done to clear the storage.

[501st-alpha1]

Under ConflictResolutionAttempts, I see one multiple errors and then a bunch of context deadline exceeded errors. Let me know if you need more details.

Thanks, I'll take a look at fs clear-conflicts.

[strib]

Ok. Unfortunately that error is a bit ambiguous -- could be that you were out of disk, or maybe just too much unflushed data on a slow cpu/disk.

Anyway, let me know if clear-conflicts works for you.

[501st-alpha1]

I tried clear-conflicts on a subfolder, and got:

▶ ERROR EOF

Log is 3e6f0b3119c2b840c1b0fe1c.

I did just upgrade Keybase to the latest version available (probably should have tried that first, sorry :grimacing:), and haven't been able to get KBFS to start up since then, but my understanding based on this comment is that clear-conflicts should work anyway.

[strib]

I'm sorry for all this, it seems like the amount of unflushed data that's built up is too big for your machine to handle -- it seems to timeout just on initializing itself. The logs show it's about 16 GB!

You can move it out of the way manually. After completely shutting down Keybase and KBFS, do:

mv $HOME/.local/share/keybase/kbfs_journal/v1/012008f229de1259e3ee6cad2e0640c1042b-df40b1c534226468 $HOME/.local/share/keybase/kbfs_journal/v1/df40b1c534226468.bkp

Then restart everything and see if it worked.

At that point you can inspect your folder and see if there's any important data that is missing, which can't be recovered from elsewhere. If so, let me know, and we'll figure out a way to restore it from the local backup you just made. If everything looks good, you can delete that .bkp directory manually to save yourself the disk space.

[501st-alpha1]

No problem, and thanks for all the help troubleshooting this.

Moving it out of the way did seem to fix it; KBFS started up fine, and the number of UnflushedPaths is 0.

AFAIK I should be able to get copies of all the unflushed data from elsewhere; I'll report back if I'm missing anything important.

[501st-alpha1]

Hmm. On one machine with this issue I've been able to add more files to Keybase and everything has been fine. UnflushedPaths goes up for a little bit, then I wait for it to go back to 0 before adding more files.

However, on my other machine I added one file and it hasn't been able to clear it yet. ConflictStatus is in conflict and stuck, but I don't see any new errors.

Log id is 6d5a9bcc69b8fc080f8e4d1c.

[strib]

Ah sorry. It still thinks it's stuck from before, because of the manual clearing we did. I think you just have to shutdown Keybase+KBFS again, and do:

rm -rf $HOME/.local/share/keybase/kbfs_conflicts

And then restart everything. Let me know if that helps.

[501st-alpha1]

That seems to have done it. Thanks!

[strib]

Great, sorry for all the issues, let me know if you see anything else like this. We fixed several bugs that might have caused this, and are putting more processes in place in the GUI to help you fix this yourself should it happen again in the future.