Restic: Very slow backup of SSHFS filesystems -- need ability to skip inode-based comparison

[McKael]

Output of restic version

restic 0.8.2

How did you run restic exactly?

% restic -r /tmp/restictest backup $SSHFSMOUNT/directory

The backup directory is a sshfs mounted filesystem.
The command is run several times to do several snapshots.
In mly case there is a huge amount of data (1TB+) in the remote directory.

What backend/server/service did you use to store the repository?

Local ext4 filesystem.

Expected behavior

I expect the first backup to be slow and transfer much data over LAN, while the next backups should be pretty fast and should not use much bandwidth.

Actual behavior

All backups take hours (if not days) even if no files have changed.

Steps to reproduce the behavior

• run a backup
• unmount sshfs filesystem
• re-mount sshfs filesystem
• run a new backup

It is not 100% reproducible but even with a small amount of data I could reproduce it.

The SFTP serrver logs show that files are completely retrieved even when they did not change.

Do you have any idea what may have caused this?

Yes: restic compares inodes to double-check if files have been modified (debug message "timestamp, size or inode changed", restic/node.go:551restic.(*Node).IsNewer11node).

However, inodes might change across filesystem mounts with sshfs (and probaly some other filesystems).

Do you have an idea how to solve the issue?

Commenting out the inode check solved the problem for me.

I'd like to have a way to disable this check; maybe a command-line flag?

Did restic help you or made you happy in any way?

Sure, it's a nice piece of software! I'm all the more pleased since I have found a workaround...
Keep up the good work!

[fd0]

Thanks for the report, this is indeed caused by restic detecting that files have changed based on the inode. For fuse-based file systems, this check is not great, we should instead only check the timestamps and the file size.

In principle this could also be detected automatically (by looking at the file system name, and keeping a black list of know-inode-unstable filesystems) so we may not even need a command-line flag.

[ebastos]

In principle this could also be detected automatically (by looking at the file system name, and keeping a black list of know-inode-unstable filesystems) so we may not even need a command-line flag.

How do you envision this? I could give it a shot...

[cyphar]

2205 was merged and is in 0.9.5, this should be closed. :wink:

[fd0]

You're right, thanks for the hint!