Git-lfs: Git checkout slow with many LFS files

[larsxschneider]

Hi,

we are using Git-LFS at my company and in general we are really happy as it nicely solves the binary files problem in Git. We started using Git-LFS for all binary files and experienced _very_ slow checkouts for a repository with +10k binaries.

E.g. consider this repo with 15k LFS files:
https://github.com/larsxschneider/lfstest-manyfiles

Checkout time on my Mac with SSD is around 10min (which is OK).
Checkout time on Windows with SSD is well over an hour (which is not so nice).

The reason for this slow checkout is, as you probably already know, the invocation of the Git-LFS process via the Git attribute filters. 10k files means 10k process executions which is, of course, horribly slow on Windows. As far as I understand the design of Git-LFS there is not much we can do about it.

However, what if we could convince the Git core team to merge a patch that (optionally) makes Git talking to a local socket instead of executing a process for Git filters? Then we could run a Git-LFS daemon kind of thing and I imagine the checkout and other operations could be much faster. Of course the daemon should be optional.

Do you think this idea is worth exploring? Do you see an easier way to tackle this problem?

Thanks,
Lars

[peff]

Where does the time go on Windows? Is the CPU pegged? Or are we getting killed by the latency of starting many processes serially?

[strich]

You can work around the single-process-per-file problem by temporarily disabling the smudge filter and then using git lfs pull to grab the files in a single pass. Example here: https://github.com/github/git-lfs/issues/911#issuecomment-169998792.

[larsxschneider]

@strich I know. However, disabling the smudge filter with a variable is still slow.

The git-lfs binary is ~10MB and it takes quite some time to run it even if it does nothing due to the GIT_LFS_SKIP_SMUDGE flag (test A). Ideally we would deactivate the LFS filter on the initial clone all together which I tried with test C. The problem with this approach is that LFS is deactivated globally and that might cause trouble with parallel Git/Git-LFS processes. Unfortunately I was not able to run git clone with a deactivated config. The best I came up with was to set the LFS filter to the cat command (test B1). See this gist for a git clone wrapper using this approach. I also tested the true command (test B2) is to measure the overhead of calling an external process.

Here are the clone timings for the test repo mentioned above (exact test run commands below):

| Test/time in sec | OS X | Windows |
| --- | --- | --- |
| A GIT_LFS_SKIP_SMUDGE=1 | 210 | 2749 |
| B1 --config filter.lfs.smudge=cat | 33 | 432 |
| B2 --config filter.lfs.smudge=true | 29 | 229 |
| C git-lfs uninstall | 4 | 37 |

As you can see the "no smudge filter" solution is more than 50 times faster than the GIT_LFS_SKIP_SMUDGE flag on my machines.

@peff: I will try to create a git-core patch which allows to deactivate a git config value on git clone (e.g. git clone --unset-config filter.lfs.smudge <repository>). Maybe it gets accepted.

@technoweenie: Do you see an issue with disabling the filter in the clone statement temporarily?

---

OS X - Test A

$ rm -rf lfstest-manyfiles && time GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/larsxschneider/lfstest-manyfiles.git
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.61 MiB/s, done.
Checking connectivity... done.
Checking out files: 100% (15001/15001), done.
GIT_LFS_SKIP_SMUDGE=1 git clone   64.56s user 133.35s system 94% cpu 3:29.81 total

OS X - Test B1/2

$ rm -rf lfstest-manyfiles && time git clone --config filter.lfs.smudge=true https://github.com/larsxschneider/lfstest-manyfiles.git
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.79 MiB/s, done.
Checking connectivity... done.
Checking out files: 100% (15001/15001), done.
git clone --config filter.lfs.smudge=true   9.85s user 16.43s system 92% cpu 28.551 total

OS X - Test C

$ rm -rf lfstest-manyfiles && git-lfs uninstall && time git clone https://github.com/larsxschneider/lfstest-manyfiles.git
Global Git LFS configuration has been removed.
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.60 MiB/s, done.
Checking connectivity... done.
git clone https://github.com/larsxschneider/lfstest-manyfiles.git  0.28s user 1.31s system 46% cpu 3.435 total

---

Windows - Test A

$ rm -rf lfstest-manyfiles && time GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/larsxschneider/lfstest-manyfiles.git
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.63 MiB/s, done.
Checking connectivity... done.
Checking out files: 100% (15001/15001), done.

real    45m48.289s
user    0m0.000s
sys     0m0.015s

Windows - Test B1/2

$ rm -rf lfstest-manyfiles && time git clone --config filter.lfs.smudge=true https://github.com/larsxschneider/lfstest-manyfiles.git
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.55 MiB/s, done.
Checking connectivity... done.
Checking out files: 100% (15001/15001), done.

real    3m48.358s
user    0m0.015s
sys     0m0.000s

Windows - Test C

$ rm -rf lfstest-manyfiles && git-lfs uninstall && time git clone https://github.com/larsxschneider/lfstest-manyfiles.git
Global Git LFS configuration has been removed.
Cloning into 'lfstest-manyfiles'...
remote: Counting objects: 15010, done.
remote: Compressing objects: 100% (15008/15008), done.
remote: Total 15010 (delta 0), reused 15010 (delta 0), pack-reused 0
Receiving objects: 100% (15010/15010), 2.02 MiB | 1.51 MiB/s, done.
Checking connectivity... done.
Checking out files: 100% (15001/15001), done.

real    0m36.816s
user    0m0.000s
sys     0m0.000s

[strich]

Good analysis @larsxschneider. In addition to what you've discussed already, can you comment on why the OSX and Windows tests are so wildly different? I'd be very interested in looking into resolving that myself.

[larsxschneider]

Process creation on Windows is just slower (see here for a few arguments). This slowness has nothing to do with git or git-lfs.

For the fun of it you can start the cat command a 1000 times on different systems:
time bash -c 'for i in {1..1000}; do cat /dev/null; done'
On my OS X machine that takes 1.5 sec and on Windows 26 sec :smile:

[peff]

@peff: I will try to create a git-core patch which allows to deactivate a git config value on git clone (e.g. git clone --unset-config filter.lfs.smudge <repository>). Maybe it gets accepted.

Unfortunately, I think you will find it quite difficult because of the way git's config code is structured. The config code reads the various config files and hands the values to an arbitrary callback in a streaming fashion. So you cannot "unset" a variable in the local config after it has been set in the user-wide config. You can only tell the callback "here is another value" and hope that it overwrites.

One thing that would probably work is to teach git -c, the per-invocation config override mechanism, to keep a blacklist of config keys and avoid passing them to the callbacks. That works because git -c config can be parsed before any of the other files (anything else would involve 2 passes over the config, or moving to a non-streaming system).

[technoweenie]

Do you see an issue with disabling the filter in the clone statement temporarily?

Are you suggesting a git lfs clone command to wrap that behavior up? I'm in favor of that. The current skip-smudge behavior is as good of a workaround as I can come up with for now. Maybe a clone command could do something strange like:

1. Bare clone (no checkout, so filters are not used)
2. Unset filter.lfs.* for the locally cloned repo
3. Checkout to a working dir
4. Run git lfs pull
5. Undo local repo config from step 2.

[strich]

I would be for that - I'm currently manually doing those exact steps for our projects with new clones.

[peff]

Anything that is clone-specific feels a bit like a hack. The real problem is "you have a lot of LFS files". So it's going to be an issue in _other_ cases, too. E.g., init + fetch + checkout. Or just checkout between two distant points in history.

Rather than a socket interface, I wonder if git could "batch" files that are going to be fed to the same filter, start the filter once, and then feed them all over its stdin. Like a socket interface, that requires defining a totally new interface for the filters, but it's going to be a lot simpler and more portable than sockets.

[peff]

The real problem is "you have a lot of LFS files".

Re-reading this, it looks like I might be implying that you're doing something wrong. You're not. I just mean "the problem is that we're moving from point A to B, and that there are a lot of different LFS files, so we have to kick off LFS a lot of times". The fact that point A is empty in a clone makes it a common place to run into this, but not the only one.

[larsxschneider]

@technoweenie @strich: see this gist for my current wrapper command: https://gist.github.com/larsxschneider/85652462dcb442cc9344

@peff: Thanks! I get your point :smile: ! Batching the filter sounds like the right solution. Although this is probably a bit more involved.

[larsxschneider]

I looked into git config for an easy way to disable certain configs. As @peff already indicated, it is a lot more complicated than I initially thought. I made quite some progress to get git -c !filter.lfs.smudge clone <repository> working when I realized that git -c filter.lfs.smudge= -c filter.lfs.required=false clone <repository> is an equally good solution and it even works with the current Git. The problem with the latter solution is that Git still tries to execute the filter and prints an error for every processed file (which adds unnecessary time to the clone execution).

Printed error using Git on OS X/Linux:

error: cannot run : No such file or directory
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

error: cannot fork to run external filter
error: external filter  failed

Printed error using Git for Windows:

error: cannot spawn : No such file or directory
error: cannot fork to run external filter
error: external filter  failed

I submitted a patch to the Git mailing list which disables filters with empty strings (topic "convert: legitimately disable clean/smudge filter with an empty override"). With this patch the clone on OS X is as fast Test C (git-lfs uninstall). I haven't compiled/tested the solution on Windows but I expect a similar result.

With the current Git we can still use the approach and ignore the errors (Test B3):

git clone --config filter.lfs.smudge= --config filter.lfs.required=false <repository> 2>&1 | sed '/error: cannot run : No such file or directory/{N;N;N;N;N;N;d;}' | sed '/error: cannot spawn : No such file or directory/{N;N;d;}'

| Test/time in sec | OS X | Windows |
| --- | --- | --- |
| A GIT_LFS_SKIP_SMUDGE=1 | 210 | 2749 |
| B1 --config filter.lfs.smudge=cat | 33 | 432 |
| B2 --config filter.lfs.smudge=true | 29 | 229 |
| B3 --config filter.lfs.smudge= --config filter.lfs.required=false | 19 | 207 |
| C git-lfs uninstall | 4 | 37 |

This makes the clone execution again faster, especially on Windows (twice as fast as the previous cat solution). The only downside is that the clone progress is gone because of the redirect. I updated the LFS fast clone gist with this code.

@technoweenie: Do you think it would make sense to add a git lfs clone command that wraps the git clone using my gist?

[technoweenie]

Yes, this seems like a better idea than the skip smudge setting.

[sinbad]

I'm having a go at this: https://github.com/sinbad/git-lfs/commit/463abf595fde8f627a170a352c2104e8a044a673

Seems to work great for me, I just need to write some integration tests. I'm out of time today, will pick this up on Monday & submit a PR.

[larsxschneider]

@sinbad Thank you! It is great to see that this approach is natively adopted by Git-LFS! :+1:

[tyen901]

When might we see a release with 'git lfs clone' built in? It's the only thing stopping my team from using git lfs for our large projects.

[rjbell4]

Hasn't this been resolved with the addition of the git lfs clone command? Or is there something still pending here? Enterprise-config-for-git still implements its own clone command: https://github.com/Autodesk/enterprise-config-for-git/blob/master/clone.sh

[larsxschneider]

@rjbell4 Yes, the git lfs clone command is the way to go. I haven't updated the the enterprise-config repo with our latest internal changes plus we do this internally ... I will tackle that soon.
BTW: are you actually using enterprise-config? If yes, then I would like to get in contact. It is not easily usable out of the box and I wonder if I can do something to improve that.

[rjbell4]

In the process of using enterprise-config, yes. (Other people here are driving it) I think we/they would absolutely love to be in contact.

[samskiter]

Having a lot of issues with this on Jenkins - anyone have any tips?

EDIT: Found this old issue on jenkins - looks like an outstanding issue there?

[ttaylorr]

Having a lot of issues with this on Jenkins - anyone have any tips?

Yes. Git LFS v1.5.0 implements support for the new process filter in Git v2.11.0. If you upgrade to both of those on your Jenkins instance, the time it takes to checkout will decrease by a factor of 80 or so (depending on platform/architecture).

[larsxschneider]

@samskiter The default Jenkins Git plugin does not take advantage of the git lfs clone command... therefore cloning repos with many LFS files takes ages. Either you perform the Git LFS commands as part of your Jenkins job yourself or you wait for this (I am working on it).

[samskiter]

I'm now using git 2.12 and git lfs 2.0.0 and still getting slow times checking out on my machine. Is there something I need to do to take advantage of the new speedups?

[ttaylorr]

@samskiter make sure you run an git lfs install in order to set the new filter.lfs.process config value.

[samskiter]

@ttaylorr I've done that already, fairly sure... do these settings look right?

$ git config --list | grep lfs
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f

[ttaylorr]

@samskiter if those settings are repository-local (<repo>/.git/config) they wont be applied to every repository on your computer that uses LFS.

That all being said: could you run a git config --list --show-origin | grep 'lfs as well as a git lfs env and paste the output of that in a new issue?

[samskiter]

@ttaylorr done :)