git gc crashes at 33% of counting objects

Created on 17 Sep 2018 · 16Comments · Source: git-for-windows/git

[x] I was not able to find an open or closed issue matching what I'm seeing

Setup

Which version of Git for Windows are you using? Is it 32-bit or 64-bit? 64-bit 2.19.01

$ git --version --build-options

git version 2.19.0.windows.1
cpu: x86_64
built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790
sizeof-long: 4
sizeof-size_t: 8

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 6.1.7601]

What options did you set as part of the installation? Or did you choose the
defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad++
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Enabled
Enable Builtin Rebase: Disabled
Enable Builtin Stash: Disabled

Any other interesting things about your environment that might be related
to the issue you're seeing?

My repository doesn't use any symlinks. I'm forced to use McAfee antivirus solutions. I'm using poshgit.

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

PowerShell Core 6

> $PSVersionTable.PSVersion
Major  Minor  Patch  PreReleas BuildLabel
                     eLabel
-----  -----  -----  --------- ----------
6      0      4

What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.

> git gc
Enumerating objects: 490566, done.
error: failed to run repack887/490566)

What did you expect to occur after running these commands?

Finished garbage collection, especially as automated crashes either.

What actually happened instead?

Problem signature:
  Problem Event Name:   APPCRASH
  Application Name: git.exe
  Application Version:  2.19.0.1
  Application Timestamp:    5b980bc7
  Fault Module Name:    ntdll.dll
  Fault Module Version: 6.1.7601.24117
  Fault Module Timestamp:   5add228d
  Exception Code:   c0000005
  Exception Offset: 0000000000032964
  OS Version:   6.1.7601.2.1.0.256.48
  Locale ID:    2057
  Additional Information 1: 335e
  Additional Information 2: 335ee83054d6c615e4a7142c362e3dd4
  Additional Information 3: 5184
  Additional Information 4: 518485c5adbc52c624cc6890056919a6

If the problem was occurring with a specific repository, can you provide the
URL to that repository to help us with testing?

It's proprietary repository belonging to my company, so I can't.

Source

michalfita

👍3

Most helpful comment

@thomas-patzig Great MCVE! The access violation is caused by using an uninitialized critical section by packing_data_lock(pack) at pack-objects.h#L377:

static inline void oe_set_delta_size(struct packing_data *pack,
                     struct object_entry *e,
                     unsigned long size)
{
    if (size < pack->oe_delta_size_limit) {
        e->delta_size_ = size;
        e->delta_size_valid = 1;
    } else {
        packing_data_lock(pack);
        if (!pack->delta_size)
            ALLOC_ARRAY(pack->delta_size, pack->nr_alloc);
        packing_data_unlock(pack);

        pack->delta_size[e - pack->objects] = size;
        e->delta_size_valid = 0;
    }
}

The code is only triggered when processing a large delta, which was created for the two big binary files in your MCVE: Total 2 (delta 1), reused 0 (delta 0), and was introduced in 9ac3f0e5b3e4 (pack-objects: fix performance issues on packing large deltas, 2017-07-22).

Moving the initialization to prepare_packing_data() seems to fix the problem:

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index d1144a8f7e..29d48f3867 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2299,7 +2299,6 @@ static void init_threaded_search(void)
        pthread_mutex_init(&cache_mutex, NULL);
        pthread_mutex_init(&progress_mutex, NULL);
        pthread_cond_init(&progress_cond, NULL);
-       pthread_mutex_init(&to_pack.lock, NULL);
        old_try_to_free_routine = set_try_to_free_routine(try_to_free_from_threads);
 }

diff --git a/pack-objects.c b/pack-objects.c
index 6ef87e5683..6070b6d565 100644
--- a/pack-objects.c
+++ b/pack-objects.c
@@ -148,6 +148,8 @@ void prepare_packing_data(struct packing_data *pdata)
                                             1U << OE_SIZE_BITS);
        pdata->oe_delta_size_limit = git_env_ulong("GIT_TEST_OE_DELTA_SIZE",
                                                   1UL << OE_DELTA_SIZE_BITS);
+
+       pthread_mutex_init(&pdata->lock, NULL);
 }

 struct object_entry *packlist_alloc(struct packing_data *pdata,

Change available on my branch pack-lock-init.

kgybels on 25 Sep 2018

👍2

All 16 comments

Just want to add that I'm seeing this too, and it affects more than just gc. I hit a variant of it with pull too:

$ git pull
remote: Enumerating objects: 3548, done.
error: git upload-pack: git-pack-objects died with error.
fatremote: aborting due to possible repository corruption on the remote side.
al: git uploadf-pack: aborting due to possible repository corruption on the remote side.
atal: protocol error: bad pack header

(Not sure why the output errors are partially interleaved...)

Rolling back to 2.18 fixes both issues for me, so I don't think it's real corruption. Running fsck doesn't show any errors.

Unfortunately, like @michalfita, I am unable to share the repo.

jholewinski on 17 Sep 2018

It seems that I am affected by this problem as well. Since 2.19.0.windows.1 I cannot execute git gc on any of my machines.

$ git --version --build-options git version 2.19.0.windows.1 cpu: x86_64 built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790 sizeof-long: 4 sizeof-size_t: 8
I am running Windows 10 64Bit (Microsoft Windows [Version 10.0.17134.285]) on both machines.

$ cat /etc/install-options.txt Editor Option: Notepad++ Path Option: Cmd Plink Path: C:\Program Files\TortoiseGit\bin\TortoiseGitPlink.exe SSH Option: Plink CURL Option: OpenSSL CRLF Option: LFOnly Bash Terminal Option: ConHost Performance Tweaks FSCache: Enabled Use Credential Manager: Enabled Enable Symlinks: Disabled Enable Builtin Rebase: Disabled Enable Builtin Stash: Disabled

The repository in question is https://github.com/ultimate-pa/ultimate -- perhaps an interesting thing is that we need long paths.

Output with GIT_TRACE=1:
$ GIT_TRACE=1 git gc 21:41:17.391689 exec-cmd.c:236 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin 21:41:17.393682 git.c:415 trace: built-in: git gc 21:41:17.402659 run-command.c:637 trace: run_command: git pack-refs --all --prune 21:41:17.418616 exec-cmd.c:236 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core 21:41:17.420635 git.c:415 trace: built-in: git pack-refs --all --prune 21:41:17.426620 run-command.c:637 trace: run_command: git reflog expire --all 21:41:17.441553 exec-cmd.c:236 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core 21:41:17.443548 git.c:415 trace: built-in: git reflog expire --all 21:41:17.479452 run-command.c:637 trace: run_command: git repack -d -l -A --unpack-unreachable=2.weeks.ago 21:41:17.492445 exec-cmd.c:236 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core 21:41:17.493415 git.c:415 trace: built-in: git repack -d -l -A --unpack-unreachable=2.weeks.ago 21:41:17.495434 run-command.c:637 trace: run_command: GIT_REF_PARANOIA=1 git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-12104-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago 21:41:17.507377 exec-cmd.c:236 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core 21:41:17.508374 git.c:415 trace: built-in: git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-12104-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago Enumerating objects: 424418, done. error: failed to run repack605/424418)

danieldietsch on 18 Sep 2018

I'm hitting this as well on a moderately large repo.
C:\repos\xxxxxx>git gc
Enumerating objects: 28426, done.
error: failed to run repack47/28426)

This is breaking gc, bundle, and push operations (not sure what else). I'm going to try going back to 1.18 to see if that fixes the issue.

C:>git --version --build-options
git version 2.19.0.windows.1
cpu: x86_64
built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790
sizeof-long: 4
sizeof-size_t: 8

C:>cmd.exe /c ver

Microsoft Windows [Version 10.0.17763.1]

C:>type "C:\Program Files\Git\etc\install-options.txt"
Editor Option: Nano
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Enabled
Enable Builtin Rebase: Enabled
Enable Builtin Stash: Enabled

andyraf on 18 Sep 2018

Good news is that I can downgrade to 2.18.x and have it working again.

michalfita on 18 Sep 2018

I see the same behavior.

git --version --build-options
git version 2.19.0.windows.1
cpu: x86_64
built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790
sizeof-long: 4
sizeof-size_t: 8

$psversiontable.psversion
Major Minor Build Revision

5 0 10586 117

type 'C:\Program Files\Git\etc\install-options.txt'
Editor Option: VIM
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Rebase: Disabled
Enable Builtin Stash: Disabled

Debugging shows
Unhandled exception at 0x0000000077842964 (ntdll.dll) in git.exe: 0xC0000005: Access violation writing location 0x0000000000000024.

The call stack -

ntdll.dll!RtlImageDirectoryEntryToData() + 228 bytes Unknown
ntdll.dll!RtlEnterCriticalSection() + 209 bytes Unknown
git.exe!000000000039d7b7() Unknown
git.exe!00000000003423e5() Unknown
git.exe!0000000000342794() Unknown
git.exe!00000000003413a5() Unknown
git.exe!000000000034150b() Unknown
kernel32.dll!BaseThreadInitThunk() + 13 bytes Unknown
ntdll.dll!RtlUserThreadStart() + 29 bytes Unknown

Dump Summary

Dump File: git.dmp : E:\temp\git.dmp
Last Write Time: 9/18/2018 4:07:38 PM
Process Name: git.exe : C:\Program Files\Git\mingw64\libexec\git-core\git.exe
Process Architecture: x64
Exception Code: 0xC0000005
Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.
Heap Information: Not Present

System Information

OS Version: 6.1.7601
CLR Version(s):

Modules

Module Name Module Path Module Version

git.exe C:\Program Files\Git\mingw64\libexec\git-core\git.exe 2.19.0.1
ntdll.dll C:\Windows\System32\ntdll.dll 6.1.7601.24214
kernel32.dll C:\Windows\System32\kernel32.dll 6.1.7601.24214
KERNELBASE.dll C:\Windows\System32\KERNELBASE.dll 6.1.7601.24214
libiconv-2.dll C:\Program Files\Git\mingw64\libexec\git-core\libiconv-2.dll 1.15.0.0
msvcrt.dll C:\Windows\System32\msvcrt.dll 7.0.7601.17744
libintl-8.dll C:\Program Files\Git\mingw64\libexec\git-core\libintl-8.dll 0.19.8.0
advapi32.dll C:\Windows\System32\advapi32.dll 6.1.7601.24214
sechost.dll C:\Windows\System32\sechost.dll 6.1.7601.18869
rpcrt4.dll C:\Windows\System32\rpcrt4.dll 6.1.7601.24214
libpcre2-8-0.dll C:\Program Files\Git\mingw64\libexec\git-core\libpcre2-8-0.dll 0.0.0.0
zlib1.dll C:\Program Files\Git\mingw64\libexec\git-core\zlib1.dll 0.0.0.0
user32.dll C:\Windows\System32\user32.dll 6.1.7601.23594
gdi32.dll C:\Windows\System32\gdi32.dll 6.1.7601.23914
lpk.dll C:\Windows\System32\lpk.dll 6.1.7601.24205
usp10.dll C:\Windows\System32\usp10.dll 1.626.7601.23894
ws2_32.dll C:\Windows\System32\ws2_32.dll 6.1.7601.23451
nsi.dll C:\Windows\System32\nsi.dll 6.1.7601.23889
libssp-0.dll C:\Program Files\Git\mingw64\libexec\git-core\libssp-0.dll 0.0.0.0
imm32.dll C:\Windows\System32\imm32.dll 6.1.7600.16385
msctf.dll C:\Windows\System32\msctf.dll 6.1.7601.23915
cryptsp.dll C:\Windows\System32\cryptsp.dll 6.1.7601.23471
rsaenh.dll C:\Windows\System32\rsaenh.dll 6.1.7600.16385
CRYPTBASE.dll C:\Windows\System32\CRYPTBASE.dll 6.1.7601.24214

CPaulButler on 18 Sep 2018

Same for me. Downgrading to 1.18 fixed the issue.

andyraf on 18 Sep 2018

I have the same issue and I can reproduce it with 2 specific binary files:

create a new git repo
copy blob1.bin, blob2.bin from that zip files into repo and add it (commit is not neccessary)
first git gc ... OK

Enumerating objects: 2, done.
Counting objects: 100% (2/2), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), done.
Total 2 (delta 1), reused 0 (delta 0)

second git gc ... FAILED!

Enumerating objects: 2, done.
error: failed to run repack)

blob1.zip
blob2.zip

I hope someone can fix it.

$ git version
git version 2.19.0.windows.1
$ cmd.exe /c ver
Microsoft Windows [Version 10.0.17134.286]

thomas-patzig on 21 Sep 2018

If you have a few minutes spare (~30+), are you able to try a bit of debugging via the SDK and wiki advice?

Also, that second fail, did it really have the trailing ) close brackets?

That would suggest that the line Counting objects: 100% (x/2) had previously been written to the terminal before the error message overwrote it (and before the , done. part is written). This may help home in on the location of the error.

Plus the fact that this was second time around. (maybe a pack hadn't been fully written correctly on Windows)

PhilipOakley on 21 Sep 2018

Sorry, I have no time at this moment. But next week I'll try the SDK.

_Also, that second fail, did it really have the trailing ) close brackets?_
-> Yes. It has a closing bracket.

May this will be helpful:

$ GIT_TRACE=1 git gc
15:00:04.210390 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
15:00:04.211387 git.c:415               trace: built-in: git gc
15:00:04.216374 run-command.c:637       trace: run_command: git pack-refs --all --prune
15:00:04.224406 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
15:00:04.225393 git.c:415               trace: built-in: git pack-refs --all --prune
15:00:04.231336 run-command.c:637       trace: run_command: git reflog expire --all
15:00:04.239314 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
15:00:04.240345 git.c:415               trace: built-in: git reflog expire --all
15:00:04.243304 run-command.c:637       trace: run_command: git repack -d -l -A --unpack-unreachable=2.weeks.ago
15:00:04.251282 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
15:00:04.252320 git.c:415               trace: built-in: git repack -d -l -A --unpack-unreachable=2.weeks.ago
15:00:04.253318 run-command.c:637       trace: run_command: GIT_REF_PARANOIA=1 git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-2628-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
15:00:04.264248 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
15:00:04.265245 git.c:415               trace: built-in: git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-2628-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
Enumerating objects: 2, done.
error: failed to run repack)

thomas-patzig on 21 Sep 2018

I have been working on this issue with @thomas-patzig and can share a few observations that may also be helpful:

This bug does only occur on windows.
This bug seems to be related to packfiles.
Writing the packfile is likely not the issue. The resulting packfile after git gc can be copied to a linux maschine and works fine. That is why the first git gc runs without error.
Producing the packfile on a linux maschine still causes the same error after copying it to the windows maschine.
Reading it seems to cause the issue. The second git gc or a push causes the error while cloning works fine.
However, unpacking the packfile works even on the windows maschine:
git unpack-objects < some_dir/*.pack after moving the packfile to some_dir.

daniel-schroeder-zeiss on 21 Sep 2018

@PhilipOakley

Hi,
I have a compiled git version:

$ git version
git version 2.19.0.windows.1.2.g42a3604fdc

...and the same issue:

$  GIT_TRACE=1 git gc
12:10:30.998489 exec-cmd.c:236          trace: resolved executable dir: C:/git-sdk-64/mingw64/bin
12:10:31.014111 git.c:415               trace: built-in: git gc
12:10:31.014111 run-command.c:637       trace: run_command: git pack-refs --all --prune
12:10:31.029732 exec-cmd.c:236          trace: resolved executable dir: C:/git-sdk-64/mingw64/libexec/git-core
12:10:31.029732 git.c:415               trace: built-in: git pack-refs --all --prune
12:10:31.045354 run-command.c:637       trace: run_command: git reflog expire --all
12:10:31.045354 exec-cmd.c:236          trace: resolved executable dir: C:/git-sdk-64/mingw64/libexec/git-core
12:10:31.061000 git.c:415               trace: built-in: git reflog expire --all
12:10:31.061000 run-command.c:637       trace: run_command: git repack -d -l -A --unpack-unreachable=2.weeks.ago
12:10:31.061000 exec-cmd.c:236          trace: resolved executable dir: C:/git-sdk-64/mingw64/libexec/git-core
12:10:31.076598 git.c:415               trace: built-in: git repack -d -l -A --unpack-unreachable=2.weeks.ago
12:10:31.076598 run-command.c:637       trace: run_command: GIT_REF_PARANOIA=1 git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-9788-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
12:10:31.092245 exec-cmd.c:236          trace: resolved executable dir: C:/git-sdk-64/mingw64/libexec/git-core
12:10:31.092245 git.c:415               trace: built-in: git pack-objects --local --delta-base-offset .git/objects/pack/.tmp-9788-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
Enumerating objects: 2, done.
error: failed to run repack)

What should I test?

thomas-patzig on 24 Sep 2018

Have a check that the write and read process (in git) do complete properly, and that there are no stray (temporary repacking) files left over in directories.

If I remember correctly, the Linux way of preparing the files may conflict with the Windows way (inodes and ownership stuff). I think the git project will be writing a temporary file, and then attempting to rename it in place to become the new pack file. It maybe that some virus scanner has an open handle on the temp file, so windows won't release it for the rename, or one of many variants on those lines.

Maybe have a look at https://github.com/git-for-windows/git/pull/1769 to gain more clues as to the GfW tribulations.

PhilipOakley on 24 Sep 2018

@thomas-patzig Great MCVE! The access violation is caused by using an uninitialized critical section by packing_data_lock(pack) at pack-objects.h#L377:

static inline void oe_set_delta_size(struct packing_data *pack,
                     struct object_entry *e,
                     unsigned long size)
{
    if (size < pack->oe_delta_size_limit) {
        e->delta_size_ = size;
        e->delta_size_valid = 1;
    } else {
        packing_data_lock(pack);
        if (!pack->delta_size)
            ALLOC_ARRAY(pack->delta_size, pack->nr_alloc);
        packing_data_unlock(pack);

        pack->delta_size[e - pack->objects] = size;
        e->delta_size_valid = 0;
    }
}

Moving the initialization to prepare_packing_data() seems to fix the problem:

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index d1144a8f7e..29d48f3867 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2299,7 +2299,6 @@ static void init_threaded_search(void)
        pthread_mutex_init(&cache_mutex, NULL);
        pthread_mutex_init(&progress_mutex, NULL);
        pthread_cond_init(&progress_cond, NULL);
-       pthread_mutex_init(&to_pack.lock, NULL);
        old_try_to_free_routine = set_try_to_free_routine(try_to_free_from_threads);
 }

diff --git a/pack-objects.c b/pack-objects.c
index 6ef87e5683..6070b6d565 100644
--- a/pack-objects.c
+++ b/pack-objects.c
@@ -148,6 +148,8 @@ void prepare_packing_data(struct packing_data *pdata)
                                             1U << OE_SIZE_BITS);
        pdata->oe_delta_size_limit = git_env_ulong("GIT_TEST_OE_DELTA_SIZE",
                                                   1UL << OE_DELTA_SIZE_BITS);
+
+       pthread_mutex_init(&pdata->lock, NULL);
 }

 struct object_entry *packlist_alloc(struct packing_data *pdata,

Change available on my branch pack-lock-init.

kgybels on 25 Sep 2018

👍2

Just to add my +1, I am also hitting this issue with Git for Windows 2.19.0. As it has for others, reverting to Git for Windows 2.18.0 fixes the problem for me.

djneades on 27 Sep 2018

👍1

I built the branch https://github.com/kgybels/git/commits/pack-lock-init, and I have not seen this issue again. At least one of my coworkers has also been using this fix successfully.

CPaulButler on 27 Sep 2018