Julia: Pkg.free deletes data on 0.6 RC3

Created on 12 Aug 2016 · 35Comments · Source: JuliaLang/julia


julia> Pkg.status()
6 required packages:
 - ExcelReaders                  0.6.0
 - Gallium                       0.0.3
 - Lora                          0.5.3
 - Mimi                          0.1.1+             model_graph (dirty)
 - NamedTuples                   1.0.0
 - TypedTables                   0.1.1+             julia-0-5
59 additional packages:
 - ASTInterpreter                0.0.3
 - AbstractTrees                 0.0.3
 - ArgParse                      0.3.1
 - BinDeps                       0.4.2
 - COFF                          0.0.1
 - CRC                           1.1.1
 - CSV                           0.0.9
 - Calculus                      0.1.15
 - CategoricalArrays             0.0.2
 - ColorTypes                    0.2.5
 - Colors                        0.6.6
 - Compat                        0.8.6
 - Conda                         0.2.3
 - DWARF                         0.0.3
 - DataArrays                    0.3.8
 - DataFrames                    0.7.4+             nl/nullable
 - DataStreams                   0.0.6
 - DataStructures                0.4.5
 - Dates                         0.4.4
 - Distributions                 0.10.2
 - DocStringExtensions           0.1.0
 - Docile                        0.5.23
 - Documenter                    0.2.0
 - ELF                           0.0.3
 - FileIO                        0.1.0
 - FixedPointNumbers             0.1.4
 - Formatting                    0.1.5
 - ForwardDiff                   0.2.2
 - FunctionWrappers              0.0.0-             master (unregistered)
 - GZip                          0.2.20
 - Graphs                        0.6.0
 - JSON                          0.6.0
 - JuliaParser                   0.7.4
 - LegacyStrings                 0.1.1
 - LibExpat                      0.2.0
 - MachO                         0.0.3
 - MacroTools                    0.3.2              master
 - NaNMath                       0.2.1
 - NullableArrays                0.0.7
 - ObjFileBase                   0.0.3
 - PDMats                        0.4.2
 - PyCall                        1.7.0
 - Reactive                      0.3.4
 - Reexport                      0.0.3
 - ReverseDiffSource             0.2.3
 - Rmath                         0.1.2
 - SHA                           0.2.0
 - SQLite                        0.3.3+             master
 - SortingAlgorithms             0.1.0
 - StatsBase                     0.9.0
 - StatsFuns                     0.3.0
 - StructIO                      0.0.2
 - TerminalUI                    0.0.1
 - TextWrap                      0.1.6
 - URIParser                     0.1.6
 - VT100                         0.0.1
 - WeakRefStrings                0.1.1
 - WinRPM                        0.2.0
 - Zlib                          0.1.12

At this point I cleaned the Mimi folder, i.e. undid all the workspace edits.

julia> Pkg.free("Mimi")
INFO: Freeing Mimi
INFO: No packages to install, update or remove

julia> Pkg.free("SQLite")
INFO: Freeing SQLite
INFO: Removing CSV v0.0.9
INFO: Removing DataStreams v0.0.6
INFO: Removing LegacyStrings v0.1.1
INFO: Removing LibExpat v0.2.0
INFO: Removing SQLite v0.3.3
INFO: Rolling back deleted LibExpat to v0.2.0
INFO: Rolling back deleted LegacyStrings to v0.1.1
INFO: Rolling back deleted DataStreams to v0.0.6
INFO: Rolling back deleted CSV to v0.0.9
ERROR: GitError(Code:ERROR, Class:Object, `HEAD` not found)
 in reset!(::Base.LibGit2.GitRepo, ::String, ::String) at .\libgit2\libgit2.jl:334
 in restore(::Base.LibGit2.State, ::Base.LibGit2.GitRepo) at .\libgit2\libgit2.jl:505
 in transact(::Base.Pkg.Entry.##22#25{Base.LibGit2.GitRepo,String}, ::Base.LibGit2.GitRepo) at .\libgit2\libgit2.jl:521
 in (::Base.Pkg.Entry.##21#24{String})(::Base.LibGit2.GitRepo) at .\pkg\entry.jl:255
 in with(::Base.Pkg.Entry.##21#24{String}, ::Base.LibGit2.GitRepo) at .\libgit2\types.jl:638
 in free(::String) at .\pkg\entry.jl:247
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at .\pkg\dir.jl:31
 in free(::String) at .\pkg\pkg.jl:183

julia>

bug help wanted libgit2 windows

Source

davidanthoff

Most helpful comment

My code above is 100% reliable to reproduce this in my computer, and I open only one session. I guess from the symptom is when Pkg.free doing something like rm -rf . while some libgit 2 thread still running, Windows locks some files and the removal failed, leaving a broken .git.

ylxdzsw on 12 Jun 2017

👍3

All 35 comments

Oh, and this is on Windows.

davidanthoff on 12 Aug 2016

And I just executed another command in the same session, this is how that looks:

julia> Pkg.status()
ERROR: GitError(Code:ENOTFOUND, Class:Repository, Could not find repository from 'SQLite')
 in Base.LibGit2.GitRepo(::String) at .\libgit2\repository.jl:11
 in installed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at .\pkg\read.jl:207
 in #status#11(::String, ::Function, ::Base.TTY) at .\pkg\entry.jl:130
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::Base.TTY, ::Vararg{Base.TTY,N}) at .\pkg\dir.jl:31
 in status(::Base.TTY) at .\pkg\pkg.jl:136 (repeats 2 times)

julia>

When I look in the SQLite folder and do a git status there, I get:

C:\Users\anthoff\.julia\v0.5\SQLite [(unknown)]> git status
fatal: Not a git repository (or any of the parent directories): .git
C:\Users\anthoff\.julia\v0.5\SQLite [(unknown)]>

There is a .git folder there, but I guess it got corrupted?

davidanthoff on 12 Aug 2016

Same (similar) here:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-rc3+0 (2016-08-22 23:43 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-unknown-linux-gnu

julia> Pkg.add("GLM")
ERROR: GitError(Code:ENOTFOUND, Class:Repository, Could not find repository from 'Compat')
 in Base.LibGit2.GitRepo(::String) at ./libgit2/repository.jl:11
 in with(::Function, ::Type{Base.LibGit2.GitRepo}, ::String, ::Vararg{String,N}) at ./libgit2/types.jl:644
 in installed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at ./pkg/read.jl:207
 in resolve at ./pkg/entry.jl:476 [inlined]
 in edit(::Function, ::String, ::Base.Pkg.Types.VersionSet, ::Vararg{Base.Pkg.Types.VersionSet,N}) at ./pkg/entry.jl:30
 in (::Base.Pkg.Entry.##2#5{String,Base.Pkg.Types.VersionSet})() at ./task.jl:360
 in sync_end() at ./task.jl:311
 in macro expansion at ./task.jl:327 [inlined]
 in add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in add(::String) at ./pkg/pkg.jl:100

julia> versioninfo()
Julia Version 0.5.0-rc3+0
Commit e6f843b (2016-08-22 23:43 UTC)
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, westmere)

floswald on 8 Sep 2016

Compat is missing a .git somehow? How did that happen? What does Pkg.status() say?

tkelman on 8 Sep 2016

same:

julia> Pkg.status()
ERROR: GitError(Code:ENOTFOUND, Class:Repository, Could not find repository from 'Compat')
 in Base.LibGit2.GitRepo(::String) at ./libgit2/repository.jl:11
 in with(::Function, ::Type{Base.LibGit2.GitRepo}, ::String, ::Vararg{String,N}) at ./libgit2/types.jl:644
 in installed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at ./pkg/read.jl:207
 in #status#11(::String, ::Function, ::Base.TTY) at ./pkg/entry.jl:130
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::Base.TTY, ::Vararg{Base.TTY,N}) at ./pkg/dir.jl:31
 in status(::Base.TTY) at ./pkg/pkg.jl:136 (repeats 2 times)

floswald on 8 Sep 2016

check the contents of Pkg.dir("Compat") manually. If there's no .git there, then whatever resulted in it getting deleted is a very bad bug that we need to figure out how to reproduce.

tkelman on 8 Sep 2016

ok hold it for a second, the problem is gone - i was running up against a quota limit (remote machine). seems i got stuck in a bad place where that last .git folder in Compat couldn't be written anymore. took me couple of runs to figure out it was the disk quota (errors messages kept changing).

Pkg.add("GLM") completes without error. this is closed for me.

floswald on 8 Sep 2016

The same thing happens to me when I try to install any package.

adrisouza on 26 Nov 2016

v0.6.0-rc2 has the same problem too. This is the code to reproduce (on a fresh windows installation):

julia> versioninfo()
Julia Version 0.6.0-rc2.0
Commit 68e911b* (2017-05-18 02:31 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

julia> Pkg.init()
INFO: Initializing package repository C:\Users\ylxdzsw\.julia\v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("LanguageServer")
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of JSON from https://github.com/JuliaIO/JSON.jl.git
INFO: Cloning cache of LanguageServer from https://github.com/JuliaEditorSupport/LanguageServer.jl.git
INFO: Cloning cache of Lint from https://github.com/tonyhffong/Lint.jl.git
INFO: Cloning cache of URIParser from https://github.com/JuliaWeb/URIParser.jl.git
INFO: Installing Compat v0.25.2
INFO: Installing JSON v0.8.3
INFO: Installing LanguageServer v0.0.1
INFO: Installing Lint v0.5.2
INFO: Installing URIParser v0.1.8
INFO: Package database updated
INFO: METADATA is out-of-date — you may not have the latest version of LanguageServer
INFO: Use `Pkg.update()` to get the latest versions of your packages

julia> Pkg.checkout("LanguageServer")
INFO: Checking out LanguageServer master...
INFO: Pulling LanguageServer latest master...
INFO: Cloning cache of CSTParser from https://github.com/ZacLN/CSTParser.jl.git
INFO: Cloning cache of Tokenize from https://github.com/KristofferC/Tokenize.jl.git
INFO: Installing CSTParser v0.0.3
INFO: Upgrading JSON: v0.8.3 => v0.12.0
INFO: Installing Tokenize v0.1.8
INFO: Removing Lint v0.5.2

julia> Pkg.rm("LanguageServer")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.free("LanguageServer")
INFO: Freeing LanguageServer
INFO: Removing CSTParser v0.0.3
INFO: Removing Compat v0.25.2
INFO: Removing JSON v0.12.0
INFO: Removing LanguageServer v0.0.1
INFO: Rolling back deleted JSON to v0.12.0
INFO: Rolling back deleted Compat to v0.25.2
INFO: Rolling back deleted CSTParser to v0.0.3
ERROR: GitError(Code:ENOTFOUND, Class:Reference, Revspec 'HEAD' not found.)
Stacktrace:
 [1] macro expansion at .\libgit2\error.jl:99 [inlined]
 [2] Base.LibGit2.GitObject(::Base.LibGit2.GitRepo, ::String) at .\libgit2\repository.jl:116
 [3] reset!(::Base.LibGit2.GitRepo, ::String, ::String) at .\libgit2\libgit2.jl:574
 [4] restore at .\libgit2\libgit2.jl:874 [inlined]
 [5] transact(::Base.Pkg.Entry.##20#23{Base.LibGit2.GitRepo,String}, ::Base.LibGit2.GitRepo) at .\libgit2\libgit2.jl:890
 [6] (::Base.Pkg.Entry.##19#22{String})(::Base.LibGit2.GitRepo) at .\pkg\entry.jl:254
 [7] with(::Base.Pkg.Entry.##19#22{String}, ::Base.LibGit2.GitRepo) at .\libgit2\types.jl:608
 [8] free(::String) at .\pkg\entry.jl:246
 [9] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}})() at .\pkg\dir.jl:36
 [10] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}}, ::String) at .\file.jl:59
 [11] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at .\pkg\dir.jl:36
 [12] free(::String) at .\pkg\pkg.jl:201

julia> Pkg.status()
ERROR: GitError(Code:ENOTFOUND, Class:Repository, Could not find repository from 'LanguageServer')
Stacktrace:
 [1] macro expansion at .\libgit2\error.jl:99 [inlined]
 [2] Base.LibGit2.GitRepo(::String) at .\libgit2\repository.jl:10
 [3] installed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at .\pkg\read.jl:211
 [4] #status#9(::String, ::Function, ::Base.TTY) at .\pkg\entry.jl:130
 [5] status(::Base.TTY) at .\pkg\entry.jl:128
 [6] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}})() at .\pkg\dir.jl:36
 [7] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#status,Tuple{Base.TTY}}, ::String) at .\file.jl:59
 [8] #cd#1(::Array{Any,1}, ::Function, ::Function, ::Base.TTY, ::Vararg{Base.TTY,N} where N) at .\pkg\dir.jl:36
 [9] status(::Base.TTY) at .\pkg\pkg.jl:154 (repeats 2 times)

After that, almost any call of Pkg.* report the same error. Manually inspecting the .git folder under LanguageServer shows

PS C:\Users\ylxdzsw\.julia\v0.6\LanguageServer> git status
fatal: Not a git repository (or any of the parent directories): .git
PS C:\Users\ylxdzsw\.julia\v0.6\LanguageServer> cd .\.git\
PS C:\Users\ylxdzsw\.julia\v0.6\LanguageServer\.git> ls


    目录: C:\Users\ylxdzsw\.julia\v0.6\LanguageServer\.git


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        2017/5/24     11:21                objects
d-----        2017/5/24     11:20                refs

while a normal .git contains the following contents:

PS C:\Users\ylxdzsw\.julia\v0.6\JSON\.git> ls


    目录: C:\Users\ylxdzsw\.julia\v0.6\JSON\.git


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        2017/5/24     11:20                hooks
d-----        2017/5/24     11:20                info
d-----        2017/5/24     11:20                logs
d-----        2017/5/24     11:20                objects
d-----        2017/5/24     11:20                refs
-a----        2017/5/24     11:20            347 config
-a----        2017/5/24     11:20             73 description
-a----        2017/5/24     11:21          19271 FETCH_HEAD
-a----        2017/5/24     11:21             41 HEAD
-a----        2017/5/24     11:21           9632 index

ylxdzsw on 24 May 2017

Thanks for the test case, that should be very helpful for tracking this down.

tkelman on 25 May 2017

I just got this again for a number of Pkg.free calls. I'm on julia 0.6-RC3 on Windows.

A Pkg.status() had this line before I did the Pkg.free:
````

Nulls 0.0.2+ master
Here is the output of the session when the problem occurred:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: https://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.6.0-rc3.0 (2017-06-07 11:53 UTC)
_/ |__'_|_|_|__'_| | Official http://julialang.org/ release
|__/ | x86_64-w64-mingw32

julia> Pkg.free("NodeJS")
INFO: Freeing NodeJS
INFO: No packages to install, update or remove

julia> Pkg.free("Nulls")
INFO: Freeing Nulls
INFO: Removing Nulls v0.0.2
ERROR: GitError(Code:ENOTFOUND, Class:Reference, Revspec 'HEAD' not found.)
Stacktrace:
[1] macro expansion at .\libgit2\error.jl:99 [inlined]
[2] Base.LibGit2.GitObject(::Base.LibGit2.GitRepo, ::String) at .\libgit2\repository.jl:116
[3] reset!(::Base.LibGit2.GitRepo, ::String, ::String) at .\libgit2\libgit2.jl:574
[4] restore at .\libgit2\libgit2.jl:874 [inlined]
[5] transact(::Base.Pkg.Entry.##20#23{Base.LibGit2.GitRepo,String}, ::Base.LibGit2.GitRepo) at .\libgit2\libgit2.jl:890
[6] (::Base.Pkg.Entry.##19#22{String})(::Base.LibGit2.GitRepo) at .\pkg\entry.jl:254
[7] with(::Base.Pkg.Entry.##19#22{String}, ::Base.LibGit2.GitRepo) at .\libgit2\types.jl:608
[8] free(::String) at .\pkg\entry.jl:246
[9] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}})() at .\pkg\dir.jl:36
[10] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#free,Tuple{String}}, ::String) at .\file.jl:59
[11] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at .\pkg\dir.jl:36
[12] free(::String) at .\pkg\pkg.jl:201

julia>
When I then enter into the folder of that package and issue a ``git status`` I get this:
C:\Users\anthoff.julia\v0.6\Nulls [(unknown)]> git status
fatal: Not a git repository (or any of the parent directories): .git
There is a ``.git`` folder under the ``Nulls`` package folder, but it seems to be missing a whole lot of things. The only files present are these:
C:\Users\anthoff.julia\v0.6\Nulls [(unknown)]> gci .git -Recurse

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git

Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/7/2017 3:40 PM objects
d----- 5/30/2017 3:23 PM refs
-a---- 5/30/2017 3:23 PM 308 packed-refs

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\objects

Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/2/2017 12:21 PM pack

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\objects\pack

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 6/2/2017 12:21 PM 1800 pack-2750b831b344bb1987a38b488915c5080b48b695.idx
-ar--- 6/2/2017 12:21 PM 5096 pack-2750b831b344bb1987a38b488915c5080b48b695.pack
-ar--- 5/30/2017 3:23 PM 7092 pack-53c017e5035c81ad7b3de9ee29b0ce0096cf732a.idx
-ar--- 5/30/2017 3:23 PM 30381 pack-53c017e5035c81ad7b3de9ee29b0ce0096cf732a.pack

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\refs

Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/2/2017 12:21 PM heads
d----- 5/30/2017 3:23 PM remotes
d----- 6/2/2017 12:21 PM tags

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\refs\heads

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 6/2/2017 12:21 PM 41 master

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\refs\remotes

Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/2/2017 12:21 PM origin

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\refs\remotes\origin

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 5/30/2017 3:23 PM 32 HEAD
-a---- 6/2/2017 12:21 PM 41 master

Directory: C:\Users\anthoff\.julia\v0.6\Nulls\.git\refs\tags

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 6/2/2017 12:21 PM 41 v0.0.2
``I believe this should be release blocking for julia 0.6. Before I didPkg.free()`` for the Nulls package I did it for another package of mine that I'm working on, and as far as I can tell it deleted e.g. one branch that I hadn't pushed anywhere, so this resulted in real data/code loss.

CC @tkelman as the release master.

davidanthoff on 8 Jun 2017

Would be good to try walking through these test sequences of operations and determine if this is windows-specific or can happen on mac or linux too

tkelman on 8 Jun 2017

This bug is really weird. I get it regularly, every couple of days, but so far I can't detect any pattern of when it occurs... For me it is always when I do a Pkg.free, but often Pkg.free works just fine...

One thing that might be a commonality (but I'm not sure!) is that I often open two julia prompts, I do a Pkg.status() in one, and then do Pkg.free() in the other one on various packages. So one thing to check might be whether there is a pattern in the sense that this bug only appears if there are two julia instances running, and one had already issued some Pkg commands in the first one. But I should say that this might be a wrong hint, I haven't carefully noted whether this was always the case when things appeared or not.

davidanthoff on 8 Jun 2017

I really think this bug should get release blocking status. This is a fairly common code path Pkg.free() deleting permanently user code/data in package directories, which seems a really nasty bug to me. Yes, it is not a regression from julia 0.5 because it has the same bug, but still...

davidanthoff on 9 Jun 2017

@tkelman should figure out if this should have the 0.6.0 tag.

ViralBShah on 9 Jun 2017

If it's been around for nearly a year and only a handful of people have hit it, I'm inclined to say it can wait until a point release. If people (especially those familiar with the libgit2 bindings and how Pkg is using them) can give it some attention and get it debugged and fixed in the next few days, great, it can make 0.6.0.

tkelman on 9 Jun 2017

and only a handful of people have hit it

We don't know that. We know how many have spoken up here.

If people (especially those familiar with the libgit2 bindings and how Pkg is using them) can give it some attention and get it debugged and fixed in the next few days, great, it can make 0.6.0.

Could someone ping those folks? I don't know who is working on this part of the code.

davidanthoff on 9 Jun 2017

Has anyone pinged the folks that are maintaining that part of the code? This is a really serious bug that deletes, permanently, work in package directories.

I was just hit by this bug again and lost another round of work. Yes, at this point I should know better and just not call Pkg.free, but that really doesn't look like a solution to me. If this can't be fixed for the 0.6 release, then Pkg.free should really just be disabled on Windows and people should be told to free packages manually with git.

My gut impression is that I've never run into this bug on my first Pkg.free() in a given julia session. Again, not a 100% sure, but that is my hunch.

davidanthoff on 11 Jun 2017

Can you confirm that every time it happened you had two Julia sessions open in parallel and run Pkg operations in each of them?

nalimilan on 11 Jun 2017

No, not a 100% sure. In fact I'm pretty sure that the last time this happened I just had one julia session open, but I am sure that I did a Pkg.status(), then at least one Pkg.free() and then the Pkg.free() with the error in a row. There might have been multiple Pkg.free() in a row, i.e. it could have been the third, fourth etc. Pkg.free. I really should write down what exactly is running when I come across this... I'll try next time.

davidanthoff on 11 Jun 2017

ylxdzsw on 12 Jun 2017

👍3

Just an observation--don't know if this is helpful:

I recently added Tabulars.jl. It was broken and unregistered. When I attempted Pkg.update() I received a similar error.

ERROR: GitError(Code:ENOTFOUND, Class:Reference, Revspec 'HEAD' not found.)

I noticed that the Compat package also was trying to update. When I removed the Tabulars.jl and then ran Pkg.update() again, the issue resolved itself.

sdmcallister on 20 Jul 2017

I have a similar problem with the installation of several packages:

Pkg.add("Documenter")
ERROR: GitError(Code:ENOTFOUND, Class:Reference, Revspec 'HEAD' not found.)
macro expansion at ./libgit2/error.jl:99 [inlined]
Base.LibGit2.GitTree(::Base.LibGit2.GitRepo, ::String) at ./libgit2/repository.jl:116
#isdiff#76(::Bool, ::Function, ::Base.LibGit2.GitRepo, ::String, ::String) at ./libgit2/libgit2.jl:147
(::Base.LibGit2.#kw##isdiff)(::Array{Any,1}, ::Base.LibGit2.#isdiff, ::Base.LibGit2.GitRepo, ::String, ::String) at ./<missing>:0
(::Base.Pkg.Read.##5#6{String})(::Base.LibGit2.GitRepo) at ./pkg/read.jl:184
with(::Base.Pkg.Read.##5#6{String}, ::Base.LibGit2.GitRepo) at ./libgit2/types.jl:608
requires_path(::String, ::Dict{VersionNumber,Base.Pkg.Types.Available}) at ./pkg/read.jl:183
fixed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}, ::Set{String}, ::VersionNumber) at ./pkg/read.jl:229
fixed(::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}) at ./pkg/read.jl:225
edit(::Function, ::String, ::Base.Pkg.Types.VersionSet, ::Vararg{Base.Pkg.Types.VersionSet,N} where N) at ./pkg/entry.jl:30
(::Base.Pkg.Entry.##1#3{String,Base.Pkg.Types.VersionSet})() at ./task.jl:335
Stacktrace:
 [1] sync_end() at ./task.jl:287
 [2] macro expansion at ./task.jl:303 [inlined]
 [3] add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51
 [4] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:36
 [5] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:70
 [6] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at ./pkg/dir.jl:36
 [7] add(::String) at ./pkg/pkg.jl:117

using Julia 0.6.0:

julia> versioninfo()
Julia Version 0.6.0
Commit 903644385b (2017-06-19 13:05 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

ghost on 21 Sep 2017

Is there a timeline for fixing this? I really want to update from 0.4 to something more recent, but this is blocking.

JaredCrean2 on 7 Oct 2017

@JaredCrean2 Do you mean you experience it all the time?

nalimilan on 7 Oct 2017

No, I think 0.4 is unaffected by this bug, so I am delaying updating to a newer version to avoid being affected.

JaredCrean2 on 7 Oct 2017

AFAICT only a limited share of Windows users are affected by this bug (it's unclear what triggers it), so I wouldn't consider it as a reason not to upgrade without trying.

nalimilan on 7 Oct 2017

@axel-sdk's post showed he was running on macOS, so the problem may be more widespread. Until more is known about what causes this, I don't think the possibility of losing research progress because of this bug is an acceptable risk.

JaredCrean2 on 7 Oct 2017

Well, you can avoid calling Pkg.free, and do that operation manually using git.

I've been able to reproduce the problem systematically under Windows 10 using the instructions given by @ylxdzsw. It's actually quite easy to reproduce at will once you've run the commands: after deleting the LanguageServer package directly and the .trash directory, running Pkg.add and the remaining commands gives the same error.

Thanks to this, I think I have made some progress with the debugging. It seems the problem comes the combination of two things:

After Pkg.checkout, Pkg.rm removes the package from REQUIRE, but the directory is not moved to .trash. This happens even after restarting Julia before calling Pkg.rm. Calling Pkg.rm a second time removes it, though (with the additional message (unregistered)).
Pkg.free applies changes to the package directory, but does not add it to REQUIRE. So when it calls resolve an attempt is made at removing the package (as can be seen from the info messages).

It's not clear to me why the second step fails, though. Listing file handles of julia.exe does not show any file under the package directory once each operation has completed. Also, restarting Julia before calling Pkg.free does not fix the problem. Overall it seems that Pkg.free has issues when a package is present but not in REQUIRE. Maybe it should just add the freed package to REQUIRE before calling resolve?

nalimilan on 7 Oct 2017

I have a fix for the corruption of git files at https://github.com/JuliaLang/julia/pull/24048. But since it calles resolve, Pkg.free still removes packages which are not in REQUIRE and not needed by any package in listed in that file. Actually, fix that fix, it will even succeed to do so instead of failing half of the way.

Can the reporters confirm that they would have expected Pkg.free to add the package to REQUIRE if it wasn't listed there for some reason (e.g. if you checked it out manually or you created it), so that the package is not removed? If so, that will require a second fix (more of a change in behavior actually).

nalimilan on 8 Oct 2017

Unregistered packages shouldn't be added to REQUIRE should they?

If a package is only installed as a dependency of something else, I don't think Pkg.free should change that.

tkelman on 9 Oct 2017

Unregistered packages shouldn't be added to REQUIRE should they?

Good catch, I guess they shouldn't.

If a package is only installed as a dependency of something else, I don't think Pkg.free should change that.

In theory I agree, but apparently people have been surprised by the behavior where Pkg.free can delete a package, by triggering a call to resolve. The fact that there was an error in the middle of the deletion, which persisted due to corrupting the git repo, only made matters more complex here, but fixing that error won't fix the data loss issue AFAICT. Or maybe there's another issue that I missed.

Overall it seems that the simplest fix would be to go back to moving removed packages to .trash, at least when the deletion happens automatically.

nalimilan on 9 Oct 2017

Is this PR getting close to being merge-able?

JaredCrean2 on 17 Oct 2017

I have the same problem, I am getting the same error:

ERROR: GitError(Code:ENOTFOUND, Class:Repository, Could not find repository from 'METADATA')

When I look in any of my folders in METADATA, none of them have a .git file. I am using a clean reinstall on Windows 10.

> versioninfo()
Julia Version 0.6.2
Commit d386e40c17* (2017-12-13 18:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

xpost at discourse