Pkg.jl: manifest_usage.toml: "duplicate key time" error

[goedman]

In both Julia 1.0.1 and 1.1.0-DEV.614 I get below error:

(v1) pkg> gc
ERROR: Pkg.TOML.ParserError(121, 125, "duplicate key time")
Stacktrace:
[1] parse(::Base.GenericIOBuffer{Array{UInt8,1}}) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:33
[2] parse at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:40 [inlined]
[3] #gc#43(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:299
[4] gc at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:290 [inlined]
[5] do_gc!(::Dict{Symbol,Any}, ::Array{String,1}, ::Dict{Symbol,Any}) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:668

I did reinstall all packages, but that didn't fix it. I also tried a bunch of packages on the release-1.0 branch, about the same time the problem started to appear, but I don't think they are related.

Any ideas?

Thanks,
Rob

[KristofferC]

You would have to look at .julia/logs/manifest_usage.toml. It seems to have gotten corrupted.

[goedman]

Thank you Kristoffer.

The usage of that file is not very clear to me, the offending lines:

lines 118 to 129 [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:23:30.122Z [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:23:35.661Z [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:27:45.159Z [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:28:02.049Z [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:28:06.712Z [["/Users/rob/.julia/environments/v1/Manifest.toml"]] time = 2018-11-02T07:32:33.656Z

After experimenting a bit (few hours) longer I gave up. Started to grasp on to ridiculous ideas like maybe this was caused by the DST change! Clearly I have no idea what this file means. Anyway, wiped out all of .julia and reinstalled.

Thanks for your help and work!

[KristofferC]

The file is a log of Manifest use so we know what projects are considered "active" and the gc doesn't delete packages that are still needed.

[StefanKarpinski]

Having a duplicate time key probably means that two writes somehow ended up getting interleaved with two time = lines in a single [[ section.

[goedman]

So maybe this did happen when I was trying release-1.0 simultaneously (just testing all my registered packages one after the other) while continuing 'normal' work in v1.1.0-DEV (in another terminal)? I also use the setup where environments v1.1 and v1.0 are symbolic links to v1, but from Kristoffers's reply I take it the log must be shared across all environments.

[StefanKarpinski]

Yes, the log is shared—this could very well be an issue of multiple instances of Julia writing to the usage log at the same time.

[goedman]

Thanks Stefan. Will close this issue for now.

[StefanKarpinski]

I'm keeping it open since it's definitely a problem even if we don't have an immediate solution.

[goedman]

Hmmm, not good. It's back.

The manifest_usage.tml only contains 3 lines:

[["/Users/rob/.julia/dev/StatisticalRethinking/Manifest.toml"]]
time = [["/Users/rob/.julia/environments/v1.0/Manifest.toml"]]
time = 2018-11-07T15:16:28.053Z

At the end go ] gc I noticed:

(v1.0) pkg> gc
    Active manifests:
        `~/.julia/environments/v1.0/Manifest.toml`
        `~/.julia/dev/StatisticalRethinking/Manifest.toml`
   Deleted `~/.julia/packages/Observables/RkKyn` (19.931 KiB)
   Deleted `~/.julia/packages/TransformVariables/G0eaF` (40.185 KiB)
   Deleted `~/.julia/packages/WebIO/QjKwf` (581.386 KiB)
ERROR: type DateTime has no field zone
Stacktrace:
 [1] getproperty(::Any, ::Symbol) at ./sysimg.jl:18
 [2] format(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Dates.DatePart{'Z'}, ::Dates.DateTime, ::Dates.DateLocale) at /Users/rob/.julia/packages/TimeZones/JvOCz/src/parse.jl:104

<snipped>

and then:

(v1.0) pkg> gc
ERROR: Pkg.TOML.ParserError(129, 133, "duplicate key `time`")
Stacktrace:
 [1] parse(::Base.GenericIOBuffer{Array{UInt8,1}}) at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:33
 [2] parse at /Users/rob/Projects/Julia/julia/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:40 [inlined]

<snipped>

This happened in Julia 1.1.0-DEV.638 and also only after I had created a symbolic link from v1.1 back to v1.0 in .julia/environments. For quite a while I just used Julia 1.0 (no symlink) without problems. Hope this helps a bit.

[goedman]

Just to see if that works, I deleted the manifest_usage.toml file. Restarted Julia (dev), did ] up (which recreated a new usage file:

[["/Users/rob/.julia/environments/v1.0/Manifest.toml"]]
time = 2018-11-07T15:38:57.651Z

and ] gc is back working:

(v1.0) pkg> up
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `https://github.com/goedman/BHAAnalysis.jl`
  Updating git-repo `https://github.com/goedman/BHAPlotRecipes.jl`
  Updating git-repo `https://github.com/goedman/UnitfulOFU.jl`
  Updating git-repo `https://github.com/goedman/BoreHoleAssemblyAnalysis.jl`
 Resolving package versions...
  Updating `~/.julia/environments/v1.0/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.0/Manifest.toml`
 [no changes]

(v1.0) pkg> gc
    Active manifests:
        `~/.julia/environments/v1.0/Manifest.toml`
   Deleted 0 package installations

No packages are actually deleted in this case.

[KristofferC]

Yeah, this is hopefully fixed by #891.

Do you happen to run multiple julia at once doing package operations?

[goedman]

When I reported it yesterday I knew I had done that in the days before I noticed the problem.

But since I reported it, and you and Stefan replied, I have not done that anymore. I noticed the problem about an hour after I (re-)created the symlink from v1.1 to v1.0 and switched to juliagit. But then again, as Stefan pointed out, the usage file is shared anyway.

[StefanKarpinski]

I guess we'll see about if #891 helps. If it doesn't then we'll have to consider locking or sharding the file somehow.

[KristofferC]

Likely fixed.

[goedman]

Cool, will keep my eyes open. The last few times it occurred was mid yesterday.

[KristofferC]

The fix is not in 1.0.2 but will be in 1.0.3. (or 1.1.0).

[asinghvi17]

Still happening, same issue on Julia v1.1

julia> versioninfo(verbose=true)
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  uname: Darwin 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64 i386
  CPU: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz:
              speed         user         nice          sys         idle          irq
       #1  2900 MHz      84648 s          0 s      83354 s     777976 s          0 s
       #2  2900 MHz       6011 s          0 s       5361 s     934241 s          0 s
       #3  2900 MHz      71737 s          0 s      43738 s     830141 s          0 s
       #4  2900 MHz       5994 s          0 s       4370 s     935248 s          0 s
       #5  2900 MHz      61511 s          0 s      32193 s     851912 s          0 s
       #6  2900 MHz       5871 s          0 s       3853 s     935888 s          0 s
       #7  2900 MHz      48224 s          0 s      22219 s     875172 s          0 s
       #8  2900 MHz       5776 s          0 s       3332 s     936504 s          0 s

  Memory: 16.0 GB (1139.7578125 MB free)
  Uptime: 192495.0 sec
  Load Avg:  1.138671875  1.31201171875  1.18505859375
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  XPC_FLAGS = 0x0
  PATH = /Users/anshul/anaconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:.:/Library/TeX/texbin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands
  TERM = xterm-256color
  HOME = /Users/anshul
  DYLD_LIBRARY_PATH = /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia

I'm going to nuke my .julia and reinstall everything, hopefully that helps.

[StefanKarpinski]

Is your home directory on a distributed file system like NFS?

[asinghvi17]

Nope, APFS.

[goedman]

Since #891 it is definitely occurring less often, but I delete manifest_usage.toml about once a week. It is very hard to figure out why it is happening. I switch regularly between Julia 1.0, 1.1 and 1.2. If I'm in 1.2 and pull & make 1.2 it is almost a given that the problem occurs after the next up.

I'm also on OSX. Over the last couple of months I've reinstalled all packages on all versions of Julia a number of times (usually as I ran into problems with unregistered packages or no compatible versions of packages), but that does not fix this problem permanently. As was suggested in November, I exit Julia, delete the manifest_usage.toml file, redo the update, followed by gc. Always seems to work.

[StefanKarpinski]

What release of 1.0.x are you using? We backported the fix to write usage info in a single block in 1.0.3, so if you're using 1.0.0-1.0.2 then you won't have that fix. Another possibility is that due to libuv/Julia's lack of POSIX-compliant open/append mode. My understanding is that what we do in append mode is open the file and fast-forward to the end and then write from there, regardless of whether another process has written more data since we fast-forwarded, whereas POSIX append mode should always write data at the current end of the file in append mode. @vtjnash, am I getting that right?

[vtjnash]

yes

[AndiMD]

I can confirm this issue exists as well for the Julia 1.0.3 64bit Linux binary
ERROR: Pkg.TOML.ParserError(121, 125, "duplicate key time")

[StefanKarpinski]

I have a very hard time understanding how people are triggering this so reliably. The file is opened for append, immediately written to, and then closed. How is it possible that multiple different Julia processes are trying to do that at the same time? Are y'all running non-interactive scripts that are doing package operations often? It's hard to imagine how a human operator could trigger this.

[AndiMD]

I noticed this issue on two Linux machines with a similar Julia 1.0.3 install. Not sure when the file was initially corrupted, I noticed it when ]gc produced said error.
I use PackageCompiler to build a Makie sysimg on both machines, and start Julia with julia -J "$HOME/.julia/packages/PackageCompiler/oT98U/sysimg/sys.so". Could this be related?

[goedman]

Stefan, I'm really sorry we (I) can't do a better job in trying to help out. I stopped reporting the problem because no-one else was reporting it so maybe this was due to the fact that I usually work on auto-installed OSX minor betas.

For testing (just to be sure, they are clearly very compatible for my type of work!) I switch often between Julia 1.0.3, 1.1 and 1.2-DEV.

I do a build at least once a day, on average maybe 4 times, always in a window separate from the REPL window:clear; git pull && make while I continue to work in the REPL 1.2-DEV until the final notice LINK usr/lib/julia/sys.dylib. Then I restart the REPL.

I run ] up followed by ] gc very frequently (20 times a day?).

Not sure if this triggers any more ideas ...

[ikselven]

I am seeing this issue on Julia 1.1.0 (x86_64 binaries from https://julialang.org) on Arch Linux. As the two commentators above me mention, it happens for me too when running ] up followed by ] gc. However, I cannot reproduce this consistently. Sometimes it works as expected without any error and sometimes it corrupts the manifest_usage.toml.

I'm using only 1.1.0 at the moment and no other Julia version in parallel.

[StefanKarpinski]

Ok, the up followed by gc thing seems like it might be a useful lead. @AndiMD, @asinghvi17, does that pattern sound familiar to you as well. It would help explain why I've never seen this—I almost never run gc, I just let my garbage build up 😂

[asinghvi17]

Yep, that's usually when it happens for me - ]up immediately followed by ]gc.

[fredrikekre]

So, something like

julia> while true
           Pkg.add(PackageSpec(name="Example", version = "0.5.0"))
           Pkg.update() # will up Example to 0.5.1
           Pkg.gc()
       end

triggers this for you? It doesn't happen for me.

[StefanKarpinski]

I can't repro either but something like this might help:

diff --git a/src/API.jl b/src/API.jl
index 946f1ea..2532f12 100644
--- a/src/API.jl
+++ b/src/API.jl
@@ -362,9 +362,11 @@ function gc(ctx::Context=Context(); kwargs...)

     # Write the new condensed usage file
     if !ctx.preview
-        open(usage_file, "w") do io
+        usage_file_tmp = "$usage_file.$(Random.randstring()).tmp"
+        open(usage_file_tmp, "w") do io
             TOML.print(io, new_usage, sorted=true)
         end
+        mv(usage_file_tmp, usage_file, force=true)
     end
     ndel = length(paths_to_delete)
     byte_save_str = ndel == 0 ? "" : (" (" * pretty_byte_str(sz) * ")")

It makes the writing of the new usage file after gc atomic by writing to a temp file and then moving it. We should probably do this anyway and see if it helps.

[goedman]

Just ran Fredrik's script parallel on v1.1 and v1.2-DEV and at the same time did the `clear; git pull && make (and it needed to rebuild, so no 'Already up to date' message). But no problems.

[asinghvi17]

Perhaps it could be tied to having two different Pkg environments? That's what I noticed happening - when switching from one Pkg environment to another (PackageCompiler sysimage or 'vanilla' Julia v1.1) and doing ]up then ]gc, the error occurs quite often.

[AndiMD]

@StefanKarpinski Not sure, I rarely use the ]gc command.

I can reproduce it with the following steps (not sure if all are required):
1) rm .julia/logs/*
2) start julia
3) ]gc
4) ]instantiate
5) using PackageCompiler
6) PackageCompiler.compile_incremental(:Makie, :AbstractPlotting, force = false)
7) start julia with the sysimg: julia -J "$HOME/.julia/packages/PackageCompiler/oT98U/sysimg/sys.so"
8) ]gc

    Active manifests:
        `~/.julia/packages/PackageCompiler/oT98U/packages/Makie/Manifest.toml`
        `~/.julia/packages/PackageCompiler/oT98U/packages/AbstractPlotting/Manifest.toml`
        `~/.julia/environments/v1.0/Manifest.toml`
        `~/.julia/packages/PackageCompiler/oT98U/packages/Manifest.toml`
ERROR: type DateTime has no field zone
Stacktrace:
 [1] getproperty(::Any, ::Symbol) at ./sysimg.jl:18
 [2] format(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Dates.DatePart{'Z'}, ::Dates.DateTime, ::Dates.DateLocale) at /home/andi/.julia/packages/TimeZones/WMDpl/src/parse.jl:104
 [3] format(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Dates.DateTime, ::Dates.DateFormat{Symbol("YYYY-mm-ddTHH:MM:SS.sssZ"),Tuple{Dates.DatePart{'Y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'Z'}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:481
 [4] format(::Dates.DateTime, ::Dates.DateFormat{Symbol("YYYY-mm-ddTHH:MM:SS.sssZ"),Tuple{Dates.DatePart{'Y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'Z'}}}, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:488
 [5] format(::Dates.DateTime, ::Dates.DateFormat{Symbol("YYYY-mm-ddTHH:MM:SS.sssZ"),Tuple{Dates.DatePart{'Y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'Z'}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:487
 [6] #format#31(::Dates.DateLocale, ::Function, ::Dates.DateTime, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:527
 [7] format at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:527 [inlined]
 [8] #printvalue#2 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/print.jl:38 [inlined]
 [9] #printvalue at ./none:0 [inlined]
 [10] #_print#3(::Int64, ::Bool, ::Bool, ::Function, ::Function, ::IOStream, ::Dict{String,Dates.DateTime}, ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/print.jl:66
 [11] (::getfield(Pkg.TOML, Symbol("#kw##_print")))(::NamedTuple{(:indent, :sorted, :by),Tuple{Int64,Bool,typeof(identity)}}, ::typeof(Pkg.TOML._print), ::IOStream, ::Dict{String,Dates.DateTime}, ::Array{String,1}) at ./none:0
 [12] #_print#3(::Int64, ::Bool, ::Bool, ::Function, ::Function, ::IOStream, ::Dict{String,Any}, ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/print.jl:99
 [13] #_print at ./none:0 [inlined]
 [14] #print#4 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/print.jl:106 [inlined]
 [15] (::getfield(Pkg.TOML, Symbol("#kw##print")))(::NamedTuple{(:sorted,),Tuple{Bool}}, ::typeof(Pkg.TOML.print), ::IOStream, ::Dict{String,Any}) at ./none:0
 [16] (::getfield(Pkg.API, Symbol("##45#48")))(::IOStream) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:397
 [17] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Pkg.API, Symbol("##45#48")), ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
 [18] open at ./iostream.jl:367 [inlined]
 [19] #gc#43(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:396
 [20] gc at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:290 [inlined]
 [21] do_gc!(::Dict{Symbol,Any}, ::Array{String,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:668
 [22] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:697
 [23] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:696
 [24] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:603
 [25] #do_cmd#33(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:577
 [26] do_cmd at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:573 [inlined]
 [27] (::getfield(Pkg.REPLMode, Symbol("##50#53")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:923
 [28] #invokelatest#1 at ./essentials.jl:697 [inlined]
 [29] invokelatest at ./essentials.jl:696 [inlined]
 [30] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/LineEdit.jl:2273
 [31] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:1034
 [32] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:191
 [33] (::getfield(Base, Symbol("##718#720")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:355
 [34] #invokelatest#1 at ./essentials.jl:697 [inlined]
 [35] invokelatest at ./essentials.jl:696 [inlined]
 [36] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:339
 [37] exec_options(::Base.JLOptions) at ./client.jl:277
 [38] _start() at ./client.jl:425

9) ]gc

ERROR: Pkg.TOML.ParserError(143, 147, "duplicate key `time`")
Stacktrace:
 [1] parse(::Base.GenericIOBuffer{Array{UInt8,1}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/TOML.jl:33
 [2] parse at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/ext/TOML/src/TOML.jl:40 [inlined]
 [3] #gc#43(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:299
 [4] gc at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:290 [inlined]
 [5] do_gc!(::Dict{Symbol,Any}, ::Array{String,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:668
 [6] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:697
 [7] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:696
 [8] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:603
 [9] #do_cmd#33(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:577
 [10] do_cmd at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:573 [inlined]
 [11] (::getfield(Pkg.REPLMode, Symbol("##50#53")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:923
 [12] #invokelatest#1 at ./essentials.jl:697 [inlined]
 [13] invokelatest at ./essentials.jl:696 [inlined]
 [14] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/LineEdit.jl:2273
 [15] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:1034
 [16] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:191
 [17] (::getfield(Base, Symbol("##718#720")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:355
 [18] #invokelatest#1 at ./essentials.jl:697 [inlined]
 [19] invokelatest at ./essentials.jl:696 [inlined]
 [20] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:339
 [21] exec_options(::Base.JLOptions) at ./client.jl:277
 [22] _start() at ./client.jl:425

[KristofferC]

Should be fixed by https://github.com/JuliaLang/Pkg.jl/pull/1004 which AFAIU have not been backported to a non-master julia version yet.

[goedman]

In fact since the middle of last week I have not seen the problem anymore! Have pretty much only worked on nightlies.

[StefanKarpinski]

Ok, has the ERROR: type DateTime has no field zone occurred for everyone else before this happened too? I had missed that fact but @goedman did mention it in https://github.com/JuliaLang/Pkg.jl/issues/885#issuecomment-436803046 back in November. It's entirely possible then that this is not a race condition problem at all but a case of and error causing incomplete TOML output to be written.

[asinghvi17]

Yep, happens for me too.

[KristofferC]

Closing until someone can repro on julia 1.2+.