Julia: Pkg.status slow on Windows

[davidanthoff]

It is of course much better on master than on 0.4, but it still takes a fair bit of time. I get the following times, with only 9 packages installed:

1) Fresh boot, wait five minutes, start julia, time Pkg.status: 24s
2) Immediately after 1), time Pkg.status() again: 2s
3) Immediately after 2), exit julia, start it again, time again: 4s
4) Immediately after 3), time again: 2s

I assume that 3) is much faster than 1) because the relevant files are still cached by the OS. All of these timings are on a fast Thinkpad X1 with a super fast SSD hard drive.

My takeaway: 1) seems really excessive. The difference between 3) and 4) seems ok, but I feel that even 2) and 4) by themselves are laggish.

Just as a comparison, pip list feels instantaneously for a much larger number of packages.

This is not a catastrophe at all, and so definitely shouldn't be a high priority case.

[lobingera]

not only on Window. I have not measured, but i've seen few-second waiting times for Pkg.status() and i wondered where the time is spend.

[wildart]

I guess if construction of in-memory copy of METADATA (Pkg.available call) can be avoided, it would greatly improve running time.

[tkelman]

Windows has a slow filesystem, and the way metadata is designed right now with so many small files is pretty hard on the filesystem to initially populate. If all versions of the packages in metadata were defined in a single file per package it might help, though that file would necessarily be more work to parse.

[lobingera]

It's worse than expected

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-linux-gnu

julia> @time Pkg.status()
29 required packages:
 - BinDeps                       0.3.21
 - Cairo                         0.2.31
 - Clang                         0.1.0+             master
 - Colors                        0.6.3
 - CxxWrap                       0.1.4+             master
 - Dates                         0.4.4
 - GR                            0.10.0
 - Gadfly                        0.4.0+             image (dirty)
 - GraphViz                      0.0.4+             master
 - Graphs                        0.6.0
 - Gtk                           0.9.3+             master
 - GtkUtilities                  0.0.8
 - HttpParser                    0.1.1
 - IJulia                        1.1.9
 - ImageView                     0.1.19+            master
 - Images                        0.5.4+             master
 - Immerse                       0.0.11+            master
 - JLD                           0.5.9
 - NearestNeighbors              0.0.3
 - PerceptualColourMaps          0.1.0
 - Plots                         0.5.4+             master
 - ProfileView                   0.1.2
 - PyCall                        1.4.0
 - PyPlot                        2.1.1
 - TimeSeries                    0.8.1
 - Tk                            0.3.7
 - Vega                          0.6.7
 - WinRPM                        0.1.15
 - Winston                       0.11.13+           master
75 additional packages: 
 - ArrayViews                    0.6.4
 - Benchmark                     0.1.0
 - Blosc                         0.1.5
 - Calculus                      0.1.14
 - Codecs                        0.1.5
 - ColorBrewer                   0.3.0
 - ColorMaps                     0.0.0-             master (unregistered)
 - ColorTypes                    0.2.2
 - ColorVectorSpace              0.1.4
 - Compat                        0.7.14
 - Compose                       0.4.2+             image
 - Conda                         0.1.9
 - ConfParser                    0.0.7
 - Contour                       0.1.0
 - Cxx                           0.0.0-             master (unregistered)
 - DataArrays                    0.2.20
 - DataFrames                    0.7.0
 - DataStructures                0.4.4
 - Devectorize                   0.4.2
 - Distances                     0.3.0+             master
 - Distributions                 0.8.10
 - Docile                        0.5.23
 - DualNumbers                   0.2.2
 - FactCheck                     0.4.2
 - FileIO                        0.0.5
 - FixedPointNumbers             0.1.3
 - FixedSizeArrays               0.1.0
 - FunctionalCollections         0.1.3
 - GZip                          0.2.18
 - Graphics                      0.1.3
 - Grid                          0.4.0
 - GtkIDE                        0.0.0-             master (unregistered)
 - GtkSourceWidget               0.0.0-             master
 - HDF5                          0.6.0
 - HTTPClient                    0.2.0
 - Hexagons                      0.0.4
 - HttpCommon                    0.2.4
 - ImmutableArrays               0.0.11
 - IniFile                       0.2.5
 - Iplot                         0.0.0-             master (unregistered, dirty)
 - Iterators                     0.1.9
 - JSON                          0.5.0
 - KernelDensity                 0.1.2
 - LaTeXStrings                  0.1.6
 - LibCURL                       0.2.0
 - LibExpat                      0.1.2
 - Loess                         0.0.6
 - MacroTools                    0.3.0
 - Measures                      0.0.2
 - NaNMath                       0.2.1
 - Nettle                        0.2.3
 - NoveltyColors                 0.1.0
 - Optim                         0.4.5
 - PDMats                        0.4.1
 - PackageEvaluator.jl           0.0.0-             master (unregistered)
 - Patchwork                     0.2.2
 - PositiveFactorizations        0.0.1
 - QML                           0.0.0-             master (unregistered)
 - Qwt                           0.0.1+             master
 - Reexport                      0.0.3
 - Requires                      0.2.2
 - Rsvg                          0.0.1+             master (dirty)
 - Rsvg_frame                    0.0.0-             master (unregistered, dirty)
 - SHA                           0.1.2
 - SIUnits                       0.0.6
 - Showoff                       0.0.6
 - SortingAlgorithms             0.0.6
 - StatsBase                     0.8.0
 - StatsFuns                     0.2.0
 - TexExtensions                 0.0.3
 - URIParser                     0.1.3
 - WoodburyMatrices              0.1.5
 - XCB                           0.0.0-             master (unregistered)
 - ZMQ                           0.3.1
 - Zlib                          0.1.12
 31.242847 seconds (3.19 M allocations: 437.844 MB, 0.48% gc time)

julia> @time Pkg.status()
...
  6.531655 seconds (2.43 M allocations: 404.623 MB, 2.28% gc time)

[wildart]

I played around the package status report. Instead of reading METADATA directory files, I used repository tree and blobs.

julia> @time Pkg.status()
9 required packages:
 - Cairo                         0.2.31
 - Compat                        0.7.18
 - DataFrames                    0.7.2
 - FactCheck                     0.4.2
 - JSON                          0.5.0
 - LMDB                          0.0.4
 - MultivariateStats             0.2.2
 - StatsBase                     0.8.1
 - YAML                          0.1.9+             nightly
19 additional packages:
 - ArrayViews                    0.6.4
 - BinDeps                       0.3.21
 - Codecs                        0.1.5
 - ColorTypes                    0.2.2
 - Colors                        0.6.4
 - DataArrays                    0.3.0
 - Dates                         0.4.4
 - Docile                        0.5.23
 - FixedPointNumbers             0.1.3
 - GZip                          0.2.18
 - Graphics                      0.1.3
 - NASA                          0.0.0-             master (unregistered)
 - NASA4                         0.0.0-             master (unregistered)
 - Reexport                      0.0.3
 - SHA                           0.1.2
 - SortingAlgorithms             0.0.6
 - StatsFuns                     0.2.1
 - URIParser                     0.1.3
 - tmp                           0.0.0-             non-repo (unregistered)
  2.718086 seconds (2.08 M allocations: 117.536 MB, 4.38% gc time)

julia> @time Pkg.cd(status,STDOUT) # new implementation
...
  0.381964 seconds (125.16 k allocations: 6.329 MB)

[tkelman]

Does that assume the git tags in the repo match the file-based METADATA tags? They don't always match exactly (they should but nothing currently checks or absolutely requires that) and the git tags aren't always present. Or are you reconstructing all of the METADATA content from an in-memory copy?

[wildart]

I do not read tags, it is the same status as in Pkg, instead all info coming from git repo tree and blobs. I just read package info when it required, not in advance. This approach can be implemented in current version of Pkg.status.

One advantage of using repo structure is that METADATA repo can be bare. No more thouthands of files.

[tkelman]

A bare repo is a neat idea for read-only usage, but we'd need some mechanism for being able to modify things. We should restructure METADATA anyway because even bare repos take a while to clone when the structure is the way we have it.

[wildart]

For users, bare METADATA is not a problem. Developers would have to use proper one.

[simonbyrne]

This has changed considerably with the new Pkg. If still an issue, please open at https://github.com/JuliaLang/Pkg.jl