Julia: error loading just-precompiled module if an imported module was loaded before precompiling

Created on 7 Aug 2015 · 22Comments · Source: JuliaLang/julia

Getting the following error messages:

WARNING: could not import Base.help into PyCall
INFO: Precompiling module LaTeXStrings...
INFO: Recompiling stale cache file xxxxx/.julia/lib/v0.4/LaTeXStrings.ji for module LaTeXStrings.
WARNING: Module Compat uuid did not match cache file

LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
while loading xxxxxx/.julia/v0.4/PyPlot/src/PyPlot.jl, in expression starting on line 555
while loading In[2], in expression starting on line 1

 in require at ./loading.jl:211
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:263
 in require at ./loading.jl:202

Julia Version 0.4.0-dev+6526
Commit dcc448b* (2015-08-07 14:14 UTC)
Platform Info:
System: Linux (x86_64-redhat-linux)
CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3
Package Directory: xxxxxxxxxxxxxx/.julia/v0.4

WARNING: unknown Lazy commit 8709dc31, metadata may be ahead of package cache

15 required packages:

ArgParse 0.2.11
ArrayViews 0.6.2
BayesNets 0.1.0
CPUTime 0.0.4
Cairo 0.2.28
DataStructures 0.3.11
Distributions 0.8.3
Escher 0.1.0
FactCheck 0.3.1
GZip 0.2.17
Gadfly 0.3.13
IJulia 0.2.5
Jewel 1.0.5
PyPlot 1.5.3
QuantEcon 0.2.0
71 additional packages:
Benchmark 0.1.0
BinDeps 0.3.14
Blosc 0.1.3
Calculus 0.1.10
Codecs 0.1.4
Color 0.4.7
Compat 0.4.12
Compose 0.3.13
Contour 0.0.7
DSP 0.0.9
DataArrays 0.2.17
DataFrames 0.6.8
Dates 0.4.4
Distances 0.2.0
Docile 0.5.14
DualNumbers 0.1.3
FixedPointNumbers 0.0.9
Formatting 0.1.3
FunctionalCollections 0.1.2
GnuTLS 0.0.4
Graphics 0.1.1
Graphs 0.5.5
Grid 0.3.10
HDF5 0.5.2
Hexagons 0.0.3
Hiccup 0.0.0
HttpCommon 0.0.12
HttpParser 0.0.11
HttpServer 0.0.12
ImmutableArrays 0.0.9
IterationManagers 0.0.1
Iterators 0.1.8
JLD 0.5.1
JSON 0.4.4
JuliaParser 0.6.2
KernelDensity 0.1.1
LNR 0.0.1
LaTeXStrings 0.1.4
Lazy 0.9.0+ master
LightGraphs 0.2.2
LightXML 0.1.13
Loess 0.0.3
MAT 0.2.12
MacroTools 0.1.0
Markdown 0.3.0
Mux 0.0.0
NaNMath 0.0.2
Nettle 0.1.10
Optim 0.4.2
Options 0.2.4
PDMats 0.3.5
Patchwork 0.1.6
Polynomials 0.0.3
PyCall 0.8.2
REPLCompletions 0.0.3
Reactive 0.2.2
Reexport 0.0.2
Requires 0.1.3
SHA 0.0.4
Showoff 0.0.4
SortingAlgorithms 0.0.5
StatsBase 0.7.0
StatsFuns 0.1.1
TextWrap 0.1.3
TikzGraphs 0.0.1
TikzPictures 0.2.2
URIParser 0.0.5
WebSockets 0.0.6
WoodburyMatrices 0.1.1
ZMQ 0.1.20
Zlib 0.1.8

precompile

Source

dbrowne

Most helpful comment

18150 should make this work much better

vtjnash on 25 Aug 2016

👍2

All 22 comments

cc @stevengj (move to PyPlot/PyCall if this is package-specific and not a general problem)

pao on 7 Aug 2015

@dbrowne, had you just done Pkg.update()? What happens if you quit Julia and try again? Or try using LaTeXStrings?

stevengj on 7 Aug 2015

👍2

I think this might happen if Compat was already loaded (from using PyPlot, which calls using Compat), but not yet precompiled, before you first tried to precompile LaTeXStrings (which happens from the using LaTeXStrings in PyPlot).

When you compile LaTeXStrings, it also precompiles Compat.ji (using a separate Julia process that doesn't know that Compat was already loaded). Then it tries to load the LaTeXStrings module from the .ji file, but fails because the UUID of the Compat that was already loaded in memory doesn't match the UUID it expects (the one from Compat.ji on disk, which was not loaded because Compat was already imported).

stevengj on 7 Aug 2015

❤1

As a workaround, for now, using PyPlot should work if you restart your Julia session (since the second time around it will load the Compat.ji file from disk.

In the short term, we should probably patch Compat to include __precompile__(true) so that it is compiled the first time it is loaded, since it is safe to precompile anyway.

In the long term, two options are:

Disallow auto-precompilation of a module unless all of its imported modules also have __precompile__(true).
Precompiling a module Foo that imports Bar does not automatically precompile Bar unless Bar has __precompile__(true) also.

The second one seems more elegant to me, but is also more technically challenging since the precompile system now seems to assume that once you precompile you precompile all the way down. @vtjnash?

stevengj on 7 Aug 2015

It's mostly a question of when you want your segfaults and incompatibilities. If you don't have tight control all of the way down, then you don't really know if some file was slightly different when it was originally loaded and might now be missing a definition which will cause the deserializer to segfault (you can try this at home by eval'ing a type or function into another module and then using that value locally -- see https://github.com/JuliaLang/julia/issues/12352

the first option is slightly annoying right now, but with precompile now being enforced by __precompile__, it's perhaps actually providing pretty good behavior.

vtjnash on 8 Aug 2015

If I quit Julia and try again I get the following:
WARNING: could not import Base.help into PyCall
WARNING: error initializing module PyPlot:
PyCall.PyError(msg=":PyImport_ImportModule", T=PyCall.PyObject(o=0x00007f8aa1ee4e60), val=PyCall.PyObject(o=0x0000000004213ea8), traceback=PyCall.PyObject(o=0x0000000000000000))
Attempting to plot returns this:
ERROR: UndefVarError: pltm not defined
in plot at /apps/infrafs1/dbrowne/.julia/v0.4/PyPlot/src/PyPlot.jl:368

dbrowne on 10 Aug 2015

@dbrowne, the "error initializing module PyPlot" sounds like an unrelated problem—the PyImport_ImportModule exception means that libpython is failing to import matplotlib or some other necessary module. Please file a followup issue at PyPlot.

stevengj on 10 Aug 2015

I raise the issue here because I'm building a new version of Julia and not PyPlot. If PyPlot were the moving target I would raise the issue there. Julia builds for the last few weeks worked without any issues.

dbrowne on 10 Aug 2015

@dbrowne, the error message is different, you are seeing a different problem. Please use separate issues to discuss distinct problems.

stevengj on 10 Aug 2015

So what's the current recommended course of action? I'm trying to get tests working for a package that depends on DataFrames (which has a bunch of dependencies, not all of which have precompilation enabled).

simonbyrne on 19 Aug 2015

@simonbyrne, just import DataFrames first, and it will precompile all its dependencies.

The only problem arises if you manually import a dependency Foo that _doesn't_ use __precompile__ first, and _then_ import a package Bar that _does_ use __precompile__(true) and which imports Foo. Then it runs into a problem because it wants the precompiled Foo but the non-precompiled Foo is already imported.

In the longer run, precompilable packages should push __precompile__() PRs to their dependencies. (Arguably, the best practice will be to push to your dependencies first, before adding __precompile__() to your own module.)

stevengj on 19 Aug 2015

Great, that fixed it, thanks!

simonbyrne on 19 Aug 2015

A related problem appears when starting several separate julia processes in parallel. When they all hit the cache at once I get similar errors to the above.

(Background: the use case is an AWS cluster where instances are commonly started from scratch. When a node is first started the node downloads and installs a zip file containing a module with all the required julia dependencies. As soon as that's complete, many jobs jump onto it all at once and start hitting the compile cache, which basically guarantees a few failures as reproduced below.)

Simple test case involving DataFrames (run from a temporary directory):

$ cat init.sh

#!/bin/bash

pkgdir=$PWD/pkg
mkdir -p $pkgdir/v0.4

export JULIA_PKGDIR=$pkgdir
julia -e "Pkg.init()"
echo DataFrames > $pkgdir/v0.4/REQUIRE
julia -e "Pkg.resolve()"

$ cat test.sh

#!/bin/bash

pkgdir=$PWD/pkg
export JULIA_PKGDIR=$pkgdir
julia -e "using DataFrames"

Usage:

$ ./init.sh

# Now run two in parallel.
$ ./test.sh & ./test.sh 
[1] 13242
INFO: Recompiling stale cache file /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataArrays.ji for module DataArrays.
WARNING: Module Reexport uuid did not match cache file
WARNING: require failed to create a precompiled cache file
WARNING: requiring "DataArrays" did not define a corresponding module.
ERROR: LoadError: UndefVarError: DataArrays not defined
 in include at ./boot.jl:260
 in include_from_node1 at ./loading.jl:271
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:284
 in _start at ./client.jl:411
while loading /home/chris/tmp/jlpkg/pkg/v0.4/DataFrames/src/DataFrames.jl, in expression starting on line 14
ERROR: Failed to precompile DataFrames to /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataFrames.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:350
 in require at ./loading.jl:217
 in process_options at ./client.jl:284
 in _start at ./client.jl:411
INFO: Recompiling stale cache file /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataFrames.ji for module DataFrames.

(julia version 0.4.0-pre+7400 commit bffe239)

c42f on 9 Sep 2015

On a clean JuliaBox 0.4 notebook (no packages installed), I just ran Pkg.add("Cbc") and got the following:

INFO: Updating cache of BinDeps...
INFO: Cloning cache of Cbc from git://github.com/JuliaOpt/Cbc.jl.git
INFO: Updating cache of Compat...
INFO: Updating cache of MathProgBase...
INFO: Updating cache of SHA...
INFO: Updating cache of URIParser...
INFO: Installing BinDeps v0.3.18
INFO: Installing Cbc v0.1.8
INFO: Installing Compat v0.7.6
INFO: Installing MathProgBase v0.3.18
INFO: Installing SHA v0.1.2
INFO: Installing URIParser v0.1.1
INFO: Building Cbc
INFO: Precompiling module URIParser...
INFO: Precompiling module SHA...
INFO: Recompiling stale cache file /home/juser/.julia/lib/v0.4/Compat.ji for module Compat.
INFO: Recompiling stale cache file /home/juser/.julia/lib/v0.4/SHA.ji for module SHA.
WARNING: Module Compat uuid did not match cache file
=================================[ ERROR: Cbc ]=================================

LoadError: LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
while loading /home/juser/.julia/v0.4/BinDeps/src/dependencies.jl, in expression starting on line 969
while loading /home/juser/.julia/v0.4/BinDeps/src/BinDeps.jl, in expression starting on line 507
while loading /home/juser/.julia/v0.4/Cbc/deps/build.jl, in expression starting on line 1

================================================================================

================================[ BUILD ERRORS ]================================

Both Compat and SHA have precompile enabled. What's the issue?

mlubin on 20 Oct 2015

@mlubin, in a notebook, the IJulia package is loaded and hence Compat has already been loaded. Hence updating it runs into the same issue above.

stevengj on 20 Oct 2015

Note that #13506 cannot be backported until there is a followup that resolves the issues with Homebrew.jl ~~and WinRPM.jl, both of~~ which ~~are~~ is broken on 0.5. (winrpm may be working now - needs more investigation)

tkelman on 20 Oct 2015

Just to report that I encountered recently similar problem with BinDeps, while building "IJulia" by Pkg.update(). I restarted Julia and then IJulia can be successfully built.

Sisyphuss on 4 Nov 2015

@tkelman, did this ever get backported? I've been seeing reports of similar problems popping up recently, e.g. JuliaLang/Roots.jl#39

stevengj on 25 Apr 2016

was what backported? this issue was never closed.

tkelman on 25 Apr 2016

@tkelman, I was referring to ~~#13458~~ #13506, which seems like it should have fixed most problems of this sort encountered in practice, e.g. the one in Roots.jl above.

stevengj on 25 Apr 2016

Oh, never mind, that was for Pkg.build. The precompilation is already done in separate processes. The original issue here still applies. Carry on then...

stevengj on 25 Apr 2016