Julia: Precompilation fails with Iterators.jl dependency

[ChrisRackauckas]

Example:

using DifferentialEquations

INFO: Recompiling stale cache file C:\Users\Chris\.julia\lib\v0.6\Combinatorics.ji for module Combinatorics.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module Combinatorics. Got:
WARNING: Module Iterators uuid did not match cache file.

Also posted at https://github.com/JunoLab/Atom.jl/issues/86. For some reason, the REPL works but Juno fails often, and if I hit it with a few usings in the REPL (for some reason, a few), and then close and re-open Juno, it's finally able to find the right precompilation caches. I'm not entirely sure if this is Juno or Base.

[jebej]

FWIW I have seen the same thing.

Not sure if this is related to that, but I have also found that I sometimes get a cache file generated for a module, which does not have any precompile statement. Sometimes the "precompilation" of my module also fails with the error above.

[ChrisRackauckas]

I'm surprised this didn't get a breaking tag? Having to close the text editor almost every time a package is changed is a really tough regression to live with. Partially saying that because I am extremely mad at the situation of having closed the text editor already 50 times this morning, but also I really hope that v0.6 doesn't ship with this prominent of a bug.

[StefanKarpinski]

This is the first we're hearing about this problem and I'm sure it wasn't done on purpose. Ideally we can figure out a way to fix this issue.

[ChrisRackauckas]

A common error message associated with this is something like:

LoadError: Declaring __precompile__(true) is only allowed in module files being imported.
while loading C:\Users\Chris\.julia\v0.6\RecursiveArrayTools\src\RecursiveArrayTools.jl, in expression starting on line 1
__precompile__(::Bool) at loading.jl:335
__precompile__() at loading.jl:331
include_from_node1(::String) at loading.jl:569
eval(::Module, ::Any) at boot.jl:235
_require(::Symbol) at loading.jl:483
require(::Symbol) at loading.jl:398
_include_from_serialized(::String) at loading.jl:157
_require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at loading.jl:200
_require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at loading.jl:236
_require(::Symbol) at loading.jl:434
require(::Symbol) at loading.jl:398
include_string(::String, ::String) at loading.jl:515
eval(::Module, ::Any) at boot.jl:235
(::Atom.##61#64)() at eval.jl:102
withpath(::Atom.##61#64, ::Void) at utils.jl:30
withpath(::Function, ::Void) at eval.jl:38
macro expansion at eval.jl:101 [inlined]
(::Atom.##60#63{Dict{String,Any}})() at task.jl:80

it seems to pick random (precompile) friendly dependents. @MikeInnes says he knew about this already. Iterators and Combinatorics seem to show up a lot though. Other than that, I'm not really sure what I'm looking for in order to debug this.

[timholy]

You surely know this already, but Iterators is both the name of a package and a module in Base. And the Combinatorics package depends on Iterators.

Perhaps try removing your Iterators package? You may have to do surgery on your REQUIRE files and/or METADATA to allow this.

[ChrisRackauckas]

I tried to get rid of every v0.6 Iterators usage by doing an if statement on the versions to choose between Iterators and Base.Iterators, but even that doesn't work. It "fixed" RecursiveArrayTools, but then Roots.jl ended up with a faulty precompile cache on Iterators, and it stayed that way even after doing this change to all of its dependencies which use Iterators.

But at least that's something. Doing a fresh install of the rc2 generic binaries on 3 separate Windows computers and then using it through Juno, it very reliably will have a precompilation problem on some (random) package which depends on iterators, either directly or indirectly. On the one hand it's great that it's very easy to repeat, but on the otherhand it's not so good that it's so easy to run into.

Given this, I tried to find out if there is anything weird that Iterators does. I did find something, if I directly do using Iterators in Juno, I get

WARNING: imported binding for Iterators overwritten in module Main

If I do it in the REPL, everything is fine. Since this precompilation problem seems to only be happening in Juno, maybe that's a clue that it's a problem with Juno using Base.Iterators for something?

[timholy]

Hmm, what happens if you add the following as the first line of your Iterators package:

error("don't use this!")

[tkelman]

Related to #21578? Would #21580 fix it? That just missed rc2 but should be in rc3.

[ChrisRackauckas]

Hmm, what happens if you add the following as the first line of your Iterators package:

Juno starts up just fine even with that.

[timholy]

Bizarre. What if you rm ~/.julia/lib/v0.6/*.ji and try again? (Nukes all precompiled packages, forcing everything to rebuild.)

[aviks]

I've seen similar bizarre precompilation issues (on 0.5) when there are multiple Julia processes started simultaneously, and they all try to load a package, and thus try to precompile a package simultaneously. Could something like that be happening here? Precompilation is far from being multi-anything safe.

[ChrisRackauckas]

I've seen similar bizarre precompilation issues (on 0.5) when there are multiple Julia processes started simultaneously, and they all try to load a package, and thus try to precompile a package simultaneously. Could something like that be happening here? Precompilation is far from being multi-anything safe.

Juno does by default start up a few processes using its "Cycler". I changed that over to "Basic" and had the same issue. Maybe this should be checked more thoroughly though. @MikeInnes @pfitzseb what do you think of that? Is there a better test of this?

[ChrisRackauckas]

Actually, deleting all of the precompilation files and then switching over to the "Basic" mode seems to be working now.

[pfitzseb]

While Juno in cycler mode certainly starts up a couple of Julia processes at the same time, only the currently used process should actually do something if you evaluate using DifferentialEquations -- unless of course that's in your .juliarc.jl or .junorc.jl, @ChrisRackauckas?

[ChrisRackauckas]

My .juliarc.jl and .junorc.jl is unchanged. As noted earlier, this is a a clean re-install of the generic binaries. You can safely assume everything is unchanged except some Pkg.adds, and then some changes to Iterators as noted above.

[MikeInnes]

FWIW, I've seen this plenty with Basic boot mode, which doesn't interleave processes at all, and at any rate I didn't see problems on 0.5.

[ChrisRackauckas]

and at any rate I didn't see problems on 0.5.

True. Me neither.

[jebej]

Note that I see the problem at the REPL, if an attempt at compilation was made in Juno previously. This also exhibits the problem where my module, which doesn't have any precompilation statements, gets precompiled (and I do end up with a ji file):

julia> using Schrodinger, BenchmarkTools
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\DiffEqBase.ji for module DiffEqBase.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module DiffEqBase. Got:

WARNING: Module Iterators uuid did not match cache file.
INFO: Precompiling module Schrodinger.
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\Combinatorics.ji for module Combinatorics.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module Combinatorics. G
ot:
WARNING: Module Iterators uuid did not match cache file.
ERROR: LoadError: Declaring __precompile__(true) is only allowed in module files being imported.

[dpsanders]

I am seeing this problem from the REPL with RC2 after deleting lib/v0.6 and doing using DifferentialEquations, with no Juno involved.

[mbauman]

Another example: https://github.com/JuliaImages/ImageAxes.jl/issues/18. We worked around it by removing the Iterators.jl dependency. That seems to be at the root of all these reports. Title changed to reflect that.

[tkelman]

then it should be fixed by backporting https://github.com/JuliaLang/julia/pull/21580. does it happen on master?

[ChrisRackauckas]

I don't seem to be having problems with this using the v0.7 nightly.

[jebej]

This is still an issue with rc3 unfortunately:

julia> using Schrodinger
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\Roots.ji for module Roots.
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\PolynomialFactors.ji for module PolynomialF
actors.
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\Combinatorics.ji for module Combinatorics.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module Combinatorics. G
ot:
WARNING: Module Iterators uuid did not match cache file.
INFO: Precompiling module Schrodinger.
INFO: Recompiling stale cache file C:\Users\Jeremy\.julia\lib\v0.6\Combinatorics.ji for module Combinatorics.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module Combinatorics. G
ot:
WARNING: Module Iterators uuid did not match cache file.
ERROR: LoadError: Declaring __precompile__(true) is only allowed in module files being imported.

[dpsanders]

Does it help to do using Schrodinger again?

[jebej]

If I close and restart julia, it works, after which there seem to be some
delay before it will give the error again. FWIT I think using Combinatorics package reproduces the error on its own.

On Wed, Jun 7, 2017 at 2:03 PM David P. Sanders [e-mail]
wrote:

Does it help to do using Schrodinger again?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JuliaLang/julia/issues/21969#issuecomment-306876874,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKoBtpk2IeLKN0oaZrOKYYsB2jA3_NOaks5sBuYGgaJpZM4Ng1Lx
.

[ChrisRackauckas]

I've found I can "recover" from this by doing:

using Iterators
using Combinatorics
using WhateverOtherPackage

after seeing the error.

[jebej]

I ended up adding import Iterators, Combinatorics to my .juliarc.jl file...

[ChrisRackauckas]

I'm getting this on rc3 as well. Maybe the easiest way is to get Iterators out of Combinatorics.

[ChrisRackauckas]

I took a package approach to "fixing this", and put in PRs to remove Iterators.jl from every dependency that was giving me problems:

https://github.com/JuliaMath/Combinatorics.jl/pull/49
https://github.com/jverzani/PolynomialFactors.jl/pull/5

If those merge and and are released, v0.6 is safe for me. Other packages that I don't use might still have issues though. But if you work in a similar package sphere to me, you can checkout those 2 PRs and you won't have the issue anymore.

[musm]

would be nice to have this fixed for 0.6

[StefanKarpinski]

@JeffBezanson did some investigation of this earlier; best if he reports what he found here.

[ChrisRackauckas]

I saw v0.6 got tagged: was there a fix for this included?

[tkelman]

Not in 0.6.0, no. @JeffBezanson can hopefully share what he found from looking into this a bit.

[prairie-guy]

I believe this has been well reported, but I add my experience using julia version 0.6.0 (OSx, downloaded as precompiled julia.app)

$ rm -fr ~/.julia
julia> Pkg.add("Combinatorics")
julia> using Combinatorics

INFO: Recompiling stale cache file /Users/cdaniels/.julia/lib/v0.6/Combinatorics.ji for module Combinatorics.
WARNING: The call to compilecache failed to create a usable precompiled cache file for module Combinatorics. Got:
WARNING: Module Iterators uuid did not match cache file.
ERROR: LoadError: Declaring __precompile__(true) is only allowed in module files being imported.
Stacktrace:
[1] __precompile__(::Bool) at ./loading.jl:335
[2] include_from_node1(::String) at ./loading.jl:569
[3] eval(::Module, ::Any) at ./boot.jl:235
[4] _require(::Symbol) at ./loading.jl:483
[5] require(::Symbol) at ./loading.jl:398
while loading /Users/cdaniels/.julia/v0.6/Combinatorics/src/Combinatorics.jl, in expression starting on line 1

I then quit the repl, restart julia, 'use Combinatorics' and it typically works.
Thereafter, I will randomly receive an error on using Combinatorics on future occasions. Though I can't always replicate the error, it does appear to be related to Pkg.update() or Pkg.add())

[ssfrr]

Not sure if this is news - but I've seen this also without having Iterators.jl installed, so it doesn't seem like it's specific to that package. I think I've seen it mostly in the context of DSP.jl not precompiling because FFTW.jl's UUID doesn't match. Probably has to do with switching branches of DSP.jl, but I don't have a consistent repro.

[tkelman]

FFTW is also having a name collision with the (deprecated) Base module. We need a real fix for this, or packages with the same name as a base module will never work reliably.

[kkmann]

+1 having the same issue on Win 7 with Iterators dependency, switching back to 0.5.2

[huckl3b3rry87]

ugg, after spending way to long getting Juno to work with issues from julia-client and ultimately HttpParser I ran into this...

[ChrisRackauckas]

Update for those who haven't followed: Iterators.jl has been deprecated in favor of IterTools.jl. That gets rid of this issue for packages that used to use Iterators.jl. Please use Pkg.update() to fix this. As @ssfrr points out, packages which use FFTW.jl will still be affected.

[huckl3b3rry87]

@ChrisRackauckas thank you, I did do a 'Pkg.update() but then I had to open and close atom a couple of times for it to finally work ok. Also, I thought this was still an issue since it is open

[ChrisRackauckas]

Well Iterators and FFTW still won't work because of the naming issue. I'll let someone else determine whether it can be considered fixed just by renaming the packages. It seems like @tkelman is looking for a real fix.

[tkelman]

I'd like for there to be one, but I don't know how to do it myself. Jeff looked into this before JuliaCon but hasn't written down what he found.

[JeffBezanson]

I think this will ultimately be solved by not using Main as the root namespace for loaded packages. IIUC, what @StefanKarpinski (and I) would like to do in Pkg3 is have using Iterators resolve the name Iterators to a UUID, and track loaded packages by UUID instead of as global names in a certain module.

The only other thing we can do is remove the Base export for Iterators. That would be a partial fix, since it only makes it less likely you will end up with another module called Iterators in Main. Then you will need an explicit using Base.Iterators before you can write Iterators.foo. I could go either way on that.

[tkelman]

Ah yes, #17997 combined with some identity for modules that's independent of the name would be great. Is there any way we can start on that change before Pkg3 is entirely ready?

[StefanKarpinski]

The change is largely independent of Pkg3. In particular, it actually can't really use UUIDs because it's possible to load code from things that aren't packages. Unless we want to eliminate the ability to load non-local code that's not a full-blown package, we need to use a path or something like that. So the general scheme would be something like using Foo means to look up where to load Foo from in a way that depends on the context of the current code, e.g. it's Pkg3 environment, instead of being uniformly and globally determined by just the name Foo. Then there would be a caching layer that hands back the pre-loaded module if that path has already been loaded. So using the entry-point path may be a good caching key. If using Foo resolves to loading different paths in different files, then it may refer to different things. We may want the same module to be loadable by different names without triggering a reload of the code; if a package changes names but continues to support being used by an older name, that could be useful. It would also be useful if we want to allow multiple entry-points to the same package.

[crstnbr]

+1 same issue here when trying to using Distributions which depends on FFTW. Have to switch back to 0.5 until this is fixed.

[ChrisRackauckas]

Well you can using FFTW; using Distributions and it'll work. But I agree that Distributions.jl should really just avoid the issue for now: it doesn't need FFTW for v0.6 to work and putting it in a v0.6 release just causes unnecessary issues except for the few people working on the nightly builds.