Julia: allow `using` when .ji file exists without corresponding .jl file

Created on 12 May 2016 · 16Comments · Source: JuliaLang/julia

In discussions with Jeff on Tuesday, he said he thought this was a reasonable change.
Use case is to ship a product with precompiled package without revealing package source.

Current behavior:

julia> using Compat
ERROR: module Compat not found in current path; you should rm("/home/sachs/.julia/lib/v0.4/Compat.ji") to remove the orphaned cache file
 in error at ./error.jl:21
 in recompile_stale at loading.jl:472
 in _require_from_serialized at loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:235

precompile

Source

josefsachsconning

👍4 👎1

Most helpful comment

It's a somewhat tricky process.

1) I've modified base/loading.jl to skip checking for the .jl files. I changed find_in_load_path to conditionally return an empty string when a task variable is set, and _require_search_from_serialized to skip the call to stale_cachefile when sourcepath is that empty string. This will require the base system image to be rebuilt.
2) With the new base system image, I prebuild the .ji files with using. Note that all packages must support precompiling.
3) After deleting the .jl files, I set the aforementioned task variable before calling using. The .ji files are then loaded without looking for the .jl's.

This is obviously not a sustainable mechanism. It broke from 0.4 to 0.5, and is obviously subject to breaking at any subsequent release. I'm hoping the Julia developers enable support (perhaps via command line option) to make this always work.

s2maki on 20 Dec 2016

👍2

All 16 comments

Maybe a specific syntax for loading from a .ji file? The default behavior of using does a lot of staleness checks that aren't well-defined without the .jl files present, so perhaps an alternate syntax could be used for force-loading a package from just the .ji file?

tkelman on 12 May 2016

Ideally, there would be a mechanism whereby julia would attempt a normal using, and fall back to loading from the .ji file if the .jl file is not present (for the convenience of developers). Could you lay out what that would look like in my julia script?

josefsachsconning on 12 May 2016

In the current mode of operation where using defaults to the automatic staleness check and recompilation, a .ji file being present without corresponding source is a sign that something went wrong, the .ji file is probably stale and shouldn't be used - the package got deleted without removing the .ji, or something like that. So it's probably safer to use a separate syntax that specifically indicates you know you want to load from a .ji file via something like usecompiled("/path/to/Compat.ji") (or usecompiled("Compat") to look in default search paths).

There isn't much room in the current syntax of using to indicate intent of whether a ji file without corresponding source is desired or a problem. Adding a usebinary Compat that works just like using Compat but allows the fallback no-source condition to succeed could also work.

tkelman on 12 May 2016

Right, I understand that, but to get the behavior that I described above, would I do

try:
    eval(:(using MyPackage))
catch:
    usecompiled("/path/to/MyPackage.ji")
end

That's not really ideal, because I'd like to get the error messages if the .jl is present but produces errors.

Above was written before your edit. usebinary sounds fine.

josefsachsconning on 12 May 2016

I don't think an alternate invocation of using would work well. That would mean that all dependent modules would also have to call usebinary. To me, it makes more sense to have a "mode" of sorts (perhaps a global variable, function call, or command line argument) that puts using into a mode of skipping stale checks.

s2maki on 12 May 2016

a .ji file being present without corresponding source is a sign that something went wrong

I think the issue is more to have a .ji (or similar) format that is defined to be loadable without the source file? I don't think it's necessary to add different syntax for binary import.

yuyichao on 13 May 2016

Perhaps we save the .ji file to a different extension that find_in_path is allowed to successfully find without having source present?

tkelman on 13 May 2016

Something like that, or a different location, or a different tag (metadata) in the file.

yuyichao on 13 May 2016

Any new related to this topic? It seems very interesting

joaquimg on 21 Sep 2016

❤1

From the above discussion it is not fully clear to me if the current .ji files are self contained or if the source files are still needed. So is this feature request a minor thing that would happen at the surface or are more complicated changed necessary to implement this.

I have a real use case where this would be really beneficial.

tknopp on 18 Dec 2016

In our experience, the .ji files are self contained. Source .jl files are not needed. However binary dependencies are still obviously required, so they sometimes have to be handled specially. For example NLopts needs a shared library. If you want to delete the .julia/v0.5 folder after generating the .ji files, the library file has to be moved and deps.jl pointed to it before precompiling the .ji.

s2maki on 18 Dec 2016

In our experience, the .ji files are self contained. Source .jl files are not needed.

Could you please clarify what this means? How do you remove the .jl files? If I remove the folder, nothing is working anymore.

tknopp on 18 Dec 2016

It's a somewhat tricky process.

s2maki on 20 Dec 2016

👍2

Thanks, yes it would be great if this could be part of Julia master itself. Actually this would also be something that might be considered for the Pkg3 revamp.

tknopp on 20 Dec 2016

Are there any news on this issue? This feature would be most useful to avoid file system congestion when running a julia application at large scale (on thousands of nodes) - a Julia discourse topic.