Plots.jl: Cannot "using Plots" from within another module

[ma-laforge]

1. Julia v1.0; Plot v0.20.2; Linux
2. Must have and actual module that you can call "using" on. Here are the steps I use to create such a module:
   a. ] generate MyPlotsMod
   b. Edit MyPlotsMod/src/MyPlotsMod.jl to include using Plots:

module MyPlotsMod
using Plots #!!!!!!!!!!!!!!!!!!You must add this line
greet() = print("Hello World!")
end # module

... And you will get a bunch of warnings on incremental compilation. One of the more readable ones is the following:

WARNING: eval from module Plots to MyPlotsMod:
Expr(:call, Expr(:., :Sys, :(:islinux)))
  ** incremental compilation may be broken for this module **

[ma-laforge]

Remark

The warnings go away if I add the following line to the very beginning of my own "dev" version of Plots.jl/src/Plots.jl:

__precompile__(false)

My guess is that the method Plots.jl uses to conditionnaly include backends causes precompile issues when including Plots.jl in another module.

[mkborregaard]

I think so too. You're not actually supposed to depend on Plots though.

[dalum]

I am running into the same issue. I would like to define a recipe for visualizing my own data using RecipesBase. However, I depend on being able to do using Plots: RGB, Segments to generate the right visualization (at least I don't know of an alternative way to do it, and this is how it is done in the PlotsRecipes package). It used to work fine when I defined the recipe in a module in 0.6, but now it won't compile.

[mkborregaard]

Yes, we're actually not sure what causes this, but believe it's an interaction between Requires and eval. It should be revisited for sure.
For now I avoid using Segments - all it does is insert an NaN into the Vector at the segment break, and you can do that easily manually for now. RGB you can/should get from Colors.

[ma-laforge]

A starting point

1. I believe that Dicts with DataType or Function cause issues with precompile.

For example: _backendSymbol.

For more information on issues with precompile, please see:

https://docs.julialang.org/en/stable/manual/modules/#Module-initialization-and-precompilation-1

1. I personally also have issues (related to precompile?) when some of my modules attempt to "conditionally" execute eval(:(import SomeOtherModule)).

I think it might be because Julia does not like import statements being executed in a module after the pre-compile process has executed.

...If this is true, it would be best to push the import command onto the user's own module by using macros. For example:

macro initialize(bk::Symbol)
    #Do some initialization here

    return :(import $bk) #Cause the "import" stmt to execute in user's module
end

If this could be achieved somehow for Plots.jl, then the @initialize macro would cause the import statement to be executed from the user's own module (assuming it is called from that module):

@initialize(PyPlot)

...Sadly I cannot verify if this is really a potential solution. I cannot seem to generate a repeatable case yet. I also don't really understand how conditional inclusion of code (backends/*.jl files) affects the precompile system either - but I am pretty certain I have been warned this was bad.

Comments

Sorry: I wish I knew where the problem actually was. The comments above are simply starting points. I do not yet fully understand the issues with precompile very well - or how we can avoid them.

[floswald]

yes same issue here. my case is exactly as in the initial example.

[piever]

This is also an issue in StatPlots where we had to manually turn off precompilation to silence the warnings: definitely something to be fixed. I also suspect the combination of @eval import BackendModule, Requires and precompilation is causing this. I think one way of making the backend mechanism healthier (not sure if it fixes the issue) would be to ask the user to explicitly import it. So for example one would do:

using Plots
import PlotlyJS
plotlyjs()

to select the PlotlyJS backend. It's a bit more verbose but definitely cleaner than evalling import PlotlyJS into Main from within Plots (and if verbosity is an issue, maybe a macro could solve it, say @backend! PlotlyJS).

[floswald]

I'm in favour of this. but i can hear the objection that one cannot just do

using Plots
plot(rand(10))

i.e. load one thing only and see a plot. I completely agree with you that given the whole Plots.jl story is about different backends, choosing one in the very beginning only seems consistent.

[piever]

GR is installed by default and would be imported by default, so using Plots would still work and plot with the GR backend.

[floswald]

ah well then this seems a great solution!

[mkborregaard]

@piever that's worth considering. It's a radical departure from the existing framework, but on the other hand it might alleviate some problems

[daschw]

I also like it a lot. @piever always has awesome ideas! I have a local branch, where I replaced gr(), pyplot() etc. with @gr, @pyplot, ... macros. It seems to work quite nicely. The WARNING: eval from module Plots to ... warnings are gone. The downside is, that I could not find a way yet to support ENV[""PLOTS_DEFAULT_BACKEND"]. So you need to do @backend to use something else than GR. I'm not even sure if ENV[""PLOTS_DEFAULT_BACKEND"] currently works, though.

I will have to do some more testing locally and update the documentation. Then I plan to push to #1730 for your reviews.

[daschw]

Oh, there's another downside of my macro implementation. Macros do not support keyword arguments, so something like gr(size = (400, 400)) will not work with @gr, but users will have to call default(size = (400, 400)) additionally.

[mkborregaard]

I think both are acceptable. But, I think most users might prefer the explicit import to the macros. Julia's userbase seems to dislike magic.

[daschw]

That's also possible:

import PyPlot, PyCall, LaTeXStrings
backend(:pyplot)

vs

@pyplot

in the current state of the branch.

[mkborregaard]

oh yeah it gets hairy for PyPlot

[piever]

Not the most elegant, but does an explicit top-level if-else do the trick? Say:

if ENV["PLOTS_DEFAULT_BACKEND"] == "pyplot"
  @pyplot
elseif ...
...

One can potentially add "keywords" to macros (by parsing @gr size = (400, 400) and returning the expression that does the right thing), but that part always struck me as a bit unnecessary, when we have default(...) which is way more intuitive.

If it's not just the backend package that needs importing, we definitely need some syntactic sugar. I'm a bit undecided between:

@pyplot

and

@backend! PyPlot

[daschw]

import GLVisualize, GeometryTypes, Reactive, GLAbstraction, GLWindow, Contour
import GeometryTypes: Point2f0, Point3f0, Vec2f0, Vec3f0, GLNormalMesh, SimpleRectangle, Point, Vec
import FileIO, Images
import Reactive: Signal
import GLAbstraction: Style
import GLVisualize: visualize
import Plots.GL
import UnicodeFun
backend(:glvisualize)

for GLVisualize :stuck_out_tongue: But I think the GLVisualize backend is not even working at the moment and it's time to deprecate it.

[daschw]

@piever What's the advantage of @backend! in your opinion?

[piever]

Normally given that macros can't be combined with multiple dispatch, I instinctively thought it was best to not "crowd macro space" by exporting too many of them. On the other hands the names are so specific (like @pyplot) that this doesn't seem to be an issue. You're probably right that we should stick to the "minimum change" solution (so going from pyplot() to @pyplot).

[daschw]

OK, so I was wrong, my macro implementation did not fix anything with respect to incremental compilation. After trying everything to rewrite the backend() code, first with macros, then getting rid of the DataType Dicts I finally found the source of the problem: We should not eval stuff in __init__(). So in the end it was a really easy fix (#1772) ... kind of annoying after all the time I spent on this.
Let's hope that the tests pass ...