Plots.jl: New package: PlotRecipesBase.jl

Created on 5 May 2016 · 46Comments · Source: JuliaPlots/Plots.jl

Moving this discussion from #222 so we don't get distracted over there...

The concept is that we'd have a tiny, minimal package for developers to be able to define custom recipes both for their own data types (such as those in MLPlots), and also to define new series types (in a manner to how I built the violin or boxplot types). This should be extremely minimal... only a few functions, and Plots will make this package a core dependency and use it when creating its own recipes (such as boxplots).

I'll use this issue to flesh the idea out a little more and come up with requirements/design.

cc @Evizero @diegozea

discussion priority

Source

tbreloff

Most helpful comment

It works! quiet and force flags are allowed. The following examples work as expected:

using Plots
gr(size=(400,300))

type T end
@recipe function plot(t::T, n::Int = 1)
    :markershape --> :auto
    :markercolor --> :red, :force
    :xrotation --> 5
    :zrotation --> 6, :quiet
    rand(10,n)
end

# plots 5 series of 10 random numbers
# markers are varied, but forced to red
# only one warning (for xrotation)
plot(T(), 5, mc = :blue)

# plots 1 series of 10 random numbers
# markers are octogons, colored red
# only one warning (for xrotation)
plot(T(), shape=:o)

tbreloff on 5 May 2016

🎉3 😄1

All 46 comments

Also... this is something that we could pitch to be included in Base Julia, as it would likely not require too much maintainence, and would make use/adoption a ton easier.

tbreloff on 5 May 2016

The name is a bit awkward and doesn't sound very professional. How about

CustomPlots
PlotsBase

Evizero on 5 May 2016

this is something that we could pitch to be included in Base Julia, as it would likely not require too much maintainence, and would make use/adoption a ton easier.

I think you should definitely have everything done first and also a reference inclusion in some package to show for. That would make a much easier sell than proposing the raw idea

Evizero on 5 May 2016

Yes certainly. I wasn't suggesting _starting_ in Base.

tbreloff on 5 May 2016

~~The case would even be stronger if this base package was so general that these user-defined receipies could even be used by plotting packages other than Plots if they choose to do so.~~

Edit: well I guess it would be by definition. I don't think I thought this through

Evizero on 5 May 2016

if this base package was so general that these user-defined receipies could even be used by plotting packages other than Plots

Yes I think this can be done without a circular dependency on Plots... meaning other packages could easily convert to their own attributes.

At its core, all the recipes do is convert the input data into a type that is recognized by Base Julia, and add attributes to describe the display. Taking an example for LearnBase:

function Plots._apply_recipe(d::KW, loss::ModelLoss, args...; kw...)
    get!(d, :xlabel, _loss_xlabel(loss))
    get!(d, :ylabel, "L(y, h(x))")
    get!(d, :label, string(loss))
    value_fun(loss), args...
end

This recipe doesn't need to know anything about anything... it only needs access to an _apply_recipe method to add it's definition to. And actually this could now be written as:

@recipe loss::ModelLoss args... begin
    :xlab => _loss_xlabel(loss))
    :ylab => "L(y, h(x))"
    :lab => string(loss)
    value_fun(loss), args...
end

tbreloff on 5 May 2016

One question, I suppose, is whether any arg/alias lists should leak into this package. Do we leave it up to the individual package maintainers to keep to conventions of attribute names?

tbreloff on 5 May 2016

personal preference: one concrete symbol per argument. i.e. always :xlabel instead :xlab. That seems much easier to maintain and keep up with

Evizero on 5 May 2016

👍2

I also prefer PlotsBase rather than PlotRecipesBase.
It's incredible that this only needs one Plots function (and only uses Julia types).
I don't like aliases, long names are better for maintaining/developing code. But I found aliases useful sometimes when I don't remember the full name.

diegozea on 5 May 2016

The name is a bit awkward

I agree, but I think it's the best description. PlotsBase seems too tightly coupled to Plots, and I want to reserve using that for a "core Plots" separation at some point. CustomPlots, PlotRecipes, and other similar names would (IMO) imply that there is worthwhile content... not just a base package defining a few simple methods. I'd rather have PlotRecipesBase which is this package, and also PlotRecipes which has some worthwhile recipes for common visualizations (I could see moving boxplot, corrplot, etc here)

one concrete symbol per argument

Yes I think that's a good thing to promote. Maybe the readme could have a list of suggested conventions and their intended use/meaning. Then people would follow Plots conventions but it still allows other graphical packages to do their own thing just by converting the keys to their own convention.

tbreloff on 5 May 2016

Maybe not the right place but I think the macro would look a little more intuitive if the argument list looked more like an actual argument list:

@recipe (loss::ModelLoss, args...) begin
    :xlab => _loss_xlabel(loss))
    :ylab => "L(y, h(x))"
    :lab => string(loss)
    value_fun(loss), args...
end

Evizero on 5 May 2016

👍1

Yeah I like that better too

tbreloff on 5 May 2016

I'd rather have PlotRecipesBase which is this package, and also PlotRecipes which has some worthwhile recipes for common visualizations

Maybe, but frankly PlotRecipesBase does simply not sound like a package I would find in Julia Base, if that argument makes sense. It doesn't have a clean ring to it. It should sound like the package that provides package maintainers with the Base Julia functionality of defining how their datastructures are plotted. I don't know the answer (I rarely do) but I think it is worth dwelling on the name a bit

Evizero on 5 May 2016

PlotRecipesBase does simply not sound like a package I would find in Julia Base

Presumably the name doesn't mean that much, and wouldn't carry into Base. Only the types/methods would be in Base, maybe: Base.@plotrecipe and Base.apply_plot_recipe? Do we even need anything else?

tbreloff on 5 May 2016

How about:

CustomPlotsBase
UserPlots

I could also maybe get behind PlotRecipes as the core macro would be @plotrecipe, but then what would we call the package that has "worthwhile recipes for common visualizations" (i.e. the one with actual recipes)?

tbreloff on 5 May 2016

In the end I guess it is a matter if taste but I think the PlotsRecipeBase bame is OK. If I understand it correctly, no user should ever want to load it directly, and in that case I think it is ok to be very specific in the name (which I think it is)

pkofod on 5 May 2016

@pkofod correct. A user of Plots need not know about it. Same with a user of LearnBase. Only the package authors would even know it exists.

tbreloff on 5 May 2016

FYI... I'm working on getting MLPlots working with a @plotrecipe macro... see the tb_recipe branch if you're curious.

tbreloff on 5 May 2016

:+1: for UserPlots

Evizero on 5 May 2016

maybe: Base.@plotrecipe and Base.apply_plot_recipe?

why call the macro @plotrecipe but the function not plot_recipe?

Evizero on 5 May 2016

The macro is what people would actually use... apply_plot_recipe is just an implementation detail. I can be swayed... I went with julia convention that short names can be squashed together, but longer names should have underscores.

So the pkg names I would consider:

UserPlots
PlotRecipes (but only if we come up with a good name for the REAL PlotRecipes)
PlotRecipesBase (my pick)

Lets have a vote!

tbreloff on 5 May 2016

In working on the macro, I found it was tricky to handle positional arguments well. I'm thinking about changing it up to use valid function syntax:

@recipe function plot(roc::Roc, auc::Bool = true)
    :legend => false
    :xlabel => "False Positive Rate"
    ...
    Any[x, 0:1], Any[y, 0:1]
end

with short form available:

type T
    n::Int
end
@recipe plot(t::T) = rand(t.n)

Thumbs up or down on this change? If thumbs down, how much do you care about optional arguments?

tbreloff on 5 May 2016

👍2

Taking extra arguments looks important to me. Has using keyword arguments the same problems than positional arguments?

diegozea on 5 May 2016

LightPlots ?

diegozea on 5 May 2016

Would be easy to add the backend logic (forced, quiet...) ?

diegozea on 5 May 2016

It would be easiest if there's existing syntax that we can steal from. For the arrows I'm doing:

            if e.head == :(=>)
                # swap out this arrow expression with a call to get!(d, ...)
                k, v = e.args[1:2]
                expr.args[i] = :(get!(d, get(Plots._keyAliases, $k, $k), $v))

which looks for expressions of a => b outside of function calls (to ignore things like Dict(a=>b)). If there's another op that I can steal for this purpose that would be easiest. Otherwise, it might be reasonable to allow:

a => b, :quiet

assuming that I can easily parse out the RHS tuple (I think I can).

tbreloff on 5 May 2016

Also do you like => or --> better as the separator?

tbreloff on 5 May 2016

I'm thinking that --> might be less confusing and less likely to clash with something. There's probably a good reason that FactCheck switched to that symbol, right?

tbreloff on 5 May 2016

Interestingly, they also parse differently!

a => b, :quiet    # parsed as Expr:  a => (b, :quiet)
a --> b, :quiet   # parsed as Tuple: ((a --> b), :quiet)

So --> turns out to be a little easier to handle as well.

tbreloff on 5 May 2016

👍1

It works! quiet and force flags are allowed. The following examples work as expected:

using Plots
gr(size=(400,300))

type T end
@recipe function plot(t::T, n::Int = 1)
    :markershape --> :auto
    :markercolor --> :red, :force
    :xrotation --> 5
    :zrotation --> 6, :quiet
    rand(10,n)
end

# plots 5 series of 10 random numbers
# markers are varied, but forced to red
# only one warning (for xrotation)
plot(T(), 5, mc = :blue)

# plots 1 series of 10 random numbers
# markers are octogons, colored red
# only one warning (for xrotation)
plot(T(), shape=:o)

tbreloff on 5 May 2016

🎉3 😄1

Great! Looks nice :)

diegozea on 5 May 2016

Yay!

INFO: Matplotlib version: 1.5.1
LearnBase
[Plots.jl] Initializing backend: pyplot
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase1.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase2.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase3.png matches.  Difference: 0.008991153975603648
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase4.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase5.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/learnbase6.png matches.  Difference: 0.0
6 facts verified.
CorrPlot
/home/tom/.julia/v0.4/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py:1288: UserWarning: findfont: Font family [u'Helvetica'] not found. Falling back to Bitstream Vera Sans
  (prop.get_family(), self.defaultFamily[fontext]))
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/corrplot.png matches.  Difference: 0.0
1 fact verified.
OnlineAI
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/onlineai1.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/onlineai2.png matches.  Difference: 0.0
2 facts verified.
ROCAnalysis
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/rocanalysis.png matches.  Difference: 0.0
1 fact verified.
ValueHistories
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/valuehistories1.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/valuehistories2.png matches.  Difference: 0.0
INFO: Reference image /home/tom/.julia/v0.4/MLPlots/test/refimg/valuehistories3.png matches.  Difference: 0.0
3 facts verified.
INFO: MLPlots tests passed
INFO: No packages to install, update or remove

tbreloff on 5 May 2016

The macro is what people would actually use... apply_plot_recipe is just an implementation detail. I can be swayed... I went with julia convention that short names can be squashed together, but longer names should have underscores.

The underscore makes sense. I am mainly talking about the apply prefix that doesn't seem to add any value

+1 for UserPlots (or 2nd PlotRecipes). A decent name for the real PlotRecipes I would say would be CustomPlots

The new macro looks nice!

Evizero on 6 May 2016

Not sure if it is still being debated, but I would consider optional arguments as very important.

I am also curious if it is at all possible to introduce custom keyword arguments that one could utilize in it's recipe. Like showmargin or something for SVM plots to specify if the separating hyperplane should be drawn. Does Plots pass them through to the recipes?

Evizero on 6 May 2016

I'll make sure keyword args work.

For the names... I just had the thought that there's no logical reason why my @recipe macro has to be applied only to plots/visualizations. At it's core, it's an argument transformer that populates a keyword dictionary with attributes. This could apply to lots more than just plots, so I think it's fair to drop the Plots from the name, and go with RecipesBase.jl. Thumbs __?

tbreloff on 6 May 2016

I got keywords working correctly, accept you can't add a type annotation to the key. I think this is acceptable, as that's more for safety than enabling features (it doesn't change dispatch).

No protests regarding the name RecipesBase.jl, so I'm going to go ahead and start it.

tbreloff on 6 May 2016

This example should showcase everything that's supported right now:

tmp

tbreloff on 6 May 2016

No protests regarding the name RecipesBase.jl, so I'm going to go ahead and start it.

yes, sounds reasonable

regarding the current macro state: looks great! any way one could avoid the kw[:myparam] trickery and write myparamdirectly? Seems confusing

Evizero on 6 May 2016

Yes it might be possible to avoid the trickery... I just made a change so that the kw dict should only have user-defined keywords, so I can probably define variables instead of setting the dictionary values.

tbreloff on 6 May 2016

out of interest, what does force and quiet do? a simple search on readthedocs didn't yield anything. I am guessing force always sets a parameter while omitting it only sets it if not specified. and quiet?

edit: quiet maybe suppresses supported warnings?

Evizero on 6 May 2016

would args... work for positional arguments?

Evizero on 6 May 2016

Yes args... works.

what does force and quiet do?

We discussed this on gitter...

what if you can add a bunch of keywords to the end of the kw: @kw ratio 1 quiet
with meanings: quiet=silently ignore if not supported, required=error if not supported, force=don't allow user override

Going to implement required now...

tbreloff on 6 May 2016

Done. Used require to match the tense of force:

tmp