Holoviews: cellmagic output and options parsing problem

[ea42gh]

#%%output backend='matplotlib'
def rectangle(x=0, y=0, width=.05, height=.05):
    return np.array([(x,y), (x+width, y), (x+width, y+height), (x, y+height)])

hv.Polygons(rectangle()).opts([ hv.opts.Polygons(color='indianred',line_alpha=.5, padding=2, backend='bokeh'),
                                hv.opts.Polygons(color='lightblue', padding=2, backend='matplotlib')])

works as expected.
Uncommenting the output backend cell magic results in
WARNING:root:main: Option 'line_alpha' for Polygons type not valid for selected backend ('matplotlib'). Option only applies to following backends: ['bokeh'] in hv 1.11.2.post3+ged1cc85a3

[philippjfr]

You also have to supply the backend to the opts method:

hv.Polygons(rectangle()).opts(hv.opts.Polygons(color='indianred',line_alpha=.5, padding=2, backend='bokeh'), backend='bokeh').opts(hv.opts.Polygons(color='lightblue', padding=2, backend='matplotlib'), backend='matplotlib')

I'd be in favor of making that be less redundant but this is not a bug.

[jbednar]

The software is working as designed, but I'd still consider it a bug from the user perspective, i.e. that the clearly declared backend information is being ignored later. Sure, the backend declaration was only intended for validation when added to hv.opts when designed, but it's very confusing that this information is lost and must be re-provided to the .opts method. I think hv.opts should store the backend in use when it was created (whether as an explicit backend= argument or simply because that's what the default backend was) so that the .opts() method can read that declaration and use it when determining how to apply the options.

[ea42gh]

Now that I know the following works
hv.Polygons(rectangle()).opts(hv.opts.Polygons(color='indianred',line_alpha=.5, padding=2, backend='bokeh'), backend='bokeh').opts(hv.opts.Polygons(color='lightblue', padding=2, backend='matplotlib'), backend='matplotlib')
I can write my code accordingly.

Awkward and unexpected, but I can live with it:
If the design stays this way, there should be a PROMINENT example in the docs
to explain it...

[philippjfr]

I have an issue labelling something that is working as designed a bug even if it is poor API design.

I think hv.opts should store the backend in use when it was created (whether as an explicit backend= argument or simply because that's what the default backend was) so that the .opts() method can read that declaration and use it when determining how to apply the options.

I would disagree here but I think it's the best approach remaining to us. My original opinion which I still stand by is that hv.opts should not be responsible for validation at all, which would get rid of the need for the backend kwarg. IMO the validation and the kwarg should only be introduced when the option is actually applied.

However, I accept that I was overruled on this initially and as long as opts is doing the validation the Options object it returns should remember that information. The only other alternative I can suggest is that the utility simply validates against all loaded backends but I think in that case I actually prefer having Options objects store the specified backend. If we decide to go that route I'd expect this to work:

object.opts(opts.Polygons(facecolor='red', backend='matplotlib'),
            opts.Polygons(fill_color='red', backend='bokeh'))

i.e. opts calls should allow specifying options for multiple backends at the same time.

[ea42gh]

| I have an issue labelling something that is working as designed a bug even if it is poor API design.
agreed.

From my user perspective, I would want semantics that specify
use these options for this backend and use these options for that backend ONCE.

[jlstevens]

I agree this is redundant but essentially I feel that magics are still the best way to do things if you have to switch backends a lot. Adding the backend keyword doesn't do anything for tab completion.

One thing I'll add that all .opts does is build an hv.Options object and you wouldn't need to specify the backend if you do this explicitly. As tab completion isn't working anyway, you might as well do that and supply the backend option only to the .opts method call.

[jlstevens]

I think I would be fine with removing validation and tab completion for hv.opts (but not hv.opts.Element) so hv.opts is just a shortcut for building a hv.Options object. Then this would be useful for working across backends (though the user would need to know the correct keywords to use each time).

[jbednar]

IMO the validation and the kwarg should only be introduced when the option is actually applied.

I strongly disagree. Shouldn't users be able to build up collections of options that include what they want to do for both backends, then apply those as needed later? If I'm writing code that I want people to be able to use with multiple backends, I want to annotate each option with the backend it is intended for, then apply them later without having to specify the backend. I don't see this as an unusual expectation at all.

[philippjfr]

As tab completion isn't working anyway, you might as well do that and supply the backend option only to the .opts method call.

But as the example here shows that you cannot do that. To be able to specify options for a backend other than the one that's currently loaded you have to supply it twice, once to ensure the validation on hv.opts doesn't error and again to ensure that the options aren't applied.

I also disagree that magics address this problem any better, in fact they do not address this problem at all. While they make it easy if you're switching backend, but they do not let you specify options for multiple backends ahead of time.

[jlstevens]

But as the example here shows that you cannot do that.

Even if you just build a hv.Options object and supply that? I was saying you don't need to use hv.opts at all.

[philippjfr]

I strongly disagree. Shouldn't users be able to build up collections of options that include what they want to do for both backends, then apply those as needed later?

I have already accepted that I was overruled on this, so I don't think this is worth discussing further.

[jlstevens]

I want to annotate each option with the backend it is intended for, then apply them later without having to specify the backend.

This could be possible if hv.Options gets an optional _backend attribute that the opts method checks for and uses. I don't really see much of an issue with this, it just hasn't been implemented yet.

[philippjfr]

Even if you just build a hv.Options object and supply that?

Sure, but that's another thing to document and introduce to users, if we can avoid making exceptions here that'd be preferable.

This could be possible if hv.Options gets a _backend attribute that the opts method checks for and uses. I don't really see much of an issue with this, it just hasn't been implemented yet.

If you're on board with this then I think we all agree that's the best solution.

[ea42gh]

To be concrete, in
https://github.com/ea42gh/HoloviewsPlayground/blob/master/Polygons.ipynb
I had tried to figure out how to add a custom element to hv.

One issue is that such an element might have its own concept of options
that internally would have to be translated to options for each backend.
An example of how to do this would be of great interest to me, and might
possibly make a use case to frame this discussion

[philippjfr]

One issue is that such an element might have its own concept of options that internally would have to be translated to options for each backend.

Having one shared options format across all backends has always been appealing but also very daunting. We should probably open an issue to capture discussing this because it comes up frequently.

I think it's something someone with some understanding of the internals of holoviews could experiment and make decent progress on this pretty quickly. The devil is of course in the details but I think playing around with a few prototypes to see how well it could be made to work would be very worthwhile. So if this is something you feel strongly about then please open an issue and I can suggest a few potential approaches that could get someone started.

[philippjfr]

I think this is a reasonable enhancement to get into the 1.11.3 release and doesn't have to wait on 1.12.0.

[ea42gh]

Not sure whether this is a make options the same across all backends issue,
but rather a how do I design a new hv.Element issue (which incidentally raises the question of
how do I specify options for this Element. Is this a discussion worth having at this time?

[philippjfr]

If it's merely about adding a new element no extra discussion is needed, the way to declare those options is well defined, plot options are parameters on the plotting class and style options are defined on the style_opts attribute of the plotting class.

[ea42gh]

Since I have never seen documentation to that effect, I can't really react. It would be nice to see a simple example :-)

[philippjfr]

Since I have never seen documentation to that effect, I can't really react. It would be nice to see a simple example :-)

We definitely still want and need that documentation, but you can look at any plot class in holoviews/plotting or one of my example notebooks on anaconda.org (e.g. https://anaconda.org/philippjfr/multitext/notebook).

[philippjfr]

I'll close this since it's sufficiently covered in https://github.com/pyviz/holoviews/issues/3466

[jlstevens]

Also #3494 has been merged which eliminates the redundancy when specifying backends - you can just supply it in hv.opts.Foo which will generate an Options object that tracks the backend (and that information is then used later as appropriate).

[jlstevens]

@ea42gh To make sure, I can confirm your original example above now works as expected.