Plots.jl: GR backend

[tbreloff]

Current TODO list (assume y = rand(10) for the examples):

• [ ] Support the grid keyword. This should display a plot without the grid lines: plot(y, grid=false)
• [ ] Support handling of NaN (gap in line segment): plot(vcat(1:10, NaN, 12:21), vcat(y,NaN,y))
• [ ] Support removal/setting of xticks/yticks: plot(y, xticks=nothing, yticks=nothing)
• [ ] Support xlim/ylim

[tbreloff]

I did some very basic initial tests. Here's some things that I consider bugs (but may be unavoidable, as I don't know GR very well):

• only one plot window can be open at a time
• if you close the window, there is a fatal error and GR is unusable for the rest of the session

In addition, there are tons of errors on the examples... I recommend doing the following:

Pkg.clone("https://github.com/tbreloff/ExamplePlots.jl.git")
using ExamplePlots
test_examples(:gr)

Note that test_examples also accepts an optional second argument which is the number of a specific example.

[tbreloff]

ps - this is the error when you close the window:

julia> XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 5740 requests (5585 known processed) with 0 events remaining.

[jheinen]

Thanks for merging the PR and doing initial tests. In terms of those results I would like to mention that:

• GR can handle multiple (and different) output devices (simultaneously). I did not implement this feature in the initial release.
• Closing the window (on Linux) will not re-open the X11 display. For applications that produce "figures" (like Plots), this behavior is inconvenient. I will provide a solution in the next GR release.
• I will try to add the missing features (plot types) ASAP

BTW: It seems to me that some of the "bugs" are simply ignored by other backends, e.g.

test_examples(:gr,4)

After analyzing the debug output It turns out, that the markersize is given as an array:

Updating plot items:
...
    markersize: 100-element Array{Float64,1}
...

Does this make sense? PyPlot seems to ignore this.

In the meantime I noticed, that Plots tries to draw marker symbols with different colors and sizes.

[tbreloff]

Thanks for the thorough respone @jheinen. I think this contribution is great, and the time-to-plot seems very quick based on my very brief tests.

BTW: It seems to me that some of the "bugs" are simply ignored by other backends

Yes, Plots is intended to be a superset of functionality of the underlying backends. This means that a user should be able to do anything they want, but they may have to switch backends to accomplish it. Ideally it will be a little easier to understand exactly which features are supported by the backends... Much of this is recorded within supported.jl, but it's not at all complete. You found a good example with PyPlot's markersize... a scalar value is supported, but not an array. I think at some point, a re-write of the "supported" logic is in order. This has been on my radar for a while... see: #7

I'm not at all asking that you fully support all the examples... just that they will help you test your code, and maybe point you to missing functionality.

Thanks again for the contribution!

[lobingera]

two small things,

• can antialiasing be turned on?
• how to resize the window?

[jheinen]

Added TODO messages (see https://github.com/jheinen/Plots.jl/commit/c5d21813888bfcf697b81f5f308f69b12287be07) for missing features - I keep trying and don't expect any problems, except for animations, which are handled internally in GR.

[jheinen]

two small things,

can antialiasing be turned on?
how to resize the window?

Anti-aliasing in GR currently only works for the Qt4 and (OS X) Quartz drivers. The next GR release will contain a cairographics driver which will add this feature in UN*X environments. Resizing, subplots and other things are still on the TODO list for the Plots backend.

[lobingera]

another small thing:

• How to time()

plot(rand(8000000),rand(8000000))

runs a few seconds (wall clock), but

julia> tic(); plot(rand(800000),rand(800000)); toc()
elapsed time: 0.013522369 seconds
0.013522369

without any visible action on screen?

[tbreloff]

Andreas, this question is likely much more complicated and far reaching than Plots/GR. I suspect it has more to do with the time macro, and when the command is considered "complete". I've seen this in the past, and assumed it has more to do with the asynchronous event architecture underlying Julia (which I admit I only have partial knowledge of). Essentially julia hands off responsibility of the command to another thread and doesn't monitor completion of the task on that thread... It just marks it as "complete" when the handoff occurs. (This is mostly guessing of course)

On Jan 5, 2016, at 12:20 PM, Andreas Lobinger [email protected] wrote:

another small thing:

How to time()
plot(rand(8000000),rand(8000000))
runs a few seconds (wall clock), but

julia> tic(); plot(rand(800000),rand(800000)); toc()
elapsed time: 0.013522369 seconds
0.013522369
without any visible action on screen?

—
Reply to this email directly or view it on GitHub.

[KristofferC]

Use tic(); display(plot(rand(800000),rand(800000))); toc()

Your code doesn't plot for the same reason that plot(rand(800000),rand(800000)); doesn't plot, (note the ;).

[tbreloff]

@KristofferC you're right I misread that... display is suppressed with a semicolon at the REPL (and always suppressed within scripts), so you need an explicit call to display (or gui(), which ensures it goes to the PlotsDisplay if you have others set).

The behavior I'm referring to is that at some part of the display pipeline (depending on the backend), julia will "hand off" processing, and will claim that the command is "complete", even though the gui window, browser window, or whatever else may not be complete. I just tested this with Gadfly, and the image was visibly loading, even though julia had already printed the toc() result.

[KristofferC]

Yeah, I guess some backends return and finish the drawing asynchronously.

[jheinen]

@tbreloff : Making progress but still some TODOs:

• linetypes :path3d, :scatter3d, :surface, :wireframe, :ohlc, :pie
• colored contour lines
• multiple figure output

http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

[tbreloff]

Looking good! Let me know when you're ready to try a merge.

On Mon, Jan 25, 2016 at 11:53 AM, Josef Heinen [email protected]
wrote:

@tbreloff https://github.com/tbreloff : Making progress but still some
TODOs:

• linetypes :path3d, :scatter3d, :surface, :wireframe, :ohlc, :pie
• colored contour lines
• multiple figure output

http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

—
Reply to this email directly or view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/108#issuecomment-174579525.

[Evizero]

This looks nice!

[tbreloff]

So that it doesn't get lost, I need to help with this request from @jheinen

@tbreloff : Need some help with the show / display logic. gr.jl has two writemime methods and both (PNG + SVG) are called when a plot is finalized (results in two files gks.png and gks.svg). Can I set the defaultOutputFormat to something other than PNG, e.g. SVG? The PNG writemime method should only be invoked when the user calls savefig with a .png file extension.

[jheinen]

Most of the stuff should work right now: http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

To create animation, I had to disable transparency when converting the PNGs to GIF (see commit 8cd6b0b). Without the -alpha off option, all images are overlayed. The change should not affect other backends.

Should I provide the output for the http://plots.readthedocs.org/en/latest/examples/gr/ section? I'd still prefer SVG :-) - it's your choice ...

[tbreloff]

These examples look really great. Nice work!

As for the doc examples, they are auto-generated with a script in ExamplePlots.jl. I have had the same thought about changing the format... For example Plotly would be best as html, since then you could see the interactivity. It's on my list and I'm open to PRs there if you have good ideas.

On Feb 13, 2016, at 3:45 AM, Josef Heinen [email protected] wrote:

Most of the stuff should work right now: http://pgi-jcns.fz-juelich.de/pub/doc/ExamplePlots.html

To create animation, I had to disable transparency when converting the PNGs to GIF (see commit 8cd6b0b). Without the -alpha off option, all images are overlayed. The change should not affect other backends.

Should I provide the output for the http://plots.readthedocs.org/en/latest/examples/gr/ section? I'd still prefer SVG :-) - it's your choice ...

—
Reply to this email directly or view it on GitHub.

[jheinen]

Can I help to provide the doc examples for GR? Would it help to force PNG for the inline format, e.g. thru a configuration option or an environment variable (PLOTSBACKEND, like MPLBACKEND for matplotlib)?

[tbreloff]

Yes you might be able to help. The examples are generated from ExamplePlots, using this method: https://github.com/tbreloff/ExamplePlots.jl/blob/master/src/example_generation.jl#L206

I tried running it and it crashed for me... Maybe you'll be able to debug? If you get it working a PR would be great.

On Feb 27, 2016, at 6:03 AM, Josef Heinen [email protected] wrote:

Can I help to provide the doc examples for GR? Would it help to force PNG for the inline format, e.g. thru a configuration option or an environment variable (PLOTSBACKEND, like MPLBACKEND for matplotlib)?

—
Reply to this email directly or view it on GitHub.

[jheinen]

Added missing marker shapes (GR native) ...

[tbreloff]

It seems like you've made a lot of progress on your fork of Plots... Any interest in submitting a PR to my dev branch soon?

On Mar 5, 2016, at 7:01 AM, Josef Heinen [email protected] wrote:

Added missing marker shapes (GR native) ...

—
Reply to this email directly or view it on GitHub.

[jheinen]

This patch requires a new GR build, which is already finished for the UN*X systems. But currently I don't have access to our Windows build system - so, next week, when I'm back in the office, I'll make a PR.

[tbreloff]

Sounds good. Thanks.

On Mar 5, 2016, at 8:19 AM, Josef Heinen [email protected] wrote:

This patch requires a new GR build, which is already finished for the UN*X systems. But currently I don't have access to our Windows build system - so, next week, when I'm back in the office, I'll make a PR.

—
Reply to this email directly or view it on GitHub.

[jheinen]

Still don't understand, why both mime methods (svg + png) are called.

Nevertheless, make a PR now ...

[tbreloff]

First: Will it be possible to have custom marker shapes?

NOTE: the first time I run this, I get the error:

KeyError: Plots.Shape([(-1,-1),(1,0),(-1,1)]) not found

and the next time I run this I get the error:

attempt to save state twice
KeyError: Plots.Shape([(-1,-1),(1,0),(-1,1)]) not found
in gr_display at /home/tom/.julia/v0.4/Plots/src/backends/gr.jl:316
in writemime at /home/tom/.julia/v0.4/Plots/src/backends/gr.jl:682
in sprint at iostream.jl:206
in display_dict at /home/tom/.julia/v0.4/IJulia/src/execute_request.jl:29

As for the second error... I wonder if a pattern of try/catch/finally would be a better solution:
Solved with @gr_state macro

julia> for i=1:10
           try
               println("save")
               rand()<0.3 && error()
           catch err
               println(err)
           finally
               println("restore")
           end
       end
save
restore
save
restore
save
restore
save
restore
save
ErrorException("")
restore
save
ErrorException("")
restore
save
restore
save
restore
save
ErrorException("")
restore
save
restore

We could even add a macro to do this:

@state begin
    # do something
end

which could expand to:

try
  GR.savestate()
  # code block
catch err
  # do something with err
finally
  GR.restorestate()
end

At a minimum, this would clean up the code significantly. Thoughts?

[tbreloff]

I added this macro (easier to do than discuss):

macro gr_state(expr::Expr)
  esc(quote
    GR.savestate()
    try
      $expr
    catch
      GR.restorestate()
      rethrow()
    end
    GR.restorestate()
  end)
end

and the state error goes away. (obviously the Shape error is still there, but at least it doesn't corrupt the session)

[tbreloff]

This drives me nuts... I'd really like to figure out a solution: Solved by clearing/rebuilding GR

julia> using Plots; gr()
Plots.GRPackage()

julia> plot(rand(10))
[Plots.jl] Initializing backend: gr

     [ I manually close the GUI window... ]

julia> XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 284 requests (195 known processed) with 0 events remaining.

julia> plot(rand(10))

    [ Hangs forever... ]

Ideally there would be cleanup that happens when the window is closed so that the session is not ruined.

[tbreloff]

For the state macro, see: https://github.com/tbreloff/Plots.jl/commit/cdff5a9039c2f51b041a819cbdfc2dc72ba67ac9

[jheinen]

It seems that your are not using the GR master branch. The XIO problems has been solved in the meantime. Please rm, free and add the GR package.

I'd prefer to "officially" support the Shape function instead of catching exceptions. What should Plots.Shape([(-1,-1),(1,0),(-1,1)]) do? Does it simply create two lines given by three points?

[jheinen]

Pls keep in mind: the GR build.jl script will not download the run-time if it finds a locally installed GR, e.g. in $HOME/gr, /usr/local/gr or as specified by the GRDIR environment.

[KristofferC]

Can confirm that I have no problems closing and opening new plots with the lastest GR.

[tbreloff]

your are not using the GR master branch

Ok reinstalling GR now.

I'd prefer to "officially" support the Shape function instead of catching exceptions.

Of course we want everything to work, but that doesn't change the need for graceful handling of errors.

What should Plots.Shape([(-1,-1),(1,0),(-1,1)]) do?

It should define a marker shape (in "unit coordinates centered at zero"). Try in another backend (for example, gadfly):

shape = Shape([(-1,-1),(1,0),(-1,1)])
plot(rand(10), marker = (shape, 20, 0.3))

[tbreloff]

After manually clearing all traces of GR from my ~/.julia directory, I was able to rebuild GR, and I confirm the window-closing-error is gone. Hurrah!

[tbreloff]

I have a temporary fix for the Shape issue here: https://github.com/tbreloff/Plots.jl/commit/9d3e0651e2848144a51a47779b983d204d59c306

[tbreloff]

I updated the first comment with a TODO list... I'll try to add/remove as we make progress.

Thanks again for working on this @jheinen... I'm looking forward to using GR for my work.

[jheinen]

... I'm getting closer (see e2e031c).

The TODOs should be easy to implement. But I'm getting sleepy ...

[tbreloff]

Just out of curiosity I added GR to the travis tests. The linux test passes, osx fails with:

[Plots.jl] Initializing backend: gr
INFO: Testing plot: gr:1:Lines
GKS: Ghostscript support not compiled in
  Error :: (line:505)
    Expression: image_comparison_tests(pkg,i,debug=debug,sigma=sigma,eps=eps) |> success --> true
    SystemError: opening file gks.png: No such file or directory
     in open at /Users/travis/julia/lib/julia/sys.dylib (repeats 2 times)
     in writemime at /Users/travis/.julia/v0.4/Plots/src/backends/gr.jl:693
     in png at /Users/travis/.julia/v0.4/Plots/src/output.jl:8
     in png at /Users/travis/.julia/v0.4/Plots/src/output.jl:11
     in anonymous at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:40
     in test_images at /Users/travis/.julia/v0.4/VisualRegressionTests/src/imgcomp.jl:79
     in image_comparison_tests at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:52
     in anonymous at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:271
     in do_fact at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:333
     [inlined code] from /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:271
     in image_comparison_facts at /Users/travis/.julia/v0.4/Plots/test/imgcomp.jl:61
     in anonymous at /Users/travis/.julia/v0.4/Plots/test/runtests.jl:33
     in facts at /Users/travis/.julia/v0.4/FactCheck/src/FactCheck.jl:448
     in include at /Users/travis/julia/lib/julia/sys.dylib
     in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
     in process_options at /Users/travis/julia/lib/julia/sys.dylib
     in _start at /Users/travis/julia/lib/julia/sys.dylib
Out of 3 total facts:
  Verified: 2
  Errored:  1

[jheinen]

Can I access the travis.ci log to verify the build step?

[tbreloff]

This is what I see for GR:

INFO: Building GR

INFO: Downloading pre-compiled GR 0.17.3 binary

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed


  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0 20.3M    0  118k    0     0   120k      0  0:02:53 --:--:--  0:02:53  120k
  7 20.3M    7 1573k    0     0   768k      0  0:00:27  0:00:02  0:00:25  769k
 17 20.3M   17 3745k    0     0  1236k      0  0:00:16  0:00:03  0:00:13 1236k
 28 20.3M   28 6008k    0     0  1502k      0  0:00:13  0:00:03  0:00:10 1502k
 40 20.3M   40 8397k    0     0  1689k      0  0:00:12  0:00:04  0:00:08 1689k
 52 20.3M   52 10.7M    0     0  1848k      0  0:00:11  0:00:05  0:00:06 2189k
 65 20.3M   65 13.3M    0     0  1965k      0  0:00:10  0:00:06  0:00:04 2464k
 80 20.3M   80 16.4M    0     0  2096k      0  0:00:09  0:00:08  0:00:01 2618k
 94 20.3M   94 19.1M    0     0  2188k      0  0:00:09  0:00:08  0:00:01 2740k
100 20.3M  100 20.3M    0     0  2239k      0  0:00:09  0:00:09 --:--:-- 2867k

[jheinen]

Ok. I should tag a new GR version ASAP - the latest pre-compiled binary is required:

$ julia --color=yes -e "Pkg.build(\"${JL_PKG}\")"
INFO: Building GR
INFO: Downloading pre-compiled GR latest binary
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.3M  100 20.3M    0     0  2860k      0  0:00:07  0:00:07 --:--:-- 3970k
19.00s$ julia --check-bounds=yes --color=yes -e "Pkg.test(\"${JL_PKG}\", coverage=true)"
INFO: Testing GR
running /Users/travis/.julia/v0.4/GR/test/ex.jl ...
running /Users/travis/.julia/v0.4/GR/test/griddata.jl ...
INFO: GR tests passed

[tbreloff]

Ok no hurry from my end... I was just curious whether it would pass.

[tbreloff]

Still don't understand, why both mime methods (svg + png) are called.

It seems this is intentional on the part of IJulia (seems like a bad decision to me, but I'm sure there's a reason for it):

https://github.com/JuliaLang/IJulia.jl/blob/master/src/inline.jl#L31-L40

I'm considering overriding this method for Plots so we're not constantly rebuilding multiple plots.

[tbreloff]

IJulia weirdness fixed on dev. See #157

[tbreloff]

I feel like it's really important we change the behavior of writing out a temporary file gks.png to the current working directory. If you need a temporary file, create one with tempname() * ".png". It drives me insane to have every directory polluted with gks.* files.

[tbreloff]

For the broken markers example:

I see this error when running from the REPL:

julia> scatter(x,y,m=(8,:auto),lab=map(string,markers),bg=:linen,xlim=(0,10),ylim=(0,10))
GKS: unable to load font

It might be unrelated, but figured I'd point it out.

[tbreloff]

To summarize what I see for the markers example:

• It works fine if you run it from the REPL and display in the window.
• When you call png(...) it returns as if successful, but the file is whatever was last saved.

Presumably there is a silent error somewhere (probably inside GR.jl? I can't find any try blocks in the Plots code), only when displaying to a png, and so the gks.png file is never written to.

Do you have any ideas of how to investigate?

[tbreloff]

Question... does GR support alpha values for colors? (either RGBA or setting the alpha seperately?)

[jheinen]

@tbreloff : I see the same behavior for the marker example (except the GKS: unable to load font error). When I remove the :auto parameter, it works as expected. If :auto is added to the marker options, the gr() methods are not called.

I still have no idea, but there is something going wrong: When I use the pyplot() backend I get the following output:

Using gks.png as the default output file (in the current directory) is not a good idea. I will change this ASAP.

[jheinen]

Question... does GR support alpha values for colors? (either RGBA or setting the alpha seperately?)

The alpha channel can be set with gr.settransparency(a), 0 <= a <=1.

[tbreloff]

Thanks for the quick response @jheinen.

When I remove the :auto parameter, it works as expected. If :auto is added to the marker options, the gr() methods are not called.

Maybe there's a discrepency between the supportedMarkers() and what is actually supported? The :auto value is replaced with a valid markershape before it gets to the GR backend code.

When I use the pyplot() backend I get the following output:

I suspect you accidentally made a vector of strings (as opposed to a 1xN matrix), so the whole vector was applied to each series.

The alpha channel can be set with gr.settransparency(a), 0 <= a <=1.

Great to hear... this should be on the list of things to add. Each of linecolor, markercolor, markerstrokecolor, and fillcolor can take an RBGA value, and there are also fields linealpha, markeralpha, markerstrokealpha, and fillalpha. You might want to add something like (untested):

function get_gr_transparency(d::KW, str)
  c, a = d[symbol(str*"color")], d[symbol(str*"alpha")]
  a == nothing ? alpha(c) : a
end

# assuming we're given a KW dictionary d, and we want to set the alpha for a line:
gr.settransparency(get_gr_transparency(d, "line"))

[tbreloff]

GR is pretty well along, so lets open new issues for anything that comes up

[ptoche]

@KristofferC, @tbreloff

I came to this thread looking for information about tic(), toc() and plot, so let me report that on my setup (Julia 0.5.1, with Juno and updated packages as of now):

using Plots
plot(sin, -2pi, pi, label="sine function")

produces a plot, while

tic()
using Plots
plot(sin, -2pi, pi, label="sine function")
toc()

does not. It does produce a timing that's probably about right, but no plot.

This is not related to any line ending with a semi-colon ; but related to the effect of wrapping the call to plot() with tic() and toc().

Apologies if this comment is in the wrong place, but it seems to be quite closely related to a discussion you had earlier on this thread. I can start a new thread if you'd prefer. Thanks.

[mkborregaard]

wrap the plot in display

[ptoche]

@mkborregaard, worked great, thanks!

48.2 seconds

[mkborregaard]

Shit. This is among other things because precompilation is currently turned off for the metadata version. Try Pkg.checkout("Plots"), that should allow the package to precompile next time you run it, so on subsequent runs it'll be (somewhat) faster.

[ptoche]

@mkborregaard, thanks! Did that and got 52.4 seconds. and 0.56 seconds on a second run. Thanks.

[mkborregaard]

It'd be nice if we could fix the issues preventing precompilation. Not a small task though at the moment.