Plots.jl: Recipe ideas

Created on 10 Jun 2016  ·  75Comments  ·  Source: JuliaPlots/Plots.jl

  • [x] stephist (#223)
  • [ ] marginal density/boxplots/ticks (#189)
  • [ ] 3D Quiver (#178)
  • [x] Graphs (connected nodes) (https://github.com/JuliaPlots/PlotRecipes.jl/issues/2, #48)
  • [ ] Aster (http://johnmyleswhite.github.io/Vega.jl/asterplot.html)
  • [ ] Horizon (http://johnmyleswhite.github.io/Vega.jl/horizon.html)
  • [ ] Population (http://johnmyleswhite.github.io/Vega.jl/popchart.html)
  • [x] [Choropleths](https://github.com/penntaylor/ChoroplethMaps.jl)
  • [x] [grouped histograms](https://github.com/tbreloff/Plots.jl/issues/254#issuecomment-218760020)
  • [ ] waterfall (https://en.wikipedia.org/wiki/Waterfall_plothttps://en.wikipedia.org/wiki/Waterfall_plot)
good first issue help wanted

All 75 comments

What about column scatter plots? They are pretty widely used in Biology since it's like a boxplot in that shows an error bar, but is also plotting the actual data values. At the bottom of this is a good example (made in R):

13227_2016_40_fig12_html

Another name is categorical scatter plot.

In that omage, also the upper bars with the significance level are useful and generally hard to make, but should be easy to make a recipe for that significance annotations with Plots.

If adding significance annotations is possible that was be fantastic, but I thought it would be hard (it would be easy to tell it that a certain plot is *** significant, but then for it to automatically place the annotations? That seems difficult).

No. Annotations should be fairly straightforward (depending on what it is
of course). You'd just create a new series with 'primary = false'.

On Sunday, June 12, 2016, Christopher Rackauckas [email protected]
wrote:

If adding significance annotations is possible that was be fantastic, but
I thought it would be hard (it would be easy to tell it that a certain plot
is *** significant, but then for it to automatically place the annotations?
That seems difficult).


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-225449195,
or mute the thread
https://github.com/notifications/unsubscribe/AA492mAV7TwyoNytWkgzG4bsly3Kj00uks5qLEBCgaJpZM4IzBp9
.

For anyone watching the repo, I just wanted to give a quick demo of features I just added: iter_segments and the curves series recipe.

tmp

If you can't figure out by the screenshot, I added a nice convenience iterator iter_segments, which takes 1 or more vectors (or anything that can be "cycled", which includes scalar values), and gives an iterator that, at each iteration, returns an integer range which is the next consecutive set of indices where the inputs are all non-NaN. I'll be replacing things like gr_polyline with this sort of iteration logic, though it has other uses:

The :curves series recipe will split the x/y/z/fillrange/line_z coordinates into segments, and rebuild each segment using the given coordinates as "control points" in a bezier curve. So you can create nice curves simply by specifying the control points separated by NaN's. The optional npoints keyword arg specifies how many line segments to use to draw each curve segment.

Rather than choropleths, or as a way of doing that, it could be nice with a recipe to plot shapefiles, i.e. the objects returned by Shapefile.jl. Allowing colour would make an easy way to do choropleths, in addition to the other uses of shapefiles.

This should be pretty straightforward. Can we add a recipe to shapefile.jl?

On Thursday, July 7, 2016, Michael Krabbe Borregaard <
[email protected]> wrote:

Rather than choropleths, or as a way of doing that, it could be nice with
a recipe to plot shapefiles, i.e. the objects returned by Shapefile.jl.
Allowing colour would make an easy way to do choropleths, in addition to
the other uses of shapefiles.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231023511,
or mute the thread
https://github.com/notifications/unsubscribe/AA492mtFmhMHtWzg-Wh_B6zAcNkbJxm-ks5qTMFKgaJpZM4IzBp9
.

Don't know, but I have the impression they welcome PRs. I made some super rudimentary headway:

using Plots, Shapefile
pyplot()

shp = open("test.shp") do fd
    read(fd, Shapefile.Handle)
end
pol = shp.shapes[1] # there may be more than one polygon, but in this case only one

xs = Float64[]
ys = Float64[]
for i in 1:pol.points
    push!(xs, i.x)
    push!(ys, i.y)
end

plot(xs, ys, aspect_ratio = :equal) 

This gives a line plot, but what we want for this type of shapefile is a polygon - cannot find in the docs how to do filled polygons.

Add 'st = :shape'! Coordinates can be NaN-separated to have multiple shapes
in the same series. I wonder if this should be added to Shapefile or
PlotRecipes?

On Thursday, July 7, 2016, Michael Krabbe Borregaard <
[email protected]> wrote:

Don't know, but I have the impression they welcome PRs. I made some super
rudimentary headway:

using Plots, Shapefile

shp = open("test.shp") do fd
read(fd, Shapefile.Handle)
end
pol = shp.shapes[1] # there may be more than one polygon, but in this case only one

xs = ys = Float64[]
for i in 1:pol.points
push!(xs, i.x)
push!(ys, i.y)
end

xs and ys become identical - this seems to be a bug, so I had to do a workaround

n = length(xs)/2
ys = xs[2_(1:n)]
xs = xs[2_(1:n)-1]

plot(xs, ys, aspect_ratio = :equal, xlim = (-70, -60))

This gives a line plot, but what we want for this type of shapefile is a
polygon - cannot find in the docs how to do filled polygons.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231049304,
or mute the thread
https://github.com/notifications/unsubscribe/AA492i7qd32NKAN0AP8ImyUvbOhqbIpyks5qTN6XgaJpZM4IzBp9
.

Oh, nice trick with the NaN to get multiple shapes, but I cannot seem to get different colours on the different shapes pic

plot([xs; NaN; xs + 3], [ys; NaN; ys], 
aspect_ratio = :equal, xlim = (-70, -60), st = :shape, 
colour = [:green :red], legend = false)

By the way, the shape file is here

I think it would make sense to put it in Shapefile, since it depends 100% on the functionality of that package.

Right now the easiest way to set different colors per shape is to make each
one a different series. Eventually this will be easier.

On Thursday, July 7, 2016, Michael Krabbe Borregaard <
[email protected]> wrote:

Oh, nice trick with the NaN to get multiple shapes, but I cannot seem to
get different colours on the shape pic
https://www.dropbox.com/s/notob86fj50m0q5/shapefile.png?dl=0

plot([xs; NaN; xs + 3], [ys; NaN; ys], aspect_ratio = :equal, xlim = (-70, -60), st = :shape, colour = [:green :red], legend = false)

By the way, the shape file is here
https://www.dropbox.com/sh/ksdncn7tntdqscm/AACZxHKh4gt-wzKkEs8DgM_Ha?dl=0

I think it would make sense to put it in Shapefile, since it depends 100%
on the functionality of that package.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231053344,
or mute the thread
https://github.com/notifications/unsubscribe/AA492hl17brvtkjVGUo5Opx4_DpQqoyUks5qTOQDgaJpZM4IzBp9
.

Nice, plot([xs (xs + 3)], [ys ys], aspect_ratio = :equal, st = :shape, colour = [:green :red], legend = false) works. I'll make a recipe and do a PR on Shapefile, see what the owners think.

Ah, no, of course making them different series does not work as the polygons will not have the same number of points. I tried using groups to get different colours, but that takes prohibitively long (I stopped my machine after a few minutes). I also tried a for loop with plot! but I could not get it to input to screen. So I will wait a bit with implementing this till I figured out how to do the shapes in different colours.

This is actually easier than you're making it. There are a few improvements to be made to colors and shapes, after which this should work well:

tmp

This is actually easier than you're making it

Story of my life, actually :-)

Anyway, that is amazing. I didn't know about the Shape class, but even if I did I would not have been able to do that so elegantly. I learn a lot from reading it. And it works!

skaermbillede 2016-07-07 kl 15 19 39

:+1: What shape file is that? All the test files seem to be very simple polygons.

That is one I am working on at the moment, of 1500 islands in the IndoPacific. It is not mine so I cannot put it online but I will PM it to you if you want it?

Oh... no need. Do you know of any good sources for other maps/shapes with a permissive license.

I think you should use this instead:

Base.convert(::Type{Plots.Shape}, s::Shapefile.ESRIShape) = Shape([pt.x for pt=s.points], [pt.y for pt=s.points])
@recipe f{T<:Shapefile.ESRIShape}(::Type{T}, shape::T) = Shape(shape)
@recipe f{T<:AVec{Shapefile.ESRIShape}}(::Type{T}, shapes::T) = Shape[s for s=shapes]
@recipe f{T<:AMat{Shapefile.ESRIShape}}(::Type{T}, shapes::T) = Shape[s for s=shapes]'

So that you can do:

tmp

Do you know of any good sources for other maps/shapes with a permissive license.

Yes, the Natural Earth site is excellent, and all data is in the public domain (no credits necessary). Take e.g. this:
http://www.naturalearthdata.com/downloads/110m-physical-vectors/110m-coastline

Cool! It would be pretty awesome to have a package which downloads/unzips this data and wraps loading through Shapefile.jl.

👍

Hey that 4-line code is fantastic, thanks. I will try and generalize it to Line and Point shapefiles as well. There is a hitch in the plotting of the shapefile I suggested, Plots gets confused in a few places about what is outside or inside. This may be a bug in Shapefiles.jl, though. The shapefile plots correctly in R.
screen shot 2016-07-07 at 21 21 06

If there are "jumps" in the same shape it will do this. I suspect the shapes need to be broken up... I can't think of a good way to automate that without domain knowledge of what the shape represents.

Sorry - I think it is a line shapefile. My bad.

Seriously, WTF? :-)

:laughing:

So they aren't actually polygons... they're partial borders. That explains the weirdness.

No there truly is something strange going on. I opened a new session and only typed this:
screen shot 2016-07-07 at 21 49 21

And now I know for sure that it is polygons that plot correctly (with fill colour) in R.

what happens if you add st = :shape to that command?

That, and specifying color instead of fill, helps a lot!
screen shot 2016-07-07 at 22 00 44

getting closer! the border isn't perfect, but I don't know if there's a Plots-solution for that.

No, I am not sure there is. It may be a slight difference in polygon ending convention among shapefile polygons and other shapes. I was just looking into whether I could define the ticks, st, aspect_ratio and legend in a plot recipe to make plotting easier - but it looks like that would require defining a specialized function, right?

you can! here's a "plot recipe" to add to your list:

@recipe function f(::Type{Val{:shapeplot}}, plt::Plot)
    legend := false
    ticks := nothing
    grid := false
    aspect_ratio := 1
    seriestype := :shape
    ()
end
@shorthands shapeplot

and then (with some bonus features that only I possess):

tmp

If we figure out how to extract closed polygons properly, then we're done :+1:

Ha ha, here is the one I just did (but it does have a specialized function name):

@recipe function f(s::Shapeplot)
    seriestype := :shape
    aspect_ratio := :equal
    grid --> false
    legend --> false
    ticks --> :none
    s.args
end

along with

@recipe f{T<:Shapefile.Handle}(::Type{T}, handle::T) = handle.shapes'

that'll work too :+1:

My plot seems to have fewer problems, that is weird.
screen shot 2016-07-07 at 23 28 30

what commit are you on? i might have introduced a bug

0.7.0 here on julia 0.4.5, I'm on my home machine. Do you want me to Pkg.update()? (and is markerstrokewidth = 0 really the best way to suppress plotting the border?)

Try and do shapeplot(shp.shapes') instead of shapeplot(shp.shapes)

Just for fun:

plot(xlim=(-200,200), ylim=(-100,100))
@gif for s in shp.shapes
    shapeplot!(s, c=:grey)
end

tmp

I thought there might have been a bug in the iter_segments logic, but I think it's ok. Not sure why some of those polygons are weird.

Hmm shapes 94 and 95 should really be combined:

tmp

For reference, this is probably the nicest way to grab the files:

fn = "https://github.com/nvkelso/natural-earth-vector/raw/master/packages/Natural_Earth_quick_start/10m_cultural/ne_10m_admin_1_states_provinces.shp"
run(`wget $fn -P /tmp/`)

Sadly they don't seem to be quite robust enough to use out of the box. I'll revisit another time.

I think the issue here is you used the old shapefile (the Lines shapefile with the coastline). I did my plots on the newer shapefile (the Polygons type). Here is how it looks:
skaermbillede 2016-07-08 kl 11 15 30
Shapefile recipes need to be specialized on the type of shapefile.
I don't think there is anything wrong with the natural earth layers, if there are issues anywhere it is probably in the Shapefile.jl package.
Would be nice to do a NaturalEarth package, yes. Does the wget command also work in Windows, though?

There are still some kinks to be ironed out but I think we could make a PR. Do you want to do it, as you wrote the code essentially, or should I? And do you want it on PlotRecipes or to Shapefile?

How about this... I'll add the shapeplot recipe to PlotRecipes, and you can
submit the Shapefile-specific type recipes to Shapefile.jl. If they don't
want it then I'll add it to PlotRecipes with an "is_installed guard".

On Friday, July 8, 2016, Michael Krabbe Borregaard [email protected]
wrote:

There are still some kinks to be ironed out but I think we could make a
PR. Do you want to do, as you wrote the code essentially, or should I? And
do you want it on PlotRecipes or to Shapefile?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231316455,
or mute the thread
https://github.com/notifications/unsubscribe/AA492tJpf8DnWzTFOnG4FZby7Ye4U-0Oks5qThitgaJpZM4IzBp9
.

Sure, let us say that.

Will that require Shapefile.jl to depend on Plots.jl because of the Plots.Shape type?

Ah. Good point. I'll rewrite them to avoid using Shape.

On Friday, July 8, 2016, Michael Krabbe Borregaard [email protected]
wrote:

Will that require Shapefile.jl to depend on Plots.jl because of the
Plots.Shape type?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231334012,
or mute the thread
https://github.com/notifications/unsubscribe/AA492ownvLyNHnF6FAfNNIrj9BEroXCeks5qTi8tgaJpZM4IzBp9
.

@mkborregaard https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231315455
I'd assume to not to have wget on windows (any revision) out of the box.
But https://github.com/JuliaWeb/Requests.jl should provide something (system independend)

👍

I'll rewrite them to avoid using Shape.

Or I could put something like: isdefined(:Plots) && isa(Plots, Module) && include("Plotsrecipe.jl") in the Shapefile module, and then putting the import Plots: Shape statement in the Plotsrecipe.jl file. I think that should keep it from adding the dependency?

No... I'll rewrite them. The shapes are immediately converted to x/y
coordinate vectors anyways... It's a wasteful step.

On Friday, July 8, 2016, Michael Krabbe Borregaard [email protected]
wrote:

I'll rewrite them to avoid using Shape.

Or I could put something like: isdefined(:Plots) && isa(Plots, Module) &&
include("Plotsrecipe.jl") in the Shapefile module, and then putting the import
Plots: Shape statement in the Plotsrecipe.jl file. I think that should
keep it from adding the dependency?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231339884,
or mute the thread
https://github.com/notifications/unsubscribe/AA492ku79lwX7RxbFo0tlSAPuwyWfV25ks5qTjhGgaJpZM4IzBp9
.

Ah-ha, and suddenly my unnecessarily complicated way of doing things may attain renewed prominence 😺
I was going to add an extra step regardless, because some Shapefile Polygons are more than one Shape. The number of shapes in a Polygon is encoded in the parts fieldname. This means that every polygon should be converted to an array of Shapes, that then gets splatted into Plots. An example of this is already in the Shapefile package, for Compose shapes: https://github.com/JuliaGeo/Shapefile.jl/blob/master/src/compose.jl

Let me know what you want me to do here. I am happy that this is getting implemented, especially when the different colours also gets into the release.

Could be easily converted to

function convert(::Type{Shape[]}, shape::Shapefile.Polygon)
    points = {}
    polygons={}
    currentpart=2
    for (i,p) in enumerate(shape.points)
        push!(points, p)
        if i==length(shape.points) || (currentpart≤length(shape.parts) && i==shape.parts[currentpart])
            push!(polygons, Shape([(p.x,p.y) for p in points]))
            currentpart += 1
            points = {}
        end
    end
    polygons
end

@mkborregaard Can you confirm what the shape.parts is? Does that break up a Shapefile.Polygon into several sub-polygons? Is parts the same length as points, or is it a list of "indices of breaks"?

Polygon.parts is an Int32[] . It has the same length as the number of shapes in each Polygon. The first element is always 0. The number is the index in Polygon.points just before the index starting each new point. So, eg if there are 5 parts with a total of 420 points, parts may be [0, 9, 34, 73, 315].
The code I have pasted above should work, though.
So yes, it is a list of indices of breaks.

Ok I'm just going to do this slightly differently so I can't copy/paste that code block. I'll post the recipes soon.

👍

Almost done... which file are you using? I'd like to download the same one as here: https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-231315455

Ok great. For reference, here are the recipes:

To be added to Shapefile.jl:

using RecipesBase
function shapefile_coords(poly::Shapefile.ESRIShape)
    start_indices = poly.parts+1
    end_indices = vcat(poly.parts[2:end], length(poly.points))
    x, y = zeros(0), zeros(0)
    for (si,ei) in zip(start_indices, end_indices)
        push!(x, NaN)
        push!(y, NaN)
        for pt in poly.points[si:ei]
            push!(x, pt.x)
            push!(y, pt.y)
        end
    end
    x, y
end

function shapefile_coords{T<:Shapefile.ESRIShape}(polys::AbstractVector{T})
    x, y = zeros(0), zeros(0)
    for poly in polys
        xpart, ypart = shapefile_coords(poly)
        append!(x, xpart)
        append!(y, ypart)
    end
    x, y
end

function shapefile_coords{T<:Shapefile.ESRIShape}(polys::AbstractMatrix{T})
    x, y = [], []
    for c in 1:size(polys,2)
        xy = shapefile_coords(vec(polys[:,c]))
        push!(x, xy[1])
        push!(y, xy[2])
    end
end

@recipe f(poly::Shapefile.ESRIShape) = (seriestype --> :shape; shapefile_coords(poly))
@recipe f{T<:Shapefile.ESRIShape}(polys::AbstractVector{T}) = (seriestype --> :shape; shapefile_coords(polys))
@recipe f{T<:Shapefile.ESRIShape}(polys::AbstractMatrix{T}) = (seriestype --> :shape; shapefile_coords(polys))

To be added to PlotRecipes:

@recipe function f(::Type{Val{:shapeplot}}, plt::Plot)
    legend --> false
    ticks --> nothing
    grid --> false
    aspect_ratio --> 1
    seriestype := :shape
    ()
end
@shorthands shapeplot

A sample script:

using PlotRecipes, Shapefile
dir = "https://github.com/nvkelso/natural-earth-vector/raw/master/110m_physical/"
fn = "ne_110m_land.shp"
run(`wget $dir/$fn -P /tmp/`)
shp = open("/tmp/$fn") do fd
    read(fd, Shapefile.Handle)
end
shapeplot(shp.shapes, c=:grey)

And some silliness:

tmp

Awesome! Do you want me to put those in the PR? It feels a bit strange, since you wrote this completely.

Just a few questions - it looks like you use NaN separation for the shapes from the same polygon. Does that mean the will retain the same attributes? E.g. will it be possible to pass a vector of colours or annotations (e.g. names) to the shapeplot function with one element per original Polygon and get the right colors out? (necessary for choropleths).

Also, does the sample script work? It shouldn't be possible to plot with just the .shp file, there are at least 4 different files in a shapefile.

Awesome! Do you want me to put those in the PR? It feels a bit strange, since you wrote this completely.

I don't mind... I'd rather not spend the time putting the PR together.

E.g. will it be possible to pass a vector of colours or annotations (e.g. names) to the shapeplot function with one element per original Polygon and get the right colors out?

This is something that is working on the tb_colors branch of Plots... you'd call it like: shapeplot(shapes, c = my_color_list). The one caveat is that the list would treat each "part" as its own polygon, so you may need to handle this specially inside the shapefile code if you want to map a color to each of the Polygon parts.

Also, does the sample script work? It shouldn't be possible to just get the .shp file, there are at least 4 different files in a shapefile.

When downloading from github, you can get just the .shp file without the .zip.

I also think it would be good with a
@recipe f{T<:Shapefile.Handle}(::Type{T}, handle::T) = handle.shapes'
which could dispatch on handle.type - this code only works for type == 5 (there are 28 of them, but I think only 1, 3, and 5 are necessary).

Sure... feel free to add/expand! I only wanted to provide the basics.

Also, as @lobingera said there is the julia alternative to wget

using Requests
tmp = get("https://github.com/nvkelso/natural-earth-vector/raw/master/110m_physical/ne_110m_land.shp")
save(tmp, "tmp.shp")
shp = open("tmp.shp") do fd
        read(fd, Shapefile.Handle)
    end

Sure... feel free to add/expand! I only wanted to provide the basics.

Awesome, I'll do the PR and expand to the other types. I would say that is essential that multi-part polygons are treated as an entity in the plotting, I will look into that once I'm 100% clear what your code is doing.

Should basically all of these go to PlotRecipes.jl and StatRecipes.jl?

@ChrisRackauckas yes

posting this to use in the readme:

tmp

Just now learning Plots myself, what would be involved in expanding the recipe used for ohlc to also overlay various average trendlines? Example from wikipedia

Should be pretty easy to add yourself. May I recommend OnlineStats.Variance
for the tracking?

On Friday, July 29, 2016, AndyGreenwell [email protected] wrote:

Just now learning Plots myself, what would be involved in expanding the
recipe used for ohlc to also overlay various average trendlines? Example
from wikipedia
https://en.wikipedia.org/wiki/Open-high-low-close_chart#/media/File:Bollinger_bands_example,_2_stddevs.png


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tbreloff/Plots.jl/issues/319#issuecomment-236091678,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA492lXRm0gyOSOOtxxqq3OlLstjzHFxks5qaYDWgaJpZM4IzBp9
.

Bump for waterfall 🌊

Bump for a 3D quiver

using Plots
plotly()
x = [1.0, 2.0, 3.0]
y = [0.0, 0.0, 0.0]
z = y
u = [1.0, 1.0, 1.0]
v = [0.0, 1.0, 0.0]
w = [1.0, 0.0, 1.0]
plot(x,y,z, quiver=(u,v,w), seriestype=:quiver)

just ignores z and w (also on gr()).

Was this page helpful?
0 / 5 - 0 ratings

Related issues

Krastanov picture Krastanov  ·  3Comments

ereday picture ereday  ·  3Comments

kleinschmidt picture kleinschmidt  ·  3Comments

apalugniok picture apalugniok  ·  3Comments

lstagner picture lstagner  ·  5Comments