Tmap: Towards tmap 4.0

[mtennekes]

In the current state, tmap is not so generic that extensions are easy. Mainly, this has to do with the fact that aesthetics are pre-specified, e.g. tm_polygons has col, tm_symbols has col, size and shape, and so on. The main problem is that the corresponding arguments, such as style, palette, labels etc are also hard-coded. This causes two problems: 1) it is hard to maintain and 2) it is increasingly hard for the users to find the right argument.

It would be more elegant to specify those elements with helper functions, e.g. like this:

tm_shape(x) +
   tm_polygons("pop_dens", setup.col = aes_col(style = "quantile", palette = "Purples"))
tm_shape(y) +
   tm_symbols(size = "count", size.setup = aes_size(sizes.legend = c(100, 200, 400), size.max = 1000))

So tm_polygons has just two main arguments, col (like in the current version) and setup.col, which specifies all aspects related to the mapping data->color. Some other arguments of tm_polygons can stay, such as id, zindex and group.
For tm_symbols there are 2x3 main arguments: col, size, and shape, and setup.col, setup.shape and setup.col.

One thing to be aware of is backwards comparability. We can work around this, to add the setup.x arguments, and make the existing arguments deprecated (so leave them for some time with a warning when a user specifies them).

What do yo think?

[Nowosad]

@mtennekes I like this idea.
I do not know how do you plan to implement this idea, but maybe it would be possible to use the syntax below:

```r
tm_shape(y) +
tm_symbols(size = "count",
size.setup = aes(legend = c(100, 200, 400), max = 1000))
````

My ideas:

1. Each argument accepting variable will have its own *.setup argument, e.g. col.setup, size.setup, etc.
2. Each *.setup argument will accept only one function, e.g., aes. In other words, aes would be universal - working for colors, shapes, etc... (On the side note: it would be probably best not to use the aes name, as it would overlap with the ggplot2 function.)
3. Having backward comparability would be great.

Again, I do not know if my ideas are possible to do, but I hope they can be useful in the discussion.

[mtennekes]

Just started a new branch 'v4'. The aim is to re-implement tmap from scratch, using code chunks from the current version if needed. It feels like taking 10 steps back and then 30 steps forward.

Why from scratch?

• The current version written in a procedural manner. Although I've reorganized code in the past, it is still unclear what function does what due to the complexity of tmap.
• The layer functions have many hard-coded arguments, and are therefore not easy to extend (see my post above).
• Although tmap already has been built on sf and stars, there are still leftovers and limitations from the implementation on sp and raster.
• There are new developments and insights that ask for re-implementation, for instance vector cubes (#458 and #497).
• The combination of these issues makes it hard to extend tmap, and to allows other developers to join.

User perspective:

• As mentioned before, the layer function arguments will be ordered in a different manner. Rather than a huge argument list per layer function, we will organize those arguments as suggested in the previous posts.
• Other than that, everything else will probably work the same.
• Backwards compatibility will be guaranteed.

Developers perspective:

• Adding new layer functions or aesthetics will become easier. E.g. fill patterns (#49), border lwd and lty (#51 and #56), gradient colors (#503), border color (#515).
• Writing functions to process layers (from data to visual variables) will become straightforward.
• The visualization part will be strictly separated and modular. There will be two visualization modules, one that uses grid graphics (the current plot mode), and one that uses leaflet (the current view mode). This separation makes it easier to extend tmap with new graphics engines (e.g. CesiumJS).

Any help is welcome:

• Currently, I'm the only one who knows about all complexity within tmap, but my aim is to make this knowledge explicit by writing proper code. Please feel free to join me.
• Also, suggestions on the user perspective are more than welcome. If you're struggling with certain elements (e.g. legends), please think about how it can be improved from a user perspective. Although backwards compatibility is important, user friendliness is at least as important in my opinion, especially on the long run.

[tim-salabim]

Ha @mtennekes I feel you! I have had similar thoughts about mapview on and off in the past. A bold step, but I truly think it is worth the effort.

One suggestion I have for your consideration is to break out the functionality to create legends into its own package. I've been toying with the idea of a leafkey package for a while now to streamline legend creation (mainly for leaflet - but that could easily be extended to grid based legends). I would love to be able to out-source the legend mess from mapview, as I am still struggling with all the corner cases that users come up with.

Pls, let me know your thoughts

[mtennekes]

@tim-salabim you mean just plotting the legend, or also the processing of the content? Both are challenging, but they are currently working (albeit improvements can be made). At the moment, I'm not sure if it should be put in a separate package, since it is an essential part of tmap (especially the data processing). However, I can image that I'll put all grid based plotting methods, including the legend, in a separate package that is imported by tmap.

For the leaflet pipeline, a package such as leafkey would be very nice to have! Is there also an package for creating popups? I mean, I have one in tmap, but perhaps it is also better to 'outsource' it.

But first my focus is to re-structure the current code into strictly defined building blocks.

[tim-salabim]

you mean just plotting the legend, or also the processing of the content?

Both. As I said, I really struggle with all the nitty-gritty details of legend layout. What you say about potentially splitting out static/grid functionality to it's own package makes total sense, also from a dependency point of view. You could have tmap.static & tmap.interactive packages for example and let tmap serve as a combination of both. Just a thought...

For the leaflet pipeline, a package such as leafkey would be very nice to have! Is there also an package for creating popups? I mean, I have one in tmap, but perhaps it is also better to 'outsource' it.

We have leafpop which you can use to create tables/graphs/images currently. There's also functionality to embed htmlwidgets content and videos, but both could use some love...

[Nowosad]

Hi @mtennekes and @tim-salabim - I've prepared a summary of possible map layer and their aesthetics. This could be useful while thinking about version 4 (is it?).

Layers and aesthetics list

Table: Map layers and their aesthetics that use data variables

|layer |aes |
|:------------------|:-----------------------------------|
|tm_polygons() |fill, color, stroke, linetype |
|tm_symbols() |fill, color, size, shape, stroke |
|tm_fill() |fill |
|tm_bubbles() |fill, color, size, shape, stroke |
|tm_square() |fill, color, size, shape, stroke |
|tm_lines() |color, stroke, linetype |
|tm_raster() |color |
|tm_text() |color, size, shape, text |
|tm_borders() |color, stroke, linetype |
|tm_dots() |color, size, shape |
|tm_markers() |color, size, shape, stroke |
|tm_iso() |color, size, linetype, text, stroke |
|tm_rgb()/tm_rgba() |color |

Questions/comments

1. Should I also list other aesthetics (e.g. alpha, colorNA, text family, text fontface, text angle)?
2. Should we distinguish between data-based aesthetics (e.g., color or size) and single value aesthetics (e.g., colorNA, text family)?
3. What to do with alpha?
   Is it in the first or the second group?
4. tm_raster()/tm_rgb()/tm_rgba() - should it use "color" or "fill"?
5. tm_dots() - should it have aesthetics like shape and color?
6. I have tried to separate some meanings.
   For example, tm_symbols() - "size" relates to the symbol size, and "stroke" relates to the border/line width.
   What do you think about it?
7. What to do with tm_iso()? "color" there could relate to the text color or line color...
   (It is the only corner case, I was able to find).
8. Also, maybe we can learn something useful from https://ggplot2-book.org/internals.html.

[mtennekes]

Great, thanks!

1. If you could list them in another column, that would be great. (See also answer to 2)
2. Yes and no. What I am planning to do, is a create a generic layer function, in which any aesthetic can be used data-driven. So, the developer can decide which aesthetic can be used data-driven and which not. Your table is a good starting point for the implementation of the layers that are included from the beginning, which I will implement. For instance, I'll probably implement tm_lines with the data-driven aesthetics color, stroke, and linetype. Aesthetics such as alpha are treated similarly, but the data-driven functionality isn't implemented yet (e.g. alpha.setup will be an empty list). If someone wants to use alpha as an aethetic as well, he/she is welcome to implement that and give me a pull request (or I could do it later). People are also welcome to add a totally new aesthetic for tm_lines, such as gradient (https://github.com/mtennekes/tmap/issues/503).
3. See previous answers. You can place alpha in the new column.
4. Good question. Pros and cons. From a methodological point of view, color would be better I think, since there normally are no raster borders. However, non-regular stars are a bit of a grey zone, since they are often regarded as polygons. But in essence, they should still be plot without borders, unless explicitly specified otherwise: e.g. the user could be able to call tm_polygons after specifying a (non-regular) stars object.
5. Originally, I had tm_dots(color), tm_bubbles(size, color), and tm_symbols(size, color, shape)`, where the non-listed aesthetic can be specified but not in a data-driven way. Not sure whether there will be any difference in the new implementation, other then different default values.
6. I like that.
7. Not sure what to do with tm_iso. Ideally, I want to wait with its implementation, since it is not a base layer, and not easy to implement. To answer you question, not sure that to do this the aesthetic names of these 'doublelayers (tm_annotatewith connection lines being another one). Maybetext.color`?
8. Yes, indeed. I've looked it and can use some of it.

[Nowosad]

I've checked all functions, and the only obvious argument that could be used as data-driven is alpha.
One question - do we assume (similarly to ggplot2) that alpha relates to all aspects of visualization (e.g., fill and borders in tm_polygons())? Or do we want to keep separate alpha for fill and separate alpha for borders?

|layer |aes |other_aes |
|:------------------|:----------------------------------------|:---------|
|tm_polygons() |fill, color, stroke, linetype |alpha |
|tm_symbols() |fill, color, size, shape, stroke |alpha |
|tm_fill() |fill |alpha |
|tm_bubbles() |fill, color, size, shape, stroke |alpha |
|tm_square() |fill, color, size, shape, stroke |alpha |
|tm_lines() |color, stroke, linetype |alpha |
|tm_raster() |color |alpha |
|tm_text() |color, size, shape, text |alpha |
|tm_borders() |color, stroke, linetype |alpha |
|tm_dots() |color, size, shape |alpha |
|tm_markers() |color, size, shape, stroke |alpha |
|tm_iso() |color?, size?, linetype?, text?, stroke? |alpha |
|tm_rgb()/tm_rgba() |color |alpha |

[mtennekes]

Thanks.

I will probably start implementing the layers and aesthetics which are currently there. For instance, for tm_symbols: fill, size and shape. So in hindsight it may be better to make no distinction between data-driven and non-data driven. Perhaps only a distinction between implemented and non-implemented, which obviously changes over time as more aesthetics will be implemented. For alpha it makes less sense from a methodological point of view, but to be as flexible as possible, the option to implement it as data-driven is there. I can image use cases in which it could be useful.

To answer your question: separate alpha, to be as flexible as possible. Moreover, for many maps, e.g. choropleths, this is quite useful.

[mtennekes]

A question for you (or any other user): how shall we let the user specify multiple column names?

There are two applications:

1. A column name for each small multiple, e.g. tm_shape(World) + tm_polygons(c("HPI", "economy")).
2. Multiple column names for one aesthetic, e.g. a bivariate choropleth (#183) or tm_rbg() which 'translates' three columns as RGB channels to one color column.

Maybe we could think of a special character syntax, like a _ separator, e.g. color = "colA _ colB". Or an alternative of the c function, say d, e.g.: d("colA", "colB").

What do you think?

[Nowosad]

@mtennekes I would start by adding another case - currently, tm_facets() arranges facets in the same fashion as facet_wrap(), however, it could be useful to have something similar to facet_grid() - allowing for two-dimensional facets.

How about:

• "colA | colB" (or just keep c("HPI", "economy"), or both) for the column name for each small multiple
• "colA _ colB" for two-dimensional facets
• "colA + colB" for multiple column names for one aesthetic

[mtennekes]

This current tm_facets example

World$HPI3 <- cut(World$HPI, breaks = c(20, 35, 50, 65), 
    labels = c("HPI low", "HPI medium", "HPI high"))
World$GDP3 <- cut(World$gdp_cap_est, breaks = c(0, 5000, 20000, Inf), 
    labels = c("GDP low", "GDP medium", "GDP high"))

tm_shape(World) + 
    tm_fill("HPI3", palette="Dark2", colorNA="grey90", legend.show = FALSE) + 
    tm_facets(c("HPI3", "GDP3"), showNA=FALSE, free.coords = FALSE, drop.units = FALSE)

is similar to facet_grid or do I miss something?

• I like the "colA | colB" notation, although math people may think of it as A given B. If we decide to use that, we can still keep the current c("colA", "colB") along.
• Do we need a special notation for grid faceting, or do we get away with the current one (so c("colA", "colB")) for tm_facets?
• How about "colA, colB"? The plus operator may suggest addition of the values.

[MarcioFCMartins]

I feel that the c("colA", "colB") notation is more intuitive and similar to what the R ecosystem tends to use - even the tidyverse. The idea of combining multiple names in a single string sounds counter-intuitive to me, while having 1 string == 1 column, 2 strings == 2 columns sounds much more like what I would expect.

Unless there's some advantage that I am missing here (my experience is limited, so I might be missing something), I would simply keep the vector approach.

[Nowosad]

@mtennekes - the above example is awesome! That is great. It would be great to think about how it could work for stars objects. And, you are right - we do not need a special notation for grid faceting.

@MarcioFCMartins - thank you a lot for your opinion! I will try to explain better what is the issue we are discussing here. Currently, when you specify two variables (e.g., c("Median_income", "Sex_ratio"), see the code below), you will get two panels. However, we plan to add bivariate choropleths (https://github.com/mtennekes/tmap/issues/183#issuecomment-670554921), where the user should also specify two variables. Therefore, our main question is how the user can have two panels, where each panel is a different bivariate choropleth? Is that explanation is clear? What is your opinion on that?

library(sf)
library(tmap)
library(spData)

tm_shape(nz) +
  tm_polygons(c("Median_income", "Sex_ratio"))

^{Created on 2020-12-23 by the reprex package (v0.3.0)}

[MarcioFCMartins]

Oh I see now, thank you for the explanation! The inclusion of bivariate scales is great, but now I see why you're having this issue.

I prefer using one function to specify small multiples and one to specify bivariate color scales, like the d suggested by @mtennekes, while keeping the current c() for small multiples. This is how it would look:

2 small multiples, single variable color scales:

tm_shape(nz) +  
    tm_polygons(c("Median_income", "Sex_ratio"))

no small multiples, bivariate color scale:

tm_shape(nz) +  
    tm_polygons(d("Median_income", "Sex_ratio"))

2 small multiples, one with single variable color scale, one with bivariate color scale:

tm_shape(nz) +  
    tm_polygons(c(d("Population", "Median_income"), "Sex_ratio"))

Maybe just use a more informative name, such as bivar().

Thanks to both of you for all your work in the R spatial ecosystem!

[KiranmayiV]

Just started a new branch 'v4'. The aim is to re-implement tmap from scratch, using code chunks from the current version if needed. It feels like taking 10 steps back and then 30 steps forward.

Why from scratch?

• The current version written in a procedural manner. Although I've reorganized code in the past, it is still unclear what function does what due to the complexity of tmap.
• The layer functions have many hard-coded arguments, and are therefore not easy to extend (see my post above).
• Although tmap already has been built on sf and stars, there are still leftovers and limitations from the implementation on sp and raster.
• There are new developments and insights that ask for re-implementation, for instance vector cubes (#458 and #497).
• The combination of these issues makes it hard to extend tmap, and to allows other developers to join.

User perspective:

• As mentioned before, the layer function arguments will be ordered in a different manner. Rather than a huge argument list per layer function, we will organize those arguments as suggested in the previous posts.
• Other than that, everything else will probably work the same.
• Backwards compatibility will be guaranteed.

Developers perspective:

• Adding new layer functions or aesthetics will become easier. E.g. fill patterns (#49), border lwd and lty (#51 and #56), gradient colors (#503), border color (#515).
• Writing functions to process layers (from data to visual variables) will become straightforward.
• The visualization part will be strictly separated and modular. There will be two visualization modules, one that uses grid graphics (the current plot mode), and one that uses leaflet (the current view mode). This separation makes it easier to extend tmap with new graphics engines (e.g. CesiumJS).

Any help is welcome:

• Currently, I'm the only one who knows about all complexity within tmap, but my aim is to make this knowledge explicit by writing proper code. Please feel free to join me.
• Also, suggestions on the user perspective are more than welcome. If you're struggling with certain elements (e.g. legends), please think about how it can be improved from a user perspective. Although backwards compatibility is important, user friendliness is at least as important in my opinion, especially on the long run.

Hi @mtennekes , thanks for tmap and this great upgrading initiative. I have been using tmap for 3 years now and I love it, its very intuitive and easy to learn for beginners - which is also its greatest strength. I just had a small suggestion to make regarding help wanted part - I am a beginner to open source coding and I have particularly found the help-wanted tags on issues of some of the other packages I follow very useful to get started. I noticed there was a publicity tag, perhaps a tag seeking contributions towards coding of this upgrade? Or may be there is a better way to go about it then I am aware, so I look forward to your thoughts on this. Thanks.

[mtennekes]

Thanks for the tip @KiranmayiV ! Definitely a good idea to use those tags. In the end, the most important thing is to make the package developers-friendly. I have had several feature requests in the past with the offer to help with the implementation, but the complexity of the code prevented actual co-coding (apart from small trivial bug fixes).

[mtennekes]

I have a specific question regarding facets for which I need your feedback from a user's perspective:

How should a user be able to map data dimensions to plotting dimensions?

Explanation of this quesion:

tmap can plot facets in maximal 3 plotting dimensions, namely rows x columns x pages:

1. (One row) or (one column) or (a 'facet wrap') or (an animation with one map).
2. (Multiple rows and multiple columns i.e. a 'facet grid') or (an animation of ((one row) or (one column) or a ('facet wrap')))
3. Multiple rows, multiple columns and multiple pages. Example:

data(World)
World$coin = sample(c("heads", "tails"), nrow(World), replace = TRUE)

tm_shape(World) +
    tm_polygons("HPI") +
    tm_facets(by = c("continent", "income_grp"), along = "coin")

There are two ways in which a user can create a data dimension, which I will refer to as data dimension types:

1. By specifying an aesthetic with two data variables, e.g. tm_shape(World) + tm_polygons(c("HPI", "life_exp"))
2. Or by using the by or along argument of tm_facets. E.g. tm_shape(World) + tm_polygons("HPI") + tm_facets(by = "continent"). Another example is the one I started with, which specified three dimensions.

Note that the current tmap 3.x implementation is restricted in the sense that mixing type 1 with the type 2/by is not allowed, e.g. tm_shape(World) + tm_polygons(c("HPI", "life_exp")) + tm_facets(by = "continent") does not work.

So the question is how should a user should be able to map the data dimensions to the plotting dimensions.

Possible answer

Instead of a by and along argument, the user should explicitly specify each plotting dimension. Examples:

tm_shape(World) +
    tm_polygons(c("HPI", "life_exp")) +
    tm_facets(rows = "VARS__", columns = "continent") 
# where "VARS__" is a special value that indicates data dimension type 1.

tm_shape(World) +
    tm_polygons("HPI") +
    tm_facets(rows = "continent", columns = "income_grp", pages = "coin")

Any other ideas how to make this as user-friendly as possible?
Pinging a sample of users: @KiranmayiV @prosoitos @cboettig @loreabad6 @elwan3 @Nowosad

[mtennekes]

Additional related question for those who like complexity:-)

How should a user specify whether aesthetic scales are free? Theoretically, we could specify a free.scales argument for each aesthetic for each plotting dimension. But this would be far too complicated I imagine. Currently, I'm thinking about one free.scales argument for each aesthetic, which is applied over all plotting dimensions. What do you think?

[Nowosad]

tm_shape(World) +
  tm_polygons("HPI") +
  tm_facets(rows = "continent", columns = "income_grp", pages = "coin")

Any other ideas how to make this as user-friendly as possible?

@mtennekes I am still wrapping my head around this idea. How could a user create "a facet wrap" using the above notation?

[mtennekes]

I think there are two options:

1. Add an argument wrap to tm_facets, which is the default " group-by" argument. If wrap is specified then rows and columns (and perhaps also pages) are ignored. Then we also need an nrow and ncol argument, which are ignored when rows and columns are used.

tm_shape(World) +
   tm_polygons("HPI") +
   tm_facets(wrap = "continent") # arguments rows and columns are NULL by default

1. Like ggplot2 create two faceting functions: tm_facets_wrap and tm_facets_grid. The first takes the arguments by, nrow and ncol, and the second takes the arguments rows, columns, and pages. In tm_facets_wrap, pages are generated when the number of facets is larger than nrow*ncol, like the current behavior in tmap 3.x.

tm_shape(World) +
   tm_polygons("HPI") +
   tm_facets_grid(rows = "continent", columns = "income_grp", pages = "coin")

and

tm_shape(World) +
   tm_polygons("HPI") +
   tm_facets_wrap(by = "continent")

The second option is probably user friendlier. What do you think?

[mtennekes]

The difficulty here is what happens when a user introduces a dimension of the first type (see earlier post), e.g. tm_polygons(c("HPI", "life_exp")) or tm_raster() where a multi-band stars object is specified. But as I think about it, it can be quite straight forward:

Without specification of tm_facets_, it will create facet wrap.

When tm_facets_grid is specified, it will map this dimension to the first unspecified plotting dimension.

Example 1:

tm_shape(World) +
    tm_polygons(c("HPI", "life_exp"))
    tm_facets_grid(columns = "income_grp")

This will map this dimension to the rows. So one row for HPI and one row for life_exp, and the columns according to income_grp.

Example 2:

tm_shape(World) +
    tm_polygons(c("HPI", "life_exp")) +
    tm_facets_grid(rows = "continent", columns = "income_grp")

This will map this dimension to the pages. So one page for HPI and one pagefor life_exp.

Example 3:

tm_shape(World) +
    tm_polygons(c("HPI", "life_exp")) +
    tm_facets_grid(rows = "continent", columns = "income_grp", pages = "coin")

This will throw an error, since there are no plotting dimensions left.

[Nowosad]

I think there are two options:

@mtennekes I like the first approach better.

[prosoitos]

Thank you for adding me to the conversation.

I don't have any strong opinion about the syntax. What I would like to add here however is that, if you are making a big revamp of the syntax, now would be a great time to add functionality.

For instance, I would love to be able to get rid of the legend title (often obvious or looking ugly if the variable has an ugly name). As a workaround, I make it as the same colour of the legend background, but when it is long, this creates a very wide legend box.

So I would love to see a legend.title.show argument accepting logical values.

I am sure that there are many similarly missing arguments and thinking deeply about all those little missing ones before releasing a big syntax change would be amazing.

Note that this is a comment about tm_layout and possibly off topic in this thread (if so, my apologies). I thought that I would still throw it out there, in case.

[mtennekes]

Thanks for your input @prosoitos

For me the direct purpose of v4 is not to add functionality, but to restructure the framework so that new features (e.g. settings, layers, plotting engines) can be added easier (and not just by me).

The legend titles are controlled in the layer functions, since each aesthetic can have its own legend. You can get rid of it by setting it to "", e.g.

tm_shape(World) +
    tm_polygons("HPI", title ="") +
    tm_layout(legend.frame = TRUE)

[prosoitos]

Ah! Wonderful! Thank you! I had not realized I could do that.

OK, no problem: do ignore my comment above then if it falls outside of your current objectives.

And thanks once more for the package!