Tmap: bulkiness of tmap: dependencies and memory usage

[mredaelli]

Unless I am mistaken, simply doing

library(tmap)

allocates 300 MB of RAM.

We are running a pool of Shiny applications using tmap, and this behaviour (plus the fact that the library is not loaded in shared memory, but duplicated for each process) really clogs us up.

Would it be possible to make a more "judicious" use of memory?

[mredaelli]

Alternatively, I would be happy to just load tmap when strictly needed, and unload it afterwards to free memory.
However, that doesn't seem to work (is this the way to do it?):

Packrat mode on. Using library in directory:
- "~/src/proj/ems/ems/packrat/lib"
> gc()
         used (Mb) gc trigger (Mb) limit (Mb) max used (Mb)
Ncells 309196 16.6     654011   35         NA   654011   35
Vcells 622274  4.8    8388608   64      16384  1957831   15
> library(tmap)
> gc()
          used  (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
Ncells 2959222 158.1    5312251 283.8         NA  4946520 264.2
Vcells 4595134  35.1   10146329  77.5      16384  8388608  64.0
> detach("package:tmap", unload=TRUE, force=TRUE)
> gc()
          used  (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
Ncells 2960717 158.2    5312251 283.8         NA  4946520 264.2
Vcells 4598885  35.1   10146329  77.5      16384  8388608  64.0
> 

(I'm not sure why, but what I see in the operating system as memory usage matches the max used column rather than the used. I'm on MacOS, if it helps.)

I have a feeling this is in good part due to packages you import, at least raster and mapview, although even after all the Required packages in DESCRIPTION are loaded, loading tmap still adds a good chunk.

[mtennekes]

Good point. tmap (with tmaptools) itself is only 4 MB. Not sure if the same memory size will be used.

My aim is to make tmap as lightweight as possible. Recently, I've have done some analysis of how many dependencies (including imports) there are recursively. There are 83 of them. However, it is a whole network of dependencies. Dropping one dependencies will not decrease the number of remaining dependencies drastically (if at all), since many indirect dependencies are used by more than direct dependency.

I used this script for it:

library(miniCRAN)
pkgs <- utils::available.packages()

x <- tools::package_dependencies(c("tmap", "tmaptools"), recursive = FALSE)
x <- intersect(setdiff(unlist(x), "tmaptools"), pkgs[,1])

y <- lapply(x, pkgDep, availPkgs = pkgs, suggests = FALSE)
y <- lapply(y, as.vector)

# dependencies (including imports)
unique(unlist(y))
#>  [1] "sf"           "Rcpp"         "DBI"          "units"       
#>  [5] "classInt"     "magrittr"     "spData"       "e1071"       
#>  [9] "class"        "MASS"         "raster"       "sp"          
#> [13] "lattice"      "rgdal"        "rgeos"        "RColorBrewer"
#> [17] "viridisLite"  "htmltools"    "digest"       "htmlwidgets" 
#> [21] "jsonlite"     "yaml"         "leaflet"      "base64enc"   
#> [25] "crosstalk"    "markdown"     "png"          "scales"      
#> [29] "viridis"      "lazyeval"     "R6"           "shiny"       
#> [33] "ggplot2"      "mime"         "dichromat"    "plyr"        
#> [37] "munsell"      "labeling"     "gridExtra"    "gtable"      
#> [41] "mgcv"         "reshape2"     "rlang"        "tibble"      
#> [45] "withr"        "colorspace"   "httpuv"       "xtable"      
#> [49] "sourcetools"  "later"        "promises"     "crayon"      
#> [53] "BH"           "nlme"         "Matrix"       "stringr"     
#> [57] "cli"          "pillar"       "assertthat"   "fansi"       
#> [61] "utf8"         "glue"         "stringi"      "mapview"     
#> [65] "satellite"    "brew"         "gdalUtils"    "webshot"     
#> [69] "svglite"      "uuid"         "foreach"      "R.utils"     
#> [73] "gdtools"      "processx"     "codetools"    "iterators"   
#> [77] "testthat"     "R.oo"         "R.methodsS3"  "praise"      
#> [81] "lwgeom"       "KernSmooth"   "XML"

z <- lapply(1:length(y), function(i) {
    unique(unlist(y[-i]))
})
names(z) <- x

# dependencies remaining when dropping one dependency
sapply(z, length)
#>           sf        units       raster        rgdal        rgeos 
#>           83           83           83           83           82 
#> RColorBrewer  viridisLite     classInt    htmltools  htmlwidgets 
#>           83           83           83           83           83 
#>      leaflet      mapview           sp       lwgeom   KernSmooth 
#>           83           66           83           82           82 
#>     magrittr    dichromat          XML 
#>           83           83           82

Created on 2018-08-08 by the reprex package (v0.2.0).

So only if mapview is dropped, the number of dependencies decreases quite a bit.

I don't how how the memory allocation works for packages.

[mredaelli]

Yeah, it looks like a dependency nightmare.

I will try investigating a bit the package unloading part, but my feeling would be that probably the right thing to do in my case is to move the code that needs tmap to a separate service, and call that from all the shiny instances. Not sure how that's doable in R, though.

But thanks for the investigation!

I also tried

> library(mapview)
> detach("package:mapview", unload=TRUE)

to see if this could be done inside in tmap, but again no difference in memory before and after.
I will try to ask them, too.

[tim-salabim]

AFAIK, tmap only requires mapview for sync, so maybe move it from Imports to Suggests and provide a proper warning when someone is trying to view interactive facets but doesn't have mapview installed. This should avoid much of the mapview overhead.

[mredaelli]

That would be already great.
However, I tried earlier to load mapview first, and then tmap, and the memory usage goes from 150MB to 300MB, so that's not the only part of the issue :/

[mtennekes]

It's correct that tmap only uses sync (actually latticeView) from mapview, but it really needs it. In view mode, sync is currently on by default when maps have the same bounding box.

A dirty trick could be to copy-paste latticeView to tmap, but I'm not sure if that's such a good idea...

[tim-salabim]

What I meant is that you could fall back to static mode when mapview is not available and provide a warning to install mapview for interactive facets.
Having the same code in two places is never a good idea IMO ...

[mtennekes]

Not really ideal IMO.

I am not sure how the dependency tree of mapview looks like and which functions depend of which packages, but isn't it an idea to start a new package with lightweight extensions to leaflet, such as facets?

[tim-salabim]

mapview naturally has hefty dependencies as it provides methods for the most common spatial classes. In the next CRAN release stars is added to that tree... maybe it is time to think about outsourcing the leaflet extensions to a lighter weight package. I am meeting with the r-spatial crew at Geostat 2018 in Prague in ten days or so and will discuss this.

[tim-salabim]

Ping @edzer

[edzer]

stars is light-weight and pure-R; all that's heavy (gdal, proj) was moved to sf - no compiled code.

[Nowosad]

The recent changes make a difference. Now:

gc()
#>          used (Mb) gc trigger (Mb) max used (Mb)
#> Ncells 477453 25.5    1034055 55.3   660191 35.3
#> Vcells 886107  6.8    8388608 64.0  1789408 13.7
library(tmap)
gc()
#>           used (Mb) gc trigger  (Mb) max used  (Mb)
#> Ncells 1518014 81.1    3009801 160.8  1972687 105.4
#> Vcells 2236160 17.1    8388608  64.0  3694550  28.2

^{Created on 2020-04-26 by the reprex package (v0.3.0)}