Stars: default for proxy in read_stars()

[edzer]

Right now, users have to set proxy=TRUE to get a stars proxy object. This will lead to many cases where a massive file is first attempted to be read in memory, with unpredictable outcome but typically of the kind of

Error: cannot allocate vector of size 110.6 Gb

which is not very helpful. I've now changed this into having proxy depend on the size of the image:

• if it has more than 1e8 cells (rows * cols * bands) then proxy is by default TRUE
• 1e8 cells is roughly 800 Mb (8 bytes per cell)
• this value 1e8 can be overridden by setting options(stars.n_proxy=1e10) or a different value

This has the side effects that st_mosaic, st_warp (when through GDAL) and st_rasterize, which use the default for read_stars, will no longer error on huge files; with the new setting, they will return a proxy object on these.

Questions:

• do you consider this an improvement, or should we stick to the proxy = TRUE to be the only way to create proxy objects?
• should the default value 1e8 be chosen differently?

[edzer]

@michaeldorman ?

[michaeldorman]

@edzer Thanks! It's a very interesting idea.

Not sure I understood the details: Will the user still be able to set proxy=TRUE or proxy=FALSE to override the automatic choice, which will be determined based on the stars.n_proxy option? Or will the proxy parameter have no effect any more, so that the only way to control the object type is though modifying stars.n_proxy?

[edzer]

Will the user still be able to set proxy=TRUE or proxy=FALSE to override the automatic choice

Yes.options(stars.n_proxy) can be set to (globally) change the default if proxy is not set.

[michaeldorman]

I understand, thanks for clarifying! Then I think it's a good addition, because that way users who are unaware of the issue will avoid attempting to read a very large raster and running into memory limitations, while users who want to decide which type of object to work with (regardless of file size) will be able to specify the proxy parameter.

IMHO it's important that the user should have full control through the proxy parameter (or any other way, for that matter) because the workflow for proxy and non-proxy objects is quite different. For instance, calculating the average value of the 1st attribute with mean(r[[1]]) will only work on a non-proxy stars object. I'll probably use proxy=FALSE parameter at all times, since usually I try to be aware of the RAM limitations, and in worst case scenario am prepared for the cannot allocate vector error :-)

By the way, you mentioned the advantageous side-effect of the proposed behaviour in that st_mosaic, st_warp and st_rasterize---when resulting in a huge file---will not fail to read it back into R since when the file size is above the threshold then proxy=TRUE will be used. Perhaps it will be useful for consistency for these functions to also accept a proxy argument, in case one wants to override the defaults (i.e., not have it determined based on file size).

I'm using stars on a daily basis by now, so thanks again for this great package!

[edzer]

By the way, you mentioned the advantageous side-effect of the proposed behaviour in that st_mosaic, st_warp and st_rasterize---when resulting in a huge file---will not fail to read it back into R since when the file size is above the threshold then proxy=TRUE will be used. Perhaps it will be useful for consistency for these functions to also accept a proxy argument, in case one wants to override the defaults (i.e., not have it determined based on file size).

I agree, and if this were trivial I had done it probably already. Feel free to raise the issue!

[mdsumner]

Only thing to maybe add is for multiple source inputs, though hard to calculate potentially

[edzer]

This happens by using prod(dim(read_stars(x, ..., proxy = TRUE)))