Streetcomplete: Enlarge maximum.cache and cache step size

[PanderMusubi]

Use case
To map frequently in several places that are far apart needs bigger cache. Contemporary smartphone have 128 GB so storage is not an issue.

Proposed Solution
Change maximum cache setting from 250 MB to 512 MB and increase cache step size from 1 MB to 32MB to more easily set maximum cache size.

[westnordost]

Increased it to 50-1000 (default 250) with step-size of 50

[goldfndr]

The "minSdkVersion" for this app is 17 (Jellybean) which includes 8 GB (~5 GB usable) phones and possibly 4 GB phones.

Personally, I liked setting the cache size to 10 and found it plenty for my needs, when I had a smaller phone. But I'm curious: how are you refreshing over ten thousand tiles on a weekly or more frequent basis? Or are your tiles simply that much larger in your regions? (mine average 20kB for each tile pair: 5 kB control file and varying data file; 250 MB for me would be ~12500 tiles) Or am I overlooking something else?

[westnordost]

To be honest, I did not look at it from the angle of what a map cache should reasonably weigh in terms of size but simply let the user choose a higher max cache size if they want it.

The cache is a simple HTTP cache, nothing fancy (MBTiles).

A typical tile in the center of Hamburg at zoom level 18 is about 40kB, see
https://map-data.de/#map=18/53.553281520507774/9.991432428359987&style=cinnabar&show_server and open the network monitor (Ctrl+Shift+E in Firefox). I don't think tiles can get much larger than that.
What do you mean with "your" tiles? In your area?

Edit1: I have no idea how much area would be 12500 tiles

Edit2: The cache control on the server is max-age=604800 (7 days)

[PanderMusubi]

Thanks for making the improvements. When you travel a lot, it is handy to have larger cache. On the other hand, I think also the maximum cache age is the reason why it looked cache was not large enough.

[westnordost]

I think one week is reasonable, considering that the tiles get updated at most 2 times a month. What do you think, @Akasch ?

[PanderMusubi]

This info will help manage user expectation. Perhaps in Settings or About the following could be mentioned:

• maximum cache age
• tile update frequency

and perhaps:

• current cache size in use

This will help in the expectations users have regarding cache and tiles and how to tune their cache settings.

[westnordost]

I turned the max cache size to 500 (= x2 as before) and the default size to 150 (= x3 as before) again.

I will not introduce texts in the app that inform about the maximum cache age because all these need to be translated and at the same time is nothing that is in my control, but of the server. Same with the tile update frequency. The tiles should in the best case be updated every 2 weeks, but in many areas around the world, they aren't at all.

[PanderMusubi]

okay, thanks! people can find the details here if they are interested.

[goldfndr]

@westnordost Use of the Cinnabar rendering style is not a valid check, as Cinnabar has a LOT more detail than the StreetComplete rendering styles. In Hamburg I'm getting smaller tiles wirth StreetComplete instead of Cinnabar. But even with your example, 40 kB is merely twice the 20 kB I was seeing; 6000+ tiles (240 MB) is still a lot to refresh, even at 10 per second.

Because I still think the default and step remain absurdly high, I request of @PanderMusubi (and other thumbs uppers) the following:

1. Look in your StreetComplete tile cache at all files. (/sdcard/Android/data/de.westnordost.streetcomplete/cache/tile_cache/) Report the count and size. (The "journal" file is probably too minor to affect the results.)
2. Look in your StreetComplete tile cache for files less than a week old (e.g. delete older files or avoid selecting them or see below). Report the count and size.
3. Use your customary method to refresh tiles you'd anticipate using within a week of wherever you might go. It's okay to be fickle (e.g. different "what if" destinations within a "trip") but it all should be limited to the extents of what you can physically travel to within a week of arrival. Optionally clear the cache beforehand, especially if some of the anticipated tiles are recent.
4. Look in your StreetComplete tile cache for files less than a week old. Report the count and size.
5. Repeat steps 3 and 4 as desired.

Oh, also, if you're using a more efficient method of refreshing than the in-app pan and zoom I'm using, report that too. If you don't have an efficient method for counting and summing recent files, you can use the following with Termux or equivalent. Note that copying files to somewhere else to check might invalidate the date stamps.

cd /sdcard/Android/data/de.westnordost.streetcomplete/cache/tile_cache
find . -mtime -7 | wc -l
wc -c `find . -mtime -7` | tail -n 1
du -m `find . -mtime -7` | tail -n 1

[westnordost]

@westnordost Use of the Cinnabar rendering style is not a valid check, as Cinnabar has a LOT more detail than the StreetComplete rendering styles.

The styles are client-side. The detail you see in the Cinnabar style are nevertheless transmitted to the client.

I am not sure what happens if you have cached map data that is older than 7 days and you are offline. I'd think that then, those are used? (tangram-es uses simply OkHttp). In that case, also older cached data is useful.

[goldfndr]

Thank you for the clarification on the tiles, my mistake.

The final comment in #846 hasn't been implemented AFAIK, and stale cached tiles aren't usable; I conducted a simple test:

1. Clear tile cache.
2. Pan and zoom in the local area to fill the tile cache.
3. Turn off data. (Wi-fi, mobile data)
4. Exit StreetComplete and dismiss it from Recent Apps.
5. Launch StreetComplete.
6. Pan and zoom to confirm the tile cache is functional.
7. Exit StreetComplete and dismiss it from Recent Apps.
8. Change the system date to a month in the future. (or 14 days or whatever, a month is easy)
9. Pan and zoom to confirm the tile cache isn't used. Since the tiles aren't shown and quests might not be shown, the GNSS position and the ability (or not) to add a note at the current zoom can be used to determine whether stale cached tiles would otherwise be shown.
10. Exit StreetComplete and dismiss it from Recent Apps.
11. Change the system date back to the current date.
12. Pan and zoom to confirm the tile cache is functional.
13. Repeat steps 7-12 as desired.

[westnordost]

Ah, too bad. Thanks for testing.

So, based on your tests, what is the cache space that would be enough in any case to store visiting a few different places a week?

Am 27. Mai 2019 01:09:30 MESZ schrieb Richard Finegold [e-mail]:

Thank you for the clarification on the tiles, my mistake.

The final comment in #846 hasn't been implemented AFAIK, and stale
cached tiles aren't usable; I conducted a simple test:

1. Clear tile cache.
2. Pan and zoom in the local area to fill the tile cache.
3. Turn off data. (Wi-fi, mobile data)
4. Exit StreetComplete and dismiss it from Recent Apps.
5. Launch StreetComplete.
6. Pan and zoom to confirm the tile cache is functional.
7. Exit StreetComplete and dismiss it from Recent Apps.
8. Change the system date to a month in the future. (or 14 days or
   whatever, a month is easy)
9. Pan and zoom to confirm the tile cache isn't used. Since the tiles
   aren't shown and quests might not be shown, the GNSS position and the
   ability (or not) to add a note at the current zoom can be used to
   determine whether stale cached tiles would otherwise be shown.
10. Exit StreetComplete and dismiss it from Recent Apps.
11. Change the system date back to the current date.
12. Pan and zoom to confirm the tile cache is functional.
13. Repeat steps 7-12 as desired.

[goldfndr]

For a lower bound for my personal self, 10 MB is more than enough to visit a few different places in the local area. It took a couple of minutes of pan and zoom to get to that point. I generally get by with a maximum of 5 MB (used=3 or so) if I'm going to a specific location; I can only walk so far in a couple of hours! :grin: Admittedly, my local area is nowhere near as complete/dense as Hamburg (e.g. some missing buildings), as my travels are through suburbs.

I can't answer the "in any case" upper bound question, but I'm hoping the OP and the :+1: people could provide empirical data points (see my earlier "request" comment). (A back-of-the-envelope estimate for me would be 5 MB per 2 hours x 4 sets per day x 7 days = 140 MB.)

As a personal aside, a year ago or so, I tried a few ideas to trick the tile cache into giving me stale tiles that I'd downloaded just ~1.2 days prior. I tried modifying the files (date, cache-control, expires) and touched the file date stamps, without success; I suspect the journal file was the key but I couldn't crack it. 7 days is a significant improvement.

[westnordost]

@PanderMusubi , could you, would you?

[PanderMusubi]

What is the question?

[westnordost]

If you could (simulate your normal usage of the app of a week by panning and zooming around and) check how much space this uses, up, see https://github.com/westnordost/StreetComplete/issues/1398#issuecomment-496029284

[westnordost]

Eh well, I think I reduced it now to the values that @goldfndr suggested

[PanderMusubi]

Sorry, did not got around to test. Fine with how this is closed.

[goldfndr]

Eh well, I think I reduced it now to the values that @goldfndr suggested

Not that I can see. Currently, in master, it's a minimum of 50 with a step of 25. and I've been purposely avoiding going into Settings due to fear of triggering these values. I humbly request a minimum of 5 ("I generally get by with a maximum of 5 MB (used=3 or so)") and a step of 5 and restore the default to 50. The app itself occupies about 30 MB on my device.

@ltog and @MegaArthur, you :+1:'d the issue, can you report any anecdotal data?

[westnordost]

I really don't want to haggle over a few MBs. Is there any good reason why a difference of 50 MB or so makes a difference to you?

[goldfndr]

I still have strong doubts about the ability for anyone to experimentally determine that this change was appropriate and not just due to a misunderstanding. @PanderMusubi, I'd love to at least learn how long you've spent populating cache prior to a trip (perhaps it was more than an hour) and what the most optimal techniques including and other than manual zoom+pan are available that I haven't learned, as I similarly go without data enabled. @PanderMusubi, could you at least grant that? Please? My speculation is frustrating and distracting.

But for my own personal use case I will resign myself to clearing cache as needed, as the 50 MB is not pre-allocated. I'll just write a shell script to take care of it.

If you'd asked me this question last year, I would've instantly said yes, as my KitKat phone was near full all the time, and I was using my rooted ability and scripting to remove inaccessible files (particularly /data/data/pkg.app.tld/app_webview/*, as they were considered app data not app cache) for dozens of (>50) apps daily. It was so full that I had taken to storing files (e.g. OsmAnd maps) in /cache (where OTA updates would go) and had given up on listening to podcasts. For the rest of 2019, my personal answer should stay as "not as much as it did".

[PanderMusubi]

Over time, more information how the cache works came forward in this issue. That also changed my perspective of the situation. Hence, my question on offering some cache info to the user in the app.

The step size of maximum cache was indeed too small. That has improved. The minimum cache size I don't really care about but understand other users find that very important. Frequently, I would add quests to places I visited just before. After I get out of the train or car. And these can be 50 to 80 km away. These were often the same places.

What I noticed is that loading map info was extremely slow (very slow, even app would sometimes hang loading underlying map), and concluded that that was because of too small cache. But since that is not entirely the case, I leave the further optimization of cache settings to you.

I can only say more useful things if I would have insight in cache behaviour while using the app. That is nog something for the normal user. Perhaps worth considering implementing and only enabling in a custom release for fine-tuning these matters.

[HolgerJeromin]

Last days I was able to load all quests of a region without too much effort.
So the current settings work for me

[goldfndr]

@HolgerJeromin That's like me saying that I was able to store a week's worth of cat food for my cat in a 55 gallon drum without too much effort, so a 55 gallon drum works for me. If you don't measure the space used, you don't know how excessive the container might be.