Cgeo: [Nightly] cgeo aborts when opening map

[geocachermgo]

Describe the bug:
cgeo aborts when opening map.

To Reproduce:
Steps to reproduce the behavior:
Open Live Map

Version of c:geo used:
2020.08.24-NB-f5f21f0

Is the problem reproducible:
Yes
I changed the map from google map to offline, that did not change something.
If I try to open the map display of a list, the same happens.

Screenshots:

System information:
--- System information --- Device: MI 8 (dipper, Xiaomi) Android version: 10 Android build: QKQ1.190828.002 test-keys c:geo version: 2020.08.24-NB-f5f21f0 Google Play services: disabled - 20.26.14 (120400-320008519) Low power mode: active Compass capabilities: yes Rotation vector sensor: present Orientation sensor: present Magnetometer & Accelerometer sensor: present Direction sensor used: rotation vector Hide caches: own/found Hide waypoints: - HW acceleration: enabled (default state) System language: de_DE System date format: dd.MM.yy Debug mode active: no System internal c:geo dir: /data/user/0/cgeo.geocaching (90,0 GB free) internal User storage c:geo dir: /storage/emulated/0/cgeo (90,0 GB free) external non-removable Geocache data: /storage/emulated/0/Android/data/cgeo.geocaching/files/GeocacheData (90,0 GB free) external non-removable Database: /data/user/0/cgeo.geocaching/databases/data (10,8 MB) on system internal storage Fine location permission: granted Write external storage permission: granted Geocaching sites enabled: geocaching.com: Logged in (Anmeldung OK) / PREMIUM Geocaching.com date format: MM/dd/yyyy Installed c:geo plugins: none BRouter connection available: false --- End of system information ---

Additional context:
e.g. Reference to other issues, projects, sources, etc.

[geocachermgo]

I installed the normal play store release (2020.08.04) and when I imported the settings: cgeo won't even open but abort when I try to open it.

[geocachermgo]

I removed the password from the settings file. Is it then save to upload it here?

[Lineflyer]

The setting might be incompatible between play and nightly...not sure.

[Lineflyer]

Not reproducible on my device with mentioned nightly version.

@geocachermgo Can you send a logfile via c:geo Settings - System

[geocachermgo]

Its not only the settings: if I restore only the backup, cgeo also aborts immeadiately.

[geocachermgo]

Is there a repository of old nightlys (e.g. yesterday)?

[geocachermgo]

@geocachermgo Can you send a logfile via c:geo Settings - System

logcat_2020-08-24_05-08.txt
That is before restoring. After restoring the app is not opening anymore.
I see now, that there is crash information included.

[moving-bits]

I installed the normal play store release (2020.08.04) and when I imported the settings: cgeo won't even open but abort when I try to open it.

When you're coming from a nightly version you won't be able to run release version with it, as there have been database changes in between.

To switch from a recent nightly to the current release version you'll need some extra steps:

• make sure you have a backup of your database and settings created and copied to a safe place
• uninstall c:geo
• place an old database backup into cgeo/backup directory - the database has to have a version less or equal to the one supported by the release version
• install release version and restore settings & database

[moving-bits]

The setting might be incompatible between play and nightly...not sure.

Breaking changes in settings are rare, and extra settings simply get ignored, so should be safe in this case.

[moving-bits]

@geocachermgo Can you send a logfile via c:geo Settings - System

logcat_2020-08-24_05-08.txt
That is before restoring. After restoring the app is not opening anymore.
I see now, that there is crash information included.

@UniQP
From the errorlog: The crash is somewhere in the RxJava code - do you have time to have a look at it?

[bekuno]

@geocachermgo
Which version did you ran before?

[bekuno]

@moving-bits Maybe there is an problem in f38728cb81d3116c18a56115309a307daab00452

[bekuno]

@geocachermgo
There is a problem shown External public cgeo directory '/storage/emulated/0/cgeo' not available
Does this folder exist?

[geocachermgo]

@geocachermgo
Which version did you ran before?

The nightly from last night:2020.08.24-NB-f5f21f0

@geocachermgo
There is a problem shown External public cgeo directory '/storage/emulated/0/cgeo' not available
Does this folder exist?

The folder cgeo exists with subdirectories "backup" "gpx" and "logfiles"

[geocachermgo]

I'm able to restore the settings into the release version. When I import the database cgeo crashes. (Up to now I tried to import the database first).

[geocachermgo]

Behaviour is unchanged with new nightly 2020.08.25-NB-cd3a8cc.
I can import the backup and it aborts when I try to open the map.
In the backup directory there is a file "cgeo.sqlite" with 12MB and a "data.corrupted" with the same size and date.

[Lineflyer]

@samueltardieu

As we seem to have some gap with rxjava knowledge, could you maybe take a look at this and at #8402 ?

[samueltardieu]

I am away for some weeks with no computer around.

Le 25 août 2020 08:34:44 Lars [e-mail] a écrit :

@samueltardieu

As we seem to have some gap with rxjava knowledge, could you maybe take a
look at this and at #8402 ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Lineflyer]

Enjoy! :)

[bekuno]

In the backup directory there is a file "cgeo.sqlite" with 12MB and a "data.corrupted" with the same size and date.

Maybe your database backup is corrupted?

see also https://github.com/cgeo/cgeo/issues/7301#issuecomment-679898834

[geocachermgo]

I tried the old (22.8. and 23.8.) apks. A restore leads to immeadiate abort with all apks, exept the one from 24.8., which only aborts when opening the map.

[geocachermgo]

I plan to "rebuild" my database by exporting the lists to gpx (with the apk from 24.8.) and read then in in a new database. If the root cause is a corrupted (however that occurred) database and not reproducable by someone else we so not need to proceed here. Shall I upload the corrupt database here?

[Lineflyer]

Please also try if on one of the APKs the map is functional without involving any restore of settings or DB.

It is not clear to me whether the DB or settings state is causing it or the APK for itself.

[geocachermgo]

If I dont restore anything all of the four (22.8.-25.8) apks works fine with default (=google) map.

[moving-bits]

If I dont restore anything all of the four (22.8.-25.8) apks works fine with default (=google) map.

only with Google Map, or with OpenStreetMap as well?

[Lineflyer]

A user on support mail with similar problem:

Since updating to 2020.08.26-NB the map cannot be opened (crash).
I have asked the user if he remembers the version upgraded from.

Did we recently change anything in the database between approx. 2020.08.20-NB and 2020.08.24-NB.
I am somehow pretty sure, that this might cause it as the DB gets corrupted.

Users system information:

--- System information ---
Device: SM-G950F (dreamltexx, samsung)
Android version: 9
Android build: PPR1.180610.011.G950FXXU9DTF1
c:geo version: 2020.08.26-NB-c1ef9e4
Google Play services: enabled - 20.26.14 (100408-320008519)
Low power mode: inactive
Compass capabilities: yes
Rotation vector sensor: present
Orientation sensor: present
Magnetometer & Accelerometer sensor: present
Direction sensor used: rotation vector
Hide caches: own/found
Hide waypoints: original visited
HW acceleration: enabled (default state)
System language: de_DE
System date format: dd.MM.yy
Debug mode active: no
System internal c:geo dir: /data/user/0/cgeo.geocaching (11,4 GB free) internal
User storage c:geo dir: /storage/emulated/0/cgeo (11,4 GB free) external non-removable
Geocache data: /storage/emulated/0/Android/data/cgeo.geocaching/files/GeocacheData (11,4 GB free) external non-removable
Database: /data/user/0/cgeo.geocaching/databases/data (384,7 MB) on system internal storage
Fine location permission: granted
Write external storage permission: granted
Geocaching sites enabled:
geocaching.com: Logged in (Anmeldung OK) / PREMIUM
Geocaching.com date format: dd/MM/yyyy
Installed c:geo plugins: contacts
BRouter connection available: true
--- End of system information ---

[moving-bits]

Did we recently change anything in the database between approx. 2020.08.20-NB and 2020.08.24-NB.
I am somehow pretty sure, that this might cause it as the DB gets corrupted.

Last DB change came with https://github.com/cgeo/cgeo/pull/8864 for redefinition of route table and migration of old individual route data:

db.execSQL("ALTER TABLE " + dbTableRoute + " RENAME TO temp_route");
db.execSQL(dbCreateRoute);
// migrate existing caches in individual route
db.execSQL("INSERT INTO " + dbTableRoute + " (precedence, type, id, latitude, longitude)"
  + " SELECT precedence, " + RouteItem.RouteItemType.GEOCACHE.ordinal() + " type, geocode id, latitude, longitude"
  + " FROM temp_route LEFT JOIN " + dbTableCaches + " USING (geocode)"
  + " WHERE temp_route.type=1");
// migrate existing waypoints in individual route
db.execSQL("INSERT INTO " + dbTableRoute + " (precedence, type, id, latitude, longitude)"
  + " SELECT precedence, " + RouteItem.RouteItemType.WAYPOINT.ordinal() + " type, " + dbTableWaypoints + ".geocode || \"-\" || PREFIX id, latitude, longitude"
  + " FROM temp_route LEFT JOIN " + dbTableWaypoints + " ON (temp_route.id = " + dbTableWaypoints + "._id)"
  + " WHERE temp_route.type=0");
// drop temp table
db.execSQL("DROP TABLE IF EXISTS temp_route");

Pretty much standard SQL, but maybe there's something SQLite version specific?
(Although Android 9 is a relatively recent version, should have a recent version of SQLite included.)

Also, if that migration is the source, then there should be an error message ("Cannot upgrade to ver. 84") in the logfile.

[Lineflyer]

Wasnt there also this downgrade thing, or is this not yet merged?

[geocachermgo]

If I dont restore anything all of the four (22.8.-25.8) apks works fine with default (=google) map.

only with Google Map, or with OpenStreetMap as well?

I tested only with the default map. The crash with opening the map (nightly 24.8.) happend with online and offline OSM-map. Shall I test something else?

[Lineflyer]

Shall I test something else?

If you dont use the map but only browser the lists/caches, which you have saved. Any crashes/problems?

[geocachermgo]

Shall I test something else?

If you dont use the map but only browser the lists/caches, which you have saved. Any crashes/problems?

Meanwhile I have the new nightly, exported all lists to gpx and imported them back again, and everything works fine and I made a new backup. For testing purposes I could go back (which APK?, I have stored 22.-24.), import the settings (first?) and the corrupt database (?). If I do all of this, all apk crash at opening of cgeo except the 24 where I can browse, export gpx (thankfully) and the crash happens only if I try to use the map (live map or list on map, (single cache on map not tested)).

[geocachermgo]

If we like to investigate this further my suggestion would be 1) is the same happening if somebody else imports that database? 2) Analysis of the database, can we determine whats wrong? 3) Is a corrupt database "allowed" to kill cgeo? Analysis of why it was written as a corrupt database/why the specific corruption is able to kill it when opening cgeo/when opening the map.

[eddiemuc]

Wasnt there also this downgrade thing, or is this not yet merged?

The downgrade thing does not modify the database in any way. As long as latest nightlys are used, DB version should be 84 and no downgrade should occur. In any case DB upgrade/downgrade errors should show up in the log.

One thing that could be investigated: if there is some error in the upgrade or downgrade, then maybe @geocachermgo s local installation still has DB version 83 but is marked as DB version 84. Could such a constellation cause the problem?

(Sorry I currently have no laptop around...)

[eddiemuc]

Also, if that migration is the source, then there should be an error message ("Cannot upgrade to ver. 84") in the logfile.

@moving-bits
The tricky thing about current upgrade routine is that each version upgrade is doing its own try-catch and logs a failure. That means, unless Log.e causes an exception itself, when a version upgrade fails this is logged but the upgrade is still marked as "done" in the DB (because onupgrade returns correctly).
Problem is: if user upgrades e.g. from 82 to 84 and this upgrade fails, then this is only logged the first time and DB is afterwards marked as version 84 despite the failure (and while in reality the DB is something in between). Subsequent starts of c:geo will then no longer try to do an upgrade or log a failure.

If, for some reason, one nightly or APK installed by @geocachermgo left her DB in such a situation, there's no way to come out of it. Analysis of the concrete DB structure and DB version of @geocachermgo s DB should reveil if this is the case.

[geocachermgo]

How do I determine the DB version? It seems not to be in the "--system information--". I deinstalled several times via Play-Store and installed the apk. Does that preserve the DB update fail?

[eddiemuc]

How do I determine the DB version? It seems not to be in the "--system information--". I deinstalled several times via Play-Store and installed the apk. Does that preserve the DB update fail?

@geocachermgo
Afaik not possible directly from c:geo. You can perform following steps:

• Install a sqlite manager took. I personally use "asqlite manager" from app store
• Open c:geo DB with this tool
• Execute query ("Abfrage"): pragma user_version -> this returns the DB version
• Execute query: select * from sqlite_master -> this returns complete DB structure

Both infos would be very helpful, if that's not too much to ask?

[eddiemuc]

..forgot to mention: I think you can only open backup DBS of c:geo with this, so you have to make such a backup first.

Below a screenshot of the second query as executed on my current installed c:geo version (the one from play store)

[moving-bits]

@geocachermgo
Do you have a chance to provide us with your database (on a separate channel, maybe dropbox-link via PM)?
Maybe even a backup from before the crashing nightly, and the corrupted version?
Then I could look into it, could try to analyze why the update has been failing (if so), and maybe even repair it.

[moving-bits]

The tricky thing about current upgrade routine is that each version upgrade is doing its own try-catch and logs a failure. That means, unless Log.e causes an exception itself

I've a small PR in work to group the "after update step" work into a separate method, and while looking at the source, also stumbled upon that problem. Log.e() does a rethrow, but only in debug mode.

Thus I propose to add a rethrow to that tiny new method I'm preparing.
This would bring a database upgrade to an immediate halt on error, without trying to do the upcoming update steps (if more than one involved).

[moving-bits]

@geocachermgo
Additional question: Can it be, that you have skipped a couple of nightlies before upgrading to the version that lead to the crash? (Nothing wrong with that, actually! But there may be an error in the update statements involved, that do not take that situation into consideration.)

[geocachermgo]

How do I determine the DB version? It seems not to be in the "--system information--". I deinstalled several times via Play-Store and installed the apk. Does that preserve the DB update fail?

@geocachermgo
Afaik not possible directly from c:geo. You can perform following steps:

* Install a sqlite manager took. I personally use "asqlite manager" from app store

* Open c:geo DB with this tool

* Execute query ("Abfrage"): `pragma user_version` -> this returns the DB version

* Execute query: `select * from sqlite_master` -> this returns complete DB structure

Both infos would be very helpful, if that's not too much to ask?

Since I'm not able to help by programming I try to help by testing and reporting. So feel free to answer for more (or less) information.
I imported the 24. DB that leads to abort (directy or with the 24 apk at opening map).
Version is 84

The current backupfile is also version 84

"database.corrupted" (same length als 24, identical?) give also v84 and the following picture (is there a way to transfer the output to the clipboard?)

[geocachermgo]

@geocachermgo
Additional question: Can it be, that you have skipped a couple of nightlies before upgrading to the version that lead to the crash?

No, if there werent several per night. Since I was caching from 13.-22. every day and I update whenever I get the "new nightly" message displays

[eddiemuc]

Since I'm not able to help by programming I try to help by testing and reporting. So feel free to answer for more (or less) information.

Hi @geocachermgo ,

if you choose to send your complete DB via PM to @moving-bits , then he should be able all the info he needs from it himself.

Only if you do not want to do that, I would ask the following from you using asqlite manager:

• Go to tables ("Tabellen") and choose sqlite_master there
• Using topright-three dots, choose "export to"
• Export to CSV, all columns selected and including headers (see screenshot).
• Paste the resulting CSV file here

[geocachermgo]

I can send the DB to moving-bits despite the risk of revealing some mystery solutions. :-) How do I send a PM in Github or how do I get his/her adress?

[moving-bits]

I can send the DB to moving-bits despite the risk of revealing some mystery solutions. :-)

I'll keep my eyes closed :-)

How do I send a PM in Github or how do I get his/her adress?

You can reach me per e-mail using xxx
If the compressed files are too large for an e-mail you may use some online storage (eg: dropbox), or I can provide you with a temporary ftp access.

[geocachermgo]

I sent a check e-mail to the assumed adress. (Space between @ and TLD filled up with name).
15:50 Sent DB, recieved.

[moving-bits]

Thanks for sending the DB. Can reproduce the exception on opening the map. A quick analysis shows that there is an invalid entry in the cg_route table, consisting only of null name and null longitude/latitude, which crashes c:geo on loading the "route".

Further investigation needs to be done by me:

• Does the migration routine lead to this invalid entry / and how to avoid it?
• Avoid crashing of c:geo on loading a route containing such an invalid route item.

[geocachermgo]

Thanks for the analysis. Why are the 22. to 23. crashing immediately and the 24. on opening the map?

[moving-bits]

Did the 22/23 versions crashed from the day you have installed it, or only after you had installed the 24 version?

If the latter, then it might be the database upgrade: After an update to the database you currently cannot downgrade to a version which expects to be a database with a lower internal version number.

[geocachermgo]

The first crashes occured with the 24 version. (Thats why I named the issue "when opening maps".) So the apk <24.8. expect a database <84 and are crashing due to the wrong DB version?
I would even include this in the description of the backup page where it says: ... but not pictures ans settings."Backups might not be usable with earlier version"

[Lineflyer]

We are talking about alpha versions here. We wont / should not spend any effort in explaining things or implementing fallbacks for such cases.@geocachermgo Nothing against you, just saying that there are certain limitations and things you need to know when you use developer snapshots.Von Samsung-Tablet gesendet

[MagpieFourtyTwo]

@geocachermgo I went through the same learning process and since then have the foll. procedure, when the New version available message appears:

1. backup database and settings (full)
2. rename directory ...\cgeo\backup to ...\cgeo\yymmdd hhmm backup (and sometimes move it to external sd)
3. put currently installed (i. e. matching) cgeo-nightly.apk into this directory
4. download and install the the new nightly (and leave the new cgeo-nightly.apk in the download folder)

I keep about three to five versions of the ...\cgeo\yymmdd hhmm backup directory.
This way I can always go back a few versions, no matter if db or settings have changed significantly.
And although I had to go back this way just once till now, as most of the times it works with just the old apk,
but better to be safe than sorry. ;)

[fm-sys]

We are talking about alpha versions here. We wont / should not spend any effort in explaining things or implementing fallbacks for such cases.@geocachermgo Nothing against you, just saying that there are certain limitations and things you need to know when you use developer snapshots.Von Samsung-Tablet gesendet

This could also happen, when a normal user tries to import a backup on a second phone which still uses an old release. I think there's nothing wrong with adding such a message...

[geocachermgo]

We are talking about alpha versions here. We wont / should not spend any effort in explaining things or implementing fallbacks for such cases.@geocachermgo Nothing against you, just saying that there are certain limitations and things you need to know when you use developer snapshots.Von Samsung-Tablet gesendet

My assumption (may be not correct) was that the 24. Apk.was crashing (at opening maps) due to usage of a corrupt database caused by alpha versions. => No consequences neccessary. But that the crashing of the 22. and 23. (on opening cgeo) was caused by an old DB version and that would also happen with release versions, whenever the DB version changes. If this is the case, I think it should be mentioned. I could understand that with one installation its safe to say that going back in cgeo version is something that almost never happens. But someone could try to use the backup to transfer all stored caches to another device with an older cgeo version which will not work if the DB versions has changed (and my assumptions are true). You could say that the backup is made for backup and not for transfer (like gpx). That is a valid argument but I would still favour to mention it.
@MagpieFourtyTwo
Thanks for the hints, but I think I'm too lazy to do this. My "main" DB is GSAK and has a proper backup. So I fine with loosing all data of cgeo on the phone occasionally. I even expected that the deinstallation will clean up the cgeo dir with the sub-dir backup.
So I was reporting here not to get my data back, but to help finding bugs.

[geocachermgo]

I have to correct myself: cgeo should never crash even with an newer DB version. There should be a message saying "newer DB version in backup, can not read backup, please update cgeo version".

[geocachermgo]

@fm-sys thanks for the short and precise summary of my verbose post, even before I submitted it :-)

[moving-bits]

Reminder to myself:

• Does the migration routine lead to this invalid entry / and how to avoid it?

Check for geocache route items that do not exist any longer during migration. (Use geocode from old routeitem instead of geocache table.)

• Avoid crashing of c:geo on loading a route containing such an invalid route item.

Adapt SQL to ignore items having name, latitude and longitude all set to null.
Check RouteItem constructor to fail gracely when name empty or null.

[moving-bits]

I have to correct myself: cgeo should never crash even with an newer DB version. There should be a message saying "newer DB version in backup, can not read backup, please update cgeo version".

I've created a new issue for this, otherwise this request might get lost in this thread.

[MagpieFourtyTwo]

Use geocode from old routeitem instead of geocache table.

Don't know exactly how you mean this, but to prevent an error we had a few days ago (and which disappeared by upgrading to next nightly): GC-Code alone may not be enough to identify a route waypoint, as a route may consist of nothing else than just waypoints of the very same cache.

Adding just waypoints of one cache always worked, but last week we shortly had the situation that adding a second waypoint of the same cache did nothing than just remove the first waypoint ...

May not be related, but perhaps you may have it in mind. ;)

[Lineflyer]

We do have more users on support, who use our nightlies and have problems opening the map.

What was the last conclusion about this problem, AFAICS a database problem concerning the individual route?
Is there any advice I can give to the users to restore functionality?

This could also happen, when a normal user tries to import a backup on a second phone which still uses an old release. I think there's nothing wrong with adding such a message...

My reply was just to mention, that such things might happen on alpha versions.
Of course we should inform users properly in case a migration fails or we detect that the DB version is higher than supported (if user tries a new DB on an older installation).

[moving-bits]

What was the last conclusion about this problem, AFAICS a database problem concerning the individual route?

I need to prepapre a fix to make c:geo more robust in that regard. Will try to do so tomorrow.

[moving-bits]

Reminder to myself:

• Does the migration routine lead to this invalid entry / and how to avoid it?

Check for geocache route items that do not exist any longer during migration. (Use geocode from old routeitem instead of geocache table.)

Root cause seems to be a waypoint referenced in an individual route, but no (longer) present in the database during migration. At least I could reproduce the faulty item creation using that way.
Non-existing geocache works flawlessly, though.

• Avoid crashing of c:geo on loading a route containing such an invalid route item.

Adapt SQL to ignore items having name, latitude and longitude all set to null.
Check RouteItem constructor to fail gracely when name empty or null.

This second part is fixed with #8900.

[moving-bits]

Fixed the several aspects with #8900, #8905 and #8906.

[Lineflyer]

Fixed the several aspects with #8900, #8905 and #8906.

Regarding the users on support mail:
Should it get back working if those users update from 2020.08.24 to tomorrows NB?

[moving-bits]

Fixed the several aspects with #8900, #8905 and #8906.

Regarding the users on support mail:
Should it get back working if those users update from 2020.08.24 to tomorrows NB?

Yes, that came with #8900. #8905 avoids creation of such invalid entries during migration.

[geocachermgo]

@eddiemuc : old (debug)apks are available for the continous builds: I guess that the countinous build should be identlical to the following nightly.
https://ci.cgeo.org/job/cgeo%20Continuous%20integration/