Streetcomplete: App is stuck with old quest

[iamnunocaldeira]

So last week a factory reset my smartphone (im a beta user of the app). connected my osm, scanned for new quests and to my surprise a lot of my previous answer were still being displayed as quests (yes i'm 100% sure they were submitted). After a few answers i submitted them, to my surprise my answer level (or the number on the right of the start...not really sure how you guys called), it always displayed the same the same value, no matter the number of answer i submit, close the app, reboot, etc.... every time its the same number.
I decreased the the size of the cache to 1mb (i usually have it at 200mb), didn't solve.

Not sure if the error is on my end or your end. let me know how to proceed.

Off topic: would be cool if in the credits all the translator would be displayed, like previously and not only a few ones and "others" have to be viewed on the poeditor.

keep up the good work

[westnordost]

The quest counter (the one with the star) does not increase when you answer a quest that turns out to be in conflict with the OSM database (because someone else or in that case you yourself did answer that before).

So, what likely happened is that for some reason, the downloaded data was old. Overpass data lags behind "original" OSM data a bit, but usually not that much. How much time passed between you submitted first answer and the factory reset, followed by your repeated answering of the quests?

If StreetComplete detects that there is a conflict in an area, it will immediately mark this area as to-be-redownloaded and will do so at next opportunity (either when you click "scan here" or if auto-sync is on, immediately). But if the Overpass data is still lagging behind, i.e. still doesn't give you the current data, the situation is the same.

So, this very probably can't be helped. You just need to be patient.

[hsimpson00]

Hi, I have a similar problem:
Since I noticed the street surface in my area was too complex to change it via StreetComplete because you had to split up the roads, I opened JOSM and made the changes.
The App asked me nearly 24h after that changes for the surface of a road piece, I deleted in that changeset. I don't know, what would happen, if I answered that question.
I reproduced this problem yesterday with a school, for wich I made an area and added the address to this area. StreetComplete is still asking for the adress of the buildings inside this area, evern though it shouldn't since the last upadte.

I think I also got the reason for this:

I didn't change my location for some time, so if I press "Hier nach Aufgaben suchen", it tells me "Keine weiteren Aufgaben verfügbar". And even if it finds new Questions, it doesn't delete the old ones.

I even tried to delete the cache of the App via the Android-Settings, but the old tasks still remain in the App even if you close it completeley.

Solution:

-Either: When pressing "Hier anch Aufgaben suchen", the App should delete the old tasks and make a complete new search.
-Or: A new button with "reload all tasks" which deletes all loaded tasks and searches for new ones.

In both cases, the tasks should expire after some time, maybe after 12 hours or less.

Regards

[westnordost]

In your case, @hsimpson00, you'd need to clear data, not cache. But this will completely reset the app as if you installed it fresh. Or, solve one of the old quests. The app will detect a conflict, drop your answer and mark surrounding quests as to-be-redownloaded.

[hsimpson00]

Ok thanks for that information!

[iamnunocaldeira]

Answers to quests were submitted 1 month and 3 weeks ago. I factory reseted my phone last week.

Also noticed a way I edited on JOSM 3 weeks ago (it was wrongly represented as two ways instead of one) and on the app it's still two ways and asks me what's the surface of it.

[westnordost]

And if you rescan for quests manually at that location, does it still show (as 2)?

On 29 August 2017 17:30:49 CEST, Nuno Caldeira [e-mail] wrote:

Answers to quests were submitted 1 month and 3 weeks ago. I factory
reseted my phone last week.

Also noticed a way I edited on JOSM 3 weeks ago (it was wrongly
represented as two ways instead of one) and on the app it's still two
ways and asks me what's the surface of it.

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/westnordost/StreetComplete/issues/562#issuecomment-325701722

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[iamnunocaldeira]

yes i have re scanned and displays 2 roads. heres the street "Rua Ribeira João Gomes" http://www.openstreetmap.org/edit#map=18/32.66033/-16.90009

[westnordost]

Okay, this is unfortunate. Someone else had the same problem and thereupon I completely re-tested the feature but could not find anything wrong. Right now, something else came to my mind which I should test, but this also works fine.
I must be on a completely wrong track where this is coming from.

Hm.

Okay, I have an idea:

Do you have Android studio an can look at the log? If not,

1. go into app settings and turn off auto sync
2. please download and _start_ a logging app like aLogcat, syslog etc.
3. go into settings and set your smartphone time to 8 days in the future
4. go back into the app and scroll to the point with 2 road surface quests where there should be only one, press "scan for quests here". Write down at which time you did this (so I can find where I have to look at in the log file)
5. Go back to the logging app to look at the log and post it here / send it via email to me (osm@ my user name .de)

[Luzandro]

First of all there should be a way to clear and refresh data without having to create a conflict. And second, it doesn't even seem to work for me. I've solved an outdated quest, I've said upload (at least this didn't actually lead to a changeset), cleared the cache, searched for new quests and there are still outdated ones nearby.

[FX7]

I think I got a similar problem as discussed indoor this issue.
My wife and I got the app installed on our mobiles. I understand that if one solves a problem that it will not automatically disappear on the other device. But after a while, in our case we are talking about days, a click on "Hier nach Aufgaben suchen" should solve the problem (shouldn't it?). But it won't. Still some already solved quests are showen (on the device that didn't solve the quest).
And there is another related problem.
On one devive much more quests are shown then in the other. For example the device of my wife isn't showing the "street with lamp" quests. I think we both have the latest f-droid version (2.3) of the app installed, but I will verify this.

[westnordost]

But after a while, in our case we are talking about days, a click on "Hier nach Aufgaben suchen" should solve the problem (shouldn't it?)

No.

On one devive much more quests are shown then in the other.

Not all possible quests are downloaded at once. They are only downloaded automatically once the more important ones are solved (or repeatedly triggering "Hier nach Aufgaben suchen").

[mmd-osm]

While testing the new changeset upload (https://www.openstreetmap.org/user/mmd/diary/44318), I also explicitly tested for Conflicts in StreetComplete, and unfortunately, I only got the following error:

Is it still the case that StreetComplete doesn't handle editing conflicts?

Here's an HTTP protocol I captured in Wireshark:

PUT /api/0.6/changeset/create HTTP/1.1
User-Agent: StreetComplete 7.0-beta1
Content-Type: text/xml
charset: utf-8
Authorization: ***
Host: 78.46.206.64
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 250

<?xml version='1.0' encoding='UTF-8' ?><osm><changeset><tag k="StreetComplete:quest_type" v="AddRoadSurface" /><tag k="comment" v="Add road surfaces" /><tag k="source" v="survey" /><tag k="created_by" v="StreetComplete 7.0-beta1" /></changeset></osm>


HTTP/1.1 200 OK
Date: Wed, 11 Jul 2018 18:15:32 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/plain; charset=utf-8
Vary: Origin,Accept-Encoding
ETag: W/"447e705152a46144c3bb86f17343f325-gzip"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: b32fa89e-d74f-4550-808c-676bd3dd3db4
X-Runtime: 0.060693
X-Frame-Options: sameorigin
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Content-Encoding: gzip
Content-Length: 26
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

2000000152


POST /api/0.6/changeset/2000000152/upload HTTP/1.1
User-Agent: StreetComplete 7.0-beta1
Content-Type: text/xml
charset: utf-8
Authorization: ***
Host: 78.46.206.64
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 344

<?xml version='1.0' encoding='UTF-8' ?><osmChange><modify><way id="26568754" version="2" changeset="2000000152"><nd ref="291292754" /><nd ref="291292679" /><nd ref="291292453" /><tag k="name" v="Bergstraße" /><tag k="highway" v="residential" /><tag k="surface" v="asphalt" /><tag k="created_by" v="Potlatch 0.10b" /></way></modify></osmChange>


HTTP/1.1 409 Conflict
Date: Wed, 11 Jul 2018 18:15:32 GMT
Server: Apache/2.4.29 (Ubuntu)
Error: Version mismatch: Provided 2, server had: 4 of Way 26568754
Cache-Control: no-cache
Content-Length: 59
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/plain

Version mismatch: Provided 2, server had: 4 of Way 26568754

[matkoniecz]

Is it still the case that StreetComplete doesn't handle editing conflicts?

StreetComplete generally handles edit conflicts by not uploading affected edits (as resolving edit conflicts is highly complicated and for StreetComplete edits usually not worth doing it) and downloading new quest data to avoid wasting editor effort.

Apparently you managed to trigger some unhandled case.

[mmd-osm]

Ok, so here's a very brief analysis of the situation. Bottom line is, that the upload thinks there's a changeset conflict, but in reality there's an object version conflict.

Steps to reproduce:

1. Load some quest in SC and identify an object to be changed later on
2. Edit the same object in another editor, like iD or JOSM + upload (this could be another user changing the object in the meantime)
3. Change object in SC and upload change

First attempt to upload changes triggers an HTTP conflict, which raises an OsmConflictException

case HttpURLConnection.HTTP_CONFLICT:
                return new OsmConflictException(error, response, description);

OsmConnection decides to disconnect!

    finally
    {
        if(connection != null) connection.disconnect();
    }

On to AOsmQuestChangesUpload...

    catch(OsmConflictException e)
    {
        return handleConflict(changesetId, quest, element, alreadyHandlingElementConflict,
                alreadyHandlingChangesetConflict, e);
    }

which then calls handleConflict

-> !changesetInfo.isOpen is true --> Changeset is closed now, probably due to the disconnect.

    if(!changesetInfo.isOpen || changesetWasOpenedByDifferentUser)
    {
        // safeguard against stack overflow in case of programming error
        if(alreadyHandlingChangesetConflict)
        {
            throw new RuntimeException("OSM server continues to report a changeset " +
                    "conflict for changeset id " + changesetId, e);
        }
        return handleChangesetConflict(quest, element, alreadyHandlingElementConflict);
    }

-> calls handleChangesetConflict, which creates a new changeset and tries to upload the same change
--> Again HTTP/409 conflict

Later: alreadyHandlingChangesetConflict is true now --> raises RuntimeException -> boom!

        // safeguard against stack overflow in case of programming error
        if(alreadyHandlingChangesetConflict)
        {
            throw new RuntimeException("OSM server continues to report a changeset " +
                    "conflict for changeset id " + changesetId, e);
        }

[westnordost]

This seems to be very valuable information you supply there. I will have a look at this immediately

[westnordost]

The assumption that the changeset is closed on disconnect is wrong, the disconnect is just the disconnect of the HTTP connection for the API call. Closing the changeset is a separate command of the OSM API.

In the case that for some reason the changeset has been closed, the logic in handleChangesetConflict is supposed to create a new open changeset.
Thus, on the next recursion, when the OSM API continues to return a HTTP/409 conflict, in the case of an element conflict, as you provoked it with the JOSM edit, the execution should continue in line 343

return handleElementConflict(changesetId, quest, alreadyHandlingChangesetConflict);

The wireshark log you provided only shows one HTTP/409 conflict reply. So, StreetComplete ran into an error already after the first conflict reply?

[westnordost]

I tried to reproduce it:

1. I loaded building quests in StreetComplete, amongst these https://www.openstreetmap.org/way/581421751
2. I downloaded the area in JOSM and made a modification to the building
3. I answered the quest and uploaded it
4. StreetComplete detected an element conflict (and decided that it is not solvable)

[mmd-osm]

StreetComplete ran into an error already after the first conflict reply?

No, the runtime error occurred during the second upload attempt, which isn’t shown in the trace (I only found out about it while debugging the code)

The assumption that the changeset is closed on disconnect is wrong

The closed changeset was the trigger to initiate another upload attempt in the code. I’m not sure why this flag is set in the first place.

I’ll double check this tomorrow...

[westnordost]

I tried another time to reproduce it, this time attempting to take exactly the same execution path as you described:

1. I loaded building quests in StreetComplete, amongst these https://www.openstreetmap.org/way/581421755
2. I downloaded the area in JOSM and made a modification to the building, uploaded that change
3. I found the previously opened changeset that StreetComplete planned to upload the quest answer to and manually closed it
4. I answered the quest and uploaded it
5. StreetComplete encountered a 409/Conflict, determined that the changeset was closed, created a new changeset, tried uploading again, encountered another 409/Conflict, determined that the changeset was still open and thus updated the element from the server, applied the answer to the updated element and tried again, and finally succeeded. No error.

So, I cannot reproduce this. Something else must have gone wrong.

[westnordost]

The only way I could think of how this error could currently come to pass would be either:

1. After encountering that the changeset the app was going to use was closed and a new changeset is created by the app, the newly created changeset is closed immediately before the app makes another attempt to upload (or, at least, the OSM API reports the new changeset as closed)
2. After encountering the element the app was going to update is old (element conflict), fetching the new element from the server, the element is edited another time by a third party before the app makes another attempt to upload the change (or, at least, the OSM API does not supply the really-latest element)

But this is extremely unlikely to happen, because the app tries to upload the element again immediately after conflict. I am only mentioning this because you mentioned you found this issue while testing the new implementation for the OSM API (diff upload method), so there is maybe a possibility of a bug (a race condition) in your implementation.

[mmd-osm]

After some further tracing, it looks like there's some issue with the current CGImap implementation, which implicitly assumes the server to run in UTC-0 timezone. If you look at the server response below, you'll notice that the changeset will only close in 1 hour, still the server claims that the changeset is already closed. This "changeset is closed" information was also the trigger for the second upload attempt I've seen during debugging.

This doesn't happen on osm.org as it is running on UTC+0, while my server is running on UTC+2.

So thanks again for providing valuable insights, your first point ("OSM API reports the new changeset as closed") was spot on! I will follow up on cgimap now...

GET /api/0.6/changeset/2000000179?include_discussion=true HTTP/1.1
User-Agent: StreetComplete 7.0-beta1
Host: 78.46.206.64
Connection: Keep-Alive
Accept-Encoding: gzip

HTTP/1.1 200 OK
Date: Thu, 12 Jul 2018 05:07:21 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Encoding: gzip
Cache-Control: private, max-age=0, must-revalidate
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/xml; charset=utf-8

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.6.0 (10976 ubuntu-4gb-fsn1-1)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <changeset id="2000000179" created_at="2018-07-12T05:05:13Z" closed_at="2018-07-12T06:05:13Z" open="false" user="mmd2" uid="100000001" comments_count="0">
  <tag k="created_by" v="StreetComplete 7.0-beta1"/>
  <tag k="source" v="survey"/>
  <tag k="comment" v="Add road surfaces"/>
  <tag k="StreetComplete:quest_type" v="AddRoadSurface"/>
  <discussion/>
 </changeset>
</osm>

[westnordost]

Cool, so analysis of the problem was well worth it :-)

On July 12, 2018 7:34:57 AM GMT+02:00, mmd [e-mail] wrote:

After some further tracing, it looks like there's some issue with the
current CGImap implementation, which implicitly assumes the server to
run in UTC-0 timezone. If you look at the server response below, you'll
notice that the changeset will only close in 1 hour, still the server
claims that the changeset is already closed. This "changeset is
closed" information was also the trigger for the second upload attempt
I've seen during debugging.

GET /api/0.6/changeset/2000000179?include_discussion=true HTTP/1.1
User-Agent: StreetComplete 7.0-beta1
Host: 78.46.206.64
Connection: Keep-Alive
Accept-Encoding: gzip

HTTP/1.1 200 OK
Date: Thu, 12 Jul 2018 05:07:21 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Encoding: gzip
Cache-Control: private, max-age=0, must-revalidate
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/xml; charset=utf-8

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.6.0 (10976 ubuntu-4gb-fsn1-1)"
copyright="OpenStreetMap and contributors"
attribution="http://www.openstreetmap.org/copyright"
license="http://opendatacommons.org/licenses/odbl/1-0/">
<changeset id="2000000179" created_at="2018-07-12T05:05:13Z"
closed_at="2018-07-12T06:05:13Z" open="false" user="mmd2"
uid="100000001" comments_count="0">
 <tag k="created_by" v="StreetComplete 7.0-beta1"/>
 <tag k="source" v="survey"/>
 <tag k="comment" v="Add road surfaces"/>
 <tag k="StreetComplete:quest_type" v="AddRoadSurface"/>
 <discussion/>
</changeset>
</osm>

[mmd-osm]

Cool, so analysis of the problem was well worth it :-)

Yes, it really turned out to be a timezone issue which disappeared once UTC+0 is used in the cgimap code. Sorry again for making some noise here, sometimes it's not clear right from the start what's really going on.

[cyclingcat]

Hi there,

some days ago when I revisited the location mentioned in https://github.com/westnordost/StreetComplete/issues/692#issuecomment-345035163, I noticed that the quest for opening times still referred to the POI which was changed in OSM more than one year ago (!). A "Kreissparkasse" was closed and a hairdresser opened in this building, however, StreetComplete (v8.2 on Android 5.1) still asks for the opening times of the "Kreissparkasse"! Clearing the app cache or invalidating the quest cache both do not help.

Now, at home again, I'm still able to reproduce this, so there seems to be a serious problem of the quest cache if information which has been outdated for more than a year is still shown.

The cycling cat

[westnordost]

Did you read the text in the dialog presented when clicking the invalidate quest cache option?

On 20 October 2018 17:23:08 CEST, cyclingcat [e-mail] wrote:

Hi there,

some days ago when I revisited the location mentioned in
https://github.com/westnordost/StreetComplete/issues/692#issuecomment-345035163,
I noticed that the quest for opening times still referred to the POI
which was changed in OSM more than one year ago (!). A
"Kreissparkasse" was closed and a hairdresser opened in this building,
however, StreetComplete (v8.2 on Android 5.1) still asks for the
opening times of the "Kreissparkasse"! Clearing the app cache or
invalidating the quest cache both do not help.

Now, at home again, I'm still able to reproduce this, so there seems to
be a serious problem of the quest cache if information which has been
outdated for more than a year is still shown.

The cycling cat

[cyclingcat]

Of course I did, but I still don't understand why the name of a POI is shown which was changed long time ago: It's more than one week since this change, so the cache should have been invalidated automatically. (Nevertheless I tried it manually without any effect.) Furthermore, I tried to download the quests manually after I invalidated the quest cache, but still no change.

So what exactly can I do in order to show the correct POI name in the text of this quest?

The cycling cat

[westnordost]

What happens when you move the opening hours quest to the top of the quest priority queue and then hit "download quests here"?

On 20 October 2018 18:09:06 CEST, cyclingcat [e-mail] wrote:

Of course I did, but I still don't understand why the name of a POI is
shown which was changed long time ago: It's more than one week since
this change, so the cache should have been invalidated automatically.
(Nevertheless I tried it manually without any effect.) Furthermore, I
tried to download the quests manually after I invalidated the quest
cache, but still no change.

So what exactly can I do in order to show the correct POI name in the
text of this quest?

The cycling cat

[cyclingcat]

Indeed, this did the trick! (Before the reorder, this quest had the third highest priority, behind postbox collection times and road surface.)

The cycling cat

[westnordost]

Actually, the app downloads I think min 8 quest types per click on that button

On 20 October 2018 19:40:21 CEST, cyclingcat [e-mail] wrote:

Indeed, this did the trick! (Before the reorder, this quest had the
third highest priority, behind postbox collection times and road
surface.)

The cycling cat

[cyclingcat]

Okay, but what can the user do in those cases? It's quite confusing to see names which have vanished one year ago over and over again... perhaps a click on "Invalidate quest cache" should do more that it currently does?

The cycling cat

[westnordost]

Perhaps it should. Or, #766 should be implemented (delete always all unsolved quests older than 1 month)

[westnordost]

Closing because several problems got mixed up in here of which those which were reproducable have been fixed.