Streetcomplete: F-Droid builds only crash: No field map of type Ljava/util/Map; in class Lorg/json/JSONObject

[oxqgit]

Workaround for F-Droid users till you can update to v22.3

Go the settings and disable the _"What is the name of this place?"_ quest type (orange icon with white label). Then, the quest download should work again like usual.

Original bug report

Tapping any "name tag" quest icon crashes the app instantly after highlighting the place.

Thread: main
App version: 22.0
Device: HUAWEI  HWANE, Android 9
Locale: de_DE

Stack trace:

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:534)

at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)

Caused by: java.lang.reflect.InvocationTargetException

at java.lang.reflect.Method.invoke(Native Method)

at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)

... 1 more

Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: No field map of type Ljava/util/Map; in class Lorg/json/JSONObject; or its superclasses (declaration of 'org.json.JSONObject' appears in /system/framework/core-libart.jar)

at java.util.concurrent.FutureTask.report(FutureTask.java:123)

at java.util.concurrent.FutureTask.get(FutureTask.java:193)

at de.westnordost.streetcomplete.quests.QuestUtilKt$getTemplateArguments$typeName$1.invoke(QuestUtil.kt:1)

at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:6)

at de.westnordost.streetcomplete.quests.place_name.AddPlaceName.getTitleArgs(AddPlaceName.kt:1)

at androidx.appcompat.app.ResourcesFlusher.getTemplateArguments(ResourcesFlusher.java:5)

at androidx.appcompat.app.ResourcesFlusher.getHtmlQuestTitle(ResourcesFlusher.java:1)

at de.westnordost.streetcomplete.quests.AbstractQuestAnswerFragment.onViewCreated(AbstractQuestAnswerFragment.kt:2)

at de.westnordost.streetcomplete.quests.AbstractQuestFormAnswerFragment.onViewCreated(AbstractQuestFormAnswerFragment.kt:1)

at de.westnordost.streetcomplete.quests.place_name.AddPlaceNameForm.onViewCreated(AddPlaceNameForm.kt:1)

at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:115)

at androidx.fragment.app.FragmentManager.moveFragmentToExpectedState(FragmentManager.java:4)

at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:375)

at androidx.fragment.app.BackStackRecord.executeOps(BackStackRecord.java:31)

at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:51)

at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:10)

at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:14)

at androidx.fragment.app.FragmentManager$4.run(FragmentManager.java:1)

at android.os.Handler.handleCallback(Handler.java:907)

at android.os.Handler.dispatchMessage(Handler.java:105)

at android.os.Looper.loop(Looper.java:216)

at android.app.ActivityThread.main(ActivityThread.java:7625)

... 3 more

Caused by: java.lang.NoSuchFieldError: No field map of type Ljava/util/Map; in class Lorg/json/JSONObject; or its superclasses (declaration of 'org.json.JSONObject' appears in /system/framework/core-libart.jar)

at de.westnordost.osmfeatures.iDFeatureCollection.parseStringMap(iDFeatureCollection.java:3)

at de.westnordost.osmfeatures.iDFeatureCollection.parseFeatures(iDFeatureCollection.java:6)

at de.westnordost.osmfeatures.iDFeatureCollection.(iDFeatureCollection.java:4)

at de.westnordost.streetcomplete.data.meta.MetadataModule$featureDictionaryFuture$1.call(MetadataModule.kt:2)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at de.westnordost.streetcomplete.StreetCompleteApplication.lambda$preload$1$StreetCompleteApplication(StreetCompleteApplication.java:1)

at de.westnordost.streetcomplete.-$$Lambda$StreetCompleteApplication$TOPCfQEhitCcMdF2wUE8N5y0WBQ.run(Unknown Source:2)

at java.lang.Thread.run(Thread.java:784)

[westnordost]

This is caused by the tags of an element or the element itself being null. Not sure why this happens, it shouldn't.

[westnordost]

So this should not be related to the place name quest, this is only how it surfaces for you. It will probably also not be reproducible on any place name quest.
Can you tell me the location of the element for which it crashes?

[westnordost]

My suspicion is that the cause for this bug is the same as for

Device: CUBOT QUEST, Android 9
Thread: main
App version: 20.2
Stack trace:
kotlin.KotlinNullPointerException
    at de.westnordost.streetcomplete.quests.surface.AddRoadSurfaceForm.getTopItems(AddRoadSurfaceForm.kt:12)
    at de.westnordost.streetcomplete.quests.AGroupedImageListQuestAnswerFragment.getInitialItems(AGroupedImageListQuestAnswerFragment.kt:93)
    at de.westnordost.streetcomplete.quests.AGroupedImageListQuestAnswerFragment.onViewCreated(AGroupedImageListQuestAnswerFragment.kt:74)
    at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:332)
    at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1187)
    at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1356)
    at androidx.fragment.app.FragmentManager.moveFragmentToExpectedState(FragmentManager.java:1434)
    at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1497)
    at androidx.fragment.app.BackStackRecord.executeOps(BackStackRecord.java:447)
    at androidx.fragment.app.FragmentManager.executeOps(FragmentManager.java:2169)
    at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:1992)
    at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:1947)
    at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:1849)
    at androidx.fragment.app.FragmentManager$4.run(FragmentManager.java:413)
    at android.os.Handler.handleCallback(Handler.java:873)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:193)
    at android.app.ActivityThread.main(ActivityThread.java:6717)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:911)

This also happens because the element (tags) are null. This happens very infrequently though, so no clue yet.

[westnordost]

I checked again all the places where elements are deleted and found no reason why/how elements could be wrongfully deleted, not even in a race condition.

So I continue to be clueless here.

I need more information on how I could reproduce this issue. Things that may be helpful:

1. For the item it crashes now, did you solve a quest for it in the past?
2. What type of element is it? Is it a node (point) or a way (roads, buildings etc)?

[oxqgit]

I don't want to post my own location here, I will give a location more far away later (searching for quests doesn't work right now).

1. No, I didn't solve quests for the places before.
2. It happens for every place name quest in my neighbourhood I tap on. It happens with nodes, but also buildings.
   All other quests work fine for now.

[HolgerJeromin]

I don't want to post my own location here,

You can mail Tobias the location:

https://wiki.openstreetmap.org/wiki/User:Westnordost has his e-Mail adress

[oxqgit]

Ok, i'll send a mail in a minute.
Might be another issue or related to this one: right now only street renaming quests and note discussion quests are loading, then the quest search fails.

[Atrate]

then the quest search fails

On what element does it fail?

[westnordost]

Okay, one element is a node, the other is a closed way. Both haven't been edited for at least two years, both have no name.

[westnordost]

I looked through the stack trace again and I think this one is not related to the spurious "element is null" error after all but parsing of the feature dictionary fails.

This means that for those who can reproduce it, it will happen on any place name quest.

It is probably also very much related to the Android Version one is using. So we got the following devices/versions so far with which this is reproducible:

• HUAWEI HWANE, Android 9
• Fairphone FP3, Android 9

I'll try to reproduce it in an Android 9 emulator.

[oxqgit]

@Atrate Different, sometimes on the place name quest, sometimes on the discussion quest or the street name quest. Today after a reboot other quests were loading again once but now it's the same again: only discussion quests and street name quests are loading and the search fails on a seemingly random element.

[westnordost]

The quest search fails with what error? If you are using the f-droid version, it should offer you to send an email to me with the crash report.

Also, unfortunately I wasn't able to reproduce it on an Android 9 emulator on v22.0.

@oxqgit could you maybe install the v22.1 from GitHub (you need to uninstall the F-Droid version first) and see if the problem persists? https://github.com/westnordost/StreetComplete/releases/download/v22.1/StreetComplete-v22.1.apk

[oxqgit]

@westnordost It fails with the download_server_error, in english _"Connection error while scanning for quests. Try again later."_
Right there, it doesn't crash. The app only crashes when tapping on name place quests and the stacktrace at the beginning is taken from the email.

[oxqgit]

Installed v22.1 right now and the previous problems are solved. No download_server_error, all kind of quests are loaded and tapping on name tag quests works fine!

[westnordost]

Well that the download fails may be related because place name quests are only generated for elements that have "feature names" (f.e. "restaurant"), so the dictionary is used to determine that. If the dictionary cannot be parsed at that point, download will fail.

[jtracey]

Still seeing this on 22.1 (f-droid, so no 22.2 yet). Same induction, selecting a naming quest. I can post the stack trace if you want, though at a glance it looks pretty much the same as the one in the first post.

EDIT: should mention, I'm running LineageOS 16 (so android 9.0/pie)

[Atrate]

22.1 is broken and banned from uploading. Wait for 22.2 to arrive on F-Droid.

[jtracey]

I know, I'm not talking about uploading, I'm talking about this bug (street complete crashes when selecting a naming quest).

[Strubbl]

I can reproduce the issue with f-droid's version 22.0.

If i click on any of the two quests, the crash happens with the stacktrace from the first post in this issue:

You can find the notes here: https://www.openstreetmap.org/#map=19/48.10044/11.62280

(i am using stock android 10 on a samsung s9)

[Strubbl]

I installed the debug version along with the release and i cannot reproduce the issue with the latest master commit da46f8665902bd933af3d49be6daf4164bc6e873

[westnordost]

Okay, that's too bad :-(

Please anyone who can reproduce this please also state your Android version and device because I can't reproduce it.

Am 14. August 2020 20:19:34 MESZ schrieb Strubbl [e-mail]:

I can reproduce the issue with f-droid's version 22.0.

If i click on any of the two quests, the crash happens with the
stacktrace from the first post in this issue:

You can find the notes here:
https://www.openstreetmap.org/#map=19/48.10044/11.62280

[westnordost]

@Strubbl and with the 22.2 release? (non debug build)

[Strubbl]

@Strubbl and with the 22.2 release? (non debug build)

i am going to test it when it is arriving in f-droid.

[westnordost]

You can also take the one from github

[Strubbl]

You can also take the one from github

Just tested this apk. Does not crash anymore when opening the mentioned quests.

[westnordost]

But 22.1 does? Because these two versions are literally the same minus the _unrelated_ one-off removal of possibly corrupted data. That's... I don't know, this seems quite impossible. Only two explanations come to my mind:

1. The build process is not clean and some artifacts when I built 22.1 were taken from when I actually compiled it for 22.0. This seemed to have happened before. But since this seemed to have happened before one time, I always make sure to issue a complete rebuild when I build the release version
2. This is a problem that actually only occurs for builds created by the F-Droid build bot.

Point 2 we can easily test. Try it again with the current F-Droid v22.1, does it work? Then, try it again with the current v22.1 here on GitHub, does it work?
@Strubbl would you do this?

[Strubbl]

I do not know about 22.1 yet. I did not test that version, but i can try right now.

To summarize:

• v22.0 from f-droid as of today afternoon, installed some days ago: crash on clicking the quest (like described in this issue in the first post)
• v22.0 from f-droid as of now, clean install: i cannot download the quests as described by @horstworst in #2017
• v22.1 from f-droid as of now: i cannot download the quests as described by @horstworst in #2017
• v22.2 from github installed via adb: searching for all quests works fine and no crash anymore on clicking the quest
• git master HEAD debug installed with Android Studio: searching for all quests works fine and no crash anymore on clicking the quest

It is the quest "what is the name of?(...)" that fails downloading with v22.0 and v22.1. And exactly this type of quest was the one, which was causing the crash, but only when i clicked on it. Maybe i had this quest still in cache from before i installed v22.0.
With a clean install, either of 22.0 or 22.1, i cannot reproduce the crash mentioned in this issue in the first post, because the app cannot download the quest which i would have to click to get the crash.

[westnordost]

Yes, it crashes because both when showing the "what is the name of ..." quest and when downloading the "what is the name of ..." quest for the first time, the feature dictionary is parsed. This fails.

Hmm, this is a strong indication that this is an issue that somehow only affects the F-Droid builds. Though to be sure, it would be good to actually compare the same version from F-Droid with the same version from Github, both as release builds.

So in short, could you check if v22.1 from GitHub produces this crash (when downloading the "what is the name of..." quest)?

[Strubbl]

Yes, it crashes because both when showing the "what is the name of ..." quest and when downloading the "what is the name of ..." quest for the first time, the feature dictionary is parsed. This fails.

So when downloading quests fails, this is also a crash? Because the app is still running and i get the toast as seen in the screenshot in #2017.

Oh, your assumption is right. I just installed 22.1 from github. It succeeds downloading the quests and it also does not crash when i click the quest, which was causing the crash.
Because i couldn't believe it, i retried the fdroid version. And yes, i can confirm the fdroid v22.1 crashes, but the github v22.1 does not. Crazy! :tada:

[westnordost]

So when downloading quests fails, this is also a crash

Well, it's the same error, but as the download happens in a background thread, the exception is caught and just a message displayed to the user. If I remember correctly, the next time you start the app or something, StreetComplete will ask you to send a bug report for that, but not sure.

Okay, thanks for the testing! Hmm, where to go from here though now? I'll think of something tomorrow, maybe unzip both APKs and toss them in some directory diff tool to see where they are different.

[westnordost]

I compared the GitHub version, a version built by running the gradle build task from source and the F-Droid version and indeed they are different. The F-Droid version includes about 2000 functions less. If that is the source of the issue I don't know, but anyway I reported it in the F-Droid issue tracker here.

If you find anything else worth noting, feel free to add it to that ticket:
https://gitlab.com/fdroid/fdroidserver/-/issues/814

[matkoniecz]

Ticket was moved to https://gitlab.com/fdroid/fdroiddata/-/issues/2132 (I will try to reproduce this on Linux outside Android Studio)

[smichel17]

Short: I have a Linux (Fedora) build environment set up, and I usually work from the command line. I can share a build if you let me know what steps you want me to take.

Long version: I'm rather busy at the moment. I do have some spare time, when I'm spent and wouldn't be productive if I were working, anyway... but that's also time when I don't really have the energy/focus to debug properly. Hence, while I watch this repo and am skimming everything that comes through my inbox, I'd appreciate spelling out what you'd like instead having to think more :sweat_smile:

[westnordost]

Okay, I was able to reproduce it and found the culprit. So this should be fixed.

[Strubbl]

Today, v22.2 arrived in f-droid. Of course this version is also affected by this issue and i am able to reproduce the crash. I am excited to see the fix when the v22.3 arrives in f-droid. :+1:

[alanz]

I am running v22.2 from f-droid on a One-plus 5 (OxygenOS 9.0.11) and am also getting "Connection error while scanning for quests, please try again later". I tried deleting all data and starting again, made no difference.

[matkoniecz]

See https://github.com/westnordost/StreetComplete/releases/tag/v22.3 - it is fixed in released v22.3

Note that F-Droid is using its own build system what delays releases there, so currently processing/delay is on F-Droid side and cannot be influenced here.

[alanz]

Thanks, I was hoping that was the case, but figured I would check in case, given the f-droid delay.

[Bubu]

Something odd was happening during the F-Droid autoupdate. I've just fixed the build metadata, it should hopefully build successfully now.

[Strubbl]

@Bubu is it possible to see the current status of the f-droid build of this app anywhere?

[Bubu]

Click on "Build Log" here: https://f-droid.org/en/packages/de.westnordost.streetcomplete/

replace the versioncode with the one you are looking for in that link.

[Strubbl]

Thank you. Is this the current status of the upcoming build 2204? Or hasn't it been updated after your fix of the build metadata yet?

Currently it says:

== Installed Android Tools ==

INFO: Creating log directory
INFO: Creating temporary directory
INFO: Creating output directory
INFO: Using git version 2.11.0
INFO: Building version 22.3 (2204) of de.westnordost.streetcomplete
INFO: Getting source for revision v22.3
INFO: Creating local.properties file at build/de.westnordost.streetcomplete/local.properties
INFO: Creating local.properties file at build/de.westnordost.streetcomplete/app/local.properties
INFO: Cleaned build.gradle of keysigning configs at build/de.westnordost.streetcomplete/app/build.gradle
INFO: Cleaning Gradle project...
ERROR: Could not build app de.westnordost.streetcomplete: Error cleaning de.westnordost.streetcomplete:22.3
==== detail begin ====
Found 6.6 via distributionUrl
No hash for gradle version 6.6! Exiting...
==== detail end ====

copied from https://f-droid.org/repo/de.westnordost.streetcomplete_2204.log.gz

From what i can see from the gradle release, the hash file is in place, see file gradle-6.6-all.zip.sha256 in https://services.gradle.org/distributions/
How is it possible to provide that hash for gradle version 6.6? Is 6.6 not supported by f-droid yet?

[Bubu]

The buildserver needs to be redeployed to include the newer gradle versions.

[westnordost]

Do we need to notify anyone about it / on the gitlab bugtracker?

[Strubbl]

The buildserver needs to be redeployed to include the newer gradle versions.

After the redeploy, is f-droid trying to build the app again automatically? For when is the redeployment for the buildserver planned?

[Bubu]

After the redeploy, is f-droid trying to build the app again automatically?

yes

For when is the redeployment for the buildserver planned?

I don't know unfortunately.

[michaelblyons]

copied from https://f-droid.org/repo/de.westnordost.streetcomplete_2204.log.gz

You can see the most recent build log on the web at https://f-droid.org/wiki/page/de.westnordost.streetcomplete/lastbuild without downloading the .gz. (Permalink to this broken build, 2204.)

_Edit: fixed, thanks @tuxayo._

[tuxayo]

You can see the most recent build log on the web at https://f-droid.org/wiki/de.westnordost.streetcomplete/lastbuild

This URL right? https://f-droid.org/wiki/page/de.westnordost.streetcomplete/lastbuild

[westnordost]

Workaround for F-Droid users till you can update to v22.3

Go the settings and disable the _"What is the name of this place?"_ quest type (orange icon with white label). Then, the quest download should work again like usual.

[westnordost]

I was thinking about pulling this app from F-Droid altogether since it has almost been one month, however I just checked the build and it looks like yesterday at 08:10 AM the build was successful. So, it should finally be deployed soon.

[Strubbl]

Thanks for not pulling it off fdroid

[alanz]

I am very happy to report that I just upgraded to v22.3 via fdroid, and as expected it does not crash downloading the entire quest set.

[tuxayo]

Indeed it works without the workaround! :D

I was thinking about pulling this app from F-Droid altogether since it has almost been one month

An alternative would have been to push a v22.2.1 to F-Droid that would have been the same source code as the latest version without regressions. (v21.2 IIUC)
To have something deployed and usable without major issues which could allow the wait to have less bad consequences.

(Still a dirty workaround though.)

[westnordost]

(So, the new version is available on F-Droid now, closing)