Streetcomplete: Adding "Ghosts" (or missing shop names) for OSMUK Quarterly Project

[atomoil]

General

Affected tag(s) to be modified/added: Name on Shops
Question asked: What is the shop name?
(ideally it would be Is the shop name correct? but I don't think that's possible)

I am one of the Directors of OpenStreetMap UK and a mobile app developer and we would love to allow mappers to use Street Complete for the Quarterly Projects that we organise.

The next topic is "Ghosts" aka Closed or Renamed Shops and Amenities and so this quest suggestion is to enable that (and start adding quests to StreetComplete 😎). Robert Whittaker has a great tool for showing Ghosts, but I wonder if we could make it broader for StreetComplete, and capture shop names as I don't _think_ there is a way to find only shop names that haven't been updated for a period of time.

I have StreetComplete downloaded and compiling in Android Studio, so can do the work to add it, although I might need some pointers as my speciality is iOS so I usually use Xcode (or React Native or Xamarin for cross platform (aka iOS & Android)), but I am very adaptable.

Checklist

Checklist for quest suggestions (see guidelines):

• [✅] 🚧 To be added tag is established and has a useful purpose
• [✅ ] 🤔 Any answer the user can give must have an equivalent tagging (Quest should not reappear to other users when solved by one)
• [✅] 🐿️ Easily answerable by everyone from the outside but a survey is necessary
• [✅] 💤 Not an overwhelming percentage of elements have the same answer (No spam)
• [✅] 🕓 Applies to a reasonable number of elements (Worth the effort)

Ideas for implementation

Element selection:
Based on looking at the tagFilters property, I _think_ it would be:
nodes, ways with ((amenity ~ post_office|pharmacy|bank|atm|bureau_de_change) or (shop)) and (!name and noname != yes)
but any/all guidance or suggestions are more than welcome.

Proposed GUI:

---

  ^
[ What is the shop name? ]

          Shop Name:
 ________________________
|________________________|

[ENT8R]

This is actually already implemented but not shown at all because there is no tag-to-name dictionary at the moment. See also #309

[rugk]

But it is interesting to know there is such a project about it.

@atomoil Note https://osmuk.org/quarterly-projects/ shows a little too much into the future:

The UK 2019 Q4 Project is “Food Hygiene Ratings revisited”.

The wiki lists that project as Q1.

So as I guess Q1 is correct "Ghosts" is then 2019 Q2?

This could be a short time for an implementation. (Okay your quarter goes up to Apr 2019, so then it's okay 😄)

---

As @ENT8R said, it is already implemented see the in dev quest list, last entry. So as for the "tag-to-name dictionary":

If you really just want to include ((amenity ~ post_office|pharmacy|bank|atm|bureau_de_change) or (shop)) that is likely easy to do though. I mean, maybe it's generally a good idea to start progressively?

[rugk]

Also note this issue has a lot of "kinda" dupes (or at least related issues):

• https://github.com/westnordost/StreetComplete/issues/22 when this has been introduced
• https://github.com/westnordost/StreetComplete/issues/653 referring even a forum post
• and some initial implementation by @ENT8R: https://github.com/westnordost/StreetComplete/issues/309#issuecomment-356354053

[atomoil]

Thanks for the speedy reply!

@atomoil Note https://osmuk.org/quarterly-projects/ shows a little too much into the future:

The UK 2019 Q4 Project is “Food Hygiene Ratings revisited”.

oops... typo fixed - thanks for letting me know 😎

309 is great and a more general approach seems better, albeit longer - is there a possibility for us to help with the tag2name piece?

The other approach for the "Ghosts" Quarterly Project is that it could very specifically target the UK and Robert's list of Ghosts (if you look on individual pages it has the overpass query used, so this could be translated into the query language fairly easily, but would be rather gross). However I'm not so sure that's very beneficial for StreetComplete in the medium to long term - do you have an opinion on this?

[matkoniecz]

is there a possibility for us to help with the tag2name piece?

See https://github.com/ENT8R/tags2names/issues/1

At this moment it requires name index that is currently an idea, without implementation or plan how to do it. And if someone would create well designed name index then almost all work would be done.

[atomoil]

See ENT8R/tags2names#1

At this moment it requires name index that is currently an idea, without implementation or plan how to do it. And if someone would create well designed name index then almost all work would be done.

ok cool, I'll wrap my head around that conversation and respond there if I have any useful thoughts.

[rugk]

However I'm not so sure that's very beneficial for StreetComplete in the medium to long term - do you have an opinion on this?

What? Why?

IMHO starting with GB and (potentially) expanding in the future is no bad thing. It's like big companies piloting/testing a feature under specific circumstances.

[atomoil]

However I'm not so sure that's very beneficial for StreetComplete in the medium to long term - do you have an opinion on this?

What? Why?

IMHO starting with GB and (potentially) expanding in the future is no bad thing. It's like big companies piloting/testing a feature under specific circumstances.

so if you look at the Ghosts table, it's a very specific list of companies that have changed their names - including "Lloyds TSB", "Poundworld", "Flying Tiger" etc.

There are 960 POIs in total in the table, which is not insubstantial, but I wonder if it passes the Applies to a reasonable number of elements checklist.

[rugk]

Ah, okay, now I understand the quest. So it's not exactly a dupe of https://github.com/westnordost/StreetComplete/issues/653 as it is not about naming old companies, but rather about closed or removed things.

That said, maybe this can be a new type of a quest that fetches some extra data of a list of renamed/removed shops. Respectively, this list can also be/include OSM queries directly... 😃

---

BTW, does Robert Whittaker have a GitHub handle? So we can mention him, so he can participate in this discussion. Especially as I think/imagine, this may not the be the only quest, that could be solved with the ideas/data/tools he has collected on https://osm.mathmos.net/.

[matkoniecz]

That said, maybe this can be a new type of a quest that fetches some extra data of a list of renamed/removed shops.

There are plans to include something like this in name-suggestion-index

https://github.com/osmlab/name-suggestion-index/issues/2415

[atomoil]

BTW, does Robert Whittaker have a GitHub handle? So we can mention him, so he can participate in this discussion. Especially as I think/imagine, this may not the be the only quest, that could be solved with the ideas/data/tools he has collected on https://osm.mathmos.net/.

I _think_ Robert Whittaker's GitHub account is @rjw62, but I don't know how active he is on GitHub. 👋 Robert 👋

[atomoil]

That said, maybe this can be a new type of a quest that fetches some extra data of a list of renamed/removed shops. Respectively, this list can also be/include OSM queries directly... 😃

That might push my native Android skills a bit far, but I'm up for trying it out if there is possibility to nudge me in the right direction / answer stupid questions. 😄

So, I think there seem to be 3 options:

1) Use our development time to help with #309
2) Add missing names only for shops and possibly only the UK (as per the Quest spec above)
3) Create a very specific GB quest limited to the Ghosts table

Have I missed anything else?

[matkoniecz]

Use our development time to help with #309

That would certainly be useful though it would require making tag2names what is a bit bigger project. Though it requires no Android skills, that would be simply a dataset

Create a very specific GB quest limited to the Ghosts table

I think that I would wait for @westnordost opinion whatever quest is a good idea. I think that main question here is - what kind of tag should be added if it turns out that according to the survey supposedly gone name is still used?

What should be done with shops/other objects that no longer exist?

[westnordost]

Regarding the ghosts table:
It would be doable to extend or duplicate the place name quest for those places that appear on the ghosts table. What we'd need here is an API to call, best with the possibility to specify a bbox. Also, why is the Ghosts table confined to the UK?
Anyway, I doubt it (create API, create quest) is worth the effort to implement though, counting the current numbers on the website you linked, this is for below 1000 elements. To put it into perspective: There are 658605 shops with no names.
Also, blocked anyway by the lack of denominating a shop with no name.

Regarding the tag2names:
Well nothing happened for quite a time in the ENT8Rs repos and to be honest, I strongly lean towards rather re-using all the translated names from iD or maybe JOSM instead of creating yet another separate translation project with yet again slighty different names and mistranslations for all the things.

And if the translations for the different shops (etc.) are not provided in the iD repository in a form that can be directly re-used because it is too iD specific, then we write a script that regularly scrapes all the translations from their repo and puts it into eligible data structures.

Though, I am not the one who would do that, at least not in the short term, got my hands on other things.

[matkoniecz]

I strongly lean towards rather re-using all the translated names from iD or maybe JOSM

Looking at https://github.com/openstreetmap/iD/blob/master/dist/locales/pl.json#L6995 it seems that iD has entries in nominative case what would require phrasing like "bakery - what is its name?"

In Polish it would be "piekarnia - jaka jest nazwa tego obiektu?" what requires a bit clunkier version to avoid masculine/nonmasculine declination.

There is danger that in some languages there would be no way to avoid broken sentences.

JOSM has data in some weird nontext format - https://josm.openstreetmap.de/browser/josm/trunk/data/pl.lang is not openable as plaintext but I suspect that declination is also missing.

[rugk]

In the worst case you possibly can always say "Ist der/die/das Bäckerei geöffnet?"?

[atomoil]

Regarding the ghosts table:
What we'd need here is an API to call, best with the possibility to specify a bbox.

What would the ideal spec be for the API call be for ease of integration with StreetComplete?
I presume a GET / POST sending bbox and returning a JSON structure.
What should the data be in the response?
I will see if I can pursue this option with Robert for any future quests.

Also, why is the Ghosts table confined to the UK?

I guess that it's likely a by product of Robert being UK based, but I'm not totally sure.

Anyway, I doubt it (create API, create quest) is worth the effort to implement though, counting the current numbers on the website you linked, this is for below 1000 elements. To put it into perspective: There are 658605 shops with no names.

Good point! So, how about returning to the idea that this is a Quest just for missing shop names? This will provide a short term solution while the longer term localisation challenges are properly considered. @westnordost what do you think of this option?

Also, blocked anyway by the lack of denominating a shop with no name.

Could this be solved by using the noname tag?

[westnordost]

The ideal spec would be GET /ghosts?bbox=x,x,x,x and return a CSV or JSON (I find it slightly easier to deal with CSV for structured data).

Good point! So, how about returning to the idea that this is a Quest just for missing shop names?
Could this be solved by using the noname tag?

noname is a different thing. I think there is a misunderstanding. What actually blocks the "What is the name of this shop?" quest, is, that it is necessary to show to the user what kind of a shop it is. E.g. "What is the name of this florist?".
Why is it not enough to show the location alone? Because that location may not be precise, the GPS location of the surveyor may not be precise and it may be in an area, where there are many small shops really close to each other.

[rjw62]

It would be easier enough for me to set up an API like that for my Ghosts tool, if wanted. However, I'm not sure quite how you'd phrase the question and deal with the answers. The entries are a mix of two things: (1) stores that have closed, and (2) stores that have (probably) been renamed/rebranded. The tool knows the difference between the two cases, so could provide this information through the API.

In case (1) you'd probably want to ask "Is the XXX still here?" and then if not, ideally you'd want to know what is there now in its place. It might be an empty unit, or it might be a new shop/amenity of a different type. As I understand it, StreetComplete isn't really set up for capturing that sort of detail.

In case (2) you might be able to ask "Has the XXX been rebranded as YYY?" although in at least one case, there is more than one YYY answer (LloydsTSB, has de-merged into Lloyds and TSB, so former branches could now be either.)

[matkoniecz]

As I understand it, StreetComplete isn't really set up for capturing that sort of detail.

In the worst case note with a photo of a current state may be opened.

[atomoil]

a CSV or JSON (I find it slightly easier to deal with CSV for structured data).

@westnordost is there an example in StreetComplete already of loading external data in CSV or JSON (or any format)? Or would that be a brand new feature to add?

noname is a different thing. I think there is a misunderstanding. What actually blocks the "What is the name of this shop?" quest, is, that it is necessary to show to the user what kind of a shop it is. E.g. "What is the name of this florist?".

@westnordost ah, I see... so am I right in saying that the challenge would still be providing good quality localisation of the question to cover the different types of shops? Actually.. where is the localisation handled in the source code? All I can see in the code is english, so does something swap it out? (forgive me if that's a super dumb question, I'm not an Android expert)

In case (1) you'd probably want to ask "Is the XXX still here?" and then if not, ideally you'd want to know what is there now in its place. It might be an empty unit, or it might be a new shop/amenity of a different type. As I understand it, StreetComplete isn't really set up for capturing that sort of detail.

@rjw62 well.. from looking at the source code, the form builder seems quite flexible, so I think capturing multiple values should be possible - does anyone on this thread know of an example Quest that demonstrates this so I can look in more detail?

In case (2) you might be able to ask "Has the XXX been rebranded as YYY?" although in at least one case, there is more than one YYY answer (LloydsTSB, has de-merged into Lloyds and TSB, so former branches could now be either.)

@rjw62 Does your database have the options for what it's been rebranded to? If so, we could show those as multiple choice (including "other") or failing that just have a text entry box for the new name.

@rjw62 generally I think it would be great to use your data sources in other things and using StreetComplete seems like a great first step. So if you have the time, then opening up an API would be great! I'm happy to discuss that in more detail here, or somewhere else if that's better for you.

[atomoil]

@westnordost actually, looking at https://taginfo.openstreetmap.org/keys/shop#values - might it be acceptable to say "What is the name of this {shop:value} shop?" for most of the values (replacing '_' for ' ') with a filter for the more popular ones where that is just too gross/weird (eg supermarket, hairdresser, optician, mall)

[matkoniecz]

@atomoil Note that this program is available not only in English.

[westnordost]

@westnordost is there an example in StreetComplete already of loading external data in CSV or JSON (or any format)? Or would that be a brand new feature to add?

For the oneway quest, something similar is done. See this folder:

• TrafficFlowSegmentsDao - parse JSON from API
• WayTrafficFlowTable, WayTrafficFlowTablesHelper, WayTrafficFlowDao - persist and access (this) downloaded data into a local database to be used offline

noname is a different thing. I think there is a misunderstanding. What actually blocks the "What is the name of this shop?" quest, is, that it is necessary to show to the user what kind of a shop it is. E.g. "What is the name of this florist?".

@westnordost ah, I see... so am I right in saying that the challenge would still be providing good quality localisation of the question to cover the different types of shops? Actually.. where is the localisation handled in the source code? All I can see in the code is english, so does something swap it out? (forgive me if that's a super dumb question, I'm not an Android expert)

Sorry, I am not sure what you are driving at. Generally, in Android, localized strings are kept in strings.xml. But it is not acceptable to add localization for any possible kind of shop to this app. This kind of shop(/amenity/...)-key to localized name dictionary should, as said, be a separate project.

@westnordost actually, looking at https://taginfo.openstreetmap.org/keys/shop#values - might it be acceptable to say "What is the name of this {shop:value} shop?"

No, because this app is aimed OSM beginners and other people who don't know the tags (or maybe not even what tags are). So, StreetComplete users should not be exposed to the raw OSM tags. As said, what is necessary is a dictionary from tag(-combination) to denomination.

[atomoil]

For the oneway quest, something similar is done.

Great, I'll take a look :-)

Actually.. where is the localisation handled in the source code?

So sorry for my confusion, I'm not used to Android Studio, and didn't realise it shows a preview of localisation strings within the source code.

Ok, from the discussion on #309 it seems there are 2 approaches to localisation:

• a solution that elegantly handles localisation, managing gender and pluralisation and can ideally work for any OSM project. Sadly this solution is sufficiently complex that it's not been solved yet.
• a solution that is slightly clunky and inelegant and doesn't handle gender or pluralisation at all, but could be realised in a shorter period of time.

(have I missed any others?)

I'm very interested in the complex localisation solution, but I know that I don't have the time or energy to dedicate to it. 😭

So - are you at all open to using the less elegant version for a shops quest?

Having looked at the iD source, it seems it would not be complex to parse their https://github.com/openstreetmap/iD/blob/master/dist/locales/ data and extract the strings we need - the question would be how to expose that data to StreetComplete but... it's not worth continuing this conversation if the simplified localisation approach is not acceptable to you.

[westnordost]

Sorry, I wanted to answer yesterday, but I had no time after all.

Yes, that would be acceptable. Taking care of all those grammatical cases it just too complex to deal with.

I envision it like this:

1. Write a program in a language of your choice that scrapes the mentioned iD directory for all the languages for the data we need for the dictionary and save them again in JSON in an eligible format. The scraped data could go into one repository.
2. Write a library that runs on a JVM (Java or Kotlin) that reads these JSON files and provides this data like a dictionary. I could help there or even write this library myself, as I got quite specific requirements and ideas generally how it should look like, see below.
3. (Optionally:) write an npm module for Javascript.
4. (Optionally:) write a small cool web-app to showcase the dictionary (@ENT8R ? ;-) )
5. (Long-Term-Goal:) If the iD data is not sufficient to cover all cases (mentioned below), try to convince iD-guys to instead of maintaining their own translations for this, source it out by importing the (our) npm module for that and create a new transifex (or other) project for that in the more general data structure we need. Then, step 1 can be scrapped.

So, regarding the (Java) library:

• given a name or synonym, it should provide a tag(-combination)
• given a tag(-combination), it should provide the name (and synonyms?)
• it should be possible to provide a tag-combination and get the name for that. For example, a amenity=social_facility + social_facility=assisted_living + social_facility:for=senior is known as a _retirement home_*
• depending on the provided tag-combination, it may occur that the result is actually a list of names. For example for amenity=restaurant + tourism=hotel. So, that list should be ordered in a meaningful way.
• depending on the provided name, it may occur that the result is actually a list of tag-combinations because the name is not unambiguous. Famous example: "bank" and "bench" translates to the same word in German. These should also be sorted/grouped in a meaningful way. For example, sort matches with the name further up than matches with a synonym

Whaddya think?

* If this is possible with the source data. I think maybe not. That would be the long-term goal anyway.

[westnordost]

(Btw, anyone knows where taginfo gets the descriptions from?)

[matkoniecz]

(Btw, anyone knows where taginfo gets the descriptions from?)

OSM Wiki, ValueDescription template in case of tag values - see https://wiki.openstreetmap.org/w/index.php?title=Tag:shop%3Dconvenience&action=edit

[westnordost]

Actually, I am digging up right now a project I started 5 years ago where already 80% or so of what we need is done, so stay tuned.

[rjw62]

In case (2) you might be able to ask "Has the XXX been rebranded as YYY?" although in at least one case, there is more than one YYY answer (LloydsTSB, has de-merged into Lloyds and TSB, so former branches could now be either.)

@rjw62 Does your database have the options for what it's been rebranded to? If so, we could show those as multiple choice (including "other") or failing that just have a text entry box for the new name.

It doesn't include this in a structured way at the moment, but it would be simple enough to add and then expose via an API.

@rjw62 generally I think it would be great to use your data sources in other things and using StreetComplete seems like a great first step. So if you have the time, then opening up an API would be great! I'm happy to discuss that in more detail here, or somewhere else if that's better for you.

It shouldn't be too difficult to do. The API would need to accept a bounding box and then output the OSM object type/ID, OSM version number, lat/lon, current name, closed/rename flag, and possible rename list. How big a bbox would it need to accept, as I might want to limit it? Have I missed anything from the output? Would you want any/all of the OSM object's tags? The data in Ghosts is only updated about once a week (it's certainly not live). Would that be a problem?

For closures, I would imagine a quest could ask a question such as:

• Is this branch of "[current name]" still present?
  
  
  
  • Yes
  
  
  • No
  
  

Although I'm not sure what you'd do if the answer is yes to mark the OSM object, and it would be useful to be able to record info about what the place is now if the answer is no. Abandoned/disused would be relatively easy to do automatically, but if the site/building has been re-purposed, it would be harder.

• Has the branch of "[current name]" been renamed/rebranded?
  
  
  
  • Yes it's now "[possible rename 1]"
  
  
  • Yes it's now "[possible rename 2]"
  
  
  • Yes, but it's something else ...
  
  
  • No, it's still "[current name]"
  
  

If it's one of the suggested renames, then you should be able to just update the OSM name=* without changing the other tagging. (Except, in the case of petrol stations, sometimes the name to change is in brand=* instead.) If it's changed to something else, then it may need a change of amenity=* or shop=* tag too, which would be more difficult to handle.

[atomoil]

Yes, that would be acceptable. Taking care of all those grammatical cases it just too complex to deal with.

Great, let's do this 😎

I envision it like this:

1. Write a program in a language of your choice that scrapes the mentioned iD directory for all the languages for the data we need for the dictionary and save them again in JSON in an eligible format. The scraped data could go into one repository.

I'm happy to make this and use node / Javascript. I'll create something now and get my head around the iD format.

The real question is what the output format should be, especially how to handle the tag-combination aspect. Have you got any particular ideas for this already?

1. (Optionally:) write an npm module for Javascript.

Happy to make an npm module once I have something up and running - sorry to be dumb, but how would the npm module get integrated into StreetComplete?

[westnordost]

I am writing the Java library at the moment.

The real question is what the output format should be, especially how to handle the tag-combination aspect. Have you got any particular ideas for this already?

The output format of that little scrape-iD-locale-jsons-script should be like this (example for just one item):

[
  {
    tags: {
      "shop": "party"
    },
    names: ["Party Supply Store"],
    keywords: ["balloons", "costumes", "decorations", "invitations"]
  },
]

This format ensures forward-compatibility for when the primary datasource is at some point not scraping the iD locale jsons anymore:

• format enables defining names for places with multiple tags (see my example above)
• format enables defining a range of synonyms for the name

Edit: Not sure if it is possible to write a (Client-Side?) Javascript to do this. What would be your tool of choice?

[westnordost]

Happy to make an npm module once I have something up and running - sorry to be dumb, but how would the npm module get integrated into StreetComplete?

Not at all. The Javascript library (npm module == Javascript library?) + little demo page would not be for StreetComplete but to showcase the library so other web-applications in the OSM-sphere get interested into using it. A further spread in this would mean a further canonicalization of localizing names which I think is a good step forward.

[atomoil]

The output format of that little scrape-iD-locale-jsons-script should be like this (example for just one item):

How does this look? https://github.com/atomoil/osm-tag-to-name/blob/master/data/en-GB.json

It's only a start - it's loading the data locally and it's not yet handling items in iD that have 3 parts to them (eg amenity/vending_machine/food) - looks like I need a lookup to handle what the key for tag food needs to be. (If anyone knows different, then do shout!).

[westnordost]

That looks already pretty good! Looks like I should hurry up with my Java library (cause I would like it to serve as a reference implementation for the JS library, no use if the implementations are completely different on different platforms, I think).
I'll upload a WIP tomorrow or so to another github repository. I am still unsure how best to integrate the name-suggestion-index into the library.

You can reliably resolve the 3 part category strings like leisure/fitness_station/horizontal_ladder without any hardcoded logic by traversing (or looking up) the directory structure of the iD repository. It is the tags key in this file:
https://github.com/openstreetmap/iD/blob/master/data/presets/presets/leisure/fitness_station/horizontal_ladder.json
Also, in the example data file, the keywords array is empty. I noticed that in the source data, the name is sometimes duplicated into the terms. Your script could fix this during parsing.

Looking at this file, I am wondering about the terms mentioned there. Why do they not show up in en-GB.json? Are those in en-GB.json supposed to be merged with the default values?

[westnordost]

Basically there will be one NamesDictionary for each Locale (created on-demand) which will have a public interface like this:

/* get dictionary entries by tags (fast) */
List<Entry> get(Map<String, String> tags) 
/* get dictionary entries by exact name (currently slow-ish, working on it being fast) */
List<Entry> get(String term) 
/* find dictionary entries with possible search options like ignore-case, ignore-diacritics (accents), search-also-keywords, search-for-term-anywhere-within-strings, etc. (slow-ish) */
List<Match> find(String term, int flags) 

and Entry is

class Entry
{
    List<String> names;
    List<String> keywords;
    Map<String, String> tags;
    int count;
}

Still todo:

• performance improvements when searching entries by name (add indices)
• memory improvements (intern-ing strings)
• add feature: being able to limit the results in find() - also as performance improvement

Open problems:

• Not sure how to merge this here with name-suggestion-index (= brand name dictionary). In the end, it would be best to combine it into one NamesDictionary with the localized names but the localized names are created for each (user) language while the brand names are dependent on the actual location (country) of a feature. Can you follow? Do you have thoughts on that?

[westnordost]

I added the library now at https://github.com/westnordost/osmnames , it's WIP.

[atomoil]

It doesn't include this in a structured way at the moment, but it would be simple enough to add and then expose via an API.

hey @rjw62 sorry for not replying sooner - looks like we can move ahead with the 'shops with missing names' quest, so I don't think we need to pursue exposing the Ghosts data via API for now.

Having said that, your projects / data sources are really awesome, so I think it's worth discussing how to make them consumable by other projects/services and hopefully be used for a future Quarterly Projects. Do you have a GitHub repo for any of your projects which might be more appropriate for a discussion, or perhaps we could switch back to email? I'm happy to discuss wherever is best for you.

[atomoil]

You can reliably resolve the 3 part category strings like leisure/fitness_station/horizontal_ladder without any hardcoded logic by traversing (or looking up) the directory structure of the iD repository. It is the tags key in this file:
https://github.com/openstreetmap/iD/blob/master/data/presets/presets/leisure/fitness_station/horizontal_ladder.json

ah... interesting... I was wondering how to best get at the iD data and looks like I'll need to access quite a few of the files, so I think it might be best to use git to check out the entire repo (and of course there is a npm module for just that purpose :-)).

Ok, I'll get on with that and report back when I have something more to show.

Oh - one more question - do you know how osnnames or streetcomplete will get the JSON files from within the osm-tag-to-name repo? Is there something I can do to expose them in a particularly useful way?

[westnordost]

This data is updated by iD maintainers and contributors all the time, so a solution that makes it easy for users to always have the most current version would be nice.

The solution I can come up with right now is:

1. you ensure that the JSON files are updated every time anything in the relevant directories of the iD repos has been changed. Not sure how to do this, a commit hook? A nightly cron job?
2. users of the library can include the JSON files in the appropriate directory via a git submodule. This means that the generated JSON files should be in an own repository that contains only those and nothing else.

[matkoniecz]

every time anything in the relevant directories of the iD repos has been changed

Or maybe to this only once iD releases? To reduce repository churn and to have supposed to be stable versions?

[rjw62]

(@atomoil No problem -- I'm not sure that my Ghosts are completely suitable for Street Complete anyway. No repos for my code, as the quality is probably somewhat embarrassing. But if you want to chat over email, my details are at https://robert.mathmos.net/ )

[westnordost]

@atomoil I found out, that all that information about presets is aggregated into this JSON file. Even the data from the name-suggestion-index is included: https://github.com/openstreetmap/iD/blob/master/data/presets/presets.json

That all those details are available in this one file actually topples my current implementation and vision of how the consumable data structure should look like a bit. Perhaps it does rather make sense to keep this "category" string there as a link to between tags and localization, so that per language, not all tags are repeated.
I need to rethink this a bit, please stay tuned.

[westnordost]

I think maybe this format would be better:

1. just reduce presets.json but keep the same format:

<category id>:
  tags
  geometry <- could be used to further limit the results of a lookup
  searchable <- default true. If explicitly false, should not be found in a lookup from name -> tags
  matchScore <- helpful for sorting results
  name <- unlocalized name, keep only if "suggestion": true (=brand names)
  addTags <- only used for brand names. Tags that should be returned when searching from name -> tags but are not required for tag -> name search.
  countryCodes <- (usually) only used for brand names. Limit by country

1. also keep the same format for [language-code].json but only keep the data in presets. I stepped back from the idea to have an array of names to accommodate synonyms a bit because the source data doesn't have it (I created a ticket suggesting that though)

<category id>:
  name
  terms

[atomoil]

I found out, that all that information about presets is aggregated into this JSON file. Even the data from the name-suggestion-index is included: https://github.com/openstreetmap/iD/blob/master/data/presets/presets.json

ah! good find - except... isn't that all in English? Or do you mean to use that as the source and combine it with the files in https://github.com/openstreetmap/iD/blob/master/dist/locales/ ?

Sorry, I think I might be a bit confused by what needs to be done 😭

[RobJN]

I'm not sure that my Ghosts are completely suitable for Street Complete anyway.

I think it would be great. It's something new for Street Complete (using external data like this) but unlocks a lot of potential.

[matkoniecz]

using external data like this

Already done with oneway quest - see https://github.com/westnordost/StreetComplete/issues/1022

[westnordost]

@atomoil Sorry for the changing requirements. But let's split this from this ticket, see https://github.com/westnordost/osmnames/issues/2

[westnordost]

This is now un-blocked, as StreetComplete will soon be able to denominate elements without a name.
So if you want, @atomoil , you can start working on it.

[atomoil]

After a bit of radio silence I'd like to pick up integrating @rjw62's APIs into a StreetComplete quest and of course have some inevitable questions for @westnordost or anyone who can answer. Hopefully you understand the sentiment of my questions, so feel free to answer the question I should be asking, instead of the ones below 😎

Can you point me to the source code for any (all ideally) quests that load external data (so I understand how that's implemented in StreetComplete / Android / Kotlin).

Do you have a particular way you'd like external data to be loaded? (I might be able to deduce this from the code, but perhaps it's best to ask for clarity)

What's the ideal / preferred relation between external data sources and communication with overpass? (and perhaps also - what other options are acceptable. what's totally unacceptable 😱)

Regarding the API - do you have any rough stats for the amount of traffic StreetComplete generates for the API(s) it consumes? (just to aid in appropriate server provisioning)

[westnordost]

Sure.

Can you point me to the source code for any (all ideally) quests that load external data (so I understand how that's implemented in StreetComplete / Android / Kotlin).

The Oneway Quest does download data from an external source. Also, it persists data to a local database.

https://github.com/westnordost/StreetComplete/tree/master/app/src/main/java/de/westnordost/streetcomplete/quests/oneway

Do you have a particular way you'd like external data to be loaded? (I might be able to deduce this from the code, but perhaps it's best to ask for clarity)

If you mean in what erm... shape the data should arrive, I am tending towards that maybe it is easier if the app would just download a list of brand names that should be obsolete and then put that into an Overpass query itself. After all, that is what rjw62's website does behind the curtains too. Also, it would/could then simply be a static file, no API.
I just wished there was a worldwide maintained list available, not just for the UK. Perhaps this could be a new github project in the future because this would also be interesting for validators like Osmosis, KeepRight and the one in iD.

What's the ideal / preferred relation between external data sources and communication with overpass? (and perhaps also - what other options are acceptable. what's totally unacceptable 😱)

External location data should be treated only as a hint because it may be old, while Overpass is "the truth" as it is usually only a few minutes behind the OSM Api. For example, for the Oneway data, the ImproveOsm data is used for preselection, but the real data is then downloaded from Overpass and re-checked.

Regarding the API - do you have any rough stats for the amount of traffic StreetComplete generates for the API(s) it consumes? (just to aid in appropriate server provisioning)

I don't have statistics, but a smallish webspace will be more than enough.

[westnordost]

I just wished there was a worldwide maintained list available, not just for the UK. Perhaps this could be a new github project in the future because this would also be interesting for validators like Osmosis, KeepRight and the one in iD.

Turns out @matkoniecz already proposed this for the name-suggestion-index: https://github.com/osmlab/name-suggestion-index/issues/2415

[matkoniecz]

Though my proposal for implementation was refused in https://github.com/osmlab/name-suggestion-index/pull/2562 and I am not enthusiastic about the alternative, so I am not working on this.

[westnordost]

The alternative proposed or a alternative you had in mind?

Am 17. Mai 2019 15:20:07 MESZ schrieb Mateusz Konieczny [e-mail]:

Though my proposal for implementation was refused in
https://github.com/osmlab/name-suggestion-index/pull/2562 and I am not
enthusiastic about the alternative, so I am not working on this.

[matkoniecz]

My PR was closed with

I'd really prefer for us to pick this information from Wikidata somehow

and for multiple reasons I would prefer to not rely on this kind of data in Wikidata (for reasons described in https://github.com/osmlab/name-suggestion-index/pull/2562#issuecomment-481753529 )

[westnordost]

This ticket did not seem to lead to anything. A PR for "UK Ghosts" is welcome, though it would probably be more worthwhile to have it work for the whole world and not only the UK