Fightpandemics: Implement & leverage geocoding to fulfill MVP requirements

[joshmorel]

Geocoding (and reverse geocoding) is required in several places for the MVP. The priority is the front-end web app but there may be similar needs for this service from the native mobile apps and in some cases the backend itself (e.g. post import from airtable).

A resolution to this issue include implementing this geocoding feature as well as potentially implementing a consistent means to it’s use in the front-end web app. Alternatively, the later could be a addressed as second issue once complete. A similar issue is #164 (although this only accounts for one place where geocoding is required on the front-end).

Two places are organization & individual profile creation. The backend model requires a structured location to facilitate filtering of posts created by that organization or individual, e.g:

address: 1600 Amphitheatre Parkway, Mountain View, CA
city: Kingston
coordinates: [37.4267861,-122.0806032]
country: United States
state: California
zip: 94043

This needs to be suggested from a free text address entered at profile creation/update for individual or organization by geocoding from a service such as Google Maps (potentially one or more depending on request limit considerations). When the suggested address is selected the structured location object as above is sent to the backend:

Additional places where geocoding is required include filtering the /feed. In such a case we also need to get the structured location if the user opts to use the browser’s geolocation API (which provides a lat/lon). In this case the coords need to also be reverse geocoded (we do currently have a reverse geocoding service using the Python reverse-geocoding library but it may not be accurate or granular enough for our requirements) to produce the same structured location object.

This is similarly required when completing the multi-step wizard to find posts to /offer-help or /need-help

[Naraujo13]

First of all, thanks @joshmorel for compiling and organizing all that info that was scattered around issues, slack channels, and team members!

Second, this will be a little bit long, but I think it will help us make a decision on which route to take.

---

What do we need?

Given the outlined requirements compiled by @joshmorel and our application, I think we only need two geo functionalities:

• Reverse Geocoding: given a coordinate, get the full address in a structured manner. This would be used to get the full location for a user when a user shares his location at registration or at posts filters.

• Geosearching: given a free text input, search for similar locations/addresses, and return a list of the most similar for the user to select one of them. This would be used at registration and feed when the user doesn't share his location and manually inputs the information.

Given that we would use the geosearch to display options for the user and that he would have to select one of them, we wouldn't need a geocoding specific point given that when the user selects one of the listed locations that would already be full (given that geosearch would have sent that).

Current Approach

Overview

• Python container that runs the offline geocoder Python lib
• It is implemented as a single endpoint to reverse geocode with the route /geo-service that receives two params, lat and long
• The reverse-geocoding lib uses checks the nearest city by Euclidian Distance to the given coordinates and returns that object
• The reverse-geocoding lib is populated with a dataset with cities with more than 1000 inhabitants taken from GeoNames

Location Structured

Current fields are:

• Country
• State
• County (only US I think)
• City

Sample

{
  Lat: 34.086681
  Long: -118.27023
  City: Silver Lake
  County: Los Angeles 
  State: Californi 
  Country: US
}

Strong Points

• No external dependencies
• No credits needed/no API costs
• Low latency

Weak Points

• Possible Reverse-Geocoding Errors: Reverse geocoding to the nearest city by Euclidian Distance WILL cause errors for any point that is near the city borders. That can cause errors from attributing the point to the wrong city, to even the wrong country. Given two cities at the border between countries, changing between them will change basically all the location info. An example case is given in this issue at the lib repo, where a Switzerland coordinate is attributed as an Italian city.
• Only up to city granularity: the reverse-geocoding lib returns only up to city granularity. It is returned country, state, county (only US I think) and city. Therefore we won't have neighborhood, zip code, or even street address.
• No geosearch: the current service does not support geosearch or geocoding. For that we would still need to fall back to Google Maps or other geolocation services.
• Processing Overhead: our servers would handle the overhead of executing the geolocation search, tbh I don't know about this specific Python lib implementation to know if that will be a lot or not.

Analysis

Architecture

I think that having the service separated from backend/frontend is the way to go since we can decouple the geolocation from the rest of the app logic, making it easier to change the geoservice in the future (we could migrate to a self-hosted or change APIs without any impact to the application).

My only question point is if this needs to be a Python application (most geoloc resources are in Python) or we could use Node to keep it coherent with the rest (if using external APIs, that should be enough).

Reverse Geocoding Endpoint

The reverse geocoding endpoint might fall short of our expectations given that it only goes to city-level, without covering neighborhood or street addresses. The possibilities for errors for addresses near borders may be problematic too.

Alternatives

As alternatives, we could use geolocation APIs, such as Google Maps, TomTom and etc..
This would make the reverse-geocoding more accurate and we would have all the info we need (and more, being able to get buildings close-by and etc.). That, of course, would come with the problem of the APIs limitations on the number of requests, implying on extra cost for the API usage.

Geosearching

This one isn't implemented yet, but I think it should be an additional endpoint at the geolocation service.

This way, the endpoint would receive the user input, call our geolocation API, and return a list of locations that are a possible match for that. These locations would follow our Location object defined at the Data Model.

This way, the frontend could call this endpoint to populate the list and just pass the select Location object to the backend API after that.

Opinion

The current service could satisfy our reverse-geocoding needs if we accept to have only information to the city-level and some error cases.

Given that we would still need to implement geosearch endpoint, using one geolocation API, I think we could redo this service with only two endpoints geosearch and reverse-geocode, using some API to fully explore our data. Using the API should be really straight-forward and simplify this process.

That approach has a single problem: How would we handle the APIs request limitations?
For that I see two options:

• Get more credits from one geolocation API (doesn't need to be google, there are several more that could fulfill our needs).
• Rotate between different services. This would make our geolocation service a bit more complex since it would be responsible for rotating the requests between different geolocation APIs, trying to distribute our requests between them, and leverage the most of our free quota. This way, we would only fallback to the paid API as a last resort. I've tried this before and it works but adds latency to the service (since it needs to hit more than one service sometimes).

With these suggestions, we would have the following location structure, with the following pros and cons

Location Structured

Current fields are:

• Country
• State
• City
  
  
  
  • Neighborhood
  
  
  • Zip
  
  
  • Street Address
  
  

Sample

 {
  "coordinates": [
    35.26,
    -32.83
  ],
  "country": "Lorem",
  "city": "Lorem",
  "neighborhood": "Lorem",
  "address": "Lorem"
}

Strong Points

• Best accuracy possible, with fewer errors at reverse-geocoding
• Information granularity up to street level
• Geosearch + Reverse-Geocoding consistent with last map database
• No overhead for processing any geolocation at our side (all processed on the API side)

Weak Points

• External dependency
• Added cost of using external APIs
• Added complexity if using rotation logics (but nothing too big)

---

Well, that's my two cents. what do you think?

[joshmorel]

The approach sounds good @Naraujo13 thanks for the analysis. I'm on board with the two endpoints. I'm fine with Python for speed of development and we can migrate functionality to Node.js backend later if it makes sense. @robinv85 does this sound okay or do you think this should be done in Node.js backend to begin with?

[robinv85]

Architecture

I think that having the service separated from backend/frontend is the way to go since we can decouple the geolocation from the rest of the app logic, making it easier to change the geoservice in the future (we could migrate to a self-hosted or change APIs without any impact to the application).

My only question point is if this needs to be a Python application (most geoloc resources are in Python) or we could use Node to keep it coherent with the rest (if using external APIs, that should be enough).

#

Yeah this is currently the case, right? It's decoupled from the Node API, as a separate service, so whether we use a self-hosted service or replace it with an external one, this would stay the same I think?

I also found a Go service for geocoding which was supposed to be faster but it didn't work that well, I couldn't get responses for all the coordinates so the Python one seemed to work quite well.

I'm also thinking, it could be possible to update the data-source to have more accurate data for the reverse-geocoding? It might not be very costly to buy that. Similar to Maxmind GeoIP databases? Those have a free version and premium ones have more accurate data.

Alternatives

As alternatives, we could use geolocation APIs, such as Google Maps, TomTom and etc..
This would make the reverse-geocoding more accurate and we would have all the info we need (and more, being able to get buildings close-by and etc.). That, of course, would come with the problem of the APIs limitations on the number of requests, implying on extra cost for the API usage.

So is this only for reverse-geocoding that we need it? Can we use our own service for geocoding, and an external service for reverse-geo-coding and geo-search? This way already saves some API requests.

The idea about using multiple services to rotate between them is also interesting.

Can we also combine both the reverse-geocoding library and use it if we have a city, but if we don't then fall back to another service instead? That way can also save us a bunch of external requests...

cc @joshmorel @Naraujo13 just tagging you so it doesn't get lost in between all the Github notifications...

[Naraujo13]

Architecture

I think that having the service separated from backend/frontend is the way to go since we can decouple the geolocation from the rest of the app logic, making it easier to change the geoservice in the future (we could migrate to a self-hosted or change APIs without any impact to the application).
My only question point is if this needs to be a Python application (most geoloc resources are in Python) or we could use Node to keep it coherent with the rest (if using external APIs, that should be enough).

Yeah this is currently the case, right? It's decoupled from the Node API, as a separate service, so whether we use a self-hosted service or replace it with an external one, this would stay the same I think?

I also found a Go service for geocoding which was supposed to be faster but it didn't work that well, I couldn't get responses for all the coordinates so the Python one seemed to work quite well.

I'm also thinking, it could be possible to update the data-source to have more accurate data for the reverse-geocoding? It might not be very costly to buy that. Similar to Maxmind GeoIP databases? Those have a free version and premium ones have more accurate data.

Alternatives
As alternatives, we could use geolocation APIs, such as Google Maps, TomTom and etc..
This would make the reverse-geocoding more accurate and we would have all the info we need (and more, being able to get buildings close-by and etc.). That, of course, would come with the problem of the APIs limitations on the number of requests, implying on extra cost for the API usage.

So is this only for reverse-geocoding that we need it? Can we use our own service for geocoding, and an external service for reverse-geo-coding and geo-search? This way already saves some API requests.

The idea about using multiple services to rotate between them is also interesting.

Can we also combine both the reverse-geocoding library and use it if we have a city, but if we don't then fall back to another service instead? That way can also save us a bunch of external requests...

cc @joshmorel @Naraujo13 just tagging you so it doesn't get lost in between all the Github notifications...

Yep, currently, it is a separate service in Python.
I was thinking and we don't need it to be an entirely separated service like that, simply decoupling the geocoding logic to specific endpoints at our API would suffice for that. I just meant to avoid having the location handling code scattered around many different endpoints and even at the frontend.

Anyway, our current backend service does only reverse geocoding. We would still need to use the API for geosearch. Also, no need for geocoding since we would use the geosearch for those (after that the user selects a location from the list, so no need to geocode that).

About updating the database, we could do that and it may solve some cases if the database is good enough, but that would still make us limited to the city level. With an API such as Google Maps, TomTom, or any other, the approximation is made at street level making it a lot less error-prone since it has more points to approximate. That would reduce our API usage because reverse-geocoding would be local, and we would only use credits for geosearch.

Anyway, I think the simpler implementation would be to add two endpoints, maybe /geosearch and reverse-geocode, using the external APIs to do the hard work.
I would propose using the rotation between services since I honestly we won't have that many requests for geosearch or reverse geocode.

I think our API usage will be low at first since we are storing the locations at our database after user signup and after that reusing all at Mongo. This way, we would only make requests for external APIs at user signup or when a user wants to filter by a location different than the one at his profile (what I think won't be that common). Still, most APIs have a limitation of 1 req/sec or 2500/day, so with 4-5 of those most of our requests will fall into the free quota.

This should be really straightforward and simple to implement and after that backend and frontend could use that to make the geo work.

In the future, as we scale, probably the cheapest solution would be to self-host a full geo service such as Nominatim. It is open source and has docker images, this way we would hit this service for all geo needs and zero our external API usage (but add the cost for the server, what is worth at a higher scale).

[Naraujo13]

Also about leaving the geolocation service in Python x doing it in Node: for consistency and to make that simpler, I would prefer Node, but tbh I have a good part of the rotation code between APIs done in Python, so that would be faster haha

[robinv85]

Yeah @Naraujo13 I think it's best to consider the reasons why you would pick one or the other, if the logic fits well in our Node app just to rotate between the different services then I would prefer that instead of putting an extra Python service for just that... we also have more developers that can probably help with the Node part if you;d like and lack some time at the moment.

[joshmorel]

I am closing this as we merged #649

Issue #164 will be closed when completing the front-end integration.