Orchardcore: Lucene and API (REST & GraphQL) Geo distance search

[stevetayloruk]

Hi,

Is it possible at the moment to do a geolocation search with Lucene/Elastic Search (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html) against some content items that would have a long/lat properties and access either via the REST or GraphQL endpoints.

Thanks

Steve

[sebastienros]

We don't support it right now, but it should be simple to add:

• https://www.elastic.co/guide/en/elasticsearch/reference/6.2/query-dsl-geo-distance-query.html
• https://www.nuget.org/packages/Lucene.Net.Spatial/

The idea would be that a new LocationField would contain Latitude and Longitude as double properties, and in its index handler we store these. Then you can provide different editor to change how locations are edited in content items, the simplest one being a set of textboxes. This is the very simple part.

Then for querying, we need to extend the DSL parser to handle the geo_point filter, and convert it to a lucene query, probably using things from the Spatial nuget package. Here is some code I found on the lucene.net repository, but not it's not using the nuget package, just custom math: https://github.com/apache/lucenenet/blob/1fdea4e86da9fc0d0fdc3481e8548e3a457eb8b7/src/Lucene.Net.Demo/Facet/DistanceFacetsExample.cs

Regarding GraphQL, we don't support Lucene queries "directly", unless if you create a Lucene Query in the queries module that is parameterized. Then it can be requested from graphql, but it won't be better than the rest api. We could also look into how to handle these queries in SQL. Or make a new type of GraphQL query that targets lucene indices.

[sebastienros]

Note that we could create a simple graphql query that would take an elastic search dsl query as the main argument, and provide the same extensions for filtering the returned properties because we would have the same content items. Should be the same logic.

@jrestall Do you see what I mean? we just accept a json document in the query, that is the exact lucene dsl that we already accept in lucene queries. We just need to define the schema for it, which should be "static". Then we call the search service to get content item ids. At that point I assume we can reuse all the classes we already use to render the parts, expansions, ...

The query could be called search.

[stevetayloruk]

Thanks @sebastienros. If its not tackled by Jan I might have a look at doing it. I'll skype you if i do to ensure no one else has.

Thanks

Steve

[petedavis]

I have already sort of started on this.

https://github.com/petedavis/OrchardCore/tree/spatial

Needs work the way that the items are indexed spatially.

But a parameratized query works

// { top:-33, left:137, bottom:-35, right:139 }
{
    "query": {
        "bool" : {
            "must" : {
                "match_all" : {}
            },
            "filter" : {
                "geo_bounding_box" : {
                    "BlogPost.Location" : {
                        "top_left" : {
                            "lat" : {{top}},
                            "lon" : {{left}}
                        },
                        "bottom_right" : {
                            "lat" : {{bottom}},
                            "lon" : {{right}}
                        }
                    }
                }
            }
        }
    }
}

[petedavis]

@sebastienros should I create a pull request to open up review/feedback etc?

[scleaver]

I need this too actually. I haven't started on the geodistance part but I have created what I have called an AddressPart. The address part allows a content item to have an address. It has a built in address search which retrieves the latitude and latitude and stores it against the fields in the addresspart.

My intention here was to then incorporate this somehow into queries so that you could in fact submit a latitude and longitude and a distance and retrieve all content items within the specified distance of the specified lat/long. This bit I haven't started working on yet but @petedavis solution looks something along the lines of what I wanted.

If you are interested in the AddressPart I am willing to share it.

[scleaver]

https://youtu.be/5vFy1rjtcSA

Here is a short screen capture of my AddressPart in action if anyone is interested.

[petedavis]

Im thinking along the same lines for geolocation of content @scleaver

I am also looking to have a map display of content like what you would see on a site like https://www.realestate.com.au/buy/in-sydney,+nsw+2000/map-1

And have a pin on a map for each content item.

[sebastienros]

@petedavis I looked at your repo and it looks great, you can create a PR please. Some comments:

• Add unit tests for the new Filter node in the DSL
• Naming: I would have expected Location for the field and the index value type. Minor, you don't have to change if you feel confident about it.
• I didn't see the distance query, just a bounding box. Is that doable?

[scleaver]

@petedavis yes that realestate.com.au example is the sort of functionality it can be used for. Often with any latitude and longitude you need to display an address so that is why I went for a more complex field than just a Location field for storing lat/long values.

Additionally the content editor needs to be able to get the lat/long from somewhere, which is why I incorporated the address autocomplete at the top.

My addresspart also has an additional editor that displays a map in the edit view to allow the content editor to fine tune the location of the pin (unfinished).

Let me know if you would like me to contribute the code for the addresspart or whether the geolocation functionality will use a more simplistic field type.

[sebastienros]

AddressPart is too custom, can't be in the core parts. But some of it might get in. Like the map as a custom Editor for the Location Field. And also as a Display mode.

[scleaver]

Ok. Location field... what are you thinking of here @sebastienros? How do you see it storing the lat/long values? Should it be added to the base field types module? Also, in order to do a map editor, most mapping solutions are going to require an API key - is there a preferred vendor?

[sebastienros]

I am thinking of exactly what @petedavis has done in the repository he linked to.

Not sure what you mean by map editor, but at least we could render the field as a map. We have Display Modes for fields which makes it extensible. If we require a key, then it can come through settings. But I assume rendering a google map at a specific location doesn't require one?

[petedavis]

@scleaver I have added a field editor using leaflet to select a position on a map. That is an example of a custom editor for the field to select lat/long. Given field editors are extensible, we can have the location field as part of core, and then anyone can create a custom editor for a location field.

https://github.com/petedavis/OrchardCore/tree/spatial/src/OrchardCore.Modules/OrchardCore.Spatial/Views

[sebastienros]

Please make a PR. Even if you need help, like for DSL unit tests, or adding more features.

[petedavis]

Hey @sebastienros , thinking about naming, a location can be stored in many formats, including a postal address. Therefore I believe that the field should be a coordinate field as it stores a lat/long coordinate that just one of many ways to describe a location.

Still thinking about distance, but with a coordinate field available it should be possible to add a service to calculate a distance between two coordinates.

Still heaps of geo search DSL's to implement. Hope we can support the distance one.

[sebastienros]

GeoLocation, like in ElasticSearch?

[petedavis]

@sebastienros I have renamed the field to GeoPoint based on https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

@stevetayloruk I have also done a rough implementation of the geo distance query. Only parsed "km" distance units and will work on the other distance units next. See pull request #2725

[stevetayloruk]

@petedavis it would be really nice if we could do this based on an array of locations. For example, a TV is in stock at three retailer locations.

[Skrypt]

@stevetayloruk Yes, at some point I will need to look into this too since I could add geo distance queries to the website I'm working on. The idea would be to return "Machinery" that are within a radius distance from a specific postal code. Could also use browser geolocation.

[douwinga]

@petedavis What work is remaining on your PR? Anything I can help with? Looking for this feature for a site getting released next month.

[Jetski5822]

If you guys have a PR, ill review it.

[scleaver]

@petedavis same as @douwinga - I have a project that is needing the GeoPoint field. Any idea how far off the first release of this feature is? Is there anything we could assist you with?

[douwinga]

I started continuing the work of @petedavis at https://github.com/douwinga/OrchardCore/tree/spatial. So far I just added support for additional distance units. I need to see how the results are sorted yet, but I think there will be work to get the results sorted by distance.

[petedavis]

@Jetski5822 There is an open pull request to start the feedback at https://github.com/OrchardCMS/OrchardCore/pull/2725

@douwinga do you know if there is a way to merge your changes into the same pull request? If it was merged into my branch, then this pull request should be updated.

This hasnt been a priortiy for me lately as is a side project need.

I have started on a map widget to point to a lucene query to get spatial query results displayed on a map.

[douwinga]

@petedavis I was planning on doing a PR to your branch that you then merge which would then update your PR to the main OrchardCore repo.

I am struggling to understand how sorting by distance should work. Did you look into it at all?

[douwinga]

https://github.com/petedavis/OrchardCore/pull/1

We will see if doing a PR to your fork will work. I did end up figuring out a way to sort by distance by setting the score. By default OrchardCore sorts by relevance (score). I am not sure how we would want to handle when a user specifies a sort order for a distance as it will be searched by string rather than score when that happens.

This PR will make it so that searching by bounding box and distance will work in the most common use case, so maybe it is ready to merge into dev?

[petedavis]

@douwinga sounds like a possible solution. I think we should try and implement the elastic api for sorting. https://www.elastic.co/guide/en/elasticsearch/reference/7.0/search-request-sort.html#geo-sorting

When displaying items on a map they dont need to be sorted, so would be ideal if they are only sorted by request.

[scleaver]

@douwinga @petedavis the scenario for sort by distance would be the typical yelp scenario where the user has the option to switch between a list or map view of the restaurants for example. Often the list view will be ordered by distance from your current location or the location you specify. Of course, the map view doesn’t need this but a list of results would.

[petedavis]

@scleaver something like this would work in both cases where the sort is optional:

{
    "sort" : [
        {
            "_geo_distance" : {
                "BlogPost.Location" : {
                    "lat" : {{centreLat}},
                    "lon" : {{centreLong}}
                },
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query": {
        "bool" : {
            "must" : {
                "match_all" : {}
            },
            "filter" : {
                "geo_bounding_box" : {
                    "BlogPost.Location" : {
                        "top_left" : {
                            "lat" : {{top}},
                            "lon" : {{left}}
                        },
                        "bottom_right" : {
                            "lat" : {{bottom}},
                            "lon" : {{right}}
                        }
                    }
                }
            }
        }
    }
}

What should also be able to be done is a distance sort on a query that does not filter on a bounding box. Just sort all results based on distance using any query results provided it finds a BlogPost.Location geo point field to sort on in the above case.