Elasticsearch: geo_shape queries not working anymore after upgrade to 6.6

[consulthys]

Elasticsearch version: 6.6.0 on ES Cloud

Description of the problem including expected versus actual behavior:

After upgrading from 6.5.4 to 6.6.0 on ES Cloud, none of our geo_shape queries work anymore. Moreover, it is not clear from the 6.6 breaking changes what needs to be done to make pre-6.6. shapes compatible with 6.6.

Steps to reproduce:

1. Create an index on a pre-6.6 ES containing a field of type geo_shape

PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "my_shape": {
          "type": "geo_shape"
        }
      }
    }
  }
}

1. index a shape

PUT test/doc/1
{
  "my_shape": {
    "type": "Polygon",
    "coordinates": [
      [
        [
          -118.37326599344239,
          34.082247752657956
        ],
        [
          -118.37326637421617,
          34.082384870862526
        ],
        [
          -118.37369602509777,
          34.08238389978479
        ],
        [
          -118.37369564361956,
          34.082246781581595
        ],
        [
          -118.37326599344239,
          34.082247752657956
        ]
      ]
    ]
  }
}

1. Find the shape using a geo_shape query

POST test/_search
{
  "query": {
    "geo_shape": {
      "my_shape": {
        "shape": {
          "type": "point",
          "coordinates": [-118.37345570325851, 34.08231046148952]
        },
        "relation": "contains"
      }
    }
  }
}

1. Migrate to ES 6.6

2. Run search from step 3 again and see that there is nothing being returned.

[oeelvik]

I have the same problem but i was thinking it might be related to me using points_only in my mapping, but you can reproduce with no additional parameters than "type": "geo_shape"?

[oeelvik]

https://discuss.elastic.co/t/elasticsearch-6-6-0-geo-shape-query-on-old-documents-not-working-after-upgrade/166962/2

[consulthys]

Yes @oeelvik, in our mapping we have just type: geo_shape and geo_shape queries stopped working just after the upgrade.

[elasticmachine]

Pinging @elastic/es-analytics-geo

[jmaupin82]

I ran into this as well. The mappings I made with a non-default geo mapping still worked after the upgrade but all geo_shape default types were broken until I reindexed to a new index. That might be a workaround in the mean time depending on the size of your index.

To elaborate the mapping type : { "type" : "geo_shape", "tree" : "quadtree", "distance_error_pct" : 0.01 }
Still allowed geo queries after the update to 6.6.0 but "type": "geo_shape" alone did not allow geo queries after the update to 6.6.0.

[consulthys]

@jmaupin82 thanks for the feedback.
Reindexing is not an option for us right now as it would take days. Hopefully, 6.6.1 will fix this issue.

[polyfractal]

Hey all, thanks for the bug report! I can confirm this is a bug, and happens on a non-cloud install as well. We did some digging and the problem, while it coincides with the new BKD-based geoshapes, it's actually a tangential bug.

Pre-6.6, the default geo_shape tree was a geohash. For a long time we have wanted to change that to quadtree, because it's just a better default for most people. When we merged the BKD geoshape work, we also switched the default of the legacy prefix tree over to quadtree.

The bug is that we're not checking the index version when constructing a default legacy prefix tree, so the pre-6.6 geohash tree is being interpreted as a quadtree... which means no terms are matching and you get empty hits.

The fix should be pretty simple (load the correct default tree type depending on version) and I don't it will require any reindexing. We should also find out why no exceptions were thrown when interpreting the geohash as quadtree and add some if they don't exist.

Sorry for the hassle! We're hoping to get this fixed soon :)

[consulthys]

Awesome, thanks @polyfractal for looking into this! Do you think this fix will/could go into 6.6.1?

[nknize]

I was able to reproduce the bug and @polyfractal is correct. The default tree for the Prefix Trees was switched to quadtree and the backcompat version check was not added. We should be able to get this in for the bug fix release.

[nknize]

Merged and backported to all versions; including 6.6.1.

[consulthys]

Thanks guys! @nknize does this mean that pre-6.6 geo shapes should work out of the box on 6.6.1 again, or do they need reindexing?

[iverase]

@consulthys it should work out of the box, you should not need to reindex.

[consulthys]

Thanks for confirming @iverase, I tried again to reproduce my initial steps and it indeed worked out. Good job guys!

[iverase]

@consulthys I have tried to reproduce the issue with the steps posted in this issue and I was unable to get it anymore. I guess you might be experiencing something different, Could you share the steps to reproduce the issue this time?

Upss I missed your last comment, thanks for confirming!

[consulthys]

@iverase see my latest comment ;-)