Mastodon: Support Elasticsearch 7.0

[angristan]

Elasticsearch 7 was released about a month ago: https://www.elastic.co/blog/elasticsearch-7-0-0-released.

Currently the elasticsearch gems are pinned at 6.0.2, so Mastodon is not compatible with ES 7.

The gems have been updated for 7.0.0 support: https://github.com/elastic/elasticsearch-ruby/releases

[angristan]

I guess more work is needed, since mapping types have been removed.

mastodon~/live> env RAILS_ENV=production ./bin/tootctl search deploy
Resetting StatusesIndex
Traceback (most recent call last):
    28: from ./bin/tootctl:5:in `<main>'
    27: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    26: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    25: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    24: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    23: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:238:in `block in subcommand'
    22: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:115:in `invoke'
    21: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    20: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    19: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    18: from /home/mastodon/live/lib/mastodon/search_cli.rb:18:in `deploy'
    17: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:63:in `upgrade'
    16: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:218:in `subscribed_task_stats'
    15: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:180:in `subscribed'
    14: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:219:in `block in subscribed_task_stats'
    13: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:180:in `subscribed'
    12: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:220:in `block (2 levels) in subscribed_task_stats'
    11: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:71:in `block in upgrade'
    10: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:71:in `each'
     9: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:73:in `block (2 levels) in upgrade'
     8: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/rake_helper.rb:299:in `reset_one'
     7: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/index/actions.rb:176:in `reset!'
     6: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/chewy-5.0.0/lib/chewy/index/actions.rb:62:in `create!'
     5: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-api-7.0.0/lib/elasticsearch/api/actions/indices/create.rb:95:in `create'
     4: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-api-7.0.0/lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'
     3: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/client.rb:160:in `perform_request'
     2: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
     1: from /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:338:in `perform_request'
/home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/elasticsearch-transport-7.0.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [status : {date_detection=false, properties={account_id={type=long}, id={type=long}, text={type=text, fields={stemmed={analyzer=content, type=text}}}, searchable_by={type=long}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [status : {date_detection=false, properties={account_id={type=long}, id={type=long}, text={type=text, fields={stemmed={analyzer=content, type=text}}}, searchable_by={type=long}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [status : {date_detection=false, properties={account_id={type=long}, id={type=long}, text={type=text, fields={stemmed={analyzer=content, type=text}}}, searchable_by={type=long}}}]"}},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest)

[noellabo]

I replied to chewy's issue dealing with the same issue.
https://github.com/toptal/chewy/issues/673#issuecomment-489293310

To solve this problem, we need to send a pull request to chewy to enable the compatibility option. Here is an example of a fix.
https://github.com/noellabo/chewy/commit/94e8a6cfc540e06c051d1217e13dda0d73442e8b

Correctly, we need to rewrite chewy without using the deprecated Type.

And there is another easy solution. In nginx, rewrite compatibility options to url parameters.
https://gist.github.com/Vincanote/2b18ba8aa33d068a9cc012b9a4d891af by @Vincanote

[Dee-2015]

Thank you for the update.

The include_type_name parameter will be removed in 8.X.

Any idea how much work it would be to rewrite chewy without the Type?

[dimlau]

Mastodon v3.0.0 rc1
The nginx side solution doesn't work any more!

[subelsky]

FYI there is a chewy PR that should fix this: https://github.com/toptal/chewy/issues/673