Rspamd: Export to ElasticSearch does not work with ElasticSearch >= v6.7.0

[gauthierc]

Classification :

• [x] Crash/Hang/Data loss
• [ ] WebUI/Usability
• [ ] Unintended behaviour (bug)
• [ ] Enhancement

Reproducibility :

• [x] Always
• [ ] Sometimes
• [ ] Rarely
• [ ] Unable
• [ ] I didn’t try
• [ ] Not applicable

Rspamd version :

• 1.9.1-1~stretch amd64

Operation system :

• Debian 9

Description :

Since version 6.7.0 of ElasticSearch, the ingest-geoip plugin is integrated into the solution.
This check in the file elastic.lua no longer works :

for node,value in pairs(obj['nodes']) do
        local plugin_found = false
        for _,plugin in pairs(value['plugins']) do
          if plugin['name'] == 'ingest-geoip' then
            plugin_found = true
            lua_util.debugm(N, "ingest-geoip plugin has been found")
          end
        end
        if not plugin_found then
          rspamd_logger.infox(rspamd_config,
              'Unable to find ingest-geoip on %1 node, disabling module', node)
          enabled = false
          return
        end
end

It is necessary to modify this test to detect the presence of the ingest-geoip functionality without checking the list of plugins.

Thanks.

Relevant logs:

Apr  9 16:16:25 rspamd-test rspamd[111042]: <75dofw>; lua; elastic.lua:259: Unable to find ingest-geoip on d5prHtFuSiR7FSsvcTl7ug node, disabling module

[vstakhov]

I'm inclined to drop ES support completely to tell the truth. I don't use it and I don't have neither time nor desire to support ES.

[gauthierc]

I find this module very useful.
I will try to propose a Pull Request to fix this little problem.

[jvkassi]

Please don't close. We need this too.

[gauthierc]

This patch is already included in Rspamd 1.9.2 and the documentation has been updated.
https://www.rspamd.com/doc/modules/elastic.html

[mxroute]

I've confirmed that the upgrade has taken place (new version of elastic.lua, with or without ingest_module = false in elastic.conf), but I'm still seeing this:

2019-04-22 20:15:13 #24848(main) <19s657>; cfg; rspamd_init_lua_filters: init lua module elastic
2019-04-22 20:15:14 #24883(controller) <19s657>; lua; elastic.lua:260: Unable to find ingest-geoip on 6Lb9Hot0S768PCAz8AjpPw node, disabling module
2019-04-22 20:15:14 #24882(rspamd_proxy) <19s657>; lua; elastic.lua:260: Unable to find ingest-geoip on 6Lb9Hot0S768PCAz8AjpPw node, disabling module
2019-04-22 20:15:14 #24884(normal) <19s657>; lua; elastic.lua:260: Unable to find ingest-geoip on 6Lb9Hot0S768PCAz8AjpPw node, disabling module

Curious if @gauthierc might have any insight. Appreciate any thoughts.

[gauthierc]

If the elasticsearch version is greater or equal to 6.7.0, specify in the configuration "ingest_module = true".
Otherwise you must install in elasticsearch the plugin ingest_geoip and specify in the configuration "ingest_module = false" or nothing.
The default configuration is "ingest_module=false" to avoid a change in behavior following the update of rspamd.