logstash-oss not starting and contain non-removable x-pack

[nean-and-i]

Hi,

it appears that the logstash-oss version from the original elastic site contain x-pack stuff under elastic license which to my understanding should not be in the oss code?
Also logstash oss isn't starting at all, so one might be forced to use the regular version.

This can be reproduced con Centos7 following the steps below:

• original oss version from website (sidenote, oss doesn’t seem to be in the repository)

curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-6.7.0.rpm -o logstash-oss-6.7.0.rpm

when starting the oss version following error pops up:

May 02 11:53:05 host logstash[8117]: [ERROR] 2019-05-02 11:53:05.366 [main] Logstash -
 java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) 
no such file to load -- x-pack/logstash_registry

• ok, just let’s remove logstash x-pack plugin:

/usr/share/logstash/bin/logstash-plugin remove x-pack

-> nope x-pack not a removable plugin :thinking:

WARNING: All illegal access operations will be denied in a future release
wAs of 6.3+ X-Pack is part of the default distribution and cannot be uninstalled;
to remove all X-Pack features, you must install the OSS-only distribution of 
Logstash. ERROR: Operation aborted, cannot remove plugin, message:
x-pack not a removable plugin

• one way to get it working is to remove the x-pack plugin manually via the ugly way:

mv -v /usr/share/logstash/x-pack /tmp/

• x-pack is handled in the logstash-core and it looks like this isn't properly working:

/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:      require("x-pack/logstash_registry")

• something is wrong LogStash::OSS
  As a workaround one can deactivate/comment load_xpack unless LogStash::OSS in the code /usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb

    def setup!
      #load_xpack unless LogStash::OSS
      load_available_plugins
      execute_universal_plugins
    end

• LogStash::OSS is handled here /usr/share/logstash/logstash-core/lib/logstash/runner.rb
  🤦‍♂

require 'pathname'
LogStash::ROOT = Pathname.new(File.join(File.expand_path(File.dirname(__FILE__)), "..", "..", "..")).cleanpath.to_s
LogStash::XPACK_PATH = File.join(LogStash::ROOT, "x-pack")
LogStash::OSS = ENV["OSS"] == "true" || !File.exists?(LogStash::XPACK_PATH)

if !LogStash::OSS
  xpack_dir = File.join(LogStash::XPACK_PATH, "lib")
  unless $LOAD_PATH.include?(xpack_dir)
    $LOAD_PATH.unshift(xpack_dir)
  end
end

Can this please get fixed else it might look like there isn’t the highest interest in creating functional oss packages?

thanks,
nean

[akshatsharma4]

Hi,

I am facing kind of similar issue related to x-pack in logstash-oss,.
My logstash-oss rpm version is 7.0.1 and it tries to contact x-pack when I enable searchguard and try to connect to elasticsearch.

Details and error logs can be found here.

Can anyone help me resolve this?

Thanks,
Akshat

[jsvd]

can you test with logstash 6.8.1? a fix in packaging was added done in https://github.com/elastic/logstash/pull/10833, that was included in 6.8.1 and will be included in 7.2.0

[akshatsharma4]

Hi @jsvd
Thanks for responding.
Actually I am using 7.0.1 version for all my components, i.e., elasticsearch-oss, logstash-oss and kibana-oss. It won't be possible to go back to 6.8.1 or to wait for 7.2.0.
Is there any fix or workaround to get 7.0.1 working?

-Akshat

[jsvd]

Right now the only workarounds are either to:

• remove the x-pack folder after installation (can be done easily with the usual deployment tools like chef/puppet,etc)
• replace a line of code in runner.rb to read instead LogStash::OSS = true
• start logstash with OSS=true bin/logstash -f ..

Because we're about to release 7.2.0 it's unlikely will ship any patch releases for previous versions of 7.x

[akshatsharma4]

@jsvd @nean-and-i

I tried option 2 and 3 but I am still facing the same issue.
Logstash is still trying to contact xpack.

error logs:
Failed to install template. {:message=>"Got response code '500' contacting Elasticsearch at URL 'https://xx.xx.xx.xxx:9200/_xpack'"

I am not understanding why it is even trying to contact xpack when I am using oss version of both elasticsearch and logstash.
Earlier I was using 6.6.1 version(oss) and I never faced this error. Only when I am trying to upgrade to 7.0.1, this error is showing up.

Note: It is only when I use elasticsearch and logstash with searchguard, I am getting this error.

Is there anything that can be done?

Thanks,
Akshat

[chaitrahegde115]

Hi,
I tried the above steps which you have mentioned. But i am still facing the same issue.
1) I cant follow the step 1 because i dont have x-pack folder in /usr/share/logstash
2) I replaced the code to read LogStash::OSS = true
3) I started logstash with OSS=true bin/logstash -f ..
The code snippet is attached. Can you please tell me if my changes are proper or do i need to add something else also.

[akshatsharma4]

@jsvd @nean-and-i
I found this file "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb" that had following code snippet:

def get_xpack_info get("/_xpack") end

I removed "/_xpack" from there and now the code is changed to:
def get_xpack_info get("") end

This resolved the error that I was getting. Now logstash is running fine and Logstash index is created on elasticsearch.

Can you please confirm if this is the correct approach.

Thnaks,
Akshat

[jsvd]

7.2.0 was released yesterday, can you check if the problem still exists?

On Wed, Jun 26, 2019, 08:28 Akshat Sharma [e-mail] wrote:

@jsvd https://github.com/jsvd @nean-and-i
https://github.com/nean-and-i
I found this file
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb"
that had following code snippet:

def get_xpack_info get("/_xpack") end

I removed "/_xpack" from there and now the code is changed to:
def get_xpack_info get("") end

This resolved the error that I was getting. Now logstash is running fine
and Logstash index is created on elasticsearch.

Can you please confirm if this is the correct approach.

Thnaks,
Akshat

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/logstash/issues/10783?email_source=notifications&email_token=AAAHYQJIZYNLUPURBOA2WB3P4MLBFA5CNFSM4HME4NQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYSTN2Y#issuecomment-505755371,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHYQJQL7CD2BJTUUWDLTDP4MLBFANCNFSM4HME4NQA
.

[akshatsharma4]

@jsvd
We are using 7.0.1 for all our components. we cannot switch to 7.2.0 at this point.
Can you please confirm if removing "/_xpack" from "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb" is the right approach?

[jsvd]

can you try instead setting ilm_enabled => false ?

[akshatsharma4]

@jsvd
In which file should I set "ilm_enabled => false" ?
can you please tell me the path and line number ?

[jsvd]

I mean in your configuration, in the elasticsearch output, like
```
output {
elasticsearch {
hosts => [.. ]
ilm_enabled => false
}
}

[akshatsharma4]

@jsvd
Yes, setting ilm_enabled => false worked.
Now it is not trying to contact xpack.

Thanks!

[nean-and-i]

@jsvd thanks for the help and fast response, good to see getting this fixed, very much appreciated!
unfortunately I can not upgrade to 7.2 atm. because of some other dependencies.

I had a look at the elastic docs and it appears ilm - index lifecycle management is exclusively for x_pack (a x-pack feature), I assume this is causing the issue on 7.0.1.

Just for my understanding, is it planed to keep code parts that are under/referring-to elastic license in the oss parts/versions or will there be a clean separation in the near future ?
I can imagine this is technically quite a challenge, and not as easy as in the high level marketing slides -> https://www.elastic.co/products/x-pack/open

[jsvd]

I assume this is causing the issue on 7.0.1

There are two problems in this issue, one is the packaging issue that has been fixed, where the contents of the non-oss package were being used in both distributions.

The other issue comes from using a plugin that alters how ES (even the ones one) responds to requests. Typically if the ES endpoint you're connecting to doesn't have the license/xpack endpoint then the feature is disabled and everything should work well. But with this plugin the error changed from 400 to a 500, throwing off the error handling code

[nean-and-i]

@jsvd before I'm going to close the issue, please allow me to ask if it is possible to apply the fix(es) in the 7.0.1 OSS packages to get working packages for those who need to stick with 7.0.1 for a while?

thanks,
nean

[jsvd]

@nean-and-i unfortunately we don't have any plans to do any more 7.0 releases (same for 7.1 actually). The 7.0 and 7.2 releases should be compatible, any reason to not upgrade to 7.2?

[nean-and-i]

@jsvd ok, atm. I have logstash-7.0.1 (oss rpm from the homepage) and elasticsearch-oss-6.7.1 and java 11 which works.
I have a lot of complex filters in place to normalize data into highly structured documents and insert it to elasticsearch-oss-6.7.1 and over the years I got very very careful with upgrading logstash.
however, if I can use 7.2 logstash without any hassle in combination with a 7.0.1 or 6.7.1 elasticsearch cluster, then I should be fine and will move on with planing a migration.

thanks

[nerijus]

I have the same issue with logstash-oss 7.4.0 and ilm_enabled => false.

[nerijus]

Oops, actually I had /usr/share/logstash/x-pack but with empty directories inside. Removing /usr/share/logstash/x-pack helped, but it seems to be a packaging problem with logstash rpm - it does not own this directory and files in it, so removing logstash package did not delete them.

[MakeHoney]

@jsvd Thank you so much!

[nean-and-i]

Hi,

can this be re-opened, it seem it causes issues under certain circumstances:

# yum info logstash-oss
Loaded plugins: langpacks, rhnplugin, versionlock
This system is receiving updates from RHN Classic or Red Hat Satellite.
Installed Packages
Name        : logstash-oss
Arch        : noarch
Epoch       : 1
Version     : 7.4.2
Release     : 1
Size        : 289 M
Repo        : installed
From repo   : elasticsearch-7.x
Summary     : An extensible logging pipeline
URL         : http://www.elasticsearch.org/overview/logstash/
License     : ASL 2.0
Description : An extensible logging pipeline

This does not start:

Nov 07 19:08:39 logstash[249857]: [ERROR] 2019-11-07 19:08:39.737 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- monitoring/monitoring

# grep -R "monitoring/monitoring" /usr/share/logstash/*
/usr/share/logstash/x-pack/lib/x-pack/logstash_registry.rb:require "monitoring/monitoring"

EDIT: it seems I got it running after I did a manual cleanup

yum install logstash
rm -rvf/usr/share/logstash/
yum install logstash-oss

[munawarsyukur]

Previously I happen to face same issue, I follow all the steps given the issue still remain.
But last night I revert all the configuration as default and manage to run the Logstash-OSS on EC2-AMI without make any changing as advise above.

I only run this command below, let me know if this command working on your environtment.

[user]$ sudo OSS=true /usr/share/logstash/bin/./logstash -f /etc/logstash/simple.conf

[educlos]

I have the same issue with logstash-oss-7.4.2.deb

OSS=true /usr/share/logstash/bin/logstash --path.settings /etc/logstash
...
LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '500' contacting Elasticsearch at URL 'https://URL:9200/_xpack'

None of the workaround proposed here work

[joelcahalan]

i had this issue on osx while trying to index on an aws es cluster even with ilm_enabled => false . I followed the stacktrace and code to def ilm_in_use? in /usr/local/Cellar/logstash/7.5.2/libexec/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.2.3-java/lib/logstash/outputs/elasticsearch/ilm.rb. Added return false as the first line in the aformentioned method and this solved the issue for me.