Logstash: NotImplementedError: block device detection unsupported or native support failed to load

Created on 28 Apr 2015 · 60Comments · Source: elastic/logstash

Issue

I am encountering the following error when trying to use the file input to watch /var/log/syslog and /var/log/auth.log

I have run the following on the log files:

setfacl -m u:logstash:r /var/log/{syslog,auth.log}

The following exception stack trace is from /var/log/logstash/logstash.err

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:230
   sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:203
        teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.9/lib/logstash/inputs/file.rb:151
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:171

If I run interactively using
sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

I get the same exception stack trace plus this:

The error reported is:
  Bad file descriptor - Bad file descriptor

I have a number of file inputs in the pipeline, but it seems only be this file input that causes a problem. Removing this file input allows it to start fine.

Environment Details

root@logstash1:/# lsb_release -d
Description:    Ubuntu 14.04.1 LTS

root@logstash1:/# /opt/logstash/bin/logstash -V
logstash 1.5.0-rc3

root@logstash1:/# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

bug

Source

mattstibbs

Most helpful comment

I also get the issue at:

alpine 3.8.1
logstatsh 6.4.2
openjdk version "1.8.0_171"

I think there might be a jruby problem，so I run：

apk add jruby

after this，the logstash works

littleji on 23 Oct 2018

👍3

All 60 comments

I have now significantly simplified the pipeline to try and narrow this problem down. Pipeline is now:

input {
  file {
    sincedb_path => [ "/var/tmp/log.sincedb" ]
    path => [ "/var/tmp/log.log" ]
    type => "syslog"
  }
}
output {
  stdout { codec => rubydebug }
}

I have also changed permissions on /var/tmp with:

sudo chmod 777 -R /var/tmp/

Running logstash with the following still generates the original error:

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

Stuck as to where to go next!

mattstibbs on 29 Apr 2015

Having started with a fresh install of Ubuntu, I am getting the same problem but am now noticing the following error intermittently as well as the reported error.

I can't find any pattern as to what causes which to appear, although it seems that if I only use RC2 I only get the below error (and not the block device error)

NotImplementedError: stat.st_dev unsupported or native support failed to load
       dev_major at org/jruby/RubyFileStat.java:188
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:150
            each at org/jruby/RubyArray.java:1613
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:132
           watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:38
            tail at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/tail.rb:68
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
            each at org/jruby/RubyArray.java:1613
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
     inputworker at /opt/logstash/lib/logstash/pipeline.rb:174
     start_input at /opt/logstash/lib/logstash/pipeline.rb:168

Also confirmed not an issue when reverting to logstash-1.4.2 using same config on same machine

mattstibbs on 29 Apr 2015

@mattstibbs I am unable to reproduce this :( It is weird because we have many folks hitting this issue -- https://github.com/elastic/logstash/issues/2962, https://github.com/elastic/logstash/issues/3033

ubuntu@ip-10-0-1-9:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.1 LTS
ubuntu@ip-10-0-1-9:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'
Logstash startup completed
{
       "message" => "Apr 30 07:00:49 ip-10-0-1-9 dhclient: DHCPREQUEST of 10.0.1.9 on eth0 to 10.0.1.1 port 67 (xid=0x54d2a924)",
      "@version" => "1",
    "@timestamp" => "2015-04-30T21:22:40.079Z",
          "host" => "ip-10-0-1-9",
          "path" => "/var/log/syslog"
}

suyograo on 30 Apr 2015

@suyograo :(

All my previous tests have been on digitalocean droplets. Have just followed below steps and am getting exactly same error:

Download Ubuntu 14.04.2 Server iso from http://www.ubuntu.com/download/server
Create standard Ubuntu install using VirtualBox (RAM 512Mb, HDD 16GB).
During startup create a user as instructed and then log into machine as that user
Run following commands in order:

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
reboot

wget http://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.0-rc3-1_all.deb
sudo dpkg -i logstash_1.5.0-rc3-1_all.deb
reboot

cd /opt/logstash
sudo bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'

Ubuntu 14.04.2 LTS
java version "1.8.0_45"
logstash 1.5.0-rc3

mattstibbs on 1 May 2015

@mattstibbs Thanks, I was able to reproduce -- we'll look into this.

ubuntu@ip-10-0-1-10:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.2 LTS
ubuntu@ip-10-0-1-10:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-10:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-10:/opt/logstash$

suyograo on 1 May 2015

@jsvd the device check was added in https://github.com/jordansissel/ruby-filewatch/pull/54. Can you take a look please?

suyograo on 2 May 2015

I tried using Oracle Java 7 or Java 8 from the repo @mattstibbs suggested in an ec2 ubuntu 14.04 instance:

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
$ bin/logstash irb
irb(main):001:0> File.blockdev? "/"
NotImplementedError: block device detection unsupported or native support failed to load
    from org/jruby/RubyFileTest.java:67:in `blockdev?'
    from (irb):1:in `evaluate'
    from org/jruby/RubyKernel.java:1107:in `eval'
    from org/jruby/RubyKernel.java:1507:in `loop'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:77:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:131:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:12:in `initialize'
irb(main):002:0>

Then tried stock openjdk 7:

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
(reverse-i-search)`': ^C
$ bin/logstash irb
irb(main):001:0> File.blockdev? "/"
=> false

So it seems to be an issue with oracle jvm, looking deeper into this..

jsvd on 6 May 2015

Same happens for Stat#dev_minor

Oracle JDK 7

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
$ bin/logstash irb
irb(main):001:0> File::Stat.new("/var/tmp/log").dev_minor
NotImplementedError: stat.st_dev unsupported or native support failed to load
    from org/jruby/RubyFileStat.java:193:in `dev_minor'
    from (irb):3:in `evaluate'
    from org/jruby/RubyKernel.java:1107:in `eval'
    from org/jruby/RubyKernel.java:1507:in `loop'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:77:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:131:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:12:in `initialize'

OpenJDK7

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
$ bin/logstash irb
irb(main):001:0> File::Stat.new("/var/tmp/log").dev_minor
=> 51713

jsvd on 6 May 2015

@mattstibbs can you try installing the package "libc6-dev" (and its dependencies) in your ubuntu instance and check if the issue goes away?

jsvd on 11 May 2015

@jordansissel is it a viable option to have libc6-dev as a dependency for the deb/rpm packages?

suyograo on 11 May 2015

@suyograo Yep! I"ll file a bug about it.

jordansissel on 11 May 2015

Do we know if this only affects Ubuntu? or Red Hat systems also?

jordansissel on 11 May 2015

@jordansissel it affects only ubuntu if using Oracle jdk (not openjdk).

jsvd on 11 May 2015

For further clarification, this is only known (for now) to reproduce with Ubuntu 14.04 _and_ Oracle Java. OpenJDK's java packages seems unaffected by this bug.

The libc6-dev package requirement had me do some investigation:

Ultimately, this is the fix:

ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

How did I find this? Here's how -

Run the reproduction case without the fix:

% strace -fo /tmp/jruby.failing -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'
NotImplementedError: block device detection unsupported or native support failed to load

Now install libc6-dev package
Rerun:

% strace -fo /tmp/jruby.working -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'
false

Now we have two files jruby.failing and jruby.working which _only_ contain file-related syscalls made by any thread (or subprocess) of java. Let's find the unique files:

% awk -F\" '{print $2}' /tmp/jruby.working | sort -u > /tmp/w
% awk -F\" '{print $2}' /tmp/jruby.failing | sort -u > /tmp/f
% diff /tmp/{f,w}
[noise trimmed for clarity]
< /lib/libcrypt.so
< /lib/x86_64-linux-gnu/libcrypt.so
< /usr/lib/libcrypt.so

So, libcrypt, eh?

# Observe the working strace
root@asdf:~/opt/logstash# grep libcrypt /tmp/jruby.working
13764 open("/root/jdk1.8.0_45/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/root/jdk1.8.0_45/bin/../lib/amd64/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = 12

# observe the failing strace
root@asdf:~/opt/logstash# grep libcrypt /tmp/jruby.failing
11222 open("/root/jdk1.8.0_45/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/root/jdk1.8.0_45/bin/../lib/amd64/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/lib/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/usr/lib/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

libcrypt.so wasn't found! Turns out Ubuntu 14.04's libc6 package doesn't have such a file:

root@asdf:~/opt/logstash# dpkg -L libc6 | grep libcrypt
/lib/x86_64-linux-gnu/libcrypt-2.19.so
/lib/x86_64-linux-gnu/libcrypt.so.1

But the libc6-dev package does:

root@asdf:~/opt/logstash# dpkg -L libc6-dev | grep libcrypt
/usr/lib/x86_64-linux-gnu/libcrypt.a
/usr/lib/x86_64-linux-gnu/libcrypt.so

Because libc6-dev transitively depends on gcc, and some users may not want gcc installed as a dependency of logstash, Ubuntu 14.04 users can patch this by doing this symlink, I think:

ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

jordansissel on 12 May 2015

🎉1

Using CentOS 7, Oracle JDK 1.8.0_45, JRuby 9.0.0.0.pre2, I cannot reproduce. File.blockdev?("/") is OK on this environment.

jordansissel on 12 May 2015

@jsvd Sorry for the delay - am on hols in Singapore!

Have just tested after installing libc6-dev and it works fine - as you probably expected!

mattstibbs on 13 May 2015

Encountered this exact error after stepping up from 1.4.2 to 1.5.0 RC4 under Solaris 10.5. Problem was an old JRE; upgraded from Oracle Java 6 to 8 and logs are once again being stashed.

KyleCartmell on 13 May 2015

I had the same issue on Debian 7.8 Wheezy, but adding the symlink solved the problem. Thanks.

pako-pl on 16 May 2015

I am seeing this as well on RHEL6, java 1.7

java -version
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

./logstash -V
logstash 1.5.0

atman9 on 8 Jun 2015

Hi, I confirm we are having the same issue with:
java version "1.7.0_79"
OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

CentOS release 6.6 (Final)

logstash-1.5.0

What about this news:
http://jruby.org/2015/05/05/jruby-1-7-20.html

Github Issues resolved for 1.7.20:

#1754 - JRuby 1.7.11 (bundled with LogStash v. 1.4.1) fails on FreeBSD 10.0 with NotImplementedError: stat.st_dev unsupported or native support failed to load

Could you guys not just use this new JRuby version?
greetings

SMMAN on 12 Jun 2015

And I can confirm the error on Solaris 10 on SPARC, logstash 1.5.0, Oracle JDK 8u45.
Oddly, it was shipping logs just fine for a week or so, but this error occurred when I restarted it.

mjadas on 12 Jun 2015

Reproduced on Debian 8.1 64b + Java 1.8.0_45

The symlink trick or installing libc6-dev solves the problem. Thanks @jordansissel

mvonthron on 20 Jun 2015

@jordansissel Awesome troubleshooting! I can confirm your soft link fixed this issue on Ubuntu 14.10 as well.

naisanza on 22 Jun 2015

I have the same error on Solaris 10 on SPARC, Logstash 1.5.1 & Oracle JDK 7u71

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:231
   sincedb_write at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:204
        teardown at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file.rb:151
     inputworker at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
     start_input at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:170

hkhail on 22 Jun 2015

have the same error in ubuntu
NotImplementedError: stat.st_gid unsupported or native support failed to load
gid at org/jruby/RubyFileStat.java:247
atomic_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/helper.rb:45
_sincedb_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:234
sincedb_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:204
teardown at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file.rb:151
inputworker at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
synchronize at org/jruby/ext/thread/Mutex.java:149
inputworker at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
start_input at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:170

srijiths on 8 Jul 2015

Still seeing this error on LS 1.5.2 even on OpenJDK and Oracle 1.8:

java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
 cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

The error:

Sending logstash logs to /db/dba/logstash/logstash/logstash.log.
NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/helper.rb:67
  _sincedb_write at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:233
   sincedb_write at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:206
        teardown at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.0/lib/logstash/inputs/file.rb:152
     inputworker at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
     start_input at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:170

Anyone hit this on Redhat and solve it with a symlink? I am seeing a libcrypt.so in /usr/lib64:

 ls -l /usr/lib64/libcrypt.*
lrwxrwxrwx. 1 root root 25 Jul  8 11:01 /usr/lib64/libcrypt.so -> ../../lib64/libcrypt.so.1

atman9 on 8 Jul 2015

I can confirm the symlink fix issue on Ubuntu 14.04 with Oracle Java 1.8!

Thanks,

ebuildy on 14 Jul 2015

LS 1.5.2 works on RHEL6 on openJDK 1.8 with no linking necessary. I had a pathing issue that was sourcing the wrong version of Java.

atman9 on 14 Jul 2015

@jordansissel hello jordan, i have the same issue with Red Hat 5.9, logstash version 1.5.2 and java 1.7.1_71, if can you help to fix this issue? thank you

Ciciloveyou on 15 Jul 2015

Thanks, Jordan.
The symlink patch appears to have solved my issue as well. For posterity. I'm including my errors as well.

Fix: ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

Ubuntu 14.04.3 LTS
logstash 1.5.2
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

stdout Error:

NotImplementedError: block device detection unsupported or native support failed to load
blockdev? at org/jruby/RubyFileTest.java:67
device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/helper.rb:67
_sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:233
sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:206
teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.0/lib/logstash/inputs/file.rb:152
inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
synchronize at org/jruby/ext/thread/Mutex.java:149
inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:170

rlwmmw on 27 Aug 2015

Appears it was a library issue on Solaris10/SPARC as well. Putting the sparcv9 lib dir at the head of the library path fixed it for me:

export LD_LIBRARY_PATH=/usr/lib/sparcv9/:$LD_LIBRARY_PATH

mjadas on 27 Aug 2015

This was a nasty surprise when attempting to upgrade from 1.4 to 1.5 on Ubuntu 14.04 with Oracle Java 1.7.0_76. Reverting back to 1.4 for now. Maybe the intro docs should be updated to say that the Oracle JVM is no longer recommended?

scottnonnenberg on 14 Sep 2015

Oracle and OpenJDK JVM are both fine. Maybe this is a bug, and not a
feature?

On Sunday, September 13, 2015, Scott Nonnenberg [email protected]
wrote:

This was a nasty surprise when attempting to upgrade from 1.4 to 1.5 on
Ubuntu 14.04 with Oracle Java 1.7.0_76. Reverting back to 1.4 for now.
Maybe the intro docs
https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html
should be updated to say that the Oracle JVM is no longer recommended?

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/logstash/issues/3127#issuecomment-139932387.

jordansissel on 14 Sep 2015

jordansissel: You must be referring to the fact that it's some sort of symlink issue, and not a problem with the JVM itself. Fair enough. I'll just be more explicit with my request. I'd like to see a page, linked to from the getting started docs, which talks about the known issues on the various platforms. Or, better, an update to the apt package to provide the symlink on ubuntu/debian.

scottnonnenberg on 14 Sep 2015

Fix in https://github.com/elastic/logstash/issues/3127#issuecomment-101068714 helped me. My system:
OS: Ubuntu 14.04
Kernel: 3.13.0-63-generic
Logstash: 1.5.4-1
Java: 1.8.0_60

jonatanblue on 22 Sep 2015

Hello,

I encounter a similar problem on Fedora 21. In my case, it seems to be the filter that causes problems and not the input.

My system:
OS: Fedora 21 x86_64
Kernel: 4.1.7-100.fc21
Logstash: 1.5.4
Java: openjdk 1.8.0_60

Executing and looking at strace, I see that it find libcrypt.so. Although, there many other files that are not found.

As there are a lot of entries, If you can tell me what I should look for I can post it.

montaropdf on 30 Sep 2015

If anybody is still having issues with this, this only happened to me when using the file input plugin.
I'm running Ubuntu 14.04 x86_64 with Oracle Java JDK and Logstash 1.5.4-1

Installing libc6-dev package resolved the issue as it provides the /usr/lib/x86_64-linux-gnu/libcrypt.so library.
apt-get install libc6-dev

tobias-urdin on 2 Oct 2015

Hi,

Any more advice to fix that issue when using OpenJDK?

I perform further investigation and it seems that the problem is using the file plugins and the grok filter that produce the error.

If I use a filter like kv, I have no error, but nothing is processed, see below:

config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@path = ["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@type = "mep_access" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@debug = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"kv", :path=>"logstash/filters/kv", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
config LogStash::Filters::KV/@field_split = " " {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@value_split = ":" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@prefix = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@source = "message" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@include_keys = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@exclude_keys = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@default_keys = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@allow_duplicate_values = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
config LogStash::Codecs::RubyDebug/@metadata = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Registering file input {:path=>["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"77", :method=>"register"}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7", :path=>["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}
_sincedb_open: /root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7: No such file or directory - /root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"215", :method=>"_sincedb_open"}
_discover_file_glob: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log: glob is: ["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
Pipeline started {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"87", :method=>"run"}
_discover_file: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log: new: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log (exclude is []) {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"141", :method=>"_discover_file"}
Logstash startup completed
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"204", :method=>"sincedb_write"}
Plugin is finished {:plugin=><LogStash::Outputs::Stdout codec=><LogStash::Codecs::RubyDebug metadata=>false>, workers=>1>, :level=>:info, :file=>"logstash/plugin.rb", :line=>"61", :method=>"finished"}
Pipeline shutdown complete. {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"101", :method=>"run"}
Logstash shutdown completed

montaropdf on 5 Nov 2015

libc6-dev solve the problem too on logstash 2.1.0

sebadiaz on 25 Nov 2015

Hi,
My system Centos 7, java-openjdk 1.7 or 1.8, logstash 2.1.0 => same error

Thanks to jordansissel post i did following things:

trace -fo /tmp/jruby.failing -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'

reading jruby.failing ... libcrypt.so was found! but file crypt wasn't found! :(

I did symlink and everything is ok now

ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/crypt

Thank you boys

ghost on 29 Nov 2015

👍1

I apply the same fix as @neradpeter , and now the file plugin is working great.

My system:
Fedora 21
java-openjdk 1.8
logstash 1.5.5

A question:

As this fix will break next time libcrypto.so is updated, who should fix its package/application?
A. The maintainer of logstash or its RPM package, to better discover the presence of libcrypto?
B. The maintainer of fedora/centos/rh/... to add the link to /usr/lib64/crypt?

Thanks.

montaropdf on 30 Nov 2015

Hi,
I have the same problem on Red Hat Enterprise Linux Server release 6.7 (Santiago)
I've tried with logstash 1.5.5 and 2.0.0 and with

jdk1.8.0_65, jdk1.8.0_60 and jdk1.8.0_51
openjdk jre-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64

Everytime, I've the same error

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:232
   sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:205
        teardown at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157
     inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
     start_input at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:171

Finally I've made the following link and it works ?!

ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/crypt

evidal on 2 Dec 2015

👍1

@Eric Vidal, have you tried the solution from neradpeter?

On Wed, Dec 2, 2015 at 12:10 PM, Eric Vidal [email protected]
wrote:

Hi,
I have the same problem on Red Hat Enterprise Linux Server release 6.7
(Santiago)
I've tried with logstash 1.5.5 and 2.0.0 and with

jdk1.8.0_65, jdk1.8.0_60 and jdk1.8.0_51

openjdk jre-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64

Everytime, I've the same error

NotImplementedError: block device detection unsupported or native support failed to load
blockdev? at org/jruby/RubyFileTest.java:67
device? at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/helper.rb:67
_sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:232
sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:205
teardown at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157
inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
synchronize at org/jruby/ext/thread/Mutex.java:149
inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
start_input at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:171

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/logstash/issues/3127#issuecomment-161263344.

montaropdf on 2 Dec 2015

Also have this issue with

logstash 1.5.6 (installed from elastic apt repo)
ubuntu 14.04 LTS
java
- oracle 1.7.0_80
- oracle 1.8.0_66
- openjdk 1.7.0_91 - (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)

The fix with the symlink does not seem to help. I am able to reproduce the steps from @jordansissel but when it comes to

grep libcrypt /tmp/jruby.working

i get
46437 open("/usr/lib/jvm/java-7-oracle/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
46437 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = 9

any idea what could be wrong with my system?

blubbi321 on 7 Jan 2016

Environment

$ lsb_release -d
Description:    Ubuntu 14.04.2 LTS

$ /opt/logstash/bin/logstash -V
logstash 2.2.0

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

I also have the symlink as suggested
lrwxrwxrwx 1 root root 35 Feb 25 2015 libcrypt.so -> /lib/x86_64-linux-gnu/libcrypt.so.1

Still see this problem.
otImplementedError: stat.st_dev unsupported or native support failed to load dev_major at org/jruby/RubyFileStat.java:205 nix_inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:28 inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:32 inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:106 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:96 _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:293 each at org/jruby/RubyArray.java:1613 each at org/jruby/RubyEnumerator.java:274 _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:284 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:95 call at org/jruby/RubyProc.java:281 synchronized at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:337 synchronize at org/jruby/ext/thread/Mutex.java:149 synchronized at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:337 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:92 tail at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/tail_base.rb:73 tail at /opt/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201 begin_tailing at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:284 each at org/jruby/RubyArray.java:1613 begin_tailing at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:284 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:288 inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.0-java/lib/logstash/pipeline.rb:316 start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.0-java/lib/logstash/pipeline.rb:310 The error reported is: Bad file descriptor - Bad file descriptor

Any idea how this can be resolved ?
This is a blocking issue since we need the file input and this happens when we use the file input.

adiworkoholic on 10 Feb 2016

Any update on the ETA for the fix ?

adiworkoholic on 17 Feb 2016

Also still see the problem with this configuration:

Description: Ubuntu 14.04.3 LTS

logstash 2.2.2

java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

lrwxrwxrwx 1 root root 35 Feb 25 2015 /usr/lib/x86_64-linux-gnu/libcrypt.so -> /lib/x86_64-linux-gnu/libcrypt.so.1

jconlon on 18 Feb 2016

For those of you seeing this problem on Ubuntu, my recommendation for a fix is still to install the libc6-dev package.

jordansissel on 18 Feb 2016

See the problem even after installing the libc6-dev package. :(

adiworkoholic on 23 Feb 2016

Same issue for me on :

Red Hat Enterprise Linux Server release 6.6 (Santiago) - Linux 2.6.32-504.8.1.el6.x86_64
LogStash 2.2.2
Tested with Oracle JDK 1.7u80 and Oracle JDK 1.8u66

I've made the following link and it works :
sudo ln -s /lib64/libcrypt.so.1 /usr/lib64/libcrypt.so

jhaeyaert on 24 Feb 2016

Is there any possibility to put the libc6-dev in the library folder of LogStash itself?
I'm asking because I have to ship logstash to a system with a special java version where I'm not permitted to install libc6-dev nor to create any symlinks.

subkowlex on 29 Feb 2016

Same problem here.
Ubuntu 14.04
Logstash 2.2.2
JDK 1.8
I have installed libc6-dev, but also I have some 32 bit libraries.. maybe is that.
I have the link everybody mentions, but I have 4 of them:

$ ls -lah /lib/i386-linux-gnu/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Feb 16 16:19 /lib/i386-linux-gnu/libcrypt.so.1 -> libcrypt-2.19.so
$ ls -lah /lib/x86_64-linux-gnu/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Feb 16 16:29 /lib/x86_64-linux-gnu/libcrypt.so.1 -> libcrypt-2.19.so
$ ls -lah /lib32/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Feb 16 16:32 /lib32/libcrypt.so.1 -> libcrypt-2.19.so
$ ls -lah /libx32/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Feb 16 16:33 /libx32/libcrypt.so.1 -> libcrypt-2.19.so

PS: My actual error is "NotImplementedError: stat.st_dev unsupported or native support failed to load". maybe is different.
PS2: I saw in #3033 that this is needed:

$ ls -ld /tmp
drwxrwxrwt 5 root root 4096 Feb 29 19:17 /tmp
$ mount | grep /tmp
$

I think badly that the problem is related to the 32 bit libraries. The problem is I need them for another app that runs there.

PS3: After running some of the commands Jordan told before, (only the failing ones), this is what I have:

$ grep libcrypt /tmp/jruby.failing
8161  open("/libx32/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 15

And then:

$ dpkg -L libc6 | grep libcrypt
/lib/x86_64-linux-gnu/libcrypt-2.19.so
/lib/x86_64-linux-gnu/libcrypt.so.1
$ dpkg -L libc6-dev | grep libcrypt
/usr/lib/x86_64-linux-gnu/libcrypt.a
/usr/lib/x86_64-linux-gnu/libcrypt.so
$ ls -lah /libx32/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Feb 16 16:33 /libx32/libcrypt.so.1 -> libcrypt-2.19.so

I think I'm one step ahead of the solution.. but don't know what slink to do or whatever is needed.

syunusic on 29 Feb 2016

I have the same results as @syunusic with the same error message.

adiworkoholic on 2 Mar 2016

At the end, what I did (NOT very classy), is just remove the symbolic link:
sudo rm /libx32/libcrypt.so.1
And logstash started to work.
I have crossed my fingers and hope anything else uses /libx32/libcrypt.so.1.

syunusic on 4 Mar 2016

👍3

For folks hitting this issue specifically on Solaris SPARC, you might find this comment has asolution. Otherwise, I have no access to SPARC hardware (and it's been a loooong time since I've used Solaris), so I'm not sure how helpful I can be on fixing it for that platform and architecture.

jordansissel on 11 May 2016

Same error at SuSE v12 (Linux on Z/OS vm) with both IBM JDK 1.7 and OpenJDK 1.7. also Any help would appreciate.

gitqizong on 23 Aug 2016

Taking a queue from @mjadas I got past this error.

$ cat /etc/*release
Oracle Linux Server release 6.4
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Oracle Linux Server release 6.4

$ /tmp/logstash-2.4.0/bin/logstash -V
logstash 2.4.0

$ ls -l /usr/lib64/libcrypt.*
lrwxrwxrwx 1 root root 25 Mar 31 11:16 /usr/lib64/libcrypt.so -> ../../lib64/libcrypt.so.1

$ /usr/bin/java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

$ echo $LD_LIBRARY_PATH
/lib:/lib:/usr/lib:/u01/app/oracle/product/agent/core/12.1.0.1.0/perl/lib:/usr/lib/oracle/12.1/client64/lib

$ export LD_LIBRARY_PATH=/lib64:/usr/lib64:$LD_LIBRARY_PATH