Elasticsearch: Elasticsearch 2.2.0 doesn't detect SSD in Docker 1.10.3

[kjelle]

Elasticsearch version: 2.2.0

Plugins installed: [license, marvel-agent]

JVM version: 1.8.0_51

OS version: CentOS 7.2.1511

Description of the problem including expected versus actual behavior:
Elasticsearchs says I am running spins:true with SSDs.

Steps to reproduce:

1. Use SSDs, launch a docker container with docker 1.10.3 and add the SSDs as volumes.
2. Launch Elasticsearch 2.2.0 in Docker 1.10.3 on CentOS 7.2.1511 using the SSDs.
3. The log for "data paths" says "spins? [possibly]", same for '/_nodes/stats/fs'

Provide logs (if relevant):
[2016-06-20 13:01:22,140][INFO ][env ] [xxx] using [4] data paths, mounts [[/grid/1 (/dev/sdb1), /grid/4 (/dev/sde1), /grid/3 (/dev/sdd1), /grid/2 (/dev/sdc1)]], net usable_space [364.9gb], net total_space [3.4tb], spins? [possibly], types [ext4]

In docker;
$ docker exec 3fe6c85ff76d cat /sys/block/sdb/queue/rotational
0

In the host-os;
$ cat /sys/block/sdb/queue/rotational
0

(same for /dev/sdc, /dev/sdd & /dev/sde)

[clintongormley]

Determining this is only attempted on linux today (https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/env/ESFileStore.java#L57). I'm wondering if Docker is not considered to be Linux by org.apache.lucene.util.Constants.

Or maybe it is and the check fails further down in the process. Could you investigate this method and see what is different about Docker that causes SSD detection to fail?

[kjelle]

Code from org.apache.lucene.util.Constants;
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/Constants.java#L40

I ran a small java program using the same System.getProperty("os.name")

$ docker exec elasticsearch-container-data-01 /bin/sh -c "cat /tmp/PrintLinux.java"
public class PrintLinux {
    public static void main(String[] args) {
        String OS_NAME = System.getProperty("os.name");
        boolean LINUX = OS_NAME.startsWith("Linux");
        System.out.println(LINUX);
    }
}

A quick test tells us the Constants.LINUX is true;

$ docker exec elasticsearch-container-data-01 /bin/sh -c "cd /tmp; /usr/java/latest/bin/javac PrintLinux.java"
$ docker exec elasticsearch-container-data-01 /bin/sh -c "cd /tmp; /usr/java/latest/bin/java PrintLinux"
true

[kjelle]

There seems to be an expcetion when calling toRealPath in spinsLinux (https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/IOUtils.java#L506)

Using the IOUtils.spinsLinux directly;

java.nio.file.NoSuchFileException: /dev/sdb1
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixPath.toRealPath(UnixPath.java:837)
        at PrintLinux.spinsLinux(PrintLinux.java:49)
        at PrintLinux.main(PrintLinux.java:15)

[clintongormley]

Apparently you can choose to mount drives in Docker - try adding an entry to fstab?

[jasontedor]

No feedback, closing. (Additionally, the SSD detection is done in Lucene; if you continue to have issues, I suggest opening an issue there.)

[ebuildy]

Is there any performance impact because of this?

[dakrone]

@ebuildy it is used to set the number of merge threads, it will default to 1 when not on SSD and 4 when on a SSD, but you can manually set this in index.merge.scheduler.max_thread_count

[lostpebble]

Hi everyone,

Running into this issue today as well. Just deployed some nodes onto Kubernetes on GKE - provisioned some SSDs for each data node. But when I check the node stats it's showing spin: true for the disk under fs. Otherwise, all the other stats of the drives are correct (the size etc.) so I think it must be seeing the correct SSD drives but just not detecting them as such.

They are not local SSD as recommended (I'll get there eventually), but they are SSD in the same data centre so hoping it won't be too much of a performance bottleneck.

I guess for now I should just use the manual setting as @dakrone recommended. Had I not been a bit pedantic and checked out all the stats stuff I might have missed it though, so something that should still probably be looked into.

Or is there a reason I should maybe not increase the number of merge threads? Since they're not completely local SSD drives.

[jasontedor]

@lostpebble In 2.x we default to assuming SSD and set the thread counts accordingly and expect users on spinning discs to adjust down accordingly. The SSD detection is for reporting purposes only and in fact as of right now we’ve removed it in 6.0.0.