Nomad: [question] exec driver error ：no such file or directory on centos6.5

[sumkire]

Nomad version

0.4.1

Operating system and Environment details

linux

Issue

run job, Driver Failure

failed to start task 'test' for alloc '8799ad5a-ef25-b3b6-be64-2386024957ea': fork/exec ../../../../../opt/shell/test.sh: no such file or directory

why path is ../../../../../opt/shell/test.sh？

client conf

data_dir = "/tmp/nomad"
region = "global"
datacenter = "dc1"
log_level = "DEBUG"
advertise {
rpc = "172.19.0.173:4647"
}

client {
enabled = true
network_speed = 10
options {
"driver.exec" = "1"
}
}

Nomad Client logs (if appropriate)

2016/11/17 08:30:49.005345 [DEBUG] http: Request /v1/jobs?region=global (29.383855ms)
2016/11/17 08:30:49.008469 [DEBUG] http: Request /v1/evaluation/56526cd5-b29c-ac0f-798c-3aa06939c847?region=global (2.112355ms)
2016/11/17 08:30:49.011236 [DEBUG] http: Request /v1/evaluation/56526cd5-b29c-ac0f-798c-3aa06939c847/allocations?region=global (1.077187ms)
2016/11/17 08:30:49.134194 [DEBUG] client: updated allocations at index 13258 (pulled 1) (filtered 0)
2016/11/17 08:30:49.136719 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)
2016/11/17 08:30:49.137924 [DEBUG] client: starting task runners for alloc '8799ad5a-ef25-b3b6-be64-2386024957ea'
2016/11/17 08:30:49.138125 [DEBUG] client: starting task context for 'test' (alloc '8799ad5a-ef25-b3b6-be64-2386024957ea')
2016/11/17 08:30:49 [DEBUG] plugin: starting plugin: /opt/nomad/nomad []string{"/opt/nomad/nomad", "executor", "/tmp/nomad/alloc/8799ad5a-ef25-b3b6-be64-2386024957ea/test/test-executor.out"}
2016/11/17 08:30:49 [DEBUG] plugin: waiting for RPC address for: /opt/nomad/nomad
2016/11/17 08:30:49 [DEBUG] plugin: nomad: 2016/11/17 08:30:49 [DEBUG] plugin: plugin address: unix /tmp/plugin210854551
2016/11/17 08:30:49 [DEBUG] plugin: /opt/nomad/nomad: plugin process exited
2016/11/17 08:30:49.235675 [INFO] client: Not restarting task: test for alloc: 8799ad5a-ef25-b3b6-be64-2386024957ea
2016/11/17 08:30:49.274968 [DEBUG] client: updated allocations at index 13260 (pulled 0) (filtered 1)
2016/11/17 08:30:49.275151 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2016/11/17 08:30:49.478172 [DEBUG] client: updated allocations at index 13261 (pulled 0) (filtered 1)
2016/11/17 08:30:49.478284 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2016/11/17 08:30:50.019284 [DEBUG] http: Request /v1/evaluation/56526cd5-b29c-ac0f-798c-3aa06939c847?region=global (6.331796ms)
2016/11/17 08:30:50.022255 [DEBUG] http: Request /v1/evaluation/56526cd5-b29c-ac0f-798c-3aa06939c847/allocations?region=global (2.046192ms)

Job file (if appropriate)

job "a5" {
datacenters = ["dc1"]
constraint {
attribute = "${attr.kernel.name}"
value = "linux"
}
update {
stagger = "10s"
max_parallel = 1
}
group "test" {
task "test" {
driver = "exec"
config {
command = "/opt/shell/test.sh"
}
service {
name = "${TASKGROUP}-test"
}
resources {
cpu = 500 # 500 MHz
memory = 256 # 256MB
}
}
}
}

[dadgar]

Hey what does your PATH look like?

[sumkire]

env?
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/root/bin

[karpeev]

I see a similar problem: fork/exec is looking for the executable at an odd path ../../../../../../absolute/path/to/actual/binary

[dadgar]

@karpeev What version of Nomad, what distro/version and are you fork/execing a downloaded binary or one that exists in the chroot?

[winduptoy]

@dadgar I've got the same symptoms as @karpeev. Ubuntu 16.10, Nomad 0.5.6, trying to exec a binary at an absolute path on the system (not from artifact), and I'm getting a strange path in the logs.

Driver Failure  failed to start task "redcloud" for alloc "df48fee8-0d9c-49f9-6637-a6ad5b581c31": failed to start command path="../../../../home/matt/dev/redcloud/bin/redcloud" --- args=["../../../../home/matt/dev/redcloud/bin/redcloud"]: fork/exec ../../../../home/matt/dev/redcloud/bin/redcloud: no such file or directory

when my job was configured to exec /home/matt/dev/redcloud/bin/redcloud.

Edit: Running the same thing with raw_exec works as expected.

[dadgar]

@winduptoy the relative prefix I agree is weird but the question to really ask is is that binary available in the chroot? I suspect it is not! You could find out by running nomad fs <alloc-id> and navigating to see if the binary is there. If it is not you will have to download the binary using artifact blocks.

[winduptoy]

@dadgar Indeed, it was not in the chroot. Thanks for the info.

[winduptoy]

@dadgar using the exec driver, how does one change the PATH to include more directories? Is this a Nomad thing or a Linux chroot/isolation thing? Running a job like this prints out a very restricted PATH that does not match the one I've specified in /etc/environment.

job "example" {
    datacenters = ["dev"]
    type = "batch"

    group "example" {
        task "printenv" {
            driver = "exec"
            config {
                command = "printenv"
            }
        }
    }
}

[dadgar]

@winduptoy Should be able to just add the path in the job:

job "example" {
    datacenters = ["dev"]
    type = "batch"

    group "example" {
        task "printenv" {
            driver = "exec"
            config {
                command = "printenv"
            }
            env {
                 PATH="MYPATH"
            }
        }
    }
}

[winduptoy]

@dadgar true, but then I need to specify that for every job that I ever run. Maybe I should have clarified that I'm looking for a way to make more PATH directories automatically available to every job. Is Nomad reading it's exec PATH from somewhere or is it hardcoded?

[dadgar]

Its inherited from the PATH of the Nomad process

[winduptoy]

Thanks for the tip.

[tgross]

Looks like this was resolved once we realized the binary wasn't in the chroot.