Nomad version
Nomad v0.5.6
Operating system and Environment details
- Ubuntu 16.04.2 LTS (Xenial Xerus)
- Docker 17.05.0-ce
Issue
Downloaded artifacts are not executable in a docker container. It works with raw_fork but not with the Docker driver.
I get the following message when using Ubuntu with Vagrant. (I also tested it with Docker for Mac):
f0b494360b302168d8334382dbba185c66b77143729a4c1b6b223d10e44a26cd: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
Reproduction steps
My go program I'm trying to run as "app":
package main
import "log"
func main() {
log.Println("DONE! :)")
}
I build this program with the following command:
GOOS=linux GOARCH=amd64 go build -o app app.go
Then I serve the binary via a simple HTTP sever and run the job file (below).
Nomad Server logs (if appropriate)
2017/05/26 19:21:59.951926 [DEBUG] http: Request /v1/evaluation/2cfa116e-4c0b-3b14-6c22-84acf6a5137f/allocations (157.131µs)
2017/05/26 19:22:00.999816 [DEBUG] driver.docker: docker pull alpine:latest succeeded
2017/05/26 19:22:01.001897 [DEBUG] driver.docker: image "alpine" (sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118) reference count incremented: 1
2017/05/26 19:22:01 [DEBUG] plugin: starting plugin: /home/ubuntu/nomad []string{"/home/ubuntu/nomad", "executor", "{\"LogFile\":\"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/executor.out\",\"LogLevel\":\"DEBUG\"}"}
2017/05/26 19:22:01 [DEBUG] plugin: waiting for RPC address for: /home/ubuntu/nomad
2017/05/26 19:22:01 [DEBUG] plugin: nomad: 2017/05/26 19:22:01 [DEBUG] plugin: plugin address: unix /tmp/plugin476509569
2017/05/26 19:22:01.043571 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin112291564
2017/05/26 19:22:01.043616 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin112291564]}
2017/05/26 19:22:01.043622 [DEBUG] driver.docker: using 268435456 bytes memory for app
2017/05/26 19:22:01.043626 [DEBUG] driver.docker: using 500 cpu shares for app
2017/05/26 19:22:01.043664 [DEBUG] driver.docker: binding directories []string{"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc:/alloc", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local:/local", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets:/secrets"} for app
2017/05/26 19:22:01.043676 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
2017/05/26 19:22:01.043684 [DEBUG] driver.docker: allocated port 192.168.178.29:40699 -> 40699 (mapped)
2017/05/26 19:22:01.043690 [DEBUG] driver.docker: exposed port 40699
2017/05/26 19:22:01.043744 [DEBUG] driver.docker: setting container startup command to: ./local/app
2017/05/26 19:22:01.043760 [DEBUG] driver.docker: setting container name to: app-1ae5668e-74c0-050d-627d-fd236faeecd1
2017/05/26 19:22:01.084600 [INFO] driver.docker: created container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112
2017/05/26 19:22:01.507951 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 1): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:02.939463 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 2): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:04.331313 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 3): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:05.746869 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 4): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:07.171912 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 5): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.579853 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 6): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.579976 [ERR] driver.docker: failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08 [DEBUG] plugin: /home/ubuntu/nomad: plugin process exited
2017/05/26 19:22:08.582071 [WARN] client: failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
2017/05/26 19:22:08.582646 [INFO] client: Not restarting task: app for alloc: 1ae5668e-74c0-050d-627d-fd236faeecd1
2017/05/26 19:22:08.582877 [DEBUG] driver.docker: image id "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118" reference count decremented: 0
2017/05/26 19:22:08.583423 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
2017/05/26 19:22:08.584152 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
2017/05/26 19:22:08.746745 [DEBUG] client: updated allocations at index 12 (total 1) (pulled 0) (filtered 1)
2017/05/26 19:22:08.746897 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
Nomad Client logs (if appropriate)
➜ Desktop ./nomad alloc-status --address=http://192.168.178.29:4646 1ae5668e
ID = 1ae5668e
Eval ID = 2cfa116e
Name = app.app[0]
Node ID = e6e35f7b
Job ID = app
Client Status = failed
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created At = 05/26/17 21:21:58 CEST
Task "app" is "dead"
Task Resources
CPU Memory Disk IOPS Addresses
500 MHz 256 MiB 300 MiB 0 db: 192.168.178.29:40699
Recent Events:
Time Type Description
05/26/17 21:22:08 CEST Not Restarting Policy allows no restarts
05/26/17 21:22:08 CEST Driver Failure failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
05/26/17 21:21:58 CEST Driver Downloading image alpine:latest
05/26/17 21:21:58 CEST Downloading Artifacts Client is downloading artifacts
05/26/17 21:21:58 CEST Task Setup Building Task Directory
05/26/17 21:21:58 CEST Received Task received by client
Job file (if appropriate)
job "app" {
datacenters = ["dc1"]
type = "batch"
update {
stagger = "10s"
max_parallel = 1
}
group "app" {
count = 1
restart {
attempts = 0
interval = "5m"
delay = "25s"
mode = "fail"
}
task "app" {
driver = "docker"
config {
image = "alpine"
command = "./local/app"
}
artifact {
source = "http://localhost:8333/app"
}
logs {
max_files = 1
max_file_size = 15
}
resources {
cpu = 500 # 500 MHz
memory = 256 # 256MB
network {
mbits = 10
port "db" {}
}
}
}
}
}
SebastianM
👍1
All 7 comments
When I look at the rights of binary in the local dir/mount, it has no x flag:
ubuntu@ubuntu-xenial:~$ cd /tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local
ubuntu@ubuntu-xenial:/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local$ ls -la
total 1600
drwxrwxrwx 2 ubuntu ubuntu 4096 May 26 19:21 .
drwxrwxrwx 5 ubuntu ubuntu 4096 May 26 19:22 ..
-rw-rw-r-- 1 ubuntu ubuntu 1627996 May 26 19:21 app
Here's the docker inspect output of the container:
ubuntu@ubuntu-xenial:~$ docker inspect f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112
[
{
"Id": "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112",
"Created": "2017-05-26T19:22:01.046659025Z",
"Path": "./local/app",
"Args": [],
"State": {
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 126,
"Error": "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118",
"ResolvConfPath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/hostname",
"HostsPath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/hosts",
"LogPath": "",
"Name": "/app-1ae5668e-74c0-050d-627d-fd236faeecd1",
"RestartCount": 0,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc:/alloc",
"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local:/local",
"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets:/secrets"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "syslog",
"Config": {
"syslog-address": "unix:///tmp/plugin112291564"
}
},
"NetworkMode": "bridge",
"PortBindings": {
"40699/tcp": [
{
"HostIp": "192.168.178.29",
"HostPort": "40699"
}
],
"40699/udp": [
{
"HostIp": "192.168.178.29",
"HostPort": "40699"
}
]
},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 500,
"Memory": 268435456,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": null,
"Name": "aufs"
},
"Mounts": [
{
"Type": "bind",
"Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc",
"Destination": "/alloc",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local",
"Destination": "/local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets",
"Destination": "/secrets",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "f909e9e0fc21",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"40699/tcp": {},
"40699/udp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"NOMAD_TASK_NAME=app",
"NOMAD_HOST_PORT_db=40699",
"NOMAD_ALLOC_DIR=/alloc",
"NOMAD_SECRETS_DIR=/secrets",
"NOMAD_TASK_DIR=/local",
"NOMAD_ALLOC_ID=1ae5668e-74c0-050d-627d-fd236faeecd1",
"NOMAD_JOB_NAME=app",
"NOMAD_MEMORY_LIMIT=256",
"NOMAD_ADDR_db=192.168.178.29:40699",
"NOMAD_ALLOC_NAME=app.app[0]",
"NOMAD_ALLOC_INDEX=0",
"NOMAD_PORT_db=40699",
"NOMAD_CPU_LIMIT=500",
"NOMAD_IP_db=192.168.178.29",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"./local/app"
],
"Image": "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "c552b7ada392cd8ed4676888f4c0705917755accc8e4e53c2d5ebca4d915ebeb",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/c552b7ada392",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "fd8fde1b517481cb2d2aa6531b2d7a1d76af0add15d79f50036d934c37a6592d",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}
}
]
It is pretty odd to download the binary you are going to be running inside a docker container. You usually bake that into the image.
We set the exec bit when using raw_exec/exec automatically. You are being hit by the lack of ability to set the perms of downloaded artifacts: https://github.com/hashicorp/go-getter/issues/63
dadgar
on 26 May 2017
@dadgar I know it sounds odd :) But I'm trying to write a CI/CD server based on nomad. The user should be able to provide the docker image for the build job and I want to run a binary inside the unknown docker image that handles the build job inside.
Is there any workaround? I tried to run something like this as the command, but that didn't work (due to lstat):
chmod +x ./local/app && ./local/app
@SebastianM I believe you can work around it by doing:
command = "/bin/bash"
args = ["-c", "chmod +x ./local/app && ./local/app"]
Oh nice! Going to open source the CI/CD?
dadgar
on 26 May 2017
@dadgar thx - works fine!
Oh nice! Going to open source the CI/CD?
Yeah that's the plan. IMO Nomad batch jobs are a perfect fit for CI workloads. I will let you know
when it's public 👍
SebastianM
on 26 May 2017
🎉3
We solved it by mounting a dir from the host containing deployment tools and scripts.
The dir keeps containerpilot binary, pre/post scripts and alike.
Works pretty well!
rickardrosen
on 7 Jun 2017
👍1
another workaround that's clean is to place the executable in a zip or tar.gz
Storing the binary in the archive will preserve the executable permission. The artifact go-getter will automatically extract.
lanefu
on 23 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
stongo
·
3Comments
DanielDent
·
3Comments
clinta
·
3Comments
funkytaco
·
3Comments
Garagoth
·
3Comments
Most helpful comment
@dadgar thx - works fine!
Yeah that's the plan. IMO Nomad batch jobs are a perfect fit for CI workloads. I will let you know
when it's public 👍