Nomad: Nomad Logs doesn't return logs on short running containers.

[sprutner]

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Nomad v0.5.4

Operating system and Environment details

Ubuntu 16.04 LTS on AWS

Issue

Fast running docker containers (a few seconds long) do not leave logs accessible with the nomad logs command. Also, the container never appears in a docker ps -a on the node. The container in question (myena/consul-backinator) just runs a simple command. The logs are, however, accessible on the node at /var/lib/nomad/alloc/[allocation-id]/alloc/logs/consul-backinator.stderr.0

Reproduction steps

Run the nomad job file below:

job "consul-backinator" {

  datacenters = ["dc1"]


  type = "service"


  update {

    stagger = "10s"


    max_parallel = 1
  }


  group "consul-backinator" {

    count = 1


    restart {
      # The number of attempts to run the job within the specified interval.
      # Runs the job every hour, adjust depending on your needs.

      attempts = 1
      interval = "1h"

      # The "delay" parameter specifies the duration to wait before restarting
      # a task after it has failed.

      delay = "1h"


      mode = "delay"
    }


    ephemeral_disk {

      size = 300
    }

  #task for consul-backinator
  task "consul-backinator" {

      driver = "docker"

      # The "config" stanza specifies the driver configuration, which is passed
      # directly to the driver to start the task. The details of configurations
      # are specific to each driver, so please see specific driver
      # documentation for more information.

      config {
        image = "myena/consul-backinator"

        args = [
        "backup", "-file", "s3://name-of-your-bucket/backups"
        ]

      }

      env {

        "AWS_ACCESS_KEY_ID"     = "AWS ACCESS KEY GOES HERE"
        "AWS_SECRET_ACCESS_KEY" = "AWS SECRET KEY GOES HERE"
        "AWS_REGION"            = "us-east-1"
        "CONSUL_HTTP_ADDR"      = "${NOMAD_IP_consulbackinator}:8500"
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
          port "consulbackinator" {}
        }
      }


    }
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Job file (if appropriate)

Quoted inline above

[dadgar]

Hey I couldn't replicate this:

nomad alloc-status b9
ID                  = b9ce3119
Eval ID             = ea9c0ec8
Name                = example.cache[0]
Node ID             = 564daec4
Job ID              = example
Client Status       = complete
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 03/20/17 17:55:27 UTC

Task "redis" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
500 MHz  256 MiB  300 MiB  0     db: 127.0.0.1:44055

Recent Events:
Time                   Type        Description
03/20/17 17:55:27 UTC  Terminated  Exit Code: 0
03/20/17 17:55:27 UTC  Started     Task started by client
03/20/17 17:55:27 UTC  Task Setup  Building Task Directory
03/20/17 17:55:27 UTC  Received    Task received by client
vagrant@nomad-server01:/opt/gopath/src/github.com/hashicorp/nomad$ nomad fs b9 alloc/logs
Mode        Size  Modified Time          Name
-rw-r--r--  0 B   03/20/17 17:55:27 UTC  redis.stderr.0
-rw-r--r--  14 B  03/20/17 17:55:27 UTC  redis.stdout.0
vagrant@nomad-server01:/opt/gopath/src/github.com/hashicorp/nomad$ nomad logs b9
hello, world!

Job:

      config {
        image = "redis:3.2"
        command = "echo"
        args = ["hello, world!"]
      }

I believe you may just be missing nomad logs -stderr <id> (the stderr flag).

[yifeikong]

I can reproduce this, logs for short running param jobs are all missing

[chrissound]

Also seem to notice this behavior

[yurii-fisakov]

v 0.9.2 also affected

[Manicqin]

same for 0.10.0-dev