Compose: Can't use linked resource with log driver

[chiefy]

I'm not sure if this is possible, but I was trying to do something like the following:

app:
  image: nginx
  ports: 
    - "80:80"
  log_driver: "gelf"
  log_opt:
    gelf-address: "udp://logstash:12201"
  links:
    - logstash

logstash:
  image: logstash
  expose:
    - "12201/udp"  

Currently this won't work, I assume because the log driver tries to initialize before any linking is done?

[dnephin]

Correct, it doesn't work because the log driver urls needs to be accessible by the daemon, not by the container. The logstash alias is never provided to the daemon. With the upcoming changes to how DNS works in docker 1.10 there might be some way to get this to work, I'm not sure.

This is a duplicate of #2182, however I think the description does a better job of demonstrating the problem, so maybe we should keep this one and close the other.

[chiefy]

thanks @dnephin I figured as much after I kept getting DNS errors.

[dnephin]

@mavenugo, @mrjana, @dave-tucker

Now that docker 1.10 is using an embedded DNS server, is there any way that the daemon could query that DNS server itself to support this use-case?

I assume you'd still need to namespace the domain name somehow.

[dnephin]

ping @mavenugo again. This seems to be a common request/issue (just had another question about it in IRC).

Is there any way the engine can use a name like service.network to access this service?

[lucj]

Hi, do you have any update on that one ?

[dave-tucker]

@dnephin I believe it's technically possible, but it's not a planned feature AFAIK.

What's the use case for running an instance of a log server per application?
As a workaround, you can use --net=host, for example:

app:
  image: nginx
  ports: 
    - "80:80"
  log_driver: "gelf"
  log_opt:
    gelf-address: "udp://localhost:12201"

logstash:
  image: logstash
  net: host
  expose:
    - "12201/udp"  

In the v2 format, you can even use depends_on to make sure that the logstash container is started first.

[lucj]

Hi @dave-tucker thanks for the update.

I'd like to have a single compose file that gathers all the services of my application:

• api
• db
• www
• reverse-proxy
• ...

and also the services used by the app:

• elk stack
• ...

Each application's service (db, api, www, ...) should send their logs to the elk stack.
Does that make sense ?

[dnephin]

@dave-tucker creating a completely isolated environment is one of the primary use cases of Compose. This is especially useful for reliable end-to-end testing.

Using the host is one workaround, but would fail on any shared host (like a jenkins CI server), so it's something I often try to avoid.

@lucj you may need to open an issue on libnetwork or docker/docker to get the feature supported.

[lucj]

@dnephin By "using the host is one workaround, but would fail on any shorted host " do you mean it cannot be used if the app and the logstash services are not deployed on the same host ?

[dnephin]

Sorry, that should have been "shared host". If you use network_mode: host you can only have one thing listening on each port on the host, so trying to run two copies of it would fail.

[lucj]

When I test with network_mode: host, I do not manage to have it working though.
My logstash service needs to have a link towards elasticsearch service (another service of the compose file). But this link is not compatible with the network_mode:

ERROR: Conflicting options: host type networking can't be used with links. This would result in undefined behavior

The part of the compose file is:

  ...
  logstash:
    image: 'myrep/logstash:latest'
    command: 'logstash -f /etc/logstash/conf.d/logstash.conf'
    network_mode: host
    links:
      - elasticsearch
    depends_on:
      - elasticsearch
    expose:
      - '5514'
    restart: on-failure
  logspout:
    image: 'myrep/logspout:latest'
    command: '-c ''/bin/logspout syslog://localhost:5514'''
    entrypoint: /bin/sh
    depends_on:
      - logstash
    restart: always
    volumes:
      - '/var/run/docker.sock:/tmp/docker.sock'

Any idea of what is wrong ?

UPDATE

@dnephin in the case I'm not using "network_mode: host", using logstash in logspout configuration seems to work fine though. Using logspout (image should be built from master branch as there is an issue in stable and latest) seems to be a good alternative to the fact service name cannot be referenced within logging part.

logstash:
  image: 'myrep/logstash:latest'
  command: 'logstash -f /etc/logstash/conf.d/logstash.conf',
  links:
    - elasticsearch
  depends_on:
    - elasticsearch
  expose:
    - '5514'
  restart: on-failure
logspout:
  image: 'myrep/logspout:latest'
  command: '-c ''/bin/logspout syslog://logstash:5514'''
  entrypoint: /bin/sh
  links:
    - logstash
  depends_on:
    - logstash
  restart: always
  volumes:
    - '/var/run/docker.sock:/tmp/docker.sock'