Logstash: Support Environment Variable in conditionals

[githubmui]

Hi,
in LS 2.3.0. I have got the problem that following expression doesn't evaluate to TRUE (if environment variable doesn't exists).

if "${LS_ENDP_JDBC:a}" == "a" { ... }

Thx in advance for reply.

[suyograo]

@githubmui The environment variable option was mostly added for substituting values, and not using it in conditional. But it seems this is a good usecase too. I'll keep this open as an enhancement.

[githubmui]

Hi,
its also an missing issue for the parameters of all the input plugins!
For example, it's not possible to use this nice feature for the URL of a "http_poller".

[IrlJidel]

As a workaround you could use mutate and metadata fields

mutate {
  add_field => { "[@metadata][LS_ENDP_JDBC]" => "${LS_ENDP_JDBC:a}" }
}

if [@metadata][LS_ENDP_JDBC] == "a" {
...
}

[czerasz]

The usage in if would make the configs more generic:

output {
  if "${OUTPUT:elasticsearch}" == "debug" {
    stdout { codec => rubydebug }
  } else {
    elasticsearch { ... }
  }
}

[maya-sh]

any update on this ? currently I am using the same approach as @IrlJidel, but it's more convenient to have the env. var. in the condition

[berglh]

I also want conditional inputs based on comparisons with environmental variables:

LS_DEV_MODE=false ./logstash --debug -e '
input { 
  if "${LS_DEV_MODE}" == "false" { 
    file { path => "/tmp/file1" sincedb_path => "/dev/null" start_position => "beginning" } 
  } 
  if "${LS_DEV_MODE}" == "true" { 
    file { path => "/tmp/file2" sincedb_path => "/dev/null" start_position => "beginning" } 
  }
}
output { 
  stdout { codec => "rubydebug" } 
}'

This is because I wish to have development and production inputs in the one configuration stack and control the mode of the pipeline using LS_DEV_MODE=(true|false).

[maxlock]

+1 I also would like to change behaviour based upon environment variables.

[PhaedrusTheGreek]

+1 for being able to use environment variables in conditionals.

[astropuffin]

Was very surprised and sad to find that I couldn't do this. I would have thought that environment variables would substitute anywhere in the config on load. Really confused as to why this isn't the case.

[mrjameshamilton]

+1

[addykim]

+1 although my desired use case is something like:

output {
    stdout { codec => rubydebug }
    if ${ELASTICSEARCH_HOST} is set {
        elasticsearch { 
            hosts => ["${ELASTICSEARCH_HOST}"]
            ...
        }
    }
}

Using psuedo code for the conditional because I'm not sure what the best way to check based on the current documentation, but that's a different topic.

[berglh]

You need to assign the environment variable to a field then do your if
statement. I assign it to metadata fields so they don't make it through the
output ie. [@metadata][es-hosts]

On 14 Feb. 2017 6:38 am, "Addy Kim" [e-mail] wrote:

+1 although my desired use case is something like:

output {
stdout { codec => rubydebug }
if ${ELASTICSEARCH_HOST} is set {
elasticsearch {
hosts => ["${ELASTICSEARCH_HOST}"]
...
}
}
}

Using psuedo code for the conditional because I'm not sure what the best
way to check based on the current documentation
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html,
but that's a different topic.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/logstash/issues/5115#issuecomment-279514773,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AL4NqhRIchkUc_uDD7yja6f698g8cXG3ks5rcL9UgaJpZM4IHD4b
.

[joniba]

Can't get @IrlJidel 's workaround working on logstash 5.2.1. My conditionals are getting ignored. E.g., this always writes to localhost:9200:

input {
...
}

filter {
mutate {
add_field => { "[@metadata][ELASTIC_NOAUTH_URL]" => "${ELASTIC_NOAUTH_URL:x}" }
}
}

output {
if [@metadata][ELASTIC_NOAUTH_URL] != "x" {
elasticsearch {
hosts => [ "${ELASTIC_NOAUTH_URL:localhost:9200}" ]
}
}
}

I've tried miriad variations to no success :(

[danizen]

+1 - just found this, and tried the other method first.

[nugend]

Can elastic please call this out in documentation or put a warning for it at least? Got blocked for most of a day wondering what the hell I was doing wrong.

[mihomir]

+1
I've been debugging a Logstash configuration and the condition that should be used to identify the header in a CSV file is supposed to you environmental variables
if ["${COLUMN_NAME_USED_TO_SKIP_HEADER_ROW}"] == "${COLUMN_NAME_USED_TO_SKIP_HEADER_ROW}" {
And it appears that it is not working because of this issue.
Which is weird, when other parts of the Logstash configuration also use those variables and work as expected.

[berglh]

Would be great to see this feature being worked on.

[duaraghav8]

Hi,
Has there been any progress on this issue?
Although it isn't a blocker for me, my workaround is pretty ugly and having this feature would make my pipeline configurations much more sane :)

[dormunis]

any progress?

[Shubydo]

Any update on this issue? This is something we would really like to use in our pipelines.

[iTaybb]

Can we please push this? It's a very useful feature that been requested for years.

[MorrieAtElastic]

I have a customer who opened a Support Ticket because his use of conditional logic on an env variable was being rejected as an error.

In fact this is a very desirable enhancement and the existing documentation, while not showing examples of using an ENV within a conditional statement, does not explicitly forbid such use.

[ceeeekay]

This is affecting me too. I just spent hours backtracking and making changes to my configs on a new build that's using a lot of variables.

Any progress on this?

[brunocascio]

Anyone using env variables with conditionals?

I need something like this:

output {
  if "${IS_LOCAL}" == "true" {
    stdout {}
  } else {
    gelf {}
  }
}

[nicklan]

Surprised to find this isn't supported. Furthermore, our use-case seems like it's basically impossible, even with the workaround above. We'd like to regex match against a pattern specified by an environment variable. It appears that even if you put the value into a field, this doesn't work:

if [field] =~ /[@metadata][regex]/

Meaning we're basically just stuck.

[luminous706]

I need this too. We use a systemd file that sets variables for our logstash pipeline. Very simple to set our variables in one place, instead of going through a logstash configuration. I just discovered that variables don't work inside a conditional, I'm baffled.