Telegraf: [inputs.exec] Python command ModuleNotFoundError

[robert-gomes]

Relevant telegraf.conf:

[[outputs.elasticsearch]]
  urls = [ "http://elasticsearch:9200" ]
  health_check_interval = "300s"
  index_name = "tct-%Y.%m"
  manage_template = false
  template_name = "telegraf"
  overwrite_template = true
  [outputs.elasticsearch.tagpass]
    send_to_elasticsearch = ["true"]

[[inputs.exec]]
  commands = [ "python3 -m api_functions.get_vcs_roots --usage unused" ]
  interval = "60s"
  timeout = "60s"
  name_override = "unused_vcs_roots"
  json_string_fields = ["name"]
  data_format = "json"
  [inputs.exec.tags]
    send_to_elasticsearch = "true"

System info:

Telegraf 1.9.0
Centos 7.6

Steps to reproduce:

In the commands section include a command that references a python module (PYTHONPATH to module root defined in /etc/environment)

Expected behavior:

Runs the command and produces json output as per input

Actual behavior:

Throws ModuleNotFoundError
2019-07-11T11:22:00Z E! [inputs.exec]: Error in plugin: exec: exit status 1 for command 'python3 -m api_functions.get_vcs_roots --usage unused': /usr/bin/python3: Error while finding module specification for 'api_functions.get_vcs_roots' (ModuleNotFoundError: No module named 'api_functions')

Additional info:

PYTHONPATH=/opt/tcs is defined in /etc/environment and points to the module root

The project has the following structure

/opt/tcs/
├── common/
│   ├── __init__.py
│   └── common_functions.py
├── config/
│   ├── __init__.py
│   └── config.py
└── api_functions/
    ├── __init__.py
    └── get_vcs_roots.py

Output of command: telegraf --config=/etc/telegraf/telegraf.d/tcscanner.conf --input-filter exec --test produces expected result

> unused_vcs_roots,host=monitor,send_to_elasticsearch=true count=0,name="A VCS (1)" 1562844830000000000
> unused_vcs_roots,host=monitor,send_to_elasticsearch=true count=0,name="B VCS (1)" 1562844830000000000
> unused_vcs_roots,host=monitor,send_to_elasticsearch=true count=0,name="C VCS (1)" 1562844830000000000

output of command: su -s /bin/bash -c 'python3 -m api_functions.get_vcs_roots --usage unused' telegraf also produces expected result

[{"id": "TCS_AVcs1", "name": "A VCS (1)", "count": 0}, {"id": "TCS_AVcs2", "name": "B VCS (1)", "count": 0}, {"id": "TCS_AVcs3", "name": "C VCS (1)", "count": 0}]

[GeorgeMac]

@robert-gomes just to clarify. Is it true that all you have to do to produce this error is to drop the --test from the telegraf command? Or are you running telegraf via some other process manager like systemd when this error occurs?

[GeorgeMac]

If you are experiencing this issue via systemd you may find it is because our default service definition doesn't reference /etc/environment and I believe that systemd would need to reference that explicitly.
https://github.com/influxdata/telegraf/blob/master/scripts/telegraf.service#L7
We do however reference /etc/default/telegraf. If this is what you are experiencing you could add PYTHONPATH=/opt/tcs to this file and try again.

[guillaumelfv]

We moved env vars from /etc/environment to /etc/default/telegraf and now it is working.

Thanks for the fix ! Issue can be closed.