influxdb logs to /var/log/messages by default

[randomeleven]

I am running (Influxdb 1.1.0), (Telegraf 1.1.0) and Grafana 3.0 (soon will run kapacitor) on a RHEL 7.0 server , and I ran into a 100% disk utilization issue because of the file /var/log/messages .

Seems like telegraf , influxdb and grafana are all logging their events to this file, resulting in a large file size quickly.

How do I direct the logs somewhere else ?
And what is the difference between info logged in /var/log/messages and /var/log/influxdb/{{ .product }} ?

The file contains event logs by influxdb, telegraf and grafana, like when a batch of points has been been written by telegraf, a point is arriving to influxdb via http or typical grafana query activity. I'll give an example:

Apr 20 10:03:32 my-server-host-name influxd: [query] 2017/04/20 10:03:32 SELECT mean(value) FROM db.rp.measure WHERE time > now() - 1h GROUP BY time(30s)
Apr 20 10:03:32 my-server-host-name influxd: [query] 2017/04/20 10:03:32 SELECT mean(value2) FROM db.rp.measure WHERE time > now() - 1h GROUP BY time(30s)
Apr 20 10:03:32 my-server-host-name influxd: [httpd] ::1 - influx_user [20/Apr/2017:10:03:32 +0400] "GET /query?db=db&epoch=ms&q=SELECT+mean....etc
Apr 20 10:03:32 my-server-host-name influxd: [httpd] ::1 - influx_user [20/Apr/2017:10:03:32 +0400] "GET /query?db=db&epoch=ms&q=SELECT+mean%......etc
Apr 20 10:03:37 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:03:37 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.11.0" 1864a5ab-258f-11e7-8309-000000000000 18886
Apr 20 10:03:41 my-server-host-name telegraf: 2017/04/20 10:03:41 I! Output [influxdb] buffer fullness: 30 / 10000 metrics. Total gathered metrics: 35263075. Total dropped metrics: 0.
Apr 20 10:03:41 my-server-host-name telegraf: 2017/04/20 10:03:41 I! Output [influxdb] wrote batch of 30 metrics in 223.127µs
Apr 20 10:04:06 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:04:06 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.4.3 CPython/2.7.9 Linux/3.16.0-4-amd64" 29b25ca8-258f-11e7-83fc-000000000000 8499
Apr 20 10:04:06 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:04:06 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.4.3 CPython/2.7.9 Linux/3.16.0-4-amd64" 29b33168-258f-11e7-83fd-000000000000 10269
......
......
more of the same

What's alarming is that the file is growing in size by about 0.5GB / 24 hours.

I am thinking about purging these logs after having directed them away from /var/log/messages

[pkittenis]

This is an issue with the system, not influx. This behaviour is not seen in RHEL6 or Ubuntu 12 and 14 distributions and I would wager this applies to all non-systemd distributions as well.

[jmvbxx]

@pkittenis That may be the case. However, for better or for worse, more and more systems are moving to systemd and logs not being properly directed by default is something though I believe should be addressed.

[chekolyn]

There is something we can do at the rsylog level (Also using RHEL 7.0)

We can create a simple rule in /etc/rsyslog.d/influxdb.conf to change the destination log:

### InfluxDB Rules ###
if $programname == 'influxd' then {
   action(type="omfile" file="/var/log/influxdb/influxd.log")
   stop
}

@pkittenis Perhaps these simple rules can be added to the packages (like we do for logrotate confs).

[pkittenis]

I am not part of the project, cannot help you there. I pointed out the issue does not occur on RHEL6 or other non-systemd based systems, which points to an issue with systemd rather than influx.

[abusayeedn]

I install grafana and influxdb latest version in same pc.
I have to monitoring another ubuntu syslog via Grafana.
And I also same monitoring mikrotik syslog too.
How can I insert remote syslog into influxdb database?

need anybody help.
Thanks

[HaoChou]

I have a same problem that influxdb logs to /var/log/messages by default .and I try some configs and failed~

[acidnag]

Same behaviour observed in influxdb:1.7.6
Putting an override in /etc/default/influxdb didn't work as per the documentation
System Info: Centos7.5.1804
Installation Info: Launched as a service

[RadekHavelka]

same problem here. Added the /etc/rsyslog.d/influxdb.conf as mentioned above solved it for me for now

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.

[vladak]

Same behaviour observed in influxdb:1.7.6
Putting an override in /etc/default/influxdb didn't work as per the documentation
System Info: Centos7.5.1804
Installation Info: Launched as a service

Thanks for the link to the documentation. I was too hit by this problem on my Raspberry Pi running Raspbian (where this is especially painful due to the SD card wearing off) with default syslog configuration. In my case /var/log/daemon.log was being blasted with entries such as:

Aug  7 21:30:48 teplota influxd[608]: ts=2019-08-07T19:30:48.898237Z lvl=info msg="Executing query" log_id=0H5zn~Kl000 service=query query="SELECT mean(cpu) FROM systemp.autogen.systemp WHERE url = 'http://localhost:5000/' AND time > now() - 5m GROUP BY time(200ms)"
Aug  7 21:31:01 teplota influxd[608]: ts=2019-08-07T19:31:01.541080Z lvl=info msg="Executing query" log_id=0H5zn~Kl000 service=query query="SELECT mean(usage_idle) FROM telegraf.autogen.cpu WHERE time >= now() - 6h GROUP BY time(20s)"
Aug  7 21:31:01 teplota influxd[608]: ts=2019-08-07T19:31:01.559447Z lvl=info msg="Executing query" log_id=0H5zn~Kl000 service=query query="SELECT mean(free) FROM telegraf.autogen.mem WHERE time >= now() - 6h GROUP BY time(20s)"
Aug  7 21:31:01 teplota influxd[608]: ts=2019-08-07T19:31:01.571080Z lvl=info msg="Executing query" log_id=0H5zn~Kl000 service=query query="SELECT mean(used_percent) FROM telegraf.autogen.disk WHERE time >= now() - 6h GROUP BY time(20s)"

I solved that by setting log level to error in /etc/influxdb/influxdb.conf and restarting the service.

[memetb]

I don't know how this issue is close to 3 years old and marked as "won't fix", when the solution is basic systems administration and should come out of the box.

In the /etc/systemd/system/influxdb.service, set the following parameters:

 StandardOutput=null
 StandardError=null

The above will simply zap the log output. But if this is not desired, one can also set these values to say append:/var/log/influxdb/influxd, which can be combined by setting /var/log/influxdb to be a sym link to some other storage medium.

This can then be combined with a logrotate integration.

As an aside, it's pretty poor form to dump http logs into syslog (which I am unable to disable at this point)

Systemd logging configuration reference.

[snenkov]

I am running influxDB version: 1.7.8 on Ubuntu machine (18.04.3 LTS).
Currently my syslog file keep increasing by 5 GB per day.
I consider this an issue with InfluxDB.
I don't understand why the DB need to log such info by default.

[jessie00]

There is something we can do at the rsylog level (Also using RHEL 7.0)

We can create a simple rule in /etc/rsyslog.d/influxdb.conf to change the destination log:

### InfluxDB Rules ###
if $programname == 'influxd' then {
   action(type="omfile" file="/var/log/influxdb/influxd.log")
   stop
}

@pkittenis Perhaps these simple rules can be added to the packages (like we do for logrotate confs).

Thanks a lot.

I added this rsyslog configuration with /dev/null. I don't have more influxdb message. It's disabled.