Timescaledb: "Telemetry Reporter" job running & failing even when telemetry disabled

[jflambert]

timescaledb.telemetry_level is off
timescaledb.max_background_workers is set to 8
I have 7 databases (including postgres)
TimescaleDB 1.6.1, PostgreSQL 11.7

_timescaledb_config.bgw_job
1 "Telemetry Reporter" "telemetry_and_version_check_if_enabled" "24:00:00" "00:01:40" -1 "01:00:00"

_timescaledb_internal.bgw_job_stat
1 "2020-03-26 17:28:01.392372+00" "2020-03-26 17:28:01.40367+00" "2020-03-27 17:28:01.40367+00" "2020-03-26 17:28:01.40367+00" true 21 "00:00:00.226514" 10 11 0 0 0

I get a steady stream of this warning message (10 per second)
WARNING: failed to launch job 1 "Telemetry Reporter": out of background workers

Seems to me like the Telemetry Reporter should NOT run when telemetry is disabled! I've confirmed that it spits out empty JSON.

[jflambert]

Can I delete bgw_job 1 or will this have disastrous side-effects?

[gayyappan]

@jflambert What command did you run to disable the job?

[gayyappan]

@jflambert How many jobs do you have per db? What is the setting for max_worker_processes?

[jflambert]

for your first question:
ALTER SYSTEM SET timescaledb.telemetry_level=off; SELECT PG_RELOAD_CONF();

For your second question, I've included the contents of _timescaledb_config.bgw_job in my post. We don't use any continuous aggregates or other types of background jobs. I'm going to try to get you the value of max_worker_processes soon, but it's probably 14 (8+3+4CPUs as per your tuning tool). But should I log another issue about that? Because my point here is that the telemetry reporter shouldn't run if it's disabled.

[gayyappan]

@jflambert yes, you can delete the job. (As an aside, note that you would have to modify max_background_workers settings if you have multiple dbs with background jobs. This setting applies across all the dbs on the instance).

[jflambert]

My max_worker_processes is 19

[jflambert]

This is still affecting me with 1.7.1-pg12, and I have much easier replication steps @gayyappan

1. docker run --rm -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password -e TIMESCALEDB_TELEMETRY=off timescale/timescaledb:latest-pg12
2. docker exec -it timescaledb psql -U postgres -d postgres -qxc "DELETE FROM _timescaledb_config.bgw_job WHERE application_name = 'Telemetry Reporter';"
3. for COUNT in {1..6}; do docker exec -it timescaledb createdb -U postgres test_$COUNT; done
4. docker exec -it timescaledb psql -U postgres -qxc "SELECT COUNT(*) AS count_databases FROM pg_database WHERE datistemplate = false;"
5. for COUNT in {1..6}; do docker exec -it timescaledb psql -U postgres -d test_$COUNT -qxc "DELETE FROM _timescaledb_config.bgw_job WHERE application_name = 'Telemetry Reporter';"; done
6. docker exec -it timescaledb psql -U postgres -qxc "SHOW timescaledb.max_background_workers;"
7. docker logs timescaledb --follow

at that point you should have 6 WARNINGS per second
WARNING: failed to launch job 1 "Telemetry Reporter": out of background workers

I don't understand. I have 7 databases and 8 max_background_workers . I have ABSOLUTELY no background jobs, not even telemetry collectors. Please explain!!!

[gayyappan]

@jflambert Thanks for the repro.

[jflambert]

@mkindahl either your fix doesn't work for me or I'm testing it wrong. Run these steps for a steady stream of WARNING: failed to launch job 1 "Telemetry Reporter": out of background workers (12 per second!)

1. docker run --rm -d --name timescaledb -e POSTGRES_PASSWORD=password -e TIMESCALEDB_TELEMETRY=off timescale/timescaledb:1.7.2-pg12
2. for COUNT in {1..6}; do docker exec -it timescaledb createdb -U postgres test_$COUNT; done
3. docker logs timescaledb --follow

[mkindahl]

I'll investigate. Thanks for checking! Reopening until we have an explanation.

[mkindahl]

@jflambert Hi, I did some checks and it seems it is not included in 1.7.2. It is on the master branch but have not been merged to 1.7.2.

[jflambert]

@mkindahl OK cool. I hope this will be resolved? Or the 1.7.2 changelog will have to change.

[mkindahl]

@jflambert We found it in 1.7.2. https://github.com/timescale/timescaledb/commit/374429ea9fad454c01f6bd6db0835a79c4cc3687 Will check if something is wrong with it.

[jflambert]

Hi @mkindahl I'm testing the official 1.7.2 with my steps and it's still happening. You may have to wait a few minutes but then the torrent of warnings begins.

should I log another issue?

[mkindahl]

@jflambert It seems to be a different problem than what is solved #1982 and this issue is quite clearly about the same as before. I'll re-open the issue.

[mkindahl]

@jflambert The issue is not specifically about the telemetry job, it is about scheduling any job when out of background workers. The option telemetry_level only control what is actually sent by the telemetry job but does not actually remove the job from being scheduled, which will then end up in the log anyway because it cannot be scheduled (even though the Telemetry job doesn't really do anything).

IMHO, the the telemetry job should be deleted from bgw_job when creating the docker image if TIMESCALEDB_TELEMETRY is off and the documentation need to be updated.

[mkindahl]

@jflambert I have created a docker image on docker hub that does that under timescaledev/timescaledb:1.7.2-pg12.1b215de51d1d584fdbe0db53de85d09a498ce822. Could you test it to check that it works as expected?

[jflambert]

@mkindahl looks good! and thanks for providing me with a docker image for testing!!!

However, I have a curve ball for you ;)

1. docker run --rm -d --name timescaledb -e POSTGRES_PASSWORD=password timescaledev/timescaledb:1.7.2-pg12.1b215de51d1d584fdbe0db53de85d09a498ce822
2. docker exec -it timescaledb psql -U postgres -d postgres -qxc "ALTER SYSTEM SET timescaledb.telemetry_level=off;"
3. docker exec -it timescaledb psql -U postgres -d postgres -qxc "SELECT PG_RELOAD_CONF();"
4. for COUNT in {1..6}; do docker exec -it timescaledb createdb -U postgres test_$COUNT; done
5. docker logs timescaledb --follow

From what I understand, using ALTER SYTEM is the only viable option once timescaledb has initialized (documentation). Unfortunately that won't hit your install shell script which deletes the jobs.

[mkindahl]

@mkindahl looks good! and thanks for providing me with a docker image for testing!!!

However, I have a curve ball for you ;)

1. `docker run --rm -d --name timescaledb -e POSTGRES_PASSWORD=password timescaledev/timescaledb:1.7.2-pg12.1b215de51d1d584fdbe0db53de85d09a498ce822`

2. `docker exec -it timescaledb psql -U postgres -d postgres -qxc "ALTER SYSTEM SET timescaledb.telemetry_level=off;"`

3. `docker exec -it timescaledb psql -U postgres -d postgres -qxc "SELECT PG_RELOAD_CONF();"`

4. `for COUNT in {1..6}; do docker exec -it timescaledb createdb -U postgres test_$COUNT; done`

5. `docker logs timescaledb --follow`

From what I understand, using ALTER SYTEM is the only viable option once timescaledb has initialized (documentation). Unfortunately that won't hit your install shell script which deletes the jobs.

That is correct. The script removes the telemetry job when creating the container, but to handle that after the setup, the job need to be explicitly deleted. Hence the documentation update above.

Thank you for verifying that it works as expected. I'll merge the Docker PR.

[jflambert]

@mkindahl will 1.7.2 contain your latest fix, or will I have to wait for 1.7.3 ?

[mkindahl]

@jflambert It is part of the Docker image, not the release, so it should already be available on the official Docker Hub page https://hub.docker.com/repository/docker/timescale/timescaledb. If not, please tell me.

[jflambert]

@mkindahl took me a while to understand but I got it now! didn't realize timescaledb and timescaledb-docker were separate projects. Now I know!

Everything works great now, thanks!

[mkindahl]

@jflambert Good to hear that it works. Thanks for reporting the issue!

[OOPMan]

I'm still seeing this issue on Timescale 2?

[jflambert]

I'm still seeing this issue on Timescale 2?

Could you expand? Very curious to hear about it. Did you tune your max-bg-workers?