Timescaledb: Possible bug: `ERROR: tried calling catalog_get when extension isn't loaded` even though the extension is loaded

[shirshendubhowmick]

Relevant system information:

• OS: Ubuntu Server 18.04.3 LTS
• PostgreSQL version (output of postgres --version): 11.6 (Ubuntu 11.6-1.pgdg18.04+1)
• TimescaleDB version (output of \dx in psql): 1.5.1
• Installation method: apt install

Describe the bug
Suddenly started getting error ERROR: tried calling catalog_get when extension isn't loaded while creating hypertable.
There was no version upgrades made to the server.
NOTE: this issue is only happening on a particular DB, another DB on the same server is working fine

To Reproduce
Omiting this as this occured suddenly, not sure how to reproduce

Postgres logs
Here is some postgres logs that might be helpful

• Restarted the postgresql service
• Tried creating a hypertable

2020-02-08 16:25:52.546 UTC [21575] LOG:  received fast shutdown request
2020-02-08 16:25:52.548 UTC [21575] LOG:  aborting any active transactions
2020-02-08 16:25:52.548 UTC [21586] LOG:  terminating TimescaleDB job scheduler due to administrator command
2020-02-08 16:25:52.548 UTC [21586] FATAL:  terminating connection due to administrator command
2020-02-08 16:25:52.554 UTC [21582] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2020-02-08 16:25:52.554 UTC [21582] FATAL:  terminating connection due to administrator command
2020-02-08 16:25:52.556 UTC [21575] LOG:  background worker "logical replication launcher" (PID 21583) exited with exit code 1
2020-02-08 16:25:52.556 UTC [21575] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 21586) exited with exit code 1
2020-02-08 16:25:52.556 UTC [21575] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 21582) exited with exit code 1
2020-02-08 16:25:52.556 UTC [21577] LOG:  shutting down
2020-02-08 16:25:52.578 UTC [21575] LOG:  database system is shut down
2020-02-08 16:25:52.742 UTC [25149] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-02-08 16:25:52.744 UTC [25149] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-02-08 16:25:52.783 UTC [25150] LOG:  database system was shut down at 2020-02-08 16:25:52 UTC
2020-02-08 16:25:52.792 UTC [25149] LOG:  database system is ready to accept connections
2020-02-08 16:25:52.794 UTC [25156] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2020-02-08 16:25:52.810 UTC [25159] ERROR:  tried calling catalog_get when extension isn't loaded
2020-02-08 16:25:52.813 UTC [25149] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 25159) exited with exit code 1
2020-02-08 16:25:53.329 UTC [25161] [unknown]@[unknown] LOG:  incomplete startup packet
2020-02-08 16:26:01.394 UTC [25197] postgres@smallcase_rum_production ERROR:  tried calling catalog_get when extension isn't loaded
2020-02-08 16:26:01.394 UTC [25197] postgres@smallcase_rum_production STATEMENT:  select create_hypertable('test', 'last_updated');

Running a \dx on the problematic DB gives:

                                      List of installed extensions
    Name     | Version |   Schema   |                            Description
-------------+---------+------------+-------------------------------------------------------------------
 plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
 timescaledb | 1.5.1   | public     | Enables scalable inserts and complex queries for time-series data
 uuid-ossp   | 1.1     | public     | generate universally unique identifiers (UUIDs)
(3 rows)

Please let me know whatever additional logs / information required.

Thanks

[bboule]

Hey @shirshendubhowmick thanks for the report, you indicate that you have this running without issue on another database within the same instance of PG and it is running without issue? Just to confirm when you connected to the "new" database you ran CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; while connected to the new database? I just wanted to confirm this before we marched ahead with additional troubleshooting!!

[shirshendubhowmick]

@bboule Yes another database on the same instance is working perfectly fine.

I tried running CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; in both the problematic DB and the working DB.

Both the DBs produced same output:

NOTICE:  extension "timescaledb" already exists, skipping
CREATE EXTENSION

[shirshendubhowmick]

Update: upgraded version to 1.6.0 the problem still persists.

[shirshendubhowmick]

Update: Tried a workaround to solve the issue, and it worked. Created a backup of the problematic DB using pg_dump dropped the database. Ran a vacuum. Created a new database, restored data using pg_restore from the backup and it worked.

[cevian]

We've run into this error message before with having timescaledb.restoring on you can double check this with show timescaledb.restoring; which should return off. Otherwise you did not complete backup/restore correctly. We will work on improving the error message.

[mkindahl]

This looks like it is related to #1844, which is caused when timescaledb.restoring is set for the database in question. Try ALTER DATABASE whatever SET timescaledb.restoring = off for the offending database before running pg_dump (or pg_restore).

[svenklemm]

Closing this since there was no activity. Feel free to reopen if the issue is still relevant.