Postgraphile: Won't reconnect after Database restart

[krzkaczor]

I'm submitting a ...

• [x] bug report
• [ ] feature request
• [ ] question

PostGraphile version: any

Steps to reproduce:

1. start postgres
2. start postgraphile server
3. stop postgres
4. start postgres

Current behavior: postgrahile won't reconnect to newly created database it will be stuck with getaddrinfo ENOTFOUND postgres postgres:5432.

We needed to restart DB on prod and unfortunatelly all api services stopped working and didnt come back after DB restart was done.

Expected behavior:
I see two options:

1. Let it crash. IMO it's often fine since restart logic should be in place anyway.
2. Properly reconnect.
3. Add hook (callback ie. onPgError) for pg errors so users can customize this behavior.

Any thoughts? I can work on a PR but help me decide what will be the best way to fix it.

Related code is: https://github.com/graphile/postgraphile/blob/d2374174adfb363c20fc068aea255ad45bbbe708/src/postgraphile/postgraphile.ts#L221 But unfortunately PgPool won't reconnect.

[benjie]

I think you're looking for the --retry-on-init-fail / retryOnInitFail: true option.

[semi-automated message] Thanks for your question; hopefully we're well on the way to helping you solve your issue. This doesn't currently seem to be a bug in the library so I'm going to close the issue, but please feel free to keep requesting help below and if it does turn out to be a bug we can definitely re-open it 👍

You can also ask for help in the #help-and-support channel in our Discord chat.

[benjie]

Hmm, actually if my understanding is correct then without that flag it should have exited anyway. I'll try and reproduce.

[benjie]

Run a clean database (so we can kill it later)

docker run --rm -p 6543:5432 postgres

Connect to the DB and set it up:

psql 'postgres://postgres@localhost:6543/template1'

create database foo;
\c foo
create table bar (id serial primary key);

Run PostGraphile:

$ npx postgraphile -c 'postgres://[email protected]:6543/foo'

PostGraphile v4.4.3 server listening on port 5000 🚀

  ‣ GraphQL API:         http://localhost:5000/graphql
  ‣ GraphiQL GUI/IDE:    http://localhost:5000/graphiql (enhance with '--enhance-graphiql')
  ‣ Postgres connection: postgres://[email protected]:6543/foo
  ‣ Postgres schema(s):  public
  ‣ Documentation:       https://graphile.org/postgraphile/introduction/
  ‣ Join Nigel Taylor in supporting PostGraphile development: https://graphile.org/sponsor/

* * *

Run a query to confirm it's working:

{
  allBars {
    nodes {
      id
    }
  }
}

{
  "data": {
    "allBars": {
      "nodes": []
    }
  }
}

Then kill the database docker server and run the query again:

{
  "errors": [
    {
      "message": "connect ECONNREFUSED 127.0.0.1:6543"
    }
  ]
}

Then restart the database server (and re-run the setup SQL, since Docker threw it away):

{
  "data": {
    "allBars": {
      "nodes": []
    }
  }
}

This runs as expected - PostGraphile will automatically reconnect to the database server when it comes back.

Can you give better reproduction instructions? Are you sure it's PostGraphile at fault here, and not the name resolution system?

[krzkaczor]

@benjie thanks for your reply! I will prepare exact repo reproducing this issue.

[krzkaczor]

@benjie that's super weird... I didn't change anything and I can't reproduce it now. Sorry for wasting your time.

I will monitor this issue and will come back if I have a real reproduction example. Thanks again!

[benjie]

No problem; thanks for letting me know :+1: You may want to check out the --retry-on-init-fail flag too.