Patroni: Superfluous error on starting replication?

[jberkus]

This is on the movebasebackup branch, but does not appear to be caused by any of the code there.

2015-10-27 17:51:05,351 INFO: Starting new HTTP connection (1): 172.17.0.5
2015-10-27 17:51:05,357 INFO: trying to bootstrap from leader
waiting for server to start....LOG: database system was interrupted; last known up at 2015-10-27 17:51:05 PDT
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/20000F0
LOG: database system is ready to accept read only connections
FATAL: could not start WAL streaming: ERROR: replication slot "postgresql1" does not exist

done
server started
2015-10-27 17:51:07,240 INFO: bootstrapped from leader

The strange thing about that error is that the slot exists, the replica is replicating, and everything is working. Seems like we could avoid having the error sent to the output by checking pg_replication_slots on the master. Not sure what code is giving rise to this, though.

[alexeyklyukin]

This is normal: the replica registers itself in the DCS and tries to connect to a given replication slot on the master. It fails for a few times initially, but once the master notices the new member and creates the slot - the error disappears.

An obvious enhancement to this would be to create the slot by the replica itself by connecting to the master with the replication protocol. The master, then, will only be responsible for removing obsolete replication slots once the corresponding members disappear from DCS.

[alexeyklyukin]

Err... won't fix? Why? We can make this a task of the replica, or, at least, suppress the warnings?

[jberkus]

This is part of a more general issue that Patroni spews a lot of alarming-looking error messages when it is, in fact, operating normally. That becomes a major UX issue, because it makes it very difficult for users to tell the difference between an error they should care about and one they shouldn't, and will lead to a lot more non-issues posted here, if we don't start supressing non-errors ...

[feikesteenbergen]

These logs are generated by PostgreSQL, they are mixed with the Patroni logs if log_destination = stdout. Most subprocesses called by Patroni are currently writing their stdout/stderr to the console straight away. We could have a look here to see if we can have multiple log streams / log verbosity.

[LappleApple]

@feikesteenbergen @alexeyklyukin Did you have a chance to take a look, and what was the result?

[alexeyklyukin]

Yes, we keep this open.

[LappleApple]

@alexeyklyukin But what was the result? What is the next action to take?

[alexeyklyukin]

This should be discussed further in this ticket.

[alexeyklyukin]

This error comes from PostgreSQL, we have no influence over it (and masking ERRORs from PostgreSQL is dangerous.)