Lisk-sdk: apt-get update & upgrade messes up postgresql

[karek314]

After doing apt-get update & apt-get upgrade postgresql has been messed up.

log 2016-07-27 18:20:20 GET /peer/list from 104.238.171.84 
log 2016-07-27 18:20:20 POST /peer/blocks from 185.92.221.251 
log 2016-07-27 18:20:33 GET /peer/signatures from 13.70.207.248 
error: Forever detected script was killed by signal: SIGKILL
info 2016-07-27 18:20:39 Lisk started: 0.0.0.0:8000 
log 2016-07-27 18:20:39 error error: FATAL:  password authentication failed for user "lisk_user"
log 2016-07-27 18:20:39 error connection: {"host":"localhost","port":5432,"database":"lisk_main","user":"lisk_user","password":"########","poolSize":20,"poolIdleTimeout":30000,"reapIntervalMillis":1000,"logEvents":["error"]}
fatal 2016-07-27 18:20:39 [Error: FATAL:  password authentication failed for user "lisk_user"
] 
log 2016-07-27 18:20:42 GET / from 176.10.98.139 
log 2016-07-27 18:20:43 GET /favicon.ico from 176.10.98.139 

[karmacoma]

A package upgrade leave any existing databases intact. Are you running both the bundled postgresql and system-wide postgresql at the same time? This could be the cause.

[karek314]

I haven't. Problem is, postegresql after upgrade does not work with lisk anymore. There should be possibility to skip upgrading postgresql or just include any auto-repair when can't connect or/and authentication failed

[ccooper21]

@karek314, I recently experienced and resolved a similar error when installing Lisk from source code. Since this was my first time working with Lisk, I'm not sure if the handling of or behavior regarding Postgres authentication has changed over time.

First, I was able to reproduce the authentication error independently of Lisk with the command line psql postgres://localhost/lisk_test. When running this, I was presented with a Password: prompt, but had not yet set a password for my user. Hence, I could not login.

I resolved this by instead connecting to the database using the psql lisk_test command line and then setting a password for my account using the \password command at the psql prompt. Note that the password being set is associated with the account, and not the database so it will affect more than just the database specified on the command line.

Lastly, I made sure the password specified in the Lisk config.json file was in-sync with the password that I had just set. From there, I was good to go.

[Isabello]

I will fix the documentation to correct the user account credential issues when running lisk from source.

As for handling apt-get update, you as an Administrator will have to be aware of what you have installed on the system when running the command.

[karek314]

@Isabello

be aware of what you have installed on the system when running the command

This is rude thing to say. Shouldn't happen in case of core team. Lisk should have in place ready mechanism to handle this properly, at least detect if db configuration is proper and if not reconfigure. Its pain to reinstall and re-download Lisk db after normal system maintenance.

[Isabello]

I disagree. Its a server administrators role to manage their OS level packages, not ours. In the case of source, you are choosing to run that version which means you are accepting all of the risks of maintaining the environment.

When you run apt-get update you are willfully telling the system to update everything and accepting the risks that comes with that operation.

What you can do is stop the service and lisk prior to executing the command, as the best practice is to stop applications that might be impacted by such an operation.

[karek314]

Im talking that lisk database get messed up after proper update. I did not make postgresql update during time when Lisk or postegresql was running.

[karmacoma]

@karek314 In production, we recommend you run the binary install. It ships with a self-contained postgresql where we _support_ upgrades. There is no need to make life complicated unless you are conducting lisk development. In which case, you should accept some responsibility and know what you are doing :P

Anyway joking apart, as you feel so strongly about it, I will reopen this issue and take a further look after we have sorted the more pressing issues.

[Isabello]

The password error has been corrected by adding a step to the installation documentation to define the password. An upgrade will not cause a database and its defined users to stop working. In the case of postgres it may change your pg_hba.conf file. If that occurs you will need to manually correct the settings.

If you upgrade from 9.5.2 to 9.6 you will need to do a pg_upgrade with the correct syntax to finish migrating between versions.

If you are running Lisk Binary on a node with Postgresql installed you will also need to be aware of potential complications between the packed in Binary DB and the OS level DB software.

[karmacoma]

Closing issue once more based on above from @Isabello. Again, I'll repeat: we recommend you run the binary install. It ships with a self-contained postgresql where we support upgrades. If not, follow our https://github.com/LiskHQ/lisk#installation which is aimed at lisk development, rather than node operation.