Kibana: SIGHUP does not reload logging configuration

[flash1293]

Kibana version: 7.3

Elasticsearch version: -

Server OS version: macOS 10.14.6

Browser version: -

Browser OS version: -

Original install method (e.g. download page, yum, from source, etc.): download page

Describe the bug:

Sending the SIGHUP signal to the Kibana process should reload the logging configuration and start logging to the new location. The logging configuration is not reloaded on macOS, but Kibana monitoring reloads properly.

Steps to reproduce:

1. Configure Kibana (no other settings):

logging.dest: /path/to/kibana.log
pid.file: /path/to/kibana.pid

1. Start Kibana via ./bin/kibana
2. Observe logs flowing into kibana.log
3. Move log file to kibana.old and observe logs still flowing into same file (as expected)
4. Send sighup by kill -HUP $(cat kibana.pid)
5. The following message is logged:

{"type":"log","@timestamp":"2019-08-19T04:33:57Z","tags":["info","monitoring","kibana-monitoring"],"pid":42073,"message":"Re-initializing Kibana Monitoring due to SIGHUP"}

1. Logs continue to flow into kibana.old

Expected behavior:

It should log to kibana.log again and log Reloading logging configuration due to SIGHUP. because of src/core/server/bootstrap.ts:91

This test should make sure it's happening: https://github.com/elastic/kibana/blob/master/src/cli/serve/integration_tests/reload_logging_config.test.js

Not sure why it doesn't fail.

[elasticmachine]

Pinging @elastic/kibana-platform

[spalger]

Closed by https://github.com/elastic/good/issues/3

[restrry]

master still uses the old version of @elastic/good. Looks like we should wait for https://github.com/elastic/kibana/pull/46604

[joshdover]

PR we're waiting on now is https://github.com/elastic/kibana/pull/48026

[restrry]

waiting for https://github.com/elastic/kibana/issues/54168

[spalger]

I gave this a good shake yesterday, trying to come up with a solution in #56660. Unfortunately, I was never able to actually reproduce it so I'm not sure if I have a fix. I'll let @joshdover take a stab at things today and unless he figures out what's going on I think we should close this and recommend that users try out a dedicated logrotate utility or something.

The exact steps I used to try and reproduce:

1. start kibana node scripts/kibana --dev

2. once kibana is started update/create the kibana.dev.yml file with:

   logging.verbose: true
   logging.dest: log.txt
   

3. wait for kibana server to restart

4. check tail log.txt to verify logs are being written to file, take note of the pid being logged

5. in a new terminal tab run the following to keep an eye on where logs are being written

   # you might need to install watch with `brew install watch`
   watch 'echo "log.txt : $(wc -l log.txt)" && echo "log2.txt: $(wc -l log2.txt)"'
   

6. move the log.txt file

   mv log.txt log2.txt
   

7. observe the logs are now being written to log2.txt

8. send SIGHUP to the process

   kill -HUP $PID
   

9. observe the logs are now being written to log.txt

[restrry]

@spalger I repeated all your steps (I used tail log.txt instead of watch) and reproduced the problem locally on both master and https://github.com/elastic/kibana/pull/56660
to note: after SIGNHUP logs in newly created log.txt starts with error

"message":"{ ValidationError: child \"xpack\" fails because [\"monitoring\" is not allowed]\n    at Object.exports.process (/Users/mikhailshustov/work/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOption

and file doesn't contain all the messages.
I suspect there is a problem on the platform side.

[joshdover]

Update: we have decided not to pursue fixing this at this time. We have a new experimental log rotation feature (checkout the logging.rotate configuration options) which you should be able to use in the meantime. We are also moving to a new logging infrastructure in a future 7.x release which will be the only logging option in 8.0+. We recommend switching to this once it is available as well.

If there is significant need from the community, please comment on this issue and we will reprioritize.