Redash: The logger is not printing every type of log name

[ankitchiplunkar]

Issue Summary

I am running a fresh redash instance using bootstrap.sh, on a fresh Ubuntu 16.04 on a Google Cloud instance

Steps to Reproduce

1. I set "REDASH_LOG_LEVEL=DEBUG"
2. Trying to see all the logs in file /var/log/supervisor/redash_server-std*
3. See logs only belonging to a certain log names
4. I am interested in seeing the logs from /opt/redash/current/redash/query_runner/bigquery.py
5. Following are the only types of visible logs

[INFO][root]
[INFO][metrics]
[DEBUG][root]
[DEBUG][metrics]
[DEBUG][redash.destinations]
[DEBUG][redash.query_runner]
[DEBUG][passlib.utils.compat]
[DEBUG][requests.packages.urllib3.connectionpool]

I expected to see all the types of debug log names in the log file.

Technical details:

• Redash Version: 3.0.0.b3134
• Browser/OS: Firefox/Ubuntu 16.04
• How did you install Redash: running bootstrap.sh from master repo

Possible reasons

1. Something unexpected is happening due to if settings.LOG_LEVEL != "DEBUG": in line 33 of current/redash/__init__.py --> setup_logging()
2. Python logging multiple modules logger not working outside main program

I would investigate this further but any help is greatly appreciated

[ankitchiplunkar]

Finally, resolved the issue

Reason

The queries are run by celery-workers hence the logs are not printed by werkzeug but rather by the celery workers.

Solution

Restart the celery workers with flag --loglevel=DEBUG to see the logs of queries.
eg.
./bin/run celery worker --app=redash.worker --beat -Qscheduled_queries,queries,celery -c2 --loglevel=DEBUG

Output

[2018-03-26 16:25:38,079][PID:12167][DEBUG][Worker-2] query_reply {u'kind': u'bigquery#getQueryResultsResponse', u'rows': [{u'f': [{u'v': u'5325577'}]}], u'jobReference': {u'projectId': u'*****', u'location': u'US', u'jobId': u'job_iD-wx8HYb11WVJvN9cXygcztt5Cv'}, u'jobComplete': True, u'totalRows': u'1', u'totalBytesProcessed': u'0', u'cacheHit': True, u'etag': u'"OhENgf8ForUUnKbYWWdbr5aJHYs/OEDjZKfwAsz8s_ImBHDzyb6HsDA"', u'schema': {u'fields': [{u'type': u'INTEGER', u'name': u'f0_', u'mode': u'NULLABLE'}]}}
[2018-03-26 16:25:38,080][PID:12167][DEBUG][Worker-2] bigquery replied: {u'kind': u'bigquery#getQueryResultsResponse', u'rows': [{u'f': [{u'v': u'5325577'}]}], u'jobReference': {u'projectId': u'*****', u'location': u'US', u'jobId': u'job_iD-wx8HYb11WVJvN9cXygcztt5Cv'}, u'jobComplete': True, u'totalRows': u'1', u'totalBytesProcessed': u'0', u'cacheHit': True, u'etag': u'"OhENgf8ForUUnKbYWWdbr5aJHYs/OEDjZKfwAsz8s_ImBHDzyb6HsDA"', u'schema': {u'fields': [{u'type': u'INTEGER', u'name': u'f0_', u'mode': u'NULLABLE'}]}}

[RichardLitt]

Thanks for both logging and following up, @ankitchiplunkar!