Boto3: Disable debug log messages

[sibelius]

Is there a way to disable or be less verbose ?

I'm using it in my celery task and boto3 is printing a lot of debug messages

[kyleknap]

Can you provide a code snippet of how you are setting up your logger? There is not really any way we can stop boto3/botocore from logging the messages to the logger as that is hard coded, but you should be able to prevent the logger from printing the messages out depending on how your handlers on you logger is set. Otherwise, we are not doing anything special we are just using the builtin logger module: https://docs.python.org/2/library/logging.html#module-logging

[sibelius]

are you using a specific name for boto logger? so I can just get it using getLogger and remove the handlers or change it to be less verbose or redirecting it to a file

[kyleknap]

Yeah for the logger, we default to 'boto3': https://github.com/boto/boto3/blob/develop/boto3/__init__.py#L35.

[sibelius]

thanks

[edwardxwu]

Each module in boto3 uses a different logger based on name. If you print the logging.Logger.manager.loggerDict after import, you can see a bunch of loggers were created like:

['nose.case', 'botocore.vendored.requests.packages', 'boto3.resources.collection', 'botocore.vendored.requests.packages.urllib3.util.retry', 'botocore.retryhandler', 'boto3', 'nose.proxy', 'botocore.utils', 'nose.loader', 'botocore.vendored.requests.packages.urllib3', 'botocore', 'botocore.vendored.requests.packages.urllib3.util', 'botocore.handlers', 'botocore.awsrequest', 'nose.config', 'botocore.credentials', 'botocore.hooks', 'botocore.parsers', 'botocore.response', 'nose.plugins.manager', 'pandas', 'botocore.endpoint', 'botocore.vendored.requests.packages.urllib3.poolmanager', 'botocore.waiter', 'boto3.resources.base', 'botocore.vendored', 'boto3.resources.model', 'boto3.resources.factory', 'nose.core', 'bcdocs', 'pandas.io.gbq', 'botocore.compat', 'boto3.resources', 'botocore.auth', 'boto3.resources.action', 'botocore.vendored.requests.packages.urllib3.connectionpool', 'nose.plugins', 'nose.importer', 'nose.failure', 'nose.result', 'botocore.vendored.requests', 'nose', 'nose.selector', 'nose.suite', 'botocore.client']

To increase the logging level, you would need to do that on all 3 top lvl loggers:

logging.getLogger('boto3').setLevel(logging.WARNING)
logging.getLogger('botocore').setLevel(logging.WARNING)
logging.getLogger('nose').setLevel(logging.WARNING)

[sibelius]

@edwardxwu may I use the same function (setLevel) to disable the logging?

[edwardxwu]

you can set it to a super high lvl like logging.CRITICAL or just disable the propagation to the root logger by logger.propagate = False

[p1nox]

@edwardxwu I tried your approach setting just CRITICAL level:

logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('nose').setLevel(logging.CRITICAL)

But I still see some logs from DEBUG level:

2016-08-18 13:46:17,065 level=DEBUG file="utils.py:412" message="Acquiring 0"
2016-08-18 13:46:17,069 level=DEBUG file="tasks.py:192" message="JoinFuturesTask({}) about to wait for the following futures [<Future at 0x7f504c23fa50 state=running>]"
2016-08-18 13:46:17,069 level=DEBUG file="tasks.py:195" message="JoinFuturesTask({}) about to wait for <Future at 0x7f504c23fa50 state=running>"
2016-08-18 13:46:17,069 level=DEBUG file="utils.py:425" message="Releasing acquire 0/None"
2016-08-18 13:46:18,065 level=DEBUG file="futures.py:248" message="Submitting task IOWriteTask({'offset': 0}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f504c22b310> for transfer request: 0."

Update:

I fixed this by adding logging.getLogger('s3transfer').setLevel(logging.CRITICAL) which is the responsible for the logs in question.

[soopanova]

@p1nox how did you find the name of the logger for s3tranfer?

[leobarcellos]

@soopanova and all that came here looking to disable log messages or set levels:
you can find all loggers like this:

import logging
loggers_dict = logging.Logger.manager.loggerDict

[adamloving]

Which leads me to...

import boto, boto3
for name in logging.Logger.manager.loggerDict.keys():
    if ('boto' in name) or ('urllib3' in name):
        logging.getLogger(name).setLevel(logging.WARNING)

[safvanck]

End up with this...

import boto, boto3
for name in logging.Logger.manager.loggerDict.keys():
    if ('boto' in name) or ('urllib3' in name) or ('s3transfer' in name) or ('boto3' in name) or ('botocore' in name) or ('nose' in name):
        logging.getLogger(name).setLevel(logging.CRITICAL)

[parinpshah94]

If you want to make sure that the logging module only collects log from your classes, avoid using root loggers. Essentially, instead of importing loggers with logging.getLogger(), import it with logging.getLogger('module_name'). This way you won't have to reset logging levels for specific module such as boto3.

[francoatmega]

@edwardxwu I tried your approach setting just CRITICAL level:

logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('nose').setLevel(logging.CRITICAL)

But I still see some logs from DEBUG level:

2016-08-18 13:46:17,065 level=DEBUG file="utils.py:412" message="Acquiring 0"
2016-08-18 13:46:17,069 level=DEBUG file="tasks.py:192" message="JoinFuturesTask({}) about to wait for the following futures [<Future at 0x7f504c23fa50 state=running>]"
2016-08-18 13:46:17,069 level=DEBUG file="tasks.py:195" message="JoinFuturesTask({}) about to wait for <Future at 0x7f504c23fa50 state=running>"
2016-08-18 13:46:17,069 level=DEBUG file="utils.py:425" message="Releasing acquire 0/None"
2016-08-18 13:46:18,065 level=DEBUG file="futures.py:248" message="Submitting task IOWriteTask({'offset': 0}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f504c22b310> for transfer request: 0."

Update:

I fixed this by adding logging.getLogger('s3transfer').setLevel(logging.CRITICAL) which is the responsible for the logs in question.

It Works! Thx

[jonathan-kosgei]

To expand on @parinpshah94 response. This is the recommended convention in the Python Logging docs.

import logging
logging.getLogger(__name__)

[8secz-johndpope]

import logging

logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('nose').setLevel(logging.CRITICAL)
logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)