Scrapy: Is possible to configure rollover log file via settings?

[Urahara]

Or the only way today to rotate logs is to implement a handle on spiders?

I configured:

LOG_FILE = 'log.txt'
LOG_LEVEL = 'INFO'

File name and log level, but unfortunately on doc i can't find anything related to rotate the logs, the only way i see that is possible is to add another handler on spiders, i want to know if is possible to change to change this on settings.py.

If is not possible i think this will be a great feature.

[leonardfrank]

Hi, @Urahara

What do you mean rotate logs? And what is it for? Would you like to explain it more clearly? That would be a great help for someone really want to solve this issue.

[Urahara]

@leonardfrank When i say rotate i mean "rollover", like:

TimedRotatingFileHandler('server.log', when='midnight', interval=1)

In this case my log will rollover when midnight, because the current configs only generate a file and never rollover making a one big file, instead of various smalls files:

log.txt.24042018
log.txt.23042018
log.txt.22042018

[leonardfrank]

Hi, @Urahara

I get your point. The answer is No. Neither the official document or the source code shows us a way to your idea.

Assuming your demand for storing large-size logs, based on you need that many files, I recommend you use Sentry to store logs.

[Urahara]

I figure out how to configure this properly:

On top of settings.py

from logging.handlers import TimedRotatingFileHandler
from scrapy.utils.log import configure_logging
import logging
logHandler = TimedRotatingFileHandler('scraping.log', when='midnight', interval=1)
configure_logging(install_root_handler=False)
logging.basicConfig(handlers=[logHandler])