Gallery-dl: Document how to use as library

[lyz-code]

Hi, I intend to use gallery-dl as a library for a program to periodically fetch the selected sources.

I already do it with youtube-dl as it's documented in their docs.

Is there a simple way to do the following with gallery-dl?

ydl_opts = {
    'format': 'bestaudio/best',
    'postprocessors': [{
        'key': 'FFmpegExtractAudio',
        'preferredcodec': 'mp3',
        'preferredquality': '192',
    }],
    'logger': MyLogger(),
    'progress_hooks': [my_hook],
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
    ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc'])

I've seen in this issue that you can process one url:

>>> from gallery_dl import job
>>> j = job.DataJob("https://imgur.com/0gybAXR")
>>> j.run()
[ ... ]

But it doesn't downloads the file, nor it works with gallery links such as https://www.deviantart.com/{{ user }}
Thank you

[mikf]

Use a DownloadJob instance to actually download stuff.
A DataJob object will only collect the data returned by its Extractor and not do much else with it.

Setting config options should be done via the functions in config.py, like config.set(), or by directly manipulating the _config dict in there. You can load config files with config.load().

For example:

from gallery_dl import config, job

config.load()  # load default config files
config.set(("extractor",), "base-directory", "/tmp/")
config.set(("extractor", "imgur"), "filename", "{id}{title:?_//}.{extension}")

for url in urls:
    job.DownloadJob(url).run()

[lyz-code]

Thank you @mikf, it helped a lot.

For others reading this issue, to know which options you need to set use the two config examples (1 and 2) with the options description. Here are some options I've set:

config.set(('extractor',), "archive", '~/.gallery-dl/archive.sql')
config.set(('extractor',), "base-directory", '~/downloads')
config.set(('extractor', 'deviantart'), "image-range", '1-10')
config.set(('extractor', 'deviantart'), "flat", False)
config.set(('extractor', 'deviantart'), "metadata", True)
config.set(
    ('extractor',),
    'postprocessors',
    [
        {
            "name": "metadata",
            "mode": "json",
        }
    ]
)

I'm still unable to configure the output, what am I doing wrong?

config.set(('output',), 'mode', 'terminal')
config.set(
    ('output',),
    'log',
    {
        "level": "info",
        "format": {
            "debug": "\u001b[0;37m{name}: {message}\u001b[0m",
            "info": "\u001b[1;37m{name}: {message}\u001b[0m",
            "warning": "\u001b[1;33m{name}: {message}\u001b[0m",
            "error": "\u001b[1;31m{name}: {message}\u001b[0m"
        }
    },
)
config.set(
    ('output',),
    'logfile',
    {
        "path": "log.txt",
        "mode": "w",
        "level": "debug"
    },
)
config.set(
    ('output',),
    "unsupportedfile",
    {
        "path": "unsupported.txt",
        "mode": "a",
        "format": "{asctime} {message}",
        "format-date": "%Y-%m-%d-%H-%M-%S"
    },
)

It produces the following config._config

 'output': {'log': {'format': {'debug': '\x1b[0;37m{name}: {message}\x1b[0m',
                               'error': '\x1b[1;31m{name}: {message}\x1b[0m',
                               'info': '\x1b[1;37m{name}: {message}\x1b[0m',
                               'warning': '\x1b[1;33m{name}: {message}\x1b[0m'},
                    'level': 'info'},
            'logfile': {'level': 'debug', 'mode': 'w', 'path': 'log.txt'},
            'mode': 'auto',
            'unsupportedfile': {'format': '{asctime} {message}',
                                'format-date': '%Y-%m-%d-%H-%M-%S',
                                'mode': 'a',
                                'path': 'unsupported.txt'}}}

Which is similar to the config example, but neither unsupported.txt, nor log.txt are being created.

Thanks

[mikf]

All logging output is done via Python's logging module.

You can use that to configure and attach your own handlers to the root logger,
or you call initialize_logging(), configure_logging(), and setup_logging_handler() from output.py after setting your output options.

Take a look at main() and search for output. to see how this is "normally" done.

For example

import logging
from gallery_dl import output

# initialze logging and setup logging handler to stderr
output.initialize_logging(logging.INFO)
# apply config options to stderr handler and create file handler
output.configure_logging(logging.INFO)
# create unsupported-file handler
output.setup_logging_handler("unsupportedfile", fmt="{message}")

[rpdelaney]

@mikf would you accept a PR documenting how to do this?

[mikf]

@rpdelaney Sure. I'd be happy about any sort of contribution, especially documentation. Let me know if you need anything or if I should explain how certain things (are supposed to) work.

[opsoyo]

All logging output is done via Python's logging module.

You can use that to configure and attach your own handlers to the root logger,
or you call initialize_logging(), configure_logging(), and setup_logging_handler() from output.py after setting your output options.

Take a look at main() and search for output. to see how this is "normally" done.

For example

import logging
from gallery_dl import output

# initialze logging and setup logging handler to stderr
output.initialize_logging(logging.INFO)
# apply config options to stderr handler and create file handler
output.configure_logging(logging.INFO)
# create unsupported-file handler
output.setup_logging_handler("unsupportedfile", fmt="{message}")

😅 I tried understanding this without success. For a split async moment, I simply use StringIO for stdout and stderr to capture and match with RegEx. Thankfully this small Discord bot won't mind the hacky method.