Gallery-dl: (Patreon) New "403 Forbidden" Cloudflare CAPTCHA error with 1.15.3

[biznizz]

Have all dependencies and gallery-dl up to date, but have been getting constant 403 errors.

Made sure that session_id cookie was up to date in config, no dice.

Exporting all Patreon cookies into a cookies.txt and updating config to point to it leads to an error which reads "No session_id set", but downloads free posts.

Cloudflare has been blocking for about a whole day at this point, and verbose doesn't really give any useful information.

[Butterfly-Dragon]

yup, can confirm

[mikf]

What version of requests and urllib3 are both of you using? There was an update for both of them in the last couple of days. Maybe that causes these problems?

pip install -U requests==2.24.0 urllib3==1.25.11 should revert to the older versions.
The 1.15.3 .exe is also using the older versions of those libraries.

Exporting all Patreon cookies into a cookies.txt and updating config to point to it leads to an error which reads "No session_id set", but downloads free posts.

There is no other warning/error message and the cookies.txt file actually contains a session_id cookie for patreon?

[Butterfly-Dragon]

oh, let me see...

[Butterfly-Dragon]

urllib3 1.26.2
requests 2.25.0
requests-oauthlib 1.3.0

[Butterfly-Dragon]

yup. can confirm it now totally works. 😅

[Butterfly-Dragon]

so... since i'm antsy about keeping old versions of the libraries, should i remove this from my daily updates and let it update only when you tell me to... or what?

[mikf]

Either that, or you create a virtualenv for gallery-dl to keep its dependencies separate from all the other Python packages.

[Butterfly-Dragon]

no, it actually solved several problems i was having the last 2 days. i would just like to skip this version, but right now i just set it to not update those 2 packages, might resume when they get updated.

[biznizz]

What version of requests and urllib3 are both of you using? There was an update for both of them in the last couple of days. Maybe that causes these problems?

Everything was as up-to-date as possible, so it would have been requests 2.25.0 and urllib3 1.26.2.

pip install -U requests==2.24.0 urllib3==1.25.11 should revert to the older versions.

Reverted and now the Cloudflare issue is gone. Will the next update of gallery-dl include support for the latest versions of urllib3 and requests? Or should I just keep the dependencies on these older versions for the foreseeable future?

There is no other warning/error message and the cookies.txt file actually contains a session_id cookie for patreon?

Yes, even when the cookies.txt had the session_id cookie in the file and the config was set to read that file, it gave an error like the cookie wasn't there, but it downloaded unlocked posts.

[biznizz]

With the latest dev build being 1.16.0.dev0, is it safe to upgrade the dependancies to the latest versions.

Pip has informed me that urllib3, requests, packaging, and cffi all have new versions that can be installed, but I don't want to risk hitting more Cloudflare issues.

[Butterfly-Dragon]

The fix was given to us 12 days ago.
Urllib3 most recent is from 15 days ago
https://github.com/urllib3/urllib3/releases
And requests is from 17 days ago
https://github.com/psf/requests/releases
Sooo... i guess not?
Packaging has updated last time 10 hours ago and cffi is not developed on github so that should be fine too...

[mikf]

It should be OK to update everything except urllib3.

The problem lies in the new default behavior when establishing a TLS connection in urllib3 version 1.26.0 (Changelog). It might be possible to monkey-patch said defaults to how they where before 1.26.0, but this is a lot of grunt work and the fact that gallery-dl doesn't even use this library directly, but only through requests, doesn't help.

[Butterfly-Dragon]

but... why should using a better encryption give us a 403 error?!? 🤔

[Butterfly-Dragon]

hmmm... apparently in TLS 2.0 the server looks for the client's certificate to continue, i guess ""we"" are not providing a certificate that works for patreon?

[mikf]

This is more about whether or not Cloudflare thinks a request comes from a browser controlled by a human being or a bot, and it uses the TLS handshake among other things to determine that. Why Cloudflare believes requests with urllib3 1.26 are from a bot, but not with 1.25 is beyond me, but at least we know what works, just not the why.

For example the latest Firefox versions only accept TLS 1.2 and 1.3, as does urllib3 1.26 (bot according to Cloudflare) in contrast to urllib3 1.25, which allows TLS 1.0, 1.1, 1.2, and 1.3 (not a bot). Maybe changing gallery-dl's user agent to some newer browser version is all that is needed to make it work with urllib3 1.26? (It currently uses Firefox 68 as user agent)

[Butterfly-Dragon]

oooh that's the problem!

I'm specifying that i'm chrome 88 as a user agent and giving cookies that come from chrome 88.

So if Urllib3 says it's firefox then they see chrome... if you ask me that's sus 🤣👍💖

[mikf]

No, urllib3 doesn't say it's a specific browser, gallery-dl by default is saying it is Firefox 68. When you've already changed gallery-dl's user agent and it still doesn't work with 1.26, we can discard my previous assumption ("Maybe changing gallery-dl's user agent ... to make it work with urllib3 1.26")

[Butterfly-Dragon]

hmmm... i have not tried "not changing" it though.

[biznizz]

I'm going to assume that even with the current 1.16.0 version of gallery-dl, that it's still not safe to upgrade to the latest version of urllib3?

[Butterfly-Dragon]

Urllib3 has been updated last time on the has been stuck on version 1.26.2 since Nov. 12

[biznizz]

Doing a quick update report:

Had upgraded chardet and requests to latest versions while keeping urllib3 at 1.25.11, got errors and downgraded back down to 3.0.4 and 2.24.0 respectively again, and error disappeared.

Gallery-dl is at the latest release version.

[AplhaSlayer1964]

I believe I have tried everything that has been suggested but still getting the error. Can anyone explain all the versions they are using that makes it work?

[biznizz]

I believe I have tried everything that has been suggested but still getting the error. Can anyone explain all the versions they are using that makes it work?

So I'm able to keep gallery-dl up to date with the current version 1.16.0.

To reduce my errors, I've had to keep the following packages at these versions.

chardet 3.0.4
requests 2.24.0
urllib3 1.25.11

There still might be the occasion where Cloudflare might start causing errors with individual post URL's, but running a generic "/posts" URL on a user page should get everything without issue.

[biznizz]

Latest report: Every dependency can be upgraded to the latest version except for urllib3 which still causes Cloudflare issues when trying to download either individual or whole user pages from Patreon.

[Butterfly-Dragon]

thanks! 👍💖

[Ogwalla]

I am also having issues with Cloudflare.

[Sjoerd82]

Yes, same here:

gallery-dl.exe -v --cookies "cookie.txt" https://www.patreon.com/posts/12345678
[gallery-dl][debug] Version 1.16.3
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.18362
[gallery-dl][debug] requests 2.25.1 - urllib3 1.25.11
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/posts/12345678'
[patreon][debug] Using PatreonPostExtractor for 'https://www.patreon.com/posts/12345678'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /posts/12345678 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/posts/12345678'

I'm not sure if the packaged executable (gallery-dl.exe) uses any of the locally installed Python (perhaps someone can enlighten me on this). In any case I ran this on two machines (same result), one has Python 3.7.0, and one has 3.9.1. -- Given that the -verbose option reports Python 3.7.9, I guess not.

[Hrxn]

The packaged executable is fully self-contained, so no, it does not use any locally installed Python or something.

[slowthgt]

Since this issue seems to still be present as of 1.16.4 (compiled exe), the workaround I found to work was to copy not only the session_id, but also the __cf_bm and __cfduid cookie, along with the correct User-Agent. This likely will not be a good method for people running this on a schedule or daemon, but in a pinch, it works.

[Archivings]

Since this issue seems to still be present as of 1.16.4 (compiled exe), the workaround I found to work was to copy not only the session_id, but also the __cf_bm and __cfduid cookie, along with the correct User-Agent. This likely will not be a good method for people running this on a schedule or daemon, but in a pinch, it works.

How do you have your config set up to read the additional cookies, and User-Agent?
Also does this allow for urllib3 to be reupdated, or is it still required to be kept 1.25.11?

[slowthgt]

How do you have your config set up to read the additional cookies, and User-Agent?

Cookies are stored on the main extractor bracket (cookies on https://github.com/mikf/gallery-dl/blob/b0cff979b1b07dfbc0309354e6ef84de5793d3bd/docs/gallery-dl.conf#L7 and user agent on https://github.com/mikf/gallery-dl/blob/b0cff979b1b07dfbc0309354e6ef84de5793d3bd/docs/gallery-dl.conf#L15).
The User-Agent should be the one where you logged in with your account to get the cookies with (this assuming that they check the User-Agent, which they might).

Also does this allow for urllib3 to be reupdated, or is it still required to be kept 1.25.11?

1.16.4 (where I tested) contains urllib3 1.25.11, which could mean that yes, it likely still requires that specific version

[Archivings]

Yep! That seems to be a working fix for the issue, for now at least.
Though I am not 100% certain on this, I unfortunately can't test it on any patron-exclusive posts at the moment, but seeing as how before I couldn't even download a users free posts it seems more likely than not that it'll work on that side of posts too

[Butterfly-Dragon]

@slowthgt if i update urllib3 it still does not work for me. Or is this a fix for the compiled exe only?

[Archivings]

@slowthgt if i update urllib3 it still does not work for me. Or is this a fix for the compiled exe only?

Aye, urllib3 still has to be kept downgraded to 1.25.11 for this to work.
I run the .exe so I can't say whether or not this would work with one of the other builds, but I don't see why it wouldn't. Just potentially requiring some more in-depth work to incorporate to whichever build you run.

[valdearg]

Did we get a specific fix for this issue in the end? I use the compiled exe version and it gives me the 403 Forbidden error.

[Butterfly-Dragon]

@valdearg use the compiled version and import a TXT cookie file (exporting cookies from your browser can be done via various extensions depending on your browser

[biznizz]

Still on urllib3 1.25.11 and up to date on gallery-dl with 1.16.5, and there are still large amounts of time where trying to rip posts or users from Patreon results in Cloudflare CAPTCHA 403 Forbiddens.

Sometimes it'll work, other times errors.

[mikf]

So, there have been some changes in this regard and a new browser option got added (https://github.com/mikf/gallery-dl/commit/cf5fa75d4c463a5b8d916147ca32a5ed8b27d691). Setting this option to "firefox" or "chrome" tells gallery-dl to use the same default HTTP headers and TLS ciphers as those two browsers, and this time it takes the local OS into account and it should work regardless of urllib3 version.

Please test this on Patreon user URLs and post URLs with and without this option enabled and let me know if this is more successful than it was before this change.

I do want to make "browser": "firefox" the default for Patreon, but wanted to get some feedback if it actually works first.

[biznizz]

I've updated urllib3 to 1.26.3 and ran several User (as in the entirety of the Patreon timeline and Posts (as in individual posts) after updating to the latest dev build of gallery-dl.

No Cloudflare issues so far, but will keep you updated.

For record, my extractor settings in config has this structure for Patreon:

"patreon": {
       "browser": "firefox",
            "cookies": {
                "session_id": "secret code"
            }
        },

[flaccidbagel]

At least so far, trying out the browser option doesn't seem to be producing any different results, neither with firefox nor chrome; downloads are still failing with a 403.

Whatever changed was very recent, as I had just utilized the tool a few days ago.

[Butterfly-Dragon]

i was able to do0wnload from patreon fine with the updated urllib3 and "chrome" setting

[biznizz]

Well, since things seem to mostly be resolved, I"m going to close this particular issue.

If anything pops up with Cloudflare and the new browser option, me or someone else will start a new ticket.

[Butterfly-Dragon]

iiit's back.

apparently i have to re-login and even then it lasts exactly for 1 session.

[biznizz]

iiit's back.

apparently i have to re-login and even then it lasts exactly for 1 session.

? Re-log in? I mean, other than replacing the session_id cookie about once a month, what exactly is it saying to do?

I've also not had any errors (so far) when running multiple runs of User or Post urls.

Can you post how you have your Patreon extractor settings are in your config?

[Butterfly-Dragon]

sute thing. General part of the config:

{
    "extractor":
    {
       "base-directory": "D:/Downloads/Downloader/Patreon/",
        "directory": ["files"],
        "filename": "{filename}.{extension}",
        "archive": "D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-patreon-archive.sqlite3",
        "cache":
            {
            "file": "D:/Downloads/Downloader//!Downloader/SQL/tmp/cache.sqlite3"
            },
        "skip": "abort:10",
        "retries": -1,
        "sleep": 0,
        "timeout": null,
        "postprocessors": null,
        "cookies": "D:/Downloads/Downloader//!Downloader/cookies.txt",
        "parent-directory": true,
        "adjust-extensions": true,
        "refresh-token": "cache",
        "browser": "chrome",
        "oauth":
        {
            "browser": true
        },

patreon specific:

        "patreon":
        {
            "directory": ["{creator[full_name]}"],
            "filename": "{creator[full_name]} {creator[vanity]} - {date} {title} {num:03} - {filename}.{extension}"
        },

ending:

    "downloader":
    {
        "part-directory": null,
        "rate": null,
        "retries": -1,
        "timeout": 30.0,
        "part": true,

        "http":
        {
            "mtime": true,
            "rate": null,
            "retries": -1,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": -1,
            "timeout": 30.0,
            "verify": true,
            "format": "bestvideo+bestaudio/best",
            "outtmpl": null
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": {
            "level": "info",
            "format": {
                "debug"  : "\u001b [debug   {name}: {message}\u001b ]",
                "info"   : "\u001b [info    {name}: {message}\u001b ]",
                "warning": "\u001b [warning {name}: {message}\u001b ]",
                "error"  : "\u001b [error   {name}: {message}\u001b ]"
            }
        },
        "logfile": {
            "path": "D:/Downloads/Downloader/RoboLogPatreon.txt",
            "mode": "w",
            "level": "debug"
        }
    },

    "cache": {
        "file": "D:/Downloads/Downloader//!Downloader/SQL/cache.sqlite3"
    },

    "netrc": false
}

[biznizz]

@Butterfly-Dragon

How strange, you don't plug your session_id cookie directly into your config file? You have it read directly from your cookies.txt file in your postprocessors setting?

Try to use the settings I posted for my Patreon extractor, where "browser: "firefox" and the cookie is directly plugged in and see if that improved anything.

[Butterfly-Dragon]

That is cookies from chrome.
I don't have firefox.

[biznizz]

I don't use Firefox either (I used Waterfox, a forked browser).

I'm thinking that the use of the browser setting is to emulate how the browser would get past the captcha. Either way, it couldn't hurt to try.

[rautamiekka]

I don't use Firefox either (I used Waterfox, a forked browser).

fork = Waterfox == Firefox. It's not another browser simply by being a fork; forks are nearly always the same type as the original, that's why you mostly can use Firefox addons in Waterfox.

[Butterfly-Dragon]

yeh, if i had cookies from microsoft edge (for... masochistic reasons, i guess) i would still need to set it as chrome, since it's a chromium browser.

[biznizz]

Well, either way, I'm sure that whatever browser you're using is moot since I'm sure that, as I said earlier, it's more about emulating browser behavior using those settings, with Chrome or Firefox settings.

I'm sure mikf can explain this. I'll reopen the issue since I started this ticket.

[mikf]

Well, I can only make assumptions. For one, the browser "emulation" isn't particularly good. gallery-dl through requests/urllib3 only uses HTTP/1.1, while all modern browsers use HTTP/2. Chrome also sends a lot of HTTP/2 specific headers (:authority, ..., sec-fetch-mode, ...) which don't get sent by gallery-dl.

Then again, both browser=firefox as well as browser=chrome work on my machine, while browser=null causes the usual 403 Forbidden. browser=firefox is actually more or less the same as before, except a slight bit better when it comes to accuracy in terms of HTTP headers and TLS handshake.

Did you at least try browser=firefox and can confirm it also doesn't work, @Butterfly-Dragon ?
Could you try it with the same setup as biznizz, i.e. only the session_id cookie for Patreon?
What about gallery-dl v1.16.5 with urllib3 1.25.11?
Can you connect to Patreon with cloudscraper, or does even that result in a CAPTCHA for you?

Also, as the others already said, cookie origin shouldn't matter here.

[Butterfly-Dragon]

i reverted to urllib3 1.25.11 this afternoon as a test __without__ re-logging into patreon and re-exporting the cookies.

It downloaded fine.

Aside from the fact any post that were blocked from my subscribed tiers would reset the count of "skip", which ... i do not know if that was the desired effect.

But yeah, worked absolutely fine with the urllib3 downgrade.

without it i have to re-login and re-export the cookies daily.

[Hrxn]

@mikf You probably already know this, but just in case: https://github.com/encode/httpx

This might be a viable alternative to move on from the requests/urllib3 combo.
I'm aware that this could require some deep rewrites of internals, but on the other hand, you've mentioned planned structural changes for a possible (maybe) future 2.0 version, if I recall that correctly. So, I don't know, but maybe this is something worth trying.

I'm not really in a position to judge here, so all I could do so far, given the presence of all these projects on this "social network" for software, was doing a bit of cross-referencing/researching/stalking the contributor page.
Seems good so far 👍 Looks like they are all involved in developing and maintaining important core projects of the Python ecosystem.

[biznizz]

I can confirm that I can still download User and individual Posts from Patreon just fine, with the settings I've posted earlier, with urllib3 still up-to-date. No Cloudflare issues with them at all since putting browser setting in the Patreon extractor.

I can't imagine why Butterfly would have to export their cookies every day, I've still only had to re-export them once a month due to them having a short lifespan.