Jackett: Cloudflare reCaptcha to hCaptcha

[philou669]

Please use the search bar at the top of the page and make sure you are not creating an already submitted issue.
Check closed issues as well, because your issue may have already been fixed.

Please read our Contributing Guidelines before submitting your issue to ensure a prompt response to your bug.

Environment

OS:

.Net Runtime: [.Net-Core/.Net-Framework/Mono]

.Net Version:

Jackett Version:

Last Working Jackett Version:

Are you using a proxy or VPN? [yes/no]

Description

[List steps to reproduce the error and details on what happens and what you expected to happen]

Logged Error Messages

[Place any relevant error messages you noticed from the logs here.]

[Make sure you attach the full logs with your personal information removed in case we need more information]

Screenshots

[Place any screenshots of the issue here if needed]

[MalaGaM]

Need more new HEADERS requests, i think.
-H 'Upgrade-Insecure-Requests: 1'
-H 'TE: Trailers'
User-Agent and cookie

work with curl.
I don't know how to modify the definition to add headers and try it for myself.

login:
method: cookie

inputs:
  cookie: "{{ .Config.cookie }}"
  user-agent: "[ .Config.useragent ]"
  TE: Trailers
 Upgrade-Insecure-Requests: 1

[garfield69]

Unfortunately I don't have any issues using the YGGCookie in my region, so there is little point in my testing this for you.

But you can test this for yourself and let us know if it works.

• shutdown the Jackett service
• edit with a plain text editor the YGGCookie yaml definition
  linux ~/.config/Jackett/Definitions\yggcookie.yml
  windows %ProgramData%\Jackett\Definitions\yggcookie.yml
• add your additional headers using the same template, you can hard-code a string instead of using a variable, for example "[1]" or "[a string of text]"
• start the Jackett service and test

[MalaGaM]

Unfortunately I don't have any issues using the YGGCookie in my region, so there is little point in my testing this for you.

But you can test this for yourself and let us know if it works.

• shutdown the Jackett service
• edit with a plain text editor the YGGCookie yaml definition
  linux ~/.config/Jackett/Definitions\yggcookie.yml
  windows %ProgramData%\Jackett\Definitions\yggcookie.yml
• add your additional headers using the same template, you can hard-code a string instead of using a variable, for example "[1]" or "[a string of text]"
• start the Jackett service and test

Tks for reply.
Like this ?

  login:
    method: cookie
    inputs:
      cookie: "{{ .Config.cookie }}"
      user-agent: "[ .Config.useragent ]"
      Accept: [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
      Accept-Language: [en-US,en;q=0.5]
      Referer: [https://www2.yggtorrent.se/engine/search?name=multi&do=search]
      Upgrade-Insecure-Requests: [1]
      TE: [Trailers]

Into login->inputs, right ?

No working by Jackett and work with curl, 
Jackett.Common.IndexerException: Exception (yggcookie2): Clearance failed after 30 attempt(s). ---> CloudflareSolverRe.Exceptions.CloudflareClearanceException: Clearance failed after 30 attempt(s).
  at CloudflareSolverRe.ClearanceHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x001f9] in <44fc4e0cb3fd4a268a136236a74d835e>:0 
  at System.Net.Http.HttpClient.FinishSendAsyncBuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x0017e] in <7ecf813f2d314058b05c6c092c47b77a>:0 
  at Jackett.Common.Utils.Clients.HttpWebClient.Run (Jackett.Common.Utils.Clients.WebRequest webRequest) [0x0048a] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Utils.Clients.WebClient.GetString (Jackett.Common.Utils.Clients.WebRequest request) [0x0010b] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Indexers.BaseWebIndexer.RequestStringWithCookies (System.String url, System.String cookieOverride, System.String referer, System.Collections.Generic.Dictionary`2[TKey,TValue] headers) [0x000cb] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Indexers.CardigannIndexer.PerformQuery (Jackett.Common.Models.TorznabQuery query) [0x0086f] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Indexers.BaseIndexer.ResultsForQuery (Jackett.Common.Models.TorznabQuery query) [0x00091] in <8f56f94e7fe949768d108e9933ee1a56>:0 
   --- End of inner exception stack trace ---
  at Jackett.Common.Indexers.BaseIndexer.ResultsForQuery (Jackett.Common.Models.TorznabQuery query) [0x000ee] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Indexers.BaseWebIndexer.ResultsForQuery (Jackett.Common.Models.TorznabQuery query) [0x0006b] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Common.Services.IndexerManagerService.TestIndexer (System.String name) [0x000a3] in <8f56f94e7fe949768d108e9933ee1a56>:0 
  at Jackett.Server.Controllers.IndexerApiController.Test () [0x000e1] in <c90afcd519fd41c891c5a20c4f452913>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+TaskOfIActionResultExecutor.Execute (Microsoft.AspNetCore.Mvc.Infrastructure.IActionResultTypeMapper mapper, Microsoft.Extensions.Internal.ObjectMethodExecutor executor, System.Object controller, System.Object[] arguments) [0x00071] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync () [0x00131] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync () [0x0009e] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Filters.ActionExecutedContext context) [0x0001b] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+State& next, Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+Scope& scope, System.Object& state, System.Boolean& isCompleted) [0x00382] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync () [0x0002f] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter () [0x0009f] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Filters.ResourceExecutedContext context) [0x0001b] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+State& next, Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+Scope& scope, System.Object& state, System.Boolean& isCompleted) [0x00840] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync () [0x0002f] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync () [0x0012e] in <b4ef600f4a594fe2865a8f97f915fb9d>:0 
  at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke (Microsoft.AspNetCore.Http.HttpContext httpContext) [0x001cb] in <6092a16d93814eba828b517a2b132f80>:0 
  at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke (Microsoft.AspNetCore.Http.HttpContext context) [0x00384] in <427697fe42b7459ba5302fb76d339d3b>:0 
  at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke (Microsoft.AspNetCore.Http.HttpContext context) [0x0043e] in <f352e566abf6421e87eafbcf57a0b237>:0 
  at Microsoft.AspNetCore.Builder.Extensions.UsePathBaseMiddleware.Invoke (Microsoft.AspNetCore.Http.HttpContext context) [0x001c8] in <dd7bbb1eb6cb4178a82cd5136b2606b8>:0 
  at Jackett.Server.Middleware.CustomExceptionHandler.Invoke (Microsoft.AspNetCore.Http.HttpContext httpContext) [0x0008a] in <c90afcd519fd41c891c5a20c4f452913>:0

curl command (copy from firefox dev tool f12)

curl 'https://www2.yggtorrent.se/engine/search?name=multi&do=search' 
-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0' 
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' 
-H 'Accept-Language: en-US,en;q=0.5' --compressed
-H 'Connection: keep-alive'
-H 'Referer: https://www2.yggtorrent.se/engine/search?name=french&do=search' 
-H 'Cookie: __cfduid=****; cf_clearance=****; __ga=****; __cf_bm=****; ygg_=***'
-H 'Upgrade-Insecure-Requests: 1'
 -H 'TE: Trailers'

[garfield69]

add double-quotes to your values "[Trailers]"

[MalaGaM]

Same with

  login:
    method: cookie
    inputs:
      cookie: "{{ .Config.cookie }}"
      user-agent: "[ .Config.useragent ]"
      Accept: "[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]"
      Accept-Language: "[en-US,en;q=0.5]"
      Referer: "[https://www2.yggtorrent.se/engine/search?name=multi&do=search]"
      Upgrade-Insecure-Requests: "[1]"
      TE: "[Trailers]"

Please enable cookies.
One more step
Please complete the security check to access www2.yggtorrent.se
Why do I have to complete a CAPTCHA?
Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.

What can I do to prevent this in the future?
If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.

If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.

Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the Firefox Add-ons Store.

Here is the message from Cloudflare when several negative tests are performed from Jackett. It is obvious that a shared IP rather than a private IP produces this problem more quickly. Like seedbox host. You may not have this problem because you have no problem attempting to connect before being successful.

Curl on login :```

curl 'https://www2.yggtorrent.se/user/login'

-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0'

-H 'Accept: /'

-H 'Accept-Language: en-US,en;q=0.5' --compressed

-H 'X-Requested-With: XMLHttpRequest'

-H 'Content-Type: multipart/form-data; boundary=---------------------------2005504662289388114127868181'

-H 'Connection: keep-alive'

-H 'Referer: https://www2.yggtorrent.se/'

-H 'Cookie: __cfduid=; cf_clearance=; __cf_bm=; __ga=; ygg_=*'

-H 'TE: Trailers' --data-binary $'-----------------------------2005504662289388114127868181\r\nContent-Disposition: form-data; name="id"\r\n\r\n\r\n-----------------------------2005504662289388114127868181\r\nContent-Disposition: form-data; name="pass"\r\n\r\n\r\n-----------------------------2005504662289388114127868181\r\nContent-Disposition: form-data; name="ci_csrf_token"\r\n\r\n\r\n-----------------------------2005504662289388114127868181--\r\n'
```

[garfield69]

@cadatoiva @ngosang any input you can provide with this issue?

[MalaGaM]

Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the Firefox Add-ons Store. -> https://addons.mozilla.org/en-US/firefox/addon/privacy-pass/

The Privacy Pass extension provides users with the ability to create and sign cryptographically blind tokens for websites that support the Privacy Pass protocol. The extension generates passes containing cryptographically "blinded" tokens that are signed by the web server when a challenge page is solved. These tokens are "unblinded" and stored by the extension for future use; they are redeemed automatically when a future challenge page is seen. The "blinding" procedure means that signed and redeemed tokens are cryptographically unlinkable from the server perspective and, as such, are suitable for usage in conjunction with external privacy measures (such as VPNs).

The solution may be on the side of the cryptographically blind tokens?

https://support.cloudflare.com/hc/en-us/articles/115001992652-Using-Privacy-Pass-with-Cloudflare

[MalaGaM]

https://privacypass.github.io/api-redeem/

to solve this problem, we can most certainly implanted this mechanism in Jackett. What do you think ?

[MalaGaM]

https://github.com/scaredos/cfresearch
Privacy Pass Update
hCaptcha now supports Privacy Pass. Head to https://www.hcaptcha.com/privacy-pass and claim 5 tokens. Clicking the hCaptcha button on any website will now use one of the tokens and solve the captcha challenge.

Challenge (Captcha) (NEW)
The new CloudFlare captcha was introduced recently in part of an attempt to block Layer7 DoS attacks
The new method is POST to ?__cf_chl_captcha_tk__=GENERATED_TOKEN. It hands a cf_clearance cookie, allowing the user to bypass captcha, to the accepted device and, as usual, a __cfuid cookie stating the CloudFlare visitor id. The cf_clearance expires 1 day after the cookie was given and is valid for over 1k requests or until CloudFlare forces you to captcha again.
The form data for the POST request is 'r', which was discovered to be used for analytics by CloudFlare, 'cf_captcha_kind', which indicates which captcha the user solved, 'id', the visitor id, 'g-recaptcha-response', which is the hCaptcha response or reCaptcha response, and 'h-recaptcha-response', which is the hCaptcha response or reCaptcha response.

[ngosang]

@MalaGaM We are using a modified version of https://github.com/RyuzakiH/CloudflareSolverRe to solve Cloudflare challenges. That library supports 2 types of Cloudflare challenges.

1. CloudflareSolverRe. Is a simple webpage with a Javascript challenge that waits for 5 seconds. It doesn't have Captcha and it's working well.
2. CloudflareSolverRe.Captcha. It's the Captcha challenge, I think that code is not working since long time ago. We only have a couple of sites with this challenge and we are requesting the user to copy the cookie from the web browser.

It's the first time I hear from Privacy Pass, but It looks like the integration should be done in https://github.com/RyuzakiH/CloudflareSolverRe instead of Jackett. We don't have enough developers to accomplish the integration. If you can help in any way, we appreciate it.

[garfield69]

to support this we would need to use the CloudflareSolverRe.Captcha library, which currently supports 2CaptchaAPI and AntiCaptchaAPI and has https://github.com/RyuzakiH/CloudflareSolverRe#implement-a-captcha-provider

[MalaGaM]

I thank you for your feedback.

I think we have found the different sources of the problems, remains to set up solutions.

I summarize the situation:
Cloudflare services leave google reCAPTCHA to use hCaptcha of Intuition Machines, Inc.

What makes obsolete "Cloudflare Javascript & reCaptcha challenge" of https://github.com/RyuzakiH/CloudflareSolverRe intended for reCaptcha.

We have to replace the mechanism for this new service.
We can find "How to switch from reCAPTCHA to hCaptcha" here: https://docs.hcaptcha.com/switch#clientside

To read, it would seem simple:

1- Replace script
<script src = "https://www.google.com/recaptcha/api.js" async defer> </script>
becomes

<script src = "https://www.hcaptcha.com/1/api.js" async defer> </script>
2- Replace parameter name
g-recaptcha becomes h-captcha

I do not know if this is a solution, we should try. I do not know how recaptcha is implemented at present. I don't think it should be used for yggtorrent yet?

I think there are several hCaptcha security levels that site owners can configure. They seem to be more interesting to put on a high level. In fact, the more regularly the hCaptcha is asked of the user, the more money they make.

I noticed that in the case of yggtorrent in addition to cookies, it would seem that certain arguments are transmitted by GET. I don't know if this was the case with reCaptcha.

{"Query string": {
"name": "french",
"do": "search",
"attempt": "1",
"__cf_chl_captcha_tk __": "***"},
"Form data":
{"r": "***",
"cf_captcha_kind": "h",
"id": "***",
"g-recaptcha-response": "***",
"h-captcha-response": "***"}}

The Privacy Pass just keeps validation in captcha resolved for longer. We can stop rating for now.

We may be able to rename this post. Cloudflare reCaptcha to hCaptcha ?

[MalaGaM]

to support this we would need to use the CloudflareSolverRe.Captcha library, which currently supports 2CaptchaAPI and AntiCaptchaAPI and has https://github.com/RyuzakiH/CloudflareSolverRe#implement-a-captcha-provider

2Captcha API support hcaptcha

https://2captcha.com/blog/solve-hcaptcha

[CyberPoison]

There will be a way to fix this issue in the future ?

[pinedours]

Hope too ! This the best website in France...! We need this :/

[tchirou]

I concur !

[jobrien2001]

Would love anticaptcha support to be implemented

[MalaGaM]

Would love anticaptcha support to be implemented

Me too ...

[garfield69]

re: anti-captcha see #5504

[ngosang]

hCaptcha can be solved with https://github.com/openbullet/CloudflareSolverRe but you need a captcha resolver service subscription. Supported providers here => https://github.com/openbullet/CaptchaSharp

[thryyy]

hCaptcha can be solved with https://github.com/openbullet/CloudflareSolverRe but you need a captcha resolver service subscription. Supported providers here => https://github.com/openbullet/CaptchaSharp

Do you mean support for those Captcha resolver services will be integrated to Jackett ?

[ngosang]

The support for those services could be integrated in Jackett with low/medium effort but I don't have accounts/resources to do the development. This project has few developers. We have to wait.

[lborruto]

YGG has done it again.

[xfouloux]

The support for those services could be integrated in Jackett with low/medium effort but I don't have accounts/resources to do the development. This project has few developers. We have to wait.

i can share with you my Anti-Captcha.com key !