Freshrss: Cannot add feed - Cloudflare protection

Created on 28 Sep 2017 · 11Comments · Source: FreshRSS/FreshRSS

Adding a Feed protected by cloudflare results in a 503 error

curl -v -A "SimplePie/1.7" https://daz3dfree.com/feed


* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
> GET /feed HTTP/1.1
> Host: daz3dfree.com
> User-Agent: SimplePie/1.7
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
< Date: Thu, 28 Sep 2017 01:40:46 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: close
< Set-Cookie: __cfduid=d3a9c091c5a7af5c2a05f2eca56b1c85e1506562846; expires=Fri, 28-Sep-18 01:40:46 GMT; path=/; domain=.daz3dfree.com; HttpOnly
< X-Frame-Options: SAMEORIGIN
< Refresh: 8;URL=/cdn-cgi/l/chk_jschl?pass=1506562850.398-PEHY94aqMK
< Cache-Control: no-cache
< Server: cloudflare-nginx
< CF-RAY: 3a52ff1dfe2b828b-ATL
<
<!DOCTYPE HTML>
<html lang="en-US">
<head>
  <meta charset="UTF-8" />
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
  <meta name="robots" content="noindex, nofollow" />
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>Just a moment...</title>
  <style type="text/css">
    html, body {width: 100%; height: 100%; margin: 0; padding: 0;}
    body {background-color: #ffffff; font-family: Helvetica, Arial, sans-serif; font-size: 100%;}
    h1 {font-size: 1.5em; color: #404040; text-align: center;}
    p {font-size: 1em; color: #404040; text-align: center; margin: 10px 0 0 0;}
    #spinner {margin: 0 auto 30px auto; display: block;}
    .attribution {margin-top: 20px;}
    @-webkit-keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    @keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    .bubbles { background-color: #404040; width:15px; height: 15px; margin:2px; border-radius:100%; -webkit-animation:bubbles 0.6s 0.07s infinite ease-in-out; animation:bubbles 0.6s 0.07s infinite ease-in-out; -webkit-animation-fill-mode:both; animation-fill-mode:both; display:inline-block; }
  </style>

    <script type="text/javascript">
  //<![CDATA[
  (function(){
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
    b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
    b(function(){
      var a = document.getElementById('cf-content');a.style.display = 'block';
      setTimeout(function(){
        var s,t,o,p,b,r,e,a,k,i,n,g,f, lxVTpdd={"btCDcyKZSc":+((!+[]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))};
        t = document.createElement('div');
        t.innerHTML="<a href='/'>x</a>";
        t = t.firstChild.href;r = t.match(/https?:\/\//)[0];
        t = t.substr(r.length); t = t.substr(0,t.length-1);
        a = document.getElementById('jschl-answer');
        f = document.getElementById('challenge-form');
        ;lxVTpdd.btCDcyKZSc*=!+[]+!![]+!![];lxVTpdd.btCDcyKZSc-=+((!+[]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));lxVTpdd.btCDcyKZSc-=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]));a.value = parseInt(lxVTpdd.btCDcyKZSc, 10) + t.length; '; 121'
        f.submit();
      }, 4000);
    }, false);
  })();
  //]]>
</script>


</head>
<body>
  <table width="100%" height="100%" cellpadding="20">
    <tr>
      <td align="center" valign="middle">
          <div class="cf-browser-verification cf-im-under-attack">
  <noscript><h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1></noscript>
  <div id="cf-content" style="display:none">

    <div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
    </div>
    <h1><span data-translate="checking_browser">Checking your browser before accessing</span> daz3dfree.com.</h1>

    <p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
    <p data-translate="allow_5_secs">Please allow up to 5 seconds&hellip;</p>
  </div>

  <form id="challenge-form" action="/cdn-cgi/l/chk_jschl" method="get">
    <input type="hidden" name="jschl_vc" value="9a3614bc707637fa448c62545811e7e2"/>
    <input type="hidden" name="pass" value="1506562850.398-PEHY94aqMK"/>
    <input type="hidden" id="jschl-answer" name="jschl_answer"/>
  </form>
</div>


          <div class="attribution">
            <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=iuam" target="_blank" style="font-size: 12px;">DDoS protection by Cloudflare</a>
            <br>
            Ray ID: 3a52ff1dfe2b828b
          </div>
      </td>

    </tr>
  </table>
</body>
</html>
* TLSv1.2 (OUT), TLS alert, Client hello (1):

Extension Feed problem help wanted

Source

ryancom16

Most helpful comment

I am a bit afraid this will become a race towards higher complexity, with Cloudflare making it more and more difficult when attacks start using libraries supporting Cloudflare.
I am not sure to be willing to go down this route: I have a bit the feeling that Cloudflare is "breaking the Web" with their approaches, so it should not be without consequences for Web site owners to activate those barriers.
P.S.: But I would for sure welcome a Cloudflare support as an extension in FreshRSS

Alkarex on 7 Jan 2018

👍3

All 11 comments

Indeed. If someone finds a cURL command that works for this feed, let me know.

Alkarex on 28 Sep 2017

If it can work with special HTTP headers, then it might be fixed with https://github.com/FreshRSS/FreshRSS/issues/1627

Alkarex on 28 Sep 2017

It looks like Cloudflare requires both cookies and JavaScript to work

Alkarex on 28 Sep 2017

I saw https://github.com/KyranRana/cloudflare-bypass which seems a bit complicated. But someone could investigate it in an add-on or something.

Frenzie on 28 Sep 2017

I am rewriting the PHP library so any suggestions are welcome :)

KyranRana on 31 Dec 2017

Alkarex on 7 Jan 2018

👍3

I have a bit the feeling that Cloudflare is "breaking the Web" with their approaches

That too, though I'm more worried about the fact that a third or so of the web is effectively "hosted" on it. Decentralized, hah! :-P

Frenzie on 8 Jan 2018

Successfully tested Cloudflare-Bypass on cloudflare protected feeds. Hoping to create an extension soon. What are my next steps? In other words, how best would it be implemented?

ryancom16 on 15 Jul 2018

👍1

Sorry for the delay @ryancom16
I first need to add an additional deeper API hook to allow overriding the cURL exec method.
The general approach will then be similar than https://github.com/FreshRSS/FreshRSS/blob/master/extensions/Tumblr-GDPR/
We then need an approach to find out when to use this hook or not. Probably an option at feed level. Maybe there are e.g. some HTTP headers in the responses which could give a hint to decide whether the hook should be used or not.

Alkarex on 23 Jul 2018

Example of cURL response for https://coinkite.com , which uses the CloudFlare UAM:

< HTTP/1.1 503 Service Temporarily Unavailable
< Date: Mon, 23 Jul 2018 21:45:11 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: close
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: __cfduid=d66e985fbc48942b64e56c6aadd92cd261532382311; expires=Tue, 23-Jul-19 21:45:11 GMT; path=/; domain=.coinkite.com; HttpOnly
< Cache-Control: no-cache
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< Server: cloudflare
< CF-RAY: 43f1552869a23d31-CPH

Probably the Status 503 together with Server: cloudflare should be enough for detecting the need.

Alkarex on 23 Jul 2018

Hey, any update for bypass cloundfare protection ?