Wappalyzer: Wappalyzer does not follow redirects and returns NO_RESPONSE

[faheel]

Platform
Which platform does the bug apply to?

• NPM

Describe the bug
Some websites like zomato.com redirect to other sub-domains or URLs, which Wappalyzer doesn't seem to follow even with redirect: true.

To Reproduce
Steps to reproduce the behavior:

1. Use the code:

const options = {
  debug: true,
  delay: 500,
  maxDepth: 3,
  maxUrls: 10,
  maxWait: 10000,
  recursive: true,
  userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
};

const wappalyzer = new Wappalyzer('https://zomato.com', options);

  wappalyzer.analyze()
  .then((json) => {
    process.stdout.write(`${JSON.stringify(json, null, 2)}\n`);
    process.exit(0);
  })
  .catch((error) => {
    process.stderr.write(`${error}\n`);
    process.exit(1);
  });

1. Wappalyzer will return the following:

[wappalyzer debug] [driver] [timer] fetch; url: https://zomato.com/; depth: 1; delay: 500ms; lapsed: 0s / 0s
[wappalyzer debug] [driver] [timer] visit start; url: https://zomato.com/; lapsed: 0.51s / 0.51s
[wappalyzer debug] [driver] [timer] visit end; url: https://zomato.com/; lapsed: 10.05s / 10.55s
[wappalyzer error] [driver] No response from server; url: https://zomato.com/

{
  "urls": {
    "https://zomato.com/": {
      "status": 0,
      "error": {
        "type": "NO_RESPONSE",
        "message": "No response from server"
      }
    }
  },
  "applications": [],
  "meta": {}
}

Expected behavior
The same web technologies as detected by the website should be detected:

Additional context
Here are the redirect response headers captured by HTTPie (http --follow --all https://zomato.com):

1.

HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 150
Content-Type: text/html
Date: Mon, 10 Feb 2020 12:21:28 GMT
Location: https://www.zomato.com:443/
Server: awselb/2.0

2.

HTTP/1.1 302 Moved Temporarily
Cache-Control: max-age=0, no-cache, no-store, no-transform
Connection: keep-alive
Content-Length: 0
Content-Security-Policy: frame-ancestors https://*.nearbuystag.in https://*.nearbuy.com 'self' https://microapps.google.com/ https://pwa.zomato.com; default-src *; font-src * data:; img-src * data:; media-src * blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' browser.sentry-cdn.com *.jwpcdn.com *.cloudflare.com *.twitter.com *.recruiterbox.com *.zdev.net *.zdev.net:8080 *.zomato.com *.tinymce.com *.gstatic.com *.googleapis.com *.google.com *.google.co.in *.facebook.com sdk.accountkit.com *.doubleclick.net *.googlesyndication.com *.nr-data.net *.newrelic.com *.google-analytics.com *.akamaihd.net *.zmtcdn.com *.googletagmanager.com *.facebook.net *.googleadservices.com *.cdninstagram.com *.googlesyndication.com *.inspectlet.com *.spreedly.com *.instagram.com *.twimg.com *.mouseflow.com *.usersnap.com d3mvnvhjmkxpjz.cloudfront.net *.serving-sys.com *.sushissl.com *.pubnub.com tsgw.tataelxsi.co.in *.branch.io app.link cdn.poll-maker.com *.ampproject.org *.smartlook.com *.hotjar.com dashboard.hypertrack.io zba.se *.googletagmanager.com *.eff.org cdn.plot.ly *.zedo.com *.bing.com *.criteo.net *.criteo.com mddigital.in static.zdassets.com ekr.zdassets.com zomato.zendesk.com *.zopim.com *.zopim.io *.cloudfront.net cdn.taboola.com trc.taboola.com; style-src * 'unsafe-inline'; worker-src 'self' https: blob:;
Content-Type: text/html; charset=UTF-8
Date: Mon, 10 Feb 2020 12:21:28 GMT
Expires: Mon, 10 Feb 2020 12:21:28 GMT
Location: https://www.zomato.com/ncr
Pragma: no-cache
Server: Apache
Set-Cookie: PHPSESSID=96c7a527aebad995c7ccc4b9277524b50d25e220; path=/; domain=.zomato.com; secure; HttpOnly
Set-Cookie: csrf=5a6e13447311f62ca6d781c93db5e85e; expires=Mon, 10-Feb-2020 12:45:28 GMT; Max-Age=1440; path=/; domain=.zomato.com
Set-Cookie: fbcity=1; expires=Sat, 08-Aug-2020 12:21:28 GMT; Max-Age=15552000; path=/; domain=.zomato.com; secure
Set-Cookie: hy-en=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=zomato.com
Set-Cookie: fre=0; expires=Sat, 22-Mar-2070 00:43:06 GMT; Max-Age=1581337298; path=/; domain=zomato.com
Set-Cookie: ttaz=0; expires=Mon, 10-Feb-2020 12:21:27 GMT; Max-Age=0; path=/; domain=zomato.com
Set-Cookie: rd=1380000; expires=Sat, 22-Mar-2070 00:43:06 GMT; Max-Age=1581337298; path=/; domain=zomato.com
Set-Cookie: lme=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=.zomato.com
Set-Cookie: zl=en; expires=Sat, 08-Aug-2020 12:21:28 GMT; Max-Age=15552000; path=/; domain=.zomato.com; secure
Set-Cookie: fbtrack=01e4db5511d3c2d3c1ae808c2fbfcefd; expires=Thu, 04-Feb-2021 12:21:28 GMT; Max-Age=31104000; path=/; domain=.zomato.com; secure
Set-Cookie: ak_bmsc=EC14C92ABB5ED6756E68D89D31BB86F417CB3F27B6150000C84A415E0A689C6B~plbdch8dUPcjEeyfx/iFZvaMagmNo68Ogsj9Rv3UNByj1dt/oAYrvxarMgFPYSuJ2OEniOeaEtaiS+huGUJoaqkotyx4eVVNd1Y2OjTgmOy8AOaXpztE3yvBlhMCuXsp3jdzm4kK8UR04kZtclyFpqbrTYkUUq/Sjcfi/IEf6nLmOJM++bTFWWJFfPcuU72Q2goRzsLeUnh/nNVhVwSmWSn2m0T/hZuAxaftGYc0m92oQ=; expires=Mon, 10 Feb 2020 14:21:28 GMT; max-age=7200; path=/; domain=.zomato.com; HttpOnly
Set-Cookie: akaas_WebCanary=1581337888~rv=36~id=1f1f3d58b532a597e5061d1d7f9f88c6; path=/; Expires=Mon, 10 Feb 2020 12:31:28 GMT; HttpOnly; Secure; SameSite=None
Set-Cookie: bm_sz=4ABF2FE62E50C3F7B33372C9968D2249~YAAQJz/LFwod1SdwAQAAnCAMLwb0dBgbS3F+gY/RASuczFBy2ymIi0w3Mfr8znSiL2lT4FRXu6SQTxTbPhqA1PGc7Btofy51nOQhcRhZ9qM9cJyAJyqSL70KBlmUgLf2IX7GtvZh245XYpMYVA7qoXzlxBVytToyutMxs1hE420AJzFgOJjCtUzAwlfK5Bgc; Domain=.zomato.com; Path=/; Expires=Mon, 10 Feb 2020 16:21:28 GMT; Max-Age=14400; HttpOnly
Set-Cookie: _abck=BE41FF38464EEBB6F905C0B0DBB6B1DE~-1~YAAQJz/LFwsd1SdwAQAAnCAMLwNdgmmGkYEY6J3RqddUZvbuICgXhJSulNGaTNp/xjXia2WohwmeH5k6E4nEjU/3Ua0HFEjqseR7J0KyTpdwdhKWK5SQKPWsAXvLEc4P5uOwNmUE7YitCtvYBrlamgOV2nsR3Z2IiPb1J17MBIQKTyca2qTKUl7XZXxXs65/1UFJZgOMArWV9gQKKTcG6+abPiAOSXqQJ4UOBqwO3teo1/3mTb1EXqlHELcmihS+Yh24RxgxNjRhU2qK12YCqb2VSmsGuQ14tuR/55hwNv7PXRotLqR6RxyK~-1~-1~-1; Domain=.zomato.com; Path=/; Expires=Tue, 09 Feb 2021 12:21:28 GMT; Max-Age=31536000; Secure
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block; report=https://www.zomato.com/cspreport.php

3.

HTTP/1.1 200 OK
Cache-Control: max-age=0, no-cache, no-store, no-transform
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 22359
Content-Security-Policy: frame-ancestors https://*.nearbuystag.in https://*.nearbuy.com 'self' https://microapps.google.com/ https://pwa.zomato.com; default-src *; font-src * data:; img-src * data:; media-src * blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' browser.sentry-cdn.com *.jwpcdn.com *.cloudflare.com *.twitter.com *.recruiterbox.com *.zdev.net *.zdev.net:8080 *.zomato.com *.tinymce.com *.gstatic.com *.googleapis.com *.google.com *.google.co.in *.facebook.com sdk.accountkit.com *.doubleclick.net *.googlesyndication.com *.nr-data.net *.newrelic.com *.google-analytics.com *.akamaihd.net *.zmtcdn.com *.googletagmanager.com *.facebook.net *.googleadservices.com *.cdninstagram.com *.googlesyndication.com *.inspectlet.com *.spreedly.com *.instagram.com *.twimg.com *.mouseflow.com *.usersnap.com d3mvnvhjmkxpjz.cloudfront.net *.serving-sys.com *.sushissl.com *.pubnub.com tsgw.tataelxsi.co.in *.branch.io app.link cdn.poll-maker.com *.ampproject.org *.smartlook.com *.hotjar.com dashboard.hypertrack.io zba.se *.googletagmanager.com *.eff.org cdn.plot.ly *.zedo.com *.bing.com *.criteo.net *.criteo.com mddigital.in static.zdassets.com ekr.zdassets.com zomato.zendesk.com *.zopim.com *.zopim.io *.cloudfront.net cdn.taboola.com trc.taboola.com; style-src * 'unsafe-inline'; worker-src 'self' https: blob:;
Content-Type: text/html; charset=UTF-8
Date: Mon, 10 Feb 2020 12:21:29 GMT
Expires: Mon, 10 Feb 2020 12:21:29 GMT
Pragma: no-cache
Server: Apache
Set-Cookie: hy-en=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=zomato.com
Set-Cookie: fre=0; expires=Sat, 22-Mar-2070 00:43:08 GMT; Max-Age=1581337299; path=/; domain=zomato.com
Set-Cookie: ttaz=0; expires=Mon, 10-Feb-2020 12:21:28 GMT; Max-Age=0; path=/; domain=zomato.com
Set-Cookie: rd=1380000; expires=Sat, 22-Mar-2070 00:43:08 GMT; Max-Age=1581337299; path=/; domain=zomato.com
Set-Cookie: lme=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=.zomato.com
Set-Cookie: akaas_WebCanary=1581337889~rv=36~id=6e93e7a9b3471472c6af173c7c009aeb; path=/; Expires=Mon, 10 Feb 2020 12:31:29 GMT; HttpOnly; Secure; SameSite=None
Set-Cookie: bm_sv=453AE18706481F6653D2A48ABAF0DFF0~ZKNtLbuVWGwIOMvniin8qJEVG3tnFyc3sv/oFndCZah0ohwPEna5csgRU8u/e0tLCk5kbhC6jN5C9ifAmgMQ8CKxDLRBJQntBjxBz59tZWcXYCfRGgN6OZXWvH10sdQJCWiY1HULOT7UJ7BBGhsYXWcyOJD5hllb/bHIzS++Aqg=; Domain=.zomato.com; Path=/; Max-Age=7199; HttpOnly
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding, User-Agent
X-Akamai-Transformed: 9 22270 0 pmb=mTOE,1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block; report=https://www.zomato.com/cspreport.php

[AliasIO]

I can't reproduce this, I'm getting a 200 and results on that URL. It looks like it may have been hitting the 10 second timeout?

[faheel]

Yes, you're right. This was a timeout issue and had nothing to do with redirection. Using a longer timeout fixes the issue.

[moerazem]

The issue actually exists.
Running wappalyzer http://www.tesla.com/ --debug --delay=500 does not follow the redirect and gives NO_RESPONSE:

{"urls":{"http://www.tesla.com/":{"status":0,"error":{"type":"NO_RESPONSE","message":"No response from server"}}},"applications":[],"meta":{}}

While if you run wappalyzer https://www.tesla.com/ --debug --delay=500 works.