Wappalyzer: Removal of never detected applications
At some point I'd like to remove applications that have had very few detections over a certain period of time. Quite a few have never been identified on any website:
AliasIO
All 58 comments
re: SailsJS -- is the header detection portion case sensitive?
x-powered-by:Sails <sailsjs.org>
The pattern needs to be updated "X-Powered-By": "^Sails$"
But I wasn't sure if the X-Powered-By portion was case sensitive, please advise.
craiglondon
on 19 Mar 2017
This is incorrect statistics.
1C-Bitrix is popular CMS. I've detected enough sites running on it over the past month.
AlexByte
on 20 Mar 2017
@AlexByte Do you have sharing of detections disabled or enabled?
craiglondon
on 20 Mar 2017
@AlexByte I checked https://www.1c-bitrix.ru/ and it was detected by Wappalyzer. I guess I'll wait a day or two to see if the metrics logged the detection.
craiglondon
on 20 Mar 2017
@craiglondon Sharing of detections is enabled.
AlexByte
on 21 Mar 2017
Strange. Can you provide a few examples of sites that use this CMS? I detected it in Chrome on the CMS homepage. But I don't know any other sites that use it.
craiglondon
on 21 Mar 2017
@alexbyte ^^^
craiglondon
on 21 Mar 2017
@craiglondon https://w3techs.com/technologies/details/cm-bitrix/all/all
https://trends.builtwith.com/websitelist/1C-Bitrix
AlexByte
on 22 Mar 2017
There is a threshold for data to be displayed on the website. I'm happy to leave 1C-Bitrix in but it looks like not many Wappalyzer users encounter it.
AliasIO
on 8 Apr 2017
Why not add the option for whose who want to track all applications?
AlexByte
on 8 Apr 2017
The motivation for removing these applications is to remove bloat.
AliasIO
on 8 Apr 2017
Whats the problem with apps.json? I guess its size is not very large, no problems with size on disk and opening.
AlexByte
on 8 Apr 2017
It would mainly be to reduce processing, i.e. the number of regular expressions evaluated on every page.
AliasIO
on 30 Apr 2017
Why not add the option for whose who want to track all applications?
AlexByte
on 30 Apr 2017
What is the minimum number of an application users to show their URLs on Wappalyzer website?
mehrshadz
on 29 May 2017
The applications needs to be detected on at least 100 websites to be displayed.
AliasIO
on 1 Jun 2017
@AliasIO Is there a threshold of the number of users?
AlexByte
on 26 Sep 2017
No, there isn't.
AliasIO
on 3 Nov 2017
Why did you close this issue @AliasIO ? Shall I open a new one to discuss technologies that might be good removal candidates?
jvoisin
on 6 Nov 2017
Ah, didn't mean to close this one. Something about the numbers seems off, not sure why Lodash has zero detections for instance.
AliasIO
on 7 Nov 2017
I'd like to propose to remove asymptix, that has not seen a single significant detection, and has 3 stars on github, and is here since fb7be84cfbb6b36ae3d68a147147571b9135d273, 18 months ago.
jvoisin
on 7 Nov 2017
Go for it.
AliasIO
on 7 Nov 2017
I propose to have two files. One with the most common applications and one with others. User could select if want to detect un-common applications too.
sagudev
on 18 Feb 2018
@AliasIO Why can it be that I detected the technology on several sites, and the site says that the technology is used on 0 websites. Sharing is enabled.
AlexByte
on 26 May 2018
@AlexByte I don't record every data point. I have rate limiting in place as well as some other methods to prevent abuse and deal with the high volume of traffic. There is something wrong with the numbers though, PHP and WooCommerce should certainly not be zero.
AliasIO
on 29 May 2018
@AlexByte I worked out what happened, the totals on that page are now up-to-date again.
AliasIO
on 1 Jun 2018
@AliasIO What was the problem?
AlexByte
on 1 Jun 2018
MySQL was struggling with large tables and timing out while trying to calculate the totals. I rebuilt the entire thing in MariaDB and it's blazing fast again.
AliasIO
on 1 Jun 2018
@AliasIO Some technology has 0 websites in the list. If I visit a site on which this technology is, how quickly should this number increase?
I visited sites with different technologies that are on 0 websites, but the number does not change.
AlexByte
on 1 Jun 2018
The summaries get updated daily but as I mentioned I'm not collecting every single data point and they need to be corroborated by multiple sources. Lodash is zero because I renamed it from Lo-dash but that should populate probably within a few weeks.
AliasIO
on 1 Jun 2018
What sources? How many sources do you need to confirm one site for technology?
AlexByte
on 1 Jun 2018
I don't want to expose all the details but it's not enough for just one user to send information and have it show up on the website. It's to prevent people from poisoning the data with fake requests.
AliasIO
on 1 Jun 2018
Why did you write that there is no threshold of the number of users? https://github.com/AliasIO/Wappalyzer/issues/1432#issuecomment-341604164
AlexByte
on 1 Jun 2018
Ha, good point. Wappalyzer has a completely different architecture now so things have changed quite a bit.
AliasIO
on 1 Jun 2018
Not surprisingly, many technologies have 0 websites, since 85% of detection sendings end in error. It is necessary not to clear the cache if the request was completed with an error and increase the interval between requests (increase > 50), so that there were fewer errors "Too Many Requests" of AWS Lmabda. What is current limit of requests per second in AWS Lambda?
AlexByte
on 7 Jun 2018
I'm processing 10 requests per second but there is also a per-user limit. I'm not sure why you're seeing that many requests, it should only ping every 50 websites or so. Did you change the code on your end? Increasing the rate and interval both affect my AWS costs, I have to balance it.
AliasIO
on 8 Jun 2018
I'm not sure why you're seeing that many requests
Because these are requests for a week.
Increasing the rate and interval both affect my AWS costs
Not increase rate. I suggest: fewer request, but with bigger request size (Lambda supports up to 6 MB). Why will prices rise if there are fewer requests?
It is necessary not to clear the cache if the request was completed with an error
What can you say about this?
AlexByte
on 8 Jun 2018
Increasing the size of the request increases the duration of the function execution and database throughput. I'll experiment with this some more though.
It is necessary not to clear the cache if the request was completed with an error
The cache will just keep growing and a request will be issued on every page load after 50. I think that will result in more issues.
AliasIO
on 8 Jun 2018
The cache will just keep growing and a request will be issued on every page load after 50. I think that will result in more issues.
You can store the limit of the number of sites in a variable and increase it by 50 if the sending was executed with an error. The cache can be stored in the repository so that when the browser is closed, the detections is not lost.
AlexByte
on 8 Jun 2018
We can try that.
AliasIO
on 8 Jun 2018
In general, at the address https://api.wappalyzer.com/ping/v1/, no more than 10 sites are sent, because adCache contains about 40 web pages. You can count the number separately for hostnameCache and adCache. adCache can be not sent at all, why is it needed?
Info about sending adCache:
AlexByte
on 8 Jun 2018
Good call, I've changed it in the latest version.
AliasIO
on 11 Jun 2018
I'm processing 10 requests per second but there is also a per-user limit.
The cache will just keep growing and a request will be issued on every page load after 50. I think that will result in more issues.
You replaced 50 by 1. In general, I do not understand why you did this, Now when parallel surfing two sites after every page loading (of opposite site) a detections of that two site is sent. You increased the number of requests in several times and worsened the problem many times.
https://github.com/AliasIO/Wappalyzer/blob/2da8ec82d33791d6837d95941dc29d1e04a90947/src/wappalyzer.js#L202-L214
AlexByte
on 13 Jun 2018
Oh, that was unintentional. I just pushed a stealth update to fix it.
https://github.com/AliasIO/Wappalyzer/commit/e6a9fcae2f04673b4dd2a52d8dc22ae09c6a70a5#diff-d7f544c6575fa5815df4b466a9e04f40
AliasIO
on 15 Jun 2018
@AliasIO If before there was no threshold of users, then why did 1c-bitrix technology have 0 sites? Is it because of a problem with sending data or before there was some threshold and sending data by one user to the server was not enough?
AlexByte
on 19 Jun 2018
Throttling was always there, just not on a per-user basis. I made a few optimisations and seem to be capturing a lot more data now, you'll probably notice fewer 429's. We can add 1C-Bitrix back in and monitor it.
AliasIO
on 19 Jun 2018
What does 429 mean?
AlexByte
on 19 Jun 2018
jvoisin
on 19 Jun 2018
@AliasIO In order to add 100 sites to the cache, you need not to close a browser for 4 days.
Previously, site detections was sent after accumulating about 10 sites in the cache.
I believe that you need to save a site cache to the storage.
AlexByte
on 19 Jun 2018
I used to have it so that it does a ping when you close the browser. I'll look into both options.
AliasIO
on 20 Jun 2018
You can save site detections to the storage when you close the browser.
AlexByte
on 21 Jun 2018
@AliasIO hi, added 1C-Bitrix back, please merge PR
khazamov
on 25 Jun 2018
You can save site detections to the storage when you close the browser.
Added that in here.
AliasIO
on 1 Jul 2018
More on topic, the numbers appear to be more accurate now. 1C-Bitrix currently has 50+ detections (thanks @khazamov).
AliasIO
on 1 Jul 2018
How are detections added if extensions are not updated to return 1c-bitrix?
AlexByte
on 5 Jul 2018
@alexbyte can you update extensions, since you already took the credit?
On Thu, Jul 5, 2018 at 1:43 AM AlexByte notifications@github.com wrote:
How are detections added if extensions are not updated to return 1c-bitrix?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AliasIO/Wappalyzer/issues/1432#issuecomment-402568374,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACcO3j5YImMGvAiXzrNS-YVBZHQuaRWaks5uDUUGgaJpZM4MaxGB
.
--
Thanks,
Omar Khazamov
khazamov
on 5 Jul 2018
How are detections added if extensions are not updated to return 1c-bitrix?
Looks like some people are still running an old version with 1C-Bitrix.
AliasIO
on 5 Jul 2018
@AliasIO Why then these sites were not added earlier?
AlexByte
on 11 Jul 2018
Related issues
KeviinCosmos
·
7Comments
faheel
·
3Comments
cfjedimaster
·
5Comments
smokeyflavor
·
8Comments
patheticcockroach
·
6Comments