Wappalyzer: What is an eligible technology to be included to Wappalyzer ?

Created on 20 Sep 2017 · 17Comments · Source: AliasIO/wappalyzer

In #1733 the issue of eligible technology is raised again.
I think it is time to add accurate, easy to understand and easy to test criteria for this.

Is the number of installations important ?
Is the age of the technology important ?
...

The goal of this would be to be able first to ask these information to be included in the pull request and then that anyone could know if its pull request could be merged or not. (we would write these criteria in the contributing part)

Why do Wappalyzer could want to reject some applications ?

Wappalyzer is intended to make discover technologies used in the real world and not to help someone to promote its web agency (this has been tried by some in the past)
Wappalyzer should keep a maintainable list and a list we could use regarding performance in the browser for example (What are the real limits today ?)

Do you have any ideas ?

Ps: keep in mind I am only a contributor and as far as I know only AliasIO is entitled to take decisions

Source

gadcam

Most helpful comment

The main limitations I see avoiding us to add an infinite number of technologies are :

the size of the extension because of the icons
Currently with all the good practices in place they are taking only 1.2Mo in the extension so it is not a problem before having 10 or 20 times more applications.
the time to process all the patterns
I do not have data about it but someone could do some tests so we can put some figures on it

Another problem could be the ability to check the consistency of the patterns to keep a good quality, however some checks are already automated : maybe we could push it further if the number of detected app would have to explode.

gadcam on 20 Sep 2017

👍3

All 17 comments

I just don't see why there be should some kind of criteria to include a technology.

Wappalyzer is described like this "Cross-platform utility that uncovers the technologies used on websites.". If someone requests the addition of a new technology, it's advantageous to the usefulness of Wappalyzer. I don't think that Wappalyzer is used by the average person. If you use this utility, it's because you are really interested in knowing the technologies behind a website even if they are not popular.

So, I don't think that we should focus on the "Marketing" side of submitting a new technology to Wappalyzer. People, and especially those who use this kind of utilities, have the ability to judge if a technology is worth discovering.

Wappalyzer has an informative role and it should focus on that. And as I said in a comment, as long as the technology is not scammy or intrusive, there is no reason for refusing a submission.

abdelbk on 20 Sep 2017

I do agree with the spirit of what you write.

However did you read that part ?

Why do Wappalyzer could want to reject some applications ?

Wappalyzer is intended to make discover technologies used in the real world and not to help someone to promote its web agency (this has been tried by some in the past)

Wappalyzer should keep a maintainable list and a list we could use regarding performance in the browser for example (What are the real limits today ?)

For the first point I see that you agree with me (you said there could be some "scammy" submission) and we need to be able to check that at least, and so set standard to make it easier.

For the second point: today Wappalyzer seems easy to use and stable but it has not always been the case in the past. We had performance issues in the past for example. It would be wonderful to detect any technology behind a webpage but if we have 100 000 detectable technologies with 3 regexes each it means we will execute 300 000 search at each page load. Can we afford that ? I don't think so. (We have currently 980 apps and something like 2200-2400 regexes)

Even if we could execute all these regexes and maintain the file where they are : do we really want to detect a technology which is used by 10 people all in the same company/group of friends/... ? I don't think there is an added value here.

That is why I think we need to set some standard, if possible backed up with data, because we need to stay realistic on one side and keep welcoming the contributions on the other.

gadcam on 20 Sep 2017

The main limitations I see avoiding us to add an infinite number of technologies are :

the size of the extension because of the icons
Currently with all the good practices in place they are taking only 1.2Mo in the extension so it is not a problem before having 10 or 20 times more applications.
the time to process all the patterns
I do not have data about it but someone could do some tests so we can put some figures on it

gadcam on 20 Sep 2017

👍3

Why add technology used on one site of the author?

AlexByte on 20 Sep 2017

😕1

I understand better the problematics now. Thank you @gadcam.

For the icons, how about defining an URL to the logo instead of uploading it ?

abdelbk on 21 Sep 2017

I don't really understand why this is problem @AlexByte.

julienszabados on 21 Sep 2017

@abdelbk
I really do not think icons are our first problem but if you want to go in that path you have to talk with @AliasIO, moreover I find it better to deploy something that is working on its own.
@AliasIO, @q--, @DaAwesomeP, @jvoisin, @craiglondon
Do you think a debate is needed on that point and do you want to take part in it ?

gadcam on 25 Sep 2017

"popular enough, with a significant (>1k stars on github, stuff like that) user base" should be enough in my opinion.

Disclaimer: I'm using Wappalyzer internally at work, and for our use-case, we only care about speed and detecting "main" web applications like wordpress, drupal, joomla, … and their versions.

jvoisin on 25 Sep 2017

The reason for rejecting applications that are not widely used is to keep Wappalyzer's footprint small. Every regular expression is evaluated on every page load, which has an impact on CPU usage and browser performance. I want to keep the threshold low but at the same time each addition should be of some value to most users.

We can go with 1K+ stars on GitHub, or a showcase of 10+ websites using the application. I'm also happy to just merge PRs that seem reasonable and remove them if they don't result in any detections for several months (as per #1432). That way we don't need to be too hostile to new contributors.

AliasIO on 26 Sep 2017

Please do write it somewhere on the wiki so we can be consistent :)

jvoisin on 26 Sep 2017

I agree with @jvoisin it would be a good thing to have both the minimum requirement for an addition and the threshold to remove an app written in a safe place.
Could we have a specific webpage listing the technologies to be removed ?
As it we would enable us to know at a glance what technologies should be removed and on which we could work to find better patterns.

gadcam on 26 Sep 2017

I'm also happy to just merge PRs that seem reasonable and remove them if they don't result in any detections for several months (as per #1432).

I'm against this because CMS Bitrix, which is used on 95 000 websites, shows that it is being used on 0 websites.

AlexByte on 26 Sep 2017

@AlexByte why would we want to keep an app we are not able to detect ? As I said in my previous comment this list could help us to find inefficient patterns and so to improve them.

gadcam on 27 Sep 2017

i. am. able. to. detect.

AlexByte on 27 Sep 2017

@AliasIO This issue is open since two weeks and nothing moved since 9 days so I think we can say the problem is solved. So do you think you could write the decision down in the wiki and then I could close this issue ?

gadcam on 5 Oct 2017

I've added to the new developer documentation.

AliasIO on 7 Oct 2017

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.