Linguist: Syntax Highlighting support for Svelte.

[umanghome]

Preliminary Steps

Please confirm you have...

• [x] reviewed How Linguist Works,
• [x] reviewed the Troubleshooting docs,
• [x] considered implementing an override,
• [x] verified an issue has not already been logged for your issue (linguist issues).

Problem Description

Syntax Highlighting isn't supported for Svelte.

URL of the affected repository:

Last modified on:

2018-11-11

Expected language:

Svelte

Detected language:

None

[pchaigno]

Please have a look at the guidelines to add support for a new language to Linguist. It Svelte meets the requirement on in-the-wild usage, we'd welcome a pull request!

In the meantime, we can't keep issues open for every languages we don't support---especially considering that we often don't have the expertise to add those languages ourselves---, so I'm going to close this.

[bdougherty]

When I search for the .svelte filename, I get 2,400 results right now. @pchaigno is that enough for a PR? Happy to take a crack at it if so.

[umanghome]

I worked on #4323 some time ago. Will update and resolve comments if there's a go-ahead for adding support.

[pchaigno]

When I search for the .svelte filename, I get 2,400 results right now. @pchaigno is that enough for a PR? Happy to take a crack at it if so.

@bdougherty That search query returns 0 .svelte and 14 svelte files. Unless you had something else in mind than adding one of these filenames, I don't think that's enough to mandate support in Linguist.

[bdougherty]

Oops, posted the wrong url. This is the one that has those results. Is there some other way you do the searches for this? That url is copied from the contributing doc, replacing the extension from that with svelte. My previous link returns over 600 repositories.

[PaulMaly]

Seems we still need to add .svelte support!

[lildude]

Seems we still need to add .svelte support!

Yup, but it's still not popular enough to meet the "each new file extension be in use in hundreds of repositories before supporting them in Linguist" requirement:

Total files found: 4372
Unique public user/repos: 171
Unique owners: 138

[PaulMaly]

@lildude That's because this extension just becomes with Svelte 3 Beta. So in the near future, the number of packages will increase exponentially. And it would be nice if GitHub will be ready for that and its users won't feel pain reading a source.

[lildude]

I agree, it would be nice, but then we'd need to do the same for _every_ language that purports "future exponential growth". This is why we have https://github.com/github/linguist/issues/4219

[umanghome]

@lildude Svelte is missing from the list in #4219. Can we get this added?

[PaulMaly]

@lildude Please check this results.

Is it not enough to be recognired as hundreds of repositories? I've check most of this pages and its all about SvelteJS and all this pages without any highlighting which is very uncomportable for the users.

[lildude]

Is it not enough to be recognired as hundreds of repositories?

@PaulMaly please see my tally of those results https://github.com/github/linguist/issues/4321#issuecomment-481166191

The total currently sits as "tens" and not "hundreds" (cos 171 is only 1 "hundred" and 70 "tens" 😉).

@lildude Svelte is missing from the list in #4219. Can we get this added?

Sure. @pchaigno manages that issue so I'll leave it to him to add when he's got the chance.

[PaulMaly]

@lildude Where we can check these numbers? Could you please provide a link to search results or something similar?

[lildude]

Unfortunately, I can't share the script I use as it abuses the API to the extent non-staff users would be blocked. https://github.com/Alhadis/Harvester is a browser-based alternative that does pretty much the same thing, but scrapes the search results as you see it and doesn't use the API. The results may vary slightly, but should be pretty similar.

[PaulMaly]

@lildude After quick research using the tool you provided, I've got these results:

Unique entries:  686

My script below:

const results = `<results string>`;

const entries = results.split('\n').map(entry => {
    const [user, repo, ...others] = entry.replace(/^(?:\/\/|[^\/]+)*\//, '').split('/');
    return `${user}/${repo}`;
});

const uniqueEntries = new Set(entries);

console.log('Unique entries: ', uniqueEntries.size);

What am I doing wrong?

[lildude]

I have no idea as I get:

Unique entries:  187

... which also matches what I get if I do it from the CLI as per https://github.com/Alhadis/Harvester#helpful-scripts:

$ grep < svelte.log -iEoe '^https?://raw\.githubusercontent\.com/([^/]+/){2}' \
| sort | uniq | sed -Ee 's,raw\.(github)usercontent,\1,g' | wc -l
     187
$ grep < svelte.log -iEoe '^https?://raw\.githubusercontent\.com/([^/]+/){2}' \
| sort | uniq | sed -Ee 's,raw\.(github)usercontent,\1,g' > unique-repos.log
$ grep < unique-repos.log -oE '^https://github\.com\/[^/]+' \
| sort | uniq > unique-users.log
$ wc -l unique-{repos,users}.log | grep -vE '\stotal$' | grep -oE '^\s*[0-9]+' |\
xargs printf '\
Unique repos: %s
Unique users: %s
'

Unique repos: 187
Unique users: 152

At a guess, I'd say your list of results varies from mine for some reason.

[PaulMaly]

@lildude Hm, strange but this proves that this method is insufficiently exact. Anyway, if we have 187 repos which is 1 "hundred" and 87 "tens" do we need just 13 additional repos to become 2 hundred and hundreds of repositories, right? I think I can create them for statistic purposes. 😄

[lildude]

For that comment, I'm putting a minimum of 300 unique user/repo for svelte :trollface:

Yes, this is an imprecise method as we're severely limited by GitHub's limited search and API rate limiting but it's all we've got, hence we keep things quite vague with the hundreds of repositories (implies unique user/repo's) requirement. This allows for imprecise measurements like this.

We hope people will be honest and work in good faith and let the popularity of the languages grow naturally towards high enough usage rather than game the system or fixate on precise numbers. People do game the system and some get really fixated on "hundreds" == 200 too, which is really not what we want.

That all said, as it stands nothing's going to magically happen to add support until someone raises a PR either 😉

As the old saying goes, all good things come to those that wait.

[umanghome]

That all said, as it stands nothing's going to magically happen to add support until someone raises a PR either

4323 was closed for being outdated. Made changes according to the comments on the PR too. Waiting for Svelte to be more popular in order to create another PR.

[PaulMaly]

@lildude I'm glad that you have a sense of humor because I don't know people who would fake popularity markup. 😉

But the main point we try to inform you that Svelte much more popular than these numbers can show us because custom extension .svelte was not mainstream until Svelte 3. Even now Svelte 3 still in beta.

So, before that, we used .html for Svelte components and GitHub really may have hundreds of repositories with these files. But even when Svelte 3 is come up and all that repos will upgrade for new version there's no reason to use .svelte with v3 if they already use .html, because .svelte is not mandatory but, in too time, has no highlighting support.

Therefore we've got a two-edged sword: 1) if we won't add .svelte support not many repos will switch to this extension; 2) if we haven't many repos with this extension we'll never add its support;

In all cases, GitHub users will most of all suffer, because they will be faced with many fake .html files which actually is Svelte components (basic html highlighting is not good for that) or with many unsupported .svelte with no highlighting at all.

I really hope you take the idea.

[lildude]

@lildude I'm glad that you have a sense of humor because I don't know people who would fake popularity markup. 😉

I do, for both 😁

So, before that, we used .html for Svelte components and GitHub really may have hundreds of repositories with these files.

If there's something particularly unique about Svelte in .html files that we can easily search for with GitHub's limited search (the more unique the keyword, the better) and can be used to differentiate from normal HTML (the only language .html is associated with at the mo) then that's certainly an option. ie add Svelte support for only the .html extension (as as you say this should already be really popular) with a heuristic to differentiate it from normal HTML.

All repos that use .svelte can then implement an override to get recognition of that extension until such time as the extension is popular enough in its own right.

This might actually be the best solution to cover current Svelte users and those eager to jump on the beta 🚌 .

[PaulMaly]

@lildude Thank you for trying to help us! Could you please describe this option more detailed?

If I got the idea correctly, we can add additional highlighting support for Svelte expressions to .html files and somehow (maybe via .gitattributes) force GitHub to recognize .svelte files as .html files with this syntax support? Am I right?

I think it would be a nice workaround before .svelte will become enough popular!

[lildude]

The idea is you add a new language as per https://github.com/github/linguist/blob/master/CONTRIBUTING.md#adding-a-language with the only extension being .html.

Add a heuristic to https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml the differentiates normal HTML from Svelte.

You'll need to add tests and samples too - all documented.

Then, when the next Linguist release is made after the PR has been merged, Svelte users will be able to add something like this to their .gitattributes file in their repos:

*.svelte linguist-language=svelte

All those .svelte files will now be identified as Svelte and will use the Svelte grammar for syntax highlighting.

If there isn't an easy way to differentiate between Svelte and normal HTML without accidentally misclassifying a load of other files, eg HTML with embedded JavaScript (I'm guessing here based on a quick look at the Svelte language), then the only option would be to wait for .svelte popularity to increase and then use the .svelte as the sole extension for the language. Once the PR has been merged, the same override can be done for .html to mark them as Svelte, ie:

*.html linguist-language=svelte

[pchaigno]

Using Harvester, I found 172 repositories by 141 users on 2236 files collected. Since there are actually 4k files (the search results limit us to ~1k files per sorting option, with duplicates), I think it's fine to assume .svelte has reached the hundreds of repositories threshold. @lildude What do you think?

[lildude]

Using Harvester, I found 172 repositories by 141 users on 2236 files collected. Since there are actually 4k files (the search results limit us to ~1k files per sorting option, with duplicates), I think it's fine to assume .svelte has reached the hundreds of repositories threshold. @lildude What do you think?

🤦‍♂️ of course. I completely missed the 4k files. If we assume the next 3000 results are similar to the first 1000, I think we can safely say .svelte meets the requirements.

@PaulMaly @umanghome feel free to open a PR to add support.

[umanghome]

PR: #4504.

[umanghome]

@lildude @pchaigno Can we assign reviewers to #4504?