Linguist: Allow specifying an ignore file for language statistics

[davidfowl]

Some repositories (like SignalR), have samples that include common javascript libraries like jQuery etc. and github ends up classifying the project as javascript instead of C# (in this particular case). Nothing is wrong with this at a high level since jQuery is javascript, but for project maintainers that want more control over statistics need a way to opt out of this behavior.

I see 2 options:

• Short term hack: Exclude commonly used js files. This will handle some scenarios but you'll have to exclude multiple versions of the library (unless you had wildcard support).
• Longer term solution: Allow a repository to have a .lignore or equivalent (I suck a naming) that uses glob syntax to exclude files to be processed for language statistics.

[DamianEdwards]

This is a must have in my view. It seems ridiculous that a repo can't configure the way stats are collected when so many contain 3rd party code.

[abevoelker]

Just to note, there is some functionality that already exists to exclude certain files and paths. Just mentioning it in case you weren't aware, since you didn't reference it.

--- README snip ---

Ignore vendored files

Checking other code into your git repo is a common practice. But this often inflates your project's language stats and may even cause your project to be labeled as another language. We are able to identify some of these files and directories and exclude them.

Linguist::FileBlob.new("vendor/plugins/foo.rb").vendored? # => true

See Linguist::BlobHelper#vendored? and lib/linguist/vendor.yml.

Generated file detection

Not all plain text files are true source files. Generated files like minified js and compiled CoffeeScript can be detected and excluded from language stats. As an extra bonus, these files are suppressed in Diffs.

Linguist::FileBlob.new("underscore.min.js").generated? # => true

See Linguist::BlobHelper#generated?.

[josh]

We've talked about this internally and we want to avoid any sort of per repo configuration. Sorry.

[davidfowl]

@josh Why? So this means our repository is broken unless we delete things from it. What about adding more libraries to the list of things to ignore?

[skoon]

Josh, could you make those internal reasons public? I'm failing to see a downside for per repo configuration in this case.

[Frogging101]

What the above asked, basically. Why would you want to actively avoid giving users the ability to configure such things?

[Droogans]

I would like it if doc directories were ignored. Min-ed jQuery typically dominates any project I that I use yardoc to publish my docs, and preview.github.io means I can host those docs online as well, which is a huge advantage.

[Chive]

• 1 for ignoring doc/docs directories

[AnonymousMeerkat]

I have 3rd party libraries that are written in a different language, but need to be included in the project itself.

True, language statistics aren't the biggest deal, but it would be very nice if I could just exclude the folder itself from being analyzed.

[danijar]

Since the language statistics come from Github rather than from Git, adding an options section to the web interface would also be a solution to think of. Just a textbox with one folder path per line. Those paths inside the repository would be excluded from the language statistics.

You can't cover all libraries by a global exclude list. What about developers of an excluded library? If they use Github, their whole project would get excluded. In my opinion some kind of repository wide configuration is the way to go.

[pgibler]

Any word on this? I have a project done mostly in Ruby. Despite this, Github has determined its a CSS/HTML project, because I wrote documentation and it's included with the project. I should be able to exclude my ./doc folder. Github should hook into this functionailty once it's in Linguist.

[cbelden]

I would love this. I track my js libraries so I can easily deploy to different environments. Right now, I have a python app that is 98% javascript.. a bit misleading.

[AlbertoMonteiro]

I made a workaround to force your repository's main language.
Example:

I want to force my repository to be marked as C# repository:

1. Create somefile.cs
2. Write this code

using System;

namespace ForceCSharpGitHub
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Console.WriteLine("This is a C# repository"); //Duplicate this line 500,000 times
        }
    }
}

Commit it, push, and you will see now that your repository is a C# repository!

[danijar]

I can't see any disadvantage of supporting an optional hint file. Where can we discuss this?

[pchaigno]

@danijar @AlbertoMonteiro Look at the current PRs, it's already being developed.

[blakev]

What's the status on this? I don't want my web projects being labeled as CSS.... -_-

[pchaigno]

@blakev It was released a few weeks ago: README#Overrides.

[danijar]

Awesome, thanks for posting the link!

[arcanis]

Will this file (.gitattributes) also be used for github stats? It's always a bit disappointing to see all stat charts destroyed when commiting vendors to the tree. On one repo I have about 935,827 ++ / 849,501 --. I'm not that productive :smile:

[pchaigno]

@arcanis It's a good idea I think. It's not something that can be implemented in Linguist however; you should suggest it to GitHub support. Might be a bit more difficult to implement for them though.

[ravicious]

To anybody still finding the issue through Google like me, Linguist now treats code under certain paths as vendored by default and doesn't include them in the stats.

[trevorhackman]

Do lines in Linguist vendored files still count towards line additions/deletions?

[lildude]

Do lines in Linguist vendored files still count towards line additions/deletions?

You can easily test this 😉

[Kroc]

I include binaries in a '/bin' directory for out-of-the-box build functionality. It seems too obvious that excluding a 'bin' directory has not already been discussed / rejected! Should I file an issue, or is this a won't-fix?

[Alhadis]

@Kroc It's a won't-fix. Linguist only scans textual formats and doesn't even acknowledge binary files. Moreover, this would severely affect repositories with executable scripts in /bin directories (which should count towards language statistics).

[Kroc]

My bin folder contains the documentation for those binaries! It's weighting the statistics entirely toward HTML, a format I don't use at all in the actual project: https://github.com/Kroc/pling (it's all assembly). Could you ignore sub-folders of /bin, or is this cutting off other use-cases?

[Alhadis]

@Kroc This is all explained in our readme. Please read the section on overrides for the solution to your problem.