Gitea: Languages Statusbar

[TimerWolf]

Are there any plans for gitea to get a status-bar that shows how much of different code its used in the git?

And when you click on it you see the different languages that are used and how many % of the code that is used of that language in the git?

Its a really neat feature that i really miss!

[tonivj5]

Here is the PR to gogs (it was not merged) implemeting that feature https://github.com/gogits/gogs/pull/2135. It could be reused to add it in gitea 😉

[lunny]

@xxxtonixxx maybe someone could send it to Gitea.

[tonivj5]

If @generaltso want, he could do it! If not, I think I could attempt it 😅

[dayvonjersen]

https://github.com/gogits/gogs/pull/2135 is certainly out of date by now as the gogs codebase has probably changed and the API for linguist has definitely changed.

Of course anyone is more than welcome to use my library but implementing this feature isn't going to be a copy/paste job.

That said, I copied and pasted the CSS I whipped up for those screenshots into a codepen: https://codepen.io/anon/pen/PjMdBy

-tso

[lafriks]

And I don't think it can be accepted in that form, stats should be generated and cached only once when repository default branch changes, not on every page load

[dayvonjersen]

@lafriks yes, exactly. probably best to have like a post-receive hook that runs in the background and stores the result in the db.
and have a setting to disable it entirely for those concerned about server resource usage
it would also be cool if the classifier could then be retrained on real-world code samples but I'm probably jumping the gun here >_>

-tso

[54]

I put up a small ($5) bounty on this one. Miss this feature!

EDIT: Here is the current pledge amount. If anyone else feels like contributing, feel free!

[dayvonjersen]

Hm, now my interest is piqued ;)

I could take another stab at it maybe tomorrow evening (I'm in EST). But be forewarned, my preliminary attempt will probably be an awful hack job. I will need to rely on the rest of the community's advice to do it right.

-tso

[54]

Sounds great! In addition to caching, the final version should probably also be limited by file size. Maybe an adjustable setting?

[dayvonjersen]

@54 Hm so you mean don't try to classify individual files that are larger than, e.g. 1MB? Most of the time what linguist does is it goes by file extension but hm, yes I see what you mean just thinking aloud... Good idea :)

[54]

@generaltso Yeah, I just figure it'd kinda suck if someone uploaded some monstrous set of files that the server had to analyze. But, I haven't looked at your linguist library. Does it ever actually do some content analysis or is it pretty much entirely based on extension?

If it is purely based on the extension, then I don't think it's necessary to add that particular limitation.

[dayvonjersen]

well it can do either.

in the reference implementation, after being filtered by linguist.ShouldIgnoreFilename() the file extension is passed to linguist.LanguageHints().

if there is more than one possible language for an extension (e.g. .php could be either PHP or Facebook's "Hack" language) then it first checks if the file is a binary blob with linguist.ShouldIgnoreContents() and then uses a bayesian classifier which has been trained on the same dataset as github/linguist to analyse the text (using a tokenizer which could use some improvement) and determine the language (the function is called linguist.Analyse())

a pretty straightforward process imo but I'm a tiny bit biased since I wrote it :p

it might be more convenient to encapsulate all the nuance into a single package-level function instead of requiring all of those steps for the typical use-case, I welcome any input in improving the library for users as well if you or anyone else have any suggestions :)

-tso

[54]

Ah, thanks for the clarification! By the way, #2108 appears to include its own submenu—minus the linguist functionality, though.

Might be worth waiting for that to get merged?

[dayvonjersen]

@54 that would seem to make the most sense in the grand scheme of things, but I'm just gonna get to hacking away at something on a separate branch based off master since I just reinstalled MySQL and setup gitea and I'm ready to go, just to get the ball moving

I'll update here with screenshots and code and stuff in a couple hours makes coffee :coffee: :grin:

[54]

Yeah, of course. I just meant since there is some overlap, probably best not to make something super polished just yet 😁

Good luck, @generaltso!

[dayvonjersen]

OK I got my feet wet in the code base and basically just implemented the design I had previously come up with some dummy placeholder results for now.

I have lots of questions but I think I made some decent progress for 1.5 hours of work

you can view my commits:
https://github.com/generaltso/go-sdk/commit/41cabf2d856c55b2202c02394777c905bd113c1f
https://github.com/generaltso/gitea/commits/feature/language-statistics

these should be probably squashed with a rebase if/when it's ready for a PR

NOTE: I had to cp -r my-local-fork-of-gitea-sdk gitea/vendor/ manually. I don't fully understand how to use vendoring with go build in this project atm

First thing I did was add a table to the DB (even though I'm not using it just yet)
https://github.com/generaltso/gitea/blob/00427a9095590b78ce85cca8c41d174b5bdf5db4/language_statistics.sql

ALTERNATIVE TO THIS would be a single ~row~ __field__ in the repository table with JSON data containing all the language stats.

Here's the Commits/Branches/Releases/Contributors bar

to complete this I need to know how to

• get the number of branches for a repo
• get the number of unique GIT_AUTHOR's for a repo

here's the language bar

It's already using linguist.LanguageColor() to use the "proper" github colors...

to complete this I need:

• to get the complete file list of the repo
• (maybe not right now but eventually) get the corresponding file sizes
• read data (file contents) from the repo
• store those results in the db
• read those results back from the db in models/repo.go or models/language_statistics.go

my biggest questions I need guidance with now are:

• how to work with the db?
• how to add hooks for e.g. post-receive? (so that language statistics can be populated and kept up-to-date in the background)

ADDITIONALLY and this is me getting way ahead of myself here but for the future:

• user data (aka user code stored with a gitea instance) should be run through the classifier again just like github does with all the code on this (incredible treasure trove and modern day library of alexandria of a) website <3 <3 <3 :octocat: :heart: :sparkling_heart:
• language statistics should be visible on other pages for example the repository list and search results

again my commits are viewable here:
https://github.com/generaltso/go-sdk/commit/41cabf2d856c55b2202c02394777c905bd113c1f
https://github.com/generaltso/gitea/commits/feature/language-statistics

I appreciate any further guidance but I think I'm going to make dinner now and relax a bit.

<3

-tso

EDIT: also note there is a transition for the commits/etc bar -> language %'s. It looks like this codepen of a 3D cube flip effect because I took it directly from that and changed the timing function (see also https://github.com/generaltso/gitea/blob/fdcd6b8b9ebfb8098ecfa81f92352c20ed58f270/templates/repo/home.tmpl#L77)

[lunny]

@genedna all db related operation on models sub module.

[genedna]

@lunny @genedna -> @generaltso ?

[lunny]

@genedna sorry for wrong mention. :) @generaltso

[dayvonjersen]

@lunny thanks I'll have to read over the code in models more in-depth then; will probably work on it some more later today :)

any other comments/feedback on what I did so far? or does it look ok

[kolaente]

@generaltso liking it so far! Any progress update?

[54]

Bumped the bounty to $20. If anyone else would like to see this, feel free to contribute to the bounty.

[alexanderadam]

I just want to add that I would _love_ to see the repository size in it (like this extension does it for GitHub).
And this is probably also easier than language recognition.

EDIT: created a new issue for it as wished

[54]

@alexanderadam Good idea as well! As far as the language recognition goes, there is already a linguist port to Go.

[kolaente]

@alexanderadam mind opening a seperate issue for that?

[54]

Looks like @lafriks started work on this a little while back 👍 Figured it'd be worth linking the PR and issue - #4824.