Definitelytyped: Reorganize the repo?

[Keats]

Hey,

The current architecture for the repo does not really scale well for the number of typings there is now.
How about a a-z-other structure to clean it up a bit ?

I think I remember reading something about it but couldn't find the issue again

[cybrown]

+1

[lokeshpeta]

+1

[Keats]

I can do a PR if needed but that will break all existing PRs obviously

[Keats]

Actually why not move them to the DefinitelyTyped org at the same time @basarat @vvakame ?
This way we could just redirect people to the new repo and clear up the issues backlog since there are some dating from a year

[basarat]

Actually why not move them to the DefinitelyTyped org at the same time

Some other day.

How about a a-z-other structure to clean it up a bit

What do you mean?

[Keats]

Basically having _ a b c d ... z as the first folders of the repo and putting libs into those folder depending on their first letter such as:

a
|_ angular
|_ ace
....
j
|_ jquery
...

[basarat]

_ a b c d ... z as the first folders of the repo and putting libs into those folder depending on their first letter

Seems like a lot of work without much gain. Is the main motivation this message:

[drinchev]

There are a couple of issues that discuss this

https://github.com/borisyankov/DefinitelyTyped/issues/4513
https://github.com/borisyankov/DefinitelyTyped/issues/2150

What I proposed ( in #2150 ) was to split every definition file to a separate repo and use a central server to register repos as a definition file. ( https://github.com/borisyankov/DefinitelyTyped/issues/2150#issuecomment-117705706 ). Similar to the way NPM works.

IMHO this is inevitable, because the benefits of centralising so much developer's work under one repo is not going to scale, not only in terms of repo-access, but also Issue tracking, releases, maintenance, etc.

Maybe TypeScript team will also suggest a solution for this.

[Keats]

I agree with @drinchev that having all the libs in the same repos can't scale really in terms of issues/PR but having one new repo for each seems a bit overkill to manage. Maybe splitting them by letters at the repo level? ie definitions-a, definitions-b
For me the best solution is something like what https://github.com/facebook/immutable-js does, ie having definitions in the repo of the lib itself (and add the tests files) but it's much harder to do and not all libs will want to do that obviously.

@basarat that's easily doable by script

[basarat]

Maybe splitting them by letters at the repo level

Having it all in one repo has been working well in terms of providing a consistent user experience e.g. many defs depend on jquery or node ... splitting these out to different projects increases chances of a version conflict in user downloads and causes issues in tools.

that's easily doable by script

Indeed, the difficult part is updating all the tools / test frameworks ... there needs to be a strong motivation

[Steve-Fenton]

Duplicate of #4513

https://github.com/DefinitelyTyped/DefinitelyTyped/issues/4513

[HarelM]

Great to see that the repository had moved to the DefinitelyTyped Company! Kudos!
See my comment in #4513. I agree with @drinchev.
In terms of jquery and node being referenced in other repositories, well git submodule is obviously the solution that comes to mind - this will allow a repo owner to define the relevant version of jquery/node his repository is using, updating the jquery repo will not take effect until the submodule is updated.
The main advantage is the ability to open an issue to a specific definition file/group.
I honestly don't see a way around it in terms of maintenance and open source code sharing (If I want to update a module all I need to do is clone the relevant repo, not the entire code tree).
to summarize: @drinchev +1.

[drinchev]

I think this started to happen. TSD is being deprecated in favor of https://github.com/typings/typings .

https://github.com/DefinitelyTyped/tsd/issues/269

They separate the typings in different repos https://github.com/typings .

[HarelM]

I'm using Nuget with visual studio, currently all the links are pointing here. I hope this typing project will take ownership of the definitely typed code. as a side note, another distribution system of code is redundant as far as I'm concerned - Nuget, npm, bower are enough...

[jozefizso]

I think the DefinitelyTyped should be split into separate individual repositories. It is so large now that Travis builds takes hours to complete.

It would be much easier to contribute to smaller repositories and maintain it.

[mhegazy]

I think the DefinitelyTyped should be split into separate individual repositories. It is so large now that Travis builds takes hours to complete.

it is not the repo organization that is to blame here. We have done it in a wrong way for Types-2.0, and we run all the tests all the time. which is frankly stupid. @andy-ms should be fixing this and going back to running it only on changed modules, like it works in master today. in master it takes a few minutes for a typical change.

Repo reorganization is a valid discussion, but i just want to make it clear that CI performance has nothing to do with it.

[jozefizso]

...running it only on changed modules, like it works in master today.

Sounds great, that will help with Types-2.0 a lot.

[cyrilgandon]

@mhegazy : That sound great to run the test only on the folders that has been updated.

On the other hand, this method can brings another problem: if the TypeScript compiler change, we should also run the tests on the whole repo. That is the error we got today due to a change in the TypeScript compiler, one of the library stops working while nothing change (https://github.com/DefinitelyTyped/DefinitelyTyped/issues/12434)

To avoid this kind of issue, maybe we should stop using the nightly build of TypeScript inside the tests, but use the latest release instead.

[isiahmeadows]

At this point, there's over a thousand not showing up. Can't this be split into a-z? It's admittedly hard to find particular packages now - I only got to Node's because I knew the format.

[blakeembrey]

You can press t to search for files on any GitHub repo, that is extremely handy. However, at the very least, it might be worth moving all definitions into a definitions/ or types/ sub-directory. Having thousands of directories on the main screen is causing it a load a little slower, and possibly causing many people to overlook the README which is full of information. I know it takes me a little while to get to it, and I know it's there.

[monolithed]

I'd like to contribute for new type definitions, but my IDE does not respect this project :(
The most problem of this project is a large and uncontrolled size.

[HarelM]

This just moves the problem one step down to the next level, not sure why you decided to close the issue, but since it wasn't fixed until now I'm not very optimistic it will be solved correctly in the future.

[andy-ms]

We need to support the ability of packages to reference each other using "baseUrl": "../". So trying to divide the packages into various categories wouldn't work well.

To find a file on github you can press t from the main page. When editing locally you can cd to the package you want. So if you stay away from that particular URL shown in your screenshot, you should be good, right?

[HarelM]

I'm not sure I follow the baseurl issue, but I'm not an expert with this repository, what I said in a previous comment is that (to my best knowledge) this big repository can be broken to little repositories, each with the relevant name, the only place that the issue with "too many repositories" might appear is the main definitelytyped page with all the repositories.
If you have a dependency on other library you can use submodule to reference it - submodule will download the relevant dependency to a defined place so I believe (from what I understand) that it should handle the baseurl issue.
I understand this requires a ton of work, but this repository won't get smaller and it will become harder and harder to maintain in the future if this operation will not be done.
I haven't tried but I can image this can be done in steps by creating a repository on a "leaf" dependency and adding it as a submodule to this repository, as so step by step breaking this huge one to smaller pieces.
https://github.com/blog/2104-working-with-submodules

[andy-ms]

The discussion on that is in Microsoft/types-publisher#4. I know people don't like a big repo, but managing 1 repo with 3000 subdirectories is way easier than managing 3000 repos. See also babel.
This issue here is about the organization of this repository.

[HarelM]

Well I generally agree with @blakeembrey in the discussion you linked.
babel is a very small repository compared to this one, If this repository had 10 folders in it I would agree that it should be kept under the same roof, the discussion linked in babel is very shallow and bias...
I don't see how what I said is not relevant to this repository.
I don't agree about the management effort -
If I want to check if there's an issue with definitions I'm using instead of opening a duplicate issue it's almost impossible.
I'm not a library owner but I would guess that as a library owner that has a few .d.ts files working with this huge repository is a nightmare, especially when it comes to issues' turn around, IMHO.

[isiahmeadows]

Idea: use dedicated repos for individual typings like what Typings uses, and have a registry of versioned symlinks to each repository URL + commit/tag/etc. The registry would be housed in a separate branch here, and the default branch would be changed to the registry branch. The existing master branch would be kept, but it would be instead auto-updated in CI to reflect changes to the registry.

This would improve people's workflow here tremendously while avoiding breaking things. BTW, we already do that in Mithril (front-end framework) to regen the bundle on commits to next (our dev branch), leveraging Travis CI's environment variables to differentiate them from PRs or other events.