Definitelytyped: Repository restructure?

[futagoza]

It seems that as more and more definition files are placed, loading the repository's main page is taking longer.

I was wondering if the repository could be restructured so that the main page just shows a list of directories which are named with a single character (a, b, c, 4, etc...), with all definitions placed in their appropriate directory.

This should provided a cleaner view and is still usable with tsd (for those that use it, excluding me :D), all while being mush faster in loading.

NOTE: _This is just a suggestion_

[horiuchi]

I thought the namespace would separate to npm, bower, or others...
And if so, we require the feature of searching defined libraries.

[futagoza]

The directory's for each project should contain the definitions for different platforms (npm, bower, github, etc...), this way it would be more manageable, but if 2 different projects on 2 different platforms (for now npm and bower) have the same name, then name-spaced directory's are a good idea.

Also, I think if a JSON based config file (maybe called tsinfo.json?) was present in each directory, then adding features like searching would be easier to implement.

As for the config's format, maybe this?:

{
  "version": {
    "=> 0.10.0": "node-0.10.d.ts",
    "=> 0.12.0": "node-0.12.d.ts",
    "=> 1.0.0": "iojs.d.ts"
  },
  "keywords": "node, iojs, js, es5, es6, io, server",
  "description": "JavaScript I/O",
  "dependencies": {
    "lib/v8": "*"
  }
}

The reason I suggest using the version field as an object is so that different definitions could be provided depending on the version condition being fulfilled. just as the dependencies field works in package.json for npm.

[cgwrench]

I agree that the performance of the repository homepage is an issue and is frustrating. However, there is a huge advantage to having all the projects with typings in a single flat list, and that is that it is easy to search the repository homepage for a package you are looking for, particularly if I'm not sure of the exact name of the folder containing a particular definiton.

For example, if I wanted to see if there was a definition for the jQuery succinct plugin, I can just search the repository homepage for "succient" and find that it is a folder called "succient". If I couldn't do, because each project were in a folder named with the first letter of the project name, then I might first look in the "j" folder, since I'm looking for a jQuery plugin.

If we do need reorganise the project structure, I'm not a fan of the use of single letter folders for organising a project. I would rather have logical groups of projects where it made sense. For example, currently, the main obvious groups of typings (to me) would be (numbers in brackets indicate the number of related projects):

• angular (29)
• boostrap (8)
• express (8)
• google (8)
• gulp (20)
• jquery (74)
• knockout (19)
• passport (7)

By consolidating related projects into the directorys listed above we would take 173 direcorties out and replace them with 8. this is a reduction of approximately 20% in the number of directories that would be listed on the project homepage.

We could even separate out these groups into their own repositories, in order to make them more managable, and simply use git submodules to include them into the main repository.

Again, this is just an idea.

[drinchev]

Have you ever considered splitting the repository? I think this issue is similar to #2150 where I just put a comment.

[enanox]

Anybody else noticed this?

[futagoza]

@enanox yep, I noticed as well, but I'm pretty sure it reached 1000+ in the root of the repo ages ago

[enlight]

However, there is a huge advantage to having all the projects with typings in a single flat list, and that is that it is easy to search the repository homepage for a package you are looking for, particularly if I'm not sure of the exact name of the folder containing a particular definiton.

Press 'T' on your keyboard while on the repository homepage, wait for the search interface to load, and then start typing what you're looking for. That lets you search the entire repository quickly.

[gregwebs]

@cgwrench that is the right idea. Using submodules will also increase the quality of code because authors will be willing to subscribe to notifications for pull requests to that subset. This is much more important than satisfying a github listing constraint.

The amazing thing is that removing 200 top-level listings is not enough anymore!

I think the suggestion mentioned here to separate out npm-only packages and browser-only packages is a good one. At the top-level

browser/
node/
anywhere/

These would be submodules to 3 separate repos. Each repo would do more groupings via submodules (jquery, angular, etc). This would begin to make things a little bit reasonable. Perhaps some more groupings could be found.

To search a flat file of all DefinitelyTyped listings, one should be able to do that on http://definitelytyped.org/

[HarelM]

With no disrespect to Boris and this awesome repository, did any of you guys thought about creating a github "company" - placing each folder as a separate project?
This is just an idea. I have recently did it with a repository I was maintaining and it feels a lot more professional. I know my example is a thousand times smaller but still...
This is the original repository:
https://github.com/HarelM/IsraelHiking
This is the new one:
https://github.com/IsraelHikingMap
This will allow contributers to fork just the relevant piece.
This will also allow issues to be open for specific definition.
What do you think?

[jbrantly]

I'm personally a big fan of @HarelM's suggestion. It would make it much easier for definition maintainers to watch repositories for PRs and issues. Right now I have to just come and do a blank search for "React" every few days to try to keep up on whats going on.

[HarelM]

@horiuchi, @futagoza, @gregwebs, @enlight, @enanox, @drinchev, @cgwrench.
Any updates on this?

[HarelM]

It look like someone already created a DefinitelyTyped "company":
https://github.com/DefinitelyTyped/
I think starting a migration process would be wise...
I'm not maintaining a project here but it look like the DefinitelyTyped has a project that converts projects to nuget packages.
Worth trying it out...

[futagoza]

@HarelM

1) I'm working on another project full-time, and although it involves JavaScript, it also involves C++, and requires hand optimised JavaScript for additional performance, so I haven't had time to work on TS related projects for a while now.

2) That's the official DefinitelyTyped "company", but the main project is still under borisyankov's repositories.

[TrabacchinLuigi]

Love the @harelm idea. Think u guys should delegate more

[nalbion]

Would be good if the structure could include frameworks which contain multiple sub-directories. Many of the visible directories have multiples prefixed by the following:

• jquery
• angular
  
  
  
  • bootstrap
  
  
  • material
  
  
• angular2
  
  
  
  • bootstrap
  
  
  • material
  
  
• backbone
• bootstrap
• chai
• cordova
• gapi
• google
• grunt
• gulp
• knockout
• node
• polymer
  
  
  
  • material
  
  
• misc

[jrylan]

I think this is worth revisiting, forking a huge repo like DefinitelyTyped can be quite imposing to developers and I feel like we're missing out on good contributions as a result.

git clone https://github.com/DefinitelyTyped/DefinitelyTyped.git
Cloning into 'DefinitelyTyped'...
remote: Counting objects: 326139, done.
remote: Total 326139 (delta 0), reused 0 (delta 0), pack-reused 326139
Receiving objects: 100% (326139/326139), 299.45 MiB | 2.84 MiB/s, done.
Resolving deltas: 100% (199971/199971), done.
Checking out files: 100% (26868/26868), done.

I know the biggest complication with migrating to an approach where each type definition has it's own repo would be difficult in the context of submissions for a package that currently has no definitions since there would be no repo for that for a developer to submit anything to.

But I think I might have a solution:

For packages with existing types:

1. Write an automated script to migrate the existing types into their own individual repos under the DefinitelyTyped GitHub org.

2. Create a publisher bot, lets call it @dtbot -- which leverages most of the existing logic used in the existing publishing process for submitting updates to npm.

3. For existing packages, developers will directly submit PRs to the repo for the package they are targeting.

4. For every merged PR to a package repo, the bot in step 2 will ensure everything is kept in sync.

For brand new packages without existing type definitions:

1. Setup a repo in the DefinitelyTyped org that is specifically for new packages.
   This could be called something along the lines of DefinitelyTyped/NewPackage.

2. A developer wishing to submit a new package will need to fork the NewPackage repo, create a new folder within it, add in their types and submit a PR back to DefinitelyTyped/NewPackage.

3. Instead of using the PR merge UI in GitHub, DefinitelyTyped admins will instead just comment on the issue and tag @dtbot with a command. Something as simple as:
   "@dtbot merge"

4. After being summoned via the comment, @dtbot would merge in the type files into a new branch that it creates (example: DefinitelyTyped/NewPackage#foobar. This would close the PR and the submitter would get the notice via GitHub that their PR is complete and they can delete their fork just like normal.
   (There are other ways in Git to handle this, but this is the only way I can think of that would keep the flow that the submitting developer would naturally expect from GitHub).

5. The new branch DefinitelyTyped/NewPackage#foobar would get picked up by the publisher bot, which would then create a new repo at DefinitelyTyped/foobar and cherry pick in all the changes from DefinitelyTyped/NewPackage#foobar then publish the npm package @types/foobar.

6. Once the new repo has been created, the publisher bot can delete the separate branch in order to save space, additionally running any available means of cleaning up or rewriting the Git history to keep the DefinitelyTyped/NewPackage repo as tiny and efficient to fork as possible.

7. Now we have a new repo and other devs can submit PRs to it and clone only that one specific package if they like.

[jakesyl]

Is there any interest in implementing this as per @jayrylan ? Clone size has gone up substantially since then: