Package-maintenance: [key packages] Pilot Packages

[Eomm]

Split 1/3 of #105

We can use to start collaborating with the authors to learn and iterate with.

Edit more info:
to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:

• explain the biggest problems and their priorities (as talk about #113 )
• give feedback to our improvements/suggestion/PR

---

Express and all its direct and transient dependencies like pillarjs and jshttp are the most mentioned package to start with.
@wesleytodd has discussed a bit of this possibility on the last Express TC meeting so I think the Express team would be on board to participating.

What to discuss here

• We need to agree or not to start with Express as a pilot package. If "no" what else?
• To define if we can and should add more "friendly" pilot packages

After that we could start to get "Directions of Help": the way package maintainers wish to receive help and define the first tasks.

Ex:
Express needs help with repo cleanup, automation, documentation and, user support.

[mhdawson]

Starting with Express sounds good to me, given that they are a key part of the ecosystem and it sounds like they are open to working with us.

[wesleytodd]

While clearly I am a fan of working with Express, I think we need at least a few others to make sure we have a variety of input. Anyone have other suggestions?

[mcollina]

I think https://github.com/mqttjs/MQTT.js and dependencies are another good place to start.

[mhdawson]

I agree we need to have several and MQTT sounds like a good one to add to the list.

[Eomm]

I don't know other friendly packages that could collaborate with us in this pilot phase.

Appling manually the basic criteria we talk about, I think that request could be a candidate: it is widely used and have a lot of issue and PR: it seems they need help to break all that work!

I looked here, there is a nice bubble schema that show the libs per usage, I took the first one with many issue&PR opened. Nothing personal 😁

[wesleytodd]

I agree request seems like a good target. @reconbot & @mikeal Does this program seem like something you would be willing to participate in? I know you two are involved in request, but if there is anyone else we should ping please feel free!

@mcollina Are you the primary maintainer for MQTT? If not, are there others we should reach out to to see if they are willing to participate?

[nschonni]

Maybe some of the plumbing packages that are depended on for the native packages like node-pre-gyp
I'm hoping to move node-sass to that in the next major version, which would bump it up close to request numbers (which is also a dependency for both)

[mcollina]

@wesleytodd I'm the only active one with publish access. I can add more if it is needed, but I'm probably the only one that knows how all the pieces fits together.

[reconbot]

I've been doing some bug triage for request, and have landed some patches but haven't yet done a release. The future of that package is a discussion. There is a desire to leave it as is and encourage the use of more modern packages even, however keeping it supported is obviously desirable. I've read the linked issue and the blog post, but I'm not sure what the work would be to be part of this project.

[wesleytodd]

I think that is a great stage for this WG to look at. I think we should be taking into consideration the entire life cycle of maintenance, including EOL like you say request is (although I feel like this might come as a shock so some users). That is to say, I don't believe this should disqualify it from consideration here.

[wesleytodd]

Also, @reconbot, maybe you will have some input on #139. Does that proposal seem like something which would help signal to users the state the maintainers think the project is in?

[chrkaatz]

One other candidate might be nedb as the creator of it asked for help himself but I know that the package is not used very much anymore these days.

[mikeal]

It’s not clear to me from the thread here and the links exactly what this pilot program means and what the goals are.

When engaging projects it would be great to have a concise write up of:

• The problems this program is designed to address.
• What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
• What is being offered by this group to these projects.

[wesleytodd]

Great points. @Eomm maybe we could edit the OP and add some of this?

The problems this program is designed to address.

The goals of this WG can be found here.

What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.

A pilot project would be the first group we try to help. The requirement would be a willingness to try some ideas and provide feedback on how they work.

What is being offered by this group to these projects.

This is in the process of being defined. And the hope is that the pilot projects help us refine and decide on what specific things would be best for us to provide.

[Eomm]

@wesleytodd edited 👍
I hope to have clarified the benefit and the effort in play.

[Eomm]

Hi, I would like to recap the topic to speed up the meeting tomorrow:

| Candidate | Approved by candidate | Main Contact |
|---|---|---|
| express | yes | @wesleytodd |
| mqtt | yes | @mcollina |
| request | need information | @mikeal |
| plumbing packages that are depended on for the native packages | - | - |
| node-sass | - | @nschonni |
| nedb | yes | @chrkaatz |

I hope I have correctly summarized 😁

[Eomm]

It’s not clear to me from the thread here and the links exactly what this pilot program _means_ and what the goals are.

When engaging projects it would be great to have a concise write up of:

• The problems this program is designed to address.
• What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
• What is being offered by this group to these projects.

Hi @mikeal, to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:

• explain the biggest problems and their priorities (as talk about #113 )
• give feedback to our improvements/suggestion/PR

I hope to be exhaustive, let us know what do you think about it
Thank you

[JamesMGreene]

I would also be interested in being a maintainer of NeDB. I try to respond to most of the newly opened issues in the existing repository, and have also forked the project as NestDB to implement some requested new features, bug fixes, and performance improvements that its sole maintainer was not keen on merging or able to give time to (a problem with maintainership that so many of us know all too well).

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

There have also been other forks of NeDB, as nedb2, nedb3, nedbhq/nedb-core, and even the now unrecognizable TeDB (TypeScript-based), so the popularity of NeDB -- and its consumers seeking for actual maintenance -- definitely has no doubt in my mind.

[mhdawson]

@wesleytodd We touched on this briefly and we wondered if we are ready to take the next steps with either express or mqtt as we continue to build out the pilot list?

[wesleytodd]

Hey, yes any time. Do we have a concrete list of the first steps? I seem to remember an issue regarding this, but I have so many unread messages at the moment (👶 time). The start was a survey right? If so I can get that posted up in the express repo so we can fill it in.

[wesleytodd]

Ok, I posted this on the express discussion repo. I will hopefully get other contributor feedback and start working on those answers soon.

https://github.com/expressjs/discussions/issues/77

[Eomm]

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

@JamesMGreene could you ping the maintainer of NeDB to join the discussion?
Because I think we can't help that package if the maintainer can't collaborate for whatever reason.

[Eomm]

In last meeting we talked to find more packages

We could start digging from this list:

The top 20, to save you a click:
debug
kind-of
supports-color
readable-stream
source-map
yargs
camelcase
minimist
strip-ansi
chalk
commander
@types/node
punycode
ansi-regex
ms
glob
define-property
semver
async
string_decoder

https://twitter.com/seldo/status/1115725616433602560?s=19

[mcollina]

https://docs.google.com/spreadsheets/d/1lZDNYsLntwD2q9XaTw-XLuG0VpHH6cOV-Uoa7Y1aTSM/edit#gid=1745448509 contains a list of the top 1000 most downloaded modules.
It would be good to do some sort of data analysis of the dependants and dependencies of these, as I guess there are a lot of overlaps and macro-trends.

[dominykas]

Here's some percentiles on last update age of top 1000 packages:

percentile | last commit | last publish | last update to .travis.yml
-----------|-------------|--------------| --------------------------
5% | 1 days old | 7 days old | 10 days old
10% | 4 days old | 11 days old | 34 days old
15% | 5 days old | 18 days old | 56 days old
20% | 5 days old | 37 days old | 92 days old
25% | 10 days old | 58 days old | 126 days old
30% | 19 days old | 88 days old | 129 days old
35% | 33 days old | 131 days old | 129 days old
40% | 57 days old | 157 days old | 160 days old
45% | 81 days old | 200 days old | 196 days old
50% | 119 days old | 246 days old | 259 days old
55% | 150 days old | 300 days old | 324 days old
60% | 194 days old | 369 days old | 387 days old
65% | 256 days old | 438 days old | 501 days old
70% | 328 days old | 559 days old | 639 days old
75% | 418 days old | 669 days old | 723 days old
80% | 563 days old | 777 days old | 891 days old
85% | 706 days old | 944 days old | 1036 days old
90% | 887 days old | 1098 days old | 1214 days old
95% | 1194 days old | 1372 days old | 1501 days old
100% | 2201 days old | 2792 days old | 2620 days old

This means that roughly half of them have not been published since last node LTS... Now, that in itself does not mean they weren't tested in that LTS (even it travis.yml wasn't update, although that's a good indicator), but it gives perspective on how much of just the top 1000 is either "abandoned" or "done".

What kind of dependents / dependencies data did you want to see @mcollina?

[dominykas]

And here's some stats from versions extracted from .travis.yml (didn't bother cleaning those up, just yet):

node version count in travis.yml

version | count
--------| ------
1 | 3
2 | 3
3 | 3
4 | 269
5 | 128
6 | 507
7 | 112
8 | 459
9 | 85
10 | 381
11 | 53
0.1 | 6
0.10 | 257
0.10.12 | 1
0.11 | 29
0.12 | 244
0.12.0 | 1
0.4 | 1
0.6 | 31
0.8 | 98
0.8.6 | 1
0.9 | 2
1.7.1 | 1
1.8 | 37
10.0 | 2
10.11 | 2
10.12 | 5
10.13 | 2
10.14 | 3
10.15 | 25
10.2 | 1
10.4 | 1
10.6 | 1
10.7 | 2
10.8 | 1
11.0 | 2
11.10 | 8
11.12 | 1
11.13 | 1
11.14 | 1
11.2 | 1
11.4 | 2
11.6 | 5
11.7 | 3
11.8 | 1
11.9 | 2
2.5 | 37
3.3 | 37
4.0 | 6
4.0.0 | 3
4.1 | 4
4.2 | 4
4.3 | 1
4.4 | 3
4.4.0 | 1
4.5 | 2
4.7 | 1
4.8 | 7
4.9 | 48
5.0 | 2
5.0.0 | 1
5.1 | 1
5.10 | 3
5.11 | 1
5.12 | 55
5.2 | 1
5.3 | 1
5.4 | 1
5.5 | 1
5.6 | 2
5.7 | 2
5.8 | 1
5.9 | 1
6.0 | 4
6.0.0 | 1
6.1 | 1
6.10 | 1
6.11 | 1
6.12 | 2
6.13 | 2
6.14 | 18
6.15 | 4
6.16 | 21
6.17 | 5
6.2 | 1
6.3 | 1
6.4 | 1
6.5 | 2
6.6 | 1
6.9 | 1
7.10 | 53
7.7 | 1
8.0 | 1
8.11 | 9
8.12 | 8
8.13 | 2
8.14 | 3
8.15 | 26
8.6 | 1
8.9 | 5
9.11 | 46
9.3 | 1
9.5 | 2
iojs | 50
iojs-1 | 1
iojs-2 | 1
iojs-3 | 1
iojs-v1.0 | 1
iojs-v1.1 | 1
iojs-v1.2 | 1
iojs-v1.3 | 1
iojs-v1.4 | 1
iojs-v1.5 | 1
iojs-v1.6 | 1
iojs-v1.7 | 1
iojs-v1.8 | 17
iojs-v2.0 | 1
iojs-v2.1 | 1
iojs-v2.2 | 1
iojs-v2.3 | 1
iojs-v2.4 | 1
iojs-v2.5 | 17
iojs-v3.0 | 1
iojs-v3.1 | 1
iojs-v3.2 | 1
iojs-v3.3 | 17
lts/* | 18
node | 196
stable | 52
v0.12 | 1
v10.* | 1
v4 | 3
v5 | 2
v6 | 2
v6.* | 1
v8.* | 1

[mcollina]

Scavenging that data is very interesting.
I’d like to see if there are some dependency relationship. For example, we know that express uses debug. This info is helpful because we can take some macro-family.

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

[ljharb]

tbh what I’d really like to see is top packages on npm with download counts of dependents and devDependents subtracted out

[dominykas]

This info is helpful because we can take some macro-family.

Sorry, not sure I'm following. You'd like to have the top1k somehow clustered into a group of deps that belong to a single dependent or smth? Or to put it differently - try to find a group that would likely get downloaded together? I can try to look into the dependency links in the top1k, but we don't have an easy, publically available method to go beyond that...

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

So getting these numbers is easy, but gaining insight is hard... Do you have any specific ideas or questions in mind?

A number of open issues on its own does not say anything (some maintainers close everything, others let issues linger, esp. support ones), the issue/pr rate over time is also meaningless - low activity could just be an indicator of done-ness and stability and I'm not sure how that is helpful?

---

What problems are we looking to solve with this? What problems can be discovered? I'm usually a fan of looking for data that can help make decisions, e.g. the numbers I posted can be used to determine which packages need some touching up to at least have CI run with newer node versions. Are there any other specific things we could be looking for?

[Eomm]

Are there any other specific things we could be looking for?

I think that this question is more related to this issue https://github.com/nodejs/package-maintenance/issues/143#issue-404515581 and here (pilot packages) I think we are looking for collaboration mainly from the maintainers of the modules and download numbers.

Adding issues/PR numbers will be useful for sure, but we could proceed step by step building the tool that we will use to list the modules to find them out so.. who are those packages in the >=50% from last travis update?

[dominykas]

who are those packages in the >=50% from last travis update?

I'll open a separate issue with the list shortly and we can take it from there.

[Eomm]

I would close the issue since the pilot packages have been selected

I don't know if we could ask to join in the #221 #239

[wesleytodd]

Do we feel that the pilot packages issues have been addressed? I know we have a lot of things in the works, but it seems to me that unless we think we are ready to move onto the next phase we should keep this open.

IMO, the goal of the pilots are to get some actual working tests for the ideas we have. From express, we are only just now starting to implement the stuff we have talked about, and have not really even gotten to give feedback on their success/failure. I dont know how the MQTT progress is on this, but I haven't seen anything concrete here yet.

Thoughts?

[Eomm]

It is true, I thought the target for this issue was to find the pilot packages.
We can keep it open to collect the results also for sure 👍

[mhdawson]

+1 to keeping it open. We are starting to make some progress but I'd agree there is more work before we can consider this part complete.