Marked: Dependents active??

[joshbruce]

Marked version: 0.3.19

Markdown flavor: n/a

Proposal type: other

What pain point are you perceiving?

3,000+ dependents...really??

What solution are you suggesting?

An automated way to iterate over the list of dependents. See what version they are using and when they last published to NPM. Hypothesis: Many of those packages are not active, reference Marked as prior art, or have a small audience base.

Tagging @UziTech, @styfle, and @Martii

See also #1123

[UziTech]

My google foo turned up this:

https://www.npmjs.com/package/dependents-cli

[styfle]

You can page through all 3000 dependendents.

But it doesn't look like there is any particular order.

@joshbruce What would we consider inactive? No downloads in the last 30 days?

[joshbruce]

@UziTech: ::sad trombone::

error getting dependents for [marked] { Error: getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:443

error getting dependents for [marked] Error: Invalid JSON (Unexpected "h" at position 1 in state STOP)

@styfle: I don't know what constitutes active, to be honest, and it's a fair question. Think I'm mainly looking for ways to even start gathering the data easily on the dependents. In some cases, paging through, the publish date would be a good first start (some were published in the last week). npm/npm is easy because they specify a version number and "give credit" when they update; so, we get notified that they're still active and actively using (aware of) Marked.

[UziTech]

@joshbruce the errors are from sending too many requests to the npm rest api. There has to be two requests per dependent. I am working on a way to fix that ... stay tuned

[UziTech]

Here is a distribution of the version numbers:

[
  [
    594,
    "^0.3.6"
  ],
  [
    450,
    "^0.3.5"
  ],
  [
    218,
    "^0.3.2"
  ],
  [
    182,
    "^0.3.3"
  ],
  [
    103,
    "~0.3.2"
  ],
  [
    77,
    "*"
  ],
  [
    76,
    "0.3.6"
  ],
  [
    59,
    "0.3.5"
  ],
  [
    54,
    "^0.3.9"
  ],
  [
    51,
    "^0.3.12"
  ],
  [
    49,
    "~0.2.9"
  ],
  [
    40,
    "0.3.2"
  ],
  [
    37,
    "~0.3.1"
  ],
  [
    35,
    "^0.3.7"
  ],
  [
    35,
    "0.2.x"
  ],
  [
    34,
    "~0.2.8"
  ],
  [
    31,
    "0.3.x"
  ],
  [
    30,
    "^0.3.17"
  ],
  [
    29,
    "~0.2.10"
  ],
  [
    23,
    "~0.3.5"
  ],
  [
    23,
    "~0.3.0"
  ],
  [
    23,
    "~0.3.3"
  ],
  [
    21,
    "^0.3.19"
  ],
  [
    21,
    "0.3.9"
  ],
  [
    20,
    "0.3.3"
  ],
  [
    18,
    "~0.2.5"
  ],
  [
    18,
    ">= 0.2.7"
  ],
  [
    17,
    "~0.3.6"
  ],
  [
    13,
    "0.2.9"
  ],
  [
    12,
    "latest"
  ],
  [
    12,
    "0.2.8"
  ],
  [
    9,
    "^0.3.16"
  ],
  [
    9,
    ""
  ],
  [
    9,
    "~0.3.9"
  ],
  [
    8,
    "0.3.12"
  ],
  [
    7,
    "0.2.5"
  ],
  [
    7,
    "0.3.0"
  ],
  [
    7,
    "^0.3.18"
  ],
  [
    6,
    "0.3.1"
  ],
  [
    6,
    ">=0.2.9"
  ],
  [
    6,
    "~0.2.7"
  ],
  [
    6,
    ">=0.2.5"
  ],
  [
    5,
    "~0.2.6"
  ],
  [
    5,
    "^0.3.4"
  ],
  [
    5,
    "0.3.*"
  ],
  [
    4,
    "^0.3.14"
  ],
  [
    4,
    "0.1.x"
  ],
  [
    4,
    "~0.3"
  ],
  [
    4,
    "0.2.6"
  ],
  [
    3,
    "~0.1.9"
  ],
  [
    3,
    "^0.2.10"
  ],
  [
    3,
    "^0.3.15"
  ],
  [
    3,
    "0.3.17"
  ],
  [
    3,
    ">=0.3.5"
  ],
  [
    3,
    "^0.3.0"
  ],
  [
    3,
    "https://github.com/mrpotes/marked/tarball/html-structure"
  ],
  [
    3,
    "x"
  ],
  [
    3,
    "0.2.7"
  ],
  [
    2,
    "~0.2.4"
  ],
  [
    2,
    "^0.x"
  ],
  [
    2,
    "0.2.4"
  ],
  [
    2,
    "=0.3.2"
  ],
  [
    2,
    "=0.2.5"
  ],
  [
    2,
    ">= 0.2.5"
  ],
  [
    2,
    ">=0.1.2"
  ],
  [
    2,
    ">= 0.2.9"
  ],
  [
    2,
    "~0.3.17"
  ],
  [
    2,
    "~0.3.7"
  ],
  [
    2,
    "~0.2"
  ],
  [
    2,
    "~0.2.x"
  ],
  [
    2,
    "0.2.10"
  ],
  [
    2,
    "^0.3"
  ],
  [
    2,
    ">=0.2.8"
  ],
  [
    2,
    ">=0.2.10"
  ],
  [
    1,
    "0.3.4"
  ],
  [
    1,
    "0.0.4"
  ],
  [
    1,
    "0"
  ],
  [
    1,
    "~x.x.x"
  ],
  [
    1,
    "https://github.com/akira-cn/marked/tarball/gitpress"
  ],
  [
    1,
    ">= 0.2.10"
  ],
  [
    1,
    ">=0.0.0"
  ],
  [
    1,
    "0.3.19"
  ],
  [
    1,
    ">= 0.1.7"
  ],
  [
    1,
    "0.3"
  ],
  [
    1,
    ">= 0.2.8"
  ],
  [
    1,
    ">=0.3.0"
  ],
  [
    1,
    "^0.3.1"
  ],
  [
    1,
    ">= 0.2.6"
  ],
  [
    1,
    "~0.3.4"
  ],
  [
    1,
    "~0.3.x"
  ],
  [
    1,
    ">=0.2.6"
  ],
  [
    1,
    "~0.3.19"
  ],
  [
    1,
    "~0.3.16"
  ],
  [
    1,
    "github:rumkin/marked"
  ],
  [
    1,
    ">= 0.3.3"
  ],
  [
    1,
    ">= 0.2.4-1"
  ],
  [
    1,
    ">= 0.3.5"
  ],
  [
    1,
    "git://github.com/twinlabs/marked.git"
  ],
  [
    1,
    "http://registry.npmjs.org/marked/-/marked-0.3.2.tgz"
  ],
  [
    1,
    ">= 0.1.9"
  ],
  [
    1,
    "~0.1.6"
  ],
  [
    1,
    "git://github.com/chjj/marked.git"
  ],
  [
    1,
    "git://github.com/chjj/marked.git#master"
  ],
  [
    1,
    "git+https://github.com/crazyguitar/marked.git"
  ],
  [
    1,
    ">=0.3.3"
  ],
  [
    1,
    "git://github.com/mgutz/marked"
  ],
  [
    1,
    "^0.2.8"
  ],
  [
    1,
    ">=0.3.2"
  ],
  [
    1,
    "== 0.2.5"
  ],
  [
    1,
    ">0.2.4"
  ],
  [
    1,
    "0.2.3"
  ],
  [
    1,
    "github:rhysd/marked#emoji"
  ],
  [
    1,
    "git://github.com/clehner/marked#8af8ae018ba452b5c7f936a474dbce265f2ae732"
  ],
  [
    1,
    ">= 0.3.0"
  ],
  [
    1,
    "0.2.109"
  ],
  [
    1,
    "=0.3.3"
  ],
  [
    1,
    ">=0.1.4"
  ],
  [
    1,
    "0.3.18"
  ],
  [
    1,
    "git+https://github.com/chester1000/marked.git"
  ],
  [
    1,
    "git://github.com/jnordberg/marked.git"
  ],
  [
    1,
    "0.3.16"
  ],
  [
    1,
    "git+https://github.com/davitv/marked.git"
  ],
  [
    1,
    "0.0.1"
  ],
  [
    1,
    "0.x"
  ],
  [
    1,
    ">=0.2.10 <0.3"
  ]
]

[UziTech]

I created a repo with the code I used to get the dependency info and some stats

[styfle]

@UziTech Very cool!

One thing to improve the results is to interpret the semver to see which version would be downloaded if they installed today.

For example,

• ^0.3.6 ^0.3.3 ^0.3.5 ^0.3.2 would actually install 0.3.19 today since that's the latest of the 0.3.x line.
• ~0.3.2 would actually install 0.3.19
• * would actually install 0.3.19

Can you aggregate the version that would be installed, not the literal string in the package.json?

[joshbruce]

@UziTech: That's pretty awesome. I took a look at the repo and, honestly, it landed somewhere between my own ignorance and "I just woke up". How much of the output can we tailor? What are the ids (they don't seem to be unique as there are multiple 7s, for example)? Can/Should we move that repo to the Marked org?

@styfle: That's a fair point. One of the things I appreciated about the supposed output of the other CLI tool was the table that basically had the name of the package and the version string. I wonder if there would be a way to get both the version specified in the .json and what that would end up being in NPM terms.

[UziTech]

What are the ids

What ids?

Can/Should we move that repo to the Marked org?

Ya that would be fine. This was just my quick PoC to check if we could/how to get info about our dependents.

If this is going to be something we actually want to use to keep track of our dependents, we should come up with some metrics that we want to make visible and I could show those metrics on the readme.md and update it every week or so.

Currently because of npm's api request throttling it takes about 2 minutes to get all the dependents info.

[joshbruce]

What ids?

  [
    1, <- Interpeting these as some form of identifier
    "0.0.1"
  ],

Ya that would be fine. This was just my quick PoC to check if we could/how to get info about our dependents.

Definitely seems like a good start.

we should come up with some metrics

Think for the purposes of user analysis, an object might look something like this (hence the question of how much we can manipulate output (this assumes everything):

{
  name: [package name],
  url: [npm package url],
  version: [package version],
  published: [last publish date to npm],
  marked: {
    version: [package.json version]
  } 
}

Those were the sorts of metrics I was looking at when I was leafing through dependents.

[UziTech]

That is not the id, that is the number of dependents that use that version.

The packages folder is where I save all dependents info:

{
    "name": "Antony",
    "version": "0.3.0",
    "depVersion": "^0.3.6",
    "maintainers": [
        {
            "name": "spiral",
            "email": "[email protected]"
        }
    ],
    "modified": "2016-09-09T16:27:48.031Z",
    "released": "2016-09-09T16:27:48.031Z",
    "downloadsLastWeek": 2
}

[styfle]

@joshbruce Can we close this issue now that there is a separate tool created to solve this problem?

[joshbruce]

@styfle: Think we're almost there.

@UziTech: Would it be possible to put the repo under the MarkedJS org? Maybe also provide a README on how to run it?

[styfle]

This page provides some valuable insights https://libraries.io/npm/marked/usage

[Martii]

That's interesting... it' polling OUJS GH dev (so GH in general). Kewl.

---

Wonder how often they reindex... because 0.4.0 isn't listed although could be with the ^... but we're fixed so guess Other.

[styfle]

@Martii Let's see if they respond on twitter 🤷‍♂️

Here's ^0.4.0 and exact 0.4.0

[joshbruce]

Deleting @uk1free comment as off topic. Hiding the other one.

[styfle]

Let's see if they [libraries.io] respond on twitter

It must be somewhat often because they have ^0.5.0 and 0.5.0 listed already.

@UziTech @joshbruce Do we still want to move UziTech/marked-dependents under the markedjs org on GitHub?

[UziTech]

I transfered ownership of marked-dependents

[styfle]

Excellent! 🎉
I'll close this issue 😄