Shields: dependency version badges for applications, from package.json

[guylepage3]

Description
I would love to be able to add dependency version badges from my package.json and have those badges change to let developers know that a dependency is outdated. I am aware of david-dm.org for example...

But it would be amazing to have a badge for each of my dependencies..

[paulmelnikow]

Hi! Thanks for the suggestion.

Is the idea that you would identify your repo, and a single dependency, and then it would show something like

jest | up to date

if that dependency was current, and

jest | outdated

when it's not?

[guylepage3]

Hey @paulmelnikow, I was thinking more along the lines of...

Up to date version is blue or green...

Outdated version would be in red..

[nottrobin]

Any movement on this? Did anyone ever find a way to do this? I would also like to be able to display the exact version of a specific dependency that's used by the project.

[paulmelnikow]

Yea, we chatted on Discord about using the dynamic badge. @guylepage3 could you point us to what you ended up with?

[chris48s]

Here's an example using dynamic json badge. The query is $.dependencies.<mydependencyname>

https://img.shields.io/badge/dynamic/json.svg?url=https%3A%2F%2Fraw.githubusercontent.com%2Fbadges%2Fshields%2Fmaster%2Fpackage.json&label=camp&query=$.dependencies.camp&colorB=blue

https://github.com/badges/shields/blob/0aee712738f0df50cfa6c62d7cf65b053c1e73fa/package.json#L25

[guylepage3]

Yeah. I was only able to execute what @chris48s is suggesting above. I would really love to remove the tilde ~ or ^ and replace it with v. But yeah. I didn't have anymore time to put into it. Sorry.

[nottrobin]

Works perfectly, thanks @chris48s, @guylepage3. I'm not sure replacing the ~ or ^ would be the right thing to do, as someone installing dependencies from package.json might not get that exact version. If the script could instead read from package-lock.json or yarn.lock, then you could confidently communicate the exact version.

Does anyone know if there's a way to get a similar badge for specific Python dependencies from requirements.txt?

[guylepage3]

For my purposes I was using the badge as a quick visual to let the viewer know which version of the dependency was currently in use. So for me it made sense. Much like how shields.io has set up my npm badge.

It's also a standard.. https://shields.io/#/examples/version

[paulmelnikow]

I can't think of a way to parse requirements.txt using the dynamic badge, though I'd really like to support custom use cases like this through a combination of #1525 and a RunKit endpoint.

Also, as I think about this more, I'd be curious to see how you're using these requirements badges to communicate with developers. Could you guys link to repos?

[nottrobin]

@paulmelnikow for me, this is just something I'm thinking of doing, so don't have examples. But these are how I'd like to use it:

• To communicate the Django version we're using in our Django website repos - e.g. https://github.com/canonical-websites/www.ubuntu.com
• The same with our Flask website repos - e.g. https://github.com/canonical-websites/snapcraft.io
• We use a styling framework, vanilla-framework, across all our sites. We'd similarly like to communicate the version of vanilla-framework on each site (upgrades represent evolution in our styling philosophy).

In general, our repositories tend to have one or two particular dependencies (usually frameworks) that we would consider fairly central, where the exact version of those dependencies is particularly interesting.

[paulmelnikow]

Ah, thanks @nottrobin, that's really helpful! I think I understand some possible use cases:

• Warn other developers that an application still uses an old version of a dependency
• Signal that some upgrade attention may be needed.

We do have a bunch of platform support badges. However the ones for Node and Python are designed for libraries. The ones for Python and Django version look at PyPI trove classifiers, and the node ones look at engines. They both look at published packages.

Have you given consideration to adopting pipenv? Pipfile uses TOML, whereas requirements.txt is completely nonstandard to parse… not to mention problematic by nature.

We have a GitHub manifest version badge, which reads the version number from package.json or manifest.json. I could see adding support to show the version of a specific dependency. We could do the same for a Pipfile. It could explicitly support the big ones, like Flask and React, sort of like we do now for Django in libraries. It could let developers specify a specific library, too.

Finally, we could also add a dynamic TOML badge for people who want to do really specific things with their Pipfile.

Do you think that would work for you?

[nottrobin]

We have looked at pipenv. The problem with it for us is that it doesn't have a way to generate lockfiles from system level dependencies. We encapsulate all our dependencies using containers, so we have no need for the pipenv user environments.

It's a shame that pipenv bundles in the new dependency format with its environment management, so you can't have one without the other, and that it's incompatible with other environment management options - like storing the dependencies at the system level.

[paulmelnikow]

System level dependencies… is that components like gunicorn and statsd? I take it they're getting installed here?

https://github.com/canonical-websites/www.ubuntu.com/blob/1116f4c8bb9d916477d40a376634e244f915ee19/Dockerfile#L12

[nottrobin]

That Dockerfile is actually only used for production. What I mean is that we do local development effectively like this (illustrative only):

docker run \
    --volume `pwd`:/srv  \
    --workdir /srv  \
    --publish 8000:8000  \
    python:3  \
    bash -c "pip install -r requirements.txt && ./manage.py runserver"

(In practice, we have a bash script at ./run in each project, which spins up the appropriate Docker containers for you, with some extra settings)

We don't actually use the python3 image, and it doesn't have to be Docker (it could easily be lxd). The point is, rather than using filesystem and $PATH tricks to mock an encapsulated environment for our Python dependencies, we use linux containers.

Using Python virtualenvs can both often have unexpected effects on the wider system (there's nothing to stop Python packages writing to or changing permissions in your home directory - and many do) and break because of quirks in your wider system (like not having specific C libraries installed or set up correctly). Using containers offers significantly more reliable encapsulation.

This allows us to have confidence that the project will run the same for each of our devs, on both Linux and macOs, with very little knowledge about the project or help from other devs. The single dependency our devs need installed is Docker. We've been doing it this way for about 4 years and it works very well.

The simplest way to do that is to run everything as root inside the container - installing everything at the system level. We could, of course, configure our containers to have a whole encapsulated user environment and still use Python virtualenvs inside the container, but it's a whole load of extra weight and complexity to our containers that we don't need.

Pipenv looks like a really promising project, and I'd love to use it (especially its lockfile functionality), but it unfortunately doesn't play at all well with actually working with dependencies that are installed at the system level. I've tried to delve into this problem with the Pipenv devs, but unfortunately I don't think they have any plans or appetite to fundamentally rearchitect Pipenv to support our workflow.

[paulmelnikow]

Ahhh I gotcha. That pipenv thread sounds more philosophical than technical… and frustrating. Heh. Anyway, I get it, you need to use requirements.txt.

I'm not really up for implementing a requirements.txt parser in Node. (The only one I found was deprecated and labeled "trash" by its author.)

I'm thinking this case could be handled using a custom endpoint. I'll resume working on reviving #1525 – that'll allow creating a badge from an arbitrary JSON endpoint which returns something like { "label": ..., "message": ..., "color": ...}.

That opens up options for the implementation. Runkit endpoints is a good one for Node. Basically you write a function like

exports.endpoint = function(request, response) {
    const badge = { schemaVersion: 1, label: ..., message: ..., color: ... }
    response.end(JSON.stringify(badge))
}

and save it in your browser and then, voila, you have a JSON endpoint. I imagine your limited case could be handled well enough in Node with a regex.

Alternatively, it might be easy enough to parse requirements.txt using some Python code. I haven't looked too far into Python endpoint options. Are you familiar with Jupyter? This approach using Jupyter Kernel Gateway seems promising. I don't know whether or not the free Jupyter notebook hosting providers like Azure Notebooks would provide something that would work with Kernel Gateway.

[nottrobin]

Interesting option. I haven't looked into it, but I'm sure parsing requirements.txt in Python must be pretty trivial.

But I suspect the simplest would be, as you say, to write a small JSON endpoint that ran a simply regex against our latest requirements.txt. Do you have an example of such an endpoint that parses a file from a GitHub repo that we could base it off?

And do you know if #1525 is likely to be merged soon?

[paulmelnikow]

I put this together: https://runkit.com/melnikow/version-from-requirements-txt

It seems to work well enough in this basic case:

• https://runkit.io/melnikow/version-from-requirements-txt/branches/master
• https://runkit.io/melnikow/version-from-requirements-txt/branches/master?dependency=metabolace

Cool stuff!

Yea, I'm working now on a replacement for #1525.

[chris48s]

I'm not really up for implementing a requirements.txt parser in Node

Writing a javascript parser for requirements.txt is way down my list of things to do as well (like way way down). That said, if someone _did_ want to do that, as noted in https://github.com/badges/shields/issues/775 there is a NPM package for dealing with PEP440 versioning: https://www.npmjs.com/package/@renovate/pep440 which has some relevant helpers and this is an example of some Node JS code that consumes that to parse a requirements.txt file: https://github.com/renovatebot/renovate/blob/bed39f10ed768c802c551cd438fceb22c0810a95/lib/manager/pip_requirements/extract.js (Rennovate Bot is similar to greenkeeper, pyup, dependabot, etc)

Pipenv looks like a really promising project, and I'd love to use it (especially its lockfile functionality), but it unfortunately doesn't play at all well with actually working with dependencies that are installed at the system level

I realise we've gone way way way off the original topic of this thread at this point, but if pipenv / Pipfile isn't a good solution for your project, poetry / pyproject.toml might be worth a look:

https://github.com/sdispater/poetry
https://poetry.eustace.io/docs/pyproject/

That said, for the love of god don't change your packaging tool if the only problem you're trying to solve is adding a badge to your README.md :D That's specialist-yak-barbershop levels of yak-shaving right there, especially given we don't even support TOML for the dynamic badge yet.

[nottrobin]

Thanks @chris48s, Poetry is very interesting. I'll try to look into it a bit further.

for the love of god don't change your packaging tool if the only problem you're trying to solve is adding a badge to your README.md :D That's specialist-yak-barbershop levels of yak-shaving right there

Lol no. We've previously been looking into alternatives to requirements.txt. The main feature we wanted at the time (and still want) is a lockfile with package signatures to lend our builds a bit more security.

We'd also like to stay up-to-date with the latest standards, so it's a shame that https://github.com/sdispater/poetry branches the possible futures for Python. But, as you can see, Pipenv isn't without its problems, so it's probably a good thing.

[guylepage3]

Hey @paulmelnikow was this resolved? 😄... If so, how may I implement this?

[paulmelnikow]

Yup! There are two badges, one for npm packages and the other for github. If you search for "dependencies" on https://shields.io/ they'll both come up:

Btw @nottrobin the Endpoint badge is still in progress. The best place to track that is #2473.

[guylepage3]

Awesome! Looking forward to this! 😄

[paulmelnikow]

Hey @nottrobin, the Endpoint badge is live! Want to give it a shot? We can continue the discussion in #2838.

[nottrobin]

@paulmelnikow works a charm!

https://github.com/canonical-websites/www.ubuntu.com/blob/master/README.md

Thanks so much!

[paulmelnikow]

Nice!! Can you share a link to your RunKit notebook?

[paulmelnikow]

Also… could you help spread the word about this?
https://twitter.com/paulmelnikow/status/1087707442824310785

[nottrobin]

@paulmelnikow this is it: https://runkit.com/nottrobin/django-from-requirements-txt. It's basically a copy of your proof-of-concept, with just one or two tweaks.

[paulmelnikow]

Just thought I'd ping this thread to let anyone watching know that there is now a dependency badge for Pipenv applications. It reads Package.lock from a GitHub repo:

[jonashackt]

Really great service for using the package.json dynamically @paulmelnikow 👍

In my case this great service didn't work, since the package.json doesn't reside in the root of my GitHub repo, but in a subdirectory called frontend - the url to the package.json is: https://github.com/jonashackt/spring-boot-vuejs/blob/master/frontend/package.json Currently I found no option to use the great tools provided at https://shields.io/category/platform-support to use it.

If someone stumbles upon this issue like me, here's my solution how to get it working though: Just use the Dynamic endpoint https://img.shields.io/badge/dynamic/json? like that to query the frontend/package.json:

https://img.shields.io/badge/dynamic/json?color=brightgreen&url=https://raw.githubusercontent.com/jonashackt/spring-boot-vuejs/master/frontend/package.json&query=$.dependencies.vue&label=vue&logo=vue.js

You simply leverage the power of JSONPATH and use query=$.dependencies.vue to query the current version of your project's dependency (I used https://jsonpath.com/ for an easy validation of my JSONPATH query). Also make sure to use the raw.githubusercontent.com version of your package.json, otherwise the shields.io service can't parse the json.

And I also had packages not defined inside my package.json, but inside the package-lock.json. Using this approach, this is also not a problem:

https://img.shields.io/badge/dynamic/json?color=brightgreen&url=https://raw.githubusercontent.com/jonashackt/spring-boot-vuejs/master/frontend/package-lock.json&query=$.dependencies.webpack.version&label=webpack&logo=webpack

[paulmelnikow]

Hi @jonashackt,

All good suggestions, including the workaround you suggested!

Per the comment above I recently added a badge for Pipenv which uses the lockfile, and it makes sense to add some corresponding support for package-lock.json. I think it’s a little tricky in a package-lock because there could be multiple versions of a particular dependency installed. I’m not sure if it makes sense to consider only the top-level dependencies, or to walk the tree and find all the installed versions.

It would be good to add a package.json dependency badge which did not depend on GitHub (similar to the other package.json badges we have).

I opened a PR #WHICH to add monorepo support to the existing GitHub badge.

So there are two open items. Feel free to open new issues for these – or to tackle them yourselves if you'd like!

• [ ] Add a GitHub dependency badge which supports package-lock.json
• [ ] Add a package.json dependency badge which works for any URL, not just GitHub