Netdata: Build packaging for debian

[titpetric]

Create a debian/ structure to build .deb packages for debian/ubuntu, along with supporting scripts (/etc/init.d/netdata,...). Building a deb package also means a smaller Docker image with just the deb package and requirements installed.

[ktsaou]

@alonbl can you help?

[alonbl]

Hi,
Unfortunately, no. This should be done by someone that actually use debian, long time since I packaged for debian. I know that ubuntu has some automation for most of the process.
BTW: even for gentoo/redhat you do not have proper init.d scripts etc... this should be completed regardless.
Alon

[ktsaou]

Thanks!
Can you give me a proper skeleton for such init scripts?
I know distros are including function libraries, etc. What is the proper way of doing it?

[alonbl]

You should provide systemd units for redhat and debian based. And OpenRC for Gentoo based. Best is to look at some examples, your init script should be very simple... so for systemd and openrc you use the no-fork mode and let the systemd or start-stop-daemon control it.

For gentoo for example replace xxx with whatever you need:

pidfile="/var/run/xxx.pid"
command="/usr/sbin/xxx"
command_args="${XXX_EXTRA_ARGS} start"
command_background="yes"
start_stop_daemon_args="--user xxx:xxx"

depend() {
    use logger
    need net
    need postgresql
}

start_pre() {
    mkdir -p /var/cache/xxxx
    chown xxx:xxx /var/cache/xxx
}

[ktsaou]

openrc added, #50

[ktsaou]

systemd service added, #51

[daveloyall]

https://bugs.debian.org/819661

You may post there simply by sending email to [email protected]. Keep in mind that there 3.8 thousand other potential packages in that same queue. See https://www.debian.org/devel/wnpp/

[ktsaou]

thanks @daveloyall !
Let me know if I can help somehow.

[HanXHX]

Hi,

I'm working on Debian packaging (https://github.com/HanXHX/netdata/tree/debian). It's not ready yet. I'll PR in few days.

Cheers,

HanXHX

[ABeaujet]

@HanXHX
Hi, would this bring proper compatibility for checkinstall ?

[HanXHX]

@ABeaujet : nop! checkinstall is a fast and dirty way to create deb package (useful for testing). It's only a wrapper for configure/make/make install.

[lhw]

@HanXHX I intented to maintain this in debian. Do you want to do a collaboration on this? debian also has a team (collab-maint) for this purpose.
In which case I would take ownership of @daveloyall bug and change it to a ITP.

Some modifications need to be done before it can be uploaded though. Many of the javascript libraries have corresponding debian packages, same with fonts. But that should be a non-issue.

[fabfurnari]

I can collaborate on debian packaging too, if you need help

My fork provides an already functioning debian/ fodler for source packaging, although not refined and lintian-correct

[mcnewton]

Hi,

Just seen this issue. I've made a basic Debian packaging this afternoon and done PR #132.

It's working fine for me, though a small number of outstanding lintian issues:

• init script for wheezy
• lots of .min.js included with no source files (some may be in other Debian packages I guess, which would require dependencies)
• couple of fonts included which are in other Debian packages
• setuid apps plugin (I think install non-suid and include comment in README to enable this if you dare)
• no man page for netdata binary

Hope it's helpful, as a starting point at least?

Tested only on jessie so far. If I get around to doing an init script I'll sbuild it for wheezy and check it over there too.

Cheers,
Matthew

[FedericoCeratto]

@HanXHX @lhw @daveloyall I can also help co-maintain the package if it's hosted on collab-maint
UPDATE: the official packaging repo is now at https://salsa.debian.org/debian/netdata

[daveloyall]

@FedericoCeratto does that mean mirroring this repo to https://anonscm.debian.org/git/collab-maint/ ?

FYI: I'm not familiar with Alioth nor collab-maint nor the Debian organization (except as a user).

[lhw]

Basically. Normally you don't import the whole history of repository but import snapshots/releases every now and then to an upstream branch and add the debian/ dir on top to master. With additional steps in between e.g. if the upstream contains copyrighted/non-free files. It's extremely easily manageable thanks to tools like git-buildpackage for importing, patching and building the debian package.

Alioth is just the code hosting for packages on debian. And collab-maint is a special "team" on the site for collaborative repositories. It is fairly easy to add new users to it even if they are not debian maintainers or developers. An actual upload to the debian ftps can only be done by a developer or an authorized maintainer (uploader tag) though.

[TheeMahn]

I wrote a bash script to package netdata for Ultimate Edition Linux and it works. I assume it will also work with Ubuntu / Debian. I will link and post the source for it on our forum after I get off work. I am going to make it an integrated part of Ultimate Edition.

Thanks for such an awesome proggie,

TheeMahn

[TheeMahn]

As promised Please file bugs for it there as far as packaging.

[ktsaou]

As promised Please file bugs for it there as far as packaging.

Nice!

[ktsaou]

Make an update - new charts, a lot of them... :)

[lhw]

I took ownership of the RFP bug on debian and added the collab-maint repo for netdata (currently empty).
My suggestion now is that we base the package for debian on the release versions of netdata
and work off of the already good contributions from @mcnewton.
If you want to join the effort please drop me a line.

[ktsaou]

I am here to help in anything you will need.
I plan to release 1.0.1 this weekend. It was to be released the last weekend, but we couldn't make it.

[TheeMahn]

I have repackaged it as 1.0.0, I would appreciate it if you could make the executable have a version support option ie

netdata --version

I have git pulled it and re-built the package, added chroot detection, so I can built it into Ultimate Edition. as can be seen in the 4/20 edit on our forum Please read the bottom of the post to see the errors that were picked up in building it.

[ktsaou]

I guess you mean these:

E: netdata: arch-independent-package-contains-binary-or-object usr/libexec/netdata/plugins.d/apps.plugin
E: netdata: unstripped-binary-or-object usr/libexec/netdata/plugins.d/apps.plugin
E: netdata: arch-independent-package-contains-binary-or-object usr/sbin/netdata
E: netdata: unstripped-binary-or-object usr/sbin/netdata
E: netdata: changelog-file-missing-in-native-package
E: netdata: no-copyright-file
E: netdata: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/netdata
E: netdata: init.d-script-does-not-implement-required-option etc/init.d/netdata force-reload
E: netdata: shell-script-fails-syntax-check etc/init.d/netdata
E: netdata: arch-independent-package-contains-binary-or-object usr/libexec/netdata/plugins.d/apps.plugin
E: netdata: arch-independent-package-contains-binary-or-object usr/sbin/netdata
E: netdata: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/netdata
E: netdata: init.d-script-does-not-implement-required-option etc/init.d/netdata force-reload
E: netdata: shell-script-fails-syntax-check etc/init.d/netdata
E: netdata: arch-independent-package-contains-binary-or-object usr/libexec/netdata/plugins.d/apps.plugin
E: netdata: arch-independent-package-contains-binary-or-object usr/sbin/netdata
E: netdata: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/netdata
E: netdata: init.d-script-does-not-implement-required-option etc/init.d/netdata force-reload
E: netdata: shell-script-fails-syntax-check etc/init.d/netdata

I need help guys... I am not sure I understand what are these...

[mcnewton]

@ktsaou they are the lintian checks and look like they are pretty much all problems with packaging of some sort. Some need fixing and some probably need overriding.

@TheeMahn I'm not sure how your packaging script works - maybe you could now copy contrib/debian/ to debian/ and try to see if that works on Ultimate Edition? I don't know how close it is to Debian though so maybe that's not practical?

[lhw]

@ktsaou For the locally build package you can ignore most of those errors. They are meant for packages actually going into debian. I'll deal with them in my package but for your build they are irrelevant.

[ktsaou]

For the locally build package you can ignore most of those errors.

ok thanks! I you need something from me, let me know.

[FedericoCeratto]

I'm populating the repository in https://anonscm.debian.org/cgit/collab-maint/netdata.git - it's currently building and installing. The javascript stuff needs to be de-vendorized. Ping to @lhw
UPDATE: the official packaging repo is now at https://salsa.debian.org/debian/netdata

[ktsaou]

Thanks!

The javascript stuff needs to be de-vendorized.

Hm... I am not sure I understand what this means...

[lhw]

@FedericoCeratto I already added some stuff to it and updated the upstream release to 1.0.0. I'll try the data/binary decoupling next.

@ktsaou some of the Javascript libraries are available as debian packages and it's the preferred method to link those in instead of the vendor supplied versions. In case of security issues and the like only one package needs updating.

[ktsaou]

ome of the Javascript libraries are available as debian packages and it's the preferred method to link those in instead of the vendor supplied versions. In case of security issues and the like only one package needs updating.

ok. Which ones? How this can be implemented (the files will be linked to the location netdata expects them?) How debian is going to handle updates on the chart libraries that will break netdata?

[mcnewton]

@ktsaou you shouldn't generally need to worry about that for official Debian packages as that is checked by the maintainer. But it's why when I did the debian package to be bundled with Netdata I included the js libraries so you won't worry about it, and any package built from the Netdata source will be good as it includes the bundled libraries.

[lhw]

@ktsaou An update of a debian javascript package can indeed cause issues with compatibility. It all requires communication with other maintainers. But they can see what depends on their package and shoot a quick warning in case something will break on an update.

And yes we would add the packages as dependencies and link their library paths into the netdata package where netdata expects them to be.

[ktsaou]

Thanks @lhw !
ok. Let's see how this will work...

[ktsaou]

@FedericoCeratto the latest version of netdata v1.4 states the version of all javascript libraries it depends.

netdata depends on a very specific version of dygraphs. The latest release of it (v1.1.1) does not support the legends properly (they are missing the hooks for updating the legends on hover), while their git version has a few issues: https://github.com/danvk/dygraphs/issues/790 https://github.com/danvk/dygraphs/issues/781

So, I ended up using commit: https://github.com/danvk/dygraphs/commit/dd74404fac69f23183ff8e540a018ca35bab4f7a

Other than the above, netdata is using the latest version of all javascript libraries: https://github.com/firehol/netdata/blob/6be563674218d92af7904c11cc368834d26e8455/web/dashboard.js#L111-L123

I see I have included a non-released version of easypiecharts and gauge.js but I think their latest released version is fine too.

[ktsaou]

And... jquery 3.x should not be used with netdata. It works for a few chart libraries, but not for others.
The latest 1.x and 2.x versions are fine.

[lhw]

netdata is currently circling in the debian NEW queue. I'll update to 1.4 as soon as it in unstable.
And holy hell javascript dependencies are a nightmare. I probably have to tighten all the dependency versions to minor releases to ensure it keeps working.

[ktsaou]

And holy hell javascript dependencies are a nightmare.

Yeap they are. It took me 2 full days to find them out...

[FedericoCeratto]

netdata is now in Debian Unstable! https://packages.debian.org/sid/netdata
It should reach Debian Testing in some days.

[lhw]

Just a short update. While netdata is now in debian unstable it will not be in the next stable release of debian as the soft-freeze occurred before the package made it into testing. It will be present in the stable release after.
I also changed the package structure to be easily backported. Which is probably a good solution for a software which still has major changes in short time frames. This way everyone can get the newest updates faster.
v1.5 is currently in unstable

[tonybolzan]

Why sid/netdata depends of libjs-bootstrap and fonts-font-awesome, netdata-data already have this dependencies.
/usr/share/netdata/web/css/bootstrap-3.3.7.css
/usr/share/netdata/web/fonts/

Without these dependencies it works fine in Debian 8 without modifications.

[arthurlogilab]

@tonybolzan probably because the package should probably not contain that code which is also available in other packages, but the maintainer hasn't got round to stripping netdata from those files.

By the way : congrats to all contributors on the debian packaging, really happy to see it in sid. Got a jessie backports compiled in no time for our internal infrastructure.

[FedericoCeratto]

@arthurlogilab Thanks. There's a backport of 1.6.0 for Jessie pending approval and I plan to do a backport for Stretch as well.

[LeeNX]

Means we will see netdata on the Raspberry Pis without needing build tool set ... Nice!

Now to get into Ubuntu ... ;-)

[FedericoCeratto]

@LeeNX Ubuntu automatically pulls most of its packages from Debian: https://packages.ubuntu.com/source/artful/netdata

[LeeNX]

@FedericoCeratto - I see that in zesty and artful, but not in trusty and xenial, which are my current running platforms. If I was not a lazy SysAdmin, I would setup a PPA to build for trusty and xenial ... ;-)

Thanks

[tonybolzan]

@arthurlogilab I believe that Netdata does not need a dependency (deb) of some Kb unique to the UI. A self contained packet should be much more portable in this case. (IMHO)

[zifnab06]

I don't actually have time (or the motivation) to maintain this going forward, but if anyone else would like to, this patch will give you a basic .deb. You can run debuild -us -uc to get a deb out. There are a few bugs (groups aren't automatically joined, for example), but works well enough for my needs.

It's not nearly as good as the previously built version, but I somehow missed this thread while I was trying to get it set up on a few different boxes...

[linview]

uuuuyu nu

[egeneralov]

Hello everybody.
I will try to write jenkins pipeline (can be multibranch, if author will add one Jenkinsfile) on this weekend. I'll need review for package. Can anybody help with feedback?

P.S.: package for what debian versions? Possible can support wheezy, jessie, stretch and sid.

[ktsaou]

I am available to help, but you will need to guide me (ie be specific about what you need and how should I do it). I can also provide a checklist (or better I can write a wiki page with all the things that need to be checked).

netdata runs everywhere, so the most distros supported the better. Can you also build for ubuntu?

[egeneralov]

Okay, i will do.

[LeeNX]

Willing too test Ubuntu PPA installs for the last three LTS releases on 64bit. Can also setup some local VM's if very basic testing is also needed. Basically begging for Ubuntu packages ... ;-)

[FedericoCeratto]

Netdata 1.7 is now in Debian Unstable. It will be backported to Stable aka "Stretch" as well.

[hahayidu]

Hi,

@FedericoCeratto @lhw

I'am not familiar with debian-backport policy. Is it posible to follow upstream version in Debian [stretch-backports] ?

[FedericoCeratto]

@hahayidu it is as long as all the required dependencies are there. Unfortunately Netdata comes with a lot of js libraries that make backporting difficult. Here are the current versions of Netdata across the archives: https://tracker.debian.org/pkg/netdata

[FooBarWidget]

I've backported Ubuntu 18.04's netdata package to Ubuntu 16.04. You can find my work here: https://github.com/phusion/netdata-ubuntu-16.04

[paulfantom]

To all interested parties: I would like to unify how netdata packages are versioned, more in #5044

[daniel-baumann]

Hi,

just curious why this hasn't been closed yet - netdata is in debian since quite a while, or is this about providing packages for debian yourself?

Regards,
Daniel

[paulfantom]

This is about providing packages by us

[daniel-baumann]

Hi,

what's the reason again for doing that? Just trying to find out why/where/how there's need to duplicate work or where we can improve on our side..

Regards,
Daniel

[paulfantom]

The reason is to provide nightly/edge channel with binary packages.

[paulkatsoulakis]

Closing this, as we will be attempting to nail this through #5963