Netdata: RPM Packaging Discussion

Created on 1 Dec 2016 · 59Comments · Source: netdata/netdata

Context: https://www.reddit.com/r/linuxadmin/comments/5fkxan/netdata_the_cli_killer_for_linux_performance/dal7ks6/?context=1

Packages: https://build.opensuse.org/package/show/home:breed808:netdata/netdata

Currently, I've got working packages for Fedora 24/25, OpenSUSE 42.1 (semi-working build for OpenSUSE 42.2), and CentOS 7. Will look into building a package for CentOS 6 due to the number of comments in the above thread requesting it.

Can someone have a look at the directory structure of the package(s) and let me know if anything needs moving? Also, the packages are shipping with an empty /etc/netdata/netdata.conf file, would it be preferable to ship with a default config file?

Other than that, let me know what is needed, be it tips on RPM packaging, requests for builds, etc.

EDIT: Just noticed the specfile _already committed to the project_. Would have saved myself a lot of time and effort by looking at this first. Will have a look at the project's spec file and get some packages building from it.

arepackaging prioritlow

Source

breed808

Most helpful comment

First thanks all the team for the amazing job.

I made a very simple CI job on gitlab in order to create RPM nightly packages:

https://gitlab.com/boube/netdata

The packages are published here:

https://cloudsmith.io/~adrian-boubeta/repos/netdata/packages/

Maybe this could be useful for anybody.

The process is fully automated, I don't touch the code at all. Feel free to suggest improvements or other repo providers.

aboubeta on 25 Mar 2019

👍2

All 59 comments

CentOS 6 build is done and appears to be working, fingers crossed.

breed808 on 1 Dec 2016

nice! Can you help me setup this to be automatically updated on every commit pushed to the main repo?
I have several demo sites where I can run things, but I don't really know where to start...

ktsaou on 4 Dec 2016

Yes, it certainly seems possible. My thinking is that when committing, you would also push commits to a remote build server which can then run a git-hook to initiate a build script similar to the example here.

Or, rather than pushing to a separate git repository, one could run a similar script to the above on a cron timer, sending a rebuild request as needed (every 15min or every hour, for example).

breed808 on 5 Dec 2016

nice!

So, to setup this you followed a series of steps. Can you identify these for me?

ktsaou on 5 Dec 2016

Well, I hadn't actually set this up yet. It's theoretical at the moment, but I could setup a proof-of-concept on a VPS and give that a whirl.

What I have done so far is use the Open Build Service to provide builds for the RPM distros, but it can also build packages for debian-based distros and Arch. You can follow the tutorial here for a quick introduction.

Essentially, you commit .spec files and tarballs to a 'project' in the OBS, and packages will be built. The OBS will automatically rebuild packages if dependencies are updated, and through scripts one can command it to build from a source repository on a timer.

breed808 on 5 Dec 2016

Yeah, I have seen this tutorial before, and tried it once, but I guess it is either too complex for me, or I didn't have the patience to follow it properly...

Can you help?

Do you want me to start a new VPS on digitalocean to set this up? Which distro?
I will need your public SSH key to give you access to it (send it via email to [email protected]).

(and I would appreciate if you track everything - so that I will be able to reproduce it - for example: I have a git repo which turns a debian system to a netdata demo site: https://github.com/firehol/netdata-demo-site, so we could follow a similar procedure for this too).

ktsaou on 5 Dec 2016

Sure, I can do that. To clarify, do you want me to setup a copy of what I've done on the Open Build Service (repository building stable releases), or what was mentioned earlier (repository building on a timer)?

The distro for the VPS could be OpenSUSE or Fedora, I've been able to use both distros with the OBS rather easily.

I've no problem with documenting what I do. I'll probably record the steps as a video as I do it and then use that to create a basic shell script.

I'll email my public key to you shortly.

breed808 on 6 Dec 2016

i have done it for fedora and centos on copr if you are interested

the spec file is not from me, i fork the project for netdata version 1.4

it work for me on fedora24 and centos 7 , i didn't test the rest

https://copr.fedorainfracloud.org/coprs/baoboa/netdata/

baoboa on 6 Dec 2016

nice! you are so nice!

I would really like to make these available to others. So, I need your help guys. I am totally ignorant of what these procedures involve.

So, I can either setup VPS for you on DigitalOcean to build the whole thing, or accept your instructions to set them up myself...

@breed808 I haven't received your public key yet. Did you send it?

ktsaou on 6 Dec 2016

@breed808 it was in spam. I found it. I am setting up the VPS now.
I see digitalocean does not have opensuse, but it has fedora. I am setting up fedora 25. If you need any other version let me know.

ktsaou on 6 Dec 2016

@breed808 your RSA ssh key is installed: ssh [email protected] with it.

ktsaou on 6 Dec 2016

@baoboa do you want me to setup a centos 7.2 for you?
I will need you public SSH key. Send it to [email protected]

ktsaou on 6 Dec 2016

@ktsaou thank you, I will login to the VPS after work and begin documenting the setup.

breed808 on 7 Dec 2016

@ktsaou

on copr there is the possibly to plug a github webhook
https://fedorahosted.org/copr/wiki/UserDocs#GitHubWebhooks

what's the goal of the VPS , just the build of package with webhook? create a demo site ?

baoboa on 7 Dec 2016

what's the goal of the VPS

orchestrate what it needs to be done.
If corp can do it without a VPS, this is even better.

Can you help me set this up, so that packages will be created automatically.

ktsaou on 7 Dec 2016

@ktsaou I have created a rough draft for the build script in the /root/ directory of the VPS.

Before I can run and test it, you will need to:

Create an openSUSE Build Service account at https://secure-www.novell.com/selfreg/jsp/createOpenSuseAccount.jsp if you don't already have one.
Log into the Build Service (https://build.opensuse.org) and choose the repositories to build against (I.E OpenSUSE, Fedora, CentOS).
Edit the setup_buildserver.sh script to change the HOME_PROJECT_NAME variable to your openSUSE Build Service login ID.

If you need assistance with the above just let me know.

Once that has been done you'll need to run the script, which will prompt you for your openSUSE Build Service credentials. If the script fails, I can test it if you're comfortable sending me the login details for the openSUSE Build account (encrypted if necessary, my GPG fingerprint is 0FD6206E5AFF0F7A).

breed808 on 7 Dec 2016

@breed808 perfect! thank you!
I am playing with it now.

ktsaou on 8 Dec 2016

I get this:

On hover it says "nothing provides systemd, libuuid, ..."
Any ideas?

ktsaou on 8 Dec 2016

Ok, I figured it out. These distros do not have these packages.
It seems it is building packages...

ktsaou on 8 Dec 2016

dependency problem ?
rhel-5 doesn't have systemd anyway

baoboa on 8 Dec 2016

@ktsaou, before I submit a pull request to update the spec file, I've got a couple questions:

Would you like me to set the extra dependencies (python, curl, bash, etc) as hard or soft dependencies?
It turns out the node.d directory was being created correctly in the git master branch, but not the 1.4.0 release. However, there are no configuration files in the /etc/netdata/node.d directory, only markdown documentation. Should these be moved to the /usr/share/doc/netdata directory?

breed808 on 21 Dec 2016

Would you like me to set the extra dependencies (python, curl, bash, etc) as hard or soft dependencies?

I think not. netdata can work without them, although external plugins (python, bash) and alarm notifications (bash, curl) may fail.

It turns out the node.d directory was being created correctly in the git master branch, but not the 1.4.0 release. However, there are no configuration files in the /etc/netdata/node.d directory, only markdown documentation. Should these be moved to the /usr/share/doc/netdata directory?

Well, my understanding is that the first thing people do when they need to configure something, is to open the configuration file in question. Since JSON files do not accept comments, it is impossible to have .conf files in these directories. So, I ended up adding markdown files that describe what can be configured.

I don't have a way to measure their effectiveness though.

The ideal solution would be to support YAML for node.js plugins too. Until this is done, I suggest to keep these files there.

ktsaou on 21 Dec 2016

Would you like me to set the extra dependencies (python, curl, bash, etc) as hard or soft dependencies?

I think not. netdata can work without them, although external plugins (python, bash) and alarm notifications (bash, curl) may fail.

Sorry, did you mean no to hard dependencies, soft dependencies, or dependencies at all?

breed808 on 22 Dec 2016

I meant no hard dependencies. i.e. allow the users to install it even without them.

ktsaou on 22 Dec 2016

Hey guys,

loving netdata, keep up the awesome work on it! I was wondering if you are familiar with the effing package manager (https://github.com/jordansissel/fpm/wiki). It should be fairly trivial to link this with a githook to create packages for various operating systems for each commit. I'd be happy to help with that.

Greg

geeww on 23 Dec 2016

Thanks! Seems interesting... Of course, if you want to help, please do and share your work with us...

ktsaou on 24 Dec 2016

why do you use http(not https!) pointing to your website instead of github in netdata.spec.in file?

hackruu on 2 Jan 2017

why do you use http(not https!) pointing to your website instead of github in netdata.spec.in file?

Where? I see it is right: https://github.com/firehol/netdata/blob/c73d3a2576417634e3cf3663fcec4378b353585a/netdata.spec.in#L36

ktsaou on 2 Jan 2017

i meant this one, and i expected github and https there.
Source0: http://firehol.org/download/netdata/releases/v@PACKAGE_VERSION@/%{name}-
@[email protected]

hackruu on 2 Jan 2017

@breed808 @ktsaou regarding the soft/hard dependency question, would it make sense to split out plugins into other packages that have hard dependencies to make them work? If you install a package you'd expect it to install everything you need to make all parts of it to work....

I.e:
netdata
netdata-alarms
netdata-plugins
netdata-plugin-

Similar way to how python modules are handled in Fedora?

Squirtukuk on 24 Jan 2017

If you install a package you'd expect it to install everything you need to make all parts of it to work....
Sorry @Squirtukuk but I disagree in this point. On linux system it seems very common that only the core part of a program will work.

For an example take a look at the git dependencies: https://www.archlinux.org/packages/extra/x86_64/git/.

Also pushing all plugins into an own package would be a lot work to maintain. Just take a look at the rate plugins are added.

simonnagl on 24 Jan 2017

@simonnagl interestingly, in fedora they are packaged separately (http://pkgs.fedoraproject.org/cgit/rpms/git.git/tree/git.spec), i.e. git and git-svn.

You could put the plugins and alarms into a "core" package like dnf does?

http://pkgs.fedoraproject.org/cgit/rpms/dnf.git/tree/dnf.spec
http://pkgs.fedoraproject.org/cgit/rpms/dnf-plugins-core.git/tree/dnf-plugins-core.spec
http://pkgs.fedoraproject.org/cgit/rpms/dnf-plugin-spacewalk.git/tree/dnf-plugin-spacewalk.spec

I'm just throwing ideas out there, I'm not sure If any one way in particular is "correct".

Squirtukuk on 24 Jan 2017

I agree with @Squirtukuk on this. It would be best to avoid soft dependencies on any package(s) as there is no standard for dealing with them between the RPM package managers and their associated front-ends.

Plugin packages could be added to the project spec file using a macro:

%define add_subpackage(n:) \
%package %{-n*} \
Summary:      %{-n*} plugin for netdata \
Requires:       %{name} =  %{version}-%{release} \
Requires:       %* \
%description %{-n*} \
Provides %{-n*} plugin for netdata monitoring service. \

%add_subpackage -n mysql python2-mysql

%files mysql
%{_netdata_plugin_dir}/mysql

This would increase complexity of the spec file, and add additional workload to the netdata team. I'm of the opinion that adding hard dependencies to the netdata package would be easiest.
If the plugins ever require bulky dependencies (E.G. Java runtime, mono environment, etc) they could be split into sub-packages on a case-by-case basis.

breed808 on 25 Jan 2017

Guys I agree. Splitting would be a solution from a packages and dependencies perspective.

However, since netdata plugins are able to detect the missing dependencies and silence themselves, I wonder what is the point of splitting them. We will just add a single python file to a package?

Can we have "virtual packages" that just install the dependencies and nothing more?

Let's see what I mean:

package|description
---------|------------
netdata|exactly what we have today
netdata-mysql|installs only the dependencies for the mysql plugins of netdata to be enabled
netdata-postgres|installs only the dependencies for the postgres plugins of netdata to be enabled
netdata-elasticsearch|as above for elasticsearch
netdata-isc-bind|as above for ISC bind
netdata-snmp|as above for SNMP

So, I don't see any value to split the files from the netdata package. Why to do that? The files are small already and provide a consistent way to distributing netdata.

Is there a way to have virtual packages (i.e. packages that install a README file and all the dependencies for the relative netdata plugins to work)?

ktsaou on 26 Jan 2017

It's a bit unorthodox, but definitely possible.
We could easily add subpackages with custom dependencies containing a README to the existing project spec file.

breed808 on 27 Jan 2017

And netdata-isc-dhcpd. Plugin needs py2-ipaddress module on python2

ilyam8 on 27 Jan 2017

Hey guys, looks like if your Linux distribution has rpm >= 4.12 then you can use weak dependencies. Looks like Suggests: would notify the user when they install netdata of suggested packages to enable other functionality. Might be worth a look.

el-psycho on 7 Apr 2017

Hey guys, looks like if your Linux distribution has rpm >= 4.12 then you can use weak dependencies. Looks like Suggests: would notify the user when they install netdata of suggested packages to enable other functionality. Might be worth a look.

nice!
@alonbl what do you think?

ktsaou on 7 Apr 2017

On 7 April 2017 at 02:48, Costa Tsaousis notifications@github.com wrote:

@alonbl what do you think?

It depends on how much backward compatibility you want to have to older systems.

alonbl on 7 Apr 2017

@alonbl @ktsaou You could also have echo commands in %post to show that information. I think that would work for most versions of rpm. For example by addding the following to the %post section:

%post
%{netdata_init_post}

echo Optional Dependencies:
echo '  lm_sensors: for hardware sensors monitoring.'
echo '  php-fpm: for PHP monitoring.'
echo '  hddtemp: for disk temperature monitoring.'
echo '  smartmontools: for disk S.M.A.R.T. monitoring.'
echo '  python2-mysql: for python 2 mysql monitoring.'
echo '  python3-mysql: for python 3 mysql monitoring.'
echo '  python2-psycopg2: for python 2 PostgreSQL monitoring.'
echo '  python3-psycopg2: for python 3 PostgreSQL monitoring.'

The rpm installation looks like this using dnf/yum:

sudo dnf install netdata-1.6.0-1.fc25.x86_64.rpm 
Last metadata expiration check: 0:14:27 ago on Thu May  4 15:45:48 2017.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                         Arch                                           Version                                                 Repository                                            Size
===================================================================================================================================================================================================================
Installing:
 netdata                                         x86_64                                         1.6.0-1.fc25                                            @commandline                                         1.9 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 1.9 M
Installed size: 6.2 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : netdata-1.6.0-1.fc25.x86_64                                                                                                                                                                    1/1 
Optional Dependencies:
  lm_sensors: for hardware sensors monitoring.
  php-fpm: for PHP monitoring.
  hddtemp: for disk temperature monitoring.
  smartmontools: for disk S.M.A.R.T. monitoring.
  python2-mysql: for python 2 mysql monitoring.
  python3-mysql: for python 3 mysql monitoring.
  python2-psycopg2: for python 2 PostgreSQL monitoring.
  python3-psycopg2: for python 3 PostgreSQL monitoring.
  Verifying   : netdata-1.6.0-1.fc25.x86_64                                                                                                                                                                    1/1 

Installed:
  netdata.x86_64 1.6.0-1.fc25

Or like this using just the rpm command:

sudo rpm -i netdata-1.6.0-1.fc25.x86_64.rpm 
Optional Dependencies:
  lm_sensors: for hardware sensors monitoring.
  php-fpm: for PHP monitoring.
  hddtemp: for disk temperature monitoring.
  smartmontools: for disk S.M.A.R.T. monitoring.
  python2-mysql: for python 2 mysql monitoring.
  python3-mysql: for python 3 mysql monitoring.
  python2-psycopg2: for python 2 PostgreSQL monitoring.
  python3-psycopg2: for python 3 PostgreSQL monitoring.

el-psycho on 5 May 2017

nice!
@alonbl what do you think? I guess these will not harm...

ktsaou on 5 May 2017

highly non standard. if that is so important, probably better to have conditionals and use the standard methods, or to support rhel-7 and above.

alonbl on 5 May 2017

highly non standard. if that is so important, probably better to have conditionals and use the standard methods, or to support rhel-7 and above.

Thanks!

The problem with supporting rhel-7+ is that it limits netdata usability. We should support all currently active distros and rhel-6 is still active and supported by redhat.

I agree this is non-standard, but it looks better than today.

So, the best solution is to do it right with conditionals, as @alonbl suggests.

Until then, I suggest to add this comment/note.

Do we agree this is the best solution from a user experience perspective?
Looking forward for a PR in either direction.

ktsaou on 5 May 2017

From a user experience perspective, reading the proposed output above, I can't tell if that message means those optional dependencies were installed or if I'm just being told they exist.

They should not be installed without warning/confirmation, so I'm fairly sure it's just informational. But the message should indicate that more clearly, since in context, it comes right after the "installing" line and appears the optional dependencies are being installed as well.

mikeweilgart on 26 Aug 2017

hello

I am a little bit out of touch on this discussion but i finally manage do to get the automatized build working (with github webhook) for redhat family distro on fedora copr, the only limitation is for the repo to contain a working spec file
https://github.com/baoboa/netdata/blob/master/packaging/redhat/fedora/netdata.spec
On release (webhook) i got an automatized rebuild for all fedora version + centos 6/7
https://copr.fedorainfracloud.org/coprs/baoboa/netdata/
I even got a badge

I hope this could be of interest for you and a copr account linked to the upstream dev will make the repo a little bit more official.

baoboa on 20 Feb 2018

I was register netdata/netdata on packagecloud.io.
I will invite Costa as contributor (or transfer this account), may be this is good start for EL repo.

k0ste on 29 Mar 2018

👍1

@k0ste this is great! I think we could build around it and provide official netdata repository for all systems not only redhat family. Do you have some more experience with using packagecloud?