Libretime: Enable install script to auto-detect distribution and release version

[bburton]

Issue

Instead of having to specify the --distribution and --release options to the install script, the script should be able to reliably detect which distribution and release it's running on and work appropriately. This detection must work on all supported and unsupported distribution and release combinations.

Assumptions

The assumption is the following distribution and release combinations are (for the install script):

• Ubuntu Trusty (14.04 LTS)
• Ubuntu Xenial (16.04 LTS)
• Debian Jessie (8.7)
• Debian Wheezy (7.11)
• CentoOS 7

As LibreTime is rather complex with a significant number of dependencies, it's better to limit to known distro and release combinations that have been tested.

Proposed solution

All of the above distribution and release combinations support the standard file /etc/os-release. This file can be sourced into any Bourne shell script to import the distribution and version configuration. There are several standard variables and some non-standard ones. The detection logic would reference the following:

• ID - distribution id - one of "ubuntu", "debian" or "centos" (for supported OS's)
• VERSION_ID - distribution version id (optional but available) - numeric version, eg. 14.04, 16.04, 7, etc.
• VERSION_CODENAME - distribution codename (optional), eg. "xenial", etc.
• PRETTY_NAME - descriptive name of the release, eg. "Ubuntu 16.04.2 LTS"

The combination of ID and VERSION_ID or VERSION_CODENAME can be used to reliably detect the distribution and release. VERSION_ID is optional but all supported distros set it to the numeric version. VERSION_CODENAME is also optional with the only supported distro that sets is Xenial. For all other distros, VERSION_ID should be used.

Proof of concept

Attached is a Bash script that detects all five releases above. Any other distribution and version combination is considered as unsupported.

os-release-test.sh.txt

This script sets the following variables:

• $dist variable to the lower case name of the distribution. It just so happens that the $ID variable already supplies the correct lower-case value.
• $code variable to the lower case code name of the distribution version, eg. "xenial", "jessie", etc.

In addition, the script also sets a boolean variable for each distribution and each distribution and release combination. This makes the testing conditional syntax much cleaner and also ensures that if there's a typo in any test that the script will fail with a syntax error instead of performing an incorrect test that always evaluates to false. For instance:

if $is_ubuntu_dist; then ...
if $is_ubuntu_xenial; then ...
if $is_centos_7; then ...
if $is_ubuntu_dist || $is_debian_dist; then ...
elif $is_centos_dist; then ...

Single line conditional statements are easy and clean:

$is_ubuntu_dist && echo "This is some version of an Ubuntu distro"
$is_debian_jessie && echo "This is Debian Jessie"
$is_ubuntu_dist || $is_debian_dist && echo "This is some version of Ubuntu or Debian"

The above approach works reliably because all the test variables are initially set to false and then only set to true when appropriate. The result is each variable has the value of true or false and no other value. Because of this, these variables can be used directly in conditions without using brackets that would otherwise be required.

Note that is preferable for the install script to test for specific features when possible instead of making too many assumptions about what configuration is used for a specific disto and version. For instance, a feature test for the init system type would detect if systemd or Upstart are active irrespective of the distro and version as it's possible for some distros to be configured to support Upstart even if systemd is the default.

Implementation Questions and Assumptions

Before implementing a patch to the install script, some issues need to be clarified:

• The core part of this script would be located after all the option parsing at around line 312 after the LibreTime ASCII art logo is displayed.
• The options --distribution and --release would no longer be needed and would be removed. I can't see any reason to have the ability to override the distro and version detection.
• Using the boolean variables for testing is a different stylistic approach which can be used or not in place of or in addition to $dist and $code variables. I think the change would be well worth it and would be willing to submit a PR once the detection logic is merged in.

Related

Have written feature detection logic for init system detection which will be submitted separately.

[ghost]

Having run support depts. My 2 cents is: choose one or max 2 supported variants - and dig in solid or our lives will be spent answering incompatibility questions. Most folks put AT on VPS or dedicated to be a stand alone station. So, when they choose the image to load up.. make it easy for them (and us).
I suggest Ubuntu Trusty & Xenial and nothing else at all.

[bburton]

The list of five distribution and release combinations was determined by seeing what was already available in the repository in one form or another. It wasn't meant to imply it's the final supported list.

There has been some discussion as part of PR #157 (Ubuntu 16.04 installer update):

1. Packages would be created for each supported distro and release combination.
2. Install script would be for development and testing.

However, in general, I agree with you as it would be preferable to limit the supported releases do to the complexity of all the dependencies. Just because the upstream version supported a release doesn't mean LibreTime needs to support it.

[ghost]

"Just because the upstream version supported a release doesn't mean LibreTime needs to support it."
Quite :)

[Lapotor]

• The options --distribution and --release would no longer be needed and would be removed. I can't see any reason to have the ability to override the distro and version detection.

This should be still availabled in some way that if you have an other not supported dist or subversion but you know that it can done the same way as for example Debian Jessie then you can say for example ./install --dist="Debian-Wheezy" and so you are still able to install it without a specific install script for your disto.

[bburton]

@Lapotor It's a minor change to allow the --distribution and --release options to override the auto-detection. However, it's a bit dangerous to support overriding unless someone knows the intimate details of every aspect of the install to know it won't install files in the wrong places, etc.

There should be some minimal check for the --distribution option so it's not possible to specify an option that's completely wrong such as "debian" on a CentOS system, etc.

If there was a --dry-run option, that would make it easier to test against any unsupported distribution.

I can think of a simple change to the place where the package file name is done to make it easy to expand the distribution selection case statement so the package file name can be shared among more than one distribution and release. For instance, it would be entirely reasonable to support installing on RedHat Enterprise Linux by using the CentOS configuration.

If feature detection is implemented well enough so the install is probing the system for each feature testing against those features instead of the distribution and release, then the install becomes significantly more robust and an override may only be needed to select the best package list.

[Lapotor]

I think the same but there should be a option to do it.
Like ./install --dist="Ubuntu-Xenial" only work when there is a second operator like ./install --dist="Ubuntu-Xenial" --trust-me-i-know-what-i-am-doing or something similar.

[Lapotor]

May we can build it, that you can add your installer script for your dist but when you execute it you get an hint that is not a offical supported installer script ?

[hairmare]

I think the reason that legacy upstream started using Ubuntu rather than the originally recommended Debian was due to upstarts auto-restart feature. Now that everyone has switched to systemd that reason doesn't really apply anymore.

I believe legacy upstream was right with their initial recommendation. IMHO Debian is simply the better choice right now. Since Ubuntu is based on Debian, things like package bump requests needs to go through Debian anyway (ie. the upcoming liquidsoap 1.3.0 release).

There are also quite a bit of unmaintained packages in Debian and Ubuntu we rely on (with liquidsoap being the prime example and silan affected as well). So far I did an NMU for silan to no effect. I'm still hoping that someone from the community is willing to take over Debian maintainership on what we need.

When I did the CentOS install I had to package lots of things. That CentOS had none of them per default resulted in loads of work but actually made the process easier as I didn't have to bump any distro packages. I think CentOS support is worth it since it opens the door for lots of RHEL shops, we also use it at our station exclusively.

I'm not sure if we really need a check if the wrong distro is specified on the cli. I think most beginners will just copy paste what the docs say and experienced folks will (or should) read the installer script before trusting it with root access.

[Lapotor]

I'm not sure if we really need a check if the wrong distro is specified on the cli. I think most beginners will just copy paste what the docs say and experienced folks will (or should) read the installer script before trusting it with root access.

Sure but it is better if the beginners get checked by the installer if they miss a thing then a installer which check it will help them.

Yeah Debian/Ubuntu will be the best dist for that but we should decide if we change everything to Debian 8/Ubuntu 16.04 because that will be the longest supported dist atm in Linux.

[Lapotor]

When I did the CentOS install I had to package lots of things. That CentOS had none of them per default resulted in loads of work but actually made the process easier as I didn't have to bump any distro packages. I think CentOS support is worth it since it opens the door for lots of RHEL shops, we also use it at our station exclusively.

But i recommand to make at first One (or two very similar) Dist(s) and than we can start with centos when it is that much work.

[hairmare]

... then we can start with centos when it is that much work.

I already did all of the grunt work when I was still installing legacy upstream and have ported that as well. I've been holding off publishing it since I want Debian to be at a similar state when we launch package based installs. I think people should have a choice rather than CentOS being the only version of LibreTime with proper packages.

[Lapotor]

@hairmare so we should start with Debian/Ubuntu at first ?

[hairmare]

@hairmare so we should start with Debian/Ubuntu at first ?

Absolutely! @paddatrapper has already started in #132 with debian packaging scripts being in LibreTime/libretime-debian-packaging.

I assume we will need a couple of releases until everything works out (and is automated).

[Lapotor]

yeah sure will try a bit with the script and impement automation if i get everything ready...

[bburton]

I've made quite a bit of progress on updating the install to add auto-detection of the distribution and release. In addition, have started to implement some feature detection to eliminate most assumptions based on distro and release.

• Sources /etc/os-release and outputs what was detected. If that file is not found, it bails if either --distribution or --release was not specified.
• Whether the user specifies either --distribution or --release options or not, outputs what is being used.
• Adds logic to detect init system by looking at the process where pid=1 following any symlinks and then looking in what package that process is found. Currently, the script does not use this information yet. All the installing, enabling, starting and stopping of services needs to have the logic updated.
• When dependency packages need to be installed, only does so for Debian-based distros (/usr/bin/apt-get exists) as there's not yet a package list file for CentOS. Also checks to ensure the package list exists and if not, bails.

Will try to get what I have checked in and pushed to a branch sometime this weekend although it will still be a work-in-progress.

[bburton]

@paddatrapper Just found the legacy upstream packaging at https://github.com/Airtime/airtime-packaging. It supports Ubuntu Trusty and some older Debian/Ubuntu releases.

[Lapotor]

I planned to start this weekend too :-D
So I wait for your branch to work with it and help you there.
But what do you think to build it in a way that you can add installation scripts from the community for additional dists and releases?
But if it is not an "offical" installer you get an message and get asked if you really want it.

For example:
The script get the dist and release and check if there is a file in install/ named like install-<dist>-<version_id>.sh if yes it include this file and run it ...
What you think ?
So we did not create for all dists and releases a install script but the community can support more dists and releases .

[bburton]

@Lapotor The problem with having multiple install scripts, is there's significant overlap in many cases such as between certain Debian and Ubuntu releases. Then, there's a series of variations between releases for the same distro. Then, there are certain things in common such as recent distros all use systemd instead of Upstart. As my time is limited, I was just trying to improve the existing install without rewriting it from scratch.

I did spend a short amount of time looking for an installation framework but didn't find much.

So I'm working on making the install test for certain features and then perform that part of the install for that feature based on what was detected. For instance, rather than assuming where the Apache root folder is located based on the distro, I have the install run apache2 -V or httpd -V as appropriate and get the root folder directly from how the executable was compiled.

Also, the install doesn't do certain things yet that will likely be problematic unless addressed:

• The install should detect if there has been a previous install via package installer either of LibreTime or Airtime and take some appropriate action.
• Conversely, the package installs should be able to detect a manual install by script and take some appropriate action.
• The install should probably record the state of certain options or features detected so they can be used by the uninstall script.

Ultimately, though, most of the install effort should be in creating and testing packages for the supported distros. The install would be primarily for development and testing new distros and releases.

[paddatrapper]

@bburton Thanks. Will definitely make use of some of the old packaging

[bburton]

Just pushed a commit for the installer with major improvements in portability between distros and releases. Please checkout the branch https://github.com/bburton/libretime/tree/install-auto-detect-distro-init. The commit message goes into great detail with all the changes made.

Have done successful install testing on Trusty, Xenial and Jessie and after performing the web install, all services start. With CentOS, could not test as much as my browser can't connect to localhost:8080 displaying: _Forbidden - You don't have permission to access / on this server._ With Wheezy, the install fails for unrelated reasons when installing some Python dependency.

The biggest change is how the different init systems are installed and deployed. The Python setup.py for each service installs both System V init and Upstart files but can't do any filtering of these files so the install script used to patch them afterward which is less than optimal. So I disabled the init file install by passing the ---no-init-script option to each setup.py and implemented the install in the shell function systemInitInstall() which only installs the necessary files for the current init system type and also performs the appropriate filtering when required. Since this is now done centrally, it's much easier to maintain instead of having to make changes to all the setup.py scripts.

Trusty is the only supported distro using Upstart. To make it easier for the install in the systemInitInstall() function, I created an Upstart config for airtime-celery so it's no longer using the System V scripts.

For Xenial, Jessie and CentOS which use systemd, after the install completes and the web install is performed, use the command:

systemctl restart airtime_analyzer airtime-celery airtime-liquidsoap airtime-playout 

to restart all services instead of the individual service commands. For Xenial and Jessie, System V init compatibility packages are installed (see respective .apt file) but they should not be needed based on these changes to the install. I did not want to remove them until more thorough tested had been done to ensure they are not required.

After that, a lot of changes were made to make the install more resilient by making fewer or no assumptions (when possible) by testing for specific versions of an executable or if it exists or not instead of making as many assumptions based on the distro and release. This should make it easier to adapt the install to unknown distros and releases as well as custom builds of certain software on supported distros and releases.

TODO - There is still much to do to clean up what's left. Some of these items should continue in this branch and others in different feature branches:

• Merge more of the centos.sh script into the install such as with installing dependencies.
• Move systemd unit files into the install folder for each Python service and update systemInitInstall() to look for them there.
• Fix the text in the web install with the service commands to start all services. Maybe there should be a script libretime-services that can start, stop or restart all the services using the appropriate init system commands such as systemctl, service or invoke-rc.d.
• More robust upgrade vs. new install detection for Apache especially.
• Update uninstall
• Detect if airtime or libretime distro package has been previously installed and abort.
• Improve detection so just running ./install with no options does the right thing whether fresh install or upgrade.
• Output simple install configuration and manifest as /etc/airtime/airtime-install.cfg which can be used by uninstall and upgrades when re-running install.
• Separate build from the install so the build can be done once and the install just uses the files already built.

[ghost]

I thought putting it in a Docker container was supposed to have solved all this?

[paddatrapper]

Not everyone wants to run LibreTime from within a Docker container. (Personally I prefer LXC for example), having distro aware install scripts and distro packages makes installation easy, no matter what the user's intended destination platform is

[ghost]

If you have a choice of one - Docker - and that is it, the support issues drop like a rock.

[hairmare]

@bburton I'm testing to see if I can figure out what is going on with CentOS (and I'll also help with merging the centos script as it was originally my hack anyway). I'll let you know what I figure out.

IMO the systemd change will make Ubuntu Xenial and Debian Jessie much easier to run and that alone is probably worth the merge :)

@paddatrapper Did you see that legacy upstream was also using lxc at some point. I'm more in the docker camp though.

[paddatrapper]

@hairmare I did, though last I looked at that was a while ago. Thanks for the link, it will come in handy when I get to setting up my new LibreTime instance at work

[hairmare]

I wasn't able to reproduce the Forbidden error with CentOS on 3179651decf962190b87280c1e4602188bb6b2db using a clean install from vagrant up centos so I'm not sure what's going on there.

I did have to do add a mkdir to the "Installing Log Files" part of the script and move the chmod on the dir slightly. The path to pypos logrotate file also needed fixing. I also created some more log dirs after the install, it looks like things are slightly out of order in install.

--- a/install
+++ b/install
@@ -891,8 +891,6 @@ verbose "...Done"

 verbose "\n * Installing airtime-celery..."
 loudCmd "python ${AIRTIMEROOT}/python_apps/airtime-celery/setup.py install --no-init-script"
-# Make the airtime log directory group-writable
-loudCmd "chmod 775 /var/log/airtime"
 # Create the Celery user
 if $is_centos_dist; then
     loudCmd "id celery 2>/dev/null || adduser --no-create-home -c 'LibreTime Celery' -r celery || true"
@@ -922,12 +920,17 @@ if [ ! -d /var/log/airtime ]; then

     verbose "\n * Copying logrotate files..."
     cp ${AIRTIMEROOT}/airtime_mvc/build/airtime-php.logrotate /etc/logrotate.d/airtime-php
-    cp ${AIRTIMEROOT}/python_apps/pypo/pypo/liquidsoap_scripts/airtime-liquidsoap.logrotate /etc/logrotate.d/airtime-liquidsoap
+    cp ${AIRTIMEROOT}/python_apps/pypo/liquidsoap/airtime-liquidsoap.logrotate /etc/logrotate.d/airtime-liquidsoap
+
+    verbose "\n * Creating /var/log/airtime..."
+    loudCmd "mkdir -p /var/log/airtime"
 fi

 verbose "\n * Setting permissions on /var/log/airtime..."
 chmod -R a+x /var/log/airtime
 chown -R ${web_user}:${web_user} /var/log/airtime/
+# Make the airtime log directory group-writable
+loudCmd "chmod 775 /var/log/airtime"

 verbose "\n * Setting permissions on /var/tmp/airtime..."
 chmod -R a+x /var/tmp/airtime

After running install before starting anything:

mkdir /var/log/airtime/pypo /var/log/airtime/pypo-liquidsoap/
chown apache /var/log/airtime/pypo-liquidsoap/ /var/log/airtime/pypo

I ran into #192 since the CentOS install in the vagrant file already has liquidsoap 1.3.0 but that's another matter. I was able to switch to the branch linked in #192 and re-run install on just pypo to fix that.

[hairmare]

Regarding the missing /var/log/airtime directory on CentOS. We could use tmpfiles.d to create dirs like /var/log/airtime/* and /var/tmp/airtime/show-recorder on distros that have systemd.

[hairmare]

I just realized that you can also skip installing std_err_override since it was made obsolete with #76 for our purposes (only media-monitor still needs it and it's slated for removal anyway). I'm having fun with liquidsoap 1.3.0 but will prepare a proper patch against your branch over the next few days.

[bburton]

@hairmare In doing further reading on systemd (which is completely new to me), it appears packages should install Unit files into /usr/lib/systemd/system as the files under /etc/systemd are reserved for local administrator customization. See the systemd.init man page under _Example 2. Overriding vendor settings_. If an administrator needs to override or extend any settings, this is easily done with systemctl edit <unit-name>. Then, systemctl cat <unit-name> shows a combined view of the original Unit plus overrides. The Unit files in /usr/lib/systemd/system can later be replaced and updated by a package update or installer without affecting the local overrides in /etc/systemd.

[hairmare]

@bburton The centos.sh installed the unit files to /etc/systemd/system since I wanted to keep the /usr/lib/systemd/system folder clean until it can be populated by proper packages.

[hairmare]

@bburton I just pushed https://github.com/radiorabe/libretime/commit/43eb777177b6edbfc8f9280ee8d6d0693e727d73 based on your branch with some small changes that help on clean installs. My changes are tested against Debian Jessie and CentOS with a manual fix for liquidsoap 1.3.0 on the latter (using freshly destroyed vagrant boxes in all cases). Feel free to merge my branch into yours so we can open a pr later.

I added mkdir calls to add dirs that were missing now that we are using --no-init-script everywhere. I also cleaned up the order the install was trying to do things in so it makes more sense.

Btw, you can't systemctl edit on Debian Jessie yet because systemd is an older version on it. If you would want to override a unit from the installer, say the liquidsoap unit, you can use a .d subdirs per unit that can contain multiple conf files like /etc/systemd/system/airtime-liquidsoap.service.d/override.conf.

[hairmare]

I did some testing with Ubuntu Xenial/16.04 and came to the conclusion that we need to downgrade to php5.6 for the time being (it's the only distro that already has php 7.0 in a stable release). I added the code to downgrade to my branch that is based on Bill's work at https://github.com/radiorabe/libretime/commit/eba625c7e813d8d80d44a27864afc0cdd20d5ef9 https://github.com/radiorabe/libretime/commit/3b3cec7bc2a18abfa72f721cd25a99321ec58ccd (edit: updated to actually use $is_ubuntu_xenial).

The php7 situation needs addressing at some point but I haven't found an easy solution so far.

[Lapotor]

@hairmare how far you are with the ppa ?
May I can help with the installer and the ppa adding.

[hairmare]

@Lapotor Sure, any help is appreciated. I actually haven't touched the debian packaging process since I have no idea of what I'm doing when I do Debian things. Maybe @paddatrapper can give us an update on the packaging situation?

[paddatrapper]

@Lapotor I haven't had a chance to start working on packaging, as I have had a lot of other things on my plate. This should hopefully die down in the next two months or so. The packaging repo is here. The current blockers is that liquidsoap packaging is orphaned (#192), so we need to decide what will happen there before we can package LT.

I wasn't intending to set up a PPA, rather just to upload to Debian repos and let it trickle down as releases happen and letting people use the install script in the meantime, but I'm not against a PPA if that is useful for people. Then we just need to be careful around package versioning in the official repos and the PPA.

[Lapotor]

My first work will be to update the installer to Xenial that this will work afterwards I can do the packaging maybe or the other installer thing where this issue is about.

[paddatrapper]

The other potentially sticky point is PHP 7, which is what is now default from Debian Stretch and Ubuntu 16.10 (I think). I haven't tested LT with PHP 7, but I am sure there will be a couple of bugs in it

[hairmare]

I haven't tested LT with PHP 7, but I am sure there will be a couple of bugs in it

That reminds me, I stomped a bad one last weekend and forgot to PR it: #226. The sucker took quite long to track down. With it heaps of stuff that was previously broken in PHP 7 should work fine. I only tested uploading and never got to testing playback. Plupload and analyzer worked which is already a win :)

[hairmare]

PHP 7, which is what is now default from Debian Stretch and Ubuntu 16.10 (I think)

I can confirm this. Ubuntu Xenial/16.10 has 5.6 backports but there are noone for Debian Stretch yet.

[hairmare]

@Lapotor I had some pending fixes for Xenials installer (ie. a php downgrade) in the branch I linked above. I need to retest this after we get #226 merged. If #226 really fixes the blocker issues with PHP7 we can probably (finally) get working on supporting modern distros. Once #226 is merged Xenial will probably install without hickups from 43eb777.

[Lapotor]

Yeah the two different PHP versions can be a problem May it can be helpful to define the PHP version and do two things for the two versions when there is no way to support both...

[hairmare]

I hope the remaining PHP 7 bugs aren't as hard to find as #226. We can use symfony/polyfill should we want to use some new php 7 features. So far I haven't found any real issues using static analysis with tools like sstalle/php7cc.

[hairmare]

I think there are still some things to clean up here so I bumped this to alpha.4 as I plan on tagging alpha.3 soonish.

[paddatrapper]

The installer now recognises the distro and version from /etc/os-release and PHP 7.2 support has been added (#670)

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please chat to us on discourse or ask for help on our chat if you have any questions or need further assistance with this issue.