Libretime: Add PPA to ensure latest Silan for Ubuntu based installations

[Robbt]

We have a number of solutions mentioned in the release notes but this requires an additional step and many individuals won't read the release notes. So if possible we should detect and perform an upgrade to a working version of silan with the install script.

[hairmare]

Debian 10+ will have a working install again, maybe we should start thinking about the possibility of getting the usable silan backported to older releases. AFAIKT it needs to be in Debian stable for this. Ubuntu Bionic already has the updated silan in it's repos. Maybe @paddatrapper has some inputs on this as he has taken over maintainership of silan in Debian.

[paddatrapper]

I may be able to backport silan into the current stable release. It depends on how many of its dependencies need to be backported. I can do this while Stretch is stable or once Buster is released in about 6 months time. It just depends on how the rest of my year plays out. My current goal is to get the lastest Liquidsoap release package and then try backport silan. Liquidsoap unfortunately is a little too complex to backport easily, so encouraging people to use the new stable in 6 months is probably the best way to get that

[hairmare]

I'm building silan for CentOS 7 which only has a really old environment so backporting silan should be doable. The same absolutely doesn't count for liquidsoap since it needs a current OCaml to build properly, so yeah... backporting liquidsoap won't be reasonable. We could consider maintaining two difference liquidsoap scripts for the time being as the changes needed aren't large and our liquidsoap scripts don't have a high churn.

[hairmare]

My version is not officially supported and I have no plans to do so. Generally overriding distro packages is a bad idea and I think this should be very explicit for the user.

[Robbt]

Well it seems like the issue will be hopefully fixed for Debian 10 and Ubuntu 18.04 and beyond thanks to @paddatrapper so I'm of the opinion that providing a fix via an unofficial package is a good idea because most people don't read the release notes. I think that adding the option to install it via the install script for specific distros that we identify as having a bad silan would probably solve most problems new users are having. Are there any liabilities introduced by the unofficial package other than overriding the distro package. Could we port or copy the files directly to the libretime project to make them more official ? Just curious as this is probably the bug that bites new users the most when they are getting started.

The other option would be to see if we can disable silan during install for distros that we have identified as having a bad silan. But we need to confirm that this doesn't break the airtime-analyzer service as I think it is explcitly called.

[paddatrapper]

That then breaks any use of distro packages, even if it does work... I think @Robbt has the right way of doing it (via installing unofficial if needed), if we can confirm that there are no issues when upgrading a distro to a version with a working Silan version packaged in it. Ideally it should prefer the distro's version in this case as this has the security support of the distro.

[paddatrapper]

@JohnnyC1951, ah yes, I see it now. I agree that a promp should be used to install, so that the end user is aware of the work around

[frecuencialibre]

what if we decided as a group that the beta release will only support ubuntu 18? the php 7.1 issue would have to be addressed by the installer, but perhaps that decision would still simplify our lives. @hairmare @Robbt
https://packages.ubuntu.com/bionic/sound/silan

[JohnnyC1951]

7.1 would go away completely as Ubuntu 18 is php7.2

[Robbt]

For the purposes of this conversation the debian based distros we currently support are
14.04 - but being deprecated due to end of LTS on April 2019 has silan 0.3.2-1 - broke
16.04 - is EOL on 4/2021 and has Php 7.0 and silan 0.3.2-1 - broke - and even though Php 7.0 is EOL by the PHP project in the past canonical has supported their php version with bug fixes see this
18.04 - will be supported until 4/2028 - see this and has silan 0.3.3-1 - which doesn't have the libretime import crippling bug.

Debian 10 will have silan 0.4 but it won't be released for a while.
Debian 9 has the broke version

I'll talk more about the 18.04 in #580

But just for the purposes of this topic I think that the workarounds we will need to do to get 18.04 working are going to be worse than coming up with a fix for 16.04 and Debian 9. So I'd probably lean against it but we can talk more about 18.04 in the relevant issue.

[paddatrapper]

Debian 10 should be released around April next year.

I want to spend some time in January backporting silan 0.4.0 to Debian 9 during the release freeze. Once that is done, users can install it from stretch-backports without issue

[hairmare]

install it from stretch-backports without issue

That sound like the point where we should automate this in the installer. I'm looking forward to this point!

[Robbt]

A little off topic but I think that we've seen issues with the liquidsoap provided by Debian 9 as well see #352 so will we be able to backport that as well ? Liquidsoap is far more essential than silan which we could easily remove without it affecting too many users negatively.

[paddatrapper]

Unfortunately backporting Liquidsoap is not possible - there are too many other packages that I would need to backport as well...

[Robbt]

Well we need to figure out #352 and determine if mp3 importing is broken out of the box with Debian because otherwise we can't support Debian until liquidsoap is upgraded.

[frecuencialibre]

would it be possible and a good idea to build silan right in the install script?
just saw this:
https://github.com/ned-kelly/docker-multicontainer-libretime/blob/e3108d7e9a424e43dee2387f93ddb6fb99c77cb2/Dockerfile#L82-L88

[Robbt]

Well I think building custom packages can make sense for docker because with docker you are ideally in the end just providing a pre-built image most of the time to end-users. But if we are trying to provide a more general distribution based upon the packages that are built into the distribution then that is introducing another thing to break down.

I guess we need to define our goals as a project more explicitly. In reality "docker" on a single container with proper instructions and the ability to host it easily would probably solve most peoples problems. The reason people are running ./install is because it is the only way to do things right now. Maybe the multi-contianer approach is actually better for something as complex as libretime.

I think one of the goals has been to depend more on the packages built into the operating system this makes it so that in theory we don't have to maintain them. But in reality as we've seen just because a package is included in the OS doesn't mean it is up to date and bug free otherwise we wouldn't have had to have @paddatrapper adopt liquidsoap and silan to get them updated in the newest version of Debian. Many of the javascript libraries we are using are siloed into specific versions.

I personally wouldn't be against treating the install script as cutting edge and putting in whatever is needed to get it working out of the box, but building a debian package automatically excludes RPM based distros that we are slowly including support for.

Another option would be to have official repos supported by the project. This is what sourcefabric did to fix all of the outstanding package incompatibility issues. They also provided a simple debian package up until 2.5.2 (when the practice was stopped) so that people could get started just by typing one command. There was basically never the effort put into resolving issues introduced when the new core lead took over and started refactoring a lot of things - @danielhjames was in charge of the Debian packages for Airtime and in this forum post he explains that 2.5.2 packages were planned to be released.

I'm not sure what the discussion was behind the scenes at SourceFabric regarding this but 2.5.2 was also the last release put out before they decided to explicitly focus their development team on the SaaS version and subsequently quit releasing code to the public. I guess this is almost just a history lesson but even though 2.5.1 was released almost half a decade ago there are possibly still stations running on it because it didn't require much intricate knowledge to get up and running with. I think that should be our goal when we go towards beta. The '''./install''' script should only be necessary for people who want to tinker and develop. It makes sense as a way to build the dependencies and provide a basis for a development version for docker and vagrant and the like but we shouldn't require users to jump through that hoop unless they are eager to dive deeper.

So sorry for the long history lesson on this but it does make me realize that we should really be gearing any "beta" release and subsequent 3.0 LibreTime around the idea that we are using the most bug-free and stable libraries. In most cases this would be the latest version but we also need to have some way of verifying that an API change or refactoring hasn't introduced new bugs. Currently we have some instability that occasionally occurs between liquidsoap and pypo and no way that I know of verifying where the problem came in see #125 & #232 for examples.

But I guess I really went on a tangent here and this could be considered off-topic so I'm going to refrain from further deliberation and/or revisit these thoughts on the more appropriate topical threads.

[Robbt]

But @frecuencialibre long and short of it, yes it is possible, but I'm not sure it's a good idea. We do have a repo built around debian packages but appears to include the entire codebase and hasn't been kept up to date with the master repo. I've never gone through the process of developing a debian package specifically for distribution so starting with LibreTime might have a steep learning curve considering how complex of a project it is but I do think wrapping the entire project in a debian file is a good idea if we want to put the effort in. A pre-built docker image might make more sense and is certainly easier to achieve. I remember this was what Albert considered a more productive use of time back when he was in charge of the project and probably one of the reasons he never put the time into building a debian package for the version he released.

[paddatrapper]

I want to spend some more time finishing the Debian package, even if it is just for our release and is not fully respecting Debian Policy. The issue is there is no Debian release that I can target - stable has broken Liquidsoap and silan, while testing and unstable have PHP 7.3, which we can't use. I'm still trying to find a PHP 7.0 repo we can piggyback, but I've come up blank so far.

Perhaps I should target stable for the moment and get a package out that mimics the install script. That way it can be tested to the same extent as the script, even if that is not fully working on a Debian release. I'm not sure when I'll get a chance to do so, but it will probably be between Christmas and New year

[Robbt]

What are your thoughts on Ubuntu packages ? Would we be able to do those as well. I think that Ubuntu 16.04 is what most of us are using currently for our development because liquidsoap works out of the box and there are unofficial packages from @hairmare that work to fix the silan. I'm still running a 14.04 instance in production and need to figure out the time to upgrade that. The benefit of Ubuntu 16.04 is that in theory Canonical will continue to support Php 7.0 with security updates for the lifetime of 16.04 which extends to 2021. link Whereas any repo we find for Debian to support 7.0 will be "unofficial" and thus the lifetime of support will be undefined. I know that Debian is a community ran project whereas Ubuntu is a corporate product that just builds off of the work of Debian but in this instance we are in an odd position where we are wanting to extend the life of software (LibreTime) built upon a shaky foundation (Zend 1) that is officially EOL. So it probably makes sense to compromise to find the best bad solution while we can work on finding a new foundation and/or rebuilding things. In my mind this is probably 16.04 with backported packages.

If on the otherhand we find that there are definite bugs that cause issues with pypo and liquidsoap 1.3.x fixes them we will need to prioritize #192 and make sure that we can provide packages for this. At this point I'm not sure that we have any methodical way of determining if liquidsoap 1.3 works better with LT than 1.1.1 which we know at least works most of the time. For all we know 1.3 might cause incompatibility that requires changes to our code and thus we would to have our code support both versions or figure out a way to get everyone running 1.3 and deprecate support for 1.1.1 - if we decide to do that and backporting 1.3 to previous versions of Debian are a pain I can only imagine that doing so with Ubuntu will also be unfeasible.

It is indeed an awkward spot for us to be in and I am only grasping at what I think might be the best solutions. Other than abandoning work on improving this version of LibreTime and focusing all of my effort on #2. I think there is an effort by @frecuencialibre to help put out a working as best we can version of the software that we can call a beta and then start planning what we can call the next-gen. I'm pretty much in support of that idea at this point but I'm also on the hook by a grant my station received to add certain functionality that doesn't currently exist so I'm focusing my attention on that as well and hope to merge a number of PRs built upon what we have in place already.

So the TLDR is would you be interested in building packages targeting Ubuntu 16.04 instead of or in addition to Debian packages ?

[paddatrapper]

I only really know about Ubuntu packages because they work the same as Debian ones. I don't have any direct experience targeting Ubuntu, but it should be straightforward enough to target Ubuntu 16.04 in the mean time. I would need help testing and around any Ubuntu specific issues, as I have never run Ubuntu in production as I just deploy Debian.

Does Ubuntu 16.04 have the required packages for us to rely on (PHP 7.0, working silan, working Liquidsoap)? I haven't yet made a request to sync Ubuntu's version of Liquidsoap with Debian (they uploaded a Ubuntu specific patch some time ago, so it doesn't automatically sync), so it still provides 1.1.1. If it is the decided target for a beta, I can start working on an all-in-one package that effectively replicates the install script. My goal would be not to include any Ubuntu specific things, but the dependency versions would effectively make 16.04 the only viable candidate.

My end goal is still to have working Buster/Buster+1 packages so that I can deploy LT in the production environments I am involved with. I would love to find some way of helping that effort, but I think migrating from Zend 1 needs more experience with PHP than I have. It would also need a resolution for #2, but that seems to be a while off.

I agree with focusing on getting a working Beta out, targeting Ubuntu 16.04 and then start working on Debian/PHP > 7.0 compatibility from there.

[Robbt]

Yeah for some reason liquidsoap 1.1.1 works under Ubuntu w/o any issues that I'm aware of. They don't have working Silan by default but we can probably easily build a working deb that can upgrade if people add the our apt repo. And they do have php 7.0 which is only supported by php until next month but in theory security updates should be continued by ubuntu if they are found until 2021.

I'd be happy to help support this effort. In theory we could also support 18.04 but this would require people to install the PPA for php 7.0 and I think that the silan fix is included in 18.04 by default. So targeting 16.04 is probably best at this point.

I'm happy to help with testing Ubuntu stuff I just don't know how to do the whole process of building and maintaining a deb package repo.

[paddatrapper]

The version in Ubuntu had a fix uploaded to it that was applied directly to the Ubuntu package, not going through Debian, that is why it works and Debian's does not. Thinking out loud - I may be able to apply the patch to Debian and upload to sloppy backports or proposed updates... I'll have to look at that.

I'm happy to build a deb that people can use to test. Once it is ready, I'll tag a release in libretime-debian-packaging and attach it there.

[ned-kelly]

Well I think building custom packages can make sense for docker because with docker you are ideally in the end just providing a pre-built image most of the time to end-users. But if we are trying to provide a more general distribution based upon the packages that are built into the distribution then that is introducing another thing to break down.

Agree here - I think there needs to be a few install options for people running on physical tin - and then there's the pre-built docker images which can have these fixes all "pre-baked into the mix"...

FWIW - I'm planning on taking all the various prices of my multi-container docker build over the holiday period and putting it into a single container for smaller deployments so I can personally run it on Heroku (i.e. an on-line streaming station only -- not a larger FM setup who should ideally be running in multiple containers and some sort of orchestration setup such as Kubernetes).

[Robbt]

This might end up solving itself if we transition to php 7.2 compatible zend at #670 by encouraging people to use Ubuntu 18.04 or Debian 10 and/or have them manually install the silan fix if they choose a older distro that has bad packages.

[paddatrapper]

Is there anything more we can do on this bug? Debian 10 has been released with a working version, Ubuntu 18.10 works and most of our efforts are focusing on these releases and later. People running 16.04 and Debian 9 should either upgrade or have enough knowledge to install silan from the PPA. Considering that LT requires a dedicated machine/VM to run, there shouldn't be too many other services running that could cause issues on upgrade

[hairmare]

I guess we can either keep this open until we phase out Debian 9 and Ubuntu 16 support in the installer or we can just close this now since it is properly documented in the release notes and there haven't been any new issues pertaining to this.

Maybe we should add a silan version check in the installer and emit a warning if a buggy silan is used as means to having actively done something to close this.

[Robbt]

So I think that we are in the boat where even silan 0.3.3 is bad and segfaults vs. providing bad information. So basically anyone using a version of Ubuntu less than 19.10 will need to manually build silan to get it working, unless we want to start hosting our own package repository to fix broken default packages.

Here are the build instructions I just posted on the forum for someone

I posted the following instructions.

git clone https://github.com/x42/silan.git
sudo apt install dh-autoreconf autogen libavcodec-dev libavformat-dev libsound1-dev
sudo apt install
cd silan
./autogen.sh
make
sudo make install
which will create a 0.4 silan in /usr/local/bin
you can then sudo apt remove silan
then run
silan --version and you should have 0.4 installed.

Probably the easiest workaround for people with bad versions from their package manager.

[paddatrapper]

I've been thinking about setting up a PPA for our Ubuntu support. It has several advantages, including building LT for many architectures, improved update management and means we could include things like a working silan version. I'll try get a chance to work on it this week or later on the month

[paddatrapper]

I have created https://launchpad.net/~libretime/+archive/ubuntu/libretime and uploaded the current alpha.8 package to test. Once it is all working, I will test backporting silan for 18.04 and upload it there too

[monomusika]

hello good day, I've been having problems with libretime for a month now that the music tracks play at half of their real time, I opened a thread and they told me it's a silan problem. I am learning have me patience please. as far as I can see I have the silan version 0.3.2 and I need to upgrade to a recent and stable version but I really don't know how to do it from SSH. Could you tell me the steps please? I am using Ubuntu 18.4 xenial. thanks and have a nice day

[Robbt]

Here are the steps to install silan 0.4 via the command line/ssh

sudo apt install git
git clone --branch v0.4.0 https://github.com/x42/silan.git
cd silan/
sudo apt install dh-autoreconf autogen libavcodec-dev libavformat-dev libsndfile1-dev
sudo apt install
./autogen.sh 
make
sudo make install
sudo apt remove silan

You will also need to update the Silan information for the tracks you've imported - see https://discourse.libretime.org/t/how-to-check-if-silan-is-working/256/13

[monomusika]

Estos son los pasos para instalar silan 0.4 a través de la línea de comando / ssh

sudo apt install git
git clone --branch v0.4.0 https://github.com/x42/silan.git
cd silan/
sudo apt install dh-autoreconf autogen libavcodec-dev libavformat-dev libsndfile1-dev
sudo apt install
./autogen.sh 
make
sudo make install
sudo apt remove silan

También deberá actualizar la información de Silan para las pistas que ha importado; consulte https://discourse.libretime.org/t/how-to-check-if-silan-is-working/256/13

Thanks you! :)

[paddatrapper]

The PPA now includes Silan version 4.4.1, which should work without people needing to compile from source.

[paddatrapper]

So the installer can add the PPA and install silan from it to ensure that people have a working version on Ubuntu

[Robbt]

So next step is to add the PPA via the installer for Ubuntu based systems ?

[paddatrapper]

Yup

[zklosko]

Can this be closed out?

[paddatrapper]

So next step is to add the PPA via the installer for Ubuntu based systems

This still needs doing