Libretime: problem: auto-playlist functionality won't schedule tracks if their is dead air for the hour before the show is scheduled

[Robbt]

The autoplaylist poll depends upon API calls made by airtime-playout/pypo in order to work. This means that if there is dead air for the whole hour before a show is supposed to air the autoplaylist will never get created.

[hairmare]

All celery tasks only get executed when there is an api call. Maybe we should let execute some of the stuff in celery regardless of a task. I.e. the autoplaylist feature mught be a better fit for pypo?

[Robbt]

That or we could execute periodic api calls via a cron/timer job or perhaps more simply pypo could be modified to make periodic API calls even if it doesn't have track info to update the database with. This would keep podcasts updating even if there is a prolonged period of dead air on the system for some reason.

[hairmare]

/api/version does almost nothing and could be called by a cron/timer. That would be quite a hacky solution though. I think piggybacking celery tasks onto the API was a bad decision in the first place.

[hairmare]

Maybe we can remove the piggybacking code and just have a timer run tasks regularly.

[Robbt]

Yeah running them on a timer seems much more logical in the long run than depending upon regular API calls from pypo.

[hairmare]

Actually, I wasn't suggesting pypo do the API calls but rather pypo do the whole scheduling proper.

[Robbt]

I was thinking that we would just have pypo trigger the celery tasks on a periodic basis directly vs. rewriting podcasts and smart blocks functionality in python. I'm not sure what the benefit of having pypo do the scheduling itself would be since it would still need to get its directions from the database which is configured by the web UI, but maybe I'm missing something.

[hairmare]

I think I get where you are coming from now. Should we move all the piggybacked tasks to pypo or let some of them stay in the API?

[Robbt]

I'm not sure. I think honestly the easiest fix would be just to have pypo regularly trigger the API call since we need to use the PHP code regardless or else would need to write code to query the DB in Python and basically refactor a bunch of modules for unknown gain. I haven't looked at the pypo code much to see if triggering a API call when it isn't doing anything works. The alternative if we aren't refactoring functionality would be to call the API via another mechanism. While using the API to trigger jobs might not be the best way of designing things I don't see how it causes any problems. The only problem I'm aware of is this one which is caused by the API not being triggered regularly. I'm open to alternative proposals but it doesn't seem necessary to factor this out at this point.

[Robbt]

In looking at it we could write another Pypo module to simply call the API every 5 minutes with a timer, this seems like a little bit of overkill but might make more sense than trying to alter the functionality of one of the other modules. The other option would be to just put a few lines of code in the main function of pypo.
There also appears to be an issue with logging, my logs only contain the startup code and never any of the details of what happens w/ pypo.

[Robbt]

I was thinking of implementing a simple API call on a 5 minute timer using this answer from stack overflow and putting it in main. Feel free to make alternative suggestions.

[hairmare]

I realized why it needs to access the database to access to run tasks. What I don't like about how it is currently done is that the tasks are triggered as a side effect of an idempotent GET request.

Maybe we can just move the whole thing to it's own action and call that. The action could even give some feedback on what it did for pypo to log.

[Robbt]

That makes sense, I think that creating a new specific API call is a good idea. Are your pypo logs working ?

[hairmare]

The log file isn't working but on systemd based systems like CentOS the stdout from the script is being logged to systemd-journal and I can look at it through journalctl. The bug that it only writes /var/log/airtime/pypo/notify.log and nothing else is something we should probably fix at some point. I'm assuming it's a case of properly configuring logging for the most part.

[Robbt]

Inspired by the well formatted issues of @bburton I'm going to clarify my approach to this before I start the coding process.

Issue

LibreTime relies upon API calls from pypo playout to trigger celery tasks such as Podcast downloading and automatic playlist scheduling. If there are no tracks scheduled and LibreTime is offline for the hour prior to a automatic playlist scheduled show the automatic playlist will never be scheduled.

Expected Behavior

LibreTime should run the celery tasks on a period basis even when there is no track playback. The frequency of this update has yet to be determined and perhaps should be user configurable. At the very least it should occur every 10 minutes but it may be desirable for it to happen as often as every minute.

Diagnosis

In discussion with @hairmare it was decided that relying upon random API calls isn't optimal and we should design more intentional system.

Proposed Solution

I will implement a new API call that will specifically trigger celery tasks and add a timed script to call as part of the pypo package independent of playout.

• [ ] Add a new API call that specifically triggers the celery tasks
• [ ] Write a sub-module in pypo that on a timer calls this celery API call every X minutes
• [ ] Remove the hook from the template file that triggers celery tasks based upon API calls

Any feedback is welcome before I begin to code it. The first part is straightforward but I'm not sure if there is a better place to put the timer or if it should be a completely independent module. It is somewhat arbitrary to place it inside of pypo but convenient and unlikely to be burdensome as pypo will remain a core component.

[Robbt]

In reviewing this issue I'm not 100% sure the current system is non-functional.
Here is the relevant line in airtime_mvc/application/controllers/plugins/PageLayoutInitPlugin.php
68 // Piggyback the TaskManager onto API calls. This provides guaranteed consistency
69 // (there is at least one API call made from pypo to Airtime every 7 minutes) and
70 // greatly reduces the chances of lock contention on cc_pref while the TaskManager runs
71 if ($controller == "api") {
72 $taskManager->runTasks();

My only issue with replacing this with a regular API call is that I want to get the timing right. Ie we don't want it to be so frequent that it causes this lock contention issue but we also don't want it to cause tasks such as podcast downloading to seem unresponsive.

There is already a poll-celery api call. this only gets called by the podcast.js file and is used to simply update the celery tasks.

I'm going to investigate the pypo application at this point and the python api client to implement the function call I'm adding to the api client.

[dosmanak]

Hello, could you clarify the title of this issue? I do not understand circumstances that lead to the problem. I am experiencing the same thing. The playlist is not filled into the show. There are only dynamic smartblocks in the playlist.

[hairmare]

I think https://github.com/LibreTime/libretime/issues/129#issuecomment-294267272 explains the issue best. You need to call the UI regularly for the needed api calls for autoplay to work because the update task gets executed as a side effect to ui calls.

[JohnnyC1951]

Would it be possible to stuff a dummy call in, every so often, to fix this? Even if its a cron job for now?

[Robbt]

I think a dummy call would probably work. I'll probably work on making this happen when I get around to working on the media-import module since they're both python code.

[Robbt]

It appears based upon the info in #422 that this dependency upon pypo scheduling can also cause issues where playlists aren't scheduled when tracks that are greater than 1 hour long are playing. We are waiting for confirmation from @JohnnyC1951 on this one, but either way replacing the call with an independent timer would be the best fix.

[Robbt]

Ok, so I think the simplest way to fix this is to ensure that pypo makes API calls at least every 7 minutes, which the code suggests that it does but in reality when nothing is scheduled this doesn't appear to happen.

I'm looking at the pypoFetch code and it does appear to manually trigger an API call every 480 seconds or 8 minutes (vs. the 7 in the commend) here - and this does appear to be happening even on a system that is starting up (I see the log)
https://github.com/LibreTime/libretime/blob/6031d953518b1f2f87d589de262e3a4a44e70ffc/python_apps/pypo/pypo/pypofetch.py#L479-L507

But this basically triggers a call to the api_client https://github.com/LibreTime/libretime/blob/6031d953518b1f2f87d589de262e3a4a44e70ffc/python_apps/api_clients/api_clients/api_client.py#L253-L257

Which requests this URL - https://github.com/LibreTime/libretime/blob/6031d953518b1f2f87d589de262e3a4a44e70ffc/python_apps/api_clients/api_clients/api_client.py#L69

So the question is whether or not this is being interpreted as an API call by the controller and thus running the task manager. I guess we can just manually call schedule/api_key/#SITEKEY# and see if this triggers the TaskManager to run etc. I suspect that for some reason it does not and if we fix this or make pypo hit an additional URL then this will resolve the issue.

[Robbt]

So oddly enough this appears to be hitting the url '/api/schedule/api_key/' - which should trigger the TaskManager to run tasks and I do see scheduling autoPlaylist logstuff in the zendPhp log with nothing on the schedule for my test PC.

So looking back at #422 - I'm going to see if it still does API calls while a long show is running w/o anyone viewing the website - I'm also curious if the issues we have been pointing here could in fact be the result of this function not always running correctly - https://github.com/LibreTime/libretime/blob/3196603dde94bd806341a546fc292eb54fbc59d2/airtime_mvc/application/common/TaskManager.php#L69-L78

The troubleshooting goes on but at least I know a little bit more about how this is working.

[JohnnyC1951]

try: return (True, self.services.export_url()) 
except:  return (False, None)

Is that perhaps resulting in a silent failure with nothing returned?

[Robbt]

I tested this overnight on my computer running vagrant and the pypofetch timeout ran and triggered an autoplaylist to build while a 3 hour show was airing. So this bug doesn't exist the way I thought it did.
There are also numerous times when it receives nothing for the schedule and it doesn't trigger a failure. So I'm starting to think that perhaps there are issues with the TaskManager::runTasks not always running when it should rather than the issue being the TaskManager itself isn't triggered (as I originally believed and had planned a work around for.

So options moving forward. We need to do more testing to find when the error occurs. It's possible and likely that if DNS resolution isn't working the pypoFetch could not be able to resolve the host and thus not be able to trigger an API but this would also result in a general failure of the playout system in general and isn't specific to periods of dead air. I'll set up another test case later today and confirm if indeed an autoplaylist is built after hours of dead air.

Moving the TaskManager triggering into a Python app or as part of Pypo is an option but lets try to determine what causes the issues we attributed to this bug before we rewrite functionality that is otherwise working.

[JohnnyC1951]

Are there any other clues from 'issues' to could point to a TaskManager failure? This must be used by many things.

[Robbt]

I haven't seen anything that points to a TaskManager::runTasks failure per se, it is just my next logical guess considering I can't duplicate this issue and that is the next logical place I can see this failing. I'm tempted to close this issue once I determine that the pypoFetch command actually accomplishes the AutoPlaylist task when there are no tracks playing and nobody visiting the website and we can leave the refactoring for the future and re-open the bug if we determine something is actually failing. This bug was initially a guess on my part and its possible that something else caused the failure of the autoplaylist and I just deduced incorrectly that the problem was TaskManager wasn't triggered.

[JohnnyC1951]

The other seemingly related oddity is RSS import? Sounds similar?

[JohnnyC1951]

Hmmm. I just realized that since installing the last public release, I do not think I saw an auto-loading failure. Maybe it already got fixed?

[Robbt]

Yeah, so on my test instance with no browser open and nothing scheduled it successfully scheduled an autoloading playlist after a few hours of dead air. So I'm going to close this issue for now and we can reopen it or a new one if we run into this problem again or are able to replicate the problem.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please chat to us on discourse or ask for help on our chat if you have any questions or need further assistance with this issue.