Azuracast: Webhooks triggering are inconsistent

[pascalla]

Using Docker installation method
Yes

AzuraCast version
Rolling Release #f62a04c (2020-09-20 13:07)

Host Operating System
Ubuntu 18.04

Describe the bug
Webhooks seem to be very inconsistent across all webhooks. I have even setup a webhook with a post dump. When testing the webhook I get a blank page

To Reproduce

1. Setup a webhook to a post dump (https://ptsv2.com/) where it triggered by playing song changes.
2. Wait for songs to go and check the post dump.
3. Wait for a few songs and see missing songs

Expected behavior
To see all the songs posted.

Relevant Logs
They are being registered by the LiquidSoap.

2020/09/24 12:21:51 [***_input_streamer:3] New metadata chunk ? -- Coolio; L.V. - Gangsta's Paradise.
2020/09/24 12:25:50 [lang:3] Authenticating DJ: ***
2020/09/24 12:25:53 [lang:3] AzuraCast DJ Auth Response: true
2020/09/24 12:25:53 [***_input_streamer:3] New metadata chunk ? -- Dua Lipa - Hotter Than Hell.
2020/09/24 12:30:08 [lang:3] Authenticating DJ: ***
2020/09/24 12:30:11 [lang:3] AzuraCast DJ Auth Response: true
2020/09/24 12:30:11 [***_input_streamer:3] New metadata chunk ? -- David Guetta - When Love Takes Over (feat. Kelly Rowland).
2020/09/24 12:33:25 [lang:3] Authenticating DJ: ***
2020/09/24 12:33:28 [lang:3] AzuraCast DJ Auth Response: true
2020/09/24 12:33:28 [***_input_streamer:3] New metadata chunk ? -- Jason Derulo; David Guetta; Nicki Minaj; Willy William - Goodbye (feat. Nicki Minaj & Willy William).
2020/09/24 12:37:20 [lang:3] Authenticating DJ: ***
2020/09/24 12:37:23 [lang:3] AzuraCast DJ Auth Response: true
2020/09/24 12:37:23 [***_input_streamer:3] New metadata chunk ? -- BTS - Dynamite.
2020/09/24 12:41:35 [lang:3] Authenticating DJ: ***
2020/09/24 12:41:39 [lang:3] AzuraCast DJ Auth Response: true
2020/09/24 12:41:39 [***_input_streamer:3] New metadata chunk ? -- Little Mix - Holiday.

It seems that in the azuralogs, it seems to be dispatching but I am recieving nothing on my post dump server.

Screenshots
Sending a test webhook:

Additional context
This is happening on all types of radio software, so seems like an issue with webhooks not triggering/sending.

[pascalla]

I can't consistently reproduce this anymore, it seems to be inconsistent. Sometimes it works and has period of not sending.

Going to try gather more info on this.

[Vaalyn]

I've found an issue with the Test button on the WebHooks page.

When the dispatcher is running the testDispatch method it calls the dispatch method with the SendWebhooks object created here: https://github.com/AzuraCast/AzuraCast/blob/master/src/Webhook/Dispatcher.php#L152-L153

When the dispatch method checks $connectorObj->shouldDispatch($event, $webhook) this returns a false. I've looked at why that might be and found that the shouldDispatch method looks at the triggers that are registered in the SendWebhooks object here: https://github.com/AzuraCast/AzuraCast/blob/master/src/Webhook/Connector/AbstractConnector.php#L26-L38

When I check what triggers are registered in the SendWebhooks and the StationWebhook objects that are provided to the shouldDispatch method this is what I get:

$webhook->getTriggers():

(
    [0] => song_changed
)

$event->getTriggers():

(
    [0] => all
)

As far as I can see the all is not checked for anywhere which leads to the WebHook not being dispatched when the test button is used. The regular dispatch does seem to be working though so I think this is most likely isolated to the test triggering.

@SlvrEagle23 I'm not completely sure which of the following 2 solutions would be preferred here, would you mind helping me decide?

Solution 1:
Change the line intestDispatch() that creates the SendWebhooks object from $event = new SendWebhooks($station, $np); to $event = new SendWebhooks($station, $np, true, $webhook->getTriggers()); so that it always contains the triggers of the WebHook when running the test.

Solution 2:
Since the constructor of SendWebhooks initialises the triggers array with all when none are supplied the hasTrigger() method that is called by shouldDispatch() could be changed to return true when it finds the all trigger.

[SlvrEagle23]

@Vaalyn I went with option #1 and went ahead and pushed it up on master just to avoid any delays. Thank you for catching that!

I think this may overall have something to do with the message queue not currently having differing priorities for various dispatched message types. This could lead to media processing queue items overwhelming the (higher-priority) webhook dispatch messages. The Symfony MessageQueue that we use has support for priorities, we just haven't implemented it yet.

@pascalla If you notice that a webhook isn't dispatching, you may be able to debug things from the host by running:

cd /var/azuracast
./docker.sh cli queue:list

and seeing if there's a long queue just sitting there waiting to process. You can also bump up the number of simultaneous queue workers to help process the queue via an environment variable in azuracast.env.

[SlvrEagle23]

@pascalla I've finished the process this evening of rewriting our message queue implementation to move back to Redis, which was far more performant and didn't experience much in the way of issues compared to our MariaDB queue implementation, which we had only switched to just so we could more easily flush the cache for media processing, which we've fixed through another implementation.

Let me know if the updated code helps at all.

[pascalla]

The queue implementation may of been part of the problem, and I appreciate fixing the webhook tests and the queues. I believe the error lays else where unrelated to this ticket, going to close the ticket and open another. <3 you've done amazing work here though