Horizon: what is the full how-to to get multiple horizons to work side-by-side on same server?

[vesper8]

Been trying to get this working for over an hour and tried many things but still doesn't work.

I have two distinct projects that use queues and I want to use horizon to monitor them separately yet they are hosted on the same Laravel Forge droplet

They are both using the latest version of the 1.0 branch and both have a unique prefix

I thought that's all it would take but even after setting the prefixes, running cache:clear and running horizon:terminate in each app, I still had one app that would just spin and spin and make my ram go nuts. The error log showed that it was trying to deal with one of the jobs that had been queued on my OTHER app. So even though they both used unique prefixes, one app is still trying to handle queues from the other app. And since the Job class doesn't exist it throws a method_exists error in the log. But on the dashboard it just spins until I am forced to force quit the browser

I couldn't really find the answer here so I looked at the other things that were identical for both apps. I renamed the supervors from supervisor-1 to prefix-supervisor-1 so they are unique on both sides

I restarted supervisord itself multiple times throughout this

That still doesn't fix it. So I tried adding the prefix in front of the queue names (default and notifications)

This confuses me though, if I change the queue names in the horizon config file, does that mean I also have to change them in the queue configuration file?

I tried all that, now my horizon dashboard won't even load on the app that has the Job class in it.

It's my understanding that Horizon monitors queues from other apps even if they have unique prefixes?

What is the solution here? Do I have to set different redis databases in the database config?

More people are going to want to do this.. please lay out all that needs to be modified on two apps for them to use horizon on the same server

tyvm

[vesper8]

it seems the reason why my dashboards are stalling is because the "Current Workload" is filled up with an infinite amount of tasks. But I have no idea why it's doing this.

It's even doing this if I have supervisord completely shutdown and have done cache:clear. Basically it should just say "inactive" but instead it stalls and sometimes it shows me that it's stuck in a huge loop but I have no idea what this workload is coming from since I just did cache:clear and nothing is running. Maybe I found a bug but I don't know what's going on here.

[vesper8]

btw both php-cli and php-fpm (via phpinfo in-browser) report phjp 7.1.8

I also tried flushing redis via Medis and the redis-cli with flushall

here's a screenshot of what I sometimes get if I wait long enough

ad infinitum

This is after running queue:flush / cache:clear, and I get this whether horizon is running or not... it was working yesterday but then I tried to get two of them to work side-by-side. Now I've even reverted my changes and rebooted multiple times.. tried different browsers, cleared cookies and local storage.. restarted valet.. really baffled

[vesper8]

ok.. prepare to have your mind blown.. I should have gone to bed hours ago.. but this was one of those bugs that wouldn't let me get some rest because of how little sense it made. There must have been something new introduced because this tiny little detail wasn't breaking horizon specifically before and I didn't change that bit

Basically, the infinite spinning was being caused by Laravel Dusk somehow. Specifically, it was being caused by package auto-discovery because it was being discovered twice?

I was manually adding "Laravel\Dusk\DuskServiceProvider" in the auto-discovery of a custom package I load in my require-dev. (Months ago I manually added a bunch of package service providers so I could use auto-discovery ahead of everyone adding it to their repos)

After hours of digging and uninstalling packages I narrowed it down to this. Once I removed it from the auto-discovery section of my custom package it stopped the infinite spinning. Tested this on more than one app too.. wth? Something must have just been added in the main laravel repo that caused this

I don't know if this also was the root cause of the problems I was having with running two Horizons.. but I'm too tired to try it now.

Would still appreciate 'best practices' for running two horizons. Is having a unique prefix enough?

Will check in tomorrow. Hope this helps someone somehow but I doubt it will.. too obscure

[vesper8]

It seems it's working fine now.. I was really bogged down by this super weird bug I mention above

With unique prefixes AND unique redis databases it seems to work perfectly separate from one-another. But if I leave it on the default 0 redis database then there is some leakage as one app tries to handle jobs created by the other app

[bbashy]

@vesper8 Did you also set the queue name for Redis different?

[vesper8]

@bbashy no I left it to "default", and I'm also using "notifications"

I don't fully understand the implications of the changing those names. I can change it in both the horizon config and the database/queue config and that's all I need to do?

What about 'notifications' ?

[bbashy]

@vesper8 It depends if both apps are using the same Redis queue names. If so, you'll run into problems.

[vesper8]

well I don't want to have to change queue names all the time.. using a unique redis db (other than 0) seems easier and that's working well

plus if I did change the queue names.. then what would happen with the default notifications queue which seems to be handled "magically".. better to use a different db. It would be nice though if you could define a prefix for your queue names in the horizon config

[elynnaie]

I have my development and pre-production run on the same server, and I ran into this issue when having two Horizons running on the same server. I thought that changing the horizon prefix was all that was needed, but it turns out I also (or perhaps instead) had to change the redis database. (I have no idea what the "redis queue default" means, so I am afraid to change it; we have a lot of queues, not just the default queue.) Thanks @vesper8 for the suggestion!

[bbashy]

@denaje Yes, you need to set a REDIS_QUEUE=someUniqueName into .env and use it in the config/queue.php file with 'queue' => env('REDIS_QUEUE', 'default'),

[CaddyDz]

Using a different redis database solved it for me but am still concerned about this
Benchmark: Shared vs. Dedicated Redis Instances
is there a better practice?