Horizon: Dashboard shows inactive status but jobs are still processing

[jgodish]

Environment: Running horizon on multiple servers with a central dedicated redis instance. Each server that runs horizon handles certain jobs. Redis version 4.0.2. Horizon version is the lastest v1.2.3

As the title states, the status on the dashboard shows "inactive" but the jobs are processing fine. It displays the "Current Workload" section but it does not list the "servers" section for some reason. Nothing has changed with our redis server or application servers this just suddenly started happening.

I did some digging and found a class RedisMasterSupervisorRepository which adds the current server into redis using the update method. The status on the dashboard seems to use this with the "names" method. It's seems like it's just not persisting this specific data in redis.

I then did a redis-cli monitor on the redis server and saw the commands from the "update" method I mentioned above.

"HMSET" "horizon:master:volt-0MoH" "name" "volt-0MoH" "pid" "4699" "status" "running" "supervisors" "[\"volt-0MoH:api\",\"volt-0MoH:api_batch\"]"
"ZADD" "horizon:masters" "1524834943" "volt-0MoH"
"EXPIRE" "horizon:master:volt-0MoH" "15"
"ZREMRANGEBYSCORE" "horizon:masters" "-inf" "1524834929"
"ZREMRANGEBYSCORE" "horizon:supervisors" "-inf" "1524834929"

I tried all the usual solutions of restarting redis, restarting the entire server but nothing has helped. Hopefully someone has some ideas on what may be causing this, I can't seem to figure it out

[ejunker]

I have the same issue with Horizon 1.4.0. I have 3 mostly identical environments and in 1 of them the web interface shows Status Inactive even though when I run artisan horizon:list it shows Status as running and artisan horizon:supervisors shows a running supervisor. I thought it might be due to some stale horizon data in Redis so I stopped horizon and deleted all the keys in my redis db and then restarted horizon. I can see keys like horizon:supervisors, horizon:master:cron01-nNbI, and horizon:supervisor:cron01-nNbI:supervisor-1 but the web interface still shows Inactive.

I'm not sure what else could cause the issue. The horizon artisan commands show it as active so it is strange that the web interface does not match.

[jgodish]

For whatever reason, the horizon dashboard randomly started working for me again. I did not do anything to try and fix it I just checked it one day and it showed everything as it should. Even though this eventually fixed itself i'm going to leave this open in case other people find what may be causing this.

[driesvints]

@ejunker can you post the horizon configurations from your three environments?

[jfoliveira]

The same issue is happening to me every 48 hours or so. The dashboard shows status Inactive whereas horizon is still running and jobs are being processed.

I came up with a workaround, that is helping me restoring the dashboard to the expected state.

1. Backup and remove supervisor/daemons configurations /etc/supervisor/conf.d/*
2. php artisan horizon:terminate to stop all horizon current instances
3. php artisan horizon to start horizon as a single process through the command line
4. ctrl + C to stop php artisan horizon that is running manually
5. Re-setup supervisor in /etc/supervisor/conf.d/*
6. Supervisor will then call horizon processes and everything will be fine

After those steps everything is working fine, although I still can't understand why. :(
I hope this helps others on figuring out the root cause of this issue.

[ejunker]

I upgraded to Horizon 2.0.0 and I no longer see this issue, though I am not sure if the upgrade fixed the issue or if it is just a coincidence.

[driesvints]

Closing this because a lack of activity from the author. Please try to upgrade to 2.0 as @ejunker suggested to see if it solves your problem.