Horizon: Failed Jobs list is empty, but failed jobs show when searching

[fgilio]

Hello!
I have this weird bug that might have been reported before here https://github.com/laravel/horizon/issues/458 and here https://github.com/laravel/horizon/issues/109
I have some failed jobs, but The Failed Jobs list is empty:

But when I search for a specific tag then it shows in the list:

Is there anything I can do to help you debug this? I'm using Horizon v2.0.3 and deleted+republished the assets when updated.

[driesvints]

Are you getting any console errors on the first screen?

[fgilio]

Well that was fast haha, thanks.
No errors in the console, but the jobs are there in the response:

[driesvints]

I was working on Horizon bugs atm :)

[driesvints]

You're looking at the wrong endpoint. The one for failed jobs is /jobs/failed. Does that one contain jobs?

[fgilio]

Let me see... will update this comment

I think the endpoint was the right one, the thing is that for some reason Chrome removes the last part of the path in the list.
Here I recorded a video with more details: https://www.dropbox.com/s/ldgm14ks4a0osg6/Kapture%202019-01-24%20at%2011.29.03.mp4?dl=0
Also in gif shape: kapture 2019-01-24 at 11 29 09

[driesvints]

I can't figure out why this is happening. Can you maybe recreate an example app which recreates this problem?

[fgilio]

Mmm I don't know how soon I can setup that, also I guess it might be something related to the load we have (?
I don't want to sound pretentious, but would it help if we do a video call and I let you control the pc?
Or maybe there is something I can debug myself (?

[fgilio]

It seems to happen when there are lots of failed jobs to show.
I have a pretty general tag (issue:pdf) that matches a ton of jobs, if I search for that one then the list is empty, even when the response from the server does contain jobs.
But if I search for a tag (issue:26165) that is related to just a couple of jobs then those are shown correctly. Even if those jobs should also be shown when searching for the general tag (issue:pdf).

[shawnhind]

I have this same problem. The requests to /horizon/api/jobs/failed even successfully gets the failed job data but it looks like the vue component just won't display the retrieved data.

I brought up this issue previously but was told to get help in help channels which I've tried without any success. https://github.com/laravel/horizon/issues/458

I'm happy to provide some more info if it helps narrow down this problem.

[shawnhind]

I wonder if the issue could maybe be the data contained in the jobs itself failing to be parsed by the component? Like something in the failed job data that breaks the component that displays the jobs. I unfortunately can't post the retrieved failed job data since it contains customer contact information.

But similar to @fgilio this does seem to happen mostly when there's a lot of failed jobs to display but that might just be due to the chance of "bad data" being higher in a larger list of failed jobs.

[fgilio]

Hi @shawnhind 👋🏼
Did you manage to solve this or figure out where the problem is? I agree with you on that it seems to be a problem parsing the data to generate the table. @driesvints maybe Horizon changed the data format on some of it's updates and the old data in Redis is no longer compatible with the frontend?

I've just tweaked my Horizon config to use Redis db 2, that one is now only used by Horizon, and suddenly have 1 failed job showing on the list. It could be that 1 job is a tiny amount of data or it could also be related to the parsing theory (cause that failed job is from just a couple of minutes ago and it's the only failed job in the entire Redis db).

[shawnhind]

@fgilio no I haven't been able to solve this. It still occurs intermittently.

I don't think it's an update to the Redis data format being the problem though. In my experience it will occur for a few days, then suddenly will be working again, and then will not display again for a few days.

I'm thinking that some type of data that my program is generating for jobs is serializing correctly to Redis but is not being able to be displayed by the horizon component. And it starts working again once that incorrect job is out of the "recent range". I can't really test this though because I'm not sure which data is causing the problems. The strange part though, if this is the case is that the recent jobs list has no problems displaying it and like @fgilio has found, when searching for the tag or viewing the failed job in the recent list I can even click and open the detailed view of the failed job.

[driesvints]

@fgilio @shawnhind which laravel, horizon & redis versions are you using?

[fgilio]

Laravel 5.7.22, Horizon 2.0.3 and Redis 5.0.3.
Manually removed Horizon assets and re published last time I updated.

[shawnhind]

I'm using Laravel 5.6 and horizon 1.4.3. We are looking into updating our production laravel to 5.7 soon but haven't yet. I know you guys don't support 5.6 anymore though. I'm not sure which version of Redis but it's the AWS Elasticache cache.m3.large Engine Version 2.8.24.

[driesvints]

Thanks.

@shawnhind neither Horizon 1 so it's indeed best if you upgrade.

@fgilio have you tried purging all data from redis, purging cache, restarting servers and horizon processes etc?

[fgilio]

Yes, I kinda just did that.
A couple of minutes ago I deployed to production this changes 👇🏼 I talked about in a prev comment:

I've just tweaked my Horizon config to use Redis db 2, that one is now only used by Horizon, and suddenly have 1 failed job showing on the list. It could be that 1 job is a tiny amount of data or it could also be related to the parsing theory (cause that failed job is from just a couple of minutes ago and it's the only failed job in the entire Redis db).

I had to manually restart the Supervisord Daemon in forge cause Horizon was not booting up, had to do the same when deployed those changes in staging.

The failed jobs log of the last 7 days is now lost and there's still no new failed job, so I'll keep an eye on this and let you guys know how it's working in the coming days.

[derekmd]

RedisJobRepository@getJobs() removes some failed job results that are missing the expected payload format. This can cause the API endpoint to return a PHP associative array. So the JSON body becomes a JavaScript object rather than the expected Array that resources/js/pages/Failed/Index.vue references.

jobs.length JS conditionals are failing because an object doesn't have a .length property.

Example Ajax response:

>> response.data.jobs
{0: {…}, 1: {…}, 2: {…}, 3: {…}, 4: {…}, 5: {…}, 6: {…}, 7: {…}, 8: {…}, 9: {…}, 10: {…},
11: {…}, 12: {…}, 13: {…}, 14: {…}, 15: {…}, 16: {…}, 17: {…}, 18: {…}, 19: {…}, 20: {…},
21: {…}, 22: {…}, 23: {…}, 24: {…}, 25: {…}, 26: {…}, 27: {…}, 28: {…}, 29: {…}, 30: {…},
31: {…}, 32: {…}, 33: {…}, 34: {…}, 35: {…}, 36: {…}, 37: {…}, 38: {…}, 39: {…}, 40: {…},
41: {…}, 42: {…}, 43: {…}, 44: {…}, 45: {…}, 46: {…}, 48: {…}, 49: {…}}

>> typeof response.data.jobs
"object"

>> _.size(response.data.jobs)
49

>> response.data.jobs.length
undefined

^ index 47 was excluded from the results due to this Collection::filter() call:

    public function getJobs(array $ids, $indexFrom = 0)
    {
        $jobs = $this->connection()->pipeline(function ($pipe) use ($ids) {
            foreach ($ids as $id) {
                $pipe->hmget($id, $this->keys);
            }
        });

        return $this->indexJobs(collect($jobs)->filter(function ($job) {
            $job = is_array($job) ? array_values($job) : null;

            return is_array($job) && $job[0] !== null;
        }), $indexFrom);
       // ^ no longer 0-indexed

When inspecting that bad payload on production, I found the excluded $job is a 0-indexed array of 11 null values from a range request for RedisJobRepository@keys:

[
  0 => null, 1 => null, 2 => null, 3 => null, 4 => null, 5 => null,
  6 => null, 7 => null, 8 => null, 9 => null, 10 => null,
]

Fix

Collection::filter() changes 0-indexed results into an associative array so call Collection::values() to force it back to that.

This testing attempt didn't make Redis return the ZRANGE / HMSET results I expected cause this problem. I wasn't able to setup any scenario using Horizon's RedisJobRepository@failed() or direct Redis calls (HMSET doesn't accept a null key) to form an all-null $job. So the paginated Collection fix seems to be the best we can do.

[fgilio]

This is awesome, thanks!