Misp: Updated Feed Scheduler not fetching feeds

[FurqanKhan1]

Feed Scheduler ,not pulling feeds

Work environment

| Questions | Answers
|---------------------------|--------------------
| Type of issue | Bug
| OS version (server) | CentOS
| OS version (client) | 7
| PHP version | 7.0,
| MISP version / git hash | 2.4.XX, hash of the commit
| Browser | If applicable

Expected behavior

I guess there has been a latest commit 12 days ago ,which has had some updates on Pull feeds ,(scheduler to automatically pull feeds.The commit id is :131f2cbf2ef12e6aa2ec8c3a12ec527a9b84267f ,with changes being done at /:
app/Model/Tasks.php
Command/ServerShell.php
Controllor/TasksControllor.php

However the updated code ,at Model/Tasks.php induces the type as fetch_feeds ,inplace of earlier pull_feeds ,how ever ,when we schedule a pull from user interface and the control flows to taskcontrollor.php :,the logic here doesnt seem to be updated and does not seem to schedule a task /job for the new type : fetch_feeds ,instead has old logic for pull_feeds
under the method _jobscheduler ,as given under 👍
private function _jobScheduler($type, $timestamp, $id) {
if ($type === 'cache_exports') $this->_cacheScheduler($timestamp, $id);
if ($type === 'pull_all') $this->_pullScheduler($timestamp, $id);
if ($type === 'push_all') $this->_pushScheduler($timestamp, $id);
if ($type === 'cache_feeds') $this->_feedScheduler($timestamp, $id, 1);
if ($type === 'pull_feeds') $this->_feedScheduler($timestamp, $id, 0);
}

Because of this ,the type fetch_feed ,is never put on the queue ,i think ,inplace of if ($type === 'pull_feeds') ,it should be if ($type === 'fetch_feeds') $

Actual behavior

Instead of the task being schedulked ,nothing would happen and ,on the UI ,we get a message Not scheduled yet.

Steps to reproduce the behavior

With the latest code base ,try to schedule a feed pull task .

Logs, screenshots, configuration dump, ...

[davidonzo]

I can confirm the bug.

[vedd3r]

I confirm the bug as well. Both MISP instances that I have are not fetching feeds and displayed "Not scheduled yet."

[davidonzo]

Just updated to the latest version (2.4.81) and nothing changed.
I should post the same screenshot above.

[Spj28]

I am having same issue. Installed 2.4.81 and unable to schedule fetching of feeds.

[OopsIMadeStool]

I am also having this issue - other tasks appear to get picked up correctly, only fetch_feeds seems to be broken.

Edit: I've tried squaring all instances of "pull_feeds" and "fetch_feeds" up, and manually updated the Tasks table in the database, but the scheduler still isn't picking the pull/fetch feeds task up. The scheduler recognizes when the scheduled time passes now though, and increments the next run time, which it wasn't doing before.

[lucamemini]

I tried to change this line:
app/Controller/TasksController.php
if ($type === 'fetch_feeds') $this->_feedScheduler($timestamp, $id, 0);
//if ($type === 'pull_feeds') $this->_feedScheduler($timestamp, $id, 0);

app/Controller/FeedsController.php
'job_type' => 'fetch_feeds',
//'job_type' => 'fetch_feed',

With this change feed fetch work for me... ;-)

[davidonzo]

Confirmed. It works for me as @lucamemini said.

[iglocska]

Thanks a lot for fixing and verifying it!

[StefanKelm]

I'm having the same issue, albeit with _cache_feeds_. The GUI claims that the task is "Not scheduled yet.". However, _Scheduled Time_ and _Next Run_ are being updated properly and resque-2018-01-03.log contains:

[2018-01-03 13:44:05] main.ERROR: {"queue":"default","id":"748e0b4d500775b0f06c434e26820c7b","class":"ServerShell","args":[{"0":"enqueueFeedCache","1":1514987040,"2":"2","3":"4","s_time":1514975087}]} failed: SQLSTATE[HY000]: General error: 1364 Field 'job_input' doesn't have a default value {"type":"fail","log":"SQLSTATE[HY000]: General error: 1364 Field 'job_input' doesn't have a default value","job_id":"748e0b4d500775b0f06c434e26820c7b","time":55,"worker":"host:3832"} []

[rommelfs]

seems to not be fixed (via #2781)

[davidonzo]

Tested using the latest VM version availables (2.4.85).
Both feeds and cache schedulers work fine.

What version are you using?
Did you check the worker tab in the server settings page? They should be up and running.

[StefanKelm]

@davidonzo I'm running 2.4.85 (not the latest _git pull_ though) and yes, all the workers are fine (otherwise I wouldn't see that message in resque-2018-01-03.log, correct?).

I'm fairly certain that the following should give a clue (at least to people who are smarter than me...):

failed: SQLSTATE[HY000]: General error: 1364 Field 'job_input' doesn't have a default value

[MattCarothers]

The error message occurs because the field job_input in the jobs table has a "NOT NULL" restriction on it with no default value specified. So when MISP tries to write a line to the table and provides a null value for the job_input, the error occurs.

I installed a fresh copy of 2.4.86 in a new docker container, and the feed cache job works fine. The job_input field gets an empty string instead of null. I'm not sure what the difference is between my two MISPs, but I fixed the problem by explicitly setting job_input to an empty string in the enqueueFeedCache() method of app/Console/Command/ServerShell.php.

Old:

$data = array(
    'worker' => 'default',
    'job_type' => 'feed_cache',
    'retries' => 0,
    'org' => $user['Organisation']['name'],
    'org_id' => $user['org_id'],
    'process_id' => 'Part of scheduled feed caching',
    'message' => 'Caching.',
);

New:

$data = array(
    'worker' => 'default',
    'job_type' => 'feed_cache',
    'job_input' => '',
    'retries' => 0,
    'org' => $user['Organisation']['name'],
    'org_id' => $user['org_id'],
    'process_id' => 'Part of scheduled feed caching',
    'message' => 'Caching.',
);

I submitted a pull request for the fix.

[iglocska]

Merged, thanks!

[dmolina213]

I am new to misp. When i try to fetch a feed, it give me an internal error on the gui.

help,

thanks,

Doug