Pkp-lib: reload scheduled tasks function for the acron plugin

[bozana]

Add the Reload Scheduled Tasks function for the Acron plugin in OJS 3 -- as it was in OJS 2.4.

Also, enable AcronPlugin::parseCronTab for DOAJ plugin.

[bozana]

PRs:
pkp-lib master: https://github.com/pkp/pkp-lib/pull/2545
ojs master: https://github.com/pkp/ojs/pull/1396

[bozana]

@ajnyga, applying these two patches and then reloading the scheduled tasks (Settings > Website > Plugins > Generic Plugins > Acron Plugin > Reload Scheduled Tasks) should add DOAJ to the list of the scheduled tasks and execute it. Thanks!!!

[ajnyga]

Hi @bozana, I am getting:
[Mon May 22 17:38:24 2017] [error] [client 192.1xxx9] PHP Fatal error: Class 'AjaxAction' not found in /public_html/lib/pkp/plugins/generic/acron/PKPAcronPlugin.inc.php on line 94

[ajnyga]

Also now when used manually, for example mark registred function, I get:
[Mon May 22 17:47:46 2017] [error] [client 192.1zddd49] PHP Fatal error: Call to undefined method PublishedArticleDAO::getByArticleId() in /classes/plugins/PubObjectsExportPlugin.inc.php on line 593

ah ok, so there are some other changes maybe as well that my installation does not support...

[bozana]

@ajnyga, thanks a lot! The first error is my fault :-\
Concerning the second error: did you copied the whole file PubObjectsExportPlugin or just the changes? Becuase that function was called differently in 3.0.2 but was then changed, not it is called like that.
The fix for the first error is coming :-)

[ajnyga]

@bozana thanks

yes I did, do you think that I could just do the two changes visible here and leave the rest as it is?
https://github.com/pkp/ojs/pull/1396/files

During this week I will try to go through all the local changes I have made since first 3.0.2 release and see that if I could switch to curretn 3.0.2 stable. At the moment my installation is something between 3.02. stable and the original 3.0.2. + some local changes that are planned for 3.1. So I have to track down the latter before switching to 3.0.2 stable. Not nice.

[bozana]

@ajnyga, yes, only the change of the register function in those two OJS classes + the two new functions in the lib/pkp/plugins/generic/acron/PKPAcronPlugin (https://github.com/pkp/pkp-lib/pull/2545/files).
I will backport it to ojs-stable-3_0_2, so you can take it from there then i.e. so you don't have to keep track of this one :-)
The fix for the first error is there now...

[ajnyga]

thanks!

I am seeing this now in the access log:
192.xxx- - [22/May/2017:18:31:11 +0300] "POST /$$$call$$$/grid/settings/plugins/settings-plugin-grid/manage?verb=reload&plugin=acronPlugin&category=generic HTTP/1.1" 200

But the plugin_settings table (acronplugin/crontab) is not updated. No visible errors.

edit: funny thing I noticed. If I open a journal backend (for example https://site.fi/journal/management/settings AND then click on the Administration link visible for the site admin, this is the url I get: https://site.fi/journal/admin/index => should that be https://site.fi/index/admin/index?)

[bozana]

@ajnyga, did you also apply the ojs changes, that change of the register function?

Concerning the URL for admin Administarion link: I would say it is OK, but maybe @asmecher or @NateWr have different opinion?

[NateWr]

If I recall, the admin area used to contain some tools that were context-specific. But I think that's no longer the case and I'm pretty sure I've changed the link so that going to the admin always bumps you to /index/admin/index now -- but this change is only in the big branch I'm working on merging soon.

[bozana]

Ah, sorry, I tested it now and it is so already in the master branch :-\ Sorry, sorry, sorry :-(

[ajnyga]

@bozana
I definitely have removed the register function from DOIPubIdExportPlugin.inc.php
And added it to PubObjectsExportPlugin
And as the third change, I edited PKPAcronPlugin

But no matter if I reload in a journal or in index, there are no changes in plugin_settings.

[bozana]

Ah, no wonder @ajnyga, I accidentally deleted the mail line of code changes ($this->_parseCrontab(); is missing in the manage > reload) :-\ Sorry, sorry, sorry :-(
I forced pushed, so could you please copy/patch the PKPAcronPlugin function manage again? THANKS!!!

[ajnyga]

Thanks!

I was wondering where the action should take place, but thought that I just do not understand OJS code that well :)

Reload works now!

However, the automatic deposits do not. I am seeing two issues here, I will post them later. I will have to eat lunch now and do a DOI workshop for our journals. I will look at the errors encountered on post them here later.

[bozana]

OK, thanks a lot!!! :-)

[ajnyga]

I think it is clearer if I do not make any pr's but just post these here:

bug 1: https://github.com/pkp/ojs/blob/master/plugins/importexport/doaj/DOAJInfoSender.inc.php#L43

bug 2: probably a situation in multijournal installations. A journal is exporting article metadata to DOAJ, enables the plugin and gives the API key. However, the Acron plugin is run by a journal that does not have DOAJ API key. This is the error I get:
[2017-05-23 11:42:14] [Warning] Deposit was not successful! The DOAJ API returned an error:: '1 - {"status": "unauthorised", "error": "An API Key is required to access this. (ref: b87b7932-3f93-11e7-86a5-04014a62b401)"}'.

Bug 3: [2017-05-23 11:42:19] [Warning] Deposit was not successful! The DOAJ API returned an error:: '1 - {"status": "bad_request", "error": "bibjson.keywords may only contain a maximum of 6 keywords (ref: bb9354fa-3f93-11e7-b0f5-04014a62b401)"}'.

[bozana]

@ajnyga, thanks a lot!!!
bug 1: clear
bug 2: I do not understand this totally, because actually it shouldn't happen, s. https://github.com/pkp/ojs/blob/master/plugins/importexport/doaj/DOAJInfoSender.inc.php#L79. I will test...
bug 3: :-O So I will only take the first 6 keywords for now...

[ajnyga]

Also, although these are only two warnings, the articles are not getting registered for the journal that has DOAJ enabled. API key is there and is working, I have exported single articles without any problem. But the automatic deposit just does not happen.

It really seems like DOAJ is only running this for the journal that triggers the Acron plugin.

Or could it be that the single article with more than 6 keywords is actully preventing the whole export?

If there is something I could debug, let me know.

[ajnyga]

@bozana nevermind number 2: I looked at our database and a journal has entered an invalid apiKey there! I will remove it and run the task again.

edit: yes only the keyword error left.

[bozana]

Yes, both are deposit errors, I think, and because of that the deposit is not working. I.e. also now it will not work because of the article with more than 6 keywords. If you would wait a second, I will post a fix...

[ajnyga]

weird limitation that 6 keywords.

[bozana]

@ajnyga, in the ojs PR above there is now a new commit (https://github.com/pkp/ojs/pull/1396/commits/5997194a4d96827e62e3a7ddf9a32bd2a1e24e45) -- it should fix the bug 1 and 3.

[ajnyga]

New error:

[2017-05-23 15:47:04] [Warning] Deposit was not successful! The DOAJ API returned an error: '1 - <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
'.

Did a re-run, the same result. Could it be that the deposit is too large. The journal has about 550 unregistered articles.

[bozana]

:-( I do not know :-(

[ajnyga]

@dommitchell do you know if 502 Bad Gateway from the DOAJ API (see above) could be because of a too large deposit?

Background: we are testing the automatic DOAJ deposits from OJS3 using the DOAJ API. The manual deposit where we choose the articles to be deposited and click register works. It also uses the DOAJ API.

However, the automatic deposit run with cron, which is trying to deposit all the unregistered articles fails with the error above. There are around 550 unregistered articles, so could it be that it is too much for the API to handle at once?

[ajnyga]

Maybe @emanuil-tolev could comment the above question?

[emanuil-tolev]

Bad gateway is not an error you should usually get from the API. Thank you for the log with timestamp. What timezone is that server in (so I can translate to GMT+1, London)?

Also I assume your regular deposit task has continued returning 502 bad gateway these past few days - is that right?

---

The keyword limitation is a bit weird and may be a limitation of a much earlier version of DOAJ (perhaps web form input). Is 100 or 1000 more appropriate? Or just none?

[emanuil-tolev]

In order to deposit, you're HTTP POST-ing to the DOAJ. Having the payload that's giving you 502 Bad Gateway would be great. Especially if the error is so consistently reproducible with that specific payload - as it sounds from this issue.

[ajnyga]

Hi,

Thanks @emanuil-tolev.

The times are Finnish time, GMT+2. And you are correct, the scheduled task has triggered during the past few days, the latest was:

[2017-05-28 15:55:02] [Warning] Deposit was not successful! The DOAJ API returned an error: '1 -

502 Bad Gateway

---

nginx/1.4.6 (Ubuntu)


'.
[2017-05-28 15:55:02] [Notice] Task process stopped.

I will contact the editor of the journal and ask her permission to send you the metadata. Thank you!

[ajnyga]

Hi @emanuil-tolev
I have the json encoded data that the plugin is trying to export to DOAJ and a permission from the journal. Where can I send it for you to see?

[emanuil-tolev]

@ajnyga [email protected]

[ajnyga]

thanks @emanuil-tolev, I sent you the data

[ajnyga]

@bozana do you have an opinion to the question concerning the max amount of keywords: https://github.com/pkp/pkp-lib/issues/2543#issuecomment-304587030
I am pretty confident that at least none of our journals will break the limit of 100.

[bozana]

@ajnyga and @emanuil-tolev, it would be great if there would be none restriction for us i.e. if some restriction is necessary, it would be great if DOAJ would then filter i.e. take only the number they need/allow... Would it be possible? THANKS!!!

[ajnyga]

Hi @bozana Emanuil mentioned in an email that he is not actually working during this week, so we can look at this issue next week.

He did not see any apparent reason why the DOAJ server would give that error while it is running without any problems. Just to be sure, am I not correct that the same depositXML function is used when you choose the articles with checkboxes and click Register? I am just trying to rule out any plugin related issue here...

[emanuil-tolev]

Hi all,

This is due to a time-out on DOAJ's side. From the payload provided I can see that the DOAJ Bulk Article create API endpoint is being used: https://doaj.org/api/v1/docs#!/Bulk_API/post_api_v1_bulk_articles . As opposed to sending the articles 1 by 1 to the CRUD Article Create endpoint.

This makes sense, except that for some reason our bulk article create endpoint is being slow processing your request. We're working on profiling the code and making it faster.

If this issue is very urgent and you can't wait for a fix on the DOAJ side, one work-around would be to loop over the articles on the plugin side, and submit each one to the individual article create endpoint: https://doaj.org/api/v1/docs#!/CRUD_Articles/post_api_v1_articles .

EDIT: Of course I'll update this issue when we have a fix live.

[ajnyga]

What do you think @bozana? I am in no hurry with this one. I can just tell the journal editor to register those articles manually and we can then wait for a fix from the DOAJ.

[bozana]

@emanuil-tolev, do you maybe approximately know how much time will that fix need? If @ajnyga does not very much need it, I think we can wait for that fix to come. Just if it is going to take veeeeery long time, I could change it so that we have a functioning version for now...

[bozana]

@emanuil-tolev, sorry to bother you this way -- if there is another/better way, please let me know -- one of our users tried to register their articles and got the "301 Moved Permanently" message. Do you maybe have any idea why, what could be the reason? (I unfortunately have to test account any more, so I cannot test anything :-( )

[emanuil-tolev]

sorry to bother you this way

Not at all!

We've implemented something that will cut processing time in half. I want to also tweak the gunicorn configuration on DOAJ's side a bit to allow for slightly higher time-outs, then I'll retry with the payload you sent me over email. (In case I didn't make it clear above, I can 100% reproduce your issue with the JSON file you sent me.)

I'll get back to you by tomorrow with an update on whether to test again or whether I've discovered further issues.

[emanuil-tolev]

one of our users tried to register their articles and got the "301 Moved Permanently" message. Do you maybe have any idea why, what could be the reason?

That sounds like a different issue. Are they accessing DOAJ via HTTP? All HTTP connections get 301 redirected to the HTTPS equivalent URL.

[ajnyga]

Hi,

The plugin uses https: https://github.com/pkp/ojs/blob/master/plugins/importexport/doaj/DOAJExportPlugin.inc.php#L22

BUT, that change was made only 2 months ago: https://github.com/pkp/ojs/commit/ada5d8840397c88d7e4d810fe3fa93f5e1fc5af1

@bozana, they probably do not have this fix yet?

[bozana]

@emanuil-tolev and @ajnyga, thanks a lot! :-))) I will ask the user to check https change...

[emanuil-tolev]

I'll get back to you by tomorrow with an update on whether to test again or whether I've discovered further issues.

Apologies for the delay. Bulk endpoint processing time is now down by half, and your payload may work, but unfortunately it doesn't seem to do so consistently - how long it takes it depends on server load. If it's under heavy load, the web worker takes too long (despite the recent speedup) and is killed.

Now, I did mention an alternative:

one work-around would be to loop over the articles on the plugin side, and submit each one to the individual article create endpoint: https://doaj.org/api/v1/docs#!/CRUD_Articles/post_api_v1_articles .

Instead of submitting in batches of up to 1000, you just submit each one. It is a lot more requests, but that endpoint will work for sure. Also, the code change is hopefully not much more complex than an additional loop in the plugin.

Is this an option with the number of articles you're managing on this platform?

[bozana]

Thanks @emanuil-tolev! I will try to change the code. Maybe then @ajnyga can test it, so we are sure it works?
I'll come back to you as soon as I did the changes...

[ajnyga]

Sure, but I am not sure if the journal still has all those articles unregistered.

[bozana]

PR:
ojs master: https://github.com/pkp/ojs/pull/1480

@ajnyga, could you maybe test it with your data and api-key? (Note the filter input type change, so that an upgrade or manual change in the DB is necessary).
I have no test account any more, so I am not able to test it properly...

[ajnyga]

Hi, I asked the journal editor a permission and if it is ok, I will test this as soon as possible. Thanks again @bozana!

[ajnyga]

Ok, so I got the permission. Will test this later today next Monday (just could not squeeze this in yesterday) .

[ajnyga]

Hi @bozana,

I am now getting this error: Deposit was not successful! The DOAJ API returned an error: '1 - {"status": "bad_request", "error": "Supplied data was not valid JSON (ref: f71e4b54-8987-11e7-acca-04014a62b401)"}'.

I tested first with a single article and used the "manual" export.

[ajnyga]

Sorry, false alarm. For some reason the change in the filter_groups table was not successful. Re-applied the change and it works now.

I will test the automatic deposist witth all the remaining articles next...

update: not getting the error, but the scheduled task has been running for 10 minutes now...

[ajnyga]

@bozana could it be that the plugin does not respect the "OJS will deposit articles automatically to DOAJ. Please note that this may take a short amount of time after publication to process. You can check for all unregistered articles." selection?

edit: task running now for 30 minutes, no new articles visible in DOAJ database for the journal I am looking at...

[ajnyga]

Ok, so @bozana the scheduled task is not working.

The scheduled task started, but it only has these two lines:

[2017-08-25 14:32:06] http://journal.fi
[2017-08-25 14:32:06] [Notice] Task process started.

If I do SELECT * FROM submission_settings WHERE setting_name = doaj::status AND setting_value = registered, I can see that no new articles are getting registered even though the task should be running. I am able to register an article by selecting it from the DOAJ plugin listing and by pressing the register button, so it is just the scheduled task that does not seem to work?

[ajnyga]

Hi @bozana. There were a couple of small typos in the code, I made a pr: https://github.com/bozana/ojs/pull/5

I can confirm that the automated article registration to DOAJ is now working! All 500+ articles are registered at least from the journal I am looking at. The Task is still running though, but I am not seeing any more errors. What might prevent the task from closing?

tag @emanuil-tolev as well.

[ajnyga]

@bozana I just checked and the task was closed succesfully. The first run just took around 30 minutes. I guess there were other journals with non registered articles as well.

@emanuil-tolev if you like to check that everything worked well in your side, the DOAJ task run between 2017-08-25 22:18:28 and 2017-08-25 22:35:01 Finnish time.

[bozana]

Thanks a lot @ajnyga -- I will then merge those changes....
THANKS. THANKS. THANKS, THANKS. THANKS. THANKS, THANKS. THANKS. THANKS, THANKS. THANKS. THANKS, THANKS. THANKS. THANKS... :-)

[ajnyga]

You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome You're Welcome.