Pkp-lib: Extend native import/export plugin to include additional entities

[asmecher]

Such as (probably not canonical)...

• Not-yet-published files
• Review assignments
• Discussions/queries
• Log entries
• Journal configuration/settings (this might involve diminishing returns)
• ... etc

[asmecher]

(FYI, @jmacgreg)

[jmacgreg]

Oh, nice. CCing @mfelczak as well. Have we discussed the potential for using this as a PLN deposit option too? Should we ping Mark?

[asmecher]

The PLN plugin already uses the native import/export. This would extend its capabilities, at the risk of increasing storage requirements, and I could see us going either way (having the PLN plugin exclude the additional data and stick closer to its core mission, or allowing the additional data in to ease the "restoration" process). @mjordan, any preference?

[mjordan]

No objections here. If we don't want to load some of the new elements into an access version of a preserved journal, some preprocessing could strip them out. Or perhaps more sensibly, we preprocess the XML to be loaded to only include galleys, metadata, etc.

[diegoabadan]

See https://github.com/lepidus/fullJournalTransfer (for OJS 2.x)

[asmecher]

@diegoabadan, thanks, that's been a useful bit of work for OJS 2.x.

For 3.x, the XML import/export plugin uses the filter framework, which essentially allows each entity to have a matching conversion filter from XML, and to XML. This will facilitate modular code -- the OJS 2.x XML import/export plugin was overly monolithic.

Do you have intentions to adapt any of the full journal transfer plugin to OJS 3.x? If so, we should combine efforts.

[mjordan]

Will this change apply to OJS 3 only, or also OJS 2? I'm asking because, having consulted with PKP Legal, we don't want to store anything that was never published in the PN or even on the staging server. Which means that the filtering out of the published stuff needs to happen within the OJS plugin.

[asmecher]

Yes, this would be OJS3. Filtering would basically involve stripping out the extra entities, either via XSL or a bit of PHP. (Having my hands in a bit of XSL these days, I'll heartily recommend PHP.)

[mjordan]

@asmecher would it make sense for Dimitris to add an integration point for this now, before we release the 3.x plugin? The callback (or whatever) to do the actual filtering could then be added later (and tested as the changes to the export XML are being developed).

[asmecher]

I don't think so -- the plugin will get installed via the plugin gallery, and we can designate plugin version compatibility with each release, and updating the filter is essentially the same as updating the plugin (unless we're imagining hosting an .xsl e.g. on a PKP server, which would be a worse kind of dependency hell, I think).

[mjordan]

OK, thanks, I thought I'd ask in case it saved us some work and testing later. But just so I understand, the plugin will get installed in the Gallery initially, then included in the next release, correct? And it can be enabled either in the plugins list or directly in the Archiving tab. That probably doesn't matter but I just want to confirm that.

[asmecher]

I'd like to keep it in the Plugin Gallery throughout. We have 80 or so plugins already included in OJS and need to decrease that -- moving something to the plugin gallery is not an indication of lesser importance.

[diegoabadan]

Thanks for the info Alec!

It is not in our short term planning. We want to prioritize other plugins before.

[mjordan]

This thread in the forums is relevant. We probably should do the work to munge personal info before we release the 3.x version of the PN plugin.

[defstat]

@asmecher for reviewRounds/ReviewAssignments

PRs
pkp-lib: pkp/pkp-lib#3722
ojs: pkp/ojs#1978

On the same branches, coming up Queries/QueryParticipants/Notes

[asmecher]

Looks good, @defstat -- I've reviewed both PRs with details.

[albig]

Dear @defstat, dear @asmecher,

we've just discussed the same requirements on the OJS workshop in Heidelberg today. And I wonder, what's the current state of the this issue and the provided pull requests? Is this a good approach to follow and to contribute to?

[asmecher]

Hi @albig!

For the OJS/OMP 3.2 release (due within a few weeks), the only changes to the import/export XML will be 1) to support metadata versioning (https://github.com/pkp/pkp-lib/issues/4880), and 2) to add more options for including/referencing file content (https://github.com/pkp/pkp-lib/pull/5524).

I don't believe the PRs above have seen attention for a while, but @defstat can update you on the specifics. There have been some underlying changes (e.g. SchemaDAO and related code) that will need consideration, and the may also open some better opportunities for including entities in import/export XML.

Are there particular entities you're interested in including in the import/export XML?

[diegoabadan]

I believe that next week we can send a PR to resolve the #4195.

[asmecher]

That sounds great, @diegoabadan! We're planning to release OJS/OMP 3.2 at the end of February, and I can't promise we can get your PR included, but we'll try.

[KBodarwe]

Hello,
I was wondering what the status of the project is? As of 3.2 the feature does not seem to have made it into the release.
Thanks!

[asmecher]

@KBodarwe, this will be a bit of a long-term project, and we're hoping to achieve it by aligning the import/export tools with the API rather than trying to maintain parallel implementations. Is there a particular entity you're interested in supporting in the import/export toolset?

[SantiMaceri]

Hello @asmecher , i'm currently working on my thesis project, the main goal its to extend the native plugin so it can export (and import) more information than it does right now (like discussions, all the review stage, etc) for version 3.2.1-1.
I am analyzing the actual code and the xmls schemas so I can figurate out how to extend it.
Any documentation about how the plugin works (especially on filters) will be helpful.
Thanks.

[asmecher]

Hi @SantiMaceri! Thanks for your offer -- we'd like to take you up on it, but before you start, I'd like to make sure we're also not digging into technical debt where we can avoid it. Is there a specific problem you need to solve with the import/export tools that requires more entities to be added?

[SantiMaceri]

Thanks for the quick answer @asmecher .
I must clarify that im working along side @gonetil, he's my thesis professor.
We would like to give the chance to different journals to migrate a bigger part from a submission/article than they have right now.
To be more specific, we would like to add:

• The participants that work in the submission workflow

• All the discussions from all stages.

• The entire revision workflow

Currently, i'm making progress with the participants. Next, it'd be the discussions, then the revision workflow.
Again, if you have any type of documentation that speeds up de project, it'd be fantastic.
And, if in any case you want to contact us to talk about the project, its also welcome. This is a thesis project, but we also want to contribute.

[dennmuel]

Hi all!

Sorry to just bump this issue, but I stumbled upon it while looking for a way to transfer a journal from one instance to another, ideally migrating more than "just" users, issues and articles (particularly the three entities mentioned by @SantiMaceri). Has there been progress "behind the scenes" or is there any other news about this?

Best regards
Dennis

[SantiMaceri]

Hi @dennmuel !
Yes! Along side @gonetil, we managed to extend the native plugin to export the three points that i've mentioned. (It's a work in progress). In fact, we're writing the thesis about this work right now.
In the following months we'are going to test it with portals from our university, and if it's ready we're going to upload it to a public repository.

Cheers
Santiago

[dennmuel]

Thanks for getting back so quickly, @SantiMaceri! Nice to hear it, please keep me posted. Until then: good luck for your thesis! :)

[defstat]

Hi @SantiMaceri! Is there a PR that we could test/review regarding that?