Pkp-lib: Strip out citation plugins and replace with CSL

[jalperin]

It should be pretty trivial to remove all the citation templates and replace them with CSL.

Using citeproc-js we shouldl be able to take a generic bibtex file, and just put it through citeproc to generate any of the other formats.

[asmecher]

Or we could do this PHP-side, using https://bitbucket.org/rjerome/citeproc-php.

[NateWr]

I've done some work on this to replace the How-To-Cite article block, which you can see in the following branches:

https://github.com/NateWr/pkp-lib/tree/i723_citation_formats
https://github.com/NateWr/ojs/tree/i723_citation_formats

I opted for the JS-based tool because the PHP one seems to be no longer in development. The work-in-progress implementation I've done feels very clunky, and definitely not "pretty trivial".

Because it's a JS-only processor, we can only generate the citations in-browser. I had to jump through quite a few hoops to at least get an initial default render of the citation without extra HTTP requests. Trying to find a way to manage this that won't require much configuration from one theme to the next is a bit of a challenge.

Before I go further, I wanted to clarify some things:

1. Do we plan to use this anywhere except the how-to-cite block on the frontend?

2. How should journal managers configure what citation formats are available in the how-to-cite block? Is this just another control to add to our settings pages?

3. What citation formats should we allow them to choose from?

[NateWr]

Oh, one thing I forgot to mention is that in order to keep the HTTP requests down, we require one adjustment to the citeproc-js lib. I'm hoping the developer will be receptive to the idea:

https://github.com/Juris-M/citeproc-js/issues/41

[NateWr]

Also (sorry), Alex said they're using Pandoc for the XML typesetting stack. Not sure if that kind of server dependency is an option for us.

I think a part of me would rather have a server-side option for this, just because when we hand off to the frontend, custom themes can do whatever, and it we'll run into compatibility issues over time I'm sure.

[asmecher]

Hrrrrrrmmm, this library is ridiculously big -- and not just the CSL database. We can use the build script to exclude stuff from the .tar.gz, but for people (including us) who are hosting from git, that'll be no help.

It looks to me (at a glance) like we just need some locale files, the main citeproc.js, and the .csl files. I don't see any sign of a "production" packaging for this -- ideal would be something for Composer e.g. on packagist.org, but as that's a PHP-centric platform, only the biggest JS-land tools have made it there.

@NateWr, if you find the author of that toolkit to be responsive, maybe he has some ideas about packaging that would be useful for us too?

[asmecher]

(I'm willing to bet you have some ideas e.g. WRT grunt and the like arising from your JS work. All ears on that front too.)

[NateWr]

Yeah I felt the same. Typically, a JS project will have a package.json file which we can use for better project management, including pulling release packages.

[asmecher]

(One other option that I'm not fond of, but just for completeness: OTS already includes CSL transformations as part of its service, so potentially we could delegate the transformation to a PKP-hosted service)

[NateWr]

I don't think we should have an external dependency for something that appears on the frontend of a journal's site. It feels like it underminds the intent of open-source and, also, think how angry everyone will be when the PKP-hosted service goes down for a day.

[asmecher]

Yep, fully agreed.

[marcbria]

I arrive to this issue from this conversation and perspective:
http://forum.pkp.sfu.ca/t/plugin-for-displaying-jats-xml-on-article-detail-page-under-abstract-for-ojs3/31053/23?u=marc

The project is very active and stable and is the source for zotero, mendelay or islandora:
http://citationstyles.org
--> http://citationstyles.org/styles

The php "transformer" (citeproc-php) is quite old but we don't expect much changes in future parsing the Citation Style Language, isn't it? I mean, CSL definition changes will be a problem for the entire citationstyles.org project...

Instead of including every citation format is it possible to let each journal pick the formats they use?

[marcbria]

citeproc-php 1.0 - updated fork in github mantained by islandora: https://github.com/Islandora/citeproc-php
citeproc-php 2.0 - Very recent refactoring (CSL 1.0.1): https://github.com/seboettg/citeproc-php

[NateWr]

@asmecher What do you think about either of the options @marcbria indicated above? Are we comfortable hitching our wagon to either of them?

On the one hand, 2.0 looks like it's the original creator revamping it. But I don't know what that means for longevity or stability.

On the other hand, are we more confident working with a copy that Islandora is using?

I'd be happy to take another stab at this, and doing things on the PHP side makes a little more sense for our frontend setup.

[asmecher]

@NateWr, neither of those looks hugely stable -- glancing e.g. at the number of stars/forks, contributors, and the commit history. Maybe take a browse of what's available over at https://packagist.org/search/?q=csl as another data point?

Low FOSS metrics don't disqualify any of those from use -- and I wouldn't expect a CSL implementation to be as antnest-like an undertaking as e.g. an application. If you prefer one usage API over the other, go for it.

[NateWr]

Done:

• Removed all citationFormat plugins
• Replaced with a new citationStyleLanguage plugin.
• Update the How-to-Cite block to fit new pattern.
• Support CSL and template-based citation formats.
• Support rendered and downloadable formats.
• Settings panel (accessed from Plugins menu) allows you to select enabled styles and select a "primary" style, which the citation will be rendered in by default.

Additional comments:

• Hooks are available to write plugins where you add your own CSL/template-based formats.
• New CSL formats (.csl) can be searched for and downloaded in the handy CSL browser.
• Many CSL formats appear to be incomplete, or poorly formatted. Just a heads-up. (There are thousands of them.)
• Version details should be added to the citation output when versioning goes in. (Hi @lilients! Here is where the citation data is compiled. The CSL standard uses a version key and expects a string.)

@asmecher citeproc-php and it's dependencies weigh in at 7.8mb. Does composer have a way of compiling production dependencies? Or what is the normal course of action in PHP world for bundling the compiled lib?

PRs:
https://github.com/pkp/ojs/pull/1498
https://github.com/pkp/pkp-lib/pull/2718

[ajnyga]

You were maybe thinking of tagging @lilients ?

[asmecher]

@asmecher citeproc-php and it's dependencies weigh in at 7.8mb. Does composer have a way of compiling production dependencies? Or what is the normal course of action in PHP world for bundling the compiled lib?

Does running Composer with the --no-dev option help at all?

[NateWr]

You were maybe thinking of tagging @lilients?

I told you I get your usernames mixed up! :)

[NateWr]

Does running Composer with the --no-dev option help at all?

Yes! Down to 2.1mb. Most of that is in citeproc-php and 1mb of it is in the citeproc-php tests. I looked at the project's releases but these are just archives of the full project source.

Would it be worth approaching him and seeing if he'll consider a release package for production? Or would this be something we add to the exclusions in buildpkg.sh?

[mtub]

Just from my bystander's perspective: This makes me happy, good to see this completed 👍

[asmecher]

I'd suggest we just start using the --no-dev option in our build script (tools/buildpkg.sh). That'll trim down other dependencies considerably, and we should be excluding dev dependencies from the distrib packages anyway.

[NateWr]

Ok, would you like to see that change as part of a separate PR or this one?

[asmecher]

Fine to sneak it in here, thanks!

[NateWr]

Snuck it in. Tests are running. :+1:

[NateWr]

@asmecher the tests are failing on some XML validation in the composer dependencies for citeproc-php. Where do we define files to exclude from XML validation? I think you've told me this before but I can't remember.

[NateWr]

@asmecher Also, the PHP tests are failing to find Guzzle for the unit tests. https://travis-ci.org/pkp/ojs/jobs/267147800#L2308

Could this be related to adding the --no-dev option to our buildpkg.sh composer command?

[asmecher]

The buildpkg.sh script isn't used for Travis testing -- the Composer dependencies are installed in .travis.yml without --no-dev. That failure looks like a temporary network error of some sort; I'd suggest restarting and seeing if it behaves differently.

[NateWr]

@asmecher it turns out that it's failing because PHPUnit is trying to run tests in the composer dependencies of citeproc-php:

https://travis-ci.org/pkp/ojs/jobs/267996944#L2298

I thought I found the place to exclude these tests, but it doesn't seem to have worked:

https://github.com/pkp/pkp-lib/pull/2718/commits/0a7b803fc0103a6eece9c9a5a54d61b2be0d24c9

Do you know how I'd go about telling PHPUnit not to run any tests under /plugins/generic/citationStyleLanguage/lib/vendor?

[asmecher]

Rather than excluding it the way you've done above (which as you say isn't working), try specifying the test directories less clumsily in the phpunit command lines. I'll bet specifying lib/pkp/plugins/*/*/tests instead of lib/pkp/plugins in https://github.com/pkp/pkp-lib/blob/ojs-3_0_2-0/tools/runAllTests.sh#L182 (and the equivalent for the app plugins path) would do it.

[NateWr]

Tests passed! @asmecher would you code review?

PRs:
https://github.com/pkp/ojs/pull/1498
https://github.com/pkp/pkp-lib/pull/2718

[asmecher]

@NateWr, this improves functionality considerably, removes a huge amount of code from our responsibility, and integrates cleanly with all our existing quirks. Great stuff. I've added a bunch of comments to individual commits (including the new plugin) -- please carry on.

[NateWr]

Thanks @asmecher! I addressed a bunch of issues and just had a few additional questions in the comments.

[NateWr]

@asmecher The outstanding comments are scattered throughout PRs and the new plugin, so it's not that easy to track them. Here's a list of outstanding questions:

https://github.com/pkp/pkp-lib/pull/2718#discussion_r136289935
https://github.com/pkp/citationStyleLanguage/commit/63076950ed0a65e9f66d22458f90732e746ff2db#commitcomment-23998106
https://github.com/pkp/citationStyleLanguage/commit/63076950ed0a65e9f66d22458f90732e746ff2db#commitcomment-24001033
https://github.com/pkp/citationStyleLanguage/commit/0ec26400b1e82ea4fad203e063434d60613ffdce#commitcomment-23997117
https://github.com/pkp/citationStyleLanguage/commit/0ec26400b1e82ea4fad203e063434d60613ffdce#commitcomment-23997940

[NateWr]

Ok, I think I've addressed all comments. It should be good to merge once the tests pass:

https://github.com/pkp/pkp-lib/pull/2718
https://github.com/pkp/ojs/pull/1498

[asmecher]

@NateWr, this will need porting to OMP too, I think.

[NateWr]

@asmecher I don't believe we have a how-to-cite block in OMP or any of the citation format plugins in place there. What would need porting?

[asmecher]

You're absolutely right, OMP never did have this! Please go ahead.

[NateWr]

Merged! :tada:

[asmecher]

:tada: indeed! Another big merge.

[marcbria]

Reading you from the dark... ;-) Congrats!! Great job done!