Joss: strengthen FOSS best practices as criterion for joss submissions

[maxheld83]

I recently an R package and came away from the process a) in love the vision of joss, and b) a little confused about the importance of FOSS best practices in joss submissions.

The software I reviewed was, I thought, too needlessly specialised, monolithic in its design, and generally lacking in software dev / FOSS best practices to be of long-term, maintainable use to the community.
(The software essentially worked only on/with the included 4 datasets, and perhaps was more of an interactive vignette overall, than a fully-fledged package).
I worry that these kinds of packages really fragment the (in this case) R ecosystem, and degrade the user experience.
They also underline the stereotype that scientific software is usually pretty poorly written software.

I understand that scientists have a lot of other obligations, and little training in software development (I am in that same situation).

However, if we really want to strengthen open source contributions as proper scientific contributions, I think we should really focus on high quality packages, with good maintainability, modularity, a clear focus and all the other evolved FOSS best practices.

Otherwise, in the extreme, we're risking a situation where people are now gaming "software contributions" with high-frequency, low-quality material, much like they have been gaming "paper contributions", to the detriment of careful scientific software developers and the overall open science enterprise.

Obviously, this is a thin needle to thread: standardise demands too much, and you end up with a calcified system (for example, I wouldn't want to only accept "tidyverse" packages for R, that'd be crazy).

For the review in question, @arfon's decision seems right to me – it'd be unfair to spring such demands on the submitting authors @davidnsousa and @joaordaniel at this point.

And I imagine that you fine people at foss have had plenty of discussions on this issue.

So I just wanted to report my experience here.

In my view, open science might be better off if we give credit for high quality research software, and be very clear about what makes for such software in advance.
Quality before quantity.

[arfon]

Thanks for posting this @maxheld83 - it's always good to hear the experience of reviewers, especially those new to JOSS.

I basically agree with everything you say here, but want to thank you for your support with the decision on https://github.com/openjournals/joss-reviews/issues/911.

However, if we really want to strengthen open source contributions as proper scientific contributions, I think we should really focus on high quality packages, with good maintainability, modularity, a clear focus and all the other evolved FOSS best practices.

Otherwise, in the extreme, we're risking a situation where people are now gaming "software contributions" with high-frequency, low-quality material, much like they have been gaming "paper contributions", to the detriment of careful scientific software developers and the overall open science enterprise.

Obviously, this is a thin needle to thread: standardise demands too much, and you end up with a calcified system (for example, I wouldn't want to only accept "tidyverse" packages for R, that'd be crazy).

Right. This turns out to be quite difficult and our submission criteria have evolved over time. I suspect you've read these already, but if you have any specific feedback on our current submission criteria that would be interesting to hear.

Thanks for all of your help thus far and your thoughtful feedback.

/ cc @openjournals/joss-editors

[maxheld83]

I would have a couple of suggestions 😄.

However, feel free to ignore / close this; I won't take any offense. I'm sure you folks have thought about this a great deal more than I have, and it's always easy to make clever suggestions from the outside.

Just hope this might be helpful.

Ideally, these are the same as general FOSS best practices, but the following seem to me to be particularly important:

• modular, composible design (aka unix philosophy), both internally (expose few, small, well-defined functions with few args) and externally (don't repeat what other packages do better, play nice with other widespread software)
• defensive programming (at least some input validation, ideally a somewhat elegant and thorough exception handling)
• unit tests, and especially important for scientific software, automatic validation against published results (to be sure that some custom algorithm does what we think it does).
• no weirdly specified data input arguments ("tidy" would be too narrow/strict, but things such as matching on column names just shouldn't be done)
• maintainable code with good comments
• at least some effort to solve a computation problem at the appropriate level of abstraction and great skepticism towards super-nichey, but otherwise trivial software. (Because this kind of software needlessly fragments the ecosystem, and will likely be hard to maintain).

I don't think this necessarily means demanding a lot more work, per se – you can save a lot of effort too, by carefully reusing code and focusing your tool.
On occasion, this might mean of course that you can't go it alone, but might rather write up a good PR to an existing project.

[jedbrown]

@maxheld83 I have similar feelings, but don't want to over-prescribe. The scope of JOSS is pretty broad, software is developed/used in many different ways, and different communities have vastly different standards. The verbiage about "maintainable extension" was our attempt to very concisely give reviewers context-dependent discretion about maintainability and suitability for research use. It's okay to be relatively niche if it's significantly contributing to research in that niche. There is expressly _not_ a "novelty" review criteria for JOSS and I suspect that if low-quality/fragmenting software becomes an issue for JOSS, the editorial board may revisit that to make a relative statement about improving software design/best practices for submissions in spaces where other packages exist.

[brainstorm]

I don't want to turn this into a full-on philosophical licensing debate, but what do you reckon would be the best strategy for software that requires non-OSS runtime dependencies? See https://github.com/openjournals/joss-reviews/issues/1104#issuecomment-443712744? I might be wrong but I believe we did not accept submissions with i.e MATLAB dependencies in the past?

[arfon]

I don't want to turn this into a full-on philosophical licensing debate, but what do you reckon would be the best strategy for software that requires non-OSS runtime dependencies? See openjournals/joss-reviews#1104 (comment)? I might be wrong but I believe we did not accept submissions with i.e MATLAB dependencies in the past?

This one we _do_ allow: https://joss.readthedocs.io/en/latest/submitting.html?highlight=prefer#submissions-using-proprietary-languages-dev-environments

[danielskatz]

All software has dependencies, but we hope those dependencies are easy enough to satisfy that we can find reviewers who can run the code.

An example is a code that has a test that runs on a large-scale HPC system, which can only be run by others who have access to that or a a similar system. The dependency here is a collection of possibly commodity components (CPUs, network, disk, etc., which note, are not open and are propriety in design) that are assembled at a scale that is unique or uncommon. If there are reviewers who have access to such as a system, we can review the software. If not, we may not be able to.