Joss-reviews: Research/academic quality of submission

[pjotrp]

I was just looking a newly submitted paper and wondered whether it actually addresses a research question, i.e., its academic value. I think we should add a standard reviewer question that says something about that. It is a box that needs to be ticked if we want to be taken seriously as a scientific journal. We don't want JOSS to end up as a basket for software projects that have no academic value.

[arfon]

I was just looking a newly submitted paper and wondered whether it actually addresses a research question, i.e., its academic value.

We already have this listed as a requirement in the submission guidelines but I agree it should probably also be in reviewer checklist.

As for the most recent submission in question (#38), I contacted the author about this exact question, that is, whether this software has a research application. Here is his response:

Well, it has a business application that has been researched. In essence, the SPDX community has not focused on how to integrate the generation of SPDX docs into the business workflow. DoSOCS does this. As a result of our work, we've been able to work with organizations in the implementation of DoSOCS. This has led to a deeper understanding of the SPDX spec in practice, resulted in our own contributions to the community, and academic papers about this very issue.

So does this fit the current author requirement of:

• Your software should have a research application

I'm not sure it does completely. Rather than reject this submission I'd rather we revisited ☝️ the language of this requirement as I'm not sure the language is good enough.

When authoring this requirement my goal was to make sure we had a good filter in place so that the only software being reviewed in JOSS had a research application and could be reasonably classified as 'research software'. In my opinion this software _does_ meet this criteria as it's the _product of academic research_.

While in this case the research software may not have an ongoing research application, it represents part of the collection of products from this academic group when conducting this research study. As such I think publishing this software in JOSS is firmly in line with the open science mission of this journal.

/ cc @openjournals/joss-editors

[danielskatz]

When authoring this requirement my goal was to make sure we had a good filter in place so that the only software being reviewed in JOSS had a research application and could be reasonably classified as 'research software'. In my opinion this software does meet this criteria as it's the product of academic research.

I'm a little concerned with this definition. Does is mean that we need the developers to be in academia?

On the other hand, I'm not as much concerned with what the application of the software is now, whether it's research of business, as if there's a case to be made that the software could help research (even if it's not being used for that now)

And finally, I think we should be fairly open in general to any software that
1 - the developers want to publish
2 - shows something interesting in what was done or how it was done

I guess this last part is the tricky one to define, and I'm sure I can't do it well.

[arfon]

I'm a little concerned with this definition. Does is mean that we need the developers to be in academia?

Fair point. That's not really a restriction I was looking to impose with that language 😞

On the other hand, I'm not as much concerned with what the application of the software is now, whether it's research of business, as if there's a case to be made that the software could help research (even if it's not being used for that now)

Can I ask, what do you think about this particular submission @danielskatz?

And finally, I think we should be fairly open in general to any software that
1 - the developers want to publish
2 - shows something interesting in what was done or how it was done

Right. I've wondered about an alternative approach here which is more along the lines of 'if this has a potential career benefit for the individual' (and it meets all of the other criteria) then we should accept it for review.

[pjotrp]

'if this has a potential career benefit for the individual' (and it meets all of the other criteria) then we should accept it for review.

I don't think this is a great idea. Personally I favour the idea of supporting people building a scientific track record through code. If we include just any project that idea is watered down and we will lose the scientists. At least there will be room for another journal ;)

[arfon]

Personally I favour the idea of supporting people building a scientific track record through code.

I think this is what I mean by "if this has a potential career benefit for the individual" - i.e. this is someone for whom building an academic/research profile is part of their job.

[arfon]

Personally I favour the idea of supporting people building a scientific track record through code.

Also, with this definition I think that the submission in question _is acceptable_. Would you agree @pjotrp? This software is an outcome/product of the research of this group.

[pjotrp]

If that is the case it should be fine. I would like the authors to respond about this in the issue tracker so it is clear to others. Also I would like to warn the authors that this publication will be visible forever and they should be convinced it will support their research track record in the future. Lack of research quality and/or code quality may be damaging to a career.

[pjotrp]

This may actually be a good strategy. Have the authors explicitly tell us in review that they are convinced this is a worthwhile contribution, that they take pride in their achievement and that publication will support their track record. I.e., what I want to avoid is the short term gain. We should all have a long term view here - authors, reviewers and editors.

[danielskatz]

I think this software is acceptable. I would like to see the paper explain what some uses of the software are, but this is hard to judge without the paper itself.

[arfon]

If that is the case it should be fine. I would like the authors to respond about this in the issue tracker so it is clear to others.

So what about this: If the origin of the software & motivation for publishing isn't clear from the paper then we could have an optional review step where we invite the authors to describe the origin/motivation in the review issue.

[germonprez]

Hi all. I'm the lead for the DoSOCS project in question. A brief history of this project:

1) Research program funded by the NSF in 2011 to perform field work to explore corporate engagement with open source communities
2) As part of this field work, we aimed to become contributing members with the communities we were aiming to learn from. We didn't want to simply scrape discussion data or bug fix data. We wanted to be true participant observers to understand corporate engagement with open source communities.
3) In doing so, we developed DoSOCS. DoSOCS is a product of our desire to understand our larger research questions. It is certainly a viable open source tool as we have presented it at various Linux Foundation events. It is a viable business tool as we have employed it within corporations. It is also a methodological tool as it has served as an important way for us to communicate with the communities we wanted to learn from.

I would not call DoSOCS a piece of scientific software. I would certainly say that DoSOCS has been an integral part of scientific project. Some directly related publications and presentations:

Germonprez, M., Kendall, K., Kendall, J., Mathiassen, L., Young, B., and Warner, B. (2016). A Theory of Responsive Design: A Field Study of Corporate Engagement with Open Source Communities, Accepted at Information Systems Research (ISR).

Germonprez, M., Kendall, J., Kendall, K., and Young, B. (2014). Collectivism, Creativity, Competition, and Control in Open Source Software Development: Reflections on the Emergent Governance of the SPDX Working Group, International Journal of Information Systems and Management, 1(1), 125-145.

Germonprez, M., Allen, JP, Warner, B., Hill, J., and McClements, G. (2013). Open Source Communities of Competitors, ACM Interactions, 20(6), 54-59.

Germonprez, M. and Warner, B. (2012). Commercial Participation in Open Innovation Communities, in Managing Open Innovation Technologies - Lundstrom, Hrastinski, Agerfalk, and Edenius eds., Springer, New York, NY.

"SPDX Open Source Tooling," Linux Collaboration Summit, Napa, CA, March 2014.

"SPDX Open Source Tooling: Integrating with Software Development Build Processes," Linux Foundation Open Compliance Summit, Yokohama, Japan, December 2013.

"Tooling Up for SPDX," Linux Collaboration Summit, San Francisco, CA, April 2013.

"FOSSology and SPDX Tooling," Hewlett Packard, Fort Collins, CO, November 2012.

[pjotrp]

Thanks for the clarification! If you have not done so, could you add that to your paper or README?

[arfon]

🎏 flagging this (and https://github.com/openjournals/joss-reviews/issues/38) as stale 🎏

[pjotrp]

I think we can close this.

[germonprez]

Thanks for the kick. If you can give me the week, I can finish this up.
Sorry for the delay.

Matt

On Oct 8, 2016 9:09 AM, "Pjotr Prins" [email protected] wrote:

I think we can close this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/39#issuecomment-252426506,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAoDUMfxb8uA2eQOU1NRks_Fh9tndnGuks5qx6QYgaJpZM4JNGoA
.

[arfon]

Thanks for the kick. If you can give me the week, I can finish this up.

👍 thanks @germonprez. Looking forward to your updates over on #38