Joss-reviews: [PRE REVIEW]: SHyP: an open source Scoreboard interface for displaying and comparing scores for Hydrometeorological Predictions

[whedon]

Submitting author: @guillaume-irstea (Guillaume Thirel)
Repository: https://github.com/hydroGR/SHyP
Version: 2.4.1
Editor: @kbarnhart
Reviewer: Pending

Author instructions

Thanks for submitting your paper to JOSS @guillaume-irstea. Currently, there isn't an JOSS editor assigned to your paper.

@guillaume-irstea if you have any suggestions for potential reviewers then please mention them here in this thread. In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission.

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

What happens now?

This submission is currently in a pre-review state which means we are waiting for an editor to be assigned and for them to find some reviewers for your submission. This may take anything between a few hours to a couple of weeks. Thanks for your patience :smile_cat:

You can help the editor by looking at this list of potential reviewers to identify individuals who might be able to review your submission (please start at the bottom of the list). Also, feel free to suggest individuals who are not on this list by mentioning their GitHub handles here.

[whedon]

Attempting to check references...

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

```Reference check summary:

OK DOIs

• 10.1016/j.jhydrol.2014.06.035 is OK
• 10.1007/978-3-642-40457-3_3-1 is OK
• 10.5194/hess-13-141-2009 is OK
• 10.1016/j.jhydrol.2014.05.028 is OK
• 10.1002/met.52 is OK
• 10.1080/02626667.2014.903331 is OK
• 10.1175/2008BAMS2619.1 is OK
• 10.1002/9781119960003 is OK
• 10.5194/hess-18-2829-2014 is OK
• 10.1016/B978-0-12-385022-5.00022-1 is OK
• 10.1002/hyp.9521 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kthyng]

Hi @guillaume-irstea. I have opened this pre-review issue to get things in order before we can review (this is standard).

I see your paper has descriptions of how SHyP is useful. Could you be specific about its research applications in particular, and what exactly SHyP does (as opposed to the software that it uses)?

[guillaume-irstea]

Hi @kthyng
Thanks for the feedback.

Regarding the proof, my only remark is about the figure. It does not appear very clearly. I would suggest rotating it by -90° and increasing its size.

The research applications of SHyP are example comparisons of the performance of different hydrological prediction systems. By hydrological prediction, I mean flood forecasts, drought forecasts, seasonal forecasts. Even if these are potential operational applications, the improvement and understanding of these systems is a large active research field (see for instance the HEPEX initiative that gathers hundred of researchers: https://hepex.irstea.fr/).
For example, if in a project different partners work with the same dataset but different models and they want to compare their performance, they can host SHyP on a server and upload their scores there. Then, they can:

• produce the same plots
• directly compare their score (see tab compare skill scores).
  In addition, a single partner can upgrade its model and show to the other partners on SHyP what the effect is.

Another application is about using, storing and analysing scores from studies that provide open data, which happens more and more. In order to verify, reproduce or improve the results, SHyP provides the perfect environment for analysis. In addition, if complete datasets are not provided but the authors are willing to provide scores, they can directly upload them.

Finally, I would finish by adding that in this field, large datasets and large number of evaluation points (outlet stations of catchments) are used. Thanks to the SQL database, requests on the interface are fast.

[kthyng]

Ok, thanks for that helpful information.

It would be great if your docs and other files did not include .docx since that is proprietary and doesn't look necessary (at least in the Instructions.docx file I looked at). Typically people will have something like markdown/rst files, Jupyter notebooks, or docs from Sphinx instead for their docs.

Do you have a way to test your code, or at least a way for a user to verify that it is running correctly?

Any reviewers of this submission would need to install the package, and it would also be preferable to not require them to root-install your package. Are there other approaches someone could use to install this?

[kthyng]

Regarding the article proof, you control how the figure looks. It looks rightside up to me, and you're right maybe it could be magnified a bit into the area of interest if you want to focus on something. You can just edit the image file itself to change it.

[guillaume-irstea]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[guillaume-irstea]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[guillaume-irstea]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[guillaume-irstea]

Hi @kthyng

I replaced all the .docx files with .md files as suggested, thanks.
In addition, the figure is now better included in the paper.

Finally, regarding the installation of the tool, it is a bit more tricky. I assume that installing R and the related packages as well as postgreSQL is no issue.
However, there is indeed one thing to somehow "root-install": the architecture of the SQL database has to be created. I therefore provide 2 files that allow to do that. Not sure if/how I can do more so far, but I'm willing to take into consideration suggestions.
Other than that, just running the App from RStudio launches the interface, there is nothing else to do.

Cheers.

[kthyng]

Thanks @guillaume-irstea.

@kbarnhart Would you be willing to be editor for this paper?

[ooo[bot]]

:wave: Hey @kthyng...

Letting you know, @kbarnhart is currently OOO until Wednesday, December 4th 2019. :heart:

[danielskatz]

@whedon remind @danielskatz in 2 days

[whedon]

@danielskatz doesn't seem to be a reviewer or author for this submission.

[danielskatz]

👋 @kbarnhart - would you be willing to edit this submission?

[kbarnhart]

Yup. I can do it.

[kbarnhart]

@whedon assign @kbarnhart as editor

[whedon]

OK, the editor is @kbarnhart

[kbarnhart]

@guillaume-irstea I wanted to follow up with a couple of additional questions that will help me understand the content of the submission better.

1. Do you have a way to test your code, or at least a way for a user to verify that it is running correctly?

2. Can you elaborate a little more on the practical aspects of re-using the software for other applications. I know that above in the conversation with @kthyng you described some other possible applications.

The research applications of SHyP are example comparisons of the performance of different hydrological prediction systems. By hydrological prediction, I mean flood forecasts, drought forecasts, seasonal forecasts.

Specifically, I'm interested in the re-use of SHyP for applications that are similar to the ones for which it was designed, but are slightly different in some way. Perhaps an application that focused on drought instead of river flooding. Would such an application be able to use SHyP without modification? Would the attribute of input data need to be the same (e.g., the providing of the columns "River_names" as is in one of the example data files and all of the information in the header)?

[guillaume-irstea]

@kbarnhart Thanks for your comment.

1. Do you have a way to test your code, or at least a way for a user to verify that it is running correctly?

It is necessary to create the structure of the SQL database, thanks to the SQL code provided (https://github.com/hydroGR/SHyP/blob/master/SQL/scoreboard_empty_structure.sql). If you are familiary with SQL this should be no problem I guess, otherwise here are some step-by-step instruction: https://github.com/hydroGR/SHyP/blob/master/Read_Me/Instructions.md
Then only running locally any of the ui.R or server.R files launches the tool.

2. Can you elaborate a little more on the practical aspects of re-using the software for other applications. I know that above in the conversation with @kthyng you described some other possible applications.

The research applications of SHyP are example comparisons of the performance of different hydrological prediction systems. By hydrological prediction, I mean flood forecasts, drought forecasts, seasonal forecasts.

Specifically, I'm interested in the re-use of SHyP for applications that are similar to the ones for which it was designed, but are slightly different in some way. Perhaps an application that focused on drought instead of river flooding. Would such an application be able to use SHyP without modification? Would the attribute of input data need to be the same (e.g., the providing of the columns "River_names" as is in one of the example data files and all of the information in the header)?

That would be an interesting way to use SHyP. Providing that you create the SQL database with what is proposed in the tool, yes you need to use the same attributes and header. Adding additional scores more related to droughts is allowed. So if you are talking about hydrological drought, the tool is ready for that.
If you are talking about meteorological or groundwater droughts, then maybe you should replace the attributes "River_names" and "Station_names" by something more adapted to your case. Modifications of https://github.com/hydroGR/SHyP/blob/master/SQL/scoreboard_empty_structure.sql could be needed. What do you have in mind?

[kbarnhart]

@guillaume-irstea thank you for this information. I want to inform you that after consulting with others on the editorial team, I have made a pre-review decision that this submission is out of scope for JOSS.

As described in the submission requirements, software submitted to JOSS must be "designed for maintainable extension (not one-off modifications)". I came to the conclusion that SHyP does not meet this requirement because the functionality of SHyP is so tied to the specific format of the input file, and that input file format is not easily modifiable to different applications.

We would encourage and welcome a submission that grew out of the current software but was designed to support either a more flexible data input or a more widely used input standard.

If you have any questions about this, please do not hesitate to post them here for discussion.