Joss-reviews: [REVIEW]: Netwulf: Interactive visualization of networks in Python

[whedon]

Submitting author: @ulfaslak (Ulf Aslak)
Repository: https://github.com/benmaier/netwulf
Version: 0.1.4
Editor: @leouieda
Reviewers: @camillescott, @vc1492a, @leouieda
Archive: 10.5281/zenodo.3402870

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/3a22c963a45dbddc8501a4b5ef4b2bf6"><img src="http://joss.theoj.org/papers/3a22c963a45dbddc8501a4b5ef4b2bf6/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/3a22c963a45dbddc8501a4b5ef4b2bf6/status.svg)](http://joss.theoj.org/papers/3a22c963a45dbddc8501a4b5ef4b2bf6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@camillescott & @vc1492a, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @leouieda know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @camillescott

Conflict of interest

• [ ] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [ ] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [ ] Repository: Is the source code for this software available at the repository url?
• [ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [ ] Version: 0.1.4
• [ ] Authorship: Has the submitting author (@ulfaslak) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Authors: Does the paper.md file include a list of authors with their affiliations?
• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @vc1492a

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: 0.1.4
• [x] Authorship: Has the submitting author (@ulfaslak) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @leouieda

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: 0.1.4
• [x] Authorship: Has the submitting author (@ulfaslak) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @camillescott, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[leouieda]

@vc1492a @camillescott thank you for volunteering your time for this review :1st_place_medal: This is the review issue where you can leave comments and check items from your checklists. If you open any issues or PRs in the software repository, please remember to link back to this issue using openjournals/joss-reviews#1425 in the issue/PR so we can keep track of progress.

@ulfaslak @vc1492a @camillescott please don't hesitate to reach to me if you have questions/concerns (here or privately, if you prefer).

[vc1492a]

Overview: This was an interesting piece of software to review and I enjoyed walking through it and testing its functionality. While I believe updates are needed in order for publication in JOSS (see below), the software is well documented and seems to fit a niche that may be useful within the network analysis community.

License: While a license is provided in the project repository (MIT), it may be beneficial to indicate the license more publicly in the readme on Github and/or in the documentation. For example, a badge could be used or a section added to the readme or documentation.

Version Matching: The version number submitted to JOSS does match the latest version of the source code in master on Github. It may be beneficial however to tag the releases on Github that are uploaded to PyPi such that users can download the software directly from Github via a Release as opposed to having to clone or pull the repository. This is helpful to users in some cases and also provides the developer with a copy of the source code for previous versions.

Functionality: I was able to verify the basic functionality, in this case installing the software, importing the software, and opening the visualization interface successfully by calling netwulf.visualize(G), where G is a networkX graph object. I was also able to verify the download PNG functionality as well as the CSV upload functionality using the example provided in the documentation. However, I was not able to get the Post to Python functionality working. More specifically, the UI indicates success and closes the browser window, but the Python kernel remains busy indefinitely and on manually stopping execution, the network_properties object is not returned. This occurred in both a Jupyter notebook as well as a terminal window. Consider opening an issue with the tag bug. I also noticed that selecting the Node size by option resizes the nodes appropriately, but deselecting it does not remove the transformation.

I'm not sure, but I think there may be a CSS formatting or similar error in the user interface. See the screenshot below:

The Upload file section open parenthesis and does not close them - seems to be missing json). No big deal, but may be worth addressing in a future release. Consider opening an issue with the tag bug.

Some potentially useful functionality would be to allow a user to start the visualization interface without the need to pass a graph object into visualize(). This may be useful in the case where the user has a file that they wish to examine and want to upload, but may not want to load first within the Python environment. By making the graph object an optional argument, the user could start the visualization interface and then explore one or more files through the upload functionality. This would be a nice feature in future releases but isn't necessary to address as part of this review. Consider opening an issue with the tag enhancement.

Additionally, I noticed a lot of use of Python's os library. While this works fine on Linux and Unix devices, it can result in errors on Windows machines. Using Pathlib in place of os would reduce the likelihood of errors on Windows machines. I am not able to verify the functionality of this software on Windows machines (the editor can decide whether this functionality on windows should be formally verified or whether this potential limitation should be listed in readme.md or documentation), but do know that Pathlib would allow the developers to normalize paths for Windows machines easily. Consider opening an issue with the tag enhancement.

Performance claims: The paper states "The interactive visualization is implemented in JavaScript, relies on d3.js for computing layouts, and uses the HTML5-object canvas for rendering. This makes it, to our knowledge, the most performant tool for interactive network visualization in Python to date." This statement implies that the word performant is used to describe the speed in which one can create and manipulate a network graph and return an object to Python that describes that graph. After using the software I believe this claim is true, but would suggest the editor or others also look into this area to help verify this claim.

Functionality documentation: This documentation is great and contains details on the API as well as cookbooks containing example code. I was able to use the example code easily to get started on verifying the functionality of the software. I noticed one small item that could be addressed in the examples. In the below screenshot, it appears that a necessary matplotlib import is missing:

I believe this should include import matplotlib.pyplot as plt.

Additionally, in the example located in the filtering section, grp = {u: 'ABCDE'[u%5] for u in G.nodes()} and nx.set_node_attributes(G, grp, 'wum') should be executed after the first visualization and before the second visualization, as opposed to before the first. Executing those lines prior to the first example results in the following error when opening the visualization interface:

The rest of the documentation looks great and is quite helpful! To gain additional feedback and user adoption, it may be interesting to show some insights gained through using the interactive visualization interface on some real-world data (not necessary though).

Statement of need (documentation): A statement of need exists in the documentation, but the focus differs from that highlighted in the generated paper. The documentation states "Netwulf is an interactive visualization tool for networkx Graph-objects, that allows you to produce beautiful looking network visualizations. Simply input a networkx.Graph object, style the network in the interactive console and either download the result as a PNG or pipe the layout back to Python for further processing." This implies the software's focus is on creating visually-pleasing network visualizations easily from networkx objects. However, the paper states "Netwulf is a light-weight Python library that provides a simple API for interactively visualizing a network and returning the computed layout and style. It is build around the philosophy that network manipulation and preprocessing should be done programmatically, but that the efficient generation of a visually appealing network is best done interactively, without code. Therefore, it offers no analysis functionality and only few exploration features, but instead focuses almost entirely on fast and intuitive layout manipulation and node/link styling." This statement implies that the development focus is still in generating visually appealing networks interactively, but is mentions a programmatic interface and lack of analysis tools which obfuscates that meaning somewhat. The two statements are not in direct conflict, but the purpose of the work must be clearly stated and in agreement between the paper and the documentation and should be revised for clarity.

Installation instructions: Instructions for installation work and are easy to follow, with the listed dependencies in setup.py being installed upon installation. However, these dependencies are in the documentation and do not appear on the Github readme and should be mentioned in both locations for clarity.

Automated tests: I did not see any automated tests in this repository that could be used to verify the functionality of the software. Revisions of this work should include a series of unit tests that can be used to verify the functionality of the software, through testing of function outputs, input and output formatting, etc. There are a variety of ways to write and use unit tests in Python, two of which are the pytest and pytest-cov packages - the former allows developers to execute unit tests (with helpful extensions for writing tests, too) while the latter provides functionality to pytest for coverage statistics, e.g. which lines of code your unit tests cover and the percentage of lines in each file (and overall, the package) that is covered by the written unit tests (this coverage statistic could be reported in readme.md). Additionally, if you're looking to have others contribute to the project more easily in the future consider setting up a continuous integration pipeline that will automatically run tests and report the test results (e.g. using Travis CI) and coverage (e.g. using Coveralls) with any pushes to master or dev and for any pull requests opened by open-source contributors. This pipeline would ease the burden on making sure new additions and/or changes to software work as intended, and would also allow maintainers to formally test against every version of Python 3 that should be supported. This would allow the authors to state "this software works in Python versions 3.5 - 3.7" as opposed to simply stating "works in Python 3".

Community guidelines: While there are some development notes on how to update the front-end from it's upstream directory when developing, there are no clear guidelines on Github (in the readme or otherwise) or in the documentation on how to best jump into development or open a pull request for bug fixes or new features. Additionally, there is no documentation on how to report issues and it's implicitly implied that the user should open a Github issue. If that's the desired error reporting behavior, this should be made more explicit in the documentation. If Github issues is the desired mechanism, it may be beneficial to the developers to open issues for some of the behaviors noted here - for the authors: let me know if you'd like me to open issues for the items identified in the Functionality section above.

References: paper.bib had all relevant references, but did not have all DOIs that are available, so added a few and opened a this pull request. Specifically for the webweb reference, the DOI is pending since it's currently under review by JOSS (https://github.com/openjournals/joss-reviews/issues/1308). The editor of this work should decide how to handle this reference and whether updating the paper.bib file post publication in JOSS is a possibility so that this reference's DOI may be included for this work once itself is reviewed and later published.

Other: Although the visuals current used are already nice and informative, they could also be improved in future versions. This would (potentially) be a nice place to collaborate with others with, and documentation on how to contribute to this and other areas would be nice to see in the repository and/or documentation. Also consider adding a changelog.md file that details changes, additions, and fixes made to the code with each release version. Lastly, it seems that semantic versioning is used but this isn't make explicitly clear in readme.md - not that it needs to be, but adding details such as these, along with a changelog.md, would go a long way towards easing the entry into development for those which may want to contribute to the package.

[leouieda]

@vc1492a thank you for the detailed review :rocket:

@ulfaslak please do your best to respond to Valentino's comments either here in this thread or in separate issues in your repository (preferred). If you open the issues/PRs, please remember to link back to this thread by including openjournals/joss-reviews#1425 in the issue/PR body.

[leouieda]

I agree with @vc1492a regarding using Pathlib and adding some form of testing. The JOSS requirements state that:

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.
Good: An automated test suite hooked up to an external service such as Travis-CI or similar
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g. a sample input file to assert behaviour)
Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works

While manual tests are acceptable, use of some form of automated test is encouraged.

[benmaier]

@vc1492a, indeed thank you very much for the super detailed review and @leouieda thank you for the thorough editing process so far. Since @ulfaslak and me are both on vacation at the moment, we will be responding to the review in detail afterwards. In the mean time, @vc1492a would you be so kind as to open an issue for the Functionality problem that you encountered? Indeed, we had a similar problem before (see https://github.com/benmaier/netwulf/issues/4 and https://github.com/benmaier/netwulf/issues/15) but thought we fixed the bug you unfortunately encountered. Thank you very much!

[leouieda]

@benmaier @ulfaslak no problem, take your time and enjoy your vacation :desert_island: :tropical_drink:

When opening issues or PRs in the software repo, please include the text openjournals/joss-reviews#1425 in the issue/PR body so that it appears here and I can keep track of what is being done.

[benmaier]

We would like to thank @vc1492a again for his incredibly helpful review which sparked a lot of ideas which actually already improved the package. We were delighted to read that he found netwulf to be an _"interesting piece of software [...] that may be useful within the network analysis community"_. He also _"believe[d] updates are needed in order for publication in JOSS"_, which we ultimately agree with. For the updated version v0.1.0, we discussed and updated things according to his remarks and will address each individually below.

License: While a license is provided in the project repository (MIT), it may be beneficial to indicate the license more publicly in the readme on Github and/or in the documentation. For example, a badge could be used or a section added to the readme or documentation.

We have added a License-section to both README.md and documentation.

Version Matching: The version number submitted to JOSS does match the latest version of the source code in master on Github. It may be beneficial however to tag the releases on Github that are uploaded to PyPi such that users can download the software directly from Github via a Release as opposed to having to clone or pull the repository. This is helpful to users in some cases and also provides the developer with a copy of the source code for previous versions.

We introduced proper versioning and release tagging. @leouieda, we will probably change the version during the review process. Can we change the ultimate version number for JOSS once the software is accepted for publication?

Functionality: I was able to verify the basic functionality, in this case installing the software, importing the software, and opening the visualization interface successfully by calling netwulf.visualize(G), where G is a networkX graph object. I was also able to verify the download PNG functionality as well as the CSV upload functionality using the example provided in the documentation. However, I was not able to get the Post to Python functionality working. More specifically, the UI indicates success and closes the browser window, but the Python kernel remains busy indefinitely and on manually stopping execution, the network_properties object is not returned. This occurred in both a Jupyter notebook as well as a terminal window. Consider opening an issue with the tag bug. I also noticed that selecting the Node size by option resizes the nodes appropriately, but deselecting it does not remove the transformation.

The Upload file section open parenthesis and does not close them - seems to be missing json). No big deal, but may be worth addressing in a future release. Consider opening an issue with the tag bug.

Some potentially useful functionality would be to allow a user to start the visualization interface without the need to pass a graph object into visualize(). This may be useful in the case where the user has a file that they wish to examine and want to upload, but may not want to load first within the Python environment. By making the graph object an optional argument, the user could start the visualization interface and then explore one or more files through the upload functionality. This would be a nice feature in future releases but isn't necessary to address as part of this review. Consider opening an issue with the tag enhancement.

This was discussed and resolved individually here: benmaier/netwulf#20 and here: benmaier/netwulf#21.

Additionally, I noticed a lot of use of Python's os library. While this works fine on Linux and Unix devices, it can result in errors on Windows machines. Using Pathlib in place of os would reduce the likelihood of errors on Windows machines. I am not able to verify the functionality of this software on Windows machines (the editor can decide whether this functionality on windows should be formally verified or whether this potential limitation should be listed in readme.md or documentation), but do know that Pathlib would allow the developers to normalize paths for Windows machines easily. Consider opening an issue with the tag enhancement.

This was discussed and resolved individually here: benmaier/netwulf#23

Performance claims: The paper states "The interactive visualization is implemented in JavaScript, relies on d3.js for computing layouts, and uses the HTML5-object canvas for rendering. This makes it, to our knowledge, the most performant tool for interactive network visualization in Python to date." This statement implies that the word performant is used to describe the speed in which one can create and manipulate a network graph and return an object to Python that describes that graph. After using the software I believe this claim is true, but would suggest the editor or others also look into this area to help verify this claim.

This point seems to be up to the editor.

Functionality documentation: This documentation is great and contains details on the API as well as cookbooks containing example code. I was able to use the example code easily to get started on verifying the functionality of the software. I noticed one small item that could be addressed in the examples. In the below screenshot, it appears that a necessary matplotlib import is missing:

I believe this should include import matplotlib.pyplot as plt.

Additionally, in the example located in the filtering section, grp = {u: 'ABCDE'[u%5] for u in G.nodes()} and nx.set_node_attributes(G, grp, 'wum') should be executed after the first visualization and before the second visualization, as opposed to before the first. Executing those lines prior to the first example results in the following error when opening the visualization interface:

The rest of the documentation looks great and is quite helpful! To gain additional feedback and user adoption, it may be interesting to show some insights gained through using the interactive visualization interface on some real-world data (not necessary though).

We fixed the bugs in the documentation code (see benmaier/netwulf#22). We think showing how netwulf can be used to obtain insights by usage of the interactive visualization is a great idea and we'll make sure to incorporate that in a later version of the documentation.

Statement of need (documentation): A statement of need exists in the documentation, but the focus differs from that highlighted in the generated paper. The documentation states "Netwulf is an interactive visualization tool for networkx Graph-objects, that allows you to produce beautiful looking network visualizations. Simply input a networkx.Graph object, style the network in the interactive console and either download the result as a PNG or pipe the layout back to Python for further processing." This implies the software's focus is on creating visually-pleasing network visualizations easily from networkx objects. However, the paper states "Netwulf is a light-weight Python library that provides a simple API for interactively visualizing a network and returning the computed layout and style. It is build around the philosophy that network manipulation and preprocessing should be done programmatically, but that the efficient generation of a visually appealing network is best done interactively, without code. Therefore, it offers no analysis functionality and only few exploration features, but instead focuses almost entirely on fast and intuitive layout manipulation and node/link styling." This statement implies that the development focus is still in generating visually appealing networks interactively, but is mentions a programmatic interface and lack of analysis tools which obfuscates that meaning somewhat. The two statements are not in direct conflict, but the purpose of the work must be clearly stated and in agreement between the paper and the documentation and should be revised for clarity.

We agree and resolved this here: benmaier/netwulf#24

Installation instructions: Instructions for installation work and are easy to follow, with the listed dependencies in setup.py being installed upon installation. However, these dependencies are in the documentation and do not appear on the Github readme and should be mentioned in both locations for clarity.

We added dependency statements in README.md, the documentation, and also added a requirements.txt.

Automated tests: I did not see any automated tests in this repository that could be used to verify the functionality of the software. Revisions of this work should include a series of unit tests that can be used to verify the functionality of the software, through testing of function outputs, input and output formatting, etc. There are a variety of ways to write and use unit tests in Python, two of which are the pytest and pytest-cov packages - the former allows developers to execute unit tests (with helpful extensions for writing tests, too) while the latter provides functionality to pytest for coverage statistics, e.g. which lines of code your unit tests cover and the percentage of lines in each file (and overall, the package) that is covered by the written unit tests (this coverage statistic could be reported in readme.md). Additionally, if you're looking to have others contribute to the project more easily in the future consider setting up a continuous integration pipeline that will automatically run tests and report the test results (e.g. using Travis CI) and coverage (e.g. using Coveralls) with any pushes to master or dev and for any pull requests opened by open-source contributors. This pipeline would ease the burden on making sure new additions and/or changes to software work as intended, and would also allow maintainers to formally test against every version of Python 3 that should be supported. This would allow the authors to state "this software works in Python versions 3.5 - 3.7" as opposed to simply stating "works in Python 3".

We discussed this here: benmaier/netwulf#26. While completely automated tests seem to be complicated to implement due to both the the human component in the network styling process as well as the interplay between Python and JavaScript in a Browser window, we introduced automated tests which can be run by using the command make test in the repository's root directory. The testing behavior is described in detail in the documentation. We tried using travis for the testing procedure but ultimately decided against it because the visual appearance should also be tested and we did not find a reliable way to do this with travis.

Community guidelines: While there are some development notes on how to update the front-end from it's upstream directory when developing, there are no clear guidelines on Github (in the readme or otherwise) or in the documentation on how to best jump into development or open a pull request for bug fixes or new features. Additionally, there is no documentation on how to report issues and it's implicitly implied that the user should open a Github issue. If that's the desired error reporting behavior, this should be made more explicit in the documentation. If Github issues is the desired mechanism, it may be beneficial to the developers to open issues for some of the behaviors noted here - for the authors: let me know if you'd like me to open issues for the items identified in the Functionality section above.

We've added a contributing guide and a code of conduct.

References: paper.bib had all relevant references, but did not have all DOIs that are available, so added a few and opened a this pull request. Specifically for the webweb reference, the DOI is pending since it's currently under review by JOSS (#1308). The editor of this work should decide how to handle this reference and whether updating the paper.bib file post publication in JOSS is a possibility so that this reference's DOI may be included for this work once itself is reviewed and later published.

@vc1492a already added the DOI's for us. Thank you! :)

Other: Although the visuals current used are already nice and informative, they could also be improved in future versions. This would (potentially) be a nice place to collaborate with others with, and documentation on how to contribute to this and other areas would be nice to see in the repository and/or documentation. Also consider adding a changelog.md file that details changes, additions, and fixes made to the code with each release version. Lastly, it seems that semantic versioning is used but this isn't make explicitly clear in readme.md - not that it needs to be, but adding details such as these, along with a changelog.md, would go a long way towards easing the entry into development for those which may want to contribute to the package.

We indeed plan to improve the visualization further, e.g. by adding directed and arrowed links (see e.g. ulfaslak/network_styling_with_d3#3 and benmaier/curved-edges ). Furthermore I'd like to have additional styling functions for trees and hierarchically clustered networks. We created an OUTLOOK.md to keep track of these ideas.

We hope that @vc1492a is content with our changes and are looking forward to @camillescott's review, too!

[vc1492a]

@benmaier Thank you for the detailed replies to my comments. Thank you for writing some automated tests and for providing a structure for writing future tests. The issue of writing tests for an interactive framework is an interesting one, and I think worth exploring in the future as you continue to improve netwulf. I'd very much welcome seeing the addition of curved edges and clustered networks in netwulf as well! Great work.

@leouieda At this stage, I believe we can consider my review of netwulf complete - please let me know if I can further assist in any way.

[leouieda]

:wave: Hi @camillescott just checking in to see if you've made any progress on the review. Do you know when you'd be able to report back to us?

[leouieda]

@vc1492a thank you for all your comments! And @benmaier thank you for being on top of it and implementing everything.

[leouieda]

Hi @camillescott just checking in again. Please let me know if you'll be able to complete the review. If, for any reason, you find that you will not be able to do this, just let me know and I can search for another reviewer.

[benmaier]

Hi @leouieda, just commenting to let you know that I've added a few features to the package because I needed them for my own research. We're now at v0.1.3. Is this going to be a problem? If it helps, everything @vc1492a reviewed is still functional in the same way he reviewed. I've also added tests and sections in the docs for the new features.

[leouieda]

@benmaier there shouldn't be a problem. Before acceptance I would ask you to make a release with the changes that came about during review anyway. We'll update the actual version number later on as well. Sorry for the delay with this. I'm trying to find another reviewer that could give me a second opinion.

[camillescott]

I'm so sorry folks, I let this completely slip my radar. I can get you in a
review by later tonight, unless you'd like to find another person.
Apologies again.

On Tue, Jun 25, 2019 at 7:00 PM Leonardo Uieda [e-mail]
wrote:

@benmaier https://github.com/benmaier there shouldn't be a problem.
Before acceptance I would ask you to make a release with the changes that
came about during review anyway. We'll update the actual version number
later on as well. Sorry for the delay with this. I'm trying to find another
reviewer that could give me a second opinion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/1425?email_source=notifications&email_token=AAWDDLKL7GROHTHGMIXVG63P4LEVPA5CNFSM4HKICLS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYSCW2A#issuecomment-505686888,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWDDLNF2FURIP35HES6E7DP4LEVPANCNFSM4HKICLSQ
.

--
Camille Scott

Graduate Group in Computer Science (GGCS)
Lab for Data Intensive Biology
University of California, Davis

[email protected] (or)
camille.scott.[email protected]

[leouieda]

Hi @camillescott, no problem at all. I know how Github notifications can get and completely understand. Take your time. It would be great to have your review (I haven't found a replacement).

[benmaier]

Any updates? Since @ulfaslak and I are both graduating soon, it would be nice to include this project in the list of published papers :)

[leouieda]

:wave: Hi @camillescott, any updates on your review?

[leouieda]

Hi @benmaier sorry for the long delay. I'm away on travel right now but as soon as I get back next week I'll have a look at the submission and see if we're good to go.

[benmaier]

any updates?

[leouieda]

@whedon add @leouieda as reviewer

[whedon]

OK, @leouieda is now a reviewer

[leouieda]

@benmaier I've been having trouble getting a second reviewer for this so I'll have a go myself. Should be straight forward since I've already had a look at the package and paper. Sorry for the delay.

[benmaier]

Cool! Thank you so much!

[leouieda]

@benmaier I have finished my review of the software and I'm very happy with all the changes made and the status of the package and paper. Together with the thorough review of @vc1492a this submission if ready for acceptance :tada:

I made a small PR to add a missing DOI (benmaier/netwulf#34). Once that is merged, there are a few actions that you'll need to take:

• Double check authors and affiliations (including ORCIDs)
• Make a release with the latest changes from the review and post the version number here.
• Archive the release on Zenodo and post the DOI here (make sure the title and author list matches the JOSS paper, if possible).

Let me know if you need help with any of these steps.

[leouieda]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

Hi @leouieda, thank you for your editing, review, and pull request! I think I've completed all the necessary steps for publishing:

• I double-checked author-names, affiliations, and ORCIDs.
• I made a new release containing all changes: v0.1.4
• I archived this release on Zenodo (DOI: 10.5281/zenodo.3402870)

Note that I also reformulated a sentence which appeared to be incorrect: In the original paper we claimed that graph-tool does not have interactive features which is evidently false. I therefore rephrased this statement.

Thank you very much, again, and let me know if there's still anything missing!

[benmaier]

@leouieda just quickly checking in asking for an update :)

[ooo[bot]]

:wave: Hey @benmaier...

Letting you know, @leouieda is currently OOO until Sunday, September 29th 2019. :heart:

[leouieda]

@whedon set v0.1.4 as version

[whedon]

OK. 0.1.4 is the version.

[leouieda]

@whedon set 10.5281/zenodo.3402870 as archive

[whedon]

OK. 10.5281/zenodo.3402870 is the archive.

[leouieda]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[leouieda]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.13140/2.1.1341.1520 is OK
• 10.1101/gr.1239303 is OK
• 10.21105/joss.01458 is OK
• 10.1109/MCSE.2007.55 is OK
• 10.1109/TVCG.2011.185 is OK
• 10.6084/m9.figshare.1164194 is OK

MISSING DOIs

• https://doi.org/10.1007/3-540-45848-4_57 may be missing for title: Graphviz — open source graph drawing tools

INVALID DOIs

• None
  ```

[leouieda]

:wave: Hi @benmaier, please change the Graphviz citation to https://link.springer.com/chapter/10.1007/3-540-45848-4_57 (same authors so I'm guessing this is a paper made from the proceedings). I was just going through the paper once more and noticed the citation for the StackOverflow survey :) I couldn't find a proper way of citing it and the current citation isn't rendering correctly with our template. Could you please change it to a hyperlink instead of a citation?

[benmaier]

oh, thanks for the hint. "Overflow, S. (2019)" should've caught our attention sooner, haha. @ulfaslak can you take care of that?

[ulfaslak]

@benmaier done!

[leouieda]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

@ulfaslak the Stack overflow reference still looks weird

[benmaier]

nvm

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

PDF failed to compile for issue #1425 with the following error:

Error reading bibliography ./paper.bib (line 80, column 5):
unexpected "d"
expecting space, ",", white space or "}"
Error running filter pandoc-citeproc:
Filter returned error status 1
Looks like we failed to compile the PDF

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

PDF failed to compile for issue #1425 with the following error:

Error reading bibliography ./paper.bib (line 80, column 5):
unexpected "d"
expecting space, ",", white space or "}"
Error running filter pandoc-citeproc:
Filter returned error status 1
Looks like we failed to compile the PDF

[benmaier]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benmaier]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.13140/2.1.1341.1520 is OK
• 10.1101/gr.1239303 is OK
• 10.21105/joss.01458 is OK
• 10.1109/MCSE.2007.55 is OK
• 10.1109/TVCG.2011.185 is OK
• 10.1007/3-540-45848-4_57 is OK
• 10.6084/m9.figshare.1164194 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[benmaier]

Sorry for the spam folks! @leouieda, this looks fine to me now.

[ooo[bot]]

:wave: Hey @benmaier...

Letting you know, @leouieda is currently OOO until Sunday, September 29th 2019. :heart:

[leouieda]

@benmaier thank you for being on top of this!

[leouieda]

@openjournals/joss-eics this submission is ready of acceptance :confetti_ball:

[kyleniemeyer]

Hi @benmaier, I fixed the NetworkX reference, please merge this PR: https://github.com/benmaier/netwulf/pull/35

[benmaier]

Done!

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kyleniemeyer]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

```Reference check summary:

OK DOIs

• 10.13140/2.1.1341.1520 is OK
• 10.1101/gr.1239303 is OK
• 10.21105/joss.01458 is OK
• 10.1109/MCSE.2007.55 is OK
• 10.1109/TVCG.2011.185 is OK
• 10.1007/3-540-45848-4_57 is OK
• 10.6084/m9.figshare.1164194 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/995

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/995, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[kyleniemeyer]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/996
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01425
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[kyleniemeyer]

Congrats @ulfaslak on your article's publication in JOSS!

Many thanks to @leouieda for editing, and @camillescott and @vc1492a for reviewing, this submission.

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01425/status.svg)](https://doi.org/10.21105/joss.01425)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01425">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01425/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01425/status.svg
   :target: https://doi.org/10.21105/joss.01425

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss