Joss-reviews: [REVIEW]: FHI-vibes: Ab initio Vibrational Simulations

[whedon]

Submitting author: @flokno (Florian Knoop)
Repository: https://gitlab.com/vibes-developers/vibes
Version: v1.0.0
Editor: @jgostick
Reviewers: @keipertk, @ajjackson
Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d"><img src="https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d/status.svg)](https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@keipertk, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jgostick know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @keipertk

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@flokno) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

---

Review checklist for @ajjackson

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@flokno) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @keipertk it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1088/1361-648X/aa680e is OK
- 10.1016/j.scriptamat.2015.07.021 is OK
- 10.1103/PhysRevB.91.094306 is OK
- 10.1016/j.cpc.2009.06.022 is OK
- 10.1016/j.cpc.2018.09.020 is OK
- 10.5334/jors.148 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1002/cpe.3505 is OK
- 10.1038/nmat3568 is OK
- 10.1103/PhysRevLett.78.4063 is OK
- 10.1103/PhysRevLett.118.175901 is OK
- 10.1002/adts.201800184 is OK
- 10.1002/anie.201812112 is OK
- 10.1021/acs.jpcc.5b11115 is OK
- 10.1038/nmat2090 is OK
- 10.1016/j.jeurceramsoc.2007.12.023 is OK
- 10.1103/PhysRevLett.96.115504 is OK
- 10.1103/physrevb.79.064301 is OK
- 10.1557/mrs.2018.208 is OK

MISSING DOIs

- 10.1103/physrevmaterials.4.083809 may be a valid DOI for title: Anharmonicity Measure for Materials
- 10.1038/s41597-020-00638-4 may be a valid DOI for title: AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

INVALID DOIs

- None

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[keipertk]

I was unable to find community guidleines for third parties wishing to contribute to the software, outside of the standard MIT license agreement. Everything else gets a check.

[jgostick]

@keipertk Thanks for this! So fast too! I still haven't even found a second reviewer yet!

[flokno]

@jgostick in case you don't find a second reviewer: I talked to [at]ajjackson at some earlier point and he was interested to review this.

@keipertk thanks for the feedback. Indeed we didn't formally include contribution guidelines, we'll add them.

[jgostick]

Thanks @flokno I will ask them.

Hi @ajjackson, your name has been suggested as a reviewer for this submission by @flokno. I am in need of a second reviewer, so your time would be much appreciated.

[ajjackson]

Hi @jgostick I will be happy to review this.

I have no COI as defined by JOSS policy. In interest of transparency I would state that I met Florian at a workshop last year for ASE developers/users and recommended submitting the code to JOSS based on presentation/discussion at that event. I have not used the code, examined it closely or otherwise been involved in its development.

[jgostick]

That sounds like a fine relationship. The review process is transparent, so if you didn't know each other before, you would afterwards!

[jgostick]

@whedon add @ajjackson as reviewer

[whedon]

OK, @ajjackson is now a reviewer

[flokno]

Hi @keipertk , would you be happy with such a contribution guideline:
https://gitlab.com/vibes-developers/vibes/-/merge_requests/23
?

[keipertk]

Yep that looks great, nice work!

[flokno]

Hi, I was waiting to merge https://gitlab.com/vibes-developers/vibes/-/merge_requests/23 until @ajjackson has made additional comments, any news there?

[ajjackson]

This is a sufficient set of contribution guidelines in my opinion, feel free to merge

[ajjackson]

@jgostick I am not able to click the check boxes in the review - I think I missed the notification to get the required permissions. Can you activate this?

[ajjackson]

Some initial comments on the manuscript:

I think this can be an exciting and useful contribution to the atomistic vibrations ecosystem. I especially welcome that the code interfaces with other active projects rather than reinventing the wheel.

The paper makes a reasonable survey of related tools, putting FHI-vibes in context. It does seem unfair to omit any mention of the MIT-licensed TDEP code which does firmly sit at the interface between lattice dynamics and MD. I don't think TDEP undermines the scholarly effort or community need for this work; the statement of need already clearly argues that the wide scope and integrated features set FHI-vibes apart from existing codes.

I am confused about the relationship between the statement of need and the actual existing feature set of FHI-vibes. The SoN gives several examples of methods that might be accessed using an integrated LD/MD toolkit: efficient MD initialisation; harmonic analysis of MD trajectories; anharmonicity analysis (Knoop et al); ab initio GK (Carbogno et al).
Of these, it is not clear from the manuscript which of these is implemented as a workflow in FHI-vibes, and which are aspirational examples of things that a user might build on top of FHI-vibes. From the documentation, I can find information about MD initialisation and anharmonicity analysis but not the other two techniques. I see green_kubo and harmonic_analysis modules in the source code so presumably these are WIP?

I will have a crack at the tutorials next.

[flokno]

Hi Adam, thanks for the initial comments!

The paper makes a reasonable survey of related tools, putting FHI-vibes in context. It does seem unfair to omit any mention of the MIT-licensed TDEP code which does firmly sit at the interface between lattice dynamics and MD. I don't think TDEP undermines the scholarly effort or community need for this work; the statement of need already clearly argues that the wide scope and integrated features set FHI-vibes apart from existing codes.

Of course we are aware of TDEP (and actually big fans of it), and we actually support dumping MD trajectories to TDEP input files, although this is kind of a hidden feature. However, since TDEP is essentially unrelated to our officially supported features (run calculations and in particular MD, compute finite-differences harmonic forceconstants, postprocessing like anharmonicity quantification) and it is outside of the python environment, I omitted it in the survey.

I am confused about the relationship between the statement of need and the actual existing feature set of FHI-vibes. The SoN gives several examples of methods that might be accessed using an integrated LD/MD toolkit: efficient MD initialisation; harmonic analysis of MD trajectories; anharmonicity analysis (Knoop et al); ab initio GK (Carbogno et al).
Of these, it is not clear from the manuscript which of these is implemented as a workflow in FHI-vibes, and which are aspirational examples of things that a user might build on top of FHI-vibes. From the documentation, I can find information about MD initialisation and anharmonicity analysis but not the other two techniques. I see green_kubo and harmonic_analysis modules in the source code so presumably these are WIP?

Yes, Green Kubo and more sophisticated harmonic analysis are WIP. These things should be finalized sometime next year. We plan a separate release (vibes 1.1.?) when GK is ready. Our plan with the paper was to write it such that it covers the envisaged scope of FHI-vibes and not just the features available in 1.0. It is stated in the paper that the officially supported features are listed in the docs/the webpage. If you wish to disentangle this more clearly in the paper, we can do so. I could for example write sth. like "In the initial release, FHI-vibes supports XYZ. We plan to add features A and B in upcoming releases. The currently supported list of features and a use guide etc. are available at vibes....de". Would you prefer that?

I will have a crack at the tutorials next.

Looking forward to that, thanks again for the good comments.

[arfon]

@whedon re-invite @ajjackson as reviewer

[whedon]

OK, the reviewer has been re-invited.

@ajjackson please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

[jgostick]

Thanks @arfon ... I should have been able to figure that out, sorry

[ajjackson]

If you wish to disentangle this more clearly in the paper, we can do so. I could for example write sth. like "In the initial release, FHI-vibes supports XYZ. We plan to add features A and B in upcoming releases. The currently supported list of features and a use guide etc. are available at vibes....de". Would you prefer that?

I think that would be ideal. @jgostick is that reasonably in line with JOSS conventions?

[jgostick]

I would suggest to leave out any reference to future functionality. It makes the paper feel like a temporary sign post rather than a destination when you talk about all the things you want to add, more like wishful thinking. If these future functionalities are important enough, then the software is incomplete without it. That's not official JOSS policy, just my opinion.

[flokno]

Thanks for these comments. I will discuss this with my coauthors as soon as the rest of the review is in.

[flokno]

Hi guys, did you already have a look at the tutorials? Looking forward to your comments.

[jgostick]

HI @ajjackson How is the review coming? Don't forget to check off the tick-boxes at the top of this thread for each aspect of the code as you look through it.

[ajjackson]

I have been working through the tutorial today. So far so good, will pick it up again once the molecular dynamics part has run. This is with a view to checking the functionality/doc quality.

There are probably some more checkboxes that I can already tick based on the manuscript, will get on those soon.

I have posted one documentation-related issue on the project board here
https://gitlab.com/vibes-developers/vibes/-/issues/21

[jgostick]

Thanks for the update.

[ajjackson]

Re:

Documentation

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

I find that the introductory paragraph in the README and landing page of the documentation does a good job of outlining the scope and problems. However, it may not be so obvious who the target audience is. The _tutorials_ do repeatedly reiterate that users are expected to have some experience of lattice dynamics and/or molecular dynamics simulations. I worry that a naive user (or, say, a supervisor directing a student) could see this introduction and think that such matters will be taken care of for them. I think this will benefit from a line clarifying that it is intended for computational materials researchers who already use atomistic simulations.

[ajjackson]

I have completed tutorials up to the Fireworks part. This will require a bit more configuration to check, I'm afraid.

Re:

Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There are quite a few aspects to the documentation here.

• The command-line tool, with autocompletion, has a useful inbuilt help capability
• The tutorials are of decent quality (I hit a couple of pain-points that I have yet to report on gitlab, but they're not a big deal.)
• Input file options are documented on the main documentation website. These pages also function as quick "how-to guides", distinct from "tutorial" content.
• Much of the python code has good docstrings, but there is no browsable API documentation as such

As such I can say that the core (command-line) functionality is documented, but there is room for improvement on the Python API. As this is only promised "for advanced analysis", I don't find the paper to be misleading on this point. And the tutorials provide a decent entrypoint.

I should be able to tick off

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

and

State of the field: Do the authors describe how this software compares to other commonly-used packages?

once the manuscript has been revised to make a clearer separation between things that are currently possible and things that we would like to do in future.

[flokno]

Hi Adam, thanks for the further comments, we're awaiting your issues on gitlab and comments on fireworks.

[flokno]

@ajjackson I understand your comments on the manuscript are complete and mainly focus on the "state of the field" section with a clearer separation between examples/potential future directions and implemented featuers? If yes I'll go back to my coauthors and revise the script to progess on that end.

[ajjackson]

That's correct. While there is truth in @jgostick 's comment about "wishful thinking" future features, the situation is a little more muddled with packages like this that present a toolkit. One can make accurate statements about how the _current state_ provides a capability for users and developers to build workflows, even if those workflows are not yet provided with the package. So I can see why unimplemented workflows belong in the "statement of need" - they are intended to illustrate the need for a good framework.

So I would like to see a clearer separation between:

• workflows that are already available elsewhere
• ways that vibes improves on or supplements those workflows
• workflows that only vibes provides
• examples of workflows that vibes facilitates but does not (yet) provide

and to avoid a long "wishlist" the bar for significance should be set fairly high for that last category. I think the examples given (harmonic analysis of MD and Green-Kubo calculations) are good, but maybe a few words are needed to explain how they benefit from the tools provided by vibes.

I still need to do the Fireworks tutorials, which hopefully will be aided by the installation instructions provided in the docs. I have played with Fireworks before, but it was some time ago on a machine I no longer have access to.

[ajjackson]

Have not had much luck today trying to follow the mongodb instructions with a Podman container. Will have to try a native installation.

[tpurcell90]

@ajjackson I might be able to help out with the mongodb installation, I never used Podman before, but I have set up the mongodb stuff for FHI-vibes.

[ajjackson]

I got it working and have completed the Fireworks tutorials. I'm pleased to report that the Vibes docs were more helpful than the Fireworks ones when it came to the commands to set up the DB, as the Mongo syntax is more up-to-date. The main issues were around the instructions for shutdown/restart/reset of the DB; this didn't quite play nicely with the container setup. But in the end it was ok and I don't have any specific recommendations for the Vibes docs on that point. The tutorials themselves were enlightening, easy to follow and not too long. There is some exciting functionality in there which should assist some good science (especially the automated convergence testing.)

I reported some minor things here but they do not impact this review https://gitlab.com/vibes-developers/vibes/-/issues/32

I have checked off the functionality documentation as satisfactory; the tutorials give a good run-down of the options and I think at this stage they are fairly essential to understand the package. I would recommend that the authors do also consider the accessibility of their reference/API documentation, especially as the package grows. An auto-generated tree from the docstrings would make this information more navigable/searchable for little development effort.

I now await:

• manuscript revision to statement of need / summary as discussed
• a minor change to the README that makes the target audience of this package slightly more obvious. Something like "Users are expected to already have some familiarity with the atomistic codes that are used to compute forces."

[flokno]

Thank you very much Adam, we'll discuss the changes and come back to you soon.

[flokno]

Hi @ajjackson , thank you very much again for going through the tutorials in this great detail, this was very helpful.

We're addressing your remaining comments in this merge request: https://gitlab.com/vibes-developers/vibes/-/merge_requests/25

In particular, we add the statement

In the documentation and tutorials, knowledge of first-principles electronic-structure theory as well as proficiency with _ab initio_ codes such as FHI-aims and high-performance computing are assumed. Additional experience with Python, the Atomic Simulation Environment (ASE), or Phonopy is helpful, but not needed.

to the README to clarify the prerequisites for running the tutorials.

In the paper, we added a dedicated subsection Features to explicitly list the available features of the 1.0 release in order to avoid possible confusion with methods mentioned in Statement of need which summarizes the scope of FHI-vibes as a framework for vibrational simulations.

Is there anything left to do from your side?

[flokno]

Hi @keipertk , also many thanks to your for reviewing the code.

We added community guidelines as discussed in this merge request: https://gitlab.com/vibes-developers/vibes/-/merge_requests/23

Is there anything else left to do from your side?

[flokno]

@whedon generate pdf from branch joss_paper_review

[whedon]

Attempting PDF compilation from custom branch joss_paper_review. Reticulating splines etc...

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left: