Joss-reviews: [REVIEW]: DynamicalSystems.jl: A Julia software library for chaos and nonlinear dynamics

Created on 27 Feb 2018 · 39Comments · Source: openjournals/joss-reviews

Submitting author: @Datseris (George Datserus)
Repository: https://github.com/JuliaDynamics/DynamicalSystems.jl
Version: v0.11.0
Editor: @kyleniemeyer
Reviewer: @dhhagan
Archive: 10.5281/zenodo.1198278

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/0ffb78b6f23fca0e94a57f68e4787bd6"><img src="http://joss.theoj.org/papers/0ffb78b6f23fca0e94a57f68e4787bd6/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/0ffb78b6f23fca0e94a57f68e4787bd6/status.svg)](http://joss.theoj.org/papers/0ffb78b6f23fca0e94a57f68e4787bd6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@dhhagan, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @kyleniemeyer know.

### Conflict of interest

[x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Version: Does the release version given match the GitHub release (v0.11.0)?
[x] Authorship: Has the submitting author (@Datseris) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

accepted published recommend-accept review

Source

whedon

All 39 comments

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @dhhagan it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

whedon on 27 Feb 2018

Attempting PDF compilation. Reticulating splines etc...

whedon on 27 Feb 2018

--> Check article proof :page_facing_up: <--

whedon on 27 Feb 2018

@Datseris @dhhagan @ahwillia @mlxd 👋 the review will take place here.

@dhhagan you are the "main" reviewer, so please fill out the checklist above, but @ahwillia @mlxd please also see if you agree that those points were satisfied (or not) and comment appropriately.

Thanks!

kyleniemeyer on 27 Feb 2018

👍1

Hi @Datseris; everything looks great with the exception of one missing DOI as mentioned in Issue 62. I believe this is the last minor hold-up!

dhhagan on 5 Mar 2018

👍1

Hello @dhhagan. Thank you for the kind words. I fixed your reported issue with the latest commit.

@kyleniemeyer I was wondering, whether it is possible to associate the paper with version v0.12 of the software, instead of the current one v0.11.

During the time this review was underway, a new feature was added (which is significant enough for a new version). I am asking you now, while v0.12 is still not tagged.

Datseris on 5 Mar 2018

@Datseris that's no problem, we expect the version to change after reviewer comments.

However, if it's a significant change, then I would ask that @dhhagan take another look, with that new feature in mind.

kyleniemeyer on 12 Mar 2018

(and @ahwillia @mlxd as well!)

kyleniemeyer on 12 Mar 2018

There is no reason for a second look, the feature was already in the
documentation before the reviewer started the review (because it comes from
another smaller package which is a dependency )

On Mar 12, 2018 7:24 AM, "Kyle Niemeyer" notifications@github.com wrote:

(and @ahwillia https://github.com/ahwillia @mlxd
https://github.com/mlxd as well!)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/598#issuecomment-372205016,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASwgYaZ4V8Hmj7aML1JAAwAaBIyJlg2uks5tdhS4gaJpZM4SVqxP
.

Datseris on 12 Mar 2018

@kyleniemeyer @Datseris Okay just had a last look and it appears good to go! Let me know if there is anything else I need to do?

dhhagan on 12 Mar 2018

@dhhagan you are good to go, thanks! I'm just waiting to see if @ahwillia and/or @mlxd are able to add any additional comments.

kyleniemeyer on 12 Mar 2018

I have it on my calendar to take a look at it this on Thursday. What would be most useful for me to focus on?

@Datseris - can you provide a short discussion about how this library differs from DifferentialEquations.jl?

@ChrisRackauckas would you also mind weighing in on this briefly? Thanks so much.

ahwillia on 12 Mar 2018

@ahwillia great, thanks! If you have time, I'd prefer for you to give the whole thing (software + paper) a look-over, independent of @dhhagan's review. At minimum, your thoughts on the software and whether it does what it advertises (etc.) would be helpful.

kyleniemeyer on 12 Mar 2018

Hi, I'm afraid I won't have much time to examine this and offer any feedback for another week at the earliest. I feel that if any of the others can weigh in on it may be best to proceed without me.

mlxd on 12 Mar 2018

👍1

@ahwillia I am not sure to respond because I don't understand the question.

Can you explain what you are asking about?

(your question implies that I have to somehow justify some difference
between the submitted package and differential equations, but that would
not make sense )

On Mar 12, 2018 6:12 PM, "Alex Williams" notifications@github.com wrote:

I have it on my calendar to take a look at it this on Thursday. What would
be most useful for me to focus on?

@Datseris https://github.com/datseris - can you provide a short
discussion about how this library differs from DifferentialEquations.jl
https://github.com/JuliaDiffEq/DifferentialEquations.jl?

@ChrisRackauckas https://github.com/chrisrackauckas would you also mind
weighing in on this briefly? Thanks so much.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/598#issuecomment-372388898,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASwgYQ54IWuyj0AuwPtx02h-pzDFzJBrks5tdqyGgaJpZM4SVqxP
.

Datseris on 12 Mar 2018

@ahwillia there's not really a similarity. DifferentialEquations.jl and JuliaDiffEq develops solvers and modeling packages. DynamicalSystems.jl is tooling for analysis of dynamical systems, such as determining Lyopunov exponents. It internally uses DifferentialEquations.jl to do the evolution of the systems, but that's only a small part of what's required for its algorithms.

ChrisRackauckas on 12 Mar 2018

Thanks Chris, this was my basic understanding. I have not had the chance to dig too deeply into this yet, but I could see new users being a bit confused.

@Datseris -- thank you for the detailed section on "Similar Existing Software". Would you mind adding a paragraph there or somewhere in the README / Documentation to make it clear to users that the numerical solvers are provided by DifferentialEquations.jl and the purpose of this package is to help characterize the behavior of solutions produced from said solvers? Something as simple as:

While many libraries exist for numerically solving differential equations (CITATION DUMP), only a few libraries focus on X, Y, and Z.

The problem is not to "justify some difference" but to make the purpose of the library as clear as possible.

ahwillia on 12 Mar 2018

It is already clear to the users that we are using DifferentialEquations.jl : https://juliadynamics.github.io/DynamicalSystems.jl/latest/definition/general/#DynamicalSystemsBase.DynamicalSystem

The documentation string of the struct DynamicalSystem is absolutely crucial for the entire library and I assume that any user that wants to use the non-data aspect of DynamicalSystems.jl has to have read it at least once. Thus, the usage of DifferentialEquations.jl is fully transparent.

(not only that, but it is also the first documentation string of the library, so it is pretty hard to miss)

and the purpose of this package is to help characterize the behavior of solutions produced from said solvers?

This is very far way from the truth. I would say that only 1/10th of the total exported functions actually use the "solutions produced from said solvers". The rest 9/10th either does pure data analysis using Base Julia (and StaticArrays), or uses NearestNeighbors, or other packages, or internal solvers for discrete systems. Don't get me wrong, DifferentialEquations.jl is spectacular, but saying that all of DynamicalSystems.jl simply "helps" characterize the behavior of solutions of DifferentialEquations.jl is very undervaluing, in my eyes at least. This is perfectly clear from the contents list, https://juliadynamics.github.io/DynamicalSystems.jl/latest/#contents

@ahwillia Have you read the documentation of the submitted package, or at least the list of features offered by it?

Your initial question on the "differences" between the submitted library and DifferentialEquations.jl hinted that you have actually not read it, and the last comment only enhances this suspicion. I sincerely believe that this is not helpful in the current review process.

Datseris on 12 Mar 2018

@Datseris I would ask to tone down the language a little bit. It is a perfectly valid reviewer comment that the differences between this and your previous package are not entirely clear based on the article/README, and it should be fairly easy to improve the description to clear this up for a reader. The differences and justification are likely quite clear to you, but (evidently) not necessarily to someone else reading about the package.

kyleniemeyer on 13 Mar 2018

👍1

Did not mean to undervalue the package at all. Something to "help" analyze dynamical systems is extremely useful. But it was unclear to me from reading the article proof whether the goal of the library was analysis or simulation or a mix of both. I also looked at the README and first front page of documentation.

I think it is reasonable to point new users to DifferentialEquations.jl and explain when & why to use that library versus DynamicalSystems.jl. I don't think a new user should have to dive into the docstrings of specific methods/types to figure this out.

I've read through the documentation more deeply now and I've actually gone back and skimmed the "E&F Chaos" paper and related packages to try to get a better sense about what this package does and the motivations behind it. It's a very technical library and I won't pretend to deeply understand some of the features (though I actually do have some experience with nonlinear dynamical systems, perhaps more than the average newcomer). The documentation seems well referenced and I was able to download the library, pass unit tests, and reproduce examples in the documentation.

I expect this package will be difficult for users without strong knowledge of this field. That being said, for those with the requisite background I could see this package being quite useful. It would be nice to incorporate more tutorials and pedagogical flow to the docs, but I do not expect this suggestion to be well received (and I do not view it as necessary). I do really appreciate the theorems which are sprinkled throughout the docs (e.g. the "shadowing theorem") and again the docs are very well-referenced with primary sources.

Consider me signed off.

ahwillia on 13 Mar 2018

I think it is reasonable to point new users to DifferentialEquations.jl and explain when & why to use that library versus DynamicalSystems.jl. I don't think a new user should have to dive into the docstrings of specific methods/types to figure this out.

@ahwillia Sure, this is a valid suggestion, but I can't think of a more natural point to do this, than the point that I introduce the concept that actually uses DifferentialEquations.jl. Doing it any point earlier does not seem very fitting, because as I said in my previous comment, only 1/10th of the exported functionality actually uses DifferentialEquations.jl.

Can you suggest a place where your suggestion would fit better?

I expect this package will be difficult for users without strong knowledge of this field. That being said, for those with the requisite background I could see this package being quite useful. It would be nice to incorporate more tutorials and pedagogical flow to the docs

Thanks for pointing this out. My topmost priority with this package is to be more like a "library", and to be helpful for new users to also be able to learn new methods, and understand new methods. To this end I have tried to set up the following:

Every function is accompanied with a short description of the underlying algorithm.
References are given to scientific articles that first introduce the method or had significant influence on the method.
The documentation sections have one or two sentences about where and why you would like to apply the methods.
In many cases, caveats and difficulties are pointed out. Most of the time knowledge of these stuff comes from hands-on experience on working with a methods in scientific projects.

Would you mind telling me first how I can improve on this and, if possible, which points were not very clear for you when you went through it?

@kyleniemeyer

@Datseris I would ask to tone down the language a little bit. It is a perfectly valid reviewer comment that the differences between this and your previous package are not entirely clear based on the article/README, and it should be fairly easy to improve the description to clear this up for a reader.

The reviewer did not comment at my previous package. I really do not understand your comment. "My" previous package (which is actually part of a GitHub organization and thus not associated with my name directly) is about billiard systems. But the reviewer never inquired about this, they inquired instead about the difference between the current submission and a solver of differential equations. I can't help but be confused. ???

Datseris on 13 Mar 2018

Following @ahwillia I've opened the following issue: https://github.com/JuliaDynamics/DynamicalSystems.jl/issues/65 which I'll solve at the near future.

Datseris on 13 Mar 2018

@Datseris ah, I misremembered the bit about your prior package, sorry about that.

Aside from the issue you just mentioned, I have a few fixes for your paper:

Could you add additional location details to your affiliation, such as city and country?
The first reference, Strogatz, is missing bibliographic details—it just has an author and title. Is this an article or book? Either way, it needs some additional information (year, DOI, etc.)
In the first sentence, the semicolon should be a comma.
In the third sentence, "This vast extend" -> "This vast extent" (I think)
In the first paragraph in the "Examples" section, there is a paragraph with "## Lyapunov spectrum of a continuous ..."; I think this was meant to be part of the code snippet that follows, rather than in the paragraph.
The citation for DynamicalBilliards doesn't look quite right; perhaps remove the publisher field from the Bibtex entry?

Please let me know when you've made all those fixes, and also any remaining changes to the software. At that point, you will need to archive the repo (using Zenodo, for example) and provide the DOI here; the JOSS paper will be associated with whatever version that corresponds to.

kyleniemeyer on 13 Mar 2018

I appreciate the desire to make the documentation more pedagogical. I really think it is a laudable goal and a great service to the field.

Would you mind telling me first how I can improve on this and, if possible, which points were not very clear for you when you went through it?

It is great that each method includes a sentence or two description and again I really appreciate the amount of primary references and citations you included.

What I do not full understand (and this is partly my ignorance), is how a user should go about using all the functionality of this package at once on a single problem. That is, I would love to see a unifying example or vignette that pulls together the simulation, data analysis, and visualization tools. Let's say I have a new dynamical system that describes population dynamics of some species. What can your package tell me about it? What should I do first, and what should my follow-up questions be?

Perhaps putting a short roadmap/description about each of the six headings on this page would also help: https://juliadynamics.github.io/DynamicalSystems.jl/latest/chaos/overview/

To summarize, the documentation of individual functions is generally good, but I had a bit of trouble connecting concepts and figuring out how everything should synergistically work together.

Just my two cents. Thanks for all the effort you are putting into the package.

ahwillia on 13 Mar 2018

@ahwillia

What I do not full understand (and this is partly my ignorance), is how a user should go about using all the functionality of this package at once on a single problem. That is, I would love to see a unifying example or vignette that pulls together the simulation, data analysis, and visualization tools. Let's say I have a new dynamical system that describes population dynamics of some species. What can your package tell me about it? What should I do first, and what should my follow-up questions be?

Unfortunately it is not possible for me to answer this question(s). It would be a great risk of misguiding someone into results they don't understand.

Each function of DynamicalSystems.jl has a well-defined purpose, mostly independent from the purpose of other functions (excluding of course cases where functions call each other). And each one can give you some information for your system. The question is, what information do you need? The user, as a scientist, should be the judge - I cannot help them in that decision.

I can only help by stating clearly what each method does (which at least in my eyes, I am fulfilling this goal, and also have an example application for any possible case). For further knowledge, one has to read papers and study. Having attended an introductory course in nonlinear dynamics and chaos is kind of mandatory. Just as having attended an introductory course in quantum mechanics is kind of mandatory for using a package about quantum mechanics, the same thing applies in using a package for nonlinear dynamics and chaos. It is an entire field of science after all (which also expands beyond physics and goes into chemistry, biology, economics, population dynamics, electronics and many more).

Mayhaps it is a good idea to show/cite some introductory textbooks on the field? I do that in the paper but not in the documentation.

(In the introduction, I do point out that the "target audience" of the library are scientists working on the field of nonlinear dynamics and chaos, in the goals section)

but I had a bit of trouble connecting concepts and figuring out how everything should synergistically work together.

You had trouble connecting concepts exactly because there isn't any direct connection. They are a lot of tools for aiding in the study of dynamical systems, but they are not inter-connected in a "simple" manner. In fact, if you had the impression that everything works synergistically together, then I would have have written a bad documentation :D

P.S.: Many of the methods are actually connected in a rigorous and formal mathematical manner. I am stating this in case some expert ever reads this review. The connection is very deep though, sometimes with quite advanced math, and way beyond the scope of a documentation to discuss about.

Datseris on 14 Mar 2018

@kyleniemeyer I have done the changes you requested and tagged a new release.

The Zenodo DOI is: 10.5281/zenodo.1198278

Datseris on 14 Mar 2018

@whedon generate pdf