Joss-reviews: [REVIEW]: Lepton

[whedon]

Submitting author: @slithiaote (Sébastien Li-Thiao-Té)
Repository: https://github.com/slithiaote/lepton
Version: v1.2
Editor: @kyleniemeyer
Reviewers: @tpetricek, @rljacobson
Archive: 10.5281/zenodo.3492221

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/5c3cb173a0e6dc2a25a775b5d133b027"><img src="http://joss.theoj.org/papers/5c3cb173a0e6dc2a25a775b5d133b027/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/5c3cb173a0e6dc2a25a775b5d133b027/status.svg)](http://joss.theoj.org/papers/5c3cb173a0e6dc2a25a775b5d133b027)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@tpetricek & @rljacobson, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @yochannah know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @tpetricek

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: v1.2
• [x] Authorship: Has the submitting author (@slithiaote) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @rljacobson

Conflict of interest

• [ ] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [ ] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [ ] Repository: Is the source code for this software available at the repository url?
• [ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [ ] Version: v1.2
• [ ] Authorship: Has the submitting author (@slithiaote) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Authors: Does the paper.md file include a list of authors with their affiliations?
• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @adammichaelwood, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[yochannah]

👋 Gentle nudge - @tpetricek, @adammichaelwood, have you had a chance to start looking at this yet?

[tpetricek]

Thanks for the nudge! I was at a conference last week, but will post my initial comments today or on Thursday.

[adammichaelwood]

Sorry - I've been a bit swamped in all areas.
I can make no promises, but I will try to carve out time for this (and my other review I volunteered for)
as soon as I can.

[tpetricek]

_[Better late than never! Below is my review. I was not able to tick all the boxes above, so the review explains what problems I had and what I think needs to be corrected. I opened one issue for Lepton tracking the most important question, i.e. how to actually compile it]_

The project is a literate programming package that allows users to write code in a classic literate programming style as proposed by Knuth. In case of Lepton, this means using LaTeX syntax with embedded chunks of code that can be combined together to define the program (this also means you can refer to a code block defined later in the file). Lepton is mostly an implementation of the original idea by Knuth, without adding much new (although the documentation is not very clear about that, so there might be new things I might have missed). However, that seems reasonable - a good implementation of the idea for OCaml would be a useful contribution.

Lepton is "implemented in itself" and my understanding is that to build it, you first need to run an existing version (to bootstrap it) and to obtain OCaml code which you can then compile to actually compile Lepton. However, I was not able to do this from scratch (see issue https://github.com/slithiaote/lepton/issues/2). I think the repository needs a Makefile of some sorts and clear instructions about how to bootstrap things (possibly for different OSes, if this cannot be universal).

Not being able to bootstrap Lepton meant that I was not able to tick the following review boxes:

• Installation - as discussed above, I'm not sure how to bootstrap the build process
• Functionality - consequently, I was not able to run Lepton and see what it does

Out of the documentation required for JOSS, I think the following are missing or not clear:

• Installation instructions - I understand that this is tricky given the nature of the project (and the fact that it is written using itself), but there should be clear instructions how to compile it (for some supported platform) and run it. I think this should also use OCaml source code rather than binary for bootstrapping.

• Example usage - There are some instructions in lepton_manual.pdf, but it is not entirely clear to me how to go from nw file to an executable (how to go from nw file to LaTeX file is however quite clear).

• Automated tests - As far as I can see, there are no tests

• Community guidelines - Besides the standard license file, which sets the basic rules, there is not much to guide community contributions. Having a standard CONTRIBUTING file would be useful.

[yochannah]

@slithiaote let me know if/when you've addressed the relevant issues and are ready to move forward with this

[slithiaote]

@yochannah I replied to the issue/report on Nov 7th. At least part of the problem seems to be Windows, which is not a supported platform. I have asked for more input, but @tpetricek did not reply and I am hesitant to ping.

[tpetricek]

@slithiaote I missed the reply. Sorry. I just added a comment on the issue.

That said, I think there are other issues (see my comment above) that need to be addressed before this can be accepted as a JOSS publication - there are a few things that the JOSS checklist requires that I think are not yet satisfied by lepton.

[EDIT] Also, if Windows is not supported platform (not even through the Ubuntu bash), then I don't think I can review this contribution.

[yochannah]

@tpetricek Sorry, I missed the edit note. I can look for another reviewer - it took me a while to find people interested in literate programming - you don't have any suggestions of possible interested colleagues or communities to approach?

[tpetricek]

@yochannah I'm happy to stay involved if the author is interested in supporting Windows, at least through the Ubuntu Bash shell - I don't see why that should be impossible and I think it would be a good thing for the project in general.

No matter what the supported platforms are, I think the project should use some form of pre-generated OCaml source code for bootstrapping rather than a checked-in executable where nobody knows what the executable might be doing. I think making that change (and adding a makefile) would likely make this work on Windows anyway.

That said, if the project can be accepted with a binary file in the repository, then it needs to be reviewed by someone who is able (and brave enough) to run it, so in that case, I won't be able to help. I sadly don't have any good alternative reviewer ideas.

[slithiaote]

@tpetricek Lepton compiles on Windows, but is mostly unusable because it requires the Unix library and Unix tools. Your comment made me realize that Windows Subsystem for Linux can provide those, although I will not be able to test it myself. I have been working on a source-based distribution for the past few days. I should be able to update this week.

@yochannah Have you tried the "reproducible research" community ?

[yochannah]

Okay - this is good - it sounds like we may be able to keep @tpetricek onboard if we can get this on Windows! 👍

I do think I generally agree about @tpetricek's comment re pre-generated OCaml source code.

[slithiaote]

I just committed to Github and tagged v1.1

[yochannah]

Hey all - just checking the status of this review. @tpetricek, has @slithiaote's update managed to resolve your issues? @adammichaelwood, do you think you might have a chance to complete your review soon? Thanks all!!

[tpetricek]

Thanks for the ping. I'm still trying to get it to build. Just added a comment there: https://github.com/slithiaote/lepton/issues/2

In addition to that, the project still does not satisfy some of the JOSS items from the check-list above, so adding those (assuming they are strictly required for acceptance) would be another good thing to do in the meantime (especially, tests, contributor guidelines)

[yochannah]

Quick note - the readme on this repo was updated recently! :) I've added some comments about build issues in https://github.com/slithiaote/lepton/issues/2

[slithiaote]

Yes, I updated the readme recently, but I did not have the time to comment as well. As far as I understand, @tpetricek expects to find installation instructions in README.md instead of lepton.pdf, so I included a copy of these instructions in README.md, as well as contributor guidelines similar to what is present in a few recent accepted JOSS papers.

The project contains two test files: an embedded lepton_manual.nw to produce the usage manual and lepton.nw itself. These two files use most of the functionality of Lepton. I believe that bootstrapping Lepton constitutes a fairly complete test of the software. Incidentally, boostrapping will automatically attempt to process lepton_manual.nw.

[yochannah]

@slithiaote I think I generally do agree - clear getting started docs in the readme is important for all open source software. I've added some comments myself in https://github.com/slithiaote/lepton/issues/2 as I also had trouble getting things to run.

[labarba]

👋 @slithiaote — Have you been able to make progress on this submission in the past month? What's your status?

[slithiaote]

Not much has evolved since my last reply to comments last month. I am unsure what the issue is. The software is written in OCaml using only the standard library, so compilation should be uneventful. I anticipate that bootstrapping may not work out-of-the-box on all systems, but I did not receive error outputs on that topic.

[yochannah]

👋 @adammichaelwood, @tpetricek - would y'all have a chance to revisit this soon & let us know where things are? Thanks!!

[yochannah]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Set the software version at the top of the issue e.g.
@whedon set v1.0.1 as version

# Open the review issue
@whedon start review

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Remind an author or reviewer to return to a review after a
# certain period of time (supported units days and weeks)
@whedon remind @reviewer in 2 weeks

# Ask Whedon to accept the paper and deposit with Crossref
@whedon accept

# Ask Whedon to check the references for missing DOIs
@whedon check references

[yochannah]

/ooo today until april 1

[ooo[bot]]

:+1: Marked @yochannah as OOO from Saturday, March 23rd 2019 to Monday, April 1st 2019. :calendar:

[tpetricek]

@slithiaote Going back to the checklist above, there are two more items "Automated tests" and "Community guidelines" that I did not see in the repo - assuming those are required by the JOSS, would you be able to add those?

[slithiaote]

@tpetricek I believe that "Automated tests" is covered by "Bootstrapping", as this exploits the functionality for which Lepton was written. Bootstrapping will in fact process two embedded files, i.e. run two test files.

I included "Community guidelines" in the "Contributing" section in Readme.md, as this is what I found in other recently published JOSS papers.

[labarba]

JOSS guidelines say this about tests:

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

• Good: An automated test suite hooked up to an external service such as Travis-CI or similar
• OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g. a sample input file to assert behaviour)
• Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works

[slithiaote]

I believe that the bootstrapping process falls into the OK category. The process is manual, and the steps are documented. This process will run two test files which cover the expected functionality of Lepton :

• it will write files to disk, e.g. the source code
• it will run commands, e.g. the compilation commands to produce the Lepton executable
• it will produce documents, e.g. the sources for the manual and tutorial, which can then be compiled to PDF with LaTeX (instructions included)
  The resulting files can be objectively checked against the ones provided in the distribution. In particular, the generated source files should be identical.

[danielskatz]

👋 @tpetricek - please confirm that the documentation for testing is OK, given the discussion above, and if so, please check off the item. If you think something else is needed to make this OK, please explain what you believe is needed.

And for other items in the checklist, what are the blockers to you?

[danielskatz]

👋 @adammichaelwood - are you still going to be able to review this?

[arfon]

I've just emailed @adammichaelwood directly to ask him to complete his review in the next couple of weeks.

[danielskatz]

👋 @tpetricek - please confirm that the documentation for testing is OK, given the discussion above, and if so, please check off the item. If you think something else is needed to make this OK, please explain what you believe is needed.

And for other items in the checklist, what are the blockers to you?

[tpetricek]

Hi @danielskatz - unfortunately, I no longer have access to a UNIX machine in order to be able to run the software, so I'm unfortunately not able to check the functionality claims.

[kyleniemeyer]

Hi @slithiaote @adammichaelwood and @tpetricek , since @yochannah is stepping down from the JOSS editorial team, I'm going to take over editing this article moving forward. Thanks @yochannah for shepherding it so far!

[kyleniemeyer]

@whedon assign @kyleniemeyer as editor

[kyleniemeyer]

It sounds like @adammichaelwood might be able to complete his review in the near future—thank you! I'll set a reminder.

@tpetricek I will check the functionality claims myself, if you are willing to complete the rest of your review.

[kyleniemeyer]

@whedon remind @adammichaelwood in 1 week

[whedon]

Reminder set for @adammichaelwood in 1 week

[tpetricek]

I think the remaining items really depend on the JOSS standards and I don't have a good sense about those. In particular:

• Example usage - The repo does have one example, which is self-hosting. Technically, that's an example usage, but it's pretty sophisticated one that does not actually teach us much about how anyone might use the system for real work. It checks the box "in principle", but I'm not sure it's a good example.

• Automated tests - Technically, self-hosting is a test - but again, this is not what I'd expect. I'd expect some automated unit testing, perhaps running using continuous integration. This is technically ticking the box, but again, I don't think it's following a very high quality standard.

• Community guidelines - There is one vague sentence in the README. This is technically a "community guideline" but it is very basic one, e.g. there is no CONTRIBUTING.md file and it does not really guide anyone interested in how they might get started contributing (e.g. no up-for-grabs issues).

So, I feel that someone with better idea about JOSS standards should decide. I personally don't think the project is "quite there" but maybe my standards are more strict than they should be...

[kyleniemeyer]

Hi @slithiaote, all of @tpetricek's comments above are valid and appropriate from my point of view. Can you address or respond to each?

[whedon]

:wave: @adammichaelwood, please update us on how your review is going.

[slithiaote]

I think that @tpetricek's comments are more appropriate for a large scale software project with ongoing development, whereas this submission is intended as a reference implementation for long-term preservation of the software.

• Example usage - Technically, there are two examples. A very simple tutorial.nw and the full-fledged lepton.nw. As in the literate programming paradigm, Lepton is intended to produce documents containing source code and executable instructions. It was used to produce this submission, so lepton.nw is a good example of real work in my opinion
• Automated tests - I think that continuous integration is suited for projects that evolve rapidly. As it is intended as a reference implementation, the source code of Lepton has not evolved in months. Moreover, testing can be done with diff to make sure that the generated .ml and .tex files are the same. In fact, this is performed automatically by git commit.
• Community guidelines - no problems have been reported for months, so there are no "up-for-grabs" issues at the moment.

[kyleniemeyer]

Hi @adammichaelwood, just checking in. Have you been able to get to this review?

[kyleniemeyer]

@slithiaote regarding your responses,

1. There may technically be two examples, but at least one reviewer found this a bit lacking. Could you improve the documentation around the examples, to make it easier for a new user? It may make sense and work easily for you, but both @yochannah and @tpetricek ran into issues.

2. All software changes, and research software needs tests. JOSS publishes research software, and some sort of automated test is a requirement.

3. I agree that the "contributing guidelines" is extremely minimal. Also, in fact problems have been reported, in an issue filed by @yochannah and @tpetricek.

I will add that in general, we expect that authors of submissions to JOSS will use the peer-review feedback to improve the software—this is one major point of the journal. So far, you have generally been rebutting the feedback, rather than improving your software / documentation / etc. In fact, @yochannah and @tpetricek even filed an issue on the repo, which you haven't satisfactorily addressed, and your repo has not been updated in months.

Most JOSS submissions are not published in the state that they were submitted. A few issues have been raised here, and we'd expect that you would take that useful feedback in the spirit it was given, and work to improve your submission.

[slithiaote]

1. Both @yochannah and @tpetricek ran into installation issues, which I have addressed by switching to a source-based distribution to support Windows Subsystem for Linux, adding a make.sh script (3 lines of code), and writing more detailed instructions and putting those in the README, instead of the main documentation file lepton.pdf. I have replied to the comments on the github issue, but I feel that it is good practice to wait for an acknowledgement that the issue is resolved for the user before closing the issue, so it is still marked as open.
   I am happy to add an intermediate example, but I will need a few days to design one that fits the "real work" criterion. I still believe that lepton.nw is not too sophisticated for that.

2. I agree that all software need tests. In the case of Lepton, I believe that bootstrapping is the best test, and the comments have not indicated flaws with this. Let me point out that boostrapping runs Lepton on a sophisticated example, and writes its output to disk. The command git commit will automatically run the diff command on the generated source files and check that these correspond exactly or show the modifications. It is the purpose of Lepton to generate these .ml and .tex files. I don't understand why this does not qualify as testing, or what benefits a continuous integration system would provide.

3. The "contributing guidelines" are taken from published JOSS papers. They are minimal, but Lepton is not such a large scale project as to require procedures beyond "write an e-mail" or "open an issue on Github". I'd rather keep this simple and welcoming.

I do appreciate feedback, and I have taken care and time in answering. On the topic of installation (open issue in the repository), I have addressed the issue and am still waiting for a followup. On the topic of "automated tests", I have replied to the comments (I claim the right to disagree), but again without a discussion. This has not been helpful to assess whether the issue is satisfactorily addressed or not and why.

[kyleniemeyer]

@whedon remove @adammichaelwood as reviewer

[whedon]

OK, @adammichaelwood is no longer a reviewer

[kyleniemeyer]

@slithiaote I appreciate your response, and apologies for the delay here—the original second reviewer was unable to complete their review, and so I had to find someone else.

Fortunately, @rljacobson has agreed to review. Thank you!

Apologies again for how long this is all taking; I'm hoping @rljacobson's review sheds light on any final changes needed, particularly for ensuring new/unfamiliar users can build and use the software.

[slithiaote]

I uploaded an initial draft of an additional example. It shows how to use Lepton to write a reproducible scientific report. Here, reproducible means that all the source code used in the report is provided, as well as data files, instructions for producing the figures. The .tex report itself can be re-generated with a single shell command, and then compiled to PDF with LaTeX.

[kyleniemeyer]

@whedon remind @rljacobson in one week

[whedon]

Reminder set for @rljacobson in one week

[slithiaote]

I uploaded an improved version of the Fibonacci example. It is now feature-complete and is ready for review.

[whedon]

:wave: @rljacobson, please update us on how your review is going.

[holtgrewe]

Hi, after reaching out to @kyleniemeyer, here are some points outside a formal review (if only out of sentiment from my original background, I'd like to see some more support for the literate programming style in any language):

• Given that Markdown/HTML are stronger on Github that classic LaTeX, it's a bit of a pity that all of the relevant information hides in binary PDF rather than text formats.
• Also, the front-page README.md file did not speak out to me as "Hi, I'm the Markdown that you're used to on a repository front page but I'm a relatively involved MD document with some YAML on the top that you're not interested in." Maybe this could be replace with a common MD document showing links to what people like: continous integration, e.g., through Travis or CircleCI?
• Maybe the PDFs could be built in continuous integration (maybe even HTML?) and be pushed into a separate branch to highlight with "living examples" that this tool can be used on modern infrastructures and setting it up is reproducible?

Although being a big fan of cameloids 🐫 🐪 and having some exposure to Haskell and the like, I'm not an OCaml expert. I'd be happy to try out all the examples and generatin all the PDFs. I'd be happy to install all the packages on all the recommended OSes, but I'm missing the necessary steps.

Maybe the author could provide the steps to install the dependencies on one popular Unixoid operating system (Ubuntu/Debian/CentOS, I'd be happy to start off FreeBSD). If you could provide step-by-step instructions in recreating the PDFs in the repository (starting off apt-get/yum install of your wonderful OCaml packages, continuing with git clone and then calling any build scripts/makefile etc.), I'd be happy to follow along.

Ayway, this is just as an "voice from the off", feel free to ignore me.

[kyleniemeyer]

Hi @rljacobson, just checking in on the status of your review. Thanks!

[slithiaote]

I agree that it would be more natural to write in Markdown for Github, and this is possible in Lepton. However, my priority when writing Lepton was scientific journals, hence LaTeX. Incidentally, the relevant information hides in the lepton.nw file, which is a LaTeX / text format, and not so much harder to read than Markdown, at least for this document.

Concerning README.md, I tried to mimick JOSS examples in the guidelines, but I can very well be wrong there.

Lepton's mindset is to let users reproduce things themselves or show them everything they need. I'd rather users do the steps instead of having a simple command to clone a prebuilt setup. I hope this will make it possible to build Lepton ten years from now, with small adjustments, and I believe that it is not a big hurdle here.

Here are some instructions for Debian

• OCaml can be installed with apt-get install ocaml.
• The git clone command is git clone https://github.com/slithiaote/lepton.git
• You can then compile Lepton with the make.sh script in the repository (run sh make.sh).
• Regenerating the PDFs will require LaTeX : sudo apt-get install texlive python-pygments should install everything. In particular, the minted package is included in texlive-latex-extra

[Kevin-Mattheus-Moerman]

@rljacobson please can you update us on review progress? If you need more time please let us know too. Thanks again for agreeing to help here!

[Kevin-Mattheus-Moerman]

@tpetricek thanks for helping us with this lengthy review process! Can you please list/summarize the main points you feel @slithiaote should focus on? Also can you please comment on (apologies if I've missed your comments somewhere) this reply by @slithiaote:

I think that @tpetricek's comments are more appropriate for a large scale software project with ongoing development, whereas this submission is intended as a reference implementation for long-term preservation of the software.

* Example usage - Technically, there are two examples. A very simple `tutorial.nw` and the full-fledged `lepton.nw`. As in the literate programming paradigm, Lepton is intended to produce documents containing source code and executable instructions. It was used to produce this submission, so `lepton.nw` is a good example of real work in my opinion

* Automated tests - I think that continuous integration is suited for projects that evolve rapidly. As it is intended as a reference implementation, the source code of Lepton has not evolved in months. Moreover, testing can be done with `diff` to make sure that the generated `.ml` and `.tex` files are the same. In fact, this is performed automatically by `git commit`.

* Community guidelines - no problems have been reported for months, so there are no "up-for-grabs" issues at the moment.

[rljacobson]

@rljacobson please can you update us on review progress? If you need more time please let us know too. Thanks again for agreeing to help here!

My sincere apologies for my absence. I had a change of employment followed immediately by health issues, and I completely forgot about this review. I will complete my review ASAP.

[kyleniemeyer]

@rljacobson no worries, thanks for the update! Hope you're doing better.

[rljacobson]

@kyleniemeyer My review is complete, but it appears I cannot edit the checklist. I have one issue:

The paper (paper.md) has an empty References section. That section should presumably include the references @LiThiaoTe20121723 and @LiThiaoTe2012439, and possibly @Knuth84literateprogramming as well.

Review checklist for @rljacobson

Conflict of interest

• [X] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [X] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [X] Repository: Is the source code for this software available at the repository url?
• [X] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

• [X] Version: Does the release version given match the GitHub release (v1.0)?

• [X] Authorship: Has the submitting author (@slithiaote) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [X] Installation: Does installation proceed as outlined in the documentation?

• [X] Functionality: Have the functional claims of the software been confirmed?

• [X] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [X] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [X] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [X] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [X] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

• [X] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?

• [X] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [X] Authors: Does the paper.md file include a list of authors with their affiliations?
• [X] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kyleniemeyer]

@rljacobson thanks! The references are automatically included in the PDF from the associated .bib file, and aren't supposed to be in the Markdown file, so this looks ok.

[rljacobson]

@kyleniemeyer Great, then my review is complete. He gets a pass from me. Looks good.

[kyleniemeyer]

@slithiaote at this point I believe we can accept this submission. I just submitted a minor PR with fixes to the paper, if you could merge this: https://github.com/slithiaote/lepton/pull/4

Then, please archive the software repository at Zenodo and let me know the resulting DOI.

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[slithiaote]

I just tagged the release v1.2 with DOI 10.5281/zenodo.3492221
I added missing DOIs to the bib file and made cosmetic LaTeX changes.
Thank you for the suggestions.

[kyleniemeyer]

@whedon set 10.5281/zenodo.3492221 as archive

[whedon]

OK. 10.5281/zenodo.3492221 is the archive.

[kyleniemeyer]

@slithiaote sorry, one last thing: can you clean up the metadata on the Zenodo archive? The title and author list should match the paper.

[slithiaote]

Done. It should be ok now.

[Kevin-Mattheus-Moerman]

@whedon set v1.2 as version

[whedon]

OK. v1.2 is the version.

[kyleniemeyer]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

PDF failed to compile for issue #1005 with the following error:

E, [2019-10-17 21:09:28#627] ERROR -- : Failed to parse BibTeX on value "This" (NAME) ["@", "Comment"]
bibtex.y:138:in on_error': Failed to parse BibTeX on value "This" (NAME) ["@", "Comment"] (BibTeX::ParseError) from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/racc/parser.rb:259:in_racc_do_parse_c'
from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/racc/parser.rb:259:in do_parse' from bibtex.y:111:inparse'
from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/bibliography.rb:67:in parse' from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/bibliography.rb:50:inopen'
from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/utilities.rb:25:in open' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/bibtex_parser.rb:36:ingenerate_citations'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/compilers.rb:222:in crossref_from_markdown' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/compilers.rb:20:ingenerate_crossref'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/processor.rb:95:in compile' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/bin/whedon:79:incompile'
from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command'
from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/bin/whedon:116:in <top (required)>' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:inload'
from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:in `

'

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[Kevin-Mattheus-Moerman]

@kyleniemeyer do you want me to take it from here? Or are you processing this one?

[kyleniemeyer]

@Kevin-Mattheus-Moerman feel free to give it another look—though I'm not sure what caused the error above, since the PDF seems to generate fine.

[Kevin-Mattheus-Moerman]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

PDF failed to compile for issue #1005 with the following error:

E, [2019-10-18 20:18:05#375] ERROR -- : Failed to parse BibTeX on value "This" (NAME) ["@", "Comment"]
bibtex.y:138:in on_error': Failed to parse BibTeX on value "This" (NAME) ["@", "Comment"] (BibTeX::ParseError) from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/racc/parser.rb:259:in_racc_do_parse_c'
from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/racc/parser.rb:259:in do_parse' from bibtex.y:111:inparse'
from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/bibliography.rb:67:in parse' from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/bibliography.rb:50:inopen'
from /app/vendor/bundle/ruby/2.4.0/gems/bibtex-ruby-5.0.0/lib/bibtex/utilities.rb:25:in open' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/bibtex_parser.rb:36:ingenerate_citations'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/compilers.rb:222:in crossref_from_markdown' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/compilers.rb:20:ingenerate_crossref'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/lib/whedon/processor.rb:95:in compile' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/bin/whedon:79:incompile'
from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command'
from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart'
from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-1e4ee47b240d/bin/whedon:116:in <top (required)>' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:inload'
from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:in `

'

[Kevin-Mattheus-Moerman]

@openjournals/dev @arfon :point_up: do you know what is going on here?

[slithiaote]

Is it perhaps related to the comment in the first line of my bibtex file ?
@Comment This file was generated from lepton.nw by lepton.bin

[arfon]

Yeah, any chance you could remove that line please @slithiaote?

[Kevin-Mattheus-Moerman]

@slithiaote can you merge this PR https://github.com/slithiaote/lepton/pull/5 to fix the above. Once you do that can you run @whedon generate pdf and tag @openjournals/joss-eics so we can resume the process of accepting your work. Thanks

[slithiaote]

I would like to keep the Bibtex comment because it makes it clear that users should not edit the bib file directly. I can remove @COMMENT and leave the text dangling outside the Bibtex entries. Both methods are official ways to add comments in Bibtex files.

Do I need to tag a new release on zenodo with tag @openjournals/joss-eics ?

[arfon]

I can remove @COMMENT and leave the text dangling outside the Bibtex entries. Both methods are official ways to add comments in Bibtex files.

That probably works. No new release is necessary.

[slithiaote]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

```Reference check summary:

OK DOIs

• 10.1093/comjnl/27.2.97 is OK
• 10.1016/j.procs.2012.04.190 is OK
• 10.1016/j.procs.2012.04.047 is OK
• 10.1109/52.311070 is OK
• 10.5281/zenodo.1311588 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1039

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1039, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[whedon]

```Reference check summary:

OK DOIs

• 10.1093/comjnl/27.2.97 is OK
• 10.1016/j.procs.2012.04.190 is OK
• 10.1016/j.procs.2012.04.047 is OK
• 10.1109/52.311070 is OK
• 10.5281/zenodo.1311588 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

```Reference check summary:

OK DOIs

• 10.1093/comjnl/27.2.97 is OK
• 10.1016/j.procs.2012.04.190 is OK
• 10.1016/j.procs.2012.04.047 is OK
• 10.1109/52.311070 is OK
• 10.5281/zenodo.1311588 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[Kevin-Mattheus-Moerman]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/1046
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01005
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[Kevin-Mattheus-Moerman]

@openjournals/dev this DOI is not resolving yet

[arfon]

@openjournals/dev this DOI is not resolving yet

Looks like the Crossref queue is backed up. It should register in the next few hours.

[kyleniemeyer]

@Kevin-Mattheus-Moerman @arfon looks good now!

[kyleniemeyer]

Congrats @slithiaote on your article's publication in JOSS—thanks for being patient over this long process. Many thanks to @tpetricek and @rljacobson for reviewing this submission!

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01005/status.svg)](https://doi.org/10.21105/joss.01005)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01005">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01005/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01005/status.svg
   :target: https://doi.org/10.21105/joss.01005

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[slithiaote]

Thanks to the editors and reviewers from JOSS for their dedication and perseverance.