Joss-reviews: [REVIEW]: Model dispersion with PRISM; an alternative to MCMC for rapid analysis of models

[whedon]

Submitting author: @1313e (Ellert van der Velden)
Repository: https://github.com/1313e/PRISM
Version: v1.0.0
Editor: @arokem
Reviewer: @fonnesbeck
Archive: 10.5281/zenodo.2572736

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf"><img src="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg)](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@fonnesbeck, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @fonnesbeck

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v1.0.0)?
• [x] Authorship: Has the submitting author (@1313e) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @fonnesbeck it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[1313e]

As mentioned in the blog here (http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing),
it must be clear to all parties involved that a JOSS submission is associated with an AAS Journals submission.
Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@fonnesbeck @arokem Do you need the AAS Journals submission reference for this by any chance?

[arfon]

Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@1313e many thanks for letting us know about this - I think you're the first!

@fonnesbeck - when you get a chance, please review this blog post from late last year: http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing . In particular, please let us know if you have any objection to JOSS receiving a small donation from AAS publishing for this.

[1313e]

@1313e many thanks for letting us know about this - I think you're the first!

Yeah, the news about that collaboration came just in time before my paper submission, so I am not surprised I am the first one.

[1313e]

@arfon May I btw mention that AAS Journals has no system in place at all that a submission is being accompanied by a JOSS submission?
I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

[arfon]

I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

Yes, I think this is still a work in progress.

/ cc @crawfordsm who is the software editor for AAS so he has some visibility into this review too.

[crawfordsm]

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

[1313e]

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

@crawfordsm Do you, by any chance, need the submission reference, to make it easier to find which paper it actually is that I submitted to AAS Journals?

[crawfordsm]

It's okay, I have the information. I am the Scientific Editor for the paper, so I can monitor it here as well as on the AAS side. I realize now that I had just overlooked the information. As of right now, we do not have a formal process for this, but appreciate your patience as we figure out the details.

[arokem]

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

[arfon]

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

That's _mostly_ correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

[arokem]

Understood! OK - @fonnesbeck - if this all sounds agreeable to you, please do go ahead with your review. Once the review process here is over, we can circle back and see what we need to do. Thanks!

[1313e]

That's _mostly_ correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

@arfon That would be required anyways, given that I am citing the AAS Journals submission for the results.
I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

I hope that is okay btw, as I couldn't find any guidelines on this.

[arfon]

I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

:+1: yes that's right. We also don't allow scientific results to be published in JOSS :-)

I hope that is okay btw, as I couldn't find any guidelines on this.

I think you've got this right. @crawfordsm and I are in the process of drafting some guidelines that should assist future authors.

[fonnesbeck]

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

[fonnesbeck]

I'm having problems running the example model. See 1313e/PRISM#7

[1313e]

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

@fonnesbeck You will have to install all requirements in requirements_dev.txt before running the pytests.
I could put that in the README for clarity, if required.

[fonnesbeck]

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

[1313e]

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

@fonnesbeck Alright, I have added it to the README.
I have also "fixed" 1313e/PRISM#7.

[fonnesbeck]

Tests now run after installing the dev requirements, but the tests fail.

[fonnesbeck]

The submission satisfies all of the evaluation criteria. Once the issue above is addressed, I'm happy to recommend ACCEPT.

[1313e]

@fonnesbeck I have solved the problem with 1313e/PRISM#8, and also improved the README and docs on the topic of running the pytests.

@arokem Do you have an idea how I can fix the formatting error that whedon gives when compiling the paper.md?
It formats my last name incorrectly in the footer of every page (I already noticed that immediately after submission, but had not asked about it yet).

[arfon]

@1313e - I’ll have to fix the name issue manually at the end (the automated proofs will continue to have this problem)

[fonnesbeck]

I am now able to run the test suite without issue. There are no other outstanding problems, so I recommend ACCEPT. cc @arfon

[arfon]

@arokem - let me know when you're ready to accept this - I'll have to manually process this one to deal with Whedon's poor handling of van der Velden as a last name.

[arokem]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arokem]

@1313e

Two comments on the manuscript:

1. The first is that the link to the software is not necessary. As you can see in the rendered PDF, there will be a link to your GitHub repo on the first page of the manuscript. This is not a required fix.

2. Could you please add the DOI for the Raftery et al. paper? (DOI: 10.1080/01621459.1995.10476529). This one is required.

[1313e]

1. Could you please add the DOI for the Raftery et al. paper? (DOI: 10.1080/01621459.1995.10476529). This one is required.

Alright, fixed that. I had actually never been able to find a DOI for that paper, so thanks for finding that. :)

@arokem - let me know when you're ready to accept this - I'll have to manually process this one to deal with Whedon's poor handling of van der Velden as a last name.

I guess we will have to wait with accepting the paper until the AAS Journals submission has been reviewed, right?

[arfon]

I guess we will have to wait with accepting the paper until the AAS Journals submission has been reviewed, right?

Yes, I think that's right. @crawfordsm - does that sound about right to you?

[arokem]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arokem]

OK - I believe that this manuscript is ready to be accepted from our end.

@arfon: should @1313e create a Zenodo archive with the code as it is now, or wait until the AAS journal review is also resolved?

[arfon]

@arfon: should @1313e create a Zenodo archive with the code as it is now, or wait until the AAS journal review is also resolved?

I think it's OK to create the archive now.

[1313e]

Alright, should I make an archive of the current state of the master branch?

[arfon]

Yes please!

[1313e]

Alright, I had to get back out of my bed for it, but here is the zenodo DOI: 10.5281/zenodo.2572736

[arfon]

@whedon set 10.5281/zenodo.2572736 as archive

[whedon]

OK. 10.5281/zenodo.2572736 is the archive.

[1313e]

So, I guess all that is remaining is linking the JOSS submission with the AAS Journals submission (which is necessary anyway for the last paragraph) and manually adjust the PDF to have my last name spelled correctly?

Btw, I am actually wondering why whedon is writing my last name this weird.
I have seen my name being spelled incorrectly in many different ways before, as most other countries do not understand that a last name can contain multiple words which should not be capitalized, but I have never seen it being spelled as "Van Velden" before.

Fun fact: I have seen no system thus far (besides those in my home country obviously) that can spell it correctly and still understand that my name should be alphabetically sorted on "Velden" rather than "van".
For my name it does not matter that much given how close the two words are together alphabetically speaking, but there are many cases in which this is not true.

[arokem]

I must be missing something. Isn't your last name already spelled correctly here?

[1313e]

@arokem Take a look at the footer of every page.

[arfon]

Just to (re)confirm. When we finally accept this paper, I will fix this manually. The library we use for handling firstnames/lastnames can't handle van der so I have to fix papers by hand when this comes up.

[arokem]

Gotcha. Sorry I missed that!

[1313e]

@crawfordsm @arfon @arokem
Alright, so I am currently in the process of revising the AAS Journals submission.
One of the things that still needs to be done, is linking the JOSS submission and the AAS Journals submission together.
However, as neither submission will get accepted before the other, no DOI exists for either submission.
This makes it hard for me to cite one submission in the other, which is necessary for both.

So, therefore, I wanted to ask: Any ideas on how I should cite either submission in the other?

PS: I coincidentally went on a three week holiday the day that I received the referee report from AAS Journals, which is why it took until now to get to it.

[arfon]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

@1313e - we actually know ahead of time what the JOSS DOI will be - in this case it's https://doi.org/10.21105/joss.01229 (this won't resolve yet though).

I also believe that AAS _should_ know their DOI towards the end of their editorial process. @crawfordsm - is that the case?

[1313e]

@arfon Right, I forgot that the DOI for JOSS submissions is actually really easy.
However, I still wanted to raise this, as I think it is a good idea to discuss how to proceed.

[arfon]

However, I still wanted to raise this, as I think it is a good idea to discuss how to proceed.

Yes, definitely worth discussing! @crawfordsm and I are overdue producing some documentation for authors such as yourself describing how all of this works 😄

[1313e]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[1313e]

Also, if I am looking at the volume and issue number logic of previously accepted papers, I am assuming that this one is going to be 4(37) (as it is probably not going to be accepted this month).
I am asking as I want to include the full citation to the JOSS paper in the AAS Journals submission, and I can pretty much guess what the entire bibtex entry is going to look like.

[arfon]

4(37) <- yes, if the paper is accepted in May then this will the be correct volume/issue.

[1313e]

@crawfordsm Now that the AAS Journals submission has been accepted, how do we proceed here?

[1313e]

@arfon I heard there are some problems with cross-refs at the moment?

[arfon]

@arfon I heard there are some problems with cross-refs at the moment?

Yeah, I suspect this will get fixed tomorrow. For now, we're not able to deposit with Crossref.

[1313e]

@arfon @crawfordsm Has it already been decided how the JOSS and AAS Journals submissions are going to be linked? Now that the AAS Journals submission has been accepted as well, I guess it is time to link them together (which can be done while cross-refs are not working anyways)?

PS: I remember a friend of mine showing me something from JOSS last Friday, that looked like some special submission page had been set up for AAS Journals and JOSS submissions. I however am unable to find it again, so I am not sure what it was.

[arfon]

@arfon @crawfordsm Has it already been decided how the JOSS and AAS Journals submissions are going to be linked? Now that the AAS Journals submission has been accepted as well, I guess it is time to link them together (which can be done while cross-refs are not working anyways)?

We're going to as you to make sure that both papers cite each other. Right now, we don't know what the AAS DOI will be so having the JOSS paper cite the AAS paper isn't likely possible for another few days.

PS: I remember a friend of mine showing me something from JOSS last Friday, that looked like some special submission page had been set up for AAS Journals and JOSS submissions. I however am unable to find it again, so I am not sure what it was.

We don't have anything like this sorry. Perhaps you're thinking of this http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing ?

[1313e]

We don't have anything like this sorry. Perhaps you're thinking of this http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing ?

No, I remember there being some kind of form with the logo of JOSS covering the entire background of it.
Problem is that I cannot ask the person who showed it to me, as he is on a holiday.
I might be wrong though, perfectly possible.

However, I am assuming that the DOI for AAS Journals will be known when the paper is actually published.
So, that would mean waiting for at least another month.

[1313e]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[1313e]

@arfon @crawfordsm Well, since I got the DOI for the ApJS manuscript this morning, I have added it to the JOSS manuscript.
I guess there are some other things that need to be changed or added, as currently it is not very clear that both manuscripts are somewhat linked.

[arfon]

@arfon @crawfordsm Well, since I got the DOI for the ApJS manuscript this morning, I have added it to the JOSS manuscript.

Excellent. You might want to add the issue/volume numbers when we have them too.

I guess there are some other things that need to be changed or added, as currently it is not very clear that both manuscripts are somewhat linked.

Right. I'm talking with AAS publishing about finding a way to show these are linked in some way. Watch this space - I should have an update on this soon. Also, thanks for your patience (and being the first to do this) - as you can tell, we have some things to work through with AAS here 😸

[1313e]

Adding the issue/volume numbers is going to be difficult I think, as that really requires the manuscript to be published.

[1313e]

@arfon Any news/updates?

[arfon]

@arfon Any news/updates?

@crawfordsm - any news from AAS on this paper?

Right. I'm talking with AAS publishing about finding a way to show these are linked in some way. Watch this space - I should have an update on this soon. Also, thanks for your patience (and being the first to do this) - as you can tell, we have some things to work through with AAS here 😸

BTW, we've decided that the JOSS papers won't need page/issue/volume numbers for the AAS papers as long as they have the correct DOI for the AAS paper in the citation.

[1313e]

@arfon So, is there going to be something like a unique/specific identifier going to be added to both papers (or something like a note) that states the papers are linked?

[arfon]

@1313e - am I correct that the AAS DOI for this paper will be 10.3847/1538-4365/ab1f7d?

If so, please add the following fields to the top of your paper.md file:

aas-doi: 10.3847/1538-4365/ab1f7d
aas-journal: Astrophysical Journal

[1313e]

@arfon Sure. However, shouldn't it be "Astrophysical Journal Supplement Series"?

[1313e]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

Yes, please change the line: aas-journal: Astrophysical Journal to aas-journal: Astrophysics Journal Supplement Series

[1313e]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

@1313e - I'm going to go ahead and accept this paper now as I believe the AAS paper is very close to being published too.

Apologies for the back and forth on this submission - you're the first author to follow the dual-publishing workflow in AAS/JOSS and we've been finding (and fixing) a few challenges along the way. Your patience has been much appreciated!

[arfon]

@fonnesbeck - many thanks for your review here and to @arokem for editing this submission ✨

@1313e - your paper is now accepted into JOSS :zap::rocket::boom:

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01229/status.svg)](https://doi.org/10.21105/joss.01229)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01229">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01229/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01229/status.svg
   :target: https://doi.org/10.21105/joss.01229

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[1313e]

@arfon Will you also make sure that my last name is properly written in the published version?

[arfon]

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

[1313e]

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

Looks fine to me.

[1313e]

@arfon I was actually just looking at the paper, and noticed that the citations are not displayed properly. The first in every sequence is, but the other ones are not.
As far as I know, this was not the case with the automatically compiled versions.
Did I write the citations wrong?

[arfon]

@1313e - could you be more specific about what you're seeing that you believe is incorrect? i.e. give me an example of expected vs actual?

[1313e]

@arfon Well, the second sentence of the third paragraph shows up as "PRISM uses the Bayes linear approach (Goldstein & Wooff, 2007), emulation technique (Craig, Goldstein, Seheult, & Smith, 1996, p. @Craig97) and history matching (Raftery, Givens, & Zeh, 1995, pp. @Craig96, @Craig97)...".
I have the feeling that a few citations here were not actually formatted.

[arfon]

Ah, right. Sorry about that. The correct syntax for multiple citations is to separate them with semi-colons (e.g. [@Raftery95; @Craig96; @Craig97]). I've fixed up the paper locally in https://github.com/openjournals/joss-papers/blob/a66fbee618680adc4825d723edcbb40165e1dda0/joss.01229/10.21105.joss.01229.pdf and it should be correct now on the live site (note it can take a few hours for the updated PDF to show properly because of caching on our website).

[1313e]

@arfon Thanks for that, but I think you will have to manually set my last name in the footer again...
Similarly, I think that the BibTeX entry for the paper will also have my name incorrectly (as that is the case for the Hickle paper, published here in JOSS).
Is there anything on my part that I can do to avoid that?

[arfon]

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

[1313e]

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

That link does not resolve.

[arfon]

Weird works fine for me. Can you try going to https://www.doi2bib.org/ and putting in 10.21105/joss.01229.

If that doesn't work, the output from the service (i.e. the bibtex entry from the Crossref metadata) is:

@article{vanderVelden2019,
  doi = {10.21105/joss.01229},
  url = {https://doi.org/10.21105/joss.01229},
  year = {2019},
  month = jun,
  publisher = {The Open Journal},
  volume = {4},
  number = {38},
  pages = {1229},
  author = {Ellert van der Velden},
  title = {Model dispersion with {PRISM}$\mathsemicolon$ an alternative to {MCMC} for rapid analysis of models},
  journal = {Journal of Open Source Software}
}

☝️does that look OK?

[1313e]

Uhm, my last name should be in braces, as otherwise it will not be formatted correctly.
Besides that, it looks good.

It does seem though that for my paper, my name is at least written correctly.
The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

[arfon]

OK, thanks. I don't have any control over where https://www.doi2bib.org puts the braces but the metadata looks correct.

The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

OK, that's fixed for 10.21105/joss.01115 in https://github.com/openjournals/joss-papers/blob/master/joss.01115/10.21105.joss.01115.crossref.xml#L43-L47 but it will take a few hours to update on https://www.doi2bib.org .

[1313e]

Alright, thanks.
I know from experience that having no braces can sometimes mess it up when using natbib for example, but I guess that if the metadata is correct, such a thing would not be hard to notice.

[1313e]

@arfon This is not that big of a deal, but I just realized that the footer in the JOSS paper says 'van der Velden et al.', even though I am the sole author on the paper.

[arfon]

@arfon This is not that big of a deal, but I just realized that the footer in the JOSS paper says 'van der Velden et al.', even though I am the sole author on the paper.

Sorry about that. This should be fixed now. Note, the new pdf can take a few hours to show up sometimes because of browser caching.