Msprime: Revamp of tutorial in the docs, especially the 'demography' section?

[gtsambos]

At the moment, the Demography section isn't particularly helpful - there's one big example (with lots of relatively complicated stuff going on), rather than a step-by-step explanation of what different types of demographic events are and what they do. I think it would be more helpful to flesh this out by porting over material from, say, here.

I'm happy to take this on but it's a not a small change, so not right now.

[petrelharp]

Awesome. It'd probably be good to post an outline of what you're thinking here before you jump into it.

[jeromekelleher]

Yes, this would be superb, thanks @gtsambos!

[molpopgen]

It may be handy to roll #766 into this?

[gtsambos]

It may be handy to roll #766 into this?

Yup, absolutely, feel free to ping things like this in here.

Awesome. It'd probably be good to post an outline of what you're thinking here before you jump into it.

Yup, will do. I won't have time to think about this properly for a few weeks but will defs let you know before I jump into any significant changes!

As a related (but possibly separate) idea, I was thinking the other day that it could be useful to set up a site/gallery (or a living book?) that holds some longer code snippets showing analyses done with tskit and msprime, sort of like the 'recipes' in the SLiM manual.

[jeromekelleher]

As a related (but possibly separate) idea, I was thinking the other day that it could be useful to set up a site/gallery (or a living book?) that holds some longer code snippets showing analyses done with tskit and msprime, sort of like the 'recipes' in the SLiM manual.

This is a bit more ambitious, and related to the tutorials we started on a while back (rendered website here). We found that converting the jupyter notebooks to HTML and the review process (in particular) for the notebooks was really tedious. Perhaps the living book thing works better?

Revamping the current tutorial a bit would be a real help though, and I don't think we need to change the toolchain for doing that.

[gtsambos]

We found that converting the jupyter notebooks to HTML and the review process (in particular) for the notebooks was really tedious. Perhaps the living book thing works better?

JupyterBook looks like it's more specifically designed for organising Jupyter notebooks and rendering them in html, so I imagine it would be better at this yes. We would defs have to check this out before embarking on anything.

But yes, I agree that improving the existing tutorial material is more important.

[jeromekelleher]

JupyterBook looks like it's more specifically designed for organising Jupyter notebooks and rendering them in html, so I imagine it would be better at this yes. We would defs have to check this out before embarking on anything.

Sounds good --- @molpopgen, might be worth looking into at some point.

[molpopgen]

@gtsambos @jeromekelleher I'd be happy to switch the back-end of the tutorials. I couldn't actually do much/anything before early November, due to my teaching schedule.

The JupyterBook looks good, in the sense that we can replace all notebooks with markdown? I'll have to play around and evaluate a bit.

Another option for this sort of thing is Rmd (R markdown). It is easier for multi-language docs (think showing examples based on the tskit C api!!!), and I've had a lot of luck with it for more complex docs. For example, here.

[gtsambos]

Another option for this sort of thing is Rmd (R markdown). It is easier for multi-language docs (think showing examples based on the tskit C api!!!), and I've had a lot of luck with it for more complex docs. For example, here.

yep, they look v similar -- I think JupyterBook is the Jupyter notebook equivalent of bookdown. I'm not sure we could render SVG images in Rmd files though, which I think would be a huge advantage for this particular application.

[molpopgen]

For the SVG images in bookdown, I use a Makefile that converts svg to 300dpi png via ImageMagick. It isn't much of a compromise, as the whole thing is built via make anyways.

[agladstein]

I wouldn't mind helping with this (especially reviewing).

[molpopgen]

Thanks @agladstein.

i think the short-term marching orders are for someone (probably me?) to evaluate JupyterBook and then do a liftover of the current tskit tutorials from notebook to markdown formats. (That should just be a notebook convert op....?)

[molpopgen]

After some monkeying around with the code for the "demo" book:

• Code blocks in markdown files do not appear to be executed
• Executed notebooks get included, and their output is shown.

So, this is cool on one hand. But it seems that the problems we've had so far still exist: changes to the content are hard to review, and the version tracking via github is very difficult. (The notebooks are bit binary blobs, and so they don't diff well.)

in other words, it doesn't let us get rid of the .ipynb files that currently exist in the tskit tutorials. What it does allow is for a more automated way to build the web version that is superior to the Makefile we currently have.

The major plus of the Rmd system is that the knitr step executes code blocks in the markdown. This means we can review code, and we "know" (via conda envs) that the outputs are current because we could have something like Circle Ci execute them.

[agladstein]

You can use ReviewNB for viewing diffs.
https://www.reviewnb.com/

But, I'm pro knitr anyways.

[molpopgen]

IIRC, we looked into ReviewNB for the tutorials. It had just come out around then. It solves one problem, but not the other, which is repo bloat.

[gtsambos]

hullo again everyone 👋 just trying to make a more concrete plan for this docs revamp now.

It'd probably be good to post an outline of what you're thinking here before you jump into it.

As a first pass, I was thinking of adding the following structure to the current Demography section:

Demography

• Population structure
  
  
  
  • see here
  
  
• Migrations
  
  
  
  • see here
  
  
• Population size and growth rates
  
  
  
  • see here
  
  
• Census events
  
  
  
  • Already in the master doc here.
  
  
• Demography Debugger
  
  
  
  • Already in the docs
  
  
• An example: OoA
  
  
  
  • Already in the docs
  
  

The section on census events that I wrote a bit earlier shows you the level of detail that I am hoping will be in each of these other sections.

Thoughts, feelings, comments?

@jeromekelleher, could you please assign this issue to the Docs project?

[jeromekelleher]

This looks like an excellent plan @gtsambos, thanks!

[agladstein]

That sounds good to me.

[agladstein]

So, have we decided what format to have the tutorials in?

[molpopgen]

I'd vote for Rmd input to md via a Makefile executing knitr. Some hints for the yml are here. I don't like the notebooks for GitHub, and the tools to look at their diffs still don't solve the repo bloat problems. I can show how to get svg images in--you can have the Makefile convert to png using ImageMagick.

[jeromekelleher]

I don't think we need to do a technology change here for the local tutorial --- hadn't we agreed to keep it as it is, and look into the other stuff for the more specialised tutorials that get hosted externally?

[molpopgen]

Oops, my bad. Yes, you're right and I got my wires crossed

On Fri, Oct 18, 2019, 8:53 AM Jerome Kelleher [e-mail]
wrote:

I don't think we need to do a technology change here for the local
tutorial --- hadn't we agreed to keep it as it is, and look into the other
stuff for the more specialised tutorials that get hosted externally?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tskit-dev/msprime/issues/808?email_source=notifications&email_token=ABQ6OH4T36CR5AHGXYILVUTQPHLUTA5CNFSM4IUVV4GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBU5PAI#issuecomment-543807361,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABQ6OHYYU4QJBYBBVFHA7U3QPHLUTANCNFSM4IUVV4GA
.

[agladstein]

So, basic tutorials will just be in restructured text on read the docs?

[molpopgen]

So, basic tutorials will just be in restructured text on read the docs?

Yep. In practice, anything else is really hard on RTD.

[jeromekelleher]

Closed in #945