Bids-specification: Table fences in PDF of specification misaligned

[VisLab]

PDF version 1.4.0 of read-the_docs --- a number of the tables are messed up
For example, on pages 13, 28, 30

[sappelhoff]

Thanks for the report --> you are probably referring to the PDFs of the spec hosted on zenodo: https://zenodo.org/record/3890439

Just FYI: These PDFs have nothing to do with readthedocs. We build them using pandoc and a python script by @Arshitha, see: https://github.com/bids-standard/bids-specification/tree/master/pdf_build_src

If somebody who reads this has time to look into this, it'd be much appreciated.

[sebastientourbier]

I took a little bit of time to look into this and it seems BIDS markdown tables are using the pipe_table syntax.
From Pandoc documentation page on extensions (https://pandoc.org/MANUAL.html#tables):

The cells of pipe tables cannot contain block elements like paragraphs and lists, and cannot span multiple lines. If a pipe table contains a row whose Markdown content is wider than the column width (see --columns), then the table will take up the full text width and the cell contents will wrap, with the relative cell widths determined by the number of dashes in the line separating the table header from the table body. (For example ---|- would make the first column 3/4 and the second column 1/4 of the full text width.) On the other hand, if no lines are wider than column width, then cell contents will not be wrapped, and the cells will be sized to their content

For instance here is the markdown code for table page 13:

| Field name | Definition |
| :---------- | :-------------------------------------------------------------------------------------------------------------- |
| LongName | Long (unabbreviated) name of the column. |
| Description | Description of the column. |
| Levels | For categorical variables: a dictionary of possible values (keys) and their descriptions (values). |
| Units | Measurement units. SI units in CMIXF formatting are RECOMMENDED (see Units). |
| TermURL | URL pointing to a formal definition of this type of data in an ontology available on the web. |

Based on the previous reference, we can see that the number of dashes for the first column is really much smaller than the number of dashes in the second and I guess a solution would be to review the proportion of dashes for the problematic tables.

[sappelhoff]

thanks for looking into this @sebastientourbier! Would you be up to trying a fix?

[sebastientourbier]

@sappelhoff Yes I would be up to.
Just to make sure I proceed correctly:
I am gonna pull the latest version of the master branch on my fork, create the branch fix_tables_in_generated_pdf, fix the tables and as soon all tables are fixed, I will open a PR to the master branch where I will mention this issue.
Is that correct?

[sappelhoff]

yes, sounds correct :+1: looking forwards to it

[sebastientourbier]

Just to keep track, here is the full list of tables with problematic layout:

• [x] page 13 (Tabular files)
• [x] page 18-19 (Derived dataset and pipeline description)
• [x] page 27-28 (Anatomy imaging data)
• [x] page 29 (Task (including resting state) imaging data)
• [x] page 30 (Required fields)
• [x] page 37 (Sidecar JSON (*_meg.json))
• [x] page 40 (Channels description (*_channels.tsv))
• [x] page 43 (Fiducials information:)
• [x] page 47 (Sidecar JSON (*_eeg.json))
• [x] page 50 (Channels description (*_channels.tsv))
• [x] page 56 (Sidecar JSON (*_ieeg.json))
• [x] page 59 (Channels description (*_channels.tsv))
• [x] page 76 (Common file level metadata fields)
• [x] page 77 (Spatial references)
• [x] page 79 (Preprocessed, coregistered and/or resampled volumes)
• [x] page 81 (Masks)
• [x] page 104 (Standard template identifiers) - ok but 1st column tight
• [x] page 106 (Nonstandard coordinate system identifiers)
• [x] page 106 (Non-template coordinate system identifiers)

[sappelhoff]

good idea, you can use a syntax like this: - [ ] to be checked to format interactive check mark boxes. That may come in handy to keep track which ones are fixed and which ones are not. They can be checked by clicking on them after the post is done. Or you can initialize them in checked mode using this syntax: - [x] checked

• [x] this one
• [x] this one too
• [ ] this one not

[sappelhoff]

@sebastientourbier I just realize: Please don't work on the pages 94-95 (Appendix IV: Entity table) because soon this table will be auto-generated, see: #475

this is also good to keep in mind when writing the script for auto-generation @yarikoptic @tsalo --> if we can find a rule on how to write markdown tables that get properly rendered as PDF using pandoc, we should encode that rule within the auto generation scripts.

[Arshitha]

@sebastientourbier Thank you for looking into it. I believe this happens after each new release because of the column width mismatch. For this to work, like @sappelhoff suggested having some rules about writing markdown tables might be the quickest fix. However, in order to automate it using just the current script, I had a couple of ideas that I'll try out over the weekend and keep y'all posted.

[sappelhoff]

@Arshitha thanks for chiming in!

Are you suggesting that the fixes @sebastientourbier is doing currently will be undone at the next release? How so?

[Arshitha]

@sappelhoff Nope, I wasn't commenting on @sebastientourbier fixes coz I assumed he'd be working on fixing the column width proportions by programmatically calculating and adding the required number of -'s wherever required. But if he plans to go into the .md files and fix the pipe tables manually by extending the column widths then it'll break during the next release. Because I'd fixed the tables manually by adding more -'s in the .md files in one of the previous release

[Arshitha]

I was just saying that this might be a quick fix:

this is also good to keep in mind when writing the script for auto-generation @yarikoptic @tsalo --> if we can find a rule on how to write markdown tables that get properly rendered as PDF using pandoc, we should encode that rule within the auto generation scripts.

as you mentioned above

[Arshitha]

One of the ways I'm thinking of fixing this issue is by finding tables in .md files and calculating the maximum character count in each column (for each table) and then replacing existing table fences with reformatted (basically more -'s) fences.

This might be a more long term solution because:

1. Authors editing or adding new .md files don't have to worry too much about formatting tables
2. Doesn't in any way affect rendering in readthedocs version

I haven't had a chance to try it out but theoretically, I believe, this should work.

[Arshitha]

Okay, I just saw the PR #553 and it seems like he fixed it manually.

[sappelhoff]

But if he plans to go into the .md files and fix the pipe tables manually by extending the column widths then it'll break during the next release. Because I'd fixed the tables manually by adding more -'s in the .md files in one of the previous release

I am just not sure how the breakage would happen during a release :thinking:

One of the ways I'm thinking of fixing this issue is by finding tables in .md files and calculating the maximum character count in each column (for each table) and then replacing existing table fences with reformatted (basically more -'s) fences.

that sounds nice indeed ... and the plus side would also be that we don't have to have the custom more wide table syntax in our source ... because it'll get generated / edited during the build only

and for this automation script, you wouldn't have to think about aligning the table fence pipes | either, because that's only stylistic and should not affect the rendering

[sebastientourbier]

When checked on readthedocs, it seems it did not affect the rendering.
But by going into this fences alignment problem, it also came to my mind that an automated way of fixing tables would be more ideal. So we should indeed correct the tables at two levels:

• To make pandoc happy (PDF generation), we need to make sure that:
  
  
  
  1. proportion of -'s in first column and
  
  
  2. the corresponding number of -'s should be bigger that the number of characters max in the first column
  
  
• To make sphinx/readthedocs and so TravisCI happy (online doc), we need to make sure that after fixing the number of -'s each row has fences aligned to the fences in the row with -'s