Is your feature request related to a problem? Please describe.
Contributing to the documentation is difficult, the documentation of the documentation is not good. We're behind on the docs release for 3.0. DocBook is difficult and we're not even using DocBook, but our own custom DTD. We have dependencies for this tooling. The docs are hosted on gitlab currently which is confusing as well.
Describe the solution you'd like
I will convert all the docs to markdown, so they can be built with the rest of DT.org website.
Alternatives
Keep doing what we're doing?
Additional context
Spoke with @aurelienpierre about this in IRC
I'd like to gain consensus from the core team members before I do the work, as this will be a large undertaking.
That will probably be a huge work but I like the idea !
I'm willing to do the work, no problem.
Great !
Will this apply to lua doc too ? iirc, lua doc is generated automatically, but I've no idea how it's done !
Anyway, thanks and good luck for all the work !
@upegelow, since you have maintained the usermanual for the longest time, are you ok with this ?
The goal would be to simply have one markdown file per manual page, laid out in sub-directories, then we thought about using Pandoc to compile the collection of .md as a HTML/PDF/ePub file (example), then update the website by simply sending the HTML on the server storage.
The benefit is markdown files can be edited directly from the Github UI in an "almost WYSIWYG" fashion. Then, it's very little admin overhead to update all targets.
I'm all for it as the current build of the documentation is far from trivial and this certainly delay the release of the documentation.
@upegelow : what's your feeling about this? You are probably the only one to be able to build it.
I don't know markdown but I have never been a big fan of docbook (to say the least). It always sucked in generating the HTML version of the manual. Right now even I cannot build it as it requires saxon which is no longer properly detected. Probably this can be fixed but I had no time yet.
The only big strength of docbook for me has been its simple design. You can easily check the structural correctness with xmllint. But maybe markdown has the same advantage then it's fine.
Concerning file structure of the manual. That's just historical. No need to stick to it.
Another point to keep in mind is that we need to offer the possibility to translate the documentation. So if rewriting is done we must be sure that this feature is kept as it is very important.
Another point to keep in mind is that we need to offer the possibility to translate the documentation. So if rewriting is done we must be sure that this feature is kept as it is very important.
With markdown, that will easy but it will need for all translators to adjust all pages by copy/paste to docbook from markdown, then adapt pagination and format all pages on markdown. So a big work on all languages.
There is this tool : https://github.com/darkcircle/po4gitbook
I will certainly want to nail down the translation part of things before I make any changes.
Sigh, currently not even the PDF user manual can be generated out of the docbook sources. This would require fop which is broken here on OpenSuse 15.1. Fails with a java error even on a simple test input.
The bug is obviously known (see here) but that's a report from April 2019.
We probably should not rely on docbook for the future.
A straight pandoc conversion using pandoc -f docbook -t markdown doc.xml won't be good enough. Lots of stuff is lost there. I'll keep digging.
For the records: I am back in the business of generating the manual. For the PDF I need a self compiled fop and for the HTML manual a third party version of the docbook4 stylesheets (the default ones no longer ship with the saxon extensions).
For the records: I am back in the business of generating the manual. For the PDF I need a self compiled fop and for the HTML manual a third party version of the docbook4 stylesheets (the default ones no longer ship with the saxon extensions).
On Debian 10 the PDF builds fine. Why not just install a VM and compile the book there? Or I can compile them as long as there is nothing to change in the code.
After reading in the Pro Git book (https://git-scm.com/book/en/v2) today I wondered in which format it is written. It turned out it is the good old asciidoc, or, rather, its new implementation asciidoctor (https://asciidoctor.org/). Reading a bit into it, I found that it has some benefits over markdown, and its features are designed with docbook in mind. Therefore I wonder if it would better suit this project as well. At least github supports the asciidoc syntax as well, and features such as image captions and tables may be handy for the darktable manual. A configurable, native html and pdf converter may ease CI as well.
That is for this. I'm considering all options. I not sure that markdown is expressive enough for the darktable docs, after looking at some of the docbook sources. And the real conceit of markdown is that you just write what Markdown's syntax doesn't cover in HTML.
I sort of wonder if markdown + HTML5 tags is even enough.
This isn't going to be a simple conversion, so I'm thinking of redoing all of it.
On January 27, 2020 3:01:17 PM PST, spaceChRiS notifications@github.com wrote:
After reading in the Pro Git book (https://git-scm.com/book/en/v2)
today I wondered in which format it is written. It turned out it is the
good old asciidoc, or, rather, its new implementation asciidoctor
(https://asciidoctor.org/). Reading a bit into it, I found that it has
some benefits over markdown, and its features are designed with docbook
in mind. Therefore I wonder if it would better suit this project as
well. At least github supports the asciidoc syntax as well, and
features such as image captions and tables may be handy for the
darktable manual. A configurable, native html and pdf converter may
ease CI as well.--
You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-578996936
markdown + HTML5 tags may fail on pdf anyway, and that's where things become difficult. For the syntaxes coverage, https://asciidoctor.org/docs/user-manual/#comparison-by-example has a simple comparison. It's not that I want to convince anybody into asciidoc, it's just 20 years of TeX experience and almost the same with more lightweight languages that make me think that markdown doesn't suit best for an entire user manual.
Besides the syntax decision, I would really love to join this effort of renovating the manual, however, with 2 little kids, a full time job and a construction site instead of a house, this will not work out any time soon. So please tell me if I should stop with my wise guy attitude here ;).
I personally think the agility to be gained from using github-flavored markdown is a very attractive idea, despite its limitations... but I wonder about a few things:
How important is the ability to generate a PDF version? Does anyone still want to use software documentation in this format anymore? (I personally don't.)
How important is fancy semantic markup? Could the manual be reasonably represented without it? For example, would it be sufficient for things like keyboard shorcuts to simply be bold text instead?
Does the manual need to include math expressions, etc?
It might be good to list the things which would be lost, evaluate their importance, and move forward from there... just a thought.
I personally think the agility to be gained from using github-flavored markdown is a very attractive idea, despite its limitations... but I wonder about a few things:
How important is the ability to generate a PDF version? Does anyone still want to use software documentation in this format anymore? (I personally don't.)
I actually prefer the pdf version.
How important is fancy semantic markup? Could the manual be reasonably represented without it? For example, would it be sufficient for things like keyboard shorcuts to simply be bold text instead?
yes or even enclosed as
I think PDF is still quite important and one of my goals is to continue to produce PDFs.
Right now I'm leaning towards a format called mDITA, which is github flavored markdown plus a subset of html5 tags. This would allow me to use the DITA Open Toolkit, a publishing pipeline I am very familiar with, but we could also use other, standard markdown pipelines.
The DITA Open Toolkit will allow us to still generate structured documents for translation. I'm hoping we can still use the PO workflow, but I haven't fleshed out that idea yet.
I'd love to hear from any translators... Can the translation workflow be better? Should we stick with makefiles and PO?
On January 27, 2020 11:39:06 PM PST, junkyardsparkle notifications@github.com wrote:
I personally think the agility to be gained from using github-flavored
markdown is a very attractive idea, despite its limitations... but I
wonder about a few things:How important is the ability to generate a PDF version? Does anyone
still want to use software documentation in this format anymore? (I
personally don't.)How important is fancy semantic markup? Could the manual be reasonably
represented without it? For example, would it be sufficient for things
like keyboard shorcuts to simply be bold text instead?Does the manual need to include math expressions, etc?
It might be good to list the things which would be lost, evaluate their
importance, and move forward from there... just a thought.--
You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-579118607
I'd love to hear from any translators... Can the translation workflow be better? Should we stick with makefiles and PO?
Much translation discussion seems to take place on the darktable-dev list, you might want to ask there.
How important is the ability to generate a PDF version? Does anyone still want to use software documentation in this format anymore? (I personally don't.)
For me it is important as well. Some documentation even deserves to end on dead trees. To learn the concepts behind something, I find it often easier not to read about them on a display.
But for printing it should at least fulfil the very basic concepts of beautiful typesetting.
I'd love to hear from any translators... Can the translation workflow be better? Should we stick with makefiles and PO?
ping @mepi0011 and @lebmich for this question as they worked on the German and French translation of the manual and maybe not seeing this issue discussion.
Proof reading parts of the German translation was not easy using PO files.
By the way, will the manual get a separate repository like darktable/dt-usermanual?
I have got used to the processes of Darktable together with Docbook, so for now I see no reason to switch to markdown (IMHO).
From a translator's point of view it is important to me that I want to limit myself to translating and proofreading. I don't want to care about the layout of the document. Therefore, I find the procedure and the use of po files very advantageous.
A change to markdown or similar should be well planned and thought through! From my point of view, the new workflow (incl. translation) has to be defined or written down so that everyone knows it before the change and nothing will be overseen. Which restrictions are there compared to docbook?
Unfortunately, I have no experience of how this works for larger projects. The documentation team of KDE and Gnome also works with docbook/
https://l10n.kde.org/docs/gettingstarted.php
Are there examples of projects that use Markdown as documentation and how is the translation workflow there?
IMHO the documentation is too much neglected and that there is currently a lack of a leader who is exclusively responsible for documentation and where everything comes together. e.g see the open change request to improve the documentation, they are open since weeks.
I have got used to the processes of Darktable together with Docbook, so for now I see no reason to switch to markdown (IMHO).
Devs procrastinating doc writing because it's a PITA and users being kept aside from contributing because it's an even greater PITA (+ Git skills needed) seems a pretty good reason to me.
IMHO the documentation is too much neglected and that there is currently a lack of a leader who is exclusively responsible for documentation and where everything comes together. e.g see the open change request to improve the documentation, they are open since weeks.
Ask yourself why…
Seems like we can do markdown -> po with this https://bitbucket.org/tagoh/gettext-markdown/src/master/
Or I can generate .xliff files for translation.
@TurboGit how are the rest of the strings translated? I was looking at setting up a web app that'll handle translation.
Everything is translated locally with Poedit, then .po/.mo files are merged from PR.
I agree with @chrik5 that translating with Poedit sometimes lack some context to handle properly strings that may have different meanings depending on context.
Just some thought as a doc-procrastinating dev :
To solve 1, markdown should allow to update different part separately, maybe this will need some tag ti indicate to which dt version we refer...
To solve 2, no idea...
After speaking with @houz in IRC, he thinks that it isn't that docbook is too hard or markdown is easier, but rather that just not enough people care. It doesn't seem like there is sufficient momentum for this idea.
@paperdigits : that's certainly wrong as most people don't even have a way to build the documentation. The build process is obscure, needs lot of dependencies... At the moment this is a single person able to build it properly. So to me the tools is really lacking simple support that will help building and so publishing the documentation.
it isn't that docbook is too hard or markdown is easier, but rather that just not enough people care.
I think there probably exists a spectrum of potential contributors to the docs, and different things are obstacles for different people. Personally, I'm a terrible multitasker and am more likely to contribute when there's a greater chance of accomplishing things in small, discrete chunks of time in between other demands on attention (I hope to have more time a week from now). This is possible with the current situation, I've made a few small updates, but it's just uncomfortable enough to make it much less likely to happen... for whatever that's worth.
That said, the translating work is obviously a very important thing to consider, and it's probably not worth making one aspect easier if it makes it even a little harder for all the translators (something I have no insight about).
As mentioned already, the impressive amount of development happening lately also complicates things slightly. Should the docs in master branch target the next release? I've never really used the 3.0.x releases, and sometimes have trouble remembering which current features were included there. In that case should the priority be to do the essential 2.6 > 3.0 updates first in a release branch, then try to "release" updated docs along with each software release (might not be too hard if the important changes are already collected in release notes)? All of this points to why increased documentation agility (whatever that might turn out to mean) is attractive.
Ideally, docs on master should either target master or the next release. That is a separate issue though.
I've been evaluating an HTML5 type of document. It is much more expressive than markdown, but not as obtuse as docbook. It's still structured, which means it should be relatively easy to translate. And HTML isn't terribly high bar. You can also use standard HTML editor tools.
The HTML5 type is called HDITA: http://docs.oasis-open.org/dita/LwDITA/v1.0/cnprd01/LwDITA-v1.0-cnprd01.html#hdita-example
Seems like it'd be a good compromise.
If you want to motivate developers to document new features, self interest should do it - don't merge new features into master without documentation.
On 5 February 2020 22:22:21 GMT+00:00, Mica notifications@github.com wrote:
Ideally, docs on master should either target master or the next
release. That is a separate issue though.--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-582642032
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
After speaking with @houz in IRC, he thinks that it isn't that docbook is too hard or markdown is easier, but rather that just not enough people care. It doesn't seem like there is sufficient momentum for this idea.
I fully agree with @houz 's statement, and this is also what I meant by my statement that the documentation is given too little attention. I don't think switching to markdown will improve the user manual situation.
IMHO the documentation is too much neglected and that there is currently a lack of a leader who is exclusively responsible for documentation and where everything comes together. e.g see the open change request to improve the documentation, they are open since weeks.
The build process is obscure, needs lot of dependencies... At the moment this is a single person able to build it properly.
This is bad that only one person knows how to build the user manual properly!
_The build process is not easy, two years ago I tried to update the user manual to docbook5 and failed with the build process._
Further above in the discussion it is mentioned that the currently used docbook is something self created. Maybe it simplifies the build process when using standard docbook.
The following should be considered when switching from docbook to markdown:
this is also what I meant by my statement that the documentation is given too little attention
To me you are just mixing cause and consequence !
Documentation is not given the needed attention because almost nobody can build it.
Let me rephrase the goal here:
Let's don' care about the technology used, just make the documentation build as easy as the software build process.
Do we agree with that ?
If someone come here and provide a ./build-doc.sh at the to of darktable repository that will build the documentation in all languages and provide the HTML ready for the Web site then we are done.
The documentation builds ok on Manjaro, if you install the right
dependencies. The pdfs were fairly straightforward, but the HTML was a bit
more of a fiddle to get the dependencies satisfied. I used the build
instructions for Archlinux on the redmine wiki, searched for missing
dependencies in the AUR repositories, installed jdk10-openjdk since jdk8 is
missing some newer object/class required by the code.
The html docset doesn't look so nice without the theme from the
darktable.org website, but if you drop the contents of the dtorg build
directory into the website's usermanual directory, it should end up looking
ok.
@matt-maguire : can you document the needed steps and provide a script to build the HTML docs?
I can make a script that contains the required build commands on linux. However, the setup of the dependencies is distribution-dependent, and I'm not sure in the end exactly which of the packages I installed are strictly needed. To produce a proper MOP, I'd need to set up a VM with a fresh manjaro install and test what is really needed.
Please merge PR #4054 into master
The docs being markdown will have no effect on the format of the screen captures.
On February 11, 2020 9:11:05 AM PST, mepi0011 notifications@github.com wrote:
Please merge [PR
4054](https://github.com/darktable-org/darktable/pull/4054) into
master
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-584743639
This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.
I am working on this... it is just quite tedious.
Don't mind the bot, it does it job.
The migration path is this:
docbook xml -> dita xml -> markdown
I've completed the docbook -> dita xml conversion.
What's left:
@TurboGit does this sound good?
@aurelienpierre @TurboGit you can see some preliminary markdown here: https://cloud.silentumbrella.com/index.php/s/j3GosN4xnZHctmM
Please let me know if you think the darkroom and lightroom files are too long. Having the files sectioned this way maintains compatibility with pandoc.
Awesome ! A would say file length is ok for now. Not sure long-term. 3000-lines docs is something, but IÂ guess it won't be to hard to split that later now that everything is converted. For now I'm more concerned about updating manual ASAP.
Thanks a lot for this awesome and tedious job !
@paperdigits : thanks for working on that. A quite big work.
I have looked at the generated HTML and it seems clear. A stupid question, what is the best way to see the final markdown rendering locally ?
Also I'm still unclear about the way translations will be updated ? Do you plan to convert the current PO in separate set of HTML ? How would the new chapters / paragraphs will be presented to translators ? Sorry, maybe basic questions to you, but all this is new to me.
@paperdigits: thanks for your great and big work on that. If you have time to answer to @TurboGit, I would like also to know what is asking. Depending on the time I will have when this work will be finished and answers to @TurboGit questions, I will probably work a little on some improvements on manual content.
Sorry I whiffed on the response to @TurboGit. I hope to wrap up the markdown this evening and submit a PR for the English content.
I'm not 100% sure on translations, it depends on where we want to push things and what tooling we want to use. Clearly we would need to transform the markdown into some sort of structured content if we want to continue to use the xml2po & similar tools. I was also thinking we could spin up a weblate instance because it looks like it natively handles markdown. TBH, markdown isn't great to translate.
The other thing we could do is use pandoc to generate other markup to translate.
On May 4, 2020 10:55:54 AM PDT, Nicolas Auffray notifications@github.com wrote:
@paperdigits: thanks for your great and big work on that. If you have
time to answer to @TurboGit, I would like also to know what is asking.
Depending on the time I will have when this work will be finished and
answers to @TurboGit questions, I will probably work a little on some
improvements on manual content.--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-623612392
Markdown is for chat. It is not for pagenated/printable/translatable user documantation.
Yes, DocBook is dificult for setup and the sucsses depends of ditro to disto, but gives you this nice PDF book you have.
DocBook gives structure of the documentation. With DockBook you can have:
With Markdown all you get is README style page, with all images centred in the middle.
@paperdigits : So I fear we went to fast on this one. As I said very early on this was that we need a clear and very simple way to propose translation. Today it is quite easy:
Certainly not perfect, that was the goal of the PR, but not very complex. The only motivation for the change is to get a new workflow for translator that would be easier. I can't help as I'm not using Markdown or whatever....
So before continuing this PR I'm proposing to sit down and try to devise a new workflow based to simplify the whole process.
With Markdown all you get is README style page, with all images centred in the middle.
Markdown is only simplified HTML, nothing prevents you from using actual (valid) HTML5 in Markdown files, and CSS stylesheets.
The current state of the fancy XML doc we have is an absurd volume of formatting markup for little content. Lists are so crowded with code that they become unlegible, and the whole markup is a custom undocumented XML that matches no standard.
As a consequence, nobody want to write in that if they don't have a gun on the forehead to motivate them. Plus, building the doc uses some clever soft that is broken once every 3rd release.
As for translations, I found this (https://okapiframework.org/wiki/index.php?title=Okapi_Filters_Plugin_for_OmegaT). Not sure how suitable it is in our project.
If you give the power of
using actual (valid) HTML5 in Markdown files, and CSS stylesheets.
on documentation writers, then you will face other problems. Countless idaes of how the documentataion should look like.
Then you will writes rules about this. But everybody knows, rules are pointles if there is no method for check them.
After that, you will make tools about it. Styp by step, the toolchain will become same piece of
clever soft that is broken once every 3rd release.
On top of that, you should consider, there is no such thing as Standard Markdown. There are tons of implementations and flavors. Special attention is needed on tools selection and vision in the future.
Testing and documentation prcess requires guns all the time. The place you go, with this Markdown migration step is the place I returning from. This migration process is huge step backward.
All you need is properly tunned docker image for the DocBook rendering process, witch can be used from CI/CD and localy as well. Editors can use every XML editor witch is capable of DTD/XmlSchema suggestion. WYSIWYG editors for DocBook exists too.
Pleace, do not kill the PDF book. I'm so proud of the fact Darktable have such a nice comunity driven book about it. This distinguishes the project from the teens around.
Thank you for sharing your concerns, I'm aware of all the arguments for and against markdown.
The fact of the matter is that nobody was updating the docbook documentation. The architecture of the document was strange, the custom DTD was completely unnecessary, and the file encoding wasn't even good, so there were entities for accented characters.
Markdown can be made into PDF, so that isn't going anywhere.
As of now, the documentation is in DITA XML, which has many of the benefits of docbook, but is more minimal in its tagging and better structured. There is a transform from DITA to markdown.
@turbogit I understand the hesitation, but the docs already are not being updated heavily, but the more commits between now and when my PR is made, the more work it is. Plus, up to date translations for an out-of-date document doesn't really do anybody much good.
On May 5, 2020 3:09:48 AM PDT, Georgi Georgiev notifications@github.com wrote:
If you give the power of
using actual (valid) HTML5 in Markdown files, and CSS stylesheets.
on documentation writers, then you will face other problems.
Countless idaes of how the documentataion should look like.Then you will writes rules about this. But everybody knows,
rules are pointles if there is no method for check them.After that, you will make tools about it. Styp by step, the toolchain
will become same piece ofclever soft that is broken once every 3rd release.
On top of that, you should consider, there is no such thing as
Standard Markdown. There are tons of implementations and flavors.
Special attention is needed on tools selection and vision in the
future.Testing and documentation prcess requires guns all the time. The place
you go, with this Markdown migration step is the place I returning
from. This migration process is huge step backward.All you need is properly tunned docker image for the DocBook rendering
process, witch can be used from CI/CD and localy as well. Editors can
use every XML editor witch is capable of DTD/XmlSchema suggestion.
WYSIWYG editors for DocBook exists too.Pleace, do not kill the PDF book. I'm so proud of the fact Darktable
have such a nice comunity driven book about it. This distinguishes the
project from the teens around.--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-623968348
First I want to say that I really don't care about the format on which the document is written. Also it is wrong to say that the current documentation is not updated. It is, slowly, probably too slowly but it is updated.
That being said, I don't think we understand each other @paperdigits.
Plus, up to date translations for an out-of-date document doesn't really do anybody much good.
But an up-to-date documentation that cannot be translated is certainly even worst.
What I want is to be sure that we switch for good reasons and that we are in better situation after than before in all regards:
And all this we easy tools especially for translator who are generally not well versed in software/IT mess :)
So before continuing this PR I'm proposing to sit down and try to devise a new workflow based to simplify the whole process.
This is exactly what I have been trying to initiate at the beginning of the discussion and implementation! (see here)
@turbogit: I think this will work: https://po4a.org/ it handles po files for things like markdown.
On May 5, 2020 8:47:55 AM PDT, Pascal Obry notifications@github.com wrote:
First I want to say that I really don't care about the format on which
the document is written. Also it is wrong to say that the current
documentation is not updated. It is, slowly, probably too slowly but it
is updated.That being said, I don't think we understand each other @paperdigits.
Plus, up to date translations for an out-of-date document doesn't
really do anybody much good.But an up-to-date documentation that cannot be translated is certainly
even worst.What I want is to be sure that we switch for good reasons and that we
are in better situation after than before in all regards:
- writing the doc
- translating the doc
- generate PDF and HTML
And all this we easy tools especially for translator who are generally
not well versed in software/IT mess :)--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/darktable-org/darktable/issues/4177#issuecomment-624135737
This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.
Well my first attempt was a failure.
Next idea is just to start over, in a separate place, pull the bits of the manual that make sense, and rewrite the rest in markdown. The new docs should separate different content types & be extremely minimal to ease maintenance and translation issues.
I also note that translation and lack of commitment from maintainers is what failed my first attempt, but looking at the PO files for the user manual, I couldn't identify any language that was translated fully. It looked like they were all half finished.
I'd also propose that we only keep docs for the current version. As content would be in git, we can tag for historical versions, but why keep those on the website?
I would approach it like this: write the reference part of the documentation first. Start with darkroom modules, then move to light room modules. Document the use of each slider. Then move on and document each tab, such as maps, print, etc. Then have a set of tutorial-like sections that cover workflow and different scenarios the user encounters.
Pretty much what I was going to suggest (i.e. start with modules and build up from there).
There is now https://github.com/elstoc/dtdocs which is a complete re-write of the docs in markdown.
Most helpful comment
There is now https://github.com/elstoc/dtdocs which is a complete re-write of the docs in markdown.