Tldr: New pages structure

[rprieto]

_(copied form the conversation at #147)_

The current folder structure looks like:

  |__ common
  |   |__ tar.md         # gnu command everyone should have
  |   |__ ssh.md         # gnu command everyone should have
  |   |__ npm.md         # just a tool that could be installed on any OS
  |__ linux
  |   |__ emerge.md      # clearly a linux-only tool
  |__ osx
      |__ ssh.md         #  OSX has a different version with different flags

This has worked so far, but

• the split can feel arbitrary
• you need to jump around to see if a command is already covered
• clients potentially need to make a lot of requests to get a given page

There seems to be a consensus among clients to move to a flatter folder structure, that would look like this:

   |__ tar.md
   |__ ssh.md
   |__ ssh.osx.md
   |__ emerge.md
   |__ npm.md

By default, we can display <command>.md, but <command>.<os>.md should have precedence if available.

This means the clients would let OSX users query Linux commands, but after all why not, they might just be curious about it. Or they might be using tldr on a Mac while SSHing on a Linux box that doesn't have it.

To clear up platform constraints, the description for emerge for example could say "for Linux" or "Gentoo specific command".

This would be a breaking change though, so we need a plan of attack. One option is to address all open PRs to get to a stable state, then copy all pages to the new structure. The old structure can live on for a while until all clients update. The PR guidelines would say to push changes to the new structure only.

[felixonmars]

+1 for keeping the old structure working while switching to the new one, as users may not upgrade their clients in time (or if they cares).

[waldyrious]

+1 with @felixonmars

[felixonmars]

Hi, any schedule on this? It has been several more months after the discussion :smiley_cat:

[rprieto]

Good point, it's still something I'm keen to look into, but requires more thinking now that there's many clients.

Should have start a conversation on gitter?

[felixonmars]

I've joined the conversation, though didn't see anything - does it keep logs?

[M3kH]

Can this done by the metadata syntax on MarkDown? maybe as additional to avoid duplicated files on categories/tags?

support:
 - osx
 - linux
 -- debian
...
---

[igorshubovych]

@M3kH in the end of this optimization process we will build man :)

[M3kH]

I would later prefer store informations in the markdown as yml instead of using the file names.
But this is my personal opinion :-)

[waldyrious]

Quoting a comment by @agnivade in #1436, which replaced all references to OS X with the new name, macOS:

Just a note that our platform folder is still named osx. And changing that might have consequences for various clients, so keeping it as is for now.

[ibnesayeed]

While we are here talking about the new page structure, we should also consider the possibility of adding custom tldrs that are not maintained by the shared repository, but a place where users can add their own overrides or additions for quick references. A note around this approach can be seen in https://github.com/tldr-pages/tldr/issues/1726#issuecomment-348537478.

[chamini2]

I like @ibnesayeed's idea. Kind of like brew taps?

[zlatanvasovic]

Any updates? Can you provide any advantages this system would give over the current one? As I see it, everything you want to have is already included in the current system. There is already a related issue in tldr-node-client: tldr-pages/tldr-node-client/issues/247

[sbrl]

Now that we have languages to contend with, this seems like even less of a good idea.

[zlatanvasovic]

Do others agree to close this?

ping @tldr-pages

[agnivade]

I agree that languages make this less appealing.

[waldyrious]

I actually think languages make this flattening more relevant. The current structure is quite noisy on the repo's root (we might end up with dozens of top-level folders as more languages are introduced by the community), and the folder-based platform division makes it hard to navigate the entire set of pages available for a given language, not to mention it can also expand over time (e.g. we're discussing the addition of a new platform folder for BSD in #816).

Here's what we have now:

|__ pages
|   |__ common
|   |   |__ foo.md
|   |   |__ bar.md
|   |__ linux
|   |   |__ baz.md
|   |__ osx
|       |__ qux.md
|__ pages.fr
|   |__ common
|   |   |__ foo.md
|   |   |__ bar.md
|   |__ linux
|   |   |__ baz.md
|   |__ osx
⋮       |__ qux.md

IMO the flat structure would be much simpler and more scalable:

|__ pages
    |__ en
    |   |__ foo.md
    |   |__ bar.md
    |   |__ baz.linux.md
    |   |__ qux.macos.md
    |__ fr
    |   |__ foo.md
    |   |__ bar.md
    |   |__ baz.linux.md
    |   |__ qux.macos.md
    ⋮

Reasons I like this:

• The hierarchy is simpler
• The top-level directory contains a single pages/ directory, rather than one for
• The English pages directory no longer has such a prominence among the other languages
• We can at a glance see whether a page exists for common or a specific platform, and they are all next to each other (we've been having issues with pages being added for specific platforms where they already exist in common)
• Adding both more languages and more platforms would add no extra complexity to the structure
• A flat listing (e.g. the output of find in the pages directory) is still able to outline the main structure:
  pages/en/foo.md pages/en/bar.md pages/en/baz.linux.md pages/en/qux.macos.md pages/fr/foo.md pages/fr/bar.md pages/fr/baz.linux.md pages/fr/qux.macos.md

So my inclination would be to consider this change even more now that we support multiple languages. We could even keep generating the zip in the current structure, and take the opportunity to generate the archive with the new structure using the .tar.gz format, as was suggested in #343.

[zlatanvasovic]

@waldyrious That structure is a bit different from the one proposed in this thread, but yes, it'd be much better than the current one.

[agnivade]

Oh you are still using the language as a directory. Yes that's okay. I thought that all the pages would be in a single directory.

[waldyrious]

I didn't realize that merging all the languages into a single folder was even being considered an option :) but even if I did, I agree it wouldn't be a good idea, and would still have proposed the structure I did in my previous comment.

[ibnesayeed]

Why can't we use some conventions in file names to accommodate for languages instead of making unnecessarily deep (and potentially repeated) directory structure? I would suggest we name files like foo.en.md and foo.fr.md. language translations are often sparse, creating folders for them will yield many folders with few files in them.

[sbrl]

Potential issue: How do we distinguish which is the master copy of a page?

@ibnesayeed: I'm not sure having it _completely_ flat is a good idea, because some PCs can get upset with a large number of files in a single directory. As tldr-pages is translated more and more, this might become a real issue.

As for how we'd actually go about making this change, I'd suggest opening a PR against the client spec that bumps it a major version first. This will almost certainly break clients, so we should give clients time to adjust to the new structure (and maybe provide an example zip of the new structure for testing purposes?).

[waldyrious]

and maybe provide an example zip of the new structure for testing purposes?

As I suggested above, this would be a good opportunity to start providing a .tar.gz file instead of a .zip one (#343), which would allow us to keep the .zip file with the current (legacy) structure, and the .tar.gz one with the new structure. That way we could simultaneously support both clients that haven't updated to the new structure and those who have (assuming they rely on the archive, that is).

[chamini2]

I would actually go for

foo.md
foo.es.md
bar.linux.md
baz.macos.md
baz.macos.es.md

[zlatanvasovic]

@chamini2 Wouldn't that kind of structure become quite unusable after some time? We already have thousands of files.

[chamini2]

Thinking a little more about it there is two things that come to mind:

The first is that I don't think having a long list of files is a problem, it let's you see what other options you have for that same command.

The second is actually against removing the current structure: what about commands that are very different in behaviour across systems but share the same name. For example, time in Linux measures the time a command took and in Windows gives you current time information. Having a list

time.md
time.es.md
time.windows.md
time.windows.es.md

And you would think they are all related...

[ibnesayeed]

My vote would be for directories for various platforms and shared/common resources, but language variants as secondary file extensions.

[sbrl]

The first is that I don't think having a long list of files is a problem

Perhaps not, but if a poor unsuspecting user navigates to the folder by accident Windows File Explorer in particular will encounter some serious performance issues :P

Note also that FAT32 has a limit of 65K files per directory. I don't know how many we've got now though.

[ibnesayeed]

Perhaps not, but if a poor unsuspecting user navigates to the folder by accident Windows File Explorer in particular will encounter some serious performance issues :P

I think the audience of this tool is somewhat tech savvy community who is familiar with CLI.

Note also that FAT32 has a limit of 65K files per directory. I don't know how many we've got now though.

If we only include a specific language variant of a specific tool, I would assume that the average number of language variants per tool will hardly be around 2 or 3, and even less if the number of tools increase as there will be many that will have no translations in place. Also, FAT32 is more common in old thumb drives with limited storage size, most windows users commonly use NTFS on their systems and Linux users use Ext4.

Many caching systems use directories, named after 4 character prefix of the hex digest of the hash of files, to uniformly distribute files in sub-folders to avoid too many files in the same folder. If I remember correctly, IPFS uses the same technique to store their content-addressed file blocks. A 4 digit hex code means 65,536 sub-folders under the main folder. So, I do not think a few thousand files flatly placed in a folder is a big deal. I have personally handled folders with over a million files in it.

[sbrl]

I think the audience of this tool is somewhat tech savvy community who is familiar with CLI.

Even in the CLI if you ls on a huge folder it can take a while to return.

I do feel that having some folder structure - even if minimal - would be helpful for navigating to the page you're after when contributing.

[zlatanvasovic]

Any structure, even minimal, makes displaying the items far more convenient and efficient (as @sbrl noted). Having a completely flat structure is completely counter-productive.

[fejx]

I feel like a decision could be made more easily if the different options would be listed clearly. Therefore, I created this little list of examples to specify the six options I could think of. If I missed something, I woud be happy to add it to the list.

If there will not be a resolution any time soon, I would suggest doing a simple vote between the maintainers.

Option A: Flat

|-- time.md
|-- time.es.md
|-- time.windows.md
|-- time.windows.es.md

Option B: Platform folders

|-- common
|-- |-- time.md
|-- |-- time.es.md
|-- windows
|-- |-- time.md
|-- |-- time.es.md

Option C: Language folders

|-- en
|-- |-- time.md
|-- |-- time.windows.md
|-- es
|-- |-- time.md
|-- |-- time.windows.md

Option D: Platform > language folders

|-- common
|-- |-- en
|-- |-- |-- time.md
|-- |-- es
|-- |-- |-- time.md
|-- windows
|-- |-- en
|-- |-- |-- time.md
|-- |-- es
|-- |-- |-- time.md

Option E: Language > platform folders

|-- en
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md
|-- es
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md

Option F: Leave it as it is

|-- common
|-- |-- time.md
|-- windows
|-- |-- time.md
pages.es
|-- common
|-- |-- time.md
|-- windows
|-- |-- time.md

[zlatanvasovic]

My vote goes to pages > en > common. It's practically the same as the current structure. I am completely against any flat structure, as it complicates the client implementations and slows down the file search.

[schneiderl]

I agree with @zdroid here. In my opinion, flat structures are impractical.
I would prefer option E on this matter.

[ivanhercaz]

Per @zdroid and @schneiderl, I also think option E seems the most practical solution.

[andrik]

I agree with you guys and option E looks better to me

[mflint]

I also prefer Option E, and would ask that we delay any change until all clients are able to work with the new structure.

(I'm maintainer of the tldr-pages iOS app)

[fejx]

When it comes to keeping backwards compatibility, I would suggest the following plan:

Stage 1: Support for both

The structure in the repository gets refactored to the new layout. When building the ZIP-file, the package structure gets copied over to the old layout as well. Since it gets compressed anyways, the ZIP-file should not increase in size notably. This means that the ZIP-file has the following structure (assuming that Option E is the way to go):

|-- en
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md
|-- es
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md
|-- common
|-- |-- time.md
|-- windows
|-- |-- time.md
pages.es
|-- common
|-- |-- time.md
|-- windows
|-- |-- time.md

Stage 2: Deprecation notice

After the files have been copied in the build script, add a line to all .md-files from the old directory structure with a notice that the users should update their client. It should be included in the regular structure of a tldr page, such as an additional example, or as part of the description in the third line of each page.

Stage 3: New system only

The build script gets simplified again so only the new structure gets zipped.

Conclusion

I am not 100 % sure that this would work since I am fairly new to the project. I assume that the clients have the platform names windows, linux and macos hardcoded, but it is possible that some clients would crash. That needs to be tested beforehand. If this idea would not work for any other reason I did not anticipate, I am open to feedback.

[ivanhercaz]

I think it is a good approach. We should think how many time should be the duplicate ther, so how many time we are going to give to perform the transition. In addition, I think we should open an issue in each client repository to warn its maintainers about this major change in tldr-pages. And I understand if we change the structure, we need to release a new version of the clients specification according with what we agree.

Thank you very much for your efforts in this issues @fejx ! 🤗

[fejx]

You are welcome! This project has become more and more useful for me, so I want it to become even better 😄. Also, old issues make me really nervous.

When opening issues on each client, be sure to add an example ZIP as well as a description of the migration plan with deadlines, so the maintainers can work on it right away.

But first of all, a decision has to be made on the new structure to finally close this issue (even though a new issue will emerge on the actual migration right afterwards).

[ivanhercaz]

When opening issues on each client, be sure to add an example ZIP as well as a description of the migration plan with deadlines, so the maintainers can work on it right away.

Yes, everything must be documented and planned to ease the transition and the development.

But first of all, a decision has to be made on the new structure to finally close this issue (even though a new issue will emerge on the actual migration right afterwards).

Yes, of course. These are just comments about a possibility, until we take a decision about it.

[zlatanvasovic]

We can keep the old architecture for some time if that's needed, but I think sending PRs to the active clients would be far more useful.

[sbrl]

Hey, thanks for contributing @fejx! No need to be nervous - old issues either have 1 of 3 problems:

• Nobody can agree on anything
• Implementing it is very complicated, and nobody has figured out how to do so safely
• Someone needs to step up and do something about it

Regarding the different options, my concern with option E there is that it becomes unclear where the master copy of pages are stored, and thus makes contributing more difficult. I propose _Option H_:

Option H

|-- common
|-- |-- time.md
|-- windows
|-- |-- time.md
translations
|-- es
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md
|-- fr
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md
|-- de
|-- |-- common
|-- |-- |-- time.md
|-- |-- windows
|-- |-- |-- time.md

This creates a clear delineation between the master copy (in English), and the translations (in other languages).

In addition, this has the added benefit that we don't break older clients that don't yet have language support (which is a fairly recent thing) that are more unlikely to be updated (thus allowing us to retain greater compatibility).

[zlatanvasovic]

Wise idea @sbrl. I still prefer E, but your proposal is more backwards-compatible.

[fejx]

When deciding on a structure, we should not consider the backwards-compatibility, because that issue can be handled separately. For instance, using the migration path I outlined here. I am not saying that I am opposed to Option H, just that we should not compromise on a solution because of backwards compatibility.

I also thought of a different approach for dealing with Stage 1 of the migration ("Support for both"):

Instead of cramming both structures into one ZIP, there should be two artifacts:

• http://tldr.sh/assets/tldr.zip
• http://tldr.sh/assets/v2/tldr.zip

This has multiple benefits compared to my original migration path:

• Also works on any future refactorings (v3, v4 etc.).
• Usage can be tracked per version by tracking the requests ("How many clients are still using v1?").
• Separate build-v1.sh file which is more idiomatic and readable.
• No mess inside the ZIP-files.

And just in case, I want to emphasize again that this approach would offer the ability to switch to the new structure in the repository immediately. The old layout would get recreated by the build-v1.sh-file, while the build-v2.sh-file would just ZIP all pages together. In other words: the legacy structure would only exist within the ZIP file served at http://tldr.sh/assets/tldr.zip, but not in the repository.

[ivanhercaz]

Interesting option the one exposed by @sbrl and the last comment by @fejx. The versioning of the zipped files might be a good option.

[sbrl]

That's an interesting approach @fejx. The other issue though as I mentioned earlier is potentially obscuring the primary 'master' copy of pages. With the proposed structure you'd need to go into pages/en to find the pages, instead of them being located directly in pages/

We can perhaps solve this with proper documentation, but I'm still cautious about it.

[fejx]

While I think your point is valid, we have to agree to something at some point.

@zdroid, @schneiderl, @ivanhercaz, @andrik and @mflint have voted for Option E. Do you guys still think that it would be the best solution, even after considering Option H?

[waldyrious]

I think option H is indeed more appropriate, not because of backward compatibility (because, as mentioned above, that can be handled more appropriately by versioning the zip files, regardless of what new structure we choose) but because it makes it easier to keep the pages in sync by marking the English copies as the master ones.

If we went with option E, I would say that would be more compatible with adopting an approach similar to Wikipedia, where each language version is completely independent from each other, and therefore non-English pages would not have to follow the English page's structure.

But I'm afraid that would make handling the pages' content and maintaining our guidelines quite hard to accomplish, and would also make room for unnecessary inconsistencies (whereas in Wikipedia these inconsistencies serve a purpose, since the same topic has different contexts depending on language).

Of course, we can still define the English pages as the master copies by convention and go with option E. I'd be fine with that approach as well.

(ps - Thanks so much @fejx for picking up this issue and moving it forward!)

[zlatanvasovic]

@waldyrious But what about the pages that will potentially be originally written in a language other than English (I'm looking at you, Mandarin)? This is not a really strange trend, and if this project grows, it will experience it as well. You may have noticed that many of the top GitHub's repositories are exclusively in Chinese. Some a bit less popular ones exist only in Japanese or Korean.

Although even with all that said, I do see substantial advantages in @sbrl's approach, as it's easier to find the original pages, as they are (all?) in English currently.

[fejx]

Why is it necessary that en-pages are considered the "master pages" to begin with? Is there any benefit to it? It seems like an unnecessary constraint.

The lookup logic defined in the client spec does not consider en-pages to be master pages. Instead, there is just an exception to the lookup rules that when you hit the language en, you should not look for pages.en, but for pages instead.

With option E, the page lookup logic would be simplified because it gets rid of this exception.

[agnivade]

I think option E is the simplest and cleanest of all. I don't think en pages should be considered as master pages. I would even go as far to say that there should not even _be_ a concept of master pages. Everybody is welcome to add pages in their own language, and that page may be in any language.

Rather the tooling should take care of figuring out which pages are missing out in which language, raise an alert when a page is edited in one language which has copies in other languages. We don't have those right now. But IMO, that way anybody who looks at the structure can immediately grok what's going on.

[waldyrious]

Good points, all. No objection on my side with treating English the same as any other language. I agree it's a cleaner approach, and if translators do want to use English as the source of their translations, they can still do so anyway (it's what happens with Wikipedia articles too, btw).

Count my vote for Option E as well!

[agnivade]

Awesome, great discussion guys !

[fejx]

Does that mean we are going for Option E? Are there any points to discuss left?

[agnivade]

Let's give it a few more days to give folks a last chance to comment. This is a big change, and there's no need to rush it.

@owenvoke @mebeim - wondering your thoughts on this.

Also, I realized this issue was opened in 2014. Took us 6 years to come to a decision. But better late than never !

[sbrl]

Hrm. I'm not averse to using option E here, but the problem arises in review. Translations are easier to review because they are based on a pre-existing English page. Allowing non-english master pages isn't a problem per say, but it would mean that we'd need to be careful that the 'original' copy of a page is properly reviewed thoroughly. For example:

• Command examples work correctly
• The style guide is followed
• The wording choice for the description of each example clearly reflects what the command is supposed to do

Unfortunately, I only know English, so I can't help with the review process for other languages before basic layout. Sometimes we get a translation in a brand-new language, so how would we locate someone else who knows that language to review it properly if it is a master copy?

Then there's the question of determining which page is the master. Is the French page the master? Or is it the Chinese one? How can we know for sure which is the original copy?

If we're not careful we'll end up with inconsistencies in pages for different languages, in which a new contributor, unaware of a page in another language, writes a totally new page that's not related to existing one in another language. Again, this isn't necessarily a problem, but it complicates matters when we need to update an example in 1 language because a mistake is found, but can't update other languages because the page layout is different and it shows different examples.

None of these are critical issues, but I think they need proper consideration - as they have abig impact on not only how the project moves forwards, but also on how we manage translations in the future. By adopting a "no language is the master copy", we effectively lock ourselves out of regular web-based translation services, as they require a main master copy in order to function.

My vote is still with my proposed Option H.

[zlatanvasovic]

@sbrl A tldr-bot may help with that, e.g. by comparing different languages. Although I really doubt we'll ever encounter such translation issue, except for the originally-Chinese pages. So it comes down to one language only.

[sbrl]

If it can happen - it will lol

But yeah, we'll need to significantly expand the tldr-bot I think to support this change if we go ahead with option E.

[waldyrious]

Well, we can certainly continuing the convention of treating the English pages as the master ones, regardless of where they're stored. The file structure was never a technical obstacle to break this anyway — it was enforced by convention, and it can continue to be if we agree that's what makes maintenance of the multilingual pages viable for the maintenance team.

Of course, it would be awesome to, on top of this, have tldr-bot perform an automatic check to make sure the example commands (not their descriptions, of course) are the same in every language version of a page — we just need to ignore the token contents, which are language-specific, and the comparison can be done pretty much verbatim. Or am I oversimplifying the problem?

[owenvoke]

I think either option E or the current situation makes the most sense personally.

I think with tldr-bot it would be good to add some GitHub Action workflows (or update the bot scripts) for various things to make this easier. :+1:

[agnivade]

Yes, the problem of reviews is a good point. But IMO it's a human obstacle, not a technical one. @waldyrious' point about treating English pages as master copies by "convention" is a good one. And some good amount of tooling via the tldr-bot to check for overall consistency of pages should take us a long way.

I would also suggest writing down language owners somewhere so that they can be added to reviews, or even have tldr-bot take care of it automatically.

@fejx - would be open to take the lead on this one and add some tooling around tldr-bot to ease this transition ? Also cc @Keating950 who is working on maintaining a list of pages needing translations.

[fejx]

I am willing to work on this issue, but I am unsure what you mean by "adding some tooling around tldr-bot". Are you referring to additional linting checks? Or are you referring to scripts to transform the existing structure into the new (as well as the other way around)?

[agnivade]

Ah I was referring to the linting checks. Things like:

• Adding a comment on a PR which modifies a page in one language, but there are copies of the page in other languages.
• @waldyrious' comment
  > have tldr-bot perform an automatic check to make sure the example commands (not their descriptions, of course) are the same in every language version of a page — we just need to ignore the token contents, which are language-specific, and the comparison can be done pretty much verbatim.
• If a page is added which has translations in other languages, post links to those pages in the PR so that it is easy to check for style consistency, and the PR submitter is made aware that other pages already exist. So the number of examples and their order has to be exactly same.
• Having language owners for non-english pages and ping them whenever a PR is created in any of those folders. (I think the GitHub CODEOWNERS file may automatically take care of this.)

The overall objective is to maintain consistency of the pages across languages.

[fejx]

Sure thing! I will try and see if I get a chance to take a look at it this weekend.

[ibnesayeed]

While this is an implementation detail that users might not care about, I personally think a scheme that groups variations related to a specific command together would be much more approachable and manageable from contributors' perspective. If the fear is that too many files in a single folder will make things slow in certain machines/OSes, I think users of this program are technically sound ones who generally have development machines that are rather recent and powerful. Even that issue can be solved by creating folders for each command and place all the variations of documents related to a specific command in its corresponding folder. If, in the future, we find there are way too many commands to manage, it can further be organized in groups of folders named after the first letter of each command.

[agnivade]

That's not a bad point. So far, we have been considering platform > language, and language > platform, but command > language is also an option. The problem that I see is that it's an inverted way of grouping things. Usually things are grouped at a larger level (platform, language), and smaller things are included inside it (pages). It's not an issue per se, but I don't see any problems from a contributor's perspective with option E too.

[fejx]

I don't see how the structure command > language would reduce the number of files in a folder.

|-- en
|-- |-- ls.md
|-- |-- grep.md
|-- |-- man.md
|-- de
|-- |-- ls.md
|-- |-- grep.md
|-- |-- man.md

These folder have three files each.

|-- ls
|-- |-- de.md
|-- |-- en.md
|-- grep
|-- |-- de.md
|-- |-- en.md
|-- man
|-- |-- de.md
|-- |-- en.md

Here, the root still has three files, they are just folders instead.

[agnivade]

It's not a matter of reducing the files. I _think_ it's about "would be much more approachable and manageable from contributors' perspective.". Although I don't see any issue with that with option E too.

[sbrl]

I like that idea of tooling @waldyrious, and thanks so much @fejx for taking the lead there!

[zlatanvasovic]

Just one thing. If the structure is language / platform / page or platform / language / page, it is easier to implement it into a tldr client. Any structure where page is before the others complicates implementing the language and platform features. This is coming from refactoring Python and C clients recently.

[fejx]

Sure, I am looking forward to implement the tooling! However, I will wait with the implementation until we have converged to a mutual decision and the issue is closed. Otherwise, I might end up implementing checkstyles for a structure we are not going to use.

[agnivade]

@ibnesayeed - Can you lay out specifically why you believe inverting the grouping by putting the page first "would be much more approachable and manageable from contributors" ? And what is the problem this solves over option E ?

Thanks.

[ibnesayeed]

One of the reasons why I think putting command name first in the hierarchy would be a better choice has to do with how semantic URIs and resources on the Web behave. Just think about it, what is the main resource we are talking about that both contributors and consumers are interested in? I think it is the command itself, not the language, not the platform, those are variations of the main resource. This is precisely how content negotiation works in web servers. The URI points to the abstract resource, from there, content type, language, content encoding, character encoding, and many other aspects are negotiated. Content negotiation often happens via respective headers, but for static files, many servers use a hierarchy of file extensions to store and serve different representations from a specific directory. This is why I think we should have one folder per command and place every variation (those we envision now or may support in the future) can be placed in that folder.

Now, lets go through some practical issues if files are organized in platform and language hierarchy and suppose a contributor or user trying to modify local copy, but he/she does not know about file globbing (i.e., wildcards) and prefers to navigate through files using GUI file managers:

• What if they are not sure which platforms support the command, should they try each platform folder and then each language folder until they find what they were looking for?
• What if they are not sure which platforms, the new command they are adding, supports, which folder should they place it in?
• What if they do not know what languages a command is documented in already and what languages are still missing, are they suppose to check every language folder?
• What if a command was earlier only supported on one platform, but later it added support for another platform and a while later it supported even more platforms, should all related files/folders be moved around in the hierarchy each time the support matrix changes or is it better to simply play around with nested file extensions when variations change or added?
• What if they want to see all the variations (i.e., supported platforms and completed languages etc.) that are available for a specific command with the intent to fill some of the cavities, should they be hunting all over the folder hierarchy?

The assumption in these examples is that they know which command they are looking for (as the command is the primary resource). There might be some counter-examples as well, such as one contributor woke up on a breezy morning and decided to add translations to a specific language, irrespective of which commands are available to translate, but I would argue, such situations will be rare and they are not impossible in the proposed hierarchy of resource first, variations next.

One folder per command gives some flexibility for future changes. For example, if a client prefers PDF, PNG, GIF, JSON, or HTML, these content types can be generated and placed in the respective commands' folders with appropriate extensions. Something like:

<cmd_name>/<cmd_name>[.<platform>][.<lang>][.<other_attrs>].<mime_ext>`

Or

<cmd_initial>/<cmd_name>/<cmd_name>[.<platform>][.<lang>][.<other_attrs>].<mime_ext>`

The latter form would be suitable if we think that the number of commands will be more than the number of files certain platforms' file systems can handle easily. This technique is used in cache stores as well where first few characters of the hash of a resource's identifier is used to place it in a sub-folder.

[agnivade]

That's great feedback. Thank you for spending the time to write this.

I think comparing asset serving in web servers with how these pages are organized are two orthogonal concerns. The URLs in a web server need not necessarily match with the actual content in a directory.

Coming to the issues with option E, yes some of them are fair concerns. I'll try to address them individually.

What if they are not sure which platforms support the command, should they try each platform folder and then each language folder until they find what they were looking for?

I think a contributor should be pretty sure which language they want to contribute in. Coming to platform, there's just 2 places to look into - the specific platform they are in, and then common.

What if they are not sure which platforms, the new command they are adding, supports, which folder should they place it in?

That is already clarified in the spec. It's common.

What if they do not know what languages a command is documented in already and what languages are still missing, are they suppose to check every language folder?

This is a weird usecase. Are you saying someone who knows like 5-6 languages decides to contribute all translations for a given page ? Maybe it's a problem then.

What if a command was earlier only supported on one platform, but later it added support for another platform and a while later it supported even more platforms, should all related files/folders be moved around in the hierarchy each time the support matrix changes or is it better to simply play around with nested file extensions when variations change or added?

Assuming we are not going with the file extension route, this problem would still exist in this model. You would have to move around files in the sub-folders.

What if they want to see all the variations (i.e., supported platforms and completed languages etc.) that are available for a specific command with the intent to fill some of the cavities, should they be hunting all over the folder hierarchy?

Right, this is slightly cumbersome. But again an edge-case IMO. Personally, I have never needed this.

I think for these edge cases, it may not be right to hinge our decision solely based on the comfort for contributors "searching files through a GUI". I agree that editing files for a given platform is possibly the only sore thumb here. But I think good amount of tooling is the right answer here.

I am not opposed to this option per se. But it seems like it is asking us to to spend too much effort for little benefit. We also have to take into account the fact that _all_ clients have to change their code to accomodate the new structure.

I will leave it to what others think.

[agnivade]

Not to sound hasty, but I'd hate to lose the steam that we have picked up here. Rarely do we see 6 year old issues being discussed actively.

If there aren't any objections, I'd like to start off with option E as the accepted model.

[fejx]

Since noone objected, I would suggest closing this issue about the decision and opening a new one regarding the execution of the new page structure.

[waldyrious]

@fejx what do we gain by opening a new issue rather than using this one?

[zlatanvasovic]

A clear discussion which doesn't require scrolling past 100 replies.

[agnivade]

In other projects, I have seen the top comment being edited and a summary being added to reflect the final decision taken. That way, anybody can get a tldr without having to read through the whole thread. I don't have any opinions either way, just excited to see some work getting started.

[waldyrious]

In other projects, I have seen the top comment being edited and a summary being added to reflect the final decision taken.

I'd prefer this too.

[fejx]

Adding a summary on the top is a good idea, but I still would like to open a separate issue.
My reasons why I want a new issue:

• The decision is a separate discussion which has nothing to do with the execution. Therefore, a separate discussion should take place in a separate issue.
• The issue is full of comments which are irrelevant for the execution.
• Closing would put a defined end to the discussion, allowing us to focus on the how not the what.
• It is easier to read for archival purposes. (New issue will link to the discussion about the decision)

[waldyrious]

Fair enough. We don't usually have separate issues for discussion vs. implementation, but your points are well taken.

Feel free to create the new issue for the implementation. I'll go ahead and close this issue with the decision of going with Option E described above.