Content: (Preparing to) move NSS docs off MDN

Created on 24 Feb 2021  ·  20Comments  ·  Source: mdn/content

We need to help the Mozilla NSS team move their documentation off MDN. To be clear, we are talking about everything underneath https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS.

The goal is to get to the stage where all of the documentation under /NSS is available as static, standalone HTML files that are free from KumaScript macros, have links fixed up as much as is possible, and can be used in the new destination (which we are hoping will be the Firefox source docs Sphinx instance)

The tasks required are:


    • [x] Fix all the flaws generated on the NSS docs (this is a list of errors found for each document, mainly broken links).


    • [x] Most of these are automatically fixable, so we should run the commands to do this.


    • [ ] Scrape all of the docs under the /NSS directory from the MDN site, and get rid of all the MDN-specific scaffolding to just leave a minimal standalone HTML document for each one (this includes removing any navigation menus). Present them in a directory structure the same as the existing directory structure in the content repo.


    • [ ] Come up with a list of manual fixes that are needed for each document. Make these fixes for each document to get them ready for use in the new system (I was thinking that a team comprising of me, another MDN writer, and Niklas from the NSS team, could do this work)


    • [ ] Once the content is prepared, the NSS team would need to do final sorting of the links inside the resource:

    • Change all internal links that link to other NSS docs like this — /en-US/docs/Mozilla/Projects/NSS/xxx > /new/relative/path/xxx

    • Change all internal links that link to other docs on MDN not inside the NSS section like so — /en-US/docs/Mozilla/xxx > https://developer.mozilla.org/en-US/docs/Mozilla/xxx


    • [ ] On the MDN side, our engineers would need to put hard redirects in place from the old locations of the pages on MDN, to the new locations of the pages.

picture chrisdavidmills

All 20 comments

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

hamishwillee picture hamishwillee on 9 Mar 2021

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

Yeah, unfortunately not, by the looks of things. I'm assuming in doing this work you are just working through the folder structure alphabetically? Do you think you can create a simple list in here that shows which directories have had their broken URLs fixed so far, and by what PR? Just so others that might work on it can easily see at a glance what else there is to be done?

chrisdavidmills picture chrisdavidmills on 9 Mar 2021

Possibly, but if you can tell me how to fix the NSPR links and confirm that we can ignore the archive links that would be a smaller task. This pretty much fixes everything else that can be fixed due to links being "accidentally broken"

hamishwillee picture hamishwillee on 9 Mar 2021

@chrisdavidmills OK, so I did a bunch more fixes in #3134.
Below is a list of what remains to be fixed. Not completely exhaustive, but enough to get a good picture on high level things that remain - a bunch of them could be bulk fixed if we knew the right targets.

Note also that many of these are actually archived and just showing up as broken link. See https://github.com/mdn/yari/issues/2675. This applies to much of the "/en-US/docs/Mozilla/Projects/NSPR" links

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_Developer_Tutorial

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module

Broken Links
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Non-FIPS_mode_of_operation 👀 line 21:15

  • FIPS is a special mode (difference here). FIPS functions are documented here
  • NON-Fips functions are not identified. They MAY just be these ones: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Functions

/en-US/docs/Using_CXX_in_Mozilla_code line 121:106

  • https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/CVS "Getting Mozilla Source Code Using CVS"

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198
INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502
INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS ("Network Security Services" - main page)

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19
INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Overview ("Overview of NSS")

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:57 /en-US/docs/Mozilla/Projects/NSPR
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:326
INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 47:15

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/FAQ "NSS FAQ"

INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 46:415
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 72:1481

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_Sources_Building_Testing "NSS Sources build testing"

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 14:338

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Introduction_to_Network_Security_Services "Introduction to Network Security Services"

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR 👀 line 25:272
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 141:15
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 142:15
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 147:109

Lots broken under the /Projects/NSS/Reference - just subset listed

The point here to see that many of these are common

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_CancelFunction

/en-US/NSC_CancelFunction line 27:19

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_CloseAllSessions

/en-US/NSC_CloseAllSessions line 27:109

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_DigestInit

/en-US/NSC_DigestInit line 28:19

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_DigestFinal

/en-US/NSC_DigestFinal line 30:105

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_DigestInit

/en-US/NSC_DigestInit line 28:19

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/FC_SetAttributeValue

/en-US/NSC_SetAttributeValue 👀 line 32:19

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/CERT_FindCertByIssuerAndSN

/en-US/NSS/CERTIssuerAndSN 👀 line 19:260

No flaws reported:

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/CERT_FindCertByDERCert
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Blank_Function
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Code_Coverage
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Encrypt_Decrypt_MAC_Keys_As_Session_Objects
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/tools/NSS_Tools_sslstrength
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_tools_:_vfyserv
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/troubleshoot
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_Certificate_Functions
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_Initialize
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_tools_:_certutil
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_Releases (and all sub docs
  • https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/SSL_functions
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_API_GUIDELINES
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/nss_tech_notes
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Building
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Tools
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_Sample_Code

    • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/nss_sample_code/NSS_Sample_Code_Sample1 to 6

  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_Third-Party_Code
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Implement
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11/Module_Specs
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/HTTP_delegation
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/TLS_Cipher_Suite_Discovery
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Certificate_Download_Specification
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/FIPS_Mode_-_an_explanation
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Functions
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/HTTP_delegation_clone
  • https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference "NSS Reference"
hamishwillee picture hamishwillee on 15 Mar 2021

Thanks @hamishwillee , that's super helpful.

I will see if I can find some time to do a few of these. Also feel free to to keep picking bits off ;-)

chrisdavidmills picture chrisdavidmills on 15 Mar 2021

Great. Will do.
FYI, I found the NSPR link - it exists, but is archived: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR

Quite a few of these might also be fixed by creating one or two little docs - e.g. aCK_SESSION_HANDLE is clearly a copy of https://www.cryptsoft.com/pkcs11doc/v100/pkcs11__all_8h.html#aCK_SESSION_HANDLE

hamishwillee picture hamishwillee on 16 Mar 2021

Yet more done in #3290.

This is what is left. I think that I'm at the end of what I can do

Broken Links
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15
/en-US/docs/Using_CXX_in_Mozilla_code line 121:106

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198
INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502
INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19
INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

/en-US/NSC_CancelFunction line 27:19

/en-US/NSC_CloseAllSessions line 27:109

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_DigestFinal line 30:105

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_SetAttributeValue 👀 line 32:19

/en-US/NSS/CERTIssuerAndSN line 19:260

hamishwillee picture hamishwillee on 19 Mar 2021

@escattone a friendly ping on this one Ryan. @hamishwillee has done a whole bunch of work to fix broken links in these docs, so the next stage is to scrape the rendered NSS docs off the site into a "pure" HTML form that the NSS team can use.

Also pinging @beurdouche for an update as to the latest progress on this.

chrisdavidmills picture chrisdavidmills on 23 Mar 2021

Thanks a lot, this is very helpful! :)

Regarding the links, if you can point the Non-FIPS mode to the page you suggested, it would be great
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Functions

I think that there is no need to worry about the rest of the links.
This list is useful and I can fix/delete/rewrite those after we move off of MDN 👍

beurdouche picture beurdouche on 23 Mar 2021

Hi @beurdouche , the link you refer to will be fixed by https://github.com/mdn/content/pull/3513.

I am not promising on my life that all links are fixed - but certainly the main docs have all been gone through and any flaws shown by the automatic tooling have been fixed up. The ones marked as Invalid above show up as broken if you build locally but work on the main site (i.e. the scaped version will work)

hamishwillee picture hamishwillee on 26 Mar 2021
❤1

Hi @beurdouche ! It looks like we've finished doing link fixes on this content, so it is ready to scrape and get into a rendered out HTML format for you folks to use.

In which case, the next stage is to create a scraper to scrape the content off of MDN. We will probably create a simple script to do this, and then hand it over to you folks so you can do the conversion and tweak it as needed.

My question here is — what do you want the HTML to look like, exactly? Do you need a full page for each, including doctype, simple document head, plus body, or do you just need the body content for each? Or something else?

chrisdavidmills picture chrisdavidmills on 1 Apr 2021

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded.
(Easiest example would be index.html)

dergoegge picture dergoegge on 1 Apr 2021

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded.
(Easiest example would be index.html)

Do you want them with the YAML front matter included?

chrisdavidmills picture chrisdavidmills on 1 Apr 2021

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

dergoegge picture dergoegge on 1 Apr 2021

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

That's correct — the scraped page would include the <h1>, as wll as all the KS macros being rendered out.

So assuming that you don't have any usage for the tags, slug, etc, then that sounds ok.

My esteemed colleague @escattone and his team would be helping with the actual scraping code here, so I'll pass this over to him. Ryan, the one thought I still have here is that we'd probably need to scrape them and provide the scraped content in a folder structure that matches the existing folder structure in the source, to help @dergoegge put them in the same kind of structure that they are currently on MDN, in their new home. This would in turn help us to put redirects to the new locations in place much more easily. Sound reasonable?

I was thinking you could start by scraping a few docs, and sharing them with Niklas to check if tey are ok for his purposes.

chrisdavidmills picture chrisdavidmills on 1 Apr 2021
👍1

@chrisdavidmills All sounds reasonable! Let me get cracking on the scraper this week, share an example with @dergoegge, and we can iterate from there.

escattone picture escattone on 13 Apr 2021
❤1

@escattone thanks, much appreciated.

chrisdavidmills picture chrisdavidmills on 13 Apr 2021

@dergoegge @beurdouche I created an initial set of "distilled" NSS docs in https://github.com/mdn/nss-docs. By "distilled", I mean I extracted <main> and <title> from the MDN version of each doc, and re-packaged in a fresh, minimal <html> package (so basically stripped out the MDN header, footer, and sidebar). Does that work for you all? I'm happy to adjust as you wish. Cheers.

escattone picture escattone on 17 Apr 2021

@escattone Could you change it so the files only contain the inner html of <article class="main-page-content"> on each page?
Besides that it looks good to me so far👍

dergoegge picture dergoegge on 20 Apr 2021
👍1

@dergoegge I've updated https://github.com/mdn/nss-docs to reflect your requested change. Hope that works for you.

escattone picture escattone on 21 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ydaniv picture ydaniv  ·  4Comments
chunkybanana picture chunkybanana  ·  3Comments
Codefeak picture Codefeak  ·  3Comments
sideshowbarker picture sideshowbarker  ·  4Comments
StephanieCunnane picture StephanieCunnane  ·  4Comments