Semgrep: Find a better home for semgrep documentation

[mschwager]

Currently semgrep documentation lives as Markdown files in the docs/ folder. We have a GitBook instance here: https://docs.semgrep.dev/, but it's unmaintained. As seen in #1004, we're starting to have some growing pains with simple Markdown files in a directory. Let's find a good documentation solution.

Some open questions:

• Where will it live? Github? GitBooks? https://r2c.dev/? Readthedocs? Host it ourselves? Other?
• What will the format be? Strongly encourage Markdown here since it's a good balance between power and usability, and it's what all our existing docs are in.
• How should we structure things? What's the hierarchy?

CC @dan5382, @pabloest, and @dlukeomalley to hear their thoughts.

[clintgibler]

• I strongly vote for Markdown.
• I think we should do something that looks nice and is easy to setup, maintain, and add content to.
• I don't think we should spend too much time on this decision, we can improve it later.
• Not sure about the final hierarchy, but my first thought is there are at least 2 "buckets" of types of content:
  
  
  
  • Core, authoritative docs that list everything you can do in a pattern, pattern-foo, etc.
  
  
  • Examples of rule writing / hands on tutorials.
  
  
• I feel like having ^ both together is useful and makes it more easily discoverable for users.

[clintgibler]

@DrewDennison was a fan of gitbook but @ievans wasn't.

[DrewDennison]

I like gitbook and docs.semgrep.dev was a thing but there was some un-debugged sync issues so we just reverted to GH as source of truth

[ievans]

I would highly recommend putting the docs as a top-level semgrep.live entry, semgrep.live/docs. Can be markdown/gitbooks integrated into semgrep.live; would make it easy to cross-link from the editor and rule packs.

EDIT: also, we can deploy the docs to semgrep.live as part of the release process when we upgrade the version of semgrep it uses by default.

[clintgibler]

+1 I like the docs living on semgrep.live/docs

[pabloest]

Here are some thoughts on documentation

Short term

In the short term I believe it’s important to:

1. Update the documentation content so it reflects the latest developments in semgrep CLI.
2. Make the docs discoverable and visible from semgrep.live.
3. Move docs out of the semgrep GitHub repo.

2 is already done thanks to a recent update. I think #3 is important because:

1. It’s rather cumbersome to navigate repo directories using the GitHub file browser.
2. There’s limited table-of-contents capabilities in the repo structure to aid with #1, and very clunky search.
3. We have no analytics insight into how users consume the content (how people got to the docs, what docs are most viewed, how long they’re viewed, if doc navigation works, what links are clicked, etc.).
4. There’s no ability to publish video content or provide interaction aids such as copy-to-clipboard buttons.
5. The visual layout is rather constrained.
6. We get no SEO benefits from the publication of this valuable Semgrep content.

Longer term

Longer term, say in a month, I think we our docs should be excellent/elegant. IMO this includes:

1. Documentation content is kept updated between now and this future state.
2. Documentation is on the same domain as semgrep.live. This is advantageous for SEO, is more discoverable, provides a smoother user experience, reinforces the product brand, allows us to provide small suggestions as people use the live editor, and lets us build examples and tutorials so users can learn and try at the same time.
3. We move large bodies of learning-focused content such as the tutorial or how-to videos out of the main product and place it alongside other documentation.

Point of discussion here may be how to accomplish the short term goals (if there’s agreement) and how far to push into the longer term goals. For example, Matt proposes moving Markdown files to a hosted Gitbook on docs.semgrep.dev.

Edit: ported this comment from another docs-related issue that is now closed.

[dlukeomalley]

Is this done now that semgrep.dev/docs exists? 🤞

[mschwager]

Yup! Good catch