Powershell: HelpSystem should consume markdown natively

[SteveL-MSFT]

HelpSystem currently relies on MAML based help. We have platyPS which converts markdown to MAML. The HelpSystem should understand markdown natively removing the conversion step.

[iSazonov]

If we'll directly read and parse Markdown files we have to rewrite all help subsystem which currently full based on xml/maml. If we'll rewrite the help sybsystem maybe use a lightweight DB engine like MongoDB? In the case we'll parse once with Update-Help and then import help info in a DB.

[SteveL-MSFT]

@iSazonov using a local noSQL DB like MongoDB may make sense. First step is to separate out the HelpSystem.

Once we can consume markdown natively, we can do some interesting stuff with rendering any markdown (not just Help) on the console

[iSazonov]

So do we need to implement a parser and a console renderer?

[SteveL-MSFT]

Would be best if we can use some existing OSS implementation

[iSazonov]

I see many parser implementations (PlatyPS?) but can not find console renderer.

[SteveL-MSFT]

Reusing PlatyPS. We will probably have to write a console renderer.

[iSazonov]

Interesting JavaScript implementation https://github.com/linuxenko/lessmd

[vors]

python implementation https://github.com/axiros/terminal_markdown_viewer

[iSazonov]

We have RFC to move HelpSystem out PowerShell Engine. So we should solve that we implement first.

[MaximoTrinidad]

Just curious!

This is not going to impact Windows PowerShell Help system?
As, Windows PowerShell is complete. This is only for PSCore6. Right!!

Also, is this also going to impact the the online Microsoft Doc information?

:)

[kvprasoon]

IMO, this will be for PowerShell Core and PowerShell cmdlet docs are already using markdown .

[MaximoTrinidad]

@kvprasoon, Thanks!

[SteveL-MSFT]

@MaximoTrinidad correct, this change is only for PSCore6 which means for Windows PowerShell we'll continue to use platyPS to convert the markdown to MAML if that help is updated.

[vors]

One thought about smoothing the transition: when we will have HelpSystem in a separate module, we can add part of platyPS there, that can convert platyPS markdown to the tradition help objects on the fly. That way, users can stop shipping maml xmls earlier and package consumers can still get it in the traditional form.

[iSazonov]

In short from discussion with @vors

• In any case we need ported console markdown renderer.
  Perhaps we could support pluggable markdown renderer modules (ex. show HTML) (maybe as part of a host implementation)

• In any case we should parse mardown help files in order to provide the ability to search and filter.
  It does not require use platyPS.
  Preffered way is direct using markdig (BSD-like license) which is fast and extensible.

• The parsing in turn implies the existence of a scheme.
  PlatyPS uses a complex scheme. Really we should simplify it because we don't need MAML anymore.
  Actually we have enough the markdown scheme which has Section.
  In other words we can realize to start something like
  Get-MarkdownHelp -Name CmdletName -Section Description

• To ensure backward compatibility we need to add an option to convert existing Help (xml) files to the markdown on the fly.
  To do this, we will need integrate platyPS package (yet it doesn't exist).

• To avoid repeated markdown parsing and maml/xml-markdown conversions, we could use a cache based on noSQL (mongoDB?).
  Update-MarkdownHelp cmdlet could update not only the source files but also the cache.

• Given that this cache will contain a list of cmdlets by modules,
  this cache could be used to speed up the search for cmdlets for module autoloading.

• This is all very similar to the roadmap for creating a new MarkdownHelpSystem that is independent of the current HelpSystem but backward compatible.

Perhaps we can not waste time and not separate the current system into a separate module.
We can work on the new system right now. The main difficulty is the lack of ported console markdown renderer.
We should consider to use https://github.com/rprichard/winpty although this may require a refactoring of the console output code.

[SteveL-MSFT]

Great discussion!

cc @daxian-dbw @adityapatwardhan who has been looking into this

[iSazonov]

Add about using winpty.

[iSazonov]

Since it is assigned to 6.3 I share follow.
First, we need decouple HelpSystem to separate dll/module.
This opens the way to the development of alternative HelpSystems.
Current HelpSystem is fully integrated into Engine.
We need design new public API for pluggable HelpSystems. Although at start we could do the separation using internal api and internal visibility.
Some internal api from the engine are also used there and it is not clear what to do with it.
Most likely we will have to convert Help function to binary cndlet.
Also there can be some (1-2) breaking changes.

I could push WIP PR but only if MSFT team is ready to lead the work.

[daxian-dbw]

@iSazonov I think @adityapatwardhan already has code changes for a prototype that separates HelpSystem from the engine.

[adityapatwardhan]

Yes, I do have one. It does not compile completely yet though.

[iSazonov]

I tried to do it three times and understand what the problems are :-)

[iSazonov]

Can you (MSFT team) share current status?
7.0 milestone is best time to decouple the help subsystem and make it pluggable and extendable.

I suggest the plan (see my comment above):

1. Move Help Sybsystem to separate dll. It requires changing internal API in engine.
2. Make it pluggable. It is breaking change and it requires new public API. It is very simple API.
3. Implement new Help module with Markdown support.

[vexx32]

@stevel-msft mentioned on Twitter that this was something he would like to look at doing for v7.

Not sure what the necessary work volume is for this, though. :)

[iSazonov]

Points 1 and 2 from list above is low volume, point 3 is high volume but can be implemented step by step without removing current Help system..

[SteveL-MSFT]

I think we should separate moving the HelpSystem out of SMA.dll as independent of this enhancement. That would be nice to have, but not critical for PS7. We should definitely have this in PS7 so we don't need MAML. However, we also need to change how we publish updateable help so that we have zips with just markdown, but need to keep MAML publishing for downlevel PS.

[iSazonov]

@SteveL-MSFT Do you mean that you have plans to integrate Markdown to _current_ HelpSystem? I'm afraid it will complicate the module significantly, make it much more difficult to move the module out SMA and further full decouple. I'd prefer that we implement _new_ module with Markdown support. Before it becomes stable, users could continue to use the old module.
In my last attempt, I advanced about 80% so that we could really do it.

[SteveL-MSFT]

@iSazonov preference would be to separate out HelpSystem as separate module, if you have a work-in-progress prototype, perhaps you can submit a draft PR and @adityapatwardhan can just work off your branch (or maybe create a new branch in this repo like we did for .NET Core 3.0 port work).

[iSazonov]

@SteveL-MSFT The PR #9841 is created.