St2: Mistral Deprecation Game Plan

[armab]

Mistral deprecation in favor of new workflow engine Orquesta was communicated for quite a long time.
StackStorm v3.2 will be latest version to ship Mistral with full removal in st2 v3.3 per Roadmap.

There is a helper Mistral -> Orquesta conversion tool available:
https://github.com/EncoreTechnologies/orquestaconvert
Special thanks @nmaludy an EncoreTechnologies for the tool

The Plan: https://github.com/orgs/StackStorm/projects/16

• [x] Identify & Migrate any missing mistral workflows in StackStorm Exchange
• [x] Remove from st2ci testing and st2cd release automation
  
  
  
  • [x] st2ci https://github.com/StackStorm/st2ci/pull/189 (@amanda11)
  
  
  • [x] st2cd https://github.com/StackStorm/st2cd/pull/440 (@amanda11)
  
  
• [x] st2 code cleanup to remove the mistral integration https://github.com/StackStorm/st2/pull/5011 (@amanda11)
• [x] st2-packages
  
  
  
  • [x] Remove deb/rpm rules from st2-packages https://github.com/StackStorm/st2-packages/pull/656 (@amanda11)
  
  
  • [x] Remove mistral from scripted curl-bash installer https://github.com/StackStorm/st2-packages/pull/657 (@amanda11)
  
  
  • [ ] Later step: move st2-packages rpm/deb scriptlets into st2 https://github.com/StackStorm/st2-packages/issues/662 (@armab)
  
  
• [x] Remove Mistral from st2tests https://github.com/StackStorm/st2tests/pull/190 (@amanda11)
• [x] Close all Mistral issues
• [x] Remove mistral from Deployments
  
  
  
  • [x] ~st2-docker~
  
  
  • [x] ~Chef~ (deprecated)
  
  
  • [x] Ansible https://github.com/StackStorm/ansible-st2/issues/270 (@amanda11)
  
  
  • [x] Puppet https://github.com/StackStorm/puppet-st2/pull/313 (@nmaludy)
  
  
• [x] Remove mistral from the launch_dev.sh script (https://github.com/StackStorm/st2/issues/5012)
• [x] Update docs
  
  
  
  • [x] https://github.com/StackStorm/st2docs/pull/1009 (@amanda11)
  
  
  • [x] https://github.com/StackStorm/st2docs/pull/1010 (@winem)
  
  
  • [x] Update any missing diagrams/pictures
  
  
  • [x] +Sync-up Roadmap
  
  
• [x] Archive mistral forks/repos https://github.com/StackStorm/mistral_dev https://github.com/StackStorm/mistral https://github.com/StackStorm/st2mistral (@armab)
• [ ] Blog post about Orquestaconvert project (@nmaludy)
• [ ] Mention deprecation in v3.3.0 release announcement (@nmaludy as a release manager)

@m4dcoder feel free to adjust the task list to make sure we don't forget any other places with mistral occurrences.

[Deepika-Kamalla]

So, would it be a valid assumption that PostgreSQL would also be removed from v3.3?

[armab]

@Deepika-Kamalla That's correct.

[Deepika-Kamalla]

Thanks, Armab!! Is there any tentative timeline when v3.3 would be GA?

[armab]

No timeline yet. This is something yet to be figured, considering recent LinuxFoundation transition.

[amanda11]

I'll start looking at the one-line installer changes.

[winem]

Just a quick note as I just noticed it while working on other PRs: We can also drop Postgres from the install docs as soon as Mistral is officially removed. Not sure if this is already clear and included in update docs.

[winem]

I started to remove mistral from st2docs and started with the reference/ha.rst and referenced configuration sample (st2/conf/HA/st2.conf.sample & st2/conf/HA/nginx/st2.conf.blueprint.sample).

Who has access to edit the HA reference deployment image to remove PostgreSQL? https://docs.google.com/drawings/d/1X6u8BB9bnWkW8C81ERBvjIKRfo9mDos4XEKeDv6YiF0/edit

An update of the deployment overview is the only thing that's missing to be done with the ha.rst.

I'll proceed with the remaining reference sites tomorrow and provide a PR once I'm done.

I would suggest to create a MR per chapter so that we can share the work a bit. Otherwise I could also offer to take care of the whole docs but this probably takes some days until I'm completely done. Let me know if you have other ideas or prefer another approach.

[amanda11]

@winem Sharing chapters sounds like a great approach. I don't know about the image updates, but if I get chance to start on some docs tomorrow - I'll let you know the chapter and I'll avoid the reference/... pages.

[blag]

@winem I removed Mistral from both the HA deployment big picture and the deployment single box drawings. I did tweak the numbers, so you'll need to update the corresponding numbers where they appear in the documentation (I'm not sure where all they show up, or I would have done that myself).

Let me know if you can't view/export them and I'll export them for you (PNG or SVG?). And feel free to ping me if you need anything else removed or tweaked.

[amanda11]

@winem I'll start on the install chapter - so far looks like all files in my PR will be in the install sub-directory except for upgrade-notes

[winem]

@blag I can't access the files yet but requested the access permissions via Google. I guess you have a mail. I'll create a draft PR later today and start with the next chapter also this evening or latest tomorrow. Will let you know which one I choose next.

[winem]

What is meant with Remove mistral from Deployments? I'm wondering if that's something where I can help.

[amanda11]

@winem I believe it's all the different deployments, ie. its just a header for the checkboxes that are under it e.g. ansible etc. All are done except puppet, and I believe @nmaludy said he'd take care of that.

[blag]

Here is the updated st2-deployment-big-picture:

And the updated deployment-single-box:

The Markdown to include them:

![st2-deployment-big-picture](https://user-images.githubusercontent.com/597113/90346098-fc7aeb00-dfda-11ea-95ad-12b95e1b17b8.png)

![deployment-single-box](https://user-images.githubusercontent.com/597113/90346101-00a70880-dfdb-11ea-9ad0-07752ad074ea.png)

But the docs are all written in RST, so you'll have to adapt it to that. I would suggest hotlinking to these images (eg: use these image URLs) rather than uploading them to GitHub a second time.

Let me know if you need anything else!

[armab]

Nice one, thanks @blag!

[winem]

Hi @blag, thanks for the updated diagrams!

Can you also update the st2-deployment-multi-node.png (source: https://docs.google.com/drawings/d/1_BJa9ZtBjFa1Dxx6cPiFlmpTS9AsNzkkvp_vuyVV3bw/edit)? That would be perfect and I would take care of the update in the docs then.

[winem]

I updated the big-picture (see #1016) but could not find the single-box diagram anywhere in the docs. It would be great if a 2nd pair of eyes can confirm that I did not missed it and my grep-foo did not fail, too.

Now we need the st2-deployment-multi-node.png to be done with the diagrams.

[winem]

I've identified another diagram that needs an update. The architecture_diagram.jpg which is also shown at https://docs.stackstorm.com/latest/overview.html still mentions mistral workflows, services and definitions. @blag do you have access to the sources?

[blag]

I missed the multi-node deployment image. The (new) Google Drawing for that is here:
https://docs.google.com/drawings/d/11YAWFFMkEBw5VJGpMGNJCtT5IiiGsdrPJcH4jGaxeeo/edit

But here it is for your convenience:

[amanda11]

Thanks @blag . PR to include that diagram is here https://github.com/StackStorm/st2docs/pull/1018. Just the architecture diagram remaining.

[blag]

And here's the architecture diagram:

I'm in the process of trying to export this image to a more universal format (like SVG!), and/or to get the TSC a traditional OmniGraffle license for free or reduced cost.

[winem]

Ok, I'll wait for an update before creating a PR. Is lucidchart an alternative? It's not really expensive, completely online and I have good experience with it. I'm not sure if everyone who needs / wants to change diagrams is a Mac user (as far as I saw Omnigraffle seems to be an iOS / MacOS only software).

[blag]

5023 includes the new Mistral architecture image.

[armab]

I believe mistral is fully removed from the codebase, diagrams, docs, ci/cd updated, repos archived and this issue could be closed.
Thanks @amanda11 for leading this work and others for help! :tada: