Manageiq: Pluggable Providers TODO

Created on 21 Apr 2017  路  16Comments  路  Source: ManageIQ/manageiq

This is a list of all open and accomplished tasks to split out providers from the main ManageIQ core repo.

_please add things you know of as comments and I'll update the list_

Extract all providers

  • [x] amazon
  • [x] azure
  • [x] vmware
  • [x] openstack
  • [x] ovirt / redhat
  • [x] hawkular
  • [x] openshift
  • [x] kubernetes
  • [x] ansible_tower
  • [x] embedded_ansible
  • [x] foreman
  • [x] google
  • [x] microsoft
  • [x] nuage

Config / Settings

Seeding

Pluggability

Couplings

Specs

UI

to be discussed / concretized

cc @Fryguy @blomquisg @bdunne @gmcculloug @gtanzillo @chessbyte
@miq-bot add_label pluggable providers

pinned pluggabilitproviders
Source
picture durandom
👍2

All 16 comments

@durandom - @juliancheal and I extracted a bunch of providers recently (Ansible Tower, Google, Foreman, Microsoft SCVMM). Please update ch

chessbyte picture chessbyte on 4 Jun 2017
👍1

@durandom et. al.

I think something that should also be considered and added to the list is the workers, and having the definitions of those living in the providers themselves versus being part of a list in MiqServer::WorkerManagement::Monitor::ClassNames (I have moved the real definition around a bit, but basically the constants defined there is what I am referring to).

This would be a bit tricky to implement, and definitely something I am still pondering on myself. This might also be related to the "base MIQ models" effort that @jrafanie and others are looking into, and would affect how we define plugins as well.

NickLaMuro picture NickLaMuro on 12 Jun 2017
👍1

I feel like providers should be able to register and enable themselves (as two steps), what models/worker classes/etc. they provide, and insert rails boot hooks if they're an engine. It's still important for them to not load all the things since we can't sustain all of our provider code and libraries being loaded in the same process. The rails engine code must not load all the things. It should only do minimal work and delay loading any gem code until it's actually used.

jrafanie picture jrafanie on 12 Jun 2017

.subclasses/descendants can get us all of the leaf workers, and so we can probably change ClassNames to do something like

MiqEmsRefreshWorker.subclasses + 
MiqEmsMetricsCollectorWorker.subclasses +
...
Fryguy picture Fryguy on 12 Jun 2017

@Fryguy That works if you want to load the world first before loading miq_worker_types, but right now in run_single_worker.rb, I am loading it by itself. This allows the command line to be snappy when doing -l, as well as not requiring everything from every provider just to boot up a worker.

Doing what you suggests means we need to load the world to determine if this is a valid worker type.

NickLaMuro picture NickLaMuro on 12 Jun 2017

The monitor has to monitor everything, so not sure why it loading the world is an issue. I would argue that if we extract worker management from MiqServer into a standalone object, then only the evmserver should be loading that class, and then accessing MiqServer won't load the world via the worker management.

Fryguy picture Fryguy on 13 Jun 2017

The monitor has to monitor everything...

That doesn't mean it has to load everything to monitor it (make sure it is running, make sure it hasn't stalled, etc.).

It needs to know about what it is at most, not what it is doing and why it is doing it. We are just re-organizing the monolith if our expectation is that it is just going to continue to load everything.

NickLaMuro picture NickLaMuro on 13 Jun 2017

@miq-bot assign juliancheal

juliancheal picture juliancheal on 8 Nov 2017

We also probably want custom Loggers to live in their own repo, which might be necessary in certain cases (see #17228 )

Fryguy picture Fryguy on 3 Apr 2018
👍1

Pluggable configuration validators should be allowed since we are allowing plugins to bring their own settings. (see https://github.com/ManageIQ/manageiq/pull/17168#issuecomment-378370350 )

Fryguy picture Fryguy on 9 Apr 2018
👍1

This issue has been automatically marked as stale because it has not been updated for at least 6 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions!

miq-bot picture miq-bot on 12 Oct 2018

@juliancheal is this still a valid issue. If not can you close.
If there's no update by next week, I'll be closing this issue.

JPrause picture JPrause on 23 Jan 2019

Closing issue. If you feel the issue needs to remain open, please let me know and it will be reopened.
@miq-bot close_issue

JPrause picture JPrause on 29 Jan 2019

https://github.com/ManageIQ/manageiq/pull/16162 should be a good first pass on getting hard-coded model names from providers out of core/ui

agrare picture agrare on 10 Oct 2019

@agrare / @juliancheal So I gave this a bit more thought since last week, and here are a few items I think could be added to this list:

  • Gemify PostgresAdmin
  • Bring back mounting code in to "core"
  • A "core gem"
  • Third-party plugin support (without bundler?)

The first two are items related to the database backup work I did last summer, and specifically with the mounting code, there are pieces for the s3 and swift portions that really should live in the provider plugins for them specifically. Carboni specifically mentioned wanting to do the PostgresAdmin bit, and I tend to agree with him there. I think we would want to make it a bit more generic than it is currently, but that should be doable.

The "core gem" idea is something @Fryguy had two summers ago, but I think that will help decouple things better. Whether it is "one gem", or a few (such as manageiq-base-models and manageiq-core) is something I haven't considered much yet, but was having multiple was brought up at options when we discussed it last.

The last one might require the core gem as a per-requisite (or a manageiq-base-provider of some sort), but the idea would be that you can put a self-contained gem in a directory somewhere to add extra providers to MIQ. While it might end up having the "npm problem" (multiple versions of the same library) by not using bundler, I think having a plugin bring all of it's dependencies (vendored and probably namespaced) avoids the core needing to provide it for them. If we go this route, I think we would need to encourage limiting dependencies to only what is needed for the provider (so aws-sdk for amazon for example).

Obviously dependencies like the core gem wouldn't need to be shipped with the providers gems, and we can assume the , and if we go more the route described here, this might be less of an issue when it comes to memory bloat. One thing we could also do for plugin compatibility is do something like Rails does for migrations:

https://guides.rubyonrails.org/v5.2/active_record_migrations.html#migration-overview

And have classes with a version in them. That way developers can write to a model spec in core, and we can deprecate/remove those versions after a few newer versions have been released.

Anyway, sorry for the :book:, but I can possibly try experimenting with the last two and comment on my findings in @juliancheal 's https://github.com/ManageIQ/manageiq/pull/16162 PR.

NickLaMuro picture NickLaMuro on 15 Oct 2019

As a lot of progress has already been made and we are starting another round of pluggability, I am going to close this issue and start a new one that represents the current state of pluggability a little better, please move any further discussion to https://github.com/ManageIQ/manageiq/issues/19440

agrare picture agrare on 28 Oct 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jawatts picture jawatts  路  8Comments
zhitongLBN picture zhitongLBN  路  6Comments
jithju31 picture jithju31  路  3Comments
zigaSRC picture zigaSRC  路  8Comments
meng-tian picture meng-tian  路  4Comments