Reaction: [EPIC] Moving the API off Meteor

[aldeed]

We have been slowly removing Meteor dependencies from Reaction API server code. We are now close enough to being done on the server side, that we can create this Epic to track the remaining work.

To be clear, the goal here is to get the _API server_ running as a non-Meteor Node app. This means all GraphQL code, all events, all background jobs, all relevant startup code. It does _not_ include necessarily all Meteor methods or publications, nor does it include operator UI client code. There will remain a Meteor app to run those things for some time yet, but it will be needed only for running the operator UI. The operator UI will eventually not need Meteor, but there is much work to be done there and no timeline yet.

Here is a preliminary list of the work that this Epic entails:

• [x] Move all SimpleSchema indexes to use collectionIndex function, and remove aldeed:schema-index Meteor package
  
  
  
  • Issue #5080
  
  
• [x] Add a standard way to register collections and their indexes
  
  
  
  • Issue #4268
  
  
  • PR #5196
  
  
• [x] Add SimpleSchemas on context
  
  
  
  • Issue #5335
  
  
• [x] De-Meteorize emailing
  
  
  
  • PR #4998
  
  
• [x] De-Meteorize the job-collection job queue
  
  
  
  • Issue #5288
  
  
• [x] Review core startup code
  
  
  
  • [x] PR #5212
  
  
• [x] Non-meteor translations loading
  
  
  
  • [x] PR #5514
  
  
• [x] Deal with template-type assets as in core/server/hooks.js
  
  
  
  • [x] PR #5430
  
  
• [x] Replace setAppVersion on startup with getting the data from the new systemInfo GraphQL
  
  
  
  • [x] PR #5429
  
  
• [x] Create a registerPackage equivalent with no Meteor deps
• [x] Update all plugins to use the non-Meteor registerPackage
• [x] Move calling of registerPackageHandlers into NodeApp, or part of registerPackage replacement.
• [x] Remove all use of context.callMeteorMethod and remove it
  
  
  
  • [x] Issue #5336
  
  
  • [x] Issue #5337 - PR #5520
  
  
  • [x] Issue #5338 - PR #5492
  
  
  • [x] Issue #5339 - PR #5493
  
  
  • [x] Issue #5340 - PR: #5488
  
  
  • [x] Issue #5341 - PR: #5484
  
  
  • [x] Removal PR: #5524
  
  
• [x] Remove all use of getGraphQLContextInMeteorMethod and remove it ( #5529 )
  
  
  
  • [x] inviteShopMember.js - #5520
  
  
  • [x] inviteShopOwner.js - #5553
  
  
  • [x] sendResetPasswordEmail.js - #5558
  
  
  • [x] sendVerificationEmail.js - TODO
  
  
  • [x] sendWelcomeEmail.js - TODO
  
  
  • [x] catalog.js - #5562
  
  
  • [x] publishProducts.js - PR #5541
  
  
  • [x] cart.js - #5562
  
  
  • [x] getLocale.js - Removed by PR #5514
  
  
  • [x] index.js - #5559
  
  
  • [x] verifySettings.js - #5531
  
  
  • [x] generate-sitemaps.js - #5530
  
  
• [x] Move ReactionError to real npm package
  
  
  
  • [x] Issue #4493
  
  
• [x] Demeteorize file upload
  
  
  
  • [x] PR #5515
  
  
  • [x] Second PR needed to port files methods and pubs to GraphQL queries and mutations
  
  
• [x] Move to a non-Meteor migrations solution
  
  
  
  • Issue #4416
  
  
• [x] Create a high-level place for API plugins, separate from Meteor plugins that are used by operator UI.
• [x] One by one, for each plugin:
  
  
  
  • Confirm full demeteorization of API files. Remove any remaining Meteor deps.
  
  
  • Remove all imports from outside the plugin. Consolidate util functions and use Babel to map those imports as if they are a package.
  
  
  • Except for methods and pubs that are still used by the operator UI, move everything into the separate API plugin/folder
  
  

This will be complete when the pure Node server, currently runnable for testing with npm run devserver, correctly handles all GraphQL requests, generates app events, creates and works background jobs, and sends emails.

Note: Long term, there will be more modular and scalable solutions for background jobs and emails, but there is no need to lump those larger projects in with this one. Porting the existing core plugins should be straightforward for now.

[zenweasel]

Epic in every sense of the word

[pmn4]

This is fantastic, @aldeed. Godspeed!

[dhonig]

This is very exciting work. Re: JobCollection library....Is there any interest in moving this to something like Agenda or Bull? https://github.com/agenda/agenda I've developed a love and hate relationship with job-collection over the course of a few projects.....

[aldeed]

@dhonig Long term it will definitely move to a different tool for queuing and scheduled jobs. It will most likely be a separate microservice, so essentially you could use any library, tool, or third-party provider you want as long as you tell Reaction how to add, monitor, and work jobs.

But that is long term and isn't on the near-term roadmap, so for this epic the goal is really just to keep the existing system working without any changes except removing Meteor dependencies.

[janus-reith]

Has Account management been left out here intentionally?

[aldeed]

@janus-reith Account management (identity provider features) will remain on Meteor, as will the Reaction Admin UI for now. Accounts are still heavily dependent on Meteor accounts packages. The plan is that eventually IDP will split to its own Meteor app, and then will be abstracted so that you can use something like Keycloak or third-party like auth0.

[janus-reith]

Ok, makes sense.
I was wondering initially how the API would deal with account data and permissions then, but realized that this is is how it works already using the the auth token.
Maybe this could allow for a simple transition: https://github.com/accounts-js/accounts

[aldeed]

Having now a release-3.0.0 branch, I'm closing this epic as done. There is some remaining work fixing bugs and getting everything working in that branch, which is now being tracked in a public GH project: https://github.com/orgs/reactioncommerce/projects/4

[vajda-media]

Sorry to ask here after closed this thread. Just wanted to know, what was the benefit of removing one of the best JS Frameworks and go on simple nodeJS? Was also the possiblity kept in focus to take ReactJS + Meteor combination?

[focusaurus]

@vajda-media There were many reasons but one key pain point was crushingly slow start/restart time for the meteor server making back end development extremely hindered. Having a GraphQL API was also a key component of our "headless" product direction and meteor being UI/back-end integrated was contrary to that.