Loopback-next: Migrating MySQL and Mongo on the same project causes a crash

[khlilturki97]

I followed this tutorial and everything worked fine. I managed to work with relations on MySQL db.

In my use case, i had to add another datasource based on Mongo and i had a bug while migrating

Unable to migrate models not associated with this data source

i had to translate it from french, the original text is

Impossible de migrer les modèles non associés à cette source de données

i digged a little in RepositoryMixin class, and found out that the problem came from migrateSchema method

i overrided the method (copy past x) ) and changed

for (const b of dsBindings) {
  const ds = await this.get(b.key);
  if (operation in ds && typeof ds[operation] === 'function') {
    debug('Migrating dataSource %s', b.key);
    await ds[operation](options.models);
  }
  else {
    debug('Skipping migration of dataSource %s', b.key);
  }
}

to

for (const b of dsBindings) {
      const ds = await this.get(b.key);
      // console.log(ds);
      // @ts-ignore
      if (operation in ds && typeof ds[operation] === 'function') {
        console.log('Migrating dataSource %s', b.key);
        // @ts-ignore
        if (!options?.models || options?.models.length === 0) {
          // @ts-ignore
          await ds[operation](options.models);
        } else {
          const models = [];
          for (const model of options?.models) {
            // @ts-ignore
            if (Object.keys(ds.definitions).includes(model)) {
              models.push(model);
            }
          }
          // @ts-ignore
          await ds[operation](models);
        }
      } else {
        // debug('Skipping migration of dataSource %s', b.key);
      }
    }

i hope this bug will be fixed soon

[deepakrkris]

@khlilturki97 how did the following help ?
we are checking for !options?.models in the if condition and the proceeding to iterate the same options?.models in the else block ?

if (!options?.models || options?.models.length === 0) {
          // @ts-ignore
          await ds[operation](options.models);
        } else {
          const models = [];
          for (const model of options?.models) {
            // @ts-ignore
            if (Object.keys(ds.definitions).includes(model)) {
              models.push(model);
            }
          }
          // @ts-ignore
          await ds[operation](models);
        }

Can you provide a sample project to recreate the issue you are facing ?

[khlilturki97]

@deepakrkris the migration script has to know if a model is part of a datasource or not

If youre not passing an array in migrate.ts, it will execute normally, but if you pass one, according to the actual implementation of the method, it will try to associate all the models in the array to each datasource which leads to a bug

I cant give access to the repo, but if it is necessary i will create a new one

[khlilturki97]

@deepakrkris here's a repository containing the issue

What i did

• i followed Todo Tutorial (i didn't integrate geocoder) and Todo List tutorial with the "Running on relational databases step" (i used MySQL instead of PostgreSQL )

until now everything works fine

• i added a new datasource (MongoDB)
• i added a new model (called mongo) with repository and controller

until now project can be compiled

when i run migrate script, here's the output

node ./dist/migrate

Migrating schemas (alter existing schema)
(node:26080) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUn
ifiedTopology: true } to the MongoClient constructor.
Cannot migrate database schema Error: Unable to migrate models not associated with this data source : TodoList Todo TodoListImage 
<--------------------------
the above message was translated from french am not sure if it's the correct translation or not
-------------------------->
    at D:\projects\loopback\todo-list\node_modules\loopback-datasource-juggler\lib\datasource.js:1146:12
    at process._tickCallback (internal/process/next_tick.js:61:11)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] migrate: `node ./dist/migrate`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] migrate script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\khlil\AppData\Roaming\npm-cache\_logs\2020-05-20T12_02_18_249Z-debug.log

Process finished with exit code 1

when i remove this line from migrate file models: ['TodoList', 'Todo', 'TodoListImage'], migration script runs but does not add foreign keys index

what i understood that Mongo connector is trying to migrate model that are not related to it, so if i override as i did in the post above, it works

i left the override commented in application.ts so you can test it

Steps to reproduce

• clone this project
• adapt db.datasouce config to yours
• create a database in MySQL with the same name in your config
• run migrate script (should fail)
• uncomment migrateSchema function in application.ts
• rerun migrate script (should pass)

Solutions

i suggest either reimplement the migrateSchema function according to what i did

Other bug

i don't know if i should open another issue or not, but when you run npm run migrate -- --rebuild when the database is not empty, the script crashes

i added

if (options.existingSchema === 'drop') {
      // @ts-ignore
      options.models = options?.models.reverse();
    }

so that the rebuild script drops the last created tables first (starting from the last model migrated)

[dhmlau]

@khlilturki97, I cloned your repo and understood your problem. Would you be interested in submitting a PR?

I'm not familiar with the code, one concern I have with your fix mentioned in the original description is that if there are multiple repository classes associated with the same model but different datasource, this case won't be taken care of. Ideally, we want to check with all repository, and migrate the datasource with the bind model instead of migrating all the models for a given datasource (which seems to be what the current code behaves).

[khlilturki97]

@dhmlau yes sure,

But can you show me how to do it ?

[dhmlau]

@khlilturki97, sorry about the late reply. We have a docs page with detailed instructions in submitting a PR: https://loopback.io/doc/en/lb4/submitting_a_pr.html. Hope it helps! Thanks.

[khlilturki97]

@dhmlau thanks for the link

i submitted a pull request #5769

[stale[bot]]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.