Mongoose: 5.10.16 is badly broken!!!! It should be rescinded/removed!

[chaeron]

We create discriminator-based documents that are highly nested, and we often create plain js objects to use in the model constructors to populate these structures in a new document when the document is first instantiated.

Something changed in 5.10.16 so that many of the fields and structures in the plain JS object we pass to the document constructor no longer end up in the document itself!

5.10.15 works fine.....so something new in .16 is the culprit.

Our code is too complex to create a test scenario easily. Sorry about that.

[dphrag]

Yes, 5.0.16 broke things here.

.Create({ ...

is throwing validation errors suddenly, even though all required fields are being passed.

[maximilianschmid]

We are also experiencing validation errors after upgrading from 5.10.15 to 5.10.16.

Example:
projects document model has a participants subdocument array

Validation error thrown by 5.10.16, but not by 5.10.15:
projects validation failed: participants.1.lastname: [] is required

participants[] is an array os subdocuments.
participants.1.lastname ist not required, but 5.10.16 throws a validation error.

Hope this helps to track the cause.

thx for your great work.

[avivshafir]

I'm also experiencing validation errors after upgrading from 5.10.15 to 5.10.16.

Example:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;


const Person = new Schema({
    items: {
        type: Array,
    },
    email: {
        type: String
    }
});

const PersonModel = mongoose.model('Person', Person);


console.log(new PersonModel({ items: undefined, email: "[email protected]" }))

Run this simple example in the new 5.10.16 version and you will see that email field is not saved
(The field/s after an undefined array field are not saved anymore)

meaning it prints:

{ items: [], _id: <ObjectId> }

[LadnovSasha]

Also query like this
this.model.find({}, { _id: 1, 'email.primary': 1}) does not return email key anymore, which breaks literally everything

[qazwsx9006]

I'm also experiencing validation errors after upgrading from 5.10.15 to 5.10.16.

Example:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;


const Person = new Schema({
    items: {
        type: Array,
    },
    email: {
        type: String
    }
});

const PersonModel = mongoose.model('Person', Person);


console.log(new PersonModel({ items: undefined, email: "[email protected]" }))

Run this simple example in the new 5.10.16 version and you will see that email field is not saved
(The field/s after an undefined array field are not saved anymore)

meaning it prints:

{ items: [], _id: <ObjectId> }

+1

[chaeron]

I believe that 5.016 has such critical issues that it should be rescinded/removed until such at time as these are fixed, so that people don't inadvertently update to use it!!!

[maximilianschmid]

@chaeron just use the previous version. stay calm and kind, open source maintainers do their best for the community.

[chaeron]

I've already reverted a while back. I am calm.....and I stand by my suggestion to revert/rescind the broken version since it is badly broken and may cause a lot of headaches for other users. Trying to be kind to the community in that regard. ;-)

[AbdelrahmanHafez]

9585.js

'use strict';
const mongoose = require('mongoose');
const { Schema } = mongoose;
const assert = require('assert');


const personSchema = new Schema({
  items: { type: Array },
  email: { type: String }
});

const Person = mongoose.model('Person', personSchema);


const person = new Person({ items: undefined, email: '[email protected]' });
assert.equal(person.email, '[email protected]');

Output

AssertionError [ERR_ASSERTION]: undefined == '[email protected]'

[AbdelrahmanHafez]

https://github.com/Automattic/mongoose/commit/ba2b75363bfd8fba080478f79a5f841e2a58f4a7 seems to be the culprit.

[AbdelrahmanHafez]

@LadnovSasha

this.model.find({}, { _id: 1, 'email.primary': 1}) does not return email key anymore, which breaks literally everything

Can you please modify the script below to reproduce the issue you're facing? I was able to fix the issue in https://github.com/Automattic/mongoose/issues/9585#issuecomment-734177784 but there might be more new issues.

9585-2.js

'use strict';
const mongoose = require('mongoose');
const { Schema } = mongoose;
const assert = require('assert');


run().catch(console.error);

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test', {
    useNewUrlParser: true,
    useUnifiedTopology: true
  });

  await mongoose.connection.dropDatabase();

  const userSchema = new Schema({
    friends: [{ type: String }],
    email: {
      secondary: String,
      primary: String
    }
  });

  const User = mongoose.model('User', userSchema);

  await User.create({
    email: {
      secondary: undefined,
      primary: '[email protected]'
    }
  });

  const user = await User.findOne({}, { 'email.primary': 1 });
  assert.equal(user.email.primary, '[email protected]');

  console.log(mongoose.version);
  console.log('All assertions passed.');
}

Output

5.10.16
All assertions passed.

Thanks in advance, Cartman.

[notaphplover]

@AbdelrahmanHafez there may be more bugs, my integration tests fails after upgrading to mongoose 5.10.16. I'm no longer obtaining the _id field of certain embedded documents, I'm trying to create a test scenario

[john-raymon]

Thankful for you all on this Thanksgiving. Is this why the include method isn't working for CoreMongooseArrays?

[LadnovSasha]

@AbdelrahmanHafez sorry, issue actually with create method, here is repro:

```const mongoose = require('mongoose');
const { Schema, Types } = mongoose;
const assert = require('assert');

run().catch(console.error);

async function run() {
await mongoose.connect('mongodb://localhost:27017/test');

await mongoose.connection.dropDatabase();

const productSchema = new Schema({
upc: { type: String },
title: { type: String },
reviewedBy: {
id: { type: Schema.Types.ObjectId, ref: 'core.users' },
name: { type: String },
email: { type: String },
},
contract: {
id: { type: Schema.Types.ObjectId, ref: 'core.contracts', index: true },
detailedProductReview: { type: Boolean },
quote: { type: Number },
specialCategory: { type: Boolean },
},
});

const Product = mongoose.model('core.products', productSchema);

await Product.create({
_id: Types.ObjectId(),
upc: undefined,
reviewedBy: undefined, // issue hidden here. If you comment out this line - assertion passes
title: 'Awesome Steel Shirt',
contract: {
id: '5fc0c23dab77d84e1c0623fc',
},
});

const user = await Product.findOne({}, { 'contract.id': 1 });
assert.strictEqual(user.contract.id.toString(), '5fc0c23dab77d84e1c0623fc');

console.log(mongoose.version);
console.log('All assertions passed.');
}
```

Output

TypeError: Cannot read property 'toString' of undefined

In database:

Thanks for looking into this

[AbdelrahmanHafez]

@LadnovSasha
It seems to be the same bug in https://github.com/Automattic/mongoose/issues/9585#issuecomment-734177784, I ran your test case on the branch with the fix, and the test is passing.

Thanks for the reproduction script.

[AbdelrahmanHafez]

@john-raymon

Can you please modify the script below to demonstrate the issue you're facing?

9585-3.js

'use strict';
const mongoose = require('mongoose');
const { Schema } = mongoose;
const assert = require('assert');


run().catch(console.error);

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test', {
    useNewUrlParser: true,
    useUnifiedTopology: true
  });

  await mongoose.connection.dropDatabase();

  const userSchema = new Schema({
    name: { type: String },
    friends: [String]
  });

  const User = mongoose.model('User', userSchema);

  await User.create({ name: 'Hafez', friends: ['Nada', 'Sam'] });

  const user = await User.findOne();
  assert.ok(user.friends.includes('Nada'));

  console.log(mongoose.version);
  console.log('All assertions passed.');
}

Output

5.10.16
All assertions passed.

[valeriansaliou]

Hello there! We saw this issue while testing this Mongoose version update over at Crisp. Some keys were not being inserted, while others were.

I've tested w/ your patch https://github.com/Automattic/mongoose/compare/master...AbdelrahmanHafez:gh-9585 and I confirm that this fixes the issue.

My quick guess is that the issue that we saw on our end is the exact same one reported here. It was not seen as of 5.10.15. Rolling back to 5.10.15 did the trick for us for now.

[vkarpov15]

My sincerest apologies for this issue. This change was meant to be a minor refactor, and our tests didn't catch this particular bug. We just released v5.10.17 with the fix :+1:

[chaeron]

Thanks for the fix!

Did you update your tests to try and catch such issues in the future? ;-)

[AbdelrahmanHafez]

@chaeron
Yes, we added a test case along with the fix to avoid regression in the future.

[valeriansaliou]

It's all right, thanks for the quick fix. I guess in this case more specific unit tests are always a good thing :) Thanks again!