Agenda: Mongoose support

[bricss]

How about to and support for Mongoose instead of Mongoskin? Because in most project where used Mongoose it would be possible to pass open singleton connection to database.

[lucasmonteverde]

+1

[imns]

+1

[lucasmonteverde]

I implemented it this way

var mongoose = require('mongoose'),
    jobs = mongoose.connection.collection('jobs'),
    agenda = require('agenda')(),
    noop = function() {};

jobs.ensureIndex({
    nextRunAt: 1, 
    lockedAt: 1, 
    name: 1, 
    priority: 1
}, noop);

agenda.mongo(jobs);

I think this should be somewhere in the doc

[mkliu]

@lucasmonteverde mongoskin can be directly switched with mongoose?

[droppedoncaprica]

Not directly, but you can overwrite some of the properties of Agenda to have it call Mongoose's calls.

We're eventually going to have to switch it out with something, with what I'm not 100% sure yet. Mongoose is well-supported, however I'm not sure what advantages we could leverage out of its features.

[linakis]

+1

[lykmapipo]

+1

[mikkotikkanen]

+1

[robotiko]

+1

[nevf]

I've refactored agenda to work directly with the MongoDB Native Driver V2. I personally can't see any benefit from using Mongoskin etc. with agenda and the added dependency for such a small use case isn't warranted. The V2 driver has full support for MongoDB V3. When time permits I'll do a fork etc. In the mean time I'm happy to provide the code.

[SidBala]

+1

[mkliu]

@nevf can you provide the code please?

[nevf]

@mkliu I've put a copy here: http://www.clibu.com/agenda-mongo_nativeV2.zip If you have any problems let me know.

[davewilliamson]

+1

[mathieumg]

:+1: +1

[droppedoncaprica]

@nevf Would you mind if I use your changes to get the native MongoDB driver working with Agenda?

[nevf]

@droppedoncaprica Fine with me. I assume you mean that you'll push my changes into Agenda.
This is a newer version: http://www.clibu.com/public_files/agenda4jun15.zip Please use it.

[Zertz]

@nevf Could you make a fork with your changes rather than a local copy?

[longseespace]

@nevf Please make a fork. Many thanks

[dmastylo]

What's the status on this?

[nevf]

Hi All, I'm afraid that at the moment I way too busy to do anything other than provide the zip (above) that you can download. When I get some spare time I'll happily do a fork.

[liamdon]

Hey all, I've published a quick fork based on @nevf's zip file - happy to accept any PRs on it too.

https://github.com/classdojo/agenda

@Zertz @longseespace

[michelob]

Hello @liamdon ... are you using your fork? Could you do a pull request here?

Also looking at your fork...a) maybe this is likely a major version bump, no, to 1.0.0? As it's substantial I think. Currently your fork is patch bumped to 0.6.35 (with this repo at 0.6.28). and b) an update to README to remove your fork addition info and replace with a small change log indicating what changed from 0.6.35 to 1.0.0 (if the major bump is agreed).

I'm looking at this from the perspective of getting rid of MongoSkin, so I can get rid of the node-mongo-native 1.4 ugly peer dependency.

(I'm on Mongo 3.0.3, and want to upgrade all modules which require Mongo to use node-mongo-native 2.0.+ as I'm occasionally running into this issue: https://jira.mongodb.org/browse/NODE-382)

[jakeorr]

Hey @liamdon, do you plan to PR your fork into this repo? It would be great to remove the dependency on mongoskin.

[Climax777]

Mongoose won't improve performance though.

On Wed, 23 Sep 2015, 22:56 Jake Orr [email protected] wrote:

Hey @liamdon https://github.com/liamdon, do you plan to PR your fork
into this repo? It would be great to remove the dependency on mongoskin.

—
Reply to this email directly or view it on GitHub
https://github.com/rschmukler/agenda/issues/156#issuecomment-142725802.

[nevf]

@Climax777 Correct, but not sure what your point is.

[bricss]

e.g. Mongoose is most powerful ODM tool for MongoDB for this days.

[lafama]

Here is my fix https://github.com/rschmukler/agenda/pull/204. Let me know what you think

[liamdon]

Hey @michelob @jakeorr, we just sent our PR, which swaps out mongoskin for mongodb-native and also addresses some performance concerns with the old queries used.

We have not been able to add the test coverage we would have liked, but we have been using this fork in production for a limited use-case. We're not explicitly using it with Mongoose, but you should be able to pass in a Mongoose instance and have that work too.

[liamdon]

I want to clarify that this is also a breaking change, requiring MongoDB 2.6+ instead of the previous 2.4+

MongoDB 2.6 has been released for almost 18 months now, do you think this is an acceptable change?

[Zertz]

@liamdon IMO this is fine because one of the primary reasons to switch to the native driver is for MongoDB 3+ SCRAM-SHA-1. If someone needs support for 2.4, then it is possible to stick to the current version. However, a bump to 1.0.0 would be needed to safely drop 2.4

[nevf]

@liamdon Thanks for your efforts with this.

[jakeorr]

:+1: :+1:

[ddennis]

Hi
I am not sure i understand how this works.

I am using mongoose and the only thing i can get working is

var agenda = new Agenda({db: {address: myMongoUri }});

I am a lillte unsure if this creates another connection to my MongoDb, besides the one Mongoose have already made? Otherwise this solution is fine,

Using Mongoose connection
This is what i have tried to reuse my mongoose connection.

agenda = new Agenda();
agenda.mongo(mongoose.connection.collection('agendajobs'))

agenda = new Agenda({mongo:mongoose.connection.collection('agendajobs')});

The first two fails right away, the last one here throws in the 'ready' listner:
TypeError: Cannot read property 'topology' of undefined

agenda = new Agenda({mongo:mongoose.connection});

I am using node 0.12.2 - mongoose 4.0.5
Any help would be much appriciated
/Dennis

[mathieumg]

Are you trying to use Agenda within an existing process? I believe Agenda should run on its own and not reuse any connection.

[ddennis]

Yes - i am trying to run it in the existing process where i an already running an express server.
I have had a look through the docs and even the Example Project Structure seems to run in the same thread.

[jakeorr]

@ddennis, Agenda will create its own connection to Mongo if you set it up with a URI like in your first example. I'm reusing an existing connection this way:

var agenda = new Agenda();
agenda.mongo(app.get('mongoose').connection.collection('[collectionName]').conn.db, [collectionName], function (err) {
    ...
});

[ddennis]

@jakeorr Thank you - that worked.

[shengbeiniao]

I found a solution.

mongoose v4.11 and agenda v0.9.1

let mongoose = require('mongoose');
let environment = require('config/environment');
let logger=require('config/logger');
let Agenda=require('agenda');

mongoose.Promise = require('bluebird');

let promise=mongoose.connect(environment.mongodb.uri,environment.mongodb.options);

promise.then(db=>{
  //use some connection with mongoose.
  let agenda = new Agenda({mongo: db});
  agenda.on('ready', function() {
  });
},err=>{
  console.error(err);
  process.exit(-1);
});

//here is mongoose options,It's not important for agenda connect.
 {
    useMongoClient:true
 }

It works well~

[vemundeldegard]

I did it by making a variable for the open connection to MongoDB like this and passing dbConnection like this to new Agenda({mongo: dbConnection}):

import mongoose from "mongoose"

export default async () => {
    mongoose
    .connect(`mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}@${process.env.MONGO_CLUSTER}/${process.env.MONGO_DB}?retryWrites=true`,
      {
        useCreateIndex: true,
        useNewUrlParser: true,
        reconnectTries: 30, // Retry up to 30 times
        reconnectInterval: 500, // Reconnect every 500ms
        poolSize: 10, // Maintain up to 10 socket connections
        bufferMaxEntries: 0 // If not connected, return errors immediately rather than waiting for reconnect
      })
    .then(connection => {
      console.log("🚀 Mongoose Lift off!")
    })
    .catch(err => {
      console.log('MongoDB connection unsuccessful, retry after 5 seconds.')
      setTimeout(mongo, 5000)
    })
}

export const dbConnection = mongoose.connection

[Amery2010]

I found a new way to connect to MongoDB like this:

// db.js
import mongoose from 'mongoose'

export const dbConnect = mongoose.connect(MONGO_URL, options)
export default mongoose

import Agenda from 'agenda'
import mongoose, { dbConnect } from './db'

const agenda = new Agenda()
// When mongoose is connected to MongoDB
dbConnect.then(() => {
  agenda.mongo(mongoose.connection.db, 'agendaJobs')
})

[IRediTOTO]

{ dbConnect }

I did it by making a variable for the open connection to MongoDB like this and passing dbConnection like this to new Agenda({mongo: dbConnection}):

import mongoose from "mongoose"

export default async () => {
    mongoose
    .connect(`mongodb+srv://${process.env.MONGO_USER}:${process.env.MONGO_PASSWORD}@${process.env.MONGO_CLUSTER}/${process.env.MONGO_DB}?retryWrites=true`,
      {
        useCreateIndex: true,
        useNewUrlParser: true,
        reconnectTries: 30, // Retry up to 30 times
        reconnectInterval: 500, // Reconnect every 500ms
        poolSize: 10, // Maintain up to 10 socket connections
        bufferMaxEntries: 0 // If not connected, return errors immediately rather than waiting for reconnect
      })
    .then(connection => {
      console.log("🚀 Mongoose Lift off!")
    })
    .catch(err => {
      console.log('MongoDB connection unsuccessful, retry after 5 seconds.')
      setTimeout(mongo, 5000)
    })
}

export const dbConnection = mongoose.connection

not work for me :c sad

[IRediTOTO]

I found a new way to connect to MongoDB like this:

// db.js
import mongoose from 'mongoose'

export const dbConnect = mongoose.connect(MONGO_URL, options)
export default mongoose

import Agenda from 'agenda'
import mongoose, { dbConnect } from './db'

const agenda = new Agenda()
// When mongoose is connected to MongoDB
dbConnect.then(() => {
  agenda.mongo(mongoose.connection.db, 'agendaJobs')
})

Not work too :( Do I must learn deep inside mongdb instead make it done faster with mongoose.

[Amery2010]

@warcraft14115 You first need to ensure that you have successfully connected to MongoDB.
Maybe you need the following code to check if your MongoDB connection is successful:

mongoose.connect(uri, options).then(
  () => { /** Connection succeeded. ready to use. The `mongoose.connect()` promise resolves to mongoose instance. */ },
  err => { /** Connection failed. handle initial connection error */ }
)

[IRediTOTO]

@Amery2010 Thank you for replie me

I tried but it got error from import :s

import mongoose, { dbConnect } from './db'
^^^^^^

SyntaxError: Cannot use import statement outside a module

const mongoose, { dbConnect } =require ('./mongoosedb')
^^^^^^^^
SyntaxError: Missing initializer in const declaration

[Amery2010]

@warcraft14115 Sorry! I used the ECMAScript Modules import syntax, you can try to use the following CommonJS Modules syntax:

// db.js
var mongoose = require('mongoose')

module.exports = {
  mongoose: mongoose,
  dbConnect: mongoose.connect(MONGO_URL, options)
}

var Agenda = require('agenda')
var db = require('./db')

var agenda = new Agenda()
// When mongoose is connected to MongoDB
db.dbConnect.then(function () {
  agenda.mongo(db.mongoose.connection.db, 'agendaJobs')
})

Maybe this can help you.
Ps: I do n’t know much about CommonJS Modules, there may be some syntax errors

[IRediTOTO]

@Amery2010 Thank you so much for replie. In that time I tried another way :) and this look work well. Hope this will help someone.

const Agenda = require('agenda');
const Users = require('./model/usersModel')
const mongoose = require('mongoose')

const connectionString = 'your url connect';

const agenda = new Agenda({
    db: { address: connectionString, collection: 'agenda', options: { useNewUrlParser: true, useUnifiedTopology: true } },
    processEvery: '5 seconds'
});
mongoose.connect(connectionString , { useNewUrlParser: true, useUnifiedTopology: true }).then((sussess) => {
    agenda.define('add user', async (job, done) => {
        const { name, email, password } = job.attrs.data
        const newUser = new Users({
            _id: new mongoose.Types.ObjectId(),
            name,
            email,
            password
        })
        await newUser.save((err, res) => {
            if (err) throw err;
            console.log(res)
            process.exit(0);
        })

    });
})

[IRediTOTO]

@warcraft14115 Sorry! I used the ECMAScript Modules import syntax, you can try to use the following CommonJS Modules syntax:

// db.js
var mongoose = require('mongoose')

module.exports = {
  mongoose: mongoose,
  dbConnect: mongoose.connect(MONGO_URL, options)
}

var Agenda = require('agenda')
var db = require('./db')

var agenda = new Agenda()
// When mongoose is connected to MongoDB
db.dbConnect.then(function () {
  agenda.mongo(db.mongoose.connection.db, 'agendaJobs')
})

Maybe this can help you.
Ps: I do n’t know much about CommonJS Modules, there may be some syntax errors

it look so short and smarter :s This connect can use everywhere