Keystone-classic: Mongo Session Store doesn't work with "add keystone to an existing Express app"

[epinapala]

Expected behavior

I am currently working on integrating keystoneJS with our existing express app. I followed the instructions at https://github.com/keystonejs/keystone/wiki/How-to-Add-Keystone-to-an-Already-Existing-Express-App. I was able to start keystone, however I am not able to use mongo session store.

Actual/Current behavior

When I use, 'session store' : 'mongo', in memory session is being used. I don't even see app_sessions collection being created in the mongo db.

Upon inspecting keystoneJS code base, issue seems to be because of the way keystone/lib/core/initExpressApp.js is written.

module.exports = function initExpressApp (customApp) {
       //When i use keystone.set('app', app);,
       //below if causes rest of the steps to be skipped.
    if (this.app) return this; 
    this.initDatabaseConfig();
    this.initExpressSession(this.mongoose);
    if (customApp) {
        this.app = customApp;
        require('../../server/createApp')(this);
    } else {
        this.app = require('../../server/createApp')(this);
    }
    return this;
};

When you look at if (this.app) return this;, When using keystone.set('app', app); this IF causes rest of the steps to be skipped, which seem to be needed for the creation of mongo based session store. Is this done intentionally for a reason?

Steps to reproduce the actual/current behavior

below is how my keystone config looks like

keystone.init({
  'name': 'rajesh',
  'brand': 'rajesh',
  'less': 'public',
  'static': 'public',
  'favicon': 'public/favicon.ico',
  'views': 'templates/views',
  'view engine': 'pug',
  'session store': 'mongo',
  'auto update': true,
  'session': true,
  'auth': true,
  'user model': 'User',
  'wysiwyg additional plugins': 'paste',
  'wysiwyg additional options': {
    paste_webkit_styles: "color font-size",
    'cookie secret': cookieSecret
  }
});

// Let keystone know where your models are defined. Here we have it at the `/models`
keystone.import('models');

// Serve your static assets
app.use(serve('./public'));

// This is where your normal routes and files are handled
app.get('/', function(req, res, next) {
  res.send('hello world');
});
keystone.app = app;
keystone.start();

Environment

| Software | Version
| ---------------- | -------
| Keystone | "4.0.0-beta.8"
| Node | v8.9.4

[gabrielgiacomini]

+1

[epinapala]

@JedWatson Can you comment on this if possible, Thanks in advance!

[gabrielgiacomini]

@epinapala, I managed to get it to work by calling keystone.initExpressSession() with my own mongoose instance. Hope it helps.

var mongoose = require('mongoose'); //<---- my own mongoose

keystone.init({
  'mongoose': mongoose, //<---- my own mongoose
  'session': true,
  'session store':'mongo', //<---- still had to add the config here
  'auth': true,
  ...
});

keystone.import('models');
keystone.initDatabaseConfig();
keystone.initExpressSession(mongoose); //<---- this is what made the difference

keystone.set('nav',{
  ...
});

//SETUP EXPRESS
var app = express();
app.set('view engine', 'jade');
app.set('views',global.__base_path+'/templates/views');
app.use(compression());

if(!headless){
  var keystoneRouter = express.Router();
  //keystoneRouter.use(favicon(keystone.getPath('favicon')));
  keystoneRouter.use(keystone.Admin.Server.createStaticRouter(keystone));
  keystoneRouter.use(keystone.get('session options').cookieParser);
  keystoneRouter.use(keystone.expressSession);
  keystoneRouter.use(keystone.session.persist);
  keystoneRouter.use(require('connect-flash')());
  keystoneRouter.use( keystone.Admin.Server.createDynamicRouter(keystone));
  app.use('/keystone',keystoneRouter);
}

...

//another routers for the api and website here, with their own middlewares

logger.info('Connecting to database...');
keystone.openDatabaseConnection(function () {
  logger.info('Starting server...');

  var server = app.listen(8000, function () {
    console.log('-------------------------------');
    console.log('Express server ready on port %d', server.address().port);
    console.log('-------------------------------');
  });
});

[epinapala]

@gabrielgiacomini Thanks for the reply. I was able to make this to work as well by doing a similar setup.
keystone.initExpressApp() = //my custom initExpressApp.js

However, I am not sure if that is a recommended way to do it. If so why was the initExpressApp.js implemented differently? I am hoping this does not have any side affects.

[gabrielgiacomini]

@epinapala, I based by initialization script in this piece of code I found on the keystone test project: advanced.js

I found easier to get full control by not having to use keystone.start(). This way I achieved a clear separation between express routers for keystone and for my application, so I'm pretty sure my routers run only the middlewares I explicitly added to my script.

[epinapala]

@gabrielgiacomini sounds good. Now that I see https://github.com/keystonejs/keystone-test-project/blob/master/advanced.js, I am more comfortable making the same changes. Solves the issue, but would be nice to know if there was a reason why this wasn't supported by the framework. May be @JedWatson would know, Anyways Thanks a lot for sharing in detail.

This way I achieved a clear separation between express routers for keystone and for my application, I agree.

[airandfingers]

Thanks @epinapala and @gabrielgiacomini for sharing how you got this to work! gabriel's code in https://github.com/keystonejs/keystone/issues/4610#issuecomment-379894721 is working for me.

One question: how are you using multer, if at all?

The "How to add keystone to an existing Express app" instructions include multer as a global middleware and say:

Make sure you're using [email protected]. The latest's multer's API changes. If you find yourself getting a TypeError app.use() middleware, this is the problem.

However, the multer docs specifically say:

WARNING: Make sure that you always handle the files that a user uploads. Never add multer as a global middleware since a malicious user could upload files to a route that you didn't anticipate. Only use this function on routes where you are handling the uploaded files.

My concern is that using multer the way the docs say introduces security vulnerabilities.. have you guys looked at this?

[gabrielgiacomini]

@airandfingers

I did manage to use multer. It was tricky though.

Most configurations would pass the file to the requet as a binary variable or stream or something (i don't know much about that), and I wanted to have the upload in a temporary file to manipulate it using the filesystem in a simple way. To get this done working setup was eventually:

app.use(multer({
    extended:true,
    includeEmptyFields: true,
    limit: '1024MB',
    dest: keystone.expandPath(global.__tmp_path), //your temporary files path
  }).single('file'));

In the request you might want to look for these variables:

    logger.info('file received',req.file);
    if(!req.file)                      return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!_.isObject(req.file))          return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.fieldname)            return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.originalname)         return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.filename)                 return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.encoding)             return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.mimetype)             return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.path)                 return callback('error_intranetUploadVideo_fileNotFound',null);
    if(!req.file.size)                 return callback('error_intranetUploadVideo_fileNotFound',null);
    if(req.file.size < 1024*1024*0.1)    return callback('error_intranetUploadVideo_fileTooSmall',null);
    if(req.file.size > 1024*1024*1024)  return callback('error_intranetUploadVideo_fileTooBig',null);

Concerning multer docs' warning, I didn't add the configuration only to specific requests because my applications have a specific node process to handle uploads (separated from all other api requests).

But you can also use different routers for different parts of your express server, which you want to isolate in terms of middlewares. Example:

var apiRouter = express.Router();
apiRouter.use(bodyParser.json({ limit: '1MB' }));
apiRouter.use(bodyParser.urlencoded({ extended:true, limit: '1MB' }));
apiRouter.use(require(global.__middlewares_path+'/shutdown'));
apiRouter.use(require(global.__middlewares_path+'/cors'));
apiRouter.use(require(global.__middlewares_path+'/responses'));
apiRouter.use(require(global.__middlewares_path+'/tracking'));
apiRouter.post(...) //requests in general
apiRouter.get(...) //requests in general
app.use('/',apiRouter);

You can use an exclusive process or router to run the keystone admin too, because you want to be sure there won't be any unneeded middlewares running in every request. So when the process is started with the option to be a keystone admin, i add:

if(!headless){
  var keystoneRouter = express.Router();
  //keystoneRouter.use(favicon(keystone.getPath('favicon')));
  keystoneRouter.use(keystone.Admin.Server.createStaticRouter(keystone));
  keystoneRouter.use(keystone.get('session options').cookieParser);
  keystoneRouter.use(keystone.expressSession);
  keystoneRouter.use(keystone.session.persist);
  keystoneRouter.use(require('connect-flash')());
  keystoneRouter.use( keystone.Admin.Server.createDynamicRouter(keystone));
  app.use('/keystone',keystoneRouter);
}

[airandfingers]

Thanks @gabrielgiacomini for the quick and thorough response!

I'm not yet sure how I want to use multer (especially given my thoughts below), but I'll keep your approach in mind and refer back to it as needed.

I noticed and appreciated the way you separate Keystone into its own router, as that allowed me to avoid including Keystone's middleware into my main Express app.

Since I only need Keystone's admin GUI - and only locally - perhaps I'll adapt your approach and run Keystone separately, avoiding any potential security issues and bloat by not including Keystone or multer in my app at all.

Thanks again!