Mongoengine: connect() called with username and password arguments returns an unauthenticated pymongo.connection.Connection object

[bmbouter]

The mongoengine documentation indicates that the mongoengine.connect() method takes keyword parameters username and password and returns a pymongo.connection.Connection object. While not explicitly stated I have an expectation that the username and password passed into connect() are authenticated immediately and that the pymongo.connection.Connection object returned is authenticated. The unexpected behavior is that it is not authentication! If authentication is required and you attempt to list the collections you'll receive the following exception:

pymongo.errors.OperationFailure: database error: not authorized for query on example_db.system.namespaces

Authentication doesn't occur until mongoengine.connection.get_db() is actually called although the mongoengine.connect() method has everything it needs to attempt authentication. Another downside of this is that the authentication error isn't explicit with the call to connect(). It will fail some time later when get_db() is called. Here is a reproducer snippet:

import mongoengine

name="example_db"
connection_kwargs={'username': 'mongodb_user', 'max_pool_size': 10, 'host': 'localhost', 'password': 'xxxxxxxx', 'port': 27017}

con = mongoengine.connect(name, **connection_kwargs)
con.example_db.collection_names()

You'll notice that this doesn't occur for normal mongoengine use, but this is still a problem. This problem affects users who are transitioning from using PyMongo directly to mongoengine. The first step in that process is to convert the connection so that it is managed using mongoengine instead of PyMongo. As soon as this happens username and password authentication will stop working for any existing code that uses the pymongo.connection.Connection returned from mongoengine.connect().

This problem affects 0.7.10+ all the way up to the latest release and master.

[lafrech]

I just got hit by this one.

This happens in my tests setup. Before creating the Flask app (using flask-mongoengine), I would like to drop the test database.

class ApiTestCase(unittest.TestCase):
    """Parent class of all API test cases"""

    def setUp(self):

        self.connection = me.connect(db='test',
                                     host='localhost',
                                     port=27017,
                                     username='user',
                                     password='pwd')

        # Workaround for 
        # https://github.com/MongoEngine/mongoengine/issues/851
        me.connection.get_db()

        self.connection.drop_database('test')

        app = create_app('testing')

Adding a call to get_db() makes it work.

Maybe I'm doing it wrong in the first place...

Anyway, like @bmbouter, I think the way the docs are formulated lets us expect an authenticated connection object. My colleague and I spent a few hours trying to figure this out.

Should we not consider this a bug?

If not, we may want to at least add a note in the docs about this behavior, but what workaround should we suggest?

[allanlei]

Also experienced this problem. Found another solution by using mongo connection URI

mongoengine.connect(username='user', password='pwd', host='localhost', port=27017, db='test')    # Does not work
mongoengine.connect(host='mongodb://user:pwd@localhost:27017/test')   # Works

I am also using flask-mongoengine and it seems flask-mongoengine parses a mongo URI into parts before passing to mongoengine.connect().

My work around is actually to patch flask-mongoengine.connection._resolve_settings() to pass the host=mongodb://.... to mongoengine.connect()

pymongo==3.3.1
mongoengine==0.10.6
flask-mongoengine==0.8

[lafrech]

@allanlei, flask-mongoengine's connection code is being reworked. Expect some changes in the coming days/weeks. Maybe you won't need your patch anymore.

Or even better, if you have time for this, have a look at this PR and see if it works for you and please provide feedback before merge/release.

[bagerard]

@bmbouter Issue is quite old but did you had a chance to test this?

[bmbouter]

@bagerard I did not.

[lopsided]

I've just hit this one too. It looks like the problem is in connection.py here: https://github.com/MongoEngine/mongoengine/blob/f0fad6df19bee4917fb01fa98924baa6ffeb5031/mongoengine/connection.py#L283 where the username and password are stripped from the connection settings with the comment "we don't care about them at this point". When I change this to leave in the username/password (commenting lines 269-270) then I can run connection.drop_database(TEST_DB_NAME) without this authentication error.

Is there a good reason for dropping the authentication details at this point?

The workaround by @allanlei of connecting with a full db uri with user/pass dets in works well though - thanks!

[bagerard]

@aparrish Could you provide a snippet showing how you were authenticating before switching to using the URI?

[lopsided]

Just going to jump in with mine...

connection = connect(
            DB_NAME,
            host=DB_HOST,  # 127.0.0.1
            port=DB_PORT,  # 27017
            username=DB_USERNAME,
            password=DB_PASSWORD,
            authentication_source='admin'
        )

which becomes:

uri = f'mongodb://{DB_USERNAME}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}?authSource=admin'

[aparrish]

@aparrish Could you provide a snippet showing how you were authenticating before switching to using the URI?

Whoops, I think you meant to tag someone else!

[bagerard]

I digged a bit into this and although connect indeed returns an unauthenticated connection, the authentication to the database is still working fine when playing with MongoEngine's Documents classes. It's indeed a problem but only if you start manipulating what's returned by connect outside of MongoEngine's context.

When using "kwargs" authentication with connect (as opposed to URI) as in conn = connect(db_name, username='myuser', password='mypass', authentication_source='admin'):

• conn will indeed be un-authenticated
• the authentication to the database is happening later on in the get_db method, this is close to what Pymongo expects us to do (they separate the connection/MongoClient creation, from the per-database authentication db.authenticate) https://api.mongodb.com/python/3.2/examples/authentication.html#scram-sha-1-rfc-5802.

So a workaround for your use case would be to call get_db() after calling connect to get the authenticated database.

URI style connection (connect(host=YOUR_URI)) indeed works differently as the authentication is happening immediately on the connect call.

The fact that there is a different behavior is confusing, I agree but somehow this is consistent with pymongo's interface...

[lopsided]

Thanks for digging into this. Calling get_db() indeed does fix the problem for me. I'm still unclear what the advantages of stripping the auth details out of the initial connection could be, but happy to run with it like this if it makes more sense to you.