Telegraf: MongoDB plugin SSL connects but fails to authenticate

[malake]

Relevant telegraf.conf:

[[inputs.mongodb]]
  servers = ["mongodb://host-1.datacenter.project.domain.com:27017,host-2.datacenter.project.domain.com:27017,host-3.datacenter.project.domain.com:27017/replicaSet=my-replSet&ssl=true&authMechanism=MONGODB-X509&authSource=$external"]

  ## When true, collect per database stats
  gather_perdb_stats = true

  ## Optional TLS Config
  ssl_ca = "/etc/telegraf/ca.pem"
  ssl_cert = "/etc/telegraf/cert.pem"
  ssl_key = "/etc/telegraf/key.pem"

  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

System info:

Telegraf Version: Telegraf 1.10.3 (git: HEAD 294bb666)
MongoDB Version: 4.0.9
OS: Ubuntu 16.06 x86_64
SSL

Steps to reproduce:

1. User is created in $external db for external x509 authentication
2. Start telegraf service with above configuration

Expected behavior:

The plugin should feed the output source (in my case influxdb) with metrics relating to the databases.

Actual behavior:

Error generated in log.
I'm able to log into the database using the same certificates provided to telegraf remotely with the mongo CLI and I'm able to successfully run the db.serverStatus() command

Additional info:

Telegraf Log

2019-05-02T16:40:30Z E! [inputs.mongodb]: Error in plugin: command serverStatus requires authentication

MongoDB auth info

{
    "_id" : "$external.CN=telegraf-monitor,OU=dev-mongo,O=Acme International Corp.,L= Arrakeen,ST= Arrakis,C=DN",
    "user" : "CN=telegraf-monitor,OU=dev-mongo,O=Acme International Corp.,L= Arrakeen,ST= Arrakis,C=DN",
    "db" : "$external",
    "roles" : [
        {
            "role" : "clusterMonitor",
            "db" : "admin"
        }
    ],
    "mechanisms" : [
        "external"
    ]
}

[malake]

I double checked my work, but i've obfuscated some details in certificates and urls, but they contain all of the same relative characters and components. If there are any inconsistencies in something like spelling, its 100% being unable to type after working on this problem for three days.

[malake]

I've also granted permissions to this user that would be a terrible idea to allow and there were no changes.

[danielnelson]

Does it work using the mongo command? Try to login an run these commands:

> use admin
> db.serverStatus()

[malake]

Yes, sir. It works as I’d expect it to. I think it’s something about how telegraf is using the connection with the certificates that may be the issue?

On May 2, 2019, at 3:08 PM, Daniel Nelson [e-mail] wrote:

Does it work using the mongo command? Try to login an run these commands:

use admin
db.serverStatus()
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[danielnelson]

Try adding a ? to the connection string to split out the options, I think this is required. If its still not working then can you add the arguments you are using to connect with mongo:

mongodb://host-1.datacenter.project.domain.com:27017,host-2.datacenter.project.domain.com:27017,host-3.datacenter.project.domain.com:27017?replicaSet=my-replSet&ssl=true&authMechanism=MONGODB-X509&authSource=$external

[malake]

I've successfully used the following mongo command to connect remotely to the database:

mongo --host "mongodb://host-1.datacenter.project.domain.com:27017,host-2.datacenter.project.domain.com:27017,host-3.datacenter.project.domain.com:27017/?ssl=true&authSource='$external'&authMechanism=MONGODB-X509" --sslPEMKeyFile ~/certs/client_path/telegraf-monitor.pem  --sslCAFile ~/certs/ca_path/myCA.pem -ssl

I tried without the -ssl command but it complained about it being required when sslCAFile and sslPEMKeyFile is in use. okay, sure. no problem.

I used the same value in the servers= ["..."] as i did in the above --host < ... >

also...
gather_perdb_stats = true
and the tls_ca, tls_cert, and tls_key all set and accessible. I know the certs are being read because if I move them, rename them, change the permissions, the error changes from "requires authentication" to "permission denied" or "no such file"

A connection is being made to the database as the user in the certificate but it's not being authenticated.

[malake]

I've seen similar activity in a node.js application connecting to the same mongo replica set. basically what i believe was happening is the node application was establishing a connection to the db and immediately moving on without waiting for the authentication piece to come back. so when the application then moved on to query the db, it would return the error not authorized. it was resolved by modifying the database connection to mongo to be async

[malake]

Went back and checked the mongo.conf file to see if there was anything in there that might cause headaches. I stumbled across the net.serviceExecutor=adaptive . (https://docs.mongodb.com/manual/reference/configuration-options/#net.serviceExecutor)

I attempted to update that to synchronous with no change in the telegraf behavior. I flipped it back to adaptive since that's what it was previously.

[danielnelson]

I don't think we have ever tested this functionality with MongoDB 4, we'll try to duplicate and see if there is a bug and how we can improve the documentation.

We might need to move to a newer client library, in particular globalsign/mgo: https://github.com/go-mgo/mgo/blob/v2-unstable/README.md

[malake]

Let me know if there is anything i can do to help!

[Mangosteen-Yang]

I meet the same error.

[danielnelson]

I believe this is a bug in go-mgo/mgo, we should test against the latest globalsign/mgo and long term we should look into moving to mongodb/mongo-go-driver.

[mengyu-dev]

@danielnelson @malake Hello, authentication x509 works in our production environment! I had the same problem as you before. After reading plugin and mgo source codes, I found the mgo driver supports the authentication x509. But it does not extract the username from the certificat like the official driver, we have pass the username explicitly in the mongo uri. If we configure as below, the plugin works:
servers = ["mongodb://CN=telegraf-monitor,OU=dev-mongo,O=Acme International%20Corp.,L=Arrakeen,ST=Arrakis,C=DN:[email protected]:27017?authMechanism=MONGODB-X509&authSource=$external
Attention, the space must be replaced by %20 and the password in the uri is just a placeholder for uri syntax. The reason why the configuration must be like this, because the telegrapf mongo plugin ignores the options in uri if user is not present. I think this is a bug, we should pass the the whole raw mongo uri to mongo driver. And the mgo driver chooses the auth mechanism by the option!

[danielnelson]

Thanks for researching this issue, could you open an issue with https://github.com/globalsign/mgo and add a link to it here?