Mongo: Cannot configure replica sets with entrypoint-initdb

[Gaff]

I'm trying to create simple replicaset enabled images using these docker images. @yosifkit had a suggestion in another thread that this can be done by calling rs.initiate() in a /docker-entrypoint-initdb.d/ script.

However if I do this I get the following:

2019-03-26T12:30:25.889+0000 I COMMAND  [conn2] initiate : no configuration specified. Using a default configuration for the set
2019-03-26T12:30:25.889+0000 I COMMAND  [conn2] created this configuration for initiation : { _id: "rs0", version: 1, members: [ { _id: 0, host: "127.0.0.1:27017" } ] }

The problem here is that since the init script binds only to localhost, the replicaset has the wrong hostname.

If I call rs.initiate() after initialization phase I get the following:

2019-03-26T12:32:16.792+0000 I COMMAND  [conn1] initiate : no configuration specified. Using a default configuration for the set
2019-03-26T12:32:16.793+0000 I COMMAND  [conn1] created this configuration for initiation : { _id: "rs0", version: 1, members: [ { _id: 0, host: "mongo:27017" } ] }

This time with the correct hostname.

Is there some way we can resolve this paradox? Either by running a script after the real startup? Or by binding to the proper interfaces during initialisation? Or by forcing the mongo server to accept a replicaset config even if it cannot resolve itself?

Thanks!

[wglambert]

https://github.com/docker-library/mongo/issues/246#issuecomment-382082775

you will need something more complex for the other nodes. But no, my hurried Dockerfile will not actually work since the host field in the replica config will be set to 127.0.0.1:27017.

https://github.com/docker-library/mongo/issues/246#issuecomment-387520572

Adding automation for setting up a replica set is not something we want to add to the image since it requires an external service like consul in order to reliably coordinate which is the first and where to join.
. . .

[Gaff]

Hmm - I did miss the "my hurried dockerfile will not actually work" point sorry!

Still I think the idea has merit; but-for the squelching of the --bind-ip arguments this would work just fine, and would have some value for single instance deployment / unit test / many other basic cases. [I realise the whole point of force binding localhost is to ensure nobody can connect until the startup scripts are complete. Not sure how to reconcile that.]

If there's a direction you think would be acceptable and could be made to work I'd be willing to have a go at implementing it.

[tianon]

According to https://docs.mongodb.com/manual/reference/method/rs.initiate/,
"rs.initiate" can accept a "configuration" argument -- is it possible
to provide the hosts list as part of that block, or does "rs.initiate"
choke on that?

(Worst case you could "rs.reconfig" after your "rs.initiate" to
provide the values you want:
https://docs.mongodb.com/manual/reference/method/rs.reconfig/)

[Gaff]

@tianon - both rs.initiate and rs.reconfig will fail if you attempt to pass a list of hosts that does not include the current node. So neither of these will work I'm afraid.

[wglambert]

The docker-entrypoint-initdb.d scripts run during an initialization period (and only if the database is empty), during which the container is only listening on localhost so trying to initiate a cluster during that period isn't possible as it won't resolve it's own container hostname.

So you'll probably need some manual intervention after everything is initialized, as using the docker-entrypoint-initdb.d will error with replSet initiate got NodeNotFound: No host described in new configuration 1 for replica set myrepl maps to this node, but then running the same rs.initiate() will work afterwards

[esetnik]

I have the same issue. Was this ever solved?

[zhangyoufu]

Somehow, I wrote an ugly hack to initialize replica set by abusing docker-entrypoint-initdb.d. Hope that helps someone coming to this issue.

[Gaff]

@esetnik - It's kinda hard to fix since Mongodb will sanity check any attempt to configure a replicaset and reject it if it looks invalid. To fix there there would have to be some 'force' option on replicasets within mongo or similar. I'd argue there's a case for this but it's probably not top of anyone's list.

In the end I wrote a sibling docker container that loops waiting for the mongodb to be available externally, configures the replicaset, then shuts down (though for non-test scopes I guess it could forever listen and set the replicasets). A better person might use consul or etcd or similar to co-ordinate all this.

@zhangyoufu's solution is terrible and beautiful at the same time! :D

[esetnik]

What a hack @zhangyoufu! I think there needs to be some support for this in the official docker image. There are several features of MongoDB only available with a replica set and this is a common production configuration. So it makes local development hard if we cannot setup a dev db with the same configuration.

[zhangyoufu]

I think that docker-entrypoint.sh keeps the initdb phase localhost-only intentionally. So that once mongod listens on whatever ip other than localhost, it means mongod is ready to serve.

To keep this mongod instance "private" (do not serve) while initdb is in progress, and circumvent the check insiders.initiate(), we could abuse /etc/hosts and point a hostname/FQDN to 127.0.0.1. After mongod shutdown in the end of initdb, just revert our changes in /etc/hosts and proceed. And maybe we should wait for the finish of the election, in case if any initdb script requires a primary node.

But modifying /etc/hosts is still too hackish to be allowed. HOSTALIASES environment variable looks pretty useful in this situation, as long as we are using glibc (not available for alpine).

[RalfLackinger]

Thank you @zhangyoufu for the hack. It worked (with minor tweaks to be able to handle authentication as well).

It felt a bit too 'hacky' for me though as I don't like abusing or modifying the original entrypoint script behavior. So I did a small workaround with a custom entrypoint that just calls the original one and after that is concluded executes the rs.initiate() command.

#!/usr/bin/env bash

# call default entrypoint
usr/local/bin/docker-entrypoint.sh "$@" &

# check if mongod is already running and the tmp init setup is done
PS_COMMAND="ps aux | grep '[m]ongod .*/etc/mongo/mongod.conf' | grep -v 'docker-entrypoint.sh'"
IS_MONGO_RUNNING=$( bash -c "${PS_COMMAND}" )
while [ -z "${IS_MONGO_RUNNING}" ]
do
  echo "[INFO] Waiting for the MongoDB setup to finish ..."
  sleep 1
  IS_MONGO_RUNNING=$( bash -c "${PS_COMMAND}" )
done
# wait for mongod to be ready for connections
sleep 3

# check if replica set is already initiated
RS_STATUS=$( mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.status().ok" )
if [[ $RS_STATUS -ne 1 ]]
then
  echo "[INFO] Replication set config invalid. Reconfiguring now."
  RS_CONFIG_STATUS=$( mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.status().codeName" )
  if [[ $RS_CONFIG_STATUS == 'InvalidReplicaSetConfig' ]]
  then
    mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin > /dev/null <<EOF
config = rs.config()
config.members[0].host = hostname()
rs.reconfig(config, {force: true})
EOF
  else
    echo "[INFO] MongoDB setup finished. Initiating replicata set."
    mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.initiate()" > /dev/null
  fi
else
  echo "[INFO] Replication set already initiated."
fi

wait

It is also very hacky, but I think will cope better with updates of the original entrypoint script. Also the code is just some quick first solution, so be aware it might be buggy. But maybe it might be helpful to someone running into the same issue.

[ldeluigi]

Maybe this can be helpful too https://gist.github.com/zhangyoufu/d1d43ac0fa268cda4dd2dfe55a8c834e#gistcomment-3554586