Clickhouse: Feature request: Add support for MongoDB clusters as external dictionaries

[uncletimmy3]

Recently I stumbled upon a problem external dictionary (linked to MongoDB) was empty. There was nothing releted to the problem in clickhouse log, but in MongoDB log I found this:

2019-01-29T12:09:24.023+0300 I NETWORK  [initandlisten] connection accepted from 192.168.231.108:54764 #3208444 (217 connections now open)
2019-01-29T12:09:24.068+0300 I QUERY    [conn3208444] assertion 13435 not master and slaveOk=false ns:NormativSrs.UserOutliers query:{}

not master and slaveOk=false causes CH not to read data, if server specified in the configuration is not a master anymore.

It seems that if you replace it:
https://github.com/yandex/ClickHouse/blob/a19d3620a3b612fc7d442368bbfc2e91a2bb8975/dbms/src/Dictionaries/MongoDBDictionarySource.cpp#L189
with this:

Connection(
    const std::string & uri,
    SocketFactory & socketFactory
);

it will be possible to set slaveOk=true.

[maddhuapplicate]

Does Clickhouse dictionary support Mongo cluster as source? Most production systems will be on clusters with SSL enabled. When we tried the cluster URI as host, Clickhouse did not connect to the cluster.

[uncletimmy3]

Hi! Dont close issue pls :)
Or may be this has been fixed?

[SychovOleh]

It is a really easy task. But, unfortunately, looks like no one wants to fix it.

[ekbfh]

Up

[alexey-milovidov]

We will try to find someone from community who are interested to implement this feature.

[manish53]

Hi alexey,
Sometime ago i faced same issue and successfully fixed it by updating poco version in clickhouse.
You can see PR https://github.com/ClickHouse/ClickHouse/pull/5387

[manish53]

And https://github.com/ClickHouse/ClickHouse/pull/5388

[alexey-milovidov]

This PR is assigned to Alexander Kuzmenkov @akuzm, he decided to take it for this week at last.

[Olegt0rr]

@alexey-milovidov @akuzm

Seems like #5388 #5387 #10915 are closed.
But still can't add MongoDB Atlas ReplicaSet as external dictionary.

Added test case covers:

Neither 3.6+ ReplicaSet connections (description):
mongodb+srv://<username>:<password>@example.dnliy.mongodb.net/test?retryWrites=true&w=majority

Nor 3.4+ ReplicaSet connections:
mongodb://<username>:<password>@shard-00-00.dnliy.mongodb.net:27017,shard-00-01.dnliy.mongodb.net:27017,shard-00-02.dnliy.mongodb.net:27017/test?ssl=true&replicaSet=atlas-10jbsv-shard-0&authSource=admin&retryWrites=true&w=majority

[akuzm]

Seems like #5388 #5387 #10915 are closed.
But still can't add MongoDB Atlas ReplicaSet as external dictionary.

Yes, I added only the very basic URI support. The hard part was updating Poco library to a version that has the necessary MongoDB API. Now the external contributors can use this API to continue working on related features. As far as I know, we don't plan to continue this work ourselves, but will happily accept contributions.

[Olegt0rr]

@akuzm, could you help to draw up the detailed requirements for the necessary contribution?
I'll try to find someone and pay for his work (ru link)

[akuzm]

@akuzm, could you help to draw up the detailed requirements for the necessary contribution?
I'll try to find someone and pay for his work (ru link)

Just implement what you need and send a PR, we don't have any particular requirements other than working code + docs + tests. Even a prototype is OK, it will let us to discuss the task in more detail.
The main difficulty may turn out to be with Poco -- if it doesn't support the kind of URI you need, it will have to be added upstream first (https://github.com/pocoproject/poco). They do accept contributions, although it may take some time. Then they'll publish a release with this feature, we'll update our Poco version to this release, and only after that we can merge the ClickHouse feature itself. In a prototype PR, Poco can be switched to whatever fork or version you need.