Metabase: ElasticSearch Driver

[CCoffie]

I'm not sure if metabase would be compatible with adding an ElasticSearch driver. If it is I would love to see it added.

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[pking74]

I would like to see ElasticSearch supported as well.

[hilaryspencer]

Another vote for ES support

[yehosef]

+1, ES is amazing

[kodyo]

One more!

[decker502]

+1

[bvolders]

+1

[javadevmtl]

+1

[wheredevel]

+1 STRONG!

[tegansnyder]

Maybe some sort of hybrid approach using this: https://github.com/NLPchina/elasticsearch-sql could help get this started.

[mzimmerman]

Heard you on the changelog, +1 for me on elasticsearch!

[kaur2k]

+1 Would use it!

[tomferreira]

+1

[adrien-candiotti]

+1

[gincor]

+1

[spa-87]

+1
We're using ELK stack for logs aggregation and monitoring. If Metabase had ES support, it would let to use different usage metrics and parsed application logs as a source data.

[kubbo]

+1

[eurotrip11]

+1

[edmarriner]

+1

[khalidsalomao]

+1

[jkarraker]

+1

[vikasgulati]

+1

[mariosergioti]

+1

[ghost]

+1

[vsreddy1602]

+1

[aksakal]

+1

[leomelzer]

+1

[alfredocambera]

+1

[juzerali]

👍 +1

[ptz0n]

+1

[coreation]

+1

[soaxelbrooke]

Grafana is doing "beautiful charts from elasticsearch" pretty well, it is a good place to take inspiration from:

http://docs.grafana.org/datasources/elasticsearch/

[tikh]

+1

[camsaul]

Hey everyone, one way we prioritize issues is by sorting them by the number of 👍 reactions on each issue's description. So instead of leaving +1 or 👍 comments, please just upvote by adding a reaction to the issue description.

[ptz0n]

@camsaul You could also count the +1's when prioritizing 😉

[camsaul]

@ptz0n, GitHub doesn't have a filter for counting +1 comments. And it notifies everyone on a given issue when a new comment is added (in this case, 41 people). So please just use 👍 reactions.

[E14]

It does not really seem to matter though, as this is the top ticket anyway ;)

May I ask what parts pose a difficulty adding this as a source - I mean, is there a technical difficulty or just a time issue?

[E14]

@camsaul

May I ask what parts pose a difficulty adding this as a source - I mean, is there a technical difficulty or just a time issue?

If it's just time and not complexity, I might just write the component myself, but writing a component that the maintainers already avoid due to complexity is probably not the best to start with -- that's why I ask.

[tlrobinson]

@E14 An Elasticsearch driver won't be as easy as adding a new SQL driver, but I don't believe it will be significantly more complex than some of our other drivers. If you want to take a look at the Druid, MongoDB, and Google Analytics drivers to see if you're up to the task that would be a good place to start. @camsaul correct me if I'm wrong.

Also note that in order to merge it into Metabase proper we'll need to make sure it's tested, including in our CI environment (CircleCI, see circle.yml and bin/ci)

[camsaul]

@E14 This is really more of a bandwidth thing than anything else -- we're a small team so we can only work on so many features at once. That being said, I'm not really sure how difficult this would be to implement. It's been probably 4 years since I've used ElasticSearch for anything so it's not really fresh in my memory right now. But I have a feeling it would be somewhat more difficult than other drivers like SQL-based ones or even Mongo since ElasticSearch's API is optimized more for search than the normal database operations we rely on for analysis.

Either way, you're welcome to give it a go. Check out our Guide to Writing a Driver for an overview on how they work.

[E14]

@tlrobinson / @camsaul Thank you for the reply and pointers! I'll have a look and see if it makes sense (for me ;)) to put time into it.

[E14]

Sorry for the missing feedback. For me there is too much documentation missing on the side of IDriver to start right into it without knowing the nuts&bolts and considering the time I have available right now.

What I can tell is that there is a Java library available for ES, so as with the Google drivers it is not necessary to implement the actual protocol, but only a wrapper around the library. Considering this, the driver might be better implemented as an optional add-in, especially since I have no idea under which license the client library can be used.

[globalundo]

For all the people waiting for this feature to be delivered - I can recommend to use a bit hacky approach for now: in our metabase installation, one of the data sources is local postgresql database installation, which had multicorn and postgres_fwd extensions installed. This way, you can access any other service as postgresql tables.
We are already using this to access data from Datadog and Mongodb installations, and plugins can be written in simple python - see https://github.com/globalundo/corndog or https://github.com/asya999/yam_fdw

[noblemavely]

+1

[Overbryd]

Please... stop posting "+1" to public isses.
Your message, and this reply, annoys all the people who are subscribed for
updates on this issue.

No one else cares if you "+1" this. If you want to express a vote on an
issue, use the "subscribe" to express interest or place a "thumbs up" emoij
on the respective issue.

+1

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/metabase/metabase/issues/1300#issuecomment-320207858,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABSdw9UyZmo8GFf86bHtj1eVi_aubOCks5sUuldgaJpZM4GTQHi
.

[camsaul]

Seconding what @Overbryd said. One way we prioritize issues is by sorting them by the number of 👍 reactions on each issue's description. So instead of leaving +1 or 👍 comments, please just upvote by adding a reaction to the issue description.

[porsovan]

elasticsearch is amazing

[je2ryw]

FYI, a WIP elasticsearch driver fork, and currently only support basic metadata discovery and native query with parameter variables. https://github.com/uxpsystems/metabase/tree/elasticsearch-driver

[prmdsharma-off]

+1

[allansene]

Elastic is experimenting with a SQL-like language. They just released a JDBC Driver.

I think that this may help:

https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-jdbc.html

[fetanchaud]

relates to #5562, as Dremio connects to ElasticSearch !

[vietvudanh]

Please add. ElasticSearch is fast and popular for analytics.

[bradjorgensen]

Also interested in an ElasticSearch driver.

[hodyroff]

The Amazon distribution of Elastic includes JDBC: https://opendistro.github.io/for-elasticsearch/downloads.html

[hadifarnoud]

It looks like this is not going to happen? it's been years since this issue created

[angjelo]

https://github.com/splitgraph/postgres-elasticsearch-fdw .... I think if you use splitgraph and have postgres you should be able to query elastic from there you can read and write too ...

[apinstein]

The Amazon distribution of Elastic includes JDBC: https://opendistro.github.io/for-elasticsearch/downloads.html

Yes.. I just ran across this, too! That actually comes from the underlying ES project, and there is an OSS JDBC driver: https://github.com/opendistro-for-elasticsearch/sql

So I am correct in thinking all that needs to be done to support ES-SQL in metabase would be to create a new "Module" per https://github.com/metabase/metabase/wiki/Writing-a-Driver:-Packaging-a-Driver-&-Metabase-Plugin-Basics and is that still the best place to start?

At this point seems it should be a low lift, so surprised no one has tackled it since it's been possible for what looks like a year now... I def am willing to try it...

[dacort]

So I am correct in thinking all that needs to be done to support ES-SQL in metabase would be to create a new "Module" per https://github.com/metabase/metabase/wiki/Writing-a-Driver:-Packaging-a-Driver-&-Metabase-Plugin-Basics and is that still the best place to start?

Yep, that'd be the approach! I'd say go up one level in the wiki to https://github.com/metabase/metabase/wiki/Writing-A-Driver - that's got some high-level detail and links to other parts of the driver docs. The documentation is still a work in progress, but should get you started.

Also feel free to take a look at https://github.com/dacort/metabase-athena-driver for an example driver that's used in several production environments. The biggest challenge is probably just keeping the driver up-to-date with Metabase releases. We do occasionally make breaking changes (see the Driver Notices section on the wiki) but there's a mailing list you can subscribe to on https://www.metabase.com/docs/latest/developers-guide-drivers.html