Clickhouse: Is it normal to store large strings in ClickHouse?
I want to use ClickHouse as storage for log data. Some log strings is pretty big (5-15 Mb).
Is it normal to store large strings (several Mb) in ClickHouse?
shalugin
All 5 comments
I think it is not suitable to store a bigger file (5-15MB), but except HDFS.
theseusyang
on 30 Nov 2017
Clickhouse is best in building reports by grouping and filtering data. If your data is 5-15Mb strings - it's quite hard to imagine scenarios how you can group / filter that huge strings. Probably you need some database with full text index.
BUT: if it's possible to extract from that huge log strings some properties which are not so long and can be grouped / sorted / filtered (for example ips, useragents, etc.) - ClickHouse can be helpful for you.
filimonov
on 30 Nov 2017
I store some log data in Clickhouse but I don't use CH to search the MSG field. I pre-split what interests me from the MSG into fields and search on those fields. I found Elasticsearch is faster and more convenient for searching logs due to the way it parses/indexes the MSG. I just wish it had as good of compression as CH.
daledude
on 1 Dec 2017
To analyze logs with ClickHouse it is indeed recommended to parse them into individual columns before ingestion instead of storing arbitrary strings/blobs as is.
@shalugin do you have any further questions?
blinkov
on 22 Aug 2018
@blinkov Thanks!
shalugin
on 30 Aug 2018
Related issues
vixa2012
路
3Comments
goranc
路
3Comments
atk91
路
3Comments
vvp83
路
3Comments
innerr
路
3Comments
Most helpful comment
Clickhouse is best in building reports by grouping and filtering data. If your data is 5-15Mb strings - it's quite hard to imagine scenarios how you can group / filter that huge strings. Probably you need some database with full text index.
BUT: if it's possible to extract from that huge log strings some properties which are not so long and can be grouped / sorted / filtered (for example ips, useragents, etc.) - ClickHouse can be helpful for you.