Loki: fluent-plugin-loki - broken or very bad documentation?

[markuman]

Describe the bug
fluent-plugin-loki does not work as documentated.

First of all the loki endpoint url uses the key url in the README https://github.com/grafana/loki/blob/master/fluentd/fluent-plugin-loki/README.md#usage
But fluentd requires the key endpoint_url

2019-05-01 13:36:02 +0000 [error]: config error file="/fluent/fluent.conf" error_class=Fluent::ConfigError error="'endpoint_url' parameter is required"

When I change it like this

<match nureintest>
  @type loki
  endpoint_url "http://loki:3100"
  extra_labels {"env":"dev"}
  flush_interval 10s
  flush_at_shutdown true
  buffer_chunk_limit 1m
</match>

Fluentd is starting correctly.
Now I put a sample log via fluent-bit to my fluentd which should be outputted to my loki server.

echo -n "{\"key\": \"some value\"}" | ./fluent-bit -i stdin -t nureintest -o forward://127.0.0.1:24224

but fluentd throws an error

2019-05-01 13:38:10 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=NoMethodError error="undefined method `each' for nil:NilClass"
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluent-plugin-loki-0.2.0/lib/fluent/plugin/out_loki.rb:174:in `format_labels'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluent-plugin-loki-0.2.0/lib/fluent/plugin/out_loki.rb:181:in `handle_record'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluent-plugin-loki-0.2.0/lib/fluent/plugin/out_loki.rb:213:in `block in write'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/event.rb:323:in `each'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/event.rb:323:in `block in each'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/memory_chunk.rb:80:in `open'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin/buffer/memory_chunk.rb:80:in `open'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/event.rb:322:in `each'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluent-plugin-loki-0.2.0/lib/fluent/plugin/out_loki.rb:212:in `write'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1125:in `try_flush'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1425:in `flush_thread_run'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:454:in `block (2 levels) in start'
2019-05-01 13:38:10 +0000 [warn]: #0 /var/lib/gems/2.5.0/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-05-01 13:38:10 +0000 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_2ad744efbfb4/587d3a04ca0167fa7b0ed5056a6d13ca.log

[markuman]

oh boy https://github.com/grafana/loki/pull/210#issuecomment-460265700

your fluent plugin collides with https://github.com/eeddaann/fluent-plugin-loki

[briangann]

your fluent plugin collides with https://github.com/eeddaann/fluent-plugin-loki

I guess we'll have to rename ours.

[hados99]

I also faced same issue - plugin collides.
Is there any update?

[RobertHeim]

Until a renaming, I wonder how to build the plugin locally and install the local build?

[shane-axiom]

@tomwilkie @briangann @eeddaann @woodsaj Any thoughts here?

TL;DR there's a gem naming conflict between the official grafana devloped fluentd loki plugin and another developed by @eeddaann.

[shane-axiom]

I'm not a ruby person and this is basically untested beyond installation, but this very manual approach seems to work.

FROM fluent/fluentd:v1.4-debian-2

USER root

RUN apt-get update \
    && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*

RUN rm -rf /tmp/loki \
    && git clone --depth 1 https://github.com/grafana/loki.git /tmp/loki \
    && cd /tmp/loki/fluentd/fluent-plugin-loki \
    && sed -i "s|^  spec.files.*|  spec.files = Dir.glob('{bin,lib}/**/*')|" fluent-plugin-loki.gemspec \
    && gem build fluent-plugin-loki.gemspec \
    && gem install fluent-plugin-loki-*.gem \
    && cd /tmp \
    && rm -rf /tmp/loki

USER fluent

[eeddaann]

Hi,
As I wrote in the issue that @RobertHeim opened in my repository, I'll be happy to assist and solve the issue.
However i'm a bit concerned with simply changing the name of the gem since I don't know how it will affect the users of the current plugin.
any help will be appreciated :smile:

[markuman]

Ha, yeah. https://github.com/eeddaann/fluent-plugin-loki/issues/3#issuecomment-495519002
This will break all users of eeddaan@fluent-plugin-loki, because then their configuratioen doesn't work anymore as documentated :)

I think this is only solveable with a new release version on both fluentd plugins.

fluent-plugin-loki < 1.0.0 -> goes to https://github.com/eeddaann/fluent-plugin-loki
fluent-plugin-http-loki >= 1.0.0 -> goes to https://github.com/eeddaann/fluent-plugin-loki
fluent-plugin-loki >= 1.0.0 -> goes to https://github.com/grafana/loki/blob/master/fluentd/fluent-plugin-loki/README.md#usage

...with some pesky deprecation warnings everywhere.

[shane-axiom]

Seems logical to me. In the meantime though, the README for the grafana plugin provides incorrect installation instructions:

$ gem install fluent-plugin-loki

I'm happy to make a PR to fix this, but need clarification what the correct installation method should be. Will the grafana plugin be published under a different name until this conflict is sorted out, or should users be instructed to install from source?

[RobertHeim]

@shane-axiom i opened a PR to at least hint people that it is currently not working https://github.com/grafana/loki/pull/618

[fernandino143]

So sad :(
I'm having the same issue. Is there a way of forcing or even a workaround for this to work?

[shane-axiom]

@fernandino143 Did you try installing from source, as demonstrated in this comment?

https://github.com/grafana/loki/issues/535#issuecomment-494162252

[fernandino143]

Took a beating, but, finally got it up in a centos7 container @shane-axiom. However, I do agree with the OP and the url parameter needs to be endpoint_url. Sorry for taking so long to reply and thanks once again

[shane-axiom]

If endpoint_url is required then you’ve installed eeddaann’s plugin and not this one...

[dawidmalina]

What is the blocker to move forward with this plugin cleanup (renaming) as I don't know why this taking so long? Would be really nice if we could finish this topic as there is a huge interest in this area.

[eeddaann]

As I understood from @markuman 's answer, you just have to change the version of Grafana's gem.
I won't increase my gem's version so it would be fine..

[markuman]

As I understood from @markuman 's answer, you just have to change the version of Grafana's gem.
I won't increase my gem's version so it would be fine..

@eeddaann that was just a suggestion :) the people of grafana must be active on this issue and do anything.

[dawidmalina]

@slim-bean, @tomwilkie is there anyone who can work on this topic?

[dawidmalina]

@woodsaj @briangann maybe we can just update plugin version (currently @eeddaann plugin version is 0.2.0) to 0.3.0 and publish new version to RubyGems.org. What do you think?

[briangann]

target is friday (6/14) to get this resolved, assigning to myself

[shane-axiom]

Plugin errors when I try to install, opened new issue #719

[RobertHeim]

Did anyone got this up and running?

[shane-axiom]

@RobertHeim Yes, gem install works using version 1.0.1 or greater.