Logstash: Refactor interaction between the Collector/Metrics with the WebApi/Pipeline/Plugin

[ph]

Currently the Metrics use a singleton instance of the Collector to save or access the metric store. This make accessing it really simple but cause other issues when testing it ( see https://github.com/elastic/logstash/issues/5291)

I suggest we do the following refactor:

• It will be the Agent responsability to create the Collector instance
• The pipeline will receive a scoped metric instance associated with right collector
• The plugin will receive a scoped metric instance from the pipeline (already done)
• The Webserver will receive an instance of the collector to access the metric store directly and wont register as an observer. This will give fresher result on the API.
• When a pipeline reload occur the collector will be cleared and the PeriodPoller restarted.
• The input metric will register to the Observer to receive notification of new snapshot.
• The input metric will use the metric accessor to gain access to the collector.

[ph]

Because of #5400 I've switched the priority to P1 and will go ahead with the refactor.

[ph]

cc @andrewvc

[andrewvc]

One other improvement would be to use a lazy cache to synchronize snapshots rather than a thread that does this periodically. It would fix initialization order issues and be a simpler design. We would need code like:

def snapshot
  @snapshot_mutex.synchronize do
     # nano time is monotonic
     return @snapshot if @snapshot.last_updated + 1sInNanos > System.nano_time

     @snapshot = make_new_snapshot
   end
end

[ph]

Even with removing the singleton, the LogStash::Agent metrics after config reloading resets the metric collector is still failling. Doing a bit more digging.

[ph]

I think it might be related to the test case and the agent doesnt reload the new configuration.

[ph]

I've moved stuff around, to know this.

• when the agent start the collector it correctly save the metrics
• the metric store is correctly cleaned when a pipeline is restarted
• Some metric are correctly saved to the new metric store like the jvm stats but newly created pipeline.

[ph]

faillings test fixed, thread safety when we were creating pipelining and settings the metric.

[ph]

I finished doing this refactor, I will run the test on a dedicated box to make sure we dont have any issue like #5400.