Sidekiq: Feature Request: Track expired jobs as a metric

[lardcanoe]

Sidekiq / Pro / Enterprise version(s): Pro

When a job is skipped due to expiring, send a metric to statsd. https://github.com/mperham/sidekiq/wiki/Pro-Metrics

Something like jobs.expired

Though, for our use case, a callback would be ideal.

[mperham]

Nice idea. More comprehensively, I need to think about how to track the metrics for various features throughout Pro and Enterprise. This is something I will ponder over the next few weeks. It could be something like an internal event pub/sub registry:

notify("cron.newjob", :schedule => schedule, :job => payload)
notify("expired", :job => payload)
notify("ratelimit.exceeded", :limiter => lmtr)

[lardcanoe]

https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md#360

[lardcanoe]

One small tweak that would be nice is to send expires_in and queue along as a tag.

tags = %W(expires_in:#{msg['expires_in']} queue:#{msg['queue']})
Sidekiq::Pro.metrics.increment 'jobs.expire', tags: tags

We need this so that we can setup proper alerts (we use Datadog). This way we can differentiate between which one of our SLA's is falling behind. We have 15min jobs, 4hr jobs, etc...

[mperham]

That's a Datadog-specific extension to Statsd. I can't support SaaS-specific extensions unless you can show it being used across multiple SaaSes. If it's a de-facto standard (like Statsd itself), I'm happy to put some work into supporting it.

[mperham]

Looks like Influx supports Statsd tags: https://www.influxdata.com/blog/getting-started-with-sending-statsd-metrics-to-telegraf-influxdb/

https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd

[mperham]

Prometheus supports Statsd tags: http://marselester.com/prometheus-via-dogstatsd.html

[lardcanoe]

I'm good with going forward with a tiny bit of custom code for this.

Thanks for 3.6.1 changes.

I'll let you close this, or leave open. Not sure your intent based on the two comments about tags.

[mperham]

Oops I think I forgot the queue tag. I’m not planning on adding the expiry as a tag but ideas on how to make the API more flexible so you can override it are welcome.

On Dec 21, 2017, at 19:01, Steve Frank [e-mail] wrote:

I'm good with going forward with a tiny bit of custom code for this.

Thanks for 3.6.1 changes.

I'll let you close this, or leave open. Not sure your intent based on the two comments about tags.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[mperham]

This is in 3.6.0 https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md#360