Horizon: Export metrics / status in prometheus scrape-able format

[ohnotnow]

Bit of a feature request. We're using docker/kubernetes more and more in our environment and Prometheus seems to be the default & recommended stats/health-check system to use. So it would be really nice to be able to scrape the metrics horizon already gathers in a prometheus-friendly format.

I'd be happy to try and work on a PR if it's something people would find useful & might be accepted :-)

[fitztrev]

I would use that. I don't know about Prometheus but a JSON response of at least the same metrics that are on the Dashboard page would be nice.

[driesvints]

Heya, thanks for suggesting this but I don't think we're going to be adding support for any external vendors like this. If we do there's really not telling which one is the next one. You may create your own package for this.

[ohnotnow]

@driesvints ah - ok. Is there any possibility of the keys etc that horizon uses being documented? If not then maintaining a package becomes a bit onerous. I'm happy to do it - but if it means constantly checking the source code of horizon for changes then that gets a bit grim :-/

[driesvints]

@ohnotnow I'm not sure if we should document anything outside the public api tbh...

[ohnotnow]

@driesvints ah, ok. Is the public api documented? Or is it just a case of tracking all the source code?

[driesvints]

@ohnotnow here are the docs: https://laravel.com/docs/5.7/horizon

[ohnotnow]

@driesvints ah- I was thinking more the public api for getting the metrics rather than how to run Horizon. If it’s not a goer just let me know and I can use a different solution :-) It’d be nice to use horizon, but if there’s no interest in kubernetes/GKE/EKS/AKS etc then I don’t want to use up your time - I’m sure you’ve got better things to do :-)

[driesvints]

@ohnotnow ah sorry, misunderstood you. No there's no api documentation like the one for laravel/framework.

[ohnotnow]

@driesvints no worries :-) I’ll look elsewhere for queue stuff :-) Have a good Xmas - thanks for all your efforts on the issues :-) Much appreciated :-)

[mfn]

We've simply looked at the keys and how the queues are structured and derived some simple redis commands which can graph each queue.

This is a script, called via collect infrastructure, which reports the size for each listed queue and the total:

#!/usr/bin/env bash

HOSTNAME="${COLLECTD_HOSTNAME:-`hostname -f`}"
INTERVAL="${COLLECTD_INTERVAL:-10}"
REDIS_HOST="${REDIS_HOST:-localhost}"
REDIS_PORT="${REDIS_PORT:-6379}"
REDIS_DB="${REDIS_DB:-0}"

# Given a Laravel queue name, it reports it's total size back on stdout
#
# Based on the Laravel implementation, see https://github.com/laravel/framework/blob/f21e942ae8a4104ffdf42c231601efe8759c4c10/src/Illuminate/Queue/LuaScripts.php#L19
laravel_queue_size() {
  local QUEUE_NAME=$1

  redis-cli \
    -h $REDIS_HOST \
    -p $REDIS_PORT \
    -n $REDIS_DB \
    eval \
    "return redis.call('llen', KEYS[1]) + redis.call('zcard', KEYS[2]) + redis.call('zcard', KEYS[3])" \
    3 \
    "queues:$QUEUE_NAME" "queues:$QUEUE_NAME:delayed" "queues:$QUEUE_NAME:reserved" \
    | sed 's/[^[:alnum:]]\+//g'
}

putval() {
  local NAME=$1
  local SIZE=$2

  if [[ -z "$SIZE" ]]; then
    SIZE=$(laravel_queue_size $NAME)
    TOTAL=$(($TOTAL + $SIZE))
  fi

  echo "PUTVAL $HOSTNAME/laravel_queue-$NAME/gauge-$NAME interval=$INTERVAL N:$SIZE"
}

while sleep "$INTERVAL"; do
  TOTAL=0
  putval queue1
  putval queue2
  putval total $TOTAL
done

HTH

[ohnotnow]

@mfn thanks for that :-) depending how drunk/lazy I am over the holidays I might put something together :-)

[LKaemmerling]

FYI i have created a package to handle this:

https://github.com/LKaemmerling/laravel-horizon-prometheus-exporter

Blog Post about this:

https://lukas-kaemmerling.de/blog/laravel-horizon-prometheus-exporter

[ohnotnow]

@LKaemmerling wow - thanks for that! I got side-tracked this year just getting our container stuff set up - this fell by the wayside :-/ Thanks for stepping up! :-D