Gluon-ts: Distributional error metrics

[alexhallam]

I am looking for a way to calculate pinball loss or rank probability score. I show some of my output below. I am curious what the QuantileLoss is. Can I use that metric to get pinball loss?

I set up some quantiles as follows:

# eval
quantile_seq = np.arange(0.1, 1 , .1)
evaluator = Evaluator(quantiles=quantile_seq)
agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(test_ds))
print(json.dumps(agg_metrics, indent=4))

the output was

    "MSE": 5.173077055395188,
    "abs_error": 1937.2383211255074,
    "abs_target_sum": 1838.0,
    "abs_target_mean": 0.9771398192450824,
    "seasonal_error": 6.693360194745236,
    "MASE": 0.16621583137977658,
    "sMAPE": 1.9511674293834367,
    "MSIS": 5.1788325358267935,
    "QuantileLoss[0.1]": 656.3897180770059,
    "Coverage[0.1]": 0.10419989367357789,
    "QuantileLoss[0.2]": 1042.5795287945307,
    "Coverage[0.2]": 0.1339712918660287,
    "QuantileLoss[0.30000000000000004]": 1352.888840725538,
    "Coverage[0.30000000000000004]": 0.15683147262094635,
    "QuantileLoss[0.4]": 1630.0921277590678,
    "Coverage[0.4]": 0.25465178096757046,
    "QuantileLoss[0.5]": 1937.2383209627733,
    "Coverage[0.5]": 0.41626794258373206,
    "QuantileLoss[0.6]": 2267.2348443405704,
    "Coverage[0.6]": 0.5614035087719298,
    "QuantileLoss[0.7000000000000001]": 2560.9445986107776,
    "Coverage[0.7000000000000001]": 0.6177565124933547,
    "QuantileLoss[0.8]": 2764.1959102034393,
    "Coverage[0.8]": 0.6544391281233386,
    "QuantileLoss[0.9]": 2777.113136820318,
    "Coverage[0.9]": 0.6985645933014354,
    "RMSE": 2.274439943237717,
    "NRMSE": 2.327650453335226,
    "ND": 1.0539925577396667,
    "wQuantileLoss[0.1]": 0.3571217182138226,
    "wQuantileLoss[0.2]": 0.5672358698555662,
    "wQuantileLoss[0.30000000000000004]": 0.7360657457701513,
    "wQuantileLoss[0.4]": 0.8868836386066745,
    "wQuantileLoss[0.5]": 1.053992557651128,
    "wQuantileLoss[0.6]": 1.2335336476281666,
    "wQuantileLoss[0.7000000000000001]": 1.3933322081669084,
    "wQuantileLoss[0.8]": 1.503915076280435,
    "wQuantileLoss[0.9]": 1.5109429471274851,
    "mean_wQuantileLoss": 1.027002601033371,
    "MAE_Coverage": 0.10114596254947134
}

[jgasthaus]

Hi @alexhallam,

the QuantileLoss here is the exactly the pinball loss, scaled by a factor of two (to make it agree with the absolute error for q=0.5), defined as:

https://github.com/awslabs/gluon-ts/blob/08ab434b1e0946c21010ddbb4248288f8f043599/src/gluonts/evaluation/_base.py#L346-L352

This is then summed over all evaluation time points and time series.

The wQuantileLoss[q] metrics are QuantileLoss[q]/abs_target_sum, and mean_wQuantileLoss is the average of these scaled quantile losses, which gives you an approximation to CRPS.

[alexhallam]

Thanks! So if I divide the QuantileLoss by 2 I get pinball. How close is mean_wQuantileLoss to CRPS? I am curious because I am comparing metrics against various other forecast methods and I would like them to have common ground.

[ColinLeverger]

Hi @alexhallam , did you learn more about your previous question :

How close is mean_wQuantileLoss to CRPS?

I'm facing the same question now!

[alexhallam]

@ColinLeverger I ended up coding my own pinball loss function outside of this package and applied it to all the forecasting methods I was interested in. Pinball Loss is more common for the community I had to present to. Also it takes three short lines of code to write. Let me know if you need anything else.

[ColinLeverger]

Thank you @alexhallam for your answer!
Then, the question still stands... Any thoughts, @jgasthaus ?