Xgboost: Interface for custom objective and evaluation functions in xgb.train and xgb.cv

[khotilov]

The obj and feval parameters for passing custom objective and evaluation functions are currently separate from the objective and eval_metric ones in the parameters list. It makes the interface cluttered and prone to errors. E.g., trying to use a custom loss function while still having 'binary:logistic' objective present within params, would mess things up.

Wouldn't it make sense to just reuse the objective and eval_metric parameters for passing those custom methods?

[tqchen]

interesting, I think your proposal totally make sense, can you create a PR on this?

[khotilov]

I can do that at some time later. Too much to do these days...

[hetong007]

I made some changes in the code with https://github.com/dmlc/xgboost/commit/19b24cf978763701b41a928679e254743c18b629, and some documentation changes in https://github.com/dmlc/xgboost/commit/8d3a7e1688f075779f7ba96ec6d68cba23a75619. Basically you can pass self-defined functions to params$objective and params$eval_metric. Considering the compatibility, I didn't remove feval and obj in the arguments.

@khotilov @tqchen Do you have further comments? Thank you.

[khotilov]

Thanks Tong!
I am currently away with poor internet access. I will be able to check it
out later in the end of the week.

On Mon, May 25, 2015 at 1:40 PM, Tong He [email protected] wrote:

I made some changes in the code with 8d3a7e1
https://github.com/dmlc/xgboost/commit/8d3a7e1688f075779f7ba96ec6d68cba23a75619.
Basically you can pass self-defined functions to params$objective and
params$eval_metric. Considering the compatibility, I didn't remove feval
and obj in the arguments.

@khotilov https://github.com/khotilov @tqchen
https://github.com/tqchen Do you have further comments? Thank you.

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/xgboost/issues/279#issuecomment-105286945.

[tqchen]

This looks good to me. Thanks! Let us update doc in the guide script to use param$objective and param$eval_metric as customized objective function

[hetong007]

I updated the demo scripts.

[ajing]

Is that possible to retrieve the index for dtrain? I was thinking about doing objective and eval_metric in xgb.cv, but I need use another outside data vector which is not part of dtrain to evaluate the model.

[ajing]

Could I ask a trivial question? For the example provided, the first gradient is 1/(1 + exp(-preds)) - labels. But the function for log loss is -log P(yt|yp) = -(yt log(yp) + (1 - yt) log(1 - yp)). Is first gradient the derivative for log loss function? I have an object function y = f(x), which I don't know the form of f(). I can get y by input x. Could I do something like (f(x+dx) - f(x-dx))/(2*dx) to get the first gradient? Thanks for the help!

[hetong007]

The last question was resovled in https://github.com/dmlc/xgboost/issues/407.

[VaibhavBehl]

Ok.. so I have looked around on a few threads, and played around with the API, but I am not able to get MAE(mean absolute error) to work with the Default objective(reg:linear). I would like to know is this even possible with R/xgb?, Theoretically it should be possible.

I am using the API like this. With this I am not getting any errors, but it shows no difference in the output.
param0 <- list("objective" = "reg:linear"
#, "eval_metric" = "rmse"
, "eta" = 0.07
, "subsample" = 0.8
, "min_child_weight" = 10
, "max_depth" = 9
, "nthreads" = 4
#, "eval_metric"=evalerror
#, "maximize"=FALSE
#, "feval"=evalerror
)
xgb.train(params = param0, data = xgtrain , nrounds =50, feval=evalerror, maximize=FALSE)

evalerror <- function(preds, dtrain) {
labels <- getinfo(dtrain, "label")
err <- as.numeric(mean(abs(labels - preds)))
print(err)
return(list(metric = "error", value = err))
}

[hadjipantelis]

@tqchen: While MAE is not nice differentiable, using pseudo-Huber (say delta = 1 to follow Wikipedia's notation) will be fine. This will resolve #829 too. Would that help? (Derivatives are trivial : a/(a^2 + 1)^(1/2) and 1/(a^2 + 1)^(1/2) - a^2/(a^2 + 1)^(3/2) for first and second derivative respectively.) Would it help if I added the relevant code in objective/regression_obj.cc or should this be treated as something that generally should be a "plug-in issue"? Robust regression is "quite standard" to offer I think.

[adamwlev]

@hadjipantelis the second derivative reduces (1 + a^2)^(-3/2)

[hadjipantelis]

@adamwlev Yes, of course. I just differentiated these out quickly for the comment. I use the form you mention in code. Thanks for mentioning it so people don't copy-paste the non-simplified form immediately.

[adamwlev]

@hadjipantelis I am a big fan of this psuedo Huber loss thanks for commenting

[aktivwater]

Thank you for the advice, but I tried to use the pseudo Huber loss function and it is not working. Not sure if it is a general problem or if I am doing something wrong.

# Define the objective
my_Huberobj <- function(preds,dtrain){ 
labels <- getinfo(dtrain, "label")
grad <- (labels-preds)/((labels-preds)^2 + 1)^(1/2)
hess <- (1 + (labels-preds)^2)^(-3/2)
return(list(grad = grad, hess = hess))
}
# run xgboost
watchlist <- list(eval = dtest, train = dtrain)
num_round <- 4
param <- list(max_depth=4, eta=0.1, nthread = 2, silent=1, objective= my_Huberobj)
xgb.train(param, dtrain, num_round, watchlist)

Results:
[1] eval-rmse:30412.744141 train-rmse:30382.330078
[2] eval-rmse:30412.744141 train-rmse:30382.330078
[3] eval-rmse:30412.744141 train-rmse:30382.330078
[4] eval-rmse:30412.744141 train-rmse:30382.330078

Doing the exact same with a quadratic loss function works fine (same results as with the build-in objective reg:linear ).

# Define the objective
my_RMSEobj <- function(preds,dtrain){ 
labels = getinfo(dtrain, "label")
grad <- preds-labels
hess <- rep(1, length(grad))
return(list(grad = grad, hess = hess))
}  
# run xgboost
watchlist <- list(eval = dtest, train = dtrain)
num_round <- 4
param <- list(max_depth=4, eta=0.1, nthread = 2, silent=1, objective= my_RMSEobj)
xgb.train(param, dtrain, num_round, watchlist)

Results:
[1] eval-rmse:27498.787109 train-rmse:27474.146484
[2] eval-rmse:24886.111328 train-rmse:24867.011719
[3] eval-rmse:22548.056641 train-rmse:22534.398438
[4] eval-rmse:20453.984375 train-rmse:20444.093750