LightGBM integration doesn't work properly if you are trying to maximize a custom metric

[volker48]

Expected behavior

If the eval metric has is_higher_better set to True then the objective should be maximized.

Environment

• Optuna version:
• Optuna Integration version:
• Python version:
• OS:
• (Optional) Other libraries and their versions:

Error messages, stack traces, or logs

def higher_is_better(self) -> bool:
        metric_name = self.lgbm_params.get("metric", "binary_logloss")
        return metric_name in ("auc", "auc_mu", "ndcg", "map", "average_precision")

This code is totally incorrect if someone is using a custom evaluation metric.

I only found out about this bug after wasting several hours tuning and the best parameters that were returned being obviously nonsense.

I then tried explicitly creating a study with direction set to 'maximize' and at least hit a warning message.

Steps to reproduce

1. Setup a study to maximize the optimization
2. Create a custom feval callback function
3. In the params explicitly set the metric to something other than ("auc", "auc_mu", "ndcg", "map", "average_precision")
4. In the call to optuna_integration.lightgbm.train, set the feval callable to the function created in step 2

def score_cb(preds, eval_data):
    score = calculate_score(preds, eval_data.label)
    return 'score', score, True

lgb_study = optuna.create_study(direction="maximize", study_name="LightGBM Auto Tune")

params = {
    "objective": "regression",
    "metric": "correlation",
    "boosting_type": "gbdt",
}

model = opt_lgb.train(params, dtrain, study=lgb_study, num_boost_round=2000, valid_sets=[dtrain, dval], valid_names=["training", "validation"], feval=score_cb, callbacks=[opt_lgb.early_stopping(stopping_rounds=30), lgb.log_evaluation(1)])

Additional context (optional)

No response

[PieterPel]

Bump, since I just ran into the same issue. Could I try to tackle this as a first issue?

[jameslamb]

👋🏻 hi, LightGBM maintainer here.

I came here in response to microsoft/LightGBM#6945. On that issue, we're discussing the possibility of LightGBM making is_higher_better available in its public API, including for custom metric functions.

I'd like to propose something that could be done on the optuna side, independent of that.... could optuna provide a way for people who provide custom metric functions to declare the optimization direction to optuna?

For example, maybe optuna could check for an attribute is_higher_better (or whatever name maintainers here would like) to register that.

And custom metrics could then provide that value like this:

class my_custom_metric:

    def __init__(self):
        self.is_higher_better = False

    def __call__ (self, env: lgb.callback.CallbackEnv):
        # ... metric logic ...

More complete example of using a class with __call__, instead of a function, to provide a custom metric: microsoft/LightGBM#6935 (comment)

This particular issue is about LightGBM, but I suspect that any estimator that optuna supports and which also supports custom metric functions would benefit from functionality like this in optuna.

(I do not use optuna much, apologies if I am saying things that are already possible in this library)

Alternatively, optuna could say that it always optimizes in one direction for custom metric functions, and anyone using this library could just flip the sign of their metrics to work as expected with that.

[jameslamb]

Also @volker48 for the future... in general, when reporting things like this, providing a commit-anchored link to the code you're referring to is very very helpful to anyone trying to see the same things you're seeing.

In this case, I think you were referring to this:

optuna-integration/optuna_integration/lightgbm/_lightgbm_tuner/optimize.py

Lines 157 to 159 in c44f35c

	def higher_is_better(self) -> bool:
	metric_name = self.lgbm_params.get("metric", "binary_logloss")
	return metric_name in ("auc", "auc_mu", "ndcg", "map", "average_precision")

I looked for a corresponding function in the xgboost integration here, and couldn't find one: https://github.com/optuna/optuna-integration/blob/c44f35cf7a3dc75b9a150c1733e626db18dfe6cb/optuna_integration/xgboost/xgboost.py. Which makes me wonder... how is this handled for xgboost in optuna?

xgboost also supports custom metric functions: https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html#customized-metric-function

[PieterPel]

Also @volker48 for the future... in general, when reporting things like this, providing a commit-anchored link to the code you're referring to is very very helpful to anyone trying to see the same things you're seeing.

In this case, I think you were referring to this:

optuna-integration/optuna_integration/lightgbm/_lightgbm_tuner/optimize.py

Lines 157 to 159 in c44f35c

def higher_is_better(self) -> bool:
metric_name = self.lgbm_params.get("metric", "binary_logloss")
return metric_name in ("auc", "auc_mu", "ndcg", "map", "average_precision")
I looked for a corresponding function in the xgboost integration here, and couldn't find one: https://github.com/optuna/optuna-integration/blob/c44f35cf7a3dc75b9a150c1733e626db18dfe6cb/optuna_integration/xgboost/xgboost.py. Which makes me wonder... how is this handled for xgboost in optuna?

xgboost also supports custom metric functions: https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html#customized-metric-function

As far as I know the XGBoost integration is much less sophisticated than it is for LightGBM, providing only a pruning callback. Hence the user always sets the direction of the Optuna study manually to "minimize" or "maximize". For LightGBM there are wrapper classes that create the study automatically. Currently the assumed direction does not consider custom evaluation functions.

👋🏻 hi, LightGBM maintainer here.

I came here in response to microsoft/LightGBM#6945. On that issue, we're discussing the possibility of LightGBM making is_higher_better available in its public API, including for custom metric functions.

I'd like to propose something that could be done on the optuna side, independent of that.... could optuna provide a way for people who provide custom metric functions to declare the optimization direction to optuna?

For example, maybe optuna could check for an attribute is_higher_better (or whatever name maintainers here would like) to register that.

And custom metrics could then provide that value like this:

class my_custom_metric:

def __init__(self):
    self.is_higher_better = False

def __call__ (self, env: lgb.callback.CallbackEnv):
    # ... metric logic ...

More complete example of using a class with __call__, instead of a function, to provide a custom metric: microsoft/LightGBM#6935 (comment)

This particular issue is about LightGBM, but I suspect that any estimator that optuna supports and which also supports custom metric functions would benefit from functionality like this in optuna.

(I do not use optuna much, apologies if I am saying things that are already possible in this library)

Alternatively, optuna could say that it always optimizes in one direction for custom metric functions, and anyone using this library could just flip the sign of their metrics to work as expected with that.

This approach could definitely be a workaround, but it would still be unclear to the user that a custom eval metric that works fine in LightGBM will do exactly the opposite for Optuna. Perhaps raising a warning at runtime that the direction is assumed when there is no is_higher_better attribute can work.

def higher_is_better(self) -> bool:
    metric_name = self.lgbm_params.get("metric", "binary_logloss")
    feval = self.lgbm_params.get("feval")

    if feval is not None:
        is_higher = getattr(feval, "is_higher_better", None)
        if is_higher is not None:
            return is_higher
        else:
            warnings.warn(
                f"Custom metric `{feval.__name__}` does not define `is_higher_better`. "
                "Assuming direction based on built-in metric names. "
                "Consider adding `is_higher_better = True/False` to your custom metric." \
                "You can do this by wrapping the metric function in a class as __call__ method."
            )

    return metric_name in ("auc", "auc_mu", "ndcg", "map", "average_precision")