Early stopping in Botorch models

[huylenguyen]

Hi all, I'm currently working on Compositional Kernel Search for botorch models, and I was wondering if early-stopping is a good idea in GPs? By this I mean computing the BIC on some validation data during optimisation, and early-terminate trials with kernel configurations that are not doing well.

I'm still just learning about GPs and Bayesian Optimization, so any insights would be greatly appreciated!

[Balandat]

Hmm I am not sure I understand what you mean by "early-terminate trials with kernel configurations that are not doing well"? Is this a setting where evaluating a proposed kernel composition itself is a very costly thing to do, so that you're running a Bayesian Optimization loop on top of this? Or are you suggesting to early-stop the inference for a particular kernel composition (i.e. stopping the numerical optimization of the MLE / the MCMC chain in a Fully Bayesian setting early?)

[huylenguyen]

Hi! What I mean is the latter: I am running many different GPs (with different kernels) on the same problem, and I'm wondering if early-stopping the MLE is a feasible strategy to save on computational costs.

[Balandat]

Hmm yeah I think it's a reasonable thing to try, but it's also potentially dangerous in that you might have cases where a lot of progress would have been made late during the optimization. I think this is largely an empirical question that requires running some comparisons of early-stopped versions against the full versions.

I made some PRs recently to allow you to early-stop numerical optimization based on a timeout parameter in case that would be helpful: #1598

[huylenguyen]

I made some PRs recently to allow you to early-stop numerical optimization based on a timeout parameter in case that would be helpful: #1598

Thank you very much, I will see if this can help with my application