Bonus: we show how to use multi-fidelity Bayesian optimization to use a smaller and faster proxy scoring model to search even more efficiently. We get the best performance by training a distilled model from our main CometKiwi model.

The candidate pool is actually a search space, and you can model your uncertainty about scores you haven't scored yet with GP regression. Use BayesOpt to search the pool for promising candidates.

This nearly gets the maximum achievable score with only 70/200 scoring calls!