Optimizing Cross-Validation Choices in Machine Learning

By John Paul Mueller, Luca Massaron

Being able to validate a machine learning hypothesis effectively allows further optimization of your chosen algorithm. The algorithm provides most of the predictive performance on your data, given its ability to detect signals from data and fit the true functional form of the predictive function without overfitting and generating much variance of the estimates. Not every machine learning algorithm is a best fit for your data, and no single algorithm can suit every problem. It’s up to you to find the right one for a specific problem.

A second source of predictive performance is the data itself when appropriately transformed and selected to enhance the learning capabilities of the chosen algorithm.

The final source of performance derives from fine-tuning the algorithm’s hyper-parameters, which are the parameters that you decide before learning happens and that aren’t learned from data. Their role is in defining a priori a hypothesis, whereas other parameters specify it a posteriori, after the algorithm interacts with the data and, by using an optimization process, finds that certain parameter values work better in obtaining good predictions.

Not all machine learning algorithms require much hyper-parameter tuning, but some of the most complex ones do, and though such algorithms still work out of the box, pulling the right levers may make a large difference in the correctness of the predictions. Even when the hyper-parameters aren’t learned from data, you should consider the data you’re working on when deciding hyper-parameters, and you should make the choice based on cross-validation and careful evaluation of possibilities.

Complex machine learning algorithms, the ones most exposed to variance of estimates, present many choices expressed in a large number of parameters. Twiddling with them makes them adapt more or less to the data they are learning from. Sometimes too much hyper-parameter twiddling may even make the algorithm detect false signals from the data. That makes hyper-parameters themselves an undetected source of variance if you start manipulating them too much based on some fixed reference like a test set or a repeated cross-validation schema.

Both R and Python offer slicing functionalities that slice your input matrix into train, test, and validation parts. In particular, for more complex testing procedures, such as cross-validation or bootstrapping, the Scikit-learn package offers an entire module, and R has a specialized package, offering functions for data splitting, preprocessing, and testing. This package is called caret.

The possible combinations of values that hyper-parameters may form make deciding where to look for optimizations hard. As described when discussing gradient descent, an optimization space may contain value combinations that perform better or worse. Even after you find a good combination, you’re not assured that it’s the best option. (This is the problem of getting stuck in local minima when minimizing the error.)

As a practical way of solving this problem, the best way to verify hyper-parameters for an algorithm applied to specific data is to test them all by cross-validation, and to pick the best combination. This simple approach, called grid-search, offers indisputable advantages by allowing you to sample the range of possible values to input into the algorithm systematically and to spot when the general minimum happens.

On the other hand, grid-search also has serious drawbacks because it’s computationally intensive (you can easily perform this task in parallel on modern multicore computers) and quite time consuming. Moreover, systematic and intensive tests enhance the possibility of incurring error because some good but fake validation results can be caused by noise present in the dataset.

Some alternatives to grid-search are available. Instead of testing everything, you can try exploring the space of possible hyper-parameter values guided by computationally heavy and mathematically complex nonlinear optimization techniques (like the Nelder-Mead method), using a Bayesian approach (where the number of tests is minimized by taking advantage of previous results) or using random search.

Surprisingly, random search works incredibly well, is simple to understand, and isn’t just based on blind luck, though it may initially appear to be. In fact, the main point of the technique is that if you pick enough random tests, you actually have enough possibilities to spot the right parameters without wasting energy on testing slightly different combinations of similarly performing combinations.

The graphical representation below explains why random search works fine. A systematic exploration, though useful, tends to test every combination, which turns into a waste of energy if some parameters don’t influence the result. A random search actually tests fewer combinations but more in the range of each hyper-parameter, a strategy that proves winning if, as often happens, certain parameters are more important than others.

grid-search vs. random search
Comparing grid-search to random search.

For randomized search to perform well, you should make from 15 to a maximum of 60 tests. It does make sense to resort to random search if a grid-search requires a larger number of experiments.