Finding a suitable machine learning algorithm for your problem is not the end of your journey because you have to use the right hyperparameters so that your model can function better. These so-called hyperparameters are quite important because they can make your model perform worse than you expect if you fail to tune them properly.
Hyperparameters is configurations you use either in data preprocessing steps or the model itself. These hyperparameters are tuned before you train your machine learning model. You can think of this configuration as the same as the arguments you pass to a function, but you pass them to a data preprocessing transformer or machine learning model constructor.
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.ensemble import RandomForestRegressor
# Assume these values as X and y train
X_train = pd.DataFrame(...)
y_train = pd.Series(...)
# Data preprocessing transformer (custom)
class MissingValuesHandler(BaseEstimator, TransformerMixin):
def __init__(self, strategy='mean', custom_value=None):
# ... (simplified)
# ... (simplified)
# Data preprocessing
handler = MissingValuesHandler(strategy='median')
X_train_transform = handler.fit_transform(X_train)
# Machine learning model
forest_reg = RandomForestRegressor(n_estimators=10, max_features=3)
forest_reg.fit(X_train_transform, y_train)
In this example, we use a total of 1 hyperparameter in the data preprocessing step (strategy) and 2 hyperparameters in the model itself (n_estimators and max_features). Random tree regressor has many more hyperparameters that you can adjust when you check the documentation.
While there is no absolute theory about how you can find these values, data scientists use a rather traditional approach to finding them (i.e. trial and error). But instead of doing it manually, they use computers to automate the process by simply providing all the combinations of parameters.
If you are critical enough, you may realize that this method will not necessarily lead you to the most optimal solution because the solution you reach may be the best only in the combination of parameters you provide.
As previously mentioned, the strategy we have to optimize (fine tune) hyperparameters is to carry out automatic trial and error. To achieve this, there are 2 strategies you can use:
A. Géron, Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. Sebastopol: O’Reilly Media, 2019.