What we will see¶
- Mathematics around model selection
- Why random-search is better than grid-search
- How to implement a random-search
- How to use MLflow to log results of a random-search
Risk Minimization¶
Setting:
- Two space of objects $X$ and $Y$
- $\pi \in \Pi$ such as $\pi : X \rightarrow Y$
- Non-negative real-valued loss function $L(\hat{y},y)$
Goals:
- Minimizing $R(\pi)$ = ${\bf E}$ $\left[ L(\pi(x),y) \right]$
- Finding $\pi^*$ = $\arg$ $\min_{\pi \in \Pi}$ $R(\pi)$
Bias-Variance Tradeoff¶
- $\Pi_m$ is the space of all models built with hyperparameters $m$
- $\pi_{m,t}$ is the model obtained after training the model $\pi_{m,0}$ with hyperparameters $m$
- $\pi_m^* = \arg \min_{\pi \in \Pi_m} R(\pi)$
Empirical Risk Minimization¶
Empirical Risk : $$R_\text{emp}(\pi) = \frac{\sum^n_{i = 1} L(\pi(x_i),y_i) }{n} \text{.}$$
McDiarmid inequality for any $\epsilon > 0$ : $$ P( |\sum^n_{i = 1} L(\pi(x_i),y_i) - nR(\pi))| \geq \epsilon) \leq \text{exp}\left( \frac{2\epsilon^2}{\sum^n_{i=1} c^2} \right)\text{,}$$ where $c > | L(\pi(x_i),y_i) - L(\pi(x_j),y_j) |$
Parameters Search¶
From Random search for hyper-parameter optimization by James Bergstra and Yoshua Bengio, JMLR vol 13, 2019
A Classification Problem¶
In [2]:
import sklearn
from sklearn import datasets
digits = datasets.load_digits()
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(digits.data, digits.target, test_size= 0.5, shuffle = True, random_state = 0)
In [3]:
from sklearn import linear_model
param = {
"max_iter" : 1000,
"tol" : 1e-3,
}
# Définition du modèle
sgd = linear_model.SGDClassifier(**param)
# Optimisation des paramètres du modèle
sgd = sgd.fit(x_train, y_train)
sklearn.metrics.accuracy_score(sgd.predict(x_test), y_test)
Out[3]:
Random Search¶
In [4]:
import numpy as np
from tqdm import tqdm_notebook as tqdm
best_model = None
max_accuracy = 0
for i in tqdm(range(100)):
param = {"max_iter" : np.random.randint(2000,100000),\
"penalty" : np.random.choice(["none", "l2", "l1", "elasticnet"]),\
"alpha": np.random.uniform(0.00001,0.001),\
"tol" : np.random.uniform(0,1e-3)}
sgd = linear_model.SGDClassifier(**param)
sgd = sgd.fit(x_train, y_train)
accuracy_test = sklearn.metrics.accuracy_score(y_test, sgd.predict(x_test))
if accuracy_test > max_accuracy:
max_accuracy = accuracy_test
best_model = sgd
max_accuracy
Out[4]:
Minimal MLflow logging¶
import mlflow
mlflow.set_experiment("Experiment")
with mlflow.start_run():
...
mlflow.log_param("param", param_value)
for i, ...
#Do Stuff
...
mlflow.log_metric("metric", metric_value, i)
artifact_path = mlflow.get_artifact_uri()
Then start the tracking server with mlflow ui -p $port
In [5]:
import mlflow, json, os
mlflow.set_experiment("Digits")
for i in tqdm(range(100)):
with mlflow.start_run():
param = {"max_iter" : np.random.randint(2000,100000),
"penalty" : np.random.choice(["none", "l2", "l1", "elasticnet"]),
"alpha": np.random.uniform(0.00001,0.001),
"tol" : np.random.uniform(0,1e-3)}
for i_key, i_value in param.items():
mlflow.log_param(i_key, i_value)
sgd = linear_model.SGDClassifier(**param)
sgd = sgd.fit(x_train, y_train)
mlflow.log_metric("acc_train",\
sklearn.metrics.accuracy_score(y_train, sgd.predict(x_train)))
mlflow.log_metric("acc_test",\
sklearn.metrics.accuracy_score(y_test, sgd.predict(x_test)))
artifact_path = mlflow.get_artifact_uri()
artifact_path = artifact_path.replace("%20", " ").replace("file://", "")
from joblib import dump, load
dump(sgd, os.path.join(artifact_path, "linear.w"))
with open(os.path.join(artifact_path, "param.json"), 'w') as f:
json.dump(param, f)
In [6]:
df = mlflow.search_runs()
df.head()
Out[6]:
In [9]:
plot = df["metrics.acc_test"].hist().plot()
In [8]:
best_idx = df["metrics.acc_test"].idxmax()
df.iloc[best_idx]
Out[8]:
Don't train on your test!¶
- Don't forget that the Random Search is somewhat part of your training
- Test your final model with data not used during the random search