概要

機械学習におけるパラメータチューニングには、グリッドサーチがよく使われる。
しかし、このような総当り的手法では計算に時間がかかる。
そこで、自動で効率よく最適なパラメータを選択する手法として、ベイズ最適化が考えられる。
今回、SVMによる分類問題にグリッドサーチとベイズ最適化を適用し、比較を行った。

結果

手法	計算時間（秒）	試行回数	F1-Score
ベイズ最適化	12.902	20	0.99
グリッドサーチ	41.947	132	0.98

結論

同等の予測精度を得るのに要した時間は、ベイズ最適化の方が短い。ベイズ最適化の効率の良さが実際に確認できた。

今後の課題

データ

scikit-learnが提供するきれいなサンプルデータ（digits）を用いた。実データでも試してみたい。

アルゴリズム

ディープラーニングのような多大な計算時間を要するアルゴリズムでこそ、ベイズ最適化の真価が発揮される。実際に試してみたい。

ライブラリ選定

今回はPythonで手軽に使えそうという理由でGPyOptを選択。しかし、他にも色々な候補があるため、比較検討の余地あり。
・skopt
・BayesianOptimization
・Spearmint
・MOE
etc...

備考

カテゴリー変数

ベイズ最適化では連続変数はもちろん、離散変数も扱うことができる。
カテゴリー変数の場合は、ダミー変数にするなどの前処理で対応するのが妥当に思える。
※今回は、単に場合分けを行った。
GPyOpt: mixing different types of variables

実行環境

minicondaをインストール後、下記コマンドにより環境構築。

conda create -n myenv --file conda.req
source activate myenv
pip install -r pip.req

conda.req

matplotlib=2.0.2
numpy=1.12.1
python=3.5.3
scikit-learn=0.18.2
scipy=0.19.1

pip.req

GPy==1.7.7
GPyOpt==1.0.3

実行結果

time python bayes_opt.py
# Tuning hyper-parameters for f1

(...省略...)

20 experiments were performed.

Best parameters set found on development set:

{'kernel': 'rbf', 'gamma': 0.0011320544666204339, 'C': 604.8637015078541}

Detailed classification report:

The model is trained on the full development set.
The scores are computed on the full evaluation set.

             precision    recall  f1-score   support

          0       1.00      1.00      1.00        89
          1       0.97      1.00      0.98        90
          2       1.00      0.98      0.99        92
          3       1.00      1.00      1.00        93
          4       1.00      1.00      1.00        76
          5       0.99      0.98      0.99       108
          6       0.99      1.00      0.99        89
          7       0.99      1.00      0.99        78
          8       1.00      0.98      0.99        92
          9       0.99      0.99      0.99        92

avg / total       0.99      0.99      0.99       899


real    0m12.902s
user    0m0.000s
sys     0m0.031s

time python grid_search.py
# Tuning hyper-parameters for f1

(...省略...)

132 experiments were performed.

Best parameters set found on development set:

{'kernel': 'rbf', 'gamma': 0.00016666666666666666, 'C': 100}

Detailed classification report:

The model is trained on the full development set.
The scores are computed on the full evaluation set.

             precision    recall  f1-score   support

          0       1.00      1.00      1.00        89
          1       0.95      1.00      0.97        90
          2       0.99      0.99      0.99        92
          3       0.97      0.99      0.98        93
          4       1.00      1.00      1.00        76
          5       0.96      0.97      0.97       108
          6       0.99      0.99      0.99        89
          7       0.99      1.00      0.99        78
          8       1.00      0.91      0.95        92
          9       0.97      0.96      0.96        92

avg / total       0.98      0.98      0.98       899


real    0m41.947s
user    0m0.000s
sys     0m0.031s

コード

bayes_opt.py

import GPyOpt
import numpy as np
from numpy.random import seed
from sklearn import datasets
from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
from sklearn.metrics import classification_report
from sklearn.svm import SVC

seed(0)

digits = datasets.load_digits()

n_samples = len(digits.images)
X = digits.images.reshape((n_samples, -1))
y = digits.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=0)

kernel2domains = {
    'rbf':[
        {'name': 'C', 'type': 'continuous', 'domain': (1, 1100)},
        {'name': 'gamma', 'type': 'continuous', 'domain': (0.00005, 0.0015)}
    ],
    'linear': [
        {'name': 'C', 'type': 'continuous', 'domain': (1, 1100)},
    ],
}

def _get_model(param, kernel):
    _param = _refine_param(param, kernel)
    model = SVC(**_param)
    return model

def _refine_param(param, kernel):
    assert kernel in ['rbf', 'linear']
    if kernel == 'rbf':
        ret = {'kernel': kernel, 'C': param[0], 'gamma': param[1]}
    else:
        ret = {'kernel': kernel, 'C': param[0]}
    return ret

def _optimize(params, kernel):
    scores = np.zeros((params.shape[0], 1))
    for i, param in enumerate(params):
        model = _get_model(param, kernel)
        y_pred = cross_val_predict(model, X_train, y_train, cv=5)
        scores[i] -= f1_score(y_train, y_pred, average='macro')
    return scores

print("# Tuning hyper-parameters for f1")
print()

bests_per_kernel = []
for k, d in kernel2domains.items():
    f = lambda x: _optimize(x, k)
    opt = GPyOpt.methods.BayesianOptimization(f=f, domain=d)
    opt.run_optimization(max_iter=15)
    idx = np.argmin(opt.Y)
    x_best = opt.X[idx]
    best_score = opt.Y[idx]
    bests_per_kernel.append((x_best, best_score, opt, k))

x_best, _, optimizer, kernel = min(bests_per_kernel, key=lambda x: x[1])

print("Grid scores on development set:")
print()
for param, score in zip(optimizer.X, optimizer.Y):
    _score = -score
    _param = _refine_param(param, kernel)
    print("%0.3f for %r" % (_score, _param))
print()

print("%d experiments were performed." % len(optimizer.X))
print()

print("Best parameters set found on development set:")
print()
print(_refine_param(x_best, kernel))
print()

print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
clf = _get_model(x_best, kernel)
clf.fit(X_train, y_train)
y_true, y_pred = y_test, clf.predict(X_test)
print(classification_report(y_true, y_pred))
print()

grid_search.py

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.svm import SVC

digits = datasets.load_digits()

n_samples = len(digits.images)
X = digits.images.reshape((n_samples, -1))
y = digits.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=0)

C = [1, 10, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]
_gamma = [1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000]
gamma = [1/i for i in _gamma]
tuned_parameters = [{'kernel': ['rbf'], 'gamma': gamma, 'C': C},
                    {'kernel': ['linear'], 'C': C}]

print("# Tuning hyper-parameters for f1")
print()

clf = GridSearchCV(SVC(C=1), tuned_parameters, cv=5, scoring='f1_macro')
clf.fit(X_train, y_train)

print("Grid scores on development set:")
print()
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r"
          % (mean, std * 2, params))
print()

print("%d experiments were performed." % len(means))
print()

print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()

print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
y_true, y_pred = y_test, clf.predict(X_test)
print(classification_report(y_true, y_pred))
print()