Why not login to Qiita and try out its useful features?

We'll deliver articles that match you.

You can read useful information later.

6
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

【備忘録】ハイパーパラメータチューニングテンプレート【Optuna + Keras】

Posted at

はじめに

Kerasでニューラルネットワークモデルを構築する際の、叩き台としてOptunaを使用してある程度の性能が出せるネットワークの設定を発見するコード。毎回作り直すのがめんどくさいため、ここに備忘録として残しておく。

コード

# 最適化アルゴリズム
optimizer = trial.suggest_categorical('optimizer', ['MomentumSGD', 'Adam'])

# optimizerを代入
model.compile(optimizer=optimizer,loss="categorical_crossentropy",metrics=["accuracy"])
  • 目的関数のreturnの部分でどんな値を返して、それを最大化(maximize)したいのか、最小化(minimize)したいのか、決定する。
  • train_test_splitなどであらかじめデータセットを分けておいて、study.optimizeに代入する
  • 入力層のインプットの形状を指定する必要はないが、出力層は指定する
  • study.optimize(lambda trial: objective(trial, 変数1、変数2...), n_trials=100)という書式で目的関数にtrial以外の引数を入れることができる
  • optuna.logging.enable_default_handler()でOptunaのログを出力できる
  • 逆にログを吐き出したくない場合はoptuna.logging.disable_default_handler()を書く
# ライブラリのインポート
import optuna
import tensorflow.keras
from  tensorflow.keras.models import Sequential
from  tensorflow.keras.layers import Dense, Dropout, Flatten
from  tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.model_selection import train_test_split

# モデルの作成
def create_model(trial):
 
    # 層の数を選択する
    n_layers = trial.suggest_int("n_layers", 3, 7)
    model = Sequential()
    
    for i in range(n_layers):
        # Dense
        num_hidden = int(trial.suggest_loguniform(f"n_units_l{i}",4,512))
        model.add(Dense(num_hidden, activation="relu"))

        # ドロップアウト層
        dropout = trial.suggest_uniform(f"dropout_l{i}", 0.2, 0.5)
        model.add(Dropout(rate=dropout))
    
    # 出力層
    model.add(Dense(1))

    # モデルのコンパイル
    model.compile(loss='mse',
              optimizer=Adam(),
              metrics=['mse'])

    return model

# 目的関数
def objective(trial,X_train, X_test, y_train, y_test):
    # 前のセッションをクリアする
    keras.backend.clear_session()

    # モデル作成
    model = create_model(trial)

    # モデルの学習
    history = model.fit(X_train,y_train,epochs=10,batch_size=1024,validation_data=(X_test, y_test))

    return 1 - history.history["val_mse"][0]

# 最適化
study = optuna.create_study(direction='maximize')
study.optimize(lambda trial: objective(trial, X_train, X_test, y_train, y_test), n_trials=100)

実際のログ(例)

  • 非常にわかりやすい
Epoch 1/10
1734/1734 [==============================] - 5s 3ms/step - loss: 0.0765 - mse: 0.0765 - val_loss: 0.0439 - val_mse: 0.0439
Epoch 2/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0206 - mse: 0.0206 - val_loss: 0.0411 - val_mse: 0.0411
Epoch 3/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0173 - mse: 0.0173 - val_loss: 0.0374 - val_mse: 0.0374
Epoch 4/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0166 - mse: 0.0166 - val_loss: 0.0387 - val_mse: 0.0387
Epoch 5/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0165 - mse: 0.0165 - val_loss: 0.0360 - val_mse: 0.0360
Epoch 6/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0164 - mse: 0.0164 - val_loss: 0.0385 - val_mse: 0.0385
Epoch 7/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0163 - mse: 0.0163 - val_loss: 0.0393 - val_mse: 0.0393
Epoch 8/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0163 - mse: 0.0163 - val_loss: 0.0415 - val_mse: 0.0415
Epoch 9/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0161 - mse: 0.0161 - val_loss: 0.0407 - val_mse: 0.0407
Epoch 10/10
1734/1734 [==============================] - 4s 2ms/step - loss: 0.0162 - mse: 0.0162 - val_loss: 0.0421 - val_mse: 0.0421
[I 2021-12-08 17:32:15,216] Trial 0 finished with value: 0.956090159714222 and parameters: {'n_layers': 4, 'n_units_l0': 136.4133153963306, 'dropout_l0': 0.2762079388284835, 'n_units_l1': 6.517867697876512, 'dropout_l1': 0.3674686044843562, 'n_units_l2': 89.82269872356912, 'dropout_l2': 0.4010645579365826, 'n_units_l3': 48.85781763881269, 'dropout_l3': 0.4415658912950136}. Best is trial 0 with value: 0.956090159714222.
Epoch 1/10
1734/1734 [==============================] - 17s 9ms/step - loss: 0.0341 - mse: 0.0341 - val_loss: 0.0972 - val_mse: 0.0972
Epoch 2/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0076 - mse: 0.0076 - val_loss: 0.0972 - val_mse: 0.0972
Epoch 3/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0053 - mse: 0.0053 - val_loss: 0.0830 - val_mse: 0.0830
Epoch 4/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0048 - mse: 0.0048 - val_loss: 0.0842 - val_mse: 0.0842
Epoch 5/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0044 - mse: 0.0044 - val_loss: 0.0686 - val_mse: 0.0686
Epoch 6/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0042 - mse: 0.0042 - val_loss: 0.0788 - val_mse: 0.0788
Epoch 7/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0040 - mse: 0.0040 - val_loss: 0.0641 - val_mse: 0.0641
Epoch 8/10
1734/1734 [==============================] - 17s 10ms/step - loss: 0.0039 - mse: 0.0039 - val_loss: 0.0769 - val_mse: 0.0769
Epoch 9/10
1734/1734 [==============================] - 17s 10ms/step - loss: 0.0038 - mse: 0.0038 - val_loss: 0.0817 - val_mse: 0.0817
Epoch 10/10
1734/1734 [==============================] - 16s 9ms/step - loss: 0.0036 - mse: 0.0036 - val_loss: 0.0796 - val_mse: 0.0796
[I 2021-12-08 17:34:59,020] Trial 1 finished with value: 0.9027610197663307 and parameters: {'n_layers': 6, 'n_units_l0': 82.35173067277402, 'dropout_l0': 0.3951302536642787, 'n_units_l1': 152.68594208166238, 'dropout_l1': 0.4686668540109759, 'n_units_l2': 51.59443308144836, 'dropout_l2': 0.24115393032237953, 'n_units_l3': 254.7841497780783, 'dropout_l3': 0.3725568565724888, 'n_units_l4': 47.12550679065958, 'dropout_l4': 0.46566366782678575, 'n_units_l5': 262.22459761792135, 'dropout_l5': 0.3287760226636271}. Best is trial 0 with value: 0.956090159714222.
............続く
6
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up

Comments

No comments

Let's comment your feelings that are more than good

6
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

Login to continue?

Login or Sign up with social account

Login or Sign up with your email address