この記事で最終的に何ができる？

あれこれ読んで, なーんだやりたかったことと違うじゃん…みたいなことあると悲しくなるので, 最初に何ができるか見てから記事を読むか判断してください.

google colabでGPU利用で機械学習
多機能なplot機能
iPhoneでハイパーパラメータの確認

- iPhoneで実行中のデータの可視化

- iPhoneで実行中の出力の確認

内容

手元に使い物になるGPUがないとき, Google Colabを使うという手があります.
そこで, 今回はGoogle Colabを便利に使うための環境を構築します.

要求内容

原則は.ipynbでなく.pyで作る

スマートフォンでリアルタイムで結果を見れるようにする
Tensorboardで可視化する

1は, .ipynb形式だと他ファイルのimportが面倒なのとlocalで回すときにjupyterを使わなければならないのを回避したいということです.

機械学習フレームワークとしてPyTorchを使います.

基本手順

最初に手順を説明します.

まずはlocalで実行するときと全く同じようにコードを書く

スマートフォンで結果を見れるようにhyperdashを, webで結果を見れるようにtensorboardを導入し, その部分のコードを追加する
これらのファイルをGoogle Driveにアップロードする(または1で最初からGoogle Driveをlocalにマウントして作成)
.pyファイルを実行するための実行スクリプトexec.ipynbを作成・実行する

hyperdashを使うとlogやパラメータ, グラフをリアルタイムでスマートフォンで見ることが出来ます.
ただし, これは1つのデータのみしか見ることが出来ず, パラメータを変えた場合の比較などが出来ません. また, plotのsmoothingやデータの取得なども出来ないので, ここは本家のtensorboardを使います.

データセットとネットワーク構造

今回はThe CIFAR-10 datasetをCNNで分類するタスクを選びました.
CNNのネットワーク構造はTrain a simple deep CNN on the CIFAR10 small images dataset.をPyTorchに書き換えました.
層の構成は次の通りです.

# network構造
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 32, 32]             896
            Conv2d-2           [-1, 32, 30, 30]           9,248
         MaxPool2d-3           [-1, 32, 15, 15]               0
           Dropout-4           [-1, 32, 15, 15]               0
            Conv2d-5           [-1, 64, 15, 15]          18,496
            Conv2d-6           [-1, 64, 13, 13]          36,928
         MaxPool2d-7             [-1, 64, 6, 6]               0
           Dropout-8             [-1, 64, 6, 6]               0
            Linear-9                  [-1, 512]       1,180,160
          Dropout-10                  [-1, 512]               0
           Linear-11                   [-1, 10]           5,130
              CNN-12                   [-1, 10]               0
================================================================

ソースコード全体

まずはファイル構成を見てください.

.
├── .hyperdash
├── datasets
├── cifar10-cnn-pytorch
        ├── exec.ipynb
        ├── log
        │   ├── 200304104116
        │   │   ├── csv
        │   │   │   ├── args.csv
        │   │   │   └── results.csv
        │   │   ├── model
        │   │   │   └── model.tar
        │   │   └── tensorboard
        │   │       └── events.out.tfevents.1583318478.9998e7dcdb7c.342.0
        │   ├── 200304111548
        .   .
        .   .
        .   .
        └── src
            ├── main.py
            ├── net.py
            └── utils.py

args.csvはargparseで指定したハイパーパラメータの保存, results.csvは出力結果の保存, model.tarはcheckpointの保存, tensorboardはtensorboardで描画するために保存するものです.

main.py

import os
import datetime
import csv
import torch
import torchvision
import torch.optim as optim
import torch.nn as nn
from torch.utils.tensorboard import SummaryWriter
from torchsummary import summary
from pathlib import Path
from time import time
from net import CNN
from utils import get_loader, plot_classes_preds, get_args, matplotlib_imshow


def train(train_loader, model, criterion, optimizer, device, writer, epoch,
          classes):
    """Train a neural networks.
    """
    correct = 0
    total = 0
    running_loss = 0.0
    model.train()
    for i, data in enumerate(train_loader):
        inputs, labels = data[0].to(device), data[1].to(device)
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        _, predicted = torch.max(outputs, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
    acc = correct / total
    writer.add_figure('prediction vs. actuals', plot_classes_preds(model,
                                                                   inputs,
                                                                   labels,
                                                                   classes),
                      global_step=epoch * len(train_loader))
    loss = running_loss / len(train_loader)
    return loss, acc


def eval(eval_loader, model, criterion, device):
    """Evaluating a trained network.
    """
    model.eval()
    correct = 0
    total = 0
    running_loss = 0.0
    with torch.no_grad():
        for data in eval_loader:
            inputs, labels = data[0].to(device), data[1].to(device)
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            running_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    acc = correct / total
    loss = running_loss / len(eval_loader)
    return loss, acc


def main():
    start_time = time()
    args = get_args()
    if args.checkpoint_dir_name:
        dir_name = args.checkpoint_dir_name
    else:
        dir_name = datetime.datetime.now().strftime('%y%m%d%H%M%S')
    path_to_dir = Path(__file__).resolve().parents[1]
    path_to_dir = os.path.join(path_to_dir, *['log', dir_name])
    os.makedirs(path_to_dir, exist_ok=True)
    # tensorboard
    path_to_tensorboard = os.path.join(path_to_dir, 'tensorboard')
    os.makedirs(path_to_tensorboard, exist_ok=True)
    writer = SummaryWriter(path_to_tensorboard)
    # model saving
    os.makedirs(os.path.join(path_to_dir, 'model'), exist_ok=True)
    path_to_model = os.path.join(path_to_dir, *['model', 'model.tar'])
    # csv
    os.makedirs(os.path.join(path_to_dir, 'csv'), exist_ok=True)
    path_to_results_csv = os.path.join(path_to_dir, *['csv', 'results.csv'])
    path_to_args_csv = os.path.join(path_to_dir, *['csv', 'args.csv'])
    if not args.checkpoint_dir_name:
        with open(path_to_args_csv, 'a') as f:
            args_dict = vars(args)
            param_writer = csv.DictWriter(f, list(args_dict.keys()))
            param_writer.writeheader()
            param_writer.writerow(args_dict)

    # logging using hyperdash
    if not args.no_hyperdash:
        from hyperdash import Experiment
        exp = Experiment('Classification task on CIFAR10 dataset with CNN')
        for key in vars(args).keys():
            exec("args.%s = exp.param('%s', args.%s)" % (key, key, key))
    else:
        exp = None

    path_to_dataset = os.path.join(Path(__file__).resolve().parents[2],
                                   'datasets')
    os.makedirs(path_to_dataset, exist_ok=True)
    train_loader, eval_loader, classes = get_loader(batch_size=args.batch_size,
                                                    num_workers=args.num_workers,
                                                    path_to_dataset=path_to_dataset)

    # show some of the training images, for fun.
    dataiter = iter(train_loader)
    images, labels = dataiter.next()
    img_grid = torchvision.utils.make_grid(images)
    matplotlib_imshow(img_grid)
    writer.add_image('four_CIFAR10_images', img_grid)

    # define a network, loss function and optimizer
    model = CNN()
    writer.add_graph(model, images)
    model = torch.nn.DataParallel(model)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=args.lr,
                          momentum=args.momentum)
    start_epoch = 0
    # resume training
    if args.checkpoint_dir_name:
        print('\nLoading the model...')
        checkpoint = torch.load(path_to_model)
        model.state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        start_epoch = checkpoint['epoch'] + 1
    summary(model, input_size=(3, 32, 32))
    model.to(args.device)

    # train the network
    print('\n--------------------')
    print('Start training and evaluating the CNN')
    for epoch in range(start_epoch, args.n_epoch):
        start_time_per_epoch = time()
        train_loss, train_acc = train(train_loader, model, criterion,
                                      optimizer, args.device, writer, epoch,
                                      classes)
        eval_loss, eval_acc = eval(eval_loader, model, criterion, args.device)
        elapsed_time_per_epoch = time() - start_time_per_epoch
        result_dict = {'epoch': epoch, 'train_loss': train_loss,
                       'eval_loss': eval_loss, 'train_acc': train_acc,
                       'eval_acc': eval_acc,
                       'elapsed time': elapsed_time_per_epoch}
        with open(path_to_results_csv, 'a') as f:
            result_writer = csv.DictWriter(f, list(result_dict.keys()))
            if epoch == 0: result_writer.writeheader()
            result_writer.writerow(result_dict)
        # checkpoint
        torch.save({'epoch': epoch, 'model_state_dict': model.state_dict(),
                    'optimizer_state_dict': optimizer.state_dict()},
                   path_to_model)
        if exp:
            exp.metric('train loss', train_loss)
            exp.metric('eval loss', eval_loss)
            exp.metric('train acc', train_acc)
            exp.metric('eval acc', eval_acc)
        else:
            print(result_dict)

        writer.add_scalar('loss/train_loss', train_loss,
                          epoch * len(train_loader))
        writer.add_scalar('loss/eval_loss', eval_loss,
                          epoch * len(eval_loader))
        writer.add_scalar('acc/train_acc', train_acc,
                          epoch * len(train_loader))
        writer.add_scalar('acc/eval_acc', eval_acc,
                          epoch * len(eval_loader))

    elapsed_time = time() - start_time
    print('\nFinished Training, elapsed time ===> %f' % elapsed_time)
    if exp:
        exp.end()
    writer.close()


if __name__ == '__main__':
    main()

utils.py

import os
import argparse
import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn.functional as F
from matplotlib import pyplot as plt


def get_loader(batch_size, num_workers, path_to_dataset):
    """Get a train loader and evaluation loader for CIFAR10.
    """
    os.makedirs(path_to_dataset, exist_ok=True)

    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
    train_data = torchvision.datasets.CIFAR10(
        root=path_to_dataset,
        train=True,
        download=True,
        transform=transform
    )
    eval_data = torchvision.datasets.CIFAR10(
        root=path_to_dataset,
        train=False,
        download=True,
        transform=transform
    )
    train_loader = torch.utils.data.DataLoader(
        train_data,
        batch_size=batch_size,
        shuffle=True,
        num_workers=num_workers
    )
    eval_loader = torch.utils.data.DataLoader(
        eval_data,
        batch_size=batch_size,
        shuffle=False,
        num_workers=num_workers
    )
    classes = (
        'plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship',
        'truck')
    return train_loader, eval_loader, classes


def images2probs(model, images):
    """
    Generates predictions and corresponding probabilities from a trained
    network and a list of images
    """
    output = model(images)
    _, predicted = torch.max(output, 1)
    preds = np.squeeze(predicted.cpu().numpy())
    probs = [F.softmax(el, dim=0)[i].item() for i, el in
             zip(preds, output)]
    return preds, probs


def plot_classes_preds(model, images, labels, classes):
    """
    Generates matplotlib Figure using a trained network, along with images
    and labels from a batch, that shows the network's top prediction along
    with its probability, alongside the actual label, coloring this
    information based on whether the prediction was correct or not.
    Uses the "images_to_probs" function.
    """
    preds, probs = images2probs(model, images)
    fig = plt.figure(figsize=(12, 48))
    for idx in range(4):
        ax = fig.add_subplot(1, 4, idx + 1, xticks=[], yticks=[])
        color = 'green' if preds[idx] == labels[idx].item() else 'red'
        title_str = '%s, %.1f \n(label: %s)' % (
            classes[preds[idx]], probs[idx] * 100.0, classes[labels[idx]])
        matplotlib_imshow(images[idx])
        ax.set_title(title_str, color=color)
    return fig


def get_args():
    """Get  hyperparameters from command line.

    Returns:
        args (argparse): Parser for command-line options.

    """
    parser = argparse.ArgumentParser(description='CNN')
    parser.add_argument('--batch_size', default=4, type=int, help='')
    parser.add_argument('--num_workers', default=2, type=int)
    parser.add_argument('--n_epoch', default=2, type=int)
    parser.add_argument('--lr', default=0.001, type=float)
    parser.add_argument('--momentum', default=0.9, type=float)
    parser.add_argument('--no_hyperdash', default=False, action='store_true')
    parser.add_argument('--checkpoint_dir_name', default=None, type=str)
    parser.add_argument('--no_cuda', action='store_true', default=False)
    args = parser.parse_args()
    args.device = torch.device('cpu')
    if not args.no_cuda and torch.cuda.is_available():
        args.device = torch.device('cuda:0')
        if torch.cuda.device_count() > 1:
            gpu_ids = [id for id in range(len(torch.cuda.device_count()))]
            args.device = torch.device(f'cuda:{gpu_ids[0]}')
    print('####################')
    print('device ===> ', args.device)
    print('####################')
    return args


def matplotlib_imshow(img, one_channnel=False):
    """Helper function to show an image.
    """
    if one_channnel:
        img = img.mean(dim=0)
    img = img / 2 + 0.5
    npimg = img.cpu().numpy()
    if one_channnel:
        plt.imshow(npimg, cmap='Greys')
    else:
        plt.imshow(np.transpose(npimg, (1, 2, 0)))

net.py

import torch.nn as nn
import torch.nn.functional as F


class CNN(nn.Module):
    """Click https://keras.io/examples/cifar10_cnn/ for reference.
    """

    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, (3, 3), padding=(1, 1))
        self.conv2 = nn.Conv2d(32, 32, (3, 3))
        self.conv3 = nn.Conv2d(32, 64, (3, 3), padding=(1, 1))
        self.conv4 = nn.Conv2d(64, 64, (3, 3))
        self.fc1 = nn.Linear(6 * 6 * 64, 512)
        self.fc2 = nn.Linear(512, 10)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.dropout1 = nn.Dropout(p=0.25)
        self.dropout2 = nn.Dropout(p=0.50)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool1(x)
        x = self.dropout1(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = self.pool1(x)
        x = self.dropout1(x)
        x = x.view(-1, 6 * 6 * 64)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)

        return x

ソースコードの実行

exec.ipynb

from google.colab import drive
drive.mount('/content/drive')

# 以下は1つのセルで実行
# optionは好きな値を設定
%%bash
nvidia-smi
pip install hyperdash
cp -r /content/drive/My\ Drive/Colab\ Notebooks/.hyperdash ~/
cd /content/drive/My\ Drive/Colab\ Notebooks/cifar10-cnn-pytorch/src
python main.py --n_epoch 150 --lr 0.001 --batch_size 64

hyperdashを使えるようにするまで

iPhone等のスマートフォンで結果を見るのにhyperdashを使いました.
tensorboardでもiOSで表示できる人もいるみたいですが, 上手く行きませんでした...
使い方は結構簡単で, 基本先のgithubのページを見れば分かります.
まずはinstallとsignupから.

# pipでinstall
$ pip install hyperdash

# gitアカウントでsign up
$ hd signup --github

あとは, 使い方.


from hyperdash import Experiment


# create
exp = Experiment('Sample')
# record the params
param = exp.param('param', param)
# record the numerical data
exp.metric('data', data)
# clean up
exp.close()

これをgoogle colab上で使うためにはAPI keyを設定する必要があります.
colab上で環境変数を設定すれば良いみたいですが, この辺り詳しくなくて上手く行きませんでした.
なので, local上~/.hyperdashをgoogle driveにuploadしてそれをcolab上で~/にコピーするやり方にしました.

!cp -r /content/drive/My\ Drive/Colab\ Notebooks/.hyperdash ~/

の部分になります.

tensorboardを使えるようにするまで

tensorboardは元々はTensorFlowで使われていたもので, PyTorchではtensorboard Xを経て, 最近公式にサポートされたものです.
smoothingしたり, plotするデータを選んだり, 画像やネットワーク構造まで描画できます.
PyTorchの公式TutorialのVISUALIZING MODELS, DATA, AND TRAINING WITH TENSORBOARDが分かりやすいです.
これも使い方は基本的にTutorialを見れば分かります.

from torch.utils.tensorboard import SummaryWriter

# create
writer = SummaryWriter(log_dir)
# record the image (from tensor, numpy array,...)
writer.add_image(tag, img_tensor)
# record the network architecuture
writer.add_graph(model, input)
# record the image (from matplotlib)
writer.add_figure(tag, figure)
# record the numerical data
writer.add_scalar(tag, scalar_value)
# clean up
writer.close()

tagの部分をloss/train_lossのようにグルーピングすると見るとき分かりやすいです.