ChainerからPyTorchへ乗り換えるため、PyTorchを完全にいちから使い始めてみた結果。MNISTの手書き数字の識別を行う学習器をとりあえず作成(NNモデルはConv4層→FC2層)、動作を確認、98%前後の精度(Accuracy)となり、良好。

PyTorchのインストール試行

成功の模様。

https://pytorch.org/get-started/locally/
1.5 Stable、Windows、Python3.7、CUDAなし(CPU)の設定のコマンドを実行。

!pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.5.0+cpu
  Downloading https://download.pytorch.org/whl/cpu/torch-1.5.0%2Bcpu-cp37-cp37m-win_amd64.whl (109.2MB)
Collecting torchvision==0.6.0+cpu
  Downloading https://download.pytorch.org/whl/cpu/torchvision-0.6.0%2Bcpu-cp37-cp37m-win_amd64.whl (456kB)
Requirement already satisfied: numpy in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torch==1.5.0+cpu) (1.16.5+mkl)
Requirement already satisfied: future in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torch==1.5.0+cpu) (0.17.0)
Requirement already satisfied: pillow>=4.1.1 in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torchvision==0.6.0+cpu) (6.1.0)
Installing collected packages: torch, torchvision
  Found existing installation: torch 1.3.1+cpu
    Uninstalling torch-1.3.1+cpu:
      Successfully uninstalled torch-1.3.1+cpu
  Found existing installation: torchvision 0.4.2+cpu
    Uninstalling torchvision-0.4.2+cpu:
      Successfully uninstalled torchvision-0.4.2+cpu
Successfully installed torch-1.5.0+cpu torchvision-0.6.0+cpu
Note: you may need to restart the kernel to use updated packages.


WARNING: You are using pip version 19.2.3, however version 20.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

!pip show torch

Name: torch
Version: 1.5.0+cpu
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages
Requires: future, numpy
Required-by: torchvision

Libs

import numpy as np
import matplotlib.pyplot as plt

import torch
import torchvision

PyTorch初歩試用

参考:

https://qiita.com/miyamotok0105/items/1fd1d5c3532b174720cd
https://qiita.com/north_redwing/items/30f9619f0ee727875250
https://www.hellocybernetics.tech/entry/2017/10/19/070522

# sec: 行列生成のテスト

torch.rand(5, 3)

tensor([[0.7130, 0.1860, 0.6266],
        [0.9275, 0.5629, 0.7507],
        [0.7500, 0.7341, 0.9597],
        [0.5301, 0.7026, 0.3152],
        [0.2197, 0.3942, 0.8452]])

# sec: 勾配計算のテスト

x = torch.arange(4, dtype=torch.float32).view(2, 2)
print(x)

x.requires_grad_(True)
print(x)

y = x**2 - 2*x + 1
print(y)

y_ave = y.mean()
print(y_ave)

y_ave.backward()
print(x.grad)
print(y.grad)

tensor([[0., 1.],
        [2., 3.]])
tensor([[0., 1.],
        [2., 3.]], requires_grad=True)
tensor([[1., 0.],
        [1., 4.]], grad_fn=<AddBackward0>)
tensor(1.5000, grad_fn=<MeanBackward0>)
tensor([[-0.5000,  0.0000],
        [ 0.5000,  1.0000]])
None

torchvision初歩試用

# sec: MNISTデータの初回呼び出し ダウンロードのテスト

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True)
ds

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw\train-images-idx3-ubyte.gz

HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

Extracting trains/pytorch-mnist\MNIST\raw\train-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw\train-labels-idx1-ubyte.gz

HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

Extracting trains/pytorch-mnist\MNIST\raw\train-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw\t10k-images-idx3-ubyte.gz

HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

Extracting trains/pytorch-mnist\MNIST\raw\t10k-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw\t10k-labels-idx1-ubyte.gz

HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

Extracting trains/pytorch-mnist\MNIST\raw\t10k-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw
Processing...

..\torch\csrc\utils\tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.

Done!

Dataset MNIST
    Number of datapoints: 60000
    Root location: trains/pytorch-mnist
    Split: Train

# sec: MNISTデータの2回目呼び出し

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True)
ds

Dataset MNIST
    Number of datapoints: 60000
    Root location: trains/pytorch-mnist
    Split: Train

# sec: MNISTデータのデータ取り出し・表示

img = np.asarray(ds[0][0])
print(img[20:25, 10:15], img.shape)

img = np.asarray(ds[0][0].convert('F'))
print(img[20:25, 10:15], img.shape)

plt.imshow(img, cmap='gray', interpolation='None')
plt.show()

[[ 24 114 221 253 253]
 [213 253 253 253 253]
 [253 253 253 195  80]
 [253 244 133  11   0]
 [132  16   0   0   0]] (28, 28)
[[ 24. 114. 221. 253. 253.]
 [213. 253. 253. 253. 253.]
 [253. 253. 253. 195.  80.]
 [253. 244. 133.  11.   0.]
 [132.  16.   0.   0.   0.]] (28, 28)

# sec: TorchのTensorを出力する

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())

# sec: MNISTデータのデータ取り出し・表示

img = ds[0][0]
print(img[0, 20:25, 10:15], ds[0][0].shape)

plt.imshow(img[0, :, :], cmap='gray', interpolation='None')
plt.show()

tensor([[0.0941, 0.4471, 0.8667, 0.9922, 0.9922],
        [0.8353, 0.9922, 0.9922, 0.9922, 0.9922],
        [0.9922, 0.9922, 0.9922, 0.7647, 0.3137],
        [0.9922, 0.9569, 0.5216, 0.0431, 0.0000],
        [0.5176, 0.0627, 0.0000, 0.0000, 0.0000]]) torch.Size([1, 28, 28])

NNモデル

import torch.nn as nn
import torch.nn.functional as F

class MyConvNet1(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 5, 3, stride=1, padding=0)
        self.conv2 = nn.Conv2d(5, 10, 3, stride=1, padding=0)
        self.conv3 = nn.Conv2d(10, 15, 3, stride=1, padding=0)
        self.conv4 = nn.Conv2d(15, 20, 3, stride=1, padding=0)
        self.bn1 = nn.BatchNorm2d(5)
        self.bn2 = nn.BatchNorm2d(10)
        self.bn3 = nn.BatchNorm2d(15)
        self.bn4 = nn.BatchNorm2d(20)
        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(20*4*4, 20)
        self.fc2 = nn.Linear(20, 10)
        self.drop1 = nn.Dropout(0.5)
        self.drop2 = nn.Dropout(0.5)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = self.pool1(x)
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.relu(self.bn4(self.conv4(x)))
        x = self.pool2(x)
        x = x.view(-1, 20*4*4)
        x = F.relu(self.fc1(self.drop1(x)))
        x = self.fc2(self.drop2(x))
        return x

TargetNet = MyConvNet1

# sec: test (順伝播計算 ゼロ値を入力)

model = TargetNet()
y = model(torch.zeros(1, 1, 28, 28))
print(y)

# sec: test (順伝播計算 MNISTデータを入力)

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
y = model(ds[0][0].view(1, 1, 28, 28))
print(y)

tensor([[ 0.2101,  0.0317, -0.0457, -0.0219,  0.1032,  0.0080, -0.2413, -0.2003,
          0.0222, -0.0821]], grad_fn=<AddmmBackward>)
tensor([[-0.1699,  0.5580, -0.1396, -0.6204,  0.3966, -0.0115, -0.0553, -0.0156,
         -0.3096, -0.1288]], grad_fn=<AddmmBackward>)

学習

参考:

# sec: データセット

ds_train = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_train = torch.utils.data.DataLoader(ds_train, batch_size=128, shuffle=True, num_workers=2)
print(len(ds_train))

# sec: 設定

model = TargetNet()
criterion = nn.CrossEntropyLoss()
# case: SGD
# optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# case: Adam
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
# case: end
print(model)

60000
MyConvNet1(
  (conv1): Conv2d(1, 5, kernel_size=(3, 3), stride=(1, 1))
  (conv2): Conv2d(5, 10, kernel_size=(3, 3), stride=(1, 1))
  (conv3): Conv2d(10, 15, kernel_size=(3, 3), stride=(1, 1))
  (conv4): Conv2d(15, 20, kernel_size=(3, 3), stride=(1, 1))
  (bn1): BatchNorm2d(5, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn2): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn3): BatchNorm2d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn4): BatchNorm2d(20, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=320, out_features=20, bias=True)
  (fc2): Linear(in_features=20, out_features=10, bias=True)
  (drop1): Dropout(p=0.5, inplace=False)
  (drop2): Dropout(p=0.5, inplace=False)
)

# sec: 学習ループ

model.train()
for i_ep in range(10): # loop over the dataset multiple times

    loss_sum = 0.0
    for ds_i in dl_train:
        
        inputs, labels = ds_i

        optimizer.zero_grad()
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        loss.backward() # 逆伝播
        optimizer.step()

        loss_sum += loss.item()
    
    # sec: print
    
    print('%dep loss: %.5f' % (i_ep + 1, loss_sum / len(ds_train)))

n_ep = i_ep + 1
print('Finished Training')

1ep loss: 0.00700
2ep loss: 0.00359
3ep loss: 0.00310
4ep loss: 0.00290
5ep loss: 0.00272
6ep loss: 0.00258
7ep loss: 0.00247
8ep loss: 0.00243
9ep loss: 0.00237
10ep loss: 0.00232
Finished Training

バリデーション

# sec: データセット

ds_vali = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
ds_vali, _ = torch.utils.data.random_split(ds_vali, [1000, len(ds_vali) - 1000]) # イマイチな書き方
dl_vali = torch.utils.data.DataLoader(ds_vali, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_vali))

# sec: バリデーション例

model.eval() # 必要 trainのままだとdropoutの効果が残り精度劣化 90.0%で低迷
loss_sum = 0
n_correct = 0
for ds_i in dl_vali:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        n_correct += (preds == labels).sum()

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_vali), float(n_correct) / len(ds_vali) * 100))

loss: 2.43e-04, accuracy: 99.03

1回の評価結果

# sec: 画像を描画

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())

x, label = ds_test[np.random.randint(0, len(ds_test))]
plt.imshow(x[0], cmap='gray', interpolation="none")
plt.show()

# sec: 1回の評価

model.eval()
x = x[None] # 1軸を追加
with torch.no_grad():
    y = model(x)

print("予測ラベル:", y.numpy().argmax(axis=1)[0], "| 真値ラベル:", label)

予測ラベル: 8 | 真値ラベル: 8

テスト

# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))

# sec: テスト例

model.eval()
loss_sum = 0
list_pred, list_label = [], []
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        list_pred.extend(preds)
        list_label.extend(labels)

list_pred = np.array(list_pred)
list_label = np.array(list_label)

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), (list_pred == list_label).sum() / len(ds_test) * 100))

loss: 2.84e-04, accuracy: 99.03

他のWeb記事のMNISTデータを用いる評価結果で「精度98.13%」程度なので、結果は妥当。
http://torch.classcat.com/2018/07/26/pytorch-040-examples-mnist-cnn/

# sec: precision, recall, fscore, supportを見る

res_prf = sklearn.metrics.precision_recall_fscore_support(list_label, list_pred)

print("precision, recall, fscore, supportを見る:")
print(res_prf)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(10) - 0.2, res_prf[0], width=0.4, label="precision")
ax.bar(np.arange(10) + 0.2, res_prf[1], width=0.4, label="recall")
ax.plot(np.arange(10), res_prf[2], 'r_', ms=12, label="f")
ax.legend(fontsize=9, framealpha=0.5, labelspacing=0.2)
ax.set_ylim([0.97, 1.0])
plt.show()

precision, recall, fscore, supportを見る:
(array([0.99084435, 0.99297012, 0.98653846, 0.99303483, 0.98681542,
       0.9877095 , 0.99267782, 0.98736638, 0.99588054, 0.9889669 ]), array([0.99387755, 0.99559471, 0.99418605, 0.98811881, 0.99083503,
       0.99103139, 0.99060543, 0.98832685, 0.99281314, 0.97720515]), array([0.99235863, 0.99428069, 0.99034749, 0.99057072, 0.98882114,
       0.98936766, 0.99164054, 0.98784638, 0.99434447, 0.98305085]), array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009],
      dtype=int64))

# sec: 混同行列を見る

sklearn.metrics.confusion_matrix(list_label, list_pred)

array([[ 974,    0,    1,    0,    0,    0,    4,    1,    0,    0],
       [   0, 1130,    4,    0,    0,    0,    0,    1,    0,    0],
       [   1,    0, 1026,    1,    0,    0,    0,    2,    2,    0],
       [   0,    0,    1,  998,    0,    5,    0,    4,    2,    0],
       [   0,    0,    0,    0,  973,    0,    1,    0,    0,    8],
       [   1,    0,    0,    4,    0,  884,    2,    1,    0,    0],
       [   3,    3,    0,    0,    0,    3,  949,    0,    0,    0],
       [   0,    5,    5,    1,    0,    0,    0, 1016,    0,    1],
       [   2,    0,    3,    0,    0,    0,    0,    0,  967,    2],
       [   2,    0,    0,    1,   13,    3,    0,    4,    0,  986]],
      dtype=int64)

NNモデルパラメータ保存・読込評価

# sec: save

torch.save(model.state_dict(), "results/pytorch-mnist/model.pkl")

# sec: load

model_e = TargetNet()
model_e.load_state_dict(torch.load("results/pytorch-mnist/model.pkl"))

<All keys matched successfully>

model_e.state_dict()

OrderedDict([('conv1.weight', tensor([[[[-0.1193, -0.1967,  0.0337],
                        [-0.2103,  0.2067,  0.3707],
                        [-0.0072,  0.2572, -0.0811]]],
              
              
                      [[[ 0.2073,  0.0639, -0.2909],
                        [ 0.2432, -0.2954, -0.1945],
                        [ 0.0062, -0.1348,  0.1976]]],
              
              
                      [[[ 0.2244,  0.2394,  0.2029],
                        [ 0.0774, -0.1356, -0.1406],
                        [-0.1890, -0.1105, -0.2486]]],
              
              
                      [[[-0.0757, -0.1669, -0.3270],
                        [ 0.2795,  0.2242,  0.1612],
                        [-0.0702,  0.2889,  0.0423]]],
              
              
                      [[[-0.0672,  0.1286, -0.2042],
                        [-0.0348, -0.2120, -0.2940],
                        [-0.3711, -0.1960,  0.1867]]]])),
             ('conv1.bias',
              tensor([ 0.0740,  0.0062,  0.1578, -0.1330,  0.2352])),
             ('conv2.weight', tensor([[[[-0.1560, -0.1605, -0.0265],
                        [ 0.2078,  0.0014, -0.1418],
                        [ 0.0111,  0.0764, -0.0443]],
              
                       [[-0.0406, -0.0301, -0.0977],
                        [-0.0876, -0.1847, -0.2042],
                        [ 0.0244,  0.1091, -0.1029]],
              
                       ...(略)...
              
                       [[-1.2396e-01, -4.7989e-02,  6.0982e-03],
                        [ 1.0323e-01,  7.4295e-02,  4.4253e-02],
                        [ 1.0707e-01,  1.2724e-01,  1.9893e-01]],
              
                       [[-5.0636e-02, -1.1169e-01, -1.8197e-02],
                        [ 8.4612e-02, -4.1535e-02,  5.4119e-03],
                        [-5.3175e-03, -9.3789e-02,  2.2890e-02]],
              
                       [[ 7.8540e-02, -1.1353e-01, -1.2255e-01],
                        [ 1.1991e-02, -6.8789e-02, -1.7400e-01],
                        [ 9.7441e-05, -8.2435e-02,  4.7667e-03]]]])),
             ('conv4.bias',
              tensor([ 0.0580, -0.0452, -0.0356, -0.0077, -0.0707, -0.0301, -0.0416,  0.0149,
                       0.1250,  0.0470,  0.0371, -0.0528, -0.0887, -0.0169,  0.0178,  0.0726,
                      -0.0794,  0.0061, -0.0065,  0.0043])),
             ('bn1.weight', tensor([0.9750, 1.0013, 0.9576, 0.9376, 1.2138])),
             ('bn1.bias',
              tensor([-0.0668, -0.0248,  0.1255,  0.1388,  0.2274])),
             ('bn1.running_mean',
              tensor([ 0.1101, -0.0276,  0.1465, -0.0831,  0.0740])),
             ('bn1.running_var',
              tensor([0.0280, 0.0244, 0.0326, 0.0298, 0.0951])),
             ('bn1.num_batches_tracked', tensor(4690)),
             ('bn2.weight',
              tensor([1.1731, 0.8839, 0.8436, 1.0764, 1.0680, 1.0759, 0.9479, 0.8768, 1.1014,
                      0.9595])),
             ('bn2.bias',
              tensor([-2.8445e-01, -4.5570e-02,  2.4080e-04, -6.8240e-03, -1.9356e-02,
                      -9.8832e-02, -3.8079e-02, -1.0984e-01, -1.7605e-01,  1.1987e-01])),
             ('bn2.running_mean',
              tensor([-0.2551, -0.3085,  0.6618,  0.3906, -0.5419, -0.8479,  0.0599, -0.3507,
                      -0.1456, -0.0181])),
             ('bn2.running_var',
              tensor([0.2021, 0.7583, 0.4969, 0.4905, 0.2947, 0.1725, 0.2593, 0.2616, 0.2963,
                      0.2185])),
             ('bn2.num_batches_tracked', tensor(4690)),
             ('bn3.weight',
              tensor([1.0696, 0.8160, 0.9060, 1.0633, 1.0389, 0.9895, 0.8915, 0.8183, 1.1316,
                      0.9516, 0.9707, 0.8907, 1.0231, 0.8617, 1.0554])),
             ('bn3.bias',
              tensor([-0.2997, -0.0583, -0.2442, -0.0981, -0.2548, -0.2014, -0.1782, -0.2454,
                      -0.3477, -0.0327, -0.2671, -0.3008, -0.3205, -0.5982, -0.1280])),
             ('bn3.running_mean',
              tensor([-0.9035,  0.2130,  0.2813, -1.3235, -0.5512,  0.0552, -0.1724,  0.1172,
                      -1.5437, -0.0501, -0.5553, -0.0547, -1.1366,  0.3776, -1.5574])),
             ('bn3.running_var',
              tensor([0.6823, 0.9488, 0.7111, 1.3630, 1.0511, 0.8177, 1.0470, 1.0047, 1.1058,
                      1.9996, 1.1460, 1.3993, 1.0603, 0.4562, 1.7725])),
             ('bn3.num_batches_tracked', tensor(4690)),
             ('bn4.weight',
              tensor([1.2947, 1.3268, 1.2583, 1.2611, 1.2759, 1.3348, 1.3097, 1.3204, 1.3608,
                      1.2124, 1.3248, 1.2563, 1.3738, 1.2736, 1.2208, 1.2850, 1.3084, 1.3131,
                      1.2993, 1.2804])),
             ('bn4.bias',
              tensor([-0.5450, -0.5422, -0.3926, -0.4930, -0.5669, -0.4537, -0.5399, -0.5409,
                      -0.5684, -0.4720, -0.5032, -0.5454, -0.5353, -0.4670, -0.4292, -0.5145,
                      -0.3628, -0.6174, -0.5858, -0.4986])),
             ('bn4.running_mean',
              tensor([ 0.4598,  0.2837, -0.1339,  0.0366, -0.3407,  0.1557,  0.2131, -0.6676,
                       0.0993,  0.9765,  0.7372,  0.2057, -0.3129, -0.2075,  0.2524, -0.2052,
                       0.7763, -0.3153, -0.8190,  0.8294])),
             ('bn4.running_var',
              tensor([0.2823, 0.3211, 0.3877, 0.3735, 0.2755, 0.3754, 0.3229, 0.3310, 0.3055,
                      0.2756, 0.4847, 0.3380, 0.3030, 0.3383, 0.3769, 0.2427, 0.5124, 0.2893,
                      0.3855, 0.3065])),
             ('bn4.num_batches_tracked', tensor(4690)),
             ('fc1.weight',
              tensor([[ 0.1167,  0.1003,  0.1363,  ..., -0.0632, -0.1001, -0.1384],
                      [ 0.0219,  0.0128,  0.0546,  ..., -0.0419,  0.0384,  0.1337],
                      [ 0.0895,  0.0410,  0.0538,  ...,  0.0491,  0.0248, -0.1246],
                      ...,
                      [ 0.0377,  0.0342,  0.0202,  ..., -0.0412,  0.0019,  0.0041],
                      [ 0.0659,  0.0783,  0.0530,  ...,  0.0059,  0.0661,  0.0319],
                      [ 0.0553,  0.0511, -0.0325,  ...,  0.0109, -0.0564, -0.1748]])),
             ('fc1.bias',
              tensor([-0.0930, -0.0096, -0.0934, -0.0489,  0.1413,  0.0640,  0.0702,  0.0313,
                       0.0491,  0.1809,  0.0214, -0.1256,  0.0015,  0.0511,  0.0580,  0.0494,
                      -0.0177, -0.0066,  0.0417,  0.0478])),
             ('fc2.weight',
              tensor([[-0.3481,  0.1943, -0.1167, -0.3848,  0.2522, -0.5797, -0.3552,  0.2499,
                       -0.4902,  0.1876, -0.6010,  0.1155,  0.1638, -0.3284, -0.3364,  0.2158,
                       -0.3198,  0.1905,  0.2642, -0.1496],
                      [-0.4746, -0.5199, -0.3840, -0.2805,  0.2469,  0.2020,  0.2321, -0.3990,
                        0.2874,  0.2119, -0.4917, -0.2028, -0.3293, -0.4795,  0.2659, -0.4735,
                       -0.3607, -0.2774, -0.3733,  0.2061],
                      [ 0.1365, -0.1580, -0.2591, -0.1550, -0.3192, -0.4425,  0.2455, -0.2653,
                       -0.3550, -0.1905,  0.0099,  0.1221,  0.2218, -0.3097,  0.3020, -0.2191,
                       -0.3595,  0.1949, -0.2289, -0.3345],
                      [ 0.1819, -0.2019, -0.5650,  0.2102, -0.2937, -0.3966,  0.1748, -0.4076,
                        0.2246,  0.2166,  0.2953, -0.4493,  0.1728, -0.2740, -0.0933,  0.2476,
                        0.2063, -0.1891, -0.1732, -0.3121],
                      [-0.4458, -0.6537,  0.1651,  0.1682, -0.2137,  0.2296,  0.1956,  0.2499,
                        0.3215, -0.3628,  0.2233,  0.1229, -0.4657,  0.0259, -0.2530, -0.4996,
                        0.1364, -0.0745, -0.2714, -0.0395],
                      [ 0.1569,  0.2018, -0.2388,  0.1967,  0.2599,  0.1737, -0.4559, -0.2330,
                       -0.1793, -0.0934, -0.1260, -0.3630, -0.2875, -0.2901, -0.4317,  0.2435,
                        0.2380, -0.1329,  0.2140,  0.2101],
                      [-0.2263, -0.0485,  0.1833, -0.5683,  0.3253, -0.1906, -0.3670,  0.0325,
                       -0.2744, -0.3122, -0.4359,  0.1418, -0.3495, -0.6060, -0.4037, -0.1927,
                        0.2340,  0.2258, -0.2329,  0.2628],
                      [-0.2519,  0.1679, -0.3764,  0.1616, -0.3912, -0.3314,  0.2476,  0.1835,
                       -0.1459,  0.1911,  0.2524, -0.3639, -0.1660,  0.1537,  0.2604, -0.2612,
                       -0.4262, -0.3990,  0.2768, -0.5895],
                      [ 0.1794,  0.2254,  0.1708, -0.1893, -0.0620, -0.2753, -0.1750, -0.0990,
                       -0.1114, -0.3117,  0.2690, -0.2005,  0.1797,  0.1494, -0.3352, -0.1336,
                       -0.0673,  0.1785, -0.1971,  0.2140],
                      [ 0.1075,  0.0880,  0.1336,  0.1675, -0.3566,  0.1433, -0.1642,  0.2479,
                        0.2795, -0.3182, -0.2066, -0.1676, -0.2736,  0.1719, -0.2884,  0.1781,
                       -0.2009, -0.2304,  0.2601, -0.2920]])),
             ('fc2.bias',
              tensor([-0.3426,  0.7404,  0.7441, -0.2981, -0.2907, -0.5408, -0.0061,  0.0115,
                      -0.1341, -0.0474]))])

# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))

# sec: 評価

model_e.eval()
loss_sum = 0
n_correct = 0
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model_e(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        n_correct += (preds == labels).sum()

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), float(n_correct) / len(ds_test) * 100))

loss: 2.43e-04, accuracy: 99.20

最適計算パラメータ保存・計算再開

# sec: save

torch.save({
    'epoch': n_ep,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss }, 
    "results/pytorch-mnist/model&opt.pkl")

ここでJupyterの実行中のPythonのカーネルをリスタート可。

Libs

import numpy as np
import matplotlib.pyplot as plt

import torch
import torchvision

NNモデモ

import torch.nn as nn
import torch.nn.functional as F

class MyConvNet1(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 5, 3, stride=1, padding=0)
        self.conv2 = nn.Conv2d(5, 10, 3, stride=1, padding=0)
        self.conv3 = nn.Conv2d(10, 15, 3, stride=1, padding=0)
        self.conv4 = nn.Conv2d(15, 20, 3, stride=1, padding=0)
        self.bn1 = nn.BatchNorm2d(5)
        self.bn2 = nn.BatchNorm2d(10)
        self.bn3 = nn.BatchNorm2d(15)
        self.bn4 = nn.BatchNorm2d(20)
        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(20*4*4, 20)
        self.fc2 = nn.Linear(20, 10)
        self.drop1 = nn.Dropout(0.5)
        self.drop2 = nn.Dropout(0.5)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = self.pool1(x)
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.relu(self.bn4(self.conv4(x)))
        x = self.pool2(x)
        x = x.view(-1, 20*4*4)
        x = F.relu(self.fc1(self.drop1(x)))
        x = self.fc2(self.drop2(x))
        return x

TargetNet = MyConvNet1

# sec: load

model = TargetNet()
optimizer = torch.optim.Adam(model.parameters())

d = torch.load("results/pytorch-mnist/model&opt.pkl")
model.load_state_dict(d['model_state_dict'])
optimizer.load_state_dict(d['optimizer_state_dict'])
n_ep = d['epoch']
loss = d['loss']

# sec: 内容を確認

optimizer.state_dict()

{'state': {2202322517592: {'step': 4690,
   'exp_avg': tensor([[[[ 4.9681e-03, -1.3392e-02, -1.1919e-02],
             [-1.8946e-02, -1.5522e-02,  3.7579e-03],
             [-2.7534e-02, -1.6930e-02, -3.9333e-05]]],
   
   
           [[[ 8.3556e-03,  8.2333e-03, -4.6917e-03],
             [ 8.6733e-03,  1.3580e-02,  1.3980e-03],
             [-1.4740e-02, -1.8268e-02, -2.1180e-02]]],
   
   
           [[[ 2.6981e-02,  2.1422e-02,  8.6973e-03],
             [ 2.2680e-02,  2.4799e-02,  1.8023e-02],
             [ 1.1921e-02,  1.9067e-02,  2.0821e-02]]],
   
   
           [[[ 2.6852e-02,  2.1411e-02,  2.4557e-02],
             [ 2.6102e-02,  1.6432e-02,  1.0814e-02],
             [ 2.2461e-02,  9.3432e-03, -1.2549e-02]]],
   
   
           [[[ 1.7876e-03,  3.0547e-03,  9.6555e-03],
             [-2.2322e-03, -2.0686e-03,  2.5197e-03],
             [-2.8117e-03, -6.4745e-03, -2.8697e-03]]]]),
   'exp_avg_sq': tensor([[[[0.0133, 0.0118, 0.0097],
             [0.0132, 0.0078, 0.0049],
             [0.0131, 0.0075, 0.0063]]],
   
   
           [[[0.0202, 0.0155, 0.0092],
             [0.0155, 0.0083, 0.0045],
             [0.0224, 0.0151, 0.0118]]],
   
   
           [[[0.0066, 0.0051, 0.0060],
             [0.0069, 0.0056, 0.0052],
             [0.0061, 0.0047, 0.0038]]],
   
   
           [[[0.0056, 0.0047, 0.0053],
             [0.0032, 0.0026, 0.0039],
             [0.0025, 0.0015, 0.0030]]],
   
   
           [[[0.0048, 0.0020, 0.0013],
             [0.0007, 0.0001, 0.0004],
             [0.0001, 0.0003, 0.0017]]]])},
  2202322517672: {'step': 4690,
   'exp_avg': tensor([-1.5162e-06, -1.2697e-06, -1.2059e-07, -4.3583e-06,  6.9085e-07]),
   'exp_avg_sq': tensor([8.3501e-10, 1.5377e-11, 3.3076e-12, 3.5197e-10, 6.3042e-12])},
  2202322517272: {'step': 4690,
   'exp_avg': tensor([[[[-3.1205e-03,  1.0635e-02,  1.7947e-02],
             [ 3.0011e-02,  4.7265e-02,  5.7637e-02],
             [ 4.3873e-02,  4.3586e-02,  5.4873e-02]],
   
            [[ 1.4990e-02,  8.2882e-03, -5.3019e-04],
             [ 6.4136e-03,  1.4983e-03, -3.0395e-03],
             [ 9.1574e-04, -3.6870e-03, -1.4301e-02]],
   
            [[ 1.5993e-02,  1.2531e-02,  8.5470e-03],
             [-3.2829e-03, -3.5228e-03, -1.0266e-02],
             [-2.1787e-03, -4.9055e-03,  8.4147e-03]],
   
            [[ 1.3000e-02,  1.4188e-02,  2.3311e-02],
             [ 5.2523e-02,  5.6909e-02,  6.5983e-02],
             [ 7.4201e-02,  6.4654e-02,  5.4315e-02]],
   
            [[-1.1624e-02, -1.2903e-02, -1.5316e-02],
             [-2.2582e-02, -1.3517e-02, -1.4352e-02],
             [-1.2509e-02, -1.4768e-02, -2.3715e-02]]],
   
           ...(略)...
           
  2202322530536: {'step': 4690,
   'exp_avg': tensor([ 0.0005, -0.0052,  0.0022,  0.0002,  0.0017,  0.0033, -0.0029,  0.0005,
           -0.0030,  0.0028]),
   'exp_avg_sq': tensor([0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001,
           0.0001])}},
 'param_groups': [{'lr': 0.001,
   'betas': (0.9, 0.999),
   'eps': 1e-08,
   'weight_decay': 0,
   'amsgrad': False,
   'params': [2202322517592,
    2202322517672,
    2202322517272,
    2202322517192,
    2202322516952,
    2202322516792,
    2202322517032,
    2202322514152,
    2202322489656,
    2202322491896,
    2202322492216,
    2202322492296,
    2202322492616,
    2202322492696,
    2202322493016,
    2202322493096,
    2202322493336,
    2202322530376,
    2202322530456,
    2202322530536]}]}

最適化の内部の状態変数もすべて保存される模様。

学習ループ再開

# sec: データセット

ds_train = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_train = torch.utils.data.DataLoader(ds_train, batch_size=128, shuffle=True, num_workers=2)
print(len(ds_train))

# sec: 学習再開

criterion = nn.CrossEntropyLoss()

model.train()
for i_ep in range(n_ep, n_ep + 10): # loop over the dataset multiple times

    loss_sum = 0.0
    for ds_i in dl_train:
        
        inputs, labels = ds_i

        optimizer.zero_grad()
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        loss.backward() # 逆伝播
        optimizer.step()

        loss_sum += loss.item()
    
    # sec: print
    
    print('%dep loss: %.5f' % (i_ep + 1, loss_sum / len(ds_train)))

n_ep = i_ep + 1
print('Finished Training')

11ep loss: 0.00227
12ep loss: 0.00225
13ep loss: 0.00220
14ep loss: 0.00220
15ep loss: 0.00215
16ep loss: 0.00211
17ep loss: 0.00207
18ep loss: 0.00207
19ep loss: 0.00204
20ep loss: 0.00204
Finished Training

初回学習では「10ep loss: 0.00232」で終わっていた状態。「11ep loss: 0.00227」から始まり、最適化の続きが開始されている模様。

# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))

# sec: テスト例

model.eval()
loss_sum = 0
list_pred, list_label = [], []
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        list_pred.extend(preds)
        list_label.extend(labels)

list_pred = np.array(list_pred)
list_label = np.array(list_label)

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), (list_pred == list_label).sum() / len(ds_test) * 100))

loss: 2.48e-04, accuracy: 99.03

他のWeb記事のMNISTデータを用いる評価結果で「精度98.13%」程度なので、結果は妥当。
http://torch.classcat.com/2018/07/26/pytorch-040-examples-mnist-cnn/

# sec: precision, recall, fscore, supportを見る

res_prf = sklearn.metrics.precision_recall_fscore_support(list_label, list_pred)

print("precision, recall, fscore, supportを見る:")
print(res_prf)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(10) - 0.2, res_prf[0], width=0.4, label="precision")
ax.bar(np.arange(10) + 0.2, res_prf[1], width=0.4, label="recall")
ax.plot(np.arange(10), res_prf[2], 'r_', ms=12, label="f")
ax.legend(fontsize=9, framealpha=0.5, labelspacing=0.2)
ax.set_ylim([0.97, 1.0])
plt.show()

precision, recall, fscore, supportを見る:
(array([0.99084435, 0.99297012, 0.98653846, 0.99303483, 0.98681542,
       0.9877095 , 0.99267782, 0.98736638, 0.99588054, 0.9889669 ]), array([0.99387755, 0.99559471, 0.99418605, 0.98811881, 0.99083503,
       0.99103139, 0.99060543, 0.98832685, 0.99281314, 0.97720515]), array([0.99235863, 0.99428069, 0.99034749, 0.99057072, 0.98882114,
       0.98936766, 0.99164054, 0.98784638, 0.99434447, 0.98305085]), array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009],
      dtype=int64))

# sec: 混同行列を見る

sklearn.metrics.confusion_matrix(list_label, list_pred)

array([[ 974,    0,    1,    0,    0,    0,    4,    1,    0,    0],
       [   0, 1130,    4,    0,    0,    0,    0,    1,    0,    0],
       [   1,    0, 1026,    1,    0,    0,    0,    2,    2,    0],
       [   0,    0,    1,  998,    0,    5,    0,    4,    2,    0],
       [   0,    0,    0,    0,  973,    0,    1,    0,    0,    8],
       [   1,    0,    0,    4,    0,  884,    2,    1,    0,    0],
       [   3,    3,    0,    0,    0,    3,  949,    0,    0,    0],
       [   0,    5,    5,    1,    0,    0,    0, 1016,    0,    1],
       [   2,    0,    3,    0,    0,    0,    0,    0,  967,    2],
       [   2,    0,    0,    1,   13,    3,    0,    4,    0,  986]],
      dtype=int64)

保存・読込

# sec: save

torch.save(model.state_dict(), "results/pytorch-mnist/model2.pkl")

# sec: load

model = TargetNet()
model.load_state_dict(torch.load("results/pytorch-mnist/model2.pkl"))

<All keys matched successfully>