0. 復習

AlexNet
AlexNetとは2012年に開かれた画像認識コンペティション2位に大差をつけて優勝したモデルである。AlexNetの登場で、ディープラーニングが大きく注目を集めた。
5層の畳み込み層およびプーリング層など、それに続く3層の全結合層から構成される。

確認テスト
・サイズ5×5の入力画像を、サイズ3×3のフィルタで畳み込んだ時の出力画像のサイズを答えよ。なおストライドは2、パディングは1とする。
→3x3

1. 再起型ニューラルネットワークの概念

RNNとは？
時系列データに対応可能な、ニューラルネットワークである。
中間層が再帰構造を持ち、前後の時刻の中間層を繋ぐ。
時間に遡って逆伝播が行われる。

時系列データとは？
時間的順序を追って一定間隔ごとに観察され、しかも相互に統計的依存関係が認められるようなデータの系列

確認テスト
・RNNのネットワークには大きくわけて3つの重みがある。1つは入力から現在の中間層を定義する際にかけられる重み、1つは中間層から出力を定義する際にかけられる重みである。残り1つの重みについて説明せよ。
残る1つの重みは、前の中間層から現在の中間層を定義する際にかけられる重みである。

BPTTとは？
RNNにおいてのパラメータ調整方法の一種
→誤差逆伝播の一種

確認テスト
・連鎖律の原理を使い、dz/dxを求めよ。
微分の連鎖率は
$$\frac{dz}{dx}=\frac{dz}{dt}\times\frac{dt}{dx}$$

\left\{
\begin{array}{l}
\frac{dz}{dt}=2t \\
\frac{dt}{dx}=1 
\end{array}
\right.

より
$$\frac{dz}{dx}=\frac{dz}{dt}\times\frac{dt}{dx}=2t\times 1=2(x+y)$$

確認テスト
・下図の$y_1$を$x_1,s_0,s_1,w_{in},w,w_{out}$を用いて数式で表わせ。また、中間層の出力にシグモイド関数を作用させよ。
$$z_1=sigmoid \left( s_0 W+x_1W_{in}+b \right)$$
$$y_1=sigmoid \left( z_1 W_{out}+c \right)$$

実装

import numpy as np
from common import functions
import matplotlib.pyplot as plt

def d_tanh(x):
    return 1/(np.cosh(x) ** 2)



# データを用意
# 2進数の桁数
binary_dim = 8
# 最大値 + 1
largest_number = pow(2, binary_dim)
# largest_numberまで2進数を用意
binary = np.unpackbits(np.array([range(largest_number)],dtype=np.uint8).T,axis=1)

input_layer_size = 2
hidden_layer_size = 16
output_layer_size = 1

weight_init_std = 1
learning_rate = 0.1

iters_num = 10000
plot_interval = 100

# ウェイト初期化 (バイアスは簡単のため省略)
W_in = weight_init_std * np.random.randn(input_layer_size, hidden_layer_size)
W_out = weight_init_std * np.random.randn(hidden_layer_size, output_layer_size)
W = weight_init_std * np.random.randn(hidden_layer_size, hidden_layer_size)

# Xavier
# W_in = np.random.randn(input_layer_size, hidden_layer_size) / (np.sqrt(input_layer_size))
# W_out = np.random.randn(hidden_layer_size, output_layer_size) / (np.sqrt(hidden_layer_size))
# W = np.random.randn(hidden_layer_size, hidden_layer_size) / (np.sqrt(hidden_layer_size))

# He
# W_in = np.random.randn(input_layer_size, hidden_layer_size) / (np.sqrt(input_layer_size)) * np.sqrt(2)
# W_out = np.random.randn(hidden_layer_size, output_layer_size) / (np.sqrt(hidden_layer_size)) * np.sqrt(2)
# W = np.random.randn(hidden_layer_size, hidden_layer_size) / (np.sqrt(hidden_layer_size)) * np.sqrt(2)




# 勾配
W_in_grad = np.zeros_like(W_in)
W_out_grad = np.zeros_like(W_out)
W_grad = np.zeros_like(W)

u = np.zeros((hidden_layer_size, binary_dim + 1))
z = np.zeros((hidden_layer_size, binary_dim + 1))
y = np.zeros((output_layer_size, binary_dim))

delta_out = np.zeros((output_layer_size, binary_dim))
delta = np.zeros((hidden_layer_size, binary_dim + 1))

all_losses = []

for i in range(iters_num):
    
    # A, B初期化 (a + b = d)
    a_int = np.random.randint(largest_number/2)
    a_bin = binary[a_int] # binary encoding
    b_int = np.random.randint(largest_number/2)
    b_bin = binary[b_int] # binary encoding
    
    # 正解データ
    d_int = a_int + b_int
    d_bin = binary[d_int]
    
    # 出力バイナリ
    out_bin = np.zeros_like(d_bin)
    
    # 時系列全体の誤差
    all_loss = 0    
    
    # 時系列ループ
    for t in range(binary_dim):
        # 入力値
        X = np.array([a_bin[ - t - 1], b_bin[ - t - 1]]).reshape(1, -1)
        # 時刻tにおける正解データ
        dd = np.array([d_bin[binary_dim - t - 1]])
        
        u[:,t+1] = np.dot(X, W_in) + np.dot(z[:,t].reshape(1, -1), W)
        z[:,t+1] = functions.sigmoid(u[:,t+1])
#         z[:,t+1] = functions.relu(u[:,t+1])
#         z[:,t+1] = np.tanh(u[:,t+1])   
        y[:,t] = functions.sigmoid(np.dot(z[:,t+1].reshape(1, -1), W_out))


        #誤差
        loss = functions.mean_squared_error(dd, y[:,t])
        
        delta_out[:,t] = functions.d_mean_squared_error(dd, y[:,t]) * functions.d_sigmoid(y[:,t])        
        
        all_loss += loss

        out_bin[binary_dim - t - 1] = np.round(y[:,t])
    
    
    for t in range(binary_dim)[::-1]:
        X = np.array([a_bin[-t-1],b_bin[-t-1]]).reshape(1, -1)        

        delta[:,t] = (np.dot(delta[:,t+1].T, W.T) + np.dot(delta_out[:,t].T, W_out.T)) * functions.d_sigmoid(u[:,t+1])
#         delta[:,t] = (np.dot(delta[:,t+1].T, W.T) + np.dot(delta_out[:,t].T, W_out.T)) * functions.d_relu(u[:,t+1])
#         delta[:,t] = (np.dot(delta[:,t+1].T, W.T) + np.dot(delta_out[:,t].T, W_out.T)) * d_tanh(u[:,t+1])    

        # 勾配更新
        W_out_grad += np.dot(z[:,t+1].reshape(-1,1), delta_out[:,t].reshape(-1,1))
        W_grad += np.dot(z[:,t].reshape(-1,1), delta[:,t].reshape(1,-1))
        W_in_grad += np.dot(X.T, delta[:,t].reshape(1,-1))
    
    # 勾配適用
    W_in -= learning_rate * W_in_grad
    W_out -= learning_rate * W_out_grad
    W -= learning_rate * W_grad
    
    W_in_grad *= 0
    W_out_grad *= 0
    W_grad *= 0
    

    if(i % plot_interval == 0):
        all_losses.append(all_loss)        
        print("iters:" + str(i))
        print("Loss:" + str(all_loss))
        print("Pred:" + str(out_bin))
        print("True:" + str(d_bin))
        out_int = 0
        for index,x in enumerate(reversed(out_bin)):
            out_int += x * pow(2, index)
        print(str(a_int) + " + " + str(b_int) + " = " + str(out_int))
        print("------------")

lists = range(0, iters_num, plot_interval)
plt.plot(lists, all_losses, label="loss")
plt.show()

iters:0
Loss:1.9973243202627657
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 1 1 1]
8 + 71 = 0

iters:100
Loss:1.0733327737788823
Pred:[0 1 1 0 0 1 1 1]
True:[1 0 0 1 1 0 1 1]
102 + 53 = 103

iters:200
Loss:0.9368876198179751
Pred:[0 0 1 1 1 0 1 1]
True:[0 0 0 1 1 1 1 0]
11 + 19 = 59

iters:300
Loss:1.028250700014706
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 1 0 1 0 1 0]
107 + 63 = 255

iters:400
Loss:1.0076000852901
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 1 0 0 1]
59 + 78 = 0

iters:500
Loss:0.9347503940439331
Pred:[0 1 1 1 1 1 1 0]
True:[0 1 0 1 1 0 1 0]
54 + 36 = 126

iters:600
Loss:0.7163765925303197
Pred:[0 0 0 1 0 0 0 1]
True:[0 0 0 1 1 0 0 1]
8 + 17 = 17

iters:700
Loss:0.9267537545833759
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 1 1 1 0]
7 + 39 = 0

iters:800
Loss:1.0286753903608468
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 1 1 0 0 0 0]
107 + 69 = 255

iters:900
Loss:0.9358021834361258
Pred:[0 0 1 1 1 1 0 0]
True:[0 1 0 1 1 0 1 0]
62 + 28 = 60

iters:1000
Loss:0.9879171112907346
Pred:[0 1 0 0 1 0 0 0]
True:[1 0 0 1 0 0 0 1]
109 + 36 = 72

iters:1100
Loss:0.9192579758337747
Pred:[1 0 0 1 0 1 1 0]
True:[1 0 1 0 0 1 1 1]
73 + 94 = 150

iters:1200
Loss:1.39564982542401
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 1 1 0 0 1 1]
123 + 120 = 0

iters:1300
Loss:0.9163083221774926
Pred:[0 1 0 0 1 0 0 0]
True:[0 1 1 1 0 1 0 0]
110 + 6 = 72

iters:1400
Loss:0.9078625536311292
Pred:[1 0 0 0 1 1 1 1]
True:[1 0 0 1 0 0 1 1]
70 + 77 = 143

iters:1500
Loss:0.9085946596622659
Pred:[1 0 1 1 0 0 1 1]
True:[1 1 0 0 0 0 0 1]
120 + 73 = 179

iters:1600
Loss:1.2937107344854766
Pred:[0 1 0 0 1 1 1 0]
True:[1 0 1 1 0 0 0 0]
61 + 115 = 78

iters:1700
Loss:0.8961435523954027
Pred:[0 0 0 1 1 1 1 0]
True:[0 0 1 0 0 0 0 0]
9 + 23 = 30

iters:1800
Loss:0.8939831289624031
Pred:[1 0 1 0 0 1 1 1]
True:[1 1 0 0 0 1 0 1]
80 + 117 = 167

iters:1900
Loss:0.7583292733236224
Pred:[1 0 0 0 0 0 0 0]
True:[1 1 1 1 1 0 0 0]
124 + 124 = 128

iters:2000
Loss:0.8471447267917033
Pred:[1 0 0 1 1 0 0 1]
True:[1 0 1 0 0 0 1 1]
95 + 68 = 153

iters:2100
Loss:1.5303908565220292
Pred:[0 1 1 1 1 1 1 0]
True:[1 0 0 0 0 0 0 0]
99 + 29 = 126

iters:2200
Loss:1.3041629743151388
Pred:[0 1 1 1 1 0 1 1]
True:[1 0 0 0 0 0 0 1]
69 + 60 = 123

iters:2300
Loss:0.4606005912389376
Pred:[0 0 1 0 1 1 1 0]
True:[0 0 1 0 1 1 1 0]
11 + 35 = 46

iters:2400
Loss:0.5416213978042581
Pred:[1 0 1 0 0 0 0 0]
True:[1 0 1 1 0 0 0 0]
104 + 72 = 160

iters:2500
Loss:0.6616124679531733
Pred:[1 0 1 1 1 1 1 1]
True:[1 0 1 0 1 1 1 1]
106 + 69 = 191

iters:2600
Loss:0.4695598987645002
Pred:[1 0 0 1 0 1 1 0]
True:[1 0 0 1 0 1 1 0]
98 + 52 = 150

iters:2700
Loss:0.43202355062711056
Pred:[1 0 0 1 1 0 1 1]
True:[1 0 0 1 1 0 0 1]
72 + 81 = 155

iters:2800
Loss:0.23939386522820452
Pred:[0 1 0 0 1 1 0 1]
True:[0 1 0 0 1 1 0 1]
2 + 75 = 77

iters:2900
Loss:0.7205441996139405
Pred:[1 1 1 1 1 1 0 0]
True:[0 1 1 1 1 1 1 0]
23 + 103 = 252

iters:3000
Loss:0.21035535868981808
Pred:[0 0 1 0 0 1 1 0]
True:[0 0 1 0 0 1 1 0]
1 + 37 = 38

iters:3100
Loss:0.1448566948938648
Pred:[0 1 1 0 1 1 0 1]
True:[0 1 1 0 1 1 0 1]
93 + 16 = 109

iters:3200
Loss:0.704861743150431
Pred:[1 1 0 1 1 1 1 1]
True:[1 1 0 0 1 1 1 1]
102 + 105 = 223

iters:3300
Loss:0.30491171674260975
Pred:[0 1 0 1 1 1 1 1]
True:[0 1 0 1 1 1 1 1]
90 + 5 = 95

iters:3400
Loss:0.30267192051138
Pred:[0 1 0 1 0 0 0 0]
True:[0 1 0 1 0 0 0 0]
50 + 30 = 80

iters:3500
Loss:0.07025791222918729
Pred:[0 0 1 1 0 1 1 0]
True:[0 0 1 1 0 1 1 0]
29 + 25 = 54

iters:3600
Loss:0.6895425275794198
Pred:[1 0 1 0 1 1 1 1]
True:[1 0 0 1 1 1 1 1]
66 + 93 = 175

iters:3700
Loss:0.3419421966230561
Pred:[1 0 0 0 0 0 0 1]
True:[1 0 0 0 0 0 0 1]
42 + 87 = 129

iters:3800
Loss:0.05976908292424251
Pred:[0 0 1 1 1 1 0 1]
True:[0 0 1 1 1 1 0 1]
39 + 22 = 61

iters:3900
Loss:0.04563214805636825
Pred:[0 1 0 0 1 0 0 0]
True:[0 1 0 0 1 0 0 0]
68 + 4 = 72

iters:4000
Loss:0.06787308211874178
Pred:[1 1 0 0 1 1 0 1]
True:[1 1 0 0 1 1 0 1]
80 + 125 = 205

iters:4100
Loss:0.07678891092654301
Pred:[1 0 0 1 1 1 1 0]
True:[1 0 0 1 1 1 1 0]
44 + 114 = 158

iters:4200
Loss:0.13568603932125967
Pred:[0 1 0 1 0 0 1 1]
True:[0 1 0 1 0 0 1 1]
24 + 59 = 83

iters:4300
Loss:0.05646865741798102
Pred:[1 0 1 1 1 1 1 0]
True:[1 0 1 1 1 1 1 0]
76 + 114 = 190

iters:4400
Loss:0.06850571680417594
Pred:[0 1 0 0 0 1 0 1]
True:[0 1 0 0 0 1 0 1]
13 + 56 = 69

iters:4500
Loss:0.08578436630662499
Pred:[1 1 0 0 0 1 1 0]
True:[1 1 0 0 0 1 1 0]
123 + 75 = 198

iters:4600
Loss:0.04356889848825188
Pred:[0 0 1 0 0 1 1 1]
True:[0 0 1 0 0 1 1 1]
39 + 0 = 39

iters:4700
Loss:0.0984666370467623
Pred:[1 0 0 0 0 1 1 0]
True:[1 0 0 0 0 1 1 0]
55 + 79 = 134

iters:4800
Loss:0.033474659731294924
Pred:[0 1 0 1 0 0 0 0]
True:[0 1 0 1 0 0 0 0]
24 + 56 = 80

iters:4900
Loss:0.015316895116949301
Pred:[1 0 1 1 1 1 0 0]
True:[1 0 1 1 1 1 0 0]
75 + 113 = 188

iters:5000
Loss:0.04664989911856581
Pred:[0 1 0 0 0 0 1 0]
True:[0 1 0 0 0 0 1 0]
14 + 52 = 66

iters:5100
Loss:0.02177424084293254
Pred:[1 0 1 0 0 0 0 0]
True:[1 0 1 0 0 0 0 0]
92 + 68 = 160

iters:5200
Loss:0.016937615702077923
Pred:[0 1 0 1 0 1 1 0]
True:[0 1 0 1 0 1 1 0]
19 + 67 = 86

iters:5300
Loss:0.022509255899109626
Pred:[0 1 1 1 1 0 0 0]
True:[0 1 1 1 1 0 0 0]
24 + 96 = 120

iters:5400
Loss:0.02858181072343992
Pred:[1 0 0 0 1 1 1 1]
True:[1 0 0 0 1 1 1 1]
107 + 36 = 143

iters:5500
Loss:0.01773096047867255
Pred:[0 0 1 1 0 0 0 0]
True:[0 0 1 1 0 0 0 0]
0 + 48 = 48

iters:5600
Loss:0.011897013007431905
Pred:[0 1 0 0 0 0 0 1]
True:[0 1 0 0 0 0 0 1]
0 + 65 = 65

iters:5700
Loss:0.018832798402217597
Pred:[1 0 1 0 1 1 1 0]
True:[1 0 1 0 1 1 1 0]
60 + 114 = 174

iters:5800
Loss:0.013614420131687352
Pred:[0 1 0 0 0 0 1 0]
True:[0 1 0 0 0 0 1 0]
2 + 64 = 66

iters:5900
Loss:0.011401270925778762
Pred:[1 0 1 1 0 0 1 0]
True:[1 0 1 1 0 0 1 0]
95 + 83 = 178

iters:6000
Loss:0.02476384037157792
Pred:[1 0 1 1 1 0 1 1]
True:[1 0 1 1 1 0 1 1]
91 + 96 = 187

iters:6100
Loss:0.009755714792342824
Pred:[1 1 0 0 1 0 0 1]
True:[1 1 0 0 1 0 0 1]
121 + 80 = 201

iters:6200
Loss:0.01391813419338415
Pred:[1 0 0 0 1 1 1 0]
True:[1 0 0 0 1 1 1 0]
28 + 114 = 142

iters:6300
Loss:0.0030927106091086336
Pred:[1 1 1 0 0 1 1 0]
True:[1 1 1 0 0 1 1 0]
125 + 105 = 230

iters:6400
Loss:0.010698261907759974
Pred:[1 0 0 1 0 1 1 0]
True:[1 0 0 1 0 1 1 0]
102 + 48 = 150

iters:6500
Loss:0.009508033208464982
Pred:[1 0 0 1 1 1 1 0]
True:[1 0 0 1 1 1 1 0]
102 + 56 = 158

iters:6600
Loss:0.007839706362070539
Pred:[1 0 0 1 0 0 1 1]
True:[1 0 0 1 0 0 1 1]
62 + 85 = 147

iters:6700
Loss:0.007783122859302097
Pred:[0 1 0 0 1 1 0 0]
True:[0 1 0 0 1 1 0 0]
31 + 45 = 76

iters:6800
Loss:0.00589730293708635
Pred:[0 1 1 1 0 0 1 0]
True:[0 1 1 1 0 0 1 0]
59 + 55 = 114

iters:6900
Loss:0.00763685551472209
Pred:[1 1 0 0 0 0 1 1]
True:[1 1 0 0 0 0 1 1]
111 + 84 = 195

iters:7000
Loss:0.008700562786829092
Pred:[1 0 1 1 0 1 1 1]
True:[1 0 1 1 0 1 1 1]
98 + 85 = 183

iters:7100
Loss:0.006435381944699292
Pred:[0 0 1 1 1 1 1 0]
True:[0 0 1 1 1 1 1 0]
50 + 12 = 62

iters:7200
Loss:0.004745044830222811
Pred:[1 0 1 0 1 0 0 1]
True:[1 0 1 0 1 0 0 1]
80 + 89 = 169

iters:7300
Loss:0.0061896161966560805
Pred:[0 1 0 1 1 1 1 0]
True:[0 1 0 1 1 1 1 0]
20 + 74 = 94

iters:7400
Loss:0.0022767708723033086
Pred:[1 1 0 0 0 0 1 0]
True:[1 1 0 0 0 0 1 0]
85 + 109 = 194

iters:7500
Loss:0.006434829400136146
Pred:[1 0 0 0 0 0 1 0]
True:[1 0 0 0 0 0 1 0]
90 + 40 = 130

iters:7600
Loss:0.004812458507285831
Pred:[0 1 1 1 0 1 1 0]
True:[0 1 1 1 0 1 1 0]
94 + 24 = 118

iters:7700
Loss:0.0015182521777489593
Pred:[1 0 0 0 0 0 1 0]
True:[1 0 0 0 0 0 1 0]
41 + 89 = 130

iters:7800
Loss:0.003828784974211416
Pred:[0 1 0 1 1 1 0 1]
True:[0 1 0 1 1 1 0 1]
43 + 50 = 93

iters:7900
Loss:0.003824075871355923
Pred:[1 0 0 0 0 0 1 0]
True:[1 0 0 0 0 0 1 0]
71 + 59 = 130

iters:8000
Loss:0.0035155706702554622
Pred:[1 0 0 0 0 1 0 1]
True:[1 0 0 0 0 1 0 1]
23 + 110 = 133

iters:8100
Loss:0.004991932221786559
Pred:[1 0 0 0 0 1 0 0]
True:[1 0 0 0 0 1 0 0]
82 + 50 = 132

iters:8200
Loss:0.009512041561193948
Pred:[1 1 0 1 1 0 0 0]
True:[1 1 0 1 1 0 0 0]
94 + 122 = 216

iters:8300
Loss:0.004048681026410974
Pred:[0 1 0 0 1 1 0 0]
True:[0 1 0 0 1 1 0 0]
60 + 16 = 76

iters:8400
Loss:0.0036692040650989754
Pred:[0 1 0 1 0 0 1 0]
True:[0 1 0 1 0 0 1 0]
32 + 50 = 82

iters:8500
Loss:0.0006075986460093813
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 0 0 1 1 1 0]
45 + 33 = 78

iters:8600
Loss:0.0010758885634611249
Pred:[1 0 0 0 0 1 1 0]
True:[1 0 0 0 0 1 1 0]
121 + 13 = 134

iters:8700
Loss:0.0024547689631443936
Pred:[1 0 0 0 1 1 1 1]
True:[1 0 0 0 1 1 1 1]
38 + 105 = 143

iters:8800
Loss:0.0022875930312221443
Pred:[0 1 0 1 1 0 0 1]
True:[0 1 0 1 1 0 0 1]
60 + 29 = 89

iters:8900
Loss:0.0012913265475893241
Pred:[1 0 1 0 1 1 0 0]
True:[1 0 1 0 1 1 0 0]
119 + 53 = 172

iters:9000
Loss:0.0023072114433282634
Pred:[1 1 0 0 0 1 0 1]
True:[1 1 0 0 0 1 0 1]
119 + 78 = 197

iters:9100
Loss:0.0012147177794085888
Pred:[1 0 1 1 0 0 0 0]
True:[1 0 1 1 0 0 0 0]
75 + 101 = 176

iters:9200
Loss:0.0005293434987758248
Pred:[1 0 0 1 1 1 1 0]
True:[1 0 0 1 1 1 1 0]
65 + 93 = 158

iters:9300
Loss:0.0021855130431205805
Pred:[1 0 0 0 0 1 0 1]
True:[1 0 0 0 0 1 0 1]
103 + 30 = 133

iters:9400
Loss:0.0027656318971855257
Pred:[1 1 0 1 0 0 0 1]
True:[1 1 0 1 0 0 0 1]
101 + 108 = 209

iters:9500
Loss:0.003534602075373821
Pred:[1 0 0 0 1 1 1 1]
True:[1 0 0 0 1 1 1 1]
17 + 126 = 143

iters:9600
Loss:0.0018605725553301703
Pred:[1 1 0 1 0 0 0 1]
True:[1 1 0 1 0 0 0 1]
109 + 100 = 209

iters:9700
Loss:0.0030274064247236706
Pred:[1 0 1 1 0 1 1 1]
True:[1 0 1 1 0 1 1 1]
125 + 58 = 183

iters:9800
Loss:0.0017619086312553943
Pred:[1 0 1 1 1 0 1 1]
True:[1 0 1 1 1 0 1 1]
94 + 93 = 187

iters:9900
Loss:0.0025667073851160685
Pred:[1 0 0 1 1 0 1 1]
True:[1 0 0 1 1 0 1 1]
33 + 122 = 155

weight_init_std = 1、learning_rate = 0.1、hidden_layer_size = 32の場合、

iters:0
Loss:2.416691778604764
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 1 0 1 0 0 0]
62 + 106 = 255

iters:100
Loss:0.9674924139426193
Pred:[1 1 1 1 1 0 1 1]
True:[0 1 1 0 1 1 0 1]
23 + 86 = 251

iters:200
Loss:0.7658558207999123
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 1 1 1 1 1 1]
102 + 89 = 255

iters:300
Loss:0.9647486668210454
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 0 0 1 0]
105 + 89 = 0

iters:400
Loss:0.8810326940427318
Pred:[1 1 1 1 1 0 0 1]
True:[1 1 0 1 0 1 0 1]
123 + 90 = 249

iters:500
Loss:0.6860695498302471
Pred:[1 1 1 1 1 1 1 0]
True:[1 1 0 1 1 1 1 0]
107 + 115 = 254

iters:600
Loss:1.1502960355686256
Pred:[0 1 1 1 1 1 1 1]
True:[1 0 0 0 0 0 1 0]
67 + 63 = 127

iters:700
Loss:1.0423215465915991
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 0 1 0]
106 + 0 = 0

iters:800
Loss:0.8346311948185425
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
16 + 96 = 0

iters:900
Loss:1.002161197318664
Pred:[1 0 0 1 0 0 0 1]
True:[1 1 0 1 1 0 0 0]
127 + 89 = 145

iters:1000
Loss:1.0048162420167215
Pred:[1 0 0 1 1 1 0 0]
True:[1 0 0 0 1 0 1 0]
97 + 41 = 156

iters:1100
Loss:1.2607465748823785
Pred:[0 0 0 0 0 0 1 1]
True:[0 1 1 1 1 0 0 0]
55 + 65 = 3

iters:1200
Loss:0.8369685743505387
Pred:[0 0 1 1 1 1 1 1]
True:[0 1 1 1 0 1 1 1]
88 + 31 = 63

iters:1300
Loss:0.9736220994578029
Pred:[0 0 0 0 0 0 1 0]
True:[0 1 0 0 1 1 0 0]
76 + 0 = 2

iters:1400
Loss:0.9229265086063995
Pred:[0 0 1 1 0 1 0 1]
True:[1 0 1 1 1 0 0 1]
90 + 95 = 53

iters:1500
Loss:1.0080148207246822
Pred:[0 1 1 1 1 1 1 1]
True:[0 0 0 1 1 0 0 1]
7 + 18 = 127

iters:1600
Loss:0.745088701593963
Pred:[0 0 1 1 0 0 1 1]
True:[1 0 1 1 1 0 1 1]
64 + 123 = 51

iters:1700
Loss:0.8060819070767945
Pred:[0 0 1 0 1 0 0 1]
True:[0 1 0 0 0 0 0 1]
55 + 10 = 41

iters:1800
Loss:0.9722186840917634
Pred:[1 0 1 1 1 1 0 1]
True:[1 0 1 1 0 0 0 1]
107 + 70 = 189

iters:1900
Loss:1.055824226145949
Pred:[0 0 0 1 1 0 0 0]
True:[0 1 1 1 1 1 0 1]
88 + 37 = 24

iters:2000
Loss:0.5436981093533196
Pred:[0 0 0 1 0 1 1 1]
True:[0 0 0 1 0 1 1 1]
4 + 19 = 23

iters:2100
Loss:0.8936229413662422
Pred:[1 0 0 1 0 0 1 0]
True:[1 0 0 1 1 0 0 0]
77 + 75 = 146

iters:2200
Loss:0.4854497701766937
Pred:[0 0 0 0 1 0 1 1]
True:[0 1 0 0 1 0 1 1]
0 + 75 = 11

iters:2300
Loss:0.9284097627607837
Pred:[0 0 0 1 1 1 1 0]
True:[0 1 0 0 0 0 0 0]
49 + 15 = 30

iters:2400
Loss:0.3471618381815352
Pred:[0 0 0 1 1 1 1 1]
True:[0 0 0 1 1 1 1 1]
29 + 2 = 31

iters:2500
Loss:0.6795401522979898
Pred:[0 0 1 0 0 1 0 0]
True:[0 1 0 0 0 1 0 0]
28 + 40 = 36

iters:2600
Loss:0.49971401804423676
Pred:[0 1 0 0 0 0 0 1]
True:[0 1 0 1 1 0 0 1]
81 + 8 = 65

iters:2700
Loss:0.4770206865926954
Pred:[0 1 0 1 0 0 0 0]
True:[0 1 0 1 1 0 0 0]
75 + 13 = 80

iters:2800
Loss:0.3597258587887581
Pred:[1 1 0 1 1 0 0 0]
True:[1 0 0 1 1 0 0 0]
52 + 100 = 216

iters:2900
Loss:0.7482754464338083
Pred:[0 1 0 0 1 0 0 1]
True:[0 1 0 1 1 0 1 1]
50 + 41 = 73

iters:3000
Loss:0.6684648417463785
Pred:[0 1 1 1 1 0 1 0]
True:[0 1 1 1 1 1 0 0]
5 + 119 = 122

iters:3100
Loss:0.1113405037831486
Pred:[1 1 1 0 0 1 1 1]
True:[1 1 1 0 0 1 1 1]
121 + 110 = 231

iters:3200
Loss:0.12077389564085679
Pred:[0 0 0 0 1 1 0 1]
True:[0 0 0 0 1 1 0 1]
5 + 8 = 13

iters:3300
Loss:0.28985902383386675
Pred:[1 0 0 0 1 0 0 0]
True:[1 0 0 0 1 0 0 0]
51 + 85 = 136

iters:3400
Loss:0.16079252036301991
Pred:[0 1 1 0 0 1 0 0]
True:[0 1 1 0 0 1 0 0]
72 + 28 = 100

iters:3500
Loss:0.19055300072082257
Pred:[1 0 1 0 0 0 1 0]
True:[1 0 1 0 0 0 1 0]
42 + 120 = 162

iters:3600
Loss:0.1320027415132536
Pred:[0 1 1 1 0 0 1 1]
True:[0 1 1 1 0 0 1 1]
3 + 112 = 115

iters:3700
Loss:0.4289206102307461
Pred:[1 0 1 1 0 0 1 1]
True:[1 0 1 0 0 0 1 1]
102 + 61 = 179

iters:3800
Loss:0.0498089999193561
Pred:[0 1 0 0 1 0 0 1]
True:[0 1 0 0 1 0 0 1]
61 + 12 = 73

iters:3900
Loss:0.07397900268174254
Pred:[0 1 1 0 1 0 0 1]
True:[0 1 1 0 1 0 0 1]
50 + 55 = 105

iters:4000
Loss:0.042699100864280144
Pred:[1 1 1 1 0 1 1 0]
True:[1 1 1 1 0 1 1 0]
122 + 124 = 246

iters:4100
Loss:0.04946036252736748
Pred:[0 0 1 1 0 1 1 1]
True:[0 0 1 1 0 1 1 1]
35 + 20 = 55

iters:4200
Loss:0.1360592083955835
Pred:[1 0 0 1 1 1 1 0]
True:[1 0 0 1 1 1 1 0]
35 + 123 = 158

iters:4300
Loss:0.0240300990198409
Pred:[1 0 1 1 0 1 0 1]
True:[1 0 1 1 0 1 0 1]
93 + 88 = 181

iters:4400
Loss:0.033981991790019046
Pred:[1 0 0 1 0 0 0 0]
True:[1 0 0 1 0 0 0 0]
43 + 101 = 144

iters:4500
Loss:0.0757600259446267
Pred:[1 1 0 0 1 0 0 0]
True:[1 1 0 0 1 0 0 0]
114 + 86 = 200

iters:4600
Loss:0.021796563578577816
Pred:[0 1 1 1 1 1 1 0]
True:[0 1 1 1 1 1 1 0]
75 + 51 = 126

iters:4700
Loss:0.07450889047932523
Pred:[1 0 0 0 0 1 1 1]
True:[1 0 0 0 0 1 1 1]
15 + 120 = 135

iters:4800
Loss:0.07358514187651208
Pred:[0 1 0 0 0 0 0 1]
True:[0 1 0 0 0 0 0 1]
14 + 51 = 65

iters:4900
Loss:0.13861529817730472
Pred:[1 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
6 + 122 = 128

iters:5000
Loss:0.013591084005834921
Pred:[1 0 1 0 0 1 0 1]
True:[1 0 1 0 0 1 0 1]
61 + 104 = 165

iters:5100
Loss:0.008527799027682323
Pred:[0 0 1 0 1 1 0 0]
True:[0 0 1 0 1 1 0 0]
12 + 32 = 44

iters:5200
Loss:0.016325040173515288
Pred:[1 0 0 1 0 0 0 1]
True:[1 0 0 1 0 0 0 1]
66 + 79 = 145

iters:5300
Loss:0.0074921196559403535
Pred:[0 1 1 1 1 1 0 1]
True:[0 1 1 1 1 1 0 1]
86 + 39 = 125

iters:5400
Loss:0.007410098613604831
Pred:[0 1 0 1 1 1 0 1]
True:[0 1 0 1 1 1 0 1]
85 + 8 = 93

iters:5500
Loss:0.0071267263809115635
Pred:[0 0 1 0 0 0 0 0]
True:[0 0 1 0 0 0 0 0]
0 + 32 = 32

iters:5600
Loss:0.005822970648813325
Pred:[1 0 0 0 0 0 1 0]
True:[1 0 0 0 0 0 1 0]
122 + 8 = 130

iters:5700
Loss:0.007985279178027745
Pred:[1 0 1 0 1 0 0 1]
True:[1 0 1 0 1 0 0 1]
75 + 94 = 169

iters:5800
Loss:0.0052486874058704745
Pred:[0 1 0 0 1 0 0 1]
True:[0 1 0 0 1 0 0 1]
54 + 19 = 73

iters:5900
Loss:0.007513193676999717
Pred:[0 1 1 1 0 1 1 1]
True:[0 1 1 1 0 1 1 1]
58 + 61 = 119

iters:6000
Loss:0.006193292583254399
Pred:[0 1 1 0 1 1 0 0]
True:[0 1 1 0 1 1 0 0]
24 + 84 = 108

iters:6100
Loss:0.006870847550646139
Pred:[0 1 1 0 0 0 0 0]
True:[0 1 1 0 0 0 0 0]
30 + 66 = 96

iters:6200
Loss:0.013591255805794563
Pred:[1 1 0 0 0 0 1 0]
True:[1 1 0 0 0 0 1 0]
76 + 118 = 194

iters:6300
Loss:0.005363006631820811
Pred:[0 1 1 1 0 1 0 0]
True:[0 1 1 1 0 1 0 0]
58 + 58 = 116

iters:6400
Loss:0.004097529358654943
Pred:[1 0 0 0 0 0 0 1]
True:[1 0 0 0 0 0 0 1]
80 + 49 = 129

iters:6500
Loss:0.002008322800876352
Pred:[0 1 0 1 0 0 1 0]
True:[0 1 0 1 0 0 1 0]
5 + 77 = 82

iters:6600
Loss:0.003662166876369808
Pred:[1 1 0 1 0 0 1 1]
True:[1 1 0 1 0 0 1 1]
110 + 101 = 211

iters:6700
Loss:0.006162035366581912
Pred:[1 0 0 0 1 0 1 0]
True:[1 0 0 0 1 0 1 0]
88 + 50 = 138

iters:6800
Loss:0.004276601795900506
Pred:[1 1 0 1 0 0 1 0]
True:[1 1 0 1 0 0 1 0]
96 + 114 = 210

iters:6900
Loss:0.0034452739116113316
Pred:[1 0 0 1 0 0 1 0]
True:[1 0 0 1 0 0 1 0]
87 + 59 = 146

iters:7000
Loss:0.0023007700449721633
Pred:[1 0 0 1 1 0 1 1]
True:[1 0 0 1 1 0 1 1]
81 + 74 = 155

iters:7100
Loss:0.0019993777166282213
Pred:[0 0 1 1 1 0 0 1]
True:[0 0 1 1 1 0 0 1]
33 + 24 = 57

iters:7200
Loss:0.004033872151076945
Pred:[0 1 1 1 1 1 0 0]
True:[0 1 1 1 1 1 0 0]
93 + 31 = 124

iters:7300
Loss:0.0036144915510509896
Pred:[1 0 0 1 1 0 0 1]
True:[1 0 0 1 1 0 0 1]
29 + 124 = 153

iters:7400
Loss:0.0017253421441266181
Pred:[0 1 0 1 1 0 1 0]
True:[0 1 0 1 1 0 1 0]
35 + 55 = 90

iters:7500
Loss:0.004265487367589737
Pred:[1 0 0 0 1 0 1 1]
True:[1 0 0 0 1 0 1 1]
68 + 71 = 139

iters:7600
Loss:0.0009192629465717398
Pred:[0 1 1 1 1 0 1 0]
True:[0 1 1 1 1 0 1 0]
73 + 49 = 122

iters:7700
Loss:0.0016109745089048148
Pred:[0 1 1 1 0 1 0 1]
True:[0 1 1 1 0 1 0 1]
25 + 92 = 117

iters:7800
Loss:0.004034188222335441
Pred:[1 1 0 0 1 1 0 0]
True:[1 1 0 0 1 1 0 0]
101 + 103 = 204

iters:7900
Loss:0.0015068901644681267
Pred:[0 0 0 1 0 1 1 1]
True:[0 0 0 1 0 1 1 1]
21 + 2 = 23

iters:8000
Loss:0.0028921206395428087
Pred:[1 0 1 0 1 1 0 0]
True:[1 0 1 0 1 1 0 0]
78 + 94 = 172

iters:8100
Loss:0.0021541052952777627
Pred:[0 1 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 0]
64 + 0 = 64

iters:8200
Loss:0.0012051110845623058
Pred:[1 1 0 1 0 0 0 1]
True:[1 1 0 1 0 0 0 1]
103 + 106 = 209

iters:8300
Loss:0.0006615377536865364
Pred:[0 1 0 0 1 0 1 0]
True:[0 1 0 0 1 0 1 0]
57 + 17 = 74

iters:8400
Loss:0.004019755435218709
Pred:[1 0 0 1 0 1 1 1]
True:[1 0 0 1 0 1 1 1]
116 + 35 = 151

iters:8500
Loss:0.0017858951508926354
Pred:[0 1 0 1 1 0 1 0]
True:[0 1 0 1 1 0 1 0]
46 + 44 = 90

iters:8600
Loss:0.001735071940815487
Pred:[1 0 1 0 1 0 0 1]
True:[1 0 1 0 1 0 0 1]
74 + 95 = 169

iters:8700
Loss:0.0015720321066657742
Pred:[1 0 0 0 0 1 1 1]
True:[1 0 0 0 0 1 1 1]
106 + 29 = 135

iters:8800
Loss:0.0008356004062380853
Pred:[0 1 1 1 1 0 0 1]
True:[0 1 1 1 1 0 0 1]
81 + 40 = 121

iters:8900
Loss:0.0006005301112898576
Pred:[1 0 1 0 0 1 1 0]
True:[1 0 1 0 0 1 1 0]
121 + 45 = 166

iters:9000
Loss:0.0015619235455995083
Pred:[1 0 1 1 0 1 1 1]
True:[1 0 1 1 0 1 1 1]
66 + 117 = 183

iters:9100
Loss:0.001275173637661595
Pred:[0 1 1 1 1 1 1 1]
True:[0 1 1 1 1 1 1 1]
89 + 38 = 127

iters:9200
Loss:0.0010671176834971556
Pred:[1 0 0 1 0 1 0 0]
True:[1 0 0 1 0 1 0 0]
113 + 35 = 148

iters:9300
Loss:0.001227511756333949
Pred:[1 0 1 0 0 0 1 0]
True:[1 0 1 0 0 0 1 0]
56 + 106 = 162

iters:9400
Loss:0.0010302120922175584
Pred:[0 0 1 0 1 0 1 0]
True:[0 0 1 0 1 0 1 0]
30 + 12 = 42

iters:9500
Loss:0.0015300698542473968
Pred:[1 0 0 1 0 1 0 0]
True:[1 0 0 1 0 1 0 0]
22 + 126 = 148

iters:9600
Loss:0.0010929006453181058
Pred:[1 1 1 0 0 0 1 1]
True:[1 1 1 0 0 0 1 1]
125 + 102 = 227

iters:9700
Loss:0.00043740150108243484
Pred:[1 0 0 0 1 1 0 0]
True:[1 0 0 0 1 1 0 0]
51 + 89 = 140

iters:9800
Loss:0.0005453999903817419
Pred:[0 1 1 0 1 1 1 0]
True:[0 1 1 0 1 1 1 0]
23 + 87 = 110

iters:9900
Loss:0.0006595713747989742
Pred:[0 1 1 0 1 1 1 1]
True:[0 1 1 0 1 1 1 1]
99 + 12 = 111

重みの初期化方法をXavierに変更

iters:0
Loss:0.8743476957541628
Pred:[0 0 0 0 1 0 0 0]
True:[0 0 1 0 1 1 0 0]
25 + 19 = 8

iters:100
Loss:1.0053812154829815
Pred:[1 0 1 1 1 1 1 1]
True:[1 0 0 0 0 0 1 1]
63 + 68 = 191

iters:200
Loss:1.0319702379493672
Pred:[0 0 0 1 0 1 1 0]
True:[1 1 0 1 0 0 0 1]
104 + 105 = 22

iters:300
Loss:1.0021175225321353
Pred:[1 1 1 0 0 1 1 1]
True:[1 0 0 1 1 0 1 1]
103 + 52 = 231

iters:400
Loss:0.6714120158541727
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 0]
8 + 56 = 0

iters:500
Loss:0.9999639955362004
Pred:[1 1 1 1 1 1 1 1]
True:[0 1 0 1 1 0 0 1]
45 + 44 = 255

iters:600
Loss:1.151977599161245
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 1 1 1]
92 + 99 = 0

iters:700
Loss:1.0076534536784902
Pred:[0 1 0 1 1 0 0 0]
True:[1 0 0 0 0 0 0 0]
72 + 56 = 88

iters:800
Loss:0.9999617480527359
Pred:[0 1 1 0 1 1 1 0]
True:[1 0 1 0 0 0 1 0]
111 + 51 = 110

iters:900
Loss:0.9948912817325073
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 0 1 1 1]
103 + 96 = 0

iters:1000
Loss:1.0277235151621884
Pred:[1 1 1 1 1 1 1 0]
True:[1 1 0 0 1 0 0 0]
109 + 91 = 254

iters:1100
Loss:0.998015378053679
Pred:[0 1 0 0 0 0 0 0]
True:[0 1 1 1 0 0 0 1]
72 + 41 = 64

iters:1200
Loss:0.9299156076704841
Pred:[1 1 1 1 1 1 1 1]
True:[0 1 1 0 0 1 1 1]
23 + 80 = 255

iters:1300
Loss:1.0312135405098686
Pred:[1 1 0 0 1 0 1 0]
True:[1 1 0 1 0 0 0 0]
107 + 101 = 202

iters:1400
Loss:0.9492293362723804
Pred:[1 1 1 0 0 1 1 0]
True:[0 1 1 1 0 1 1 0]
2 + 116 = 230

iters:1500
Loss:0.9927002161396645
Pred:[0 0 1 0 0 1 0 0]
True:[0 1 1 0 1 1 1 1]
53 + 58 = 36

iters:1600
Loss:1.0570927589331494
Pred:[1 1 1 1 1 1 1 1]
True:[1 1 0 1 1 0 0 0]
123 + 93 = 255

iters:1700
Loss:1.1546911807150908
Pred:[0 1 1 1 1 1 1 0]
True:[1 0 0 1 0 0 0 1]
126 + 19 = 126

iters:1800
Loss:0.9800591900763591
Pred:[0 0 0 0 0 1 0 1]
True:[0 1 0 1 0 1 1 1]
13 + 74 = 5

iters:1900
Loss:0.9599307061795821
Pred:[0 0 1 1 1 1 1 0]
True:[0 1 1 0 0 1 1 0]
62 + 40 = 62

iters:2000
Loss:1.200425920790059
Pred:[0 1 1 0 1 1 1 1]
True:[1 0 1 1 0 0 0 0]
101 + 75 = 111

iters:2100
Loss:0.8527808848776284
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 0 1 1 0 1]
1 + 12 = 0

iters:2200
Loss:0.8263101165071016
Pred:[0 1 0 0 0 0 0 0]
True:[0 1 0 1 0 0 0 0]
32 + 48 = 64

iters:2300
Loss:1.1019464879169547
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 1 1 0 1]
110 + 111 = 0

iters:2400
Loss:0.9929333934635693
Pred:[0 1 1 0 0 0 0 0]
True:[1 0 1 0 1 1 0 0]
50 + 122 = 96

iters:2500
Loss:1.0197083734216665
Pred:[0 1 1 0 1 0 0 0]
True:[1 0 0 0 1 1 0 0]
120 + 20 = 104

iters:2600
Loss:0.8558754082331951
Pred:[0 1 1 1 0 0 1 1]
True:[0 1 0 1 0 0 1 0]
49 + 33 = 115

iters:2700
Loss:1.0611572421729796
Pred:[1 1 1 1 0 1 1 1]
True:[1 0 0 1 0 1 0 1]
58 + 91 = 247

iters:2800
Loss:0.9953747507223866
Pred:[0 0 0 1 0 0 0 0]
True:[0 1 1 1 1 1 1 0]
18 + 108 = 16

iters:2900
Loss:1.0743980836724327
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 1 1 0 0 1 0]
39 + 75 = 78

iters:3000
Loss:1.0938455069427802
Pred:[0 1 1 1 0 1 0 0]
True:[1 0 0 0 0 1 1 0]
124 + 10 = 116

iters:3100
Loss:1.0483936523804354
Pred:[1 1 0 1 1 1 1 1]
True:[0 1 0 1 1 0 0 0]
73 + 15 = 223

iters:3200
Loss:0.8847395492259285
Pred:[0 1 1 0 0 0 0 1]
True:[1 0 0 1 0 0 0 1]
25 + 120 = 97

iters:3300
Loss:0.8055155476673519
Pred:[0 0 0 1 0 0 0 1]
True:[0 0 0 1 1 1 0 1]
1 + 28 = 17

iters:3400
Loss:0.565890454652329
Pred:[0 0 1 0 1 0 0 0]
True:[0 0 1 0 1 1 0 0]
28 + 16 = 40

iters:3500
Loss:0.9729084203790712
Pred:[0 1 1 0 0 0 1 1]
True:[1 0 0 0 1 0 1 1]
82 + 57 = 99

iters:3600
Loss:0.5780496463184954
Pred:[1 0 0 1 1 1 1 1]
True:[1 0 0 1 1 0 1 1]
79 + 76 = 159

iters:3700
Loss:0.8863674711223063
Pred:[0 0 0 0 1 0 1 0]
True:[0 1 0 1 0 1 1 0]
2 + 84 = 10

iters:3800
Loss:0.6952969614115788
Pred:[1 0 0 0 0 0 0 0]
True:[1 0 1 0 1 0 0 1]
96 + 73 = 128

iters:3900
Loss:0.6743904879310779
Pred:[1 1 1 1 1 0 1 0]
True:[0 1 1 1 1 0 1 0]
64 + 58 = 250

iters:4000
Loss:0.5990060790294038
Pred:[1 1 0 1 1 1 0 0]
True:[1 1 0 0 1 1 0 0]
104 + 100 = 220

iters:4100
Loss:0.8547995722331307
Pred:[0 1 1 1 0 1 0 0]
True:[1 0 0 0 0 1 0 0]
58 + 74 = 116

iters:4200
Loss:0.7554846179577659
Pred:[1 0 0 1 1 0 1 1]
True:[1 0 1 1 1 1 1 1]
73 + 118 = 155

iters:4300
Loss:0.4714927185069871
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 0 0 1 1 1 0]
2 + 76 = 78

iters:4400
Loss:0.4695208669568356
Pred:[0 1 1 1 1 0 1 1]
True:[0 1 1 1 1 0 1 1]
118 + 5 = 123

iters:4500
Loss:0.8881960709413956
Pred:[1 1 1 0 1 1 0 0]
True:[1 0 0 0 0 0 0 0]
109 + 19 = 236

iters:4600
Loss:0.77198466697289
Pred:[1 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
125 + 3 = 128

iters:4700
Loss:0.681090076859481
Pred:[1 1 1 1 1 1 1 1]
True:[0 1 1 1 1 1 1 1]
44 + 83 = 255

iters:4800
Loss:0.5251281244313108
Pred:[0 1 0 1 0 0 0 1]
True:[0 1 1 0 0 0 0 1]
57 + 40 = 81

iters:4900
Loss:0.36887244988389095
Pred:[1 0 0 0 1 1 1 0]
True:[1 0 0 0 1 1 1 0]
45 + 97 = 142

iters:5000
Loss:0.6380427367148315
Pred:[1 0 0 1 0 1 1 1]
True:[1 1 0 1 0 1 1 1]
124 + 91 = 151

iters:5100
Loss:0.356849352715217
Pred:[1 0 1 0 1 1 1 1]
True:[1 0 1 0 1 1 1 1]
94 + 81 = 175

iters:5200
Loss:0.27805454402757335
Pred:[0 1 0 1 1 1 1 0]
True:[0 1 0 1 1 1 1 0]
91 + 3 = 94

iters:5300
Loss:0.2031920270758424
Pred:[0 1 1 1 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
36 + 76 = 112

iters:5400
Loss:0.06951250234415077
Pred:[0 1 0 1 1 1 1 0]
True:[0 1 0 1 1 1 1 0]
62 + 32 = 94

iters:5500
Loss:0.0813105102311296
Pred:[0 1 1 0 1 0 0 1]
True:[0 1 1 0 1 0 0 1]
97 + 8 = 105

iters:5600
Loss:0.3275755161660727
Pred:[1 0 1 1 0 0 1 1]
True:[1 0 1 1 0 0 1 1]
124 + 55 = 179

iters:5700
Loss:0.12956675923751834
Pred:[1 0 1 0 1 1 1 1]
True:[1 0 1 0 1 1 1 1]
83 + 92 = 175

iters:5800
Loss:0.1473134729174252
Pred:[1 0 1 0 0 0 1 1]
True:[1 0 1 0 0 0 1 1]
78 + 85 = 163

iters:5900
Loss:0.16935594615831542
Pred:[0 1 0 0 1 1 1 1]
True:[0 1 0 0 1 1 1 1]
64 + 15 = 79

iters:6000
Loss:0.10972886446961741
Pred:[1 0 1 0 0 1 1 0]
True:[1 0 1 0 0 1 1 0]
75 + 91 = 166

iters:6100
Loss:0.20344560496960687
Pred:[0 1 1 0 0 0 0 0]
True:[0 1 1 0 0 0 0 0]
63 + 33 = 96

iters:6200
Loss:0.0543118877663375
Pred:[0 1 1 1 1 1 0 1]
True:[0 1 1 1 1 1 0 1]
120 + 5 = 125

iters:6300
Loss:0.03713534784058617
Pred:[0 1 0 1 1 0 0 0]
True:[0 1 0 1 1 0 0 0]
44 + 44 = 88

iters:6400
Loss:0.043641433649592776
Pred:[0 1 1 0 0 1 1 0]
True:[0 1 1 0 0 1 1 0]
99 + 3 = 102

iters:6500
Loss:0.02079408170900693
Pred:[0 1 1 1 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
40 + 72 = 112

iters:6600
Loss:0.021725142439090524
Pred:[1 0 1 1 1 1 0 0]
True:[1 0 1 1 1 1 0 0]
120 + 68 = 188

iters:6700
Loss:0.03485277363369012
Pred:[1 1 1 0 1 1 0 1]
True:[1 1 1 0 1 1 0 1]
115 + 122 = 237

iters:6800
Loss:0.022207617705429464
Pred:[0 1 1 0 0 1 0 0]
True:[0 1 1 0 0 1 0 0]
99 + 1 = 100

iters:6900
Loss:0.18307604724221657
Pred:[1 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
83 + 45 = 128

iters:7000
Loss:0.028194497285152244
Pred:[1 0 0 1 1 0 1 1]
True:[1 0 0 1 1 0 1 1]
118 + 37 = 155

iters:7100
Loss:0.015800738073792085
Pred:[0 1 0 0 1 0 1 0]
True:[0 1 0 0 1 0 1 0]
48 + 26 = 74

iters:7200
Loss:0.014657687459295892
Pred:[1 0 1 0 1 1 0 0]
True:[1 0 1 0 1 1 0 0]
56 + 116 = 172

iters:7300
Loss:0.006618812543858797
Pred:[0 0 1 0 0 0 1 0]
True:[0 0 1 0 0 0 1 0]
21 + 13 = 34

iters:7400
Loss:0.0315388486599356
Pred:[1 0 0 0 0 1 1 0]
True:[1 0 0 0 0 1 1 0]
79 + 55 = 134

iters:7500
Loss:0.004386680761037107
Pred:[1 0 0 0 1 0 1 0]
True:[1 0 0 0 1 0 1 0]
69 + 69 = 138

iters:7600
Loss:0.009869016619752483
Pred:[0 1 1 0 1 0 1 1]
True:[0 1 1 0 1 0 1 1]
10 + 97 = 107

iters:7700
Loss:0.00792968120276659
Pred:[0 1 1 1 1 1 0 1]
True:[0 1 1 1 1 1 0 1]
84 + 41 = 125

iters:7800
Loss:0.012842519744262103
Pred:[0 0 0 0 1 1 1 1]
True:[0 0 0 0 1 1 1 1]
5 + 10 = 15

iters:7900
Loss:0.010239315242899194
Pred:[1 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
16 + 112 = 128

iters:8000
Loss:0.013440888832475927
Pred:[1 0 1 0 1 0 0 0]
True:[1 0 1 0 1 0 0 0]
123 + 45 = 168

iters:8100
Loss:0.013142651514623165
Pred:[0 0 1 0 0 0 0 0]
True:[0 0 1 0 0 0 0 0]
26 + 6 = 32

iters:8200
Loss:0.002266578654678048
Pred:[1 0 0 0 1 1 1 0]
True:[1 0 0 0 1 1 1 0]
25 + 117 = 142

iters:8300
Loss:0.0021136471152542985
Pred:[0 1 1 0 0 1 1 0]
True:[0 1 1 0 0 1 1 0]
21 + 81 = 102

iters:8400
Loss:0.0024043356562257114
Pred:[1 0 0 1 1 1 0 0]
True:[1 0 0 1 1 1 0 0]
65 + 91 = 156

iters:8500
Loss:0.010450117623833292
Pred:[0 1 1 1 0 1 1 0]
True:[0 1 1 1 0 1 1 0]
31 + 87 = 118

iters:8600
Loss:0.008728823853354701
Pred:[0 1 1 0 0 0 0 0]
True:[0 1 1 0 0 0 0 0]
92 + 4 = 96

iters:8700
Loss:0.009194059578535911
Pred:[1 0 0 1 0 0 0 1]
True:[1 0 0 1 0 0 0 1]
119 + 26 = 145

iters:8800
Loss:0.007894882797475845
Pred:[0 1 1 1 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
0 + 112 = 112

iters:8900
Loss:0.006411823266184286
Pred:[0 0 1 1 1 0 0 1]
True:[0 0 1 1 1 0 0 1]
37 + 20 = 57

iters:9000
Loss:0.0058656677758067936
Pred:[1 0 0 0 1 1 1 1]
True:[1 0 0 0 1 1 1 1]
86 + 57 = 143

iters:9100
Loss:0.0010680732243936806
Pred:[0 1 0 1 0 1 1 0]
True:[0 1 0 1 0 1 1 0]
13 + 73 = 86

iters:9200
Loss:0.002734324116076846
Pred:[1 0 0 0 1 1 0 0]
True:[1 0 0 0 1 1 0 0]
45 + 95 = 140

iters:9300
Loss:0.0061421479028298176
Pred:[0 0 1 0 0 1 1 0]
True:[0 0 1 0 0 1 1 0]
18 + 20 = 38

iters:9400
Loss:0.0015680054169851232
Pred:[1 1 0 1 0 0 1 0]
True:[1 1 0 1 0 0 1 0]
85 + 125 = 210

iters:9500
Loss:0.0044470989325449675
Pred:[1 0 0 1 1 1 0 1]
True:[1 0 0 1 1 1 0 1]
89 + 68 = 157

iters:9600
Loss:0.003379983157143373
Pred:[0 0 0 1 1 1 0 1]
True:[0 0 0 1 1 1 0 1]
22 + 7 = 29

iters:9700
Loss:0.003678196254456772
Pred:[1 0 0 0 1 0 0 1]
True:[1 0 0 0 1 0 0 1]
28 + 109 = 137

iters:9800
Loss:0.0035800081447896256
Pred:[0 1 1 0 0 1 1 1]
True:[0 1 1 0 0 1 1 1]
24 + 79 = 103

iters:9900
Loss:0.0022785260619156743
Pred:[1 1 0 0 1 1 0 0]
True:[1 1 0 0 1 1 0 0]
115 + 89 = 204

重みの初期化方法をHeに変更

iters:0
Loss:1.0200468831981315
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 1 0 0]
103 + 61 = 0

iters:100
Loss:0.9375384526416728
Pred:[0 1 0 1 1 1 1 1]
True:[1 0 0 1 1 1 1 1]
88 + 71 = 95

iters:200
Loss:0.8714428733881315
Pred:[0 1 1 1 1 1 1 1]
True:[0 0 1 1 1 1 1 1]
10 + 53 = 127

iters:300
Loss:0.9582727580785477
Pred:[0 0 0 0 1 0 1 1]
True:[0 1 0 0 1 0 1 1]
62 + 13 = 11

iters:400
Loss:0.8874921879189306
Pred:[1 1 1 1 1 1 1 1]
True:[0 1 1 1 1 1 1 0]
117 + 9 = 255

iters:500
Loss:0.9989939756846553
Pred:[1 1 1 1 1 1 1 0]
True:[0 0 1 0 0 0 1 0]
34 + 0 = 254

iters:600
Loss:1.0988568262200276
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 0 0 0 1 0 1]
30 + 103 = 255

iters:700
Loss:1.0032447469648518
Pred:[0 1 0 1 1 0 1 1]
True:[0 1 0 1 0 1 0 0]
37 + 47 = 91

iters:800
Loss:0.770734965371308
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 0 1 0 0]
0 + 36 = 0

iters:900
Loss:1.143190247601115
Pred:[1 1 1 1 1 1 1 1]
True:[1 0 0 0 0 1 0 1]
22 + 111 = 255

iters:1000
Loss:0.8732133064271415
Pred:[0 1 0 0 0 0 1 1]
True:[1 0 0 0 0 0 1 1]
98 + 33 = 67

iters:1100
Loss:0.8926725464755759
Pred:[0 1 1 0 1 1 1 1]
True:[0 1 1 1 1 0 1 1]
23 + 100 = 111

iters:1200
Loss:1.0146193600618603
Pred:[0 1 0 0 1 1 0 1]
True:[0 1 1 1 0 0 0 1]
74 + 39 = 77

iters:1300
Loss:1.0076497680614185
Pred:[1 0 1 0 0 0 0 0]
True:[1 0 1 1 0 1 0 1]
96 + 85 = 160

iters:1400
Loss:0.8910033755584573
Pred:[0 0 1 1 1 0 1 0]
True:[0 0 1 1 1 1 0 1]
56 + 5 = 58

iters:1500
Loss:1.1444351560439063
Pred:[1 1 1 1 1 0 1 1]
True:[1 0 1 0 1 1 0 0]
51 + 121 = 251

iters:1600
Loss:0.920635477562876
Pred:[1 1 0 0 1 0 0 1]
True:[1 0 0 1 1 0 0 1]
37 + 116 = 201

iters:1700
Loss:0.9317291118677138
Pred:[0 1 1 1 1 1 0 1]
True:[0 0 1 1 1 0 0 1]
18 + 39 = 125

iters:1800
Loss:0.8626339046477632
Pred:[1 1 1 1 1 1 0 0]
True:[0 1 1 1 1 1 1 0]
36 + 90 = 252

iters:1900
Loss:0.8296859065715564
Pred:[0 0 1 0 1 1 0 1]
True:[0 1 0 0 1 1 1 1]
29 + 50 = 45

iters:2000
Loss:1.0142714336160408
Pred:[1 0 1 0 1 0 1 0]
True:[1 1 0 0 1 1 0 0]
87 + 117 = 170

iters:2100
Loss:1.450692970387027
Pred:[0 1 1 1 1 1 1 0]
True:[1 0 0 0 0 0 0 0]
107 + 21 = 126

iters:2200
Loss:0.6267670543476008
Pred:[0 0 1 1 1 1 1 1]
True:[0 0 1 1 1 1 1 1]
33 + 30 = 63

iters:2300
Loss:0.8783956916310335
Pred:[1 1 0 1 1 1 1 1]
True:[0 1 0 1 0 1 0 1]
16 + 69 = 223

iters:2400
Loss:0.9339382730068279
Pred:[1 1 1 0 1 1 0 1]
True:[1 0 0 0 1 1 0 1]
18 + 123 = 237

iters:2500
Loss:0.6406188244895901
Pred:[0 1 0 0 0 0 0 0]
True:[1 0 0 0 0 0 1 0]
32 + 98 = 64

iters:2600
Loss:0.804859653068406
Pred:[1 1 0 1 1 0 1 1]
True:[1 1 1 0 1 0 1 1]
109 + 126 = 219

iters:2700
Loss:0.749449479737911
Pred:[0 1 1 1 1 1 1 0]
True:[0 1 1 1 1 1 1 0]
95 + 31 = 126

iters:2800
Loss:0.7478273252732904
Pred:[0 1 1 1 1 0 1 1]
True:[0 1 0 1 1 0 1 1]
31 + 60 = 123

iters:2900
Loss:0.7776606433561271
Pred:[0 1 1 1 1 0 1 0]
True:[0 1 1 1 0 1 1 0]
52 + 66 = 122

iters:3000
Loss:0.38989394970538754
Pred:[1 0 1 0 1 1 1 1]
True:[1 0 1 0 1 1 1 1]
95 + 80 = 175

iters:3100
Loss:1.0551807764467769
Pred:[0 1 0 1 1 1 1 0]
True:[0 1 1 0 0 0 0 0]
91 + 5 = 94

iters:3200
Loss:0.5897786303597473
Pred:[1 1 1 1 1 1 1 0]
True:[0 1 1 1 1 1 1 0]
5 + 121 = 254

iters:3300
Loss:0.574757479268692
Pred:[0 1 1 1 1 0 1 1]
True:[0 1 1 1 1 0 1 1]
16 + 107 = 123

iters:3400
Loss:0.3279757873195445
Pred:[0 1 1 1 1 1 0 0]
True:[0 1 1 1 1 1 0 0]
58 + 66 = 124

iters:3500
Loss:0.2759865807372145
Pred:[0 1 1 0 1 0 1 0]
True:[0 1 1 0 1 0 1 0]
49 + 57 = 106

iters:3600
Loss:0.4700202480697368
Pred:[0 1 1 0 1 0 0 0]
True:[0 1 1 0 1 1 0 0]
69 + 39 = 104

iters:3700
Loss:0.5064117003770365
Pred:[0 1 1 0 1 1 1 1]
True:[0 1 1 0 1 1 1 1]
70 + 41 = 111

iters:3800
Loss:0.6497630798406375
Pred:[0 1 1 1 1 1 1 1]
True:[0 1 1 1 1 1 1 1]
56 + 71 = 127

iters:3900
Loss:0.77276151662407
Pred:[1 0 0 0 0 1 0 0]
True:[1 0 0 0 1 0 0 0]
13 + 123 = 132

iters:4000
Loss:0.41073830625074953
Pred:[0 1 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 0]
37 + 27 = 64

iters:4100
Loss:0.2564845320885512
Pred:[0 0 0 1 1 0 0 0]
True:[0 0 0 1 0 0 0 0]
12 + 4 = 24

iters:4200
Loss:0.3410619647031007
Pred:[0 1 1 0 1 0 0 0]
True:[0 1 1 0 1 0 0 0]
43 + 61 = 104

iters:4300
Loss:0.09441940800325974
Pred:[0 0 1 1 0 1 0 0]
True:[0 0 1 1 0 1 0 0]
27 + 25 = 52

iters:4400
Loss:0.17055846076879955
Pred:[0 0 0 0 1 1 1 1]
True:[0 0 0 0 1 1 1 1]
5 + 10 = 15

iters:4500
Loss:0.020873164682359377
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 0 0 1 1 1 0]
77 + 1 = 78

iters:4600
Loss:0.13579197187954592
Pred:[0 1 1 0 0 1 0 0]
True:[0 1 1 0 0 1 0 0]
60 + 40 = 100

iters:4700
Loss:0.07910620724853322
Pred:[1 1 0 0 1 0 1 1]
True:[1 1 0 0 1 0 1 1]
117 + 86 = 203

iters:4800
Loss:0.12627494810251955
Pred:[0 1 0 0 1 1 0 1]
True:[0 1 0 0 1 1 0 1]
46 + 31 = 77

iters:4900
Loss:0.019257025484386188
Pred:[0 1 1 0 1 1 0 0]
True:[0 1 1 0 1 1 0 0]
75 + 33 = 108

iters:5000
Loss:0.029616775355105554
Pred:[1 0 0 0 1 1 0 1]
True:[1 0 0 0 1 1 0 1]
92 + 49 = 141

iters:5100
Loss:0.04476281948653035
Pred:[1 0 0 1 0 1 0 0]
True:[1 0 0 1 0 1 0 0]
53 + 95 = 148

iters:5200
Loss:0.02153144821063555
Pred:[1 0 0 0 1 0 0 1]
True:[1 0 0 0 1 0 0 1]
49 + 88 = 137

iters:5300
Loss:0.011237082972040447
Pred:[0 1 0 0 0 1 1 0]
True:[0 1 0 0 0 1 1 0]
13 + 57 = 70

iters:5400
Loss:0.012832588883085944
Pred:[1 0 0 0 1 1 1 0]
True:[1 0 0 0 1 1 1 0]
51 + 91 = 142

iters:5500
Loss:0.011485822079607449
Pred:[0 1 1 0 1 1 0 1]
True:[0 1 1 0 1 1 0 1]
93 + 16 = 109

iters:5600
Loss:0.021825032347262782
Pred:[1 0 1 0 0 1 0 1]
True:[1 0 1 0 0 1 0 1]
126 + 39 = 165

iters:5700
Loss:0.015441305868100882
Pred:[1 0 1 1 1 1 1 1]
True:[1 0 1 1 1 1 1 1]
73 + 118 = 191

iters:5800
Loss:0.007499289320730973
Pred:[1 0 1 0 1 1 1 0]
True:[1 0 1 0 1 1 1 0]
53 + 121 = 174

iters:5900
Loss:0.014289688628773195
Pred:[0 1 0 0 0 0 0 1]
True:[0 1 0 0 0 0 0 1]
10 + 55 = 65

iters:6000
Loss:0.010080331289304715
Pred:[1 0 0 0 1 0 1 1]
True:[1 0 0 0 1 0 1 1]
90 + 49 = 139

iters:6100
Loss:0.01766729288553398
Pred:[0 1 1 0 0 0 0 0]
True:[0 1 1 0 0 0 0 0]
22 + 74 = 96

iters:6200
Loss:0.005520327068884506
Pred:[0 1 1 1 1 0 1 0]
True:[0 1 1 1 1 0 1 0]
101 + 21 = 122

iters:6300
Loss:0.011268748962385404
Pred:[1 0 1 0 1 0 1 1]
True:[1 0 1 0 1 0 1 1]
111 + 60 = 171

iters:6400
Loss:0.004932772595621007
Pred:[0 0 0 0 1 1 0 1]
True:[0 0 0 0 1 1 0 1]
1 + 12 = 13

iters:6500
Loss:0.0019175404949832169
Pred:[1 0 0 1 1 0 1 0]
True:[1 0 0 1 1 0 1 0]
41 + 113 = 154

iters:6600
Loss:0.0072700182320300675
Pred:[0 1 1 1 1 1 0 1]
True:[0 1 1 1 1 1 0 1]
59 + 66 = 125

iters:6700
Loss:0.005144391704583873
Pred:[0 1 1 1 0 1 1 0]
True:[0 1 1 1 0 1 1 0]
0 + 118 = 118

iters:6800
Loss:0.004609841036868185
Pred:[1 0 0 0 0 0 0 1]
True:[1 0 0 0 0 0 0 1]
124 + 5 = 129

iters:6900
Loss:0.002924814853004286
Pred:[0 1 1 1 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
9 + 103 = 112

iters:7000
Loss:0.003306036729174898
Pred:[0 1 0 1 1 0 0 0]
True:[0 1 0 1 1 0 0 0]
59 + 29 = 88

iters:7100
Loss:0.0005717439160445153
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 0 0 1 1 1 0]
77 + 1 = 78

iters:7200
Loss:0.005151603484671542
Pred:[0 1 1 0 1 0 0 0]
True:[0 1 1 0 1 0 0 0]
72 + 32 = 104

iters:7300
Loss:0.00838481612268285
Pred:[1 1 0 0 1 1 0 0]
True:[1 1 0 0 1 1 0 0]
94 + 110 = 204

iters:7400
Loss:0.001477568712596599
Pred:[1 0 0 0 0 0 1 0]
True:[1 0 0 0 0 0 1 0]
125 + 5 = 130

iters:7500
Loss:0.002793380493892691
Pred:[1 0 0 0 1 1 0 1]
True:[1 0 0 0 1 1 0 1]
109 + 32 = 141

iters:7600
Loss:0.00542133885551524
Pred:[0 1 1 1 0 1 0 0]
True:[0 1 1 1 0 1 0 0]
12 + 104 = 116

iters:7700
Loss:0.0026173592682335652
Pred:[0 1 0 0 0 0 1 0]
True:[0 1 0 0 0 0 1 0]
15 + 51 = 66

iters:7800
Loss:0.0007621087853581949
Pred:[0 0 1 1 1 1 1 0]
True:[0 0 1 1 1 1 1 0]
23 + 39 = 62

iters:7900
Loss:0.004462834698619396
Pred:[0 1 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 0]
8 + 56 = 64

iters:8000
Loss:0.0014987219510469332
Pred:[1 0 0 1 0 1 1 0]
True:[1 0 0 1 0 1 1 0]
79 + 71 = 150

iters:8100
Loss:0.0004915558643186704
Pred:[0 0 0 1 0 1 0 0]
True:[0 0 0 1 0 1 0 0]
3 + 17 = 20

iters:8200
Loss:0.0022227223699103557
Pred:[1 1 0 1 0 1 0 1]
True:[1 1 0 1 0 1 0 1]
111 + 102 = 213

iters:8300
Loss:0.0021477590790413273
Pred:[1 0 0 0 0 0 0 1]
True:[1 0 0 0 0 0 0 1]
105 + 24 = 129

iters:8400
Loss:0.00033420726900565114
Pred:[0 1 0 0 1 1 1 0]
True:[0 1 0 0 1 1 1 0]
45 + 33 = 78

iters:8500
Loss:0.0028691025429902414
Pred:[1 0 0 0 0 1 1 1]
True:[1 0 0 0 0 1 1 1]
105 + 30 = 135

iters:8600
Loss:0.0005643278952914295
Pred:[1 1 0 1 1 1 0 0]
True:[1 1 0 1 1 1 0 0]
101 + 119 = 220

iters:8700
Loss:0.0016707415443318527
Pred:[1 0 1 1 1 1 1 1]
True:[1 0 1 1 1 1 1 1]
66 + 125 = 191

iters:8800
Loss:0.0009173822344912654
Pred:[0 0 1 1 1 1 0 0]
True:[0 0 1 1 1 1 0 0]
29 + 31 = 60

iters:8900
Loss:0.00028471526040969245
Pred:[0 1 1 0 0 1 0 0]
True:[0 1 1 0 0 1 0 0]
33 + 67 = 100

iters:9000
Loss:0.002168118728939596
Pred:[0 1 1 0 0 1 1 1]
True:[0 1 1 0 0 1 1 1]
79 + 24 = 103

iters:9100
Loss:0.002445075871485941
Pred:[0 1 0 1 0 0 1 0]
True:[0 1 0 1 0 0 1 0]
64 + 18 = 82

iters:9200
Loss:0.0010848400370720456
Pred:[1 1 1 0 1 1 0 0]
True:[1 1 1 0 1 1 0 0]
109 + 127 = 236

iters:9300
Loss:0.0015476370603994458
Pred:[1 0 0 0 0 1 0 1]
True:[1 0 0 0 0 1 0 1]
45 + 88 = 133

iters:9400
Loss:0.001388684414631984
Pred:[0 1 0 0 0 0 0 1]
True:[0 1 0 0 0 0 0 1]
19 + 46 = 65

iters:9500
Loss:0.0007317443308026744
Pred:[0 0 1 0 0 0 0 0]
True:[0 0 1 0 0 0 0 0]
15 + 17 = 32

iters:9600
Loss:0.0026223242734541075
Pred:[0 1 0 1 0 1 0 0]
True:[0 1 0 1 0 1 0 0]
32 + 52 = 84

iters:9700
Loss:0.0010913408417998194
Pred:[0 1 1 0 0 1 0 1]
True:[0 1 1 0 0 1 0 1]
51 + 50 = 101

iters:9800
Loss:0.00015990629218724477
Pred:[0 1 1 1 1 0 1 0]
True:[0 1 1 1 1 0 1 0]
77 + 45 = 122

iters:9900
Loss:0.0005424952109513927
Pred:[0 1 1 1 0 1 0 0]
True:[0 1 1 1 0 1 0 0]
13 + 103 = 116

中間層の活性化関数をReLUに変更してみる

iters:0
Loss:2.5207671175350304
Pred:[0 1 0 1 0 0 0 0]
True:[0 0 1 0 1 0 1 1]
28 + 15 = 80

iters:100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 1 0 0 1]
49 + 88 = 0

iters:200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 0 1 1 0]
75 + 43 = 0

iters:300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 1 0 0 0]
112 + 8 = 0

iters:400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 1 1]
3 + 108 = 0

iters:500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 0 1 1 1]
89 + 110 = 0

iters:600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 0 0 1 1]
13 + 6 = 0

iters:700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 0 1 1 0]
108 + 74 = 0

iters:800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 1 1]
97 + 54 = 0

iters:900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 1 1 0 1]
88 + 37 = 0

iters:1000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 0 1 0]
115 + 31 = 0

iters:1100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 0 0 0 0]
41 + 71 = 0

iters:1200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 1 1 0 0]
113 + 107 = 0

iters:1300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 0 1 0]
75 + 15 = 0

iters:1400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 1 0 1 0]
4 + 22 = 0

iters:1500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 1 0 1 1]
93 + 62 = 0

iters:1600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 0 0 1 1]
127 + 84 = 0

iters:1700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 0 1 0]
29 + 61 = 0

iters:1800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 0 1 0]
92 + 14 = 0

iters:1900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 0 0 0 1]
106 + 71 = 0

iters:2000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 1 1 0 0]
60 + 112 = 0

iters:2100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 0 0 1 1]
112 + 3 = 0

iters:2200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 1 1 0 0]
83 + 89 = 0

iters:2300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 1 1]
98 + 13 = 0

iters:2400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 0 0 1 1]
84 + 127 = 0

iters:2500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 1 1 0]
12 + 66 = 0

iters:2600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 1 0 0 0 1]
12 + 37 = 0

iters:2700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 1 1 0 0 1 0]
119 + 123 = 0

iters:2800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 0 1 1 1]
40 + 31 = 0

iters:2900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 0 0 1]
27 + 46 = 0

iters:3000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 1 0 0 0]
11 + 13 = 0

iters:3100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 1 0 1 1]
24 + 19 = 0

iters:3200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 0 1 1 0]
6 + 112 = 0

iters:3300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 0 0 1 0]
61 + 117 = 0

iters:3400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 0 0 1]
48 + 41 = 0

iters:3500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 1 0 1 1]
53 + 118 = 0

iters:3600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 1 0]
57 + 53 = 0

iters:3700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 1 0 1 1]
105 + 98 = 0

iters:3800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 0 0 0 0]
103 + 89 = 0

iters:3900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 1 0 0 0]
52 + 84 = 0

iters:4000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 0 0 0]
88 + 96 = 0

iters:4100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 1 0 1 1 0]
36 + 18 = 0

iters:4200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 0 0]
101 + 47 = 0

iters:4300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 1 0 0 0]
44 + 92 = 0

iters:4400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 1 1 1]
70 + 9 = 0

iters:4500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 1 1 0]
31 + 63 = 0

iters:4600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 1 1 0 0 0 1]
121 + 120 = 0

iters:4700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 1 0 0 1]
36 + 5 = 0

iters:4800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 0 0 1 1]
62 + 37 = 0

iters:4900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 0 1 1 0]
23 + 15 = 0

iters:5000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 0 1 0 0]
35 + 1 = 0

iters:5100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 1 1]
90 + 61 = 0

iters:5200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 0 1]
78 + 71 = 0

iters:5300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 0 1 0 1]
101 + 112 = 0

iters:5400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 0 1]
23 + 86 = 0

iters:5500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 1 0 0 1]
97 + 24 = 0

iters:5600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 1 0 0 0 0 1]
111 + 114 = 0

iters:5700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 1 1 0]
76 + 114 = 0

iters:5800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 1 0]
106 + 44 = 0

iters:5900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 0 0 1 1]
11 + 72 = 0

iters:6000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 0 1]
109 + 0 = 0

iters:6100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 0 0 1]
38 + 123 = 0

iters:6200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 1 1 1]
45 + 122 = 0

iters:6300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 1 0 0 1 0]
20 + 30 = 0

iters:6400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 1 0 0 0]
4 + 20 = 0

iters:6500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 0 0 0]
11 + 61 = 0

iters:6600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 1 0]
60 + 90 = 0

iters:6700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 0 1 1 0]
107 + 27 = 0

iters:6800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 0 1 1 1]
125 + 90 = 0

iters:6900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 0]
36 + 28 = 0

iters:7000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 1 0 1 0 1]
89 + 124 = 0

iters:7100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 0 0 1 1 0 0]
127 + 77 = 0

iters:7200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 0 0 1]
97 + 88 = 0

iters:7300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 1 0 1]
101 + 64 = 0

iters:7400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 1 0 0 0]
12 + 28 = 0

iters:7500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 0 0 0]
101 + 83 = 0

iters:7600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 0 0 1 1]
49 + 18 = 0

iters:7700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 0 1 1]
68 + 95 = 0

iters:7800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 1 1 1 1]
104 + 55 = 0

iters:7900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 0 1 0 0]
60 + 24 = 0

iters:8000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 1 0 0 0 0 0]
32 + 0 = 0

iters:8100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 1 0 0 0]
118 + 34 = 0

iters:8200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 0 1 1]
6 + 101 = 0

iters:8300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 0 1 1 1 0]
103 + 7 = 0

iters:8400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 1 0 0 0]
100 + 52 = 0

iters:8500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
87 + 41 = 0

iters:8600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 1 0 0 0]
62 + 106 = 0

iters:8700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 0 0 0]
117 + 27 = 0

iters:8800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 1 1 1 0 1 1]
106 + 17 = 0

iters:8900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 1 0 1 1]
19 + 8 = 0

iters:9000
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 0 1 0]
7 + 83 = 0

iters:9100
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 0 0 0 1]
23 + 42 = 0

iters:9200
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 1 1 1 0 0]
54 + 38 = 0

iters:9300
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 1 0 0 1 0 1 1]
9 + 66 = 0

iters:9400
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 0 0 1 1 1]
78 + 89 = 0

iters:9500
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[0 0 0 1 1 1 0 0]
2 + 26 = 0

iters:9600
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 1 1 0 0 1 0 1]
118 + 111 = 0

iters:9700
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 1 0 0 0]
39 + 113 = 0

iters:9800
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 1 0 1 1 0]
58 + 92 = 0

iters:9900
Loss:1.0
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 1 1 1 1 0 1]
76 + 113 = 0

勾配爆発

中間層の活性化関数をtanhに変更してみる

iters:0
Loss:2.3132948464427683
Pred:[0 1 0 0 0 0 0 0]
True:[1 1 0 0 1 1 1 1]
124 + 83 = 64

iters:100
Loss:1.0062824901483274
Pred:[1 0 1 0 1 0 0 0]
True:[1 0 0 1 1 0 1 0]
84 + 70 = 168

iters:200
Loss:1.1087730618558893
Pred:[0 0 0 0 0 1 1 1]
True:[0 1 1 0 1 1 0 1]
72 + 37 = 7

iters:300
Loss:1.0663857912252155
Pred:[0 0 1 1 0 1 0 1]
True:[0 1 0 1 1 1 1 0]
63 + 31 = 53

iters:400
Loss:0.9948407094326963
Pred:[0 1 0 0 0 0 0 0]
True:[0 0 1 0 0 0 0 0]
16 + 16 = 64

iters:500
Loss:1.1351286565741545
Pred:[0 1 0 0 1 0 0 0]
True:[0 1 1 1 1 1 0 1]
87 + 38 = 72

iters:600
Loss:1.0886459689850974
Pred:[1 1 0 1 0 0 1 0]
True:[0 1 0 1 1 1 1 0]
6 + 88 = 210

iters:700
Loss:0.9394990901658775
Pred:[1 1 0 1 1 0 0 0]
True:[0 1 0 1 1 0 0 0]
62 + 26 = 216

iters:800
Loss:1.2880328550543154
Pred:[1 0 1 1 1 1 1 0]
True:[1 0 0 0 0 0 1 0]
12 + 118 = 190

iters:900
Loss:0.7877801091507073
Pred:[0 0 1 0 1 1 1 1]
True:[0 1 1 0 1 1 0 1]
50 + 59 = 47

iters:1000
Loss:1.175936555933746
Pred:[0 0 1 1 1 0 1 0]
True:[1 0 1 0 0 1 1 0]
103 + 63 = 58

iters:1100
Loss:0.8165234068860712
Pred:[1 0 1 1 0 1 0 0]
True:[0 1 1 1 0 1 0 0]
38 + 78 = 180

iters:1200
Loss:0.9913095232990881
Pred:[0 0 1 1 1 0 0 1]
True:[0 1 1 1 0 1 0 1]
67 + 50 = 57

iters:1300
Loss:0.9497946170064058
Pred:[0 1 0 1 1 0 0 1]
True:[0 1 0 0 0 1 0 1]
21 + 48 = 89

iters:1400
Loss:1.156473061386165
Pred:[1 0 0 1 0 0 1 0]
True:[1 1 0 0 1 0 1 0]
122 + 80 = 146

iters:1500
Loss:1.2066053971536685
Pred:[0 1 1 0 1 0 1 1]
True:[1 0 0 0 0 0 0 1]
50 + 79 = 107

iters:1600
Loss:0.8754790793829794
Pred:[0 0 0 0 0 0 0 0]
True:[1 0 0 0 0 0 0 0]
32 + 96 = 0

iters:1700
Loss:0.9007051238072981
Pred:[1 0 0 1 0 1 0 1]
True:[0 1 1 1 0 1 1 1]
39 + 80 = 149

iters:1800
Loss:0.4803840297162036
Pred:[1 0 1 0 1 1 0 0]
True:[1 0 1 0 1 1 0 0]
45 + 127 = 172

iters:1900
Loss:0.7811528276462031
Pred:[0 1 0 1 1 0 0 0]
True:[0 1 1 1 1 0 0 0]
24 + 96 = 88

iters:2000
Loss:0.7871764409357176
Pred:[1 1 0 1 0 1 1 1]
True:[1 1 0 1 1 0 1 1]
113 + 106 = 215

iters:2100
Loss:0.5557118265892158
Pred:[1 0 1 0 1 0 1 1]
True:[1 0 1 1 1 1 1 1]
103 + 88 = 171

iters:2200
Loss:1.0300319190061586
Pred:[0 1 0 0 1 0 1 1]
True:[0 0 1 1 1 1 0 1]
55 + 6 = 75

iters:2300
Loss:0.9783533676665803
Pred:[0 1 1 0 1 1 0 0]
True:[0 0 1 1 0 0 0 0]
6 + 42 = 108

iters:2400
Loss:0.6528903229564557
Pred:[1 0 1 0 1 0 1 0]
True:[1 1 1 0 0 0 1 0]
101 + 125 = 170

iters:2500
Loss:0.760741133494967
Pred:[1 1 1 1 1 0 1 0]
True:[1 0 1 1 1 1 1 0]
105 + 85 = 250

iters:2600
Loss:0.6448304342769581
Pred:[0 0 1 0 1 0 0 0]
True:[1 0 0 1 1 0 0 0]
25 + 127 = 40

iters:2700
Loss:0.6963135409868314
Pred:[1 1 0 0 1 1 1 0]
True:[1 1 0 0 0 1 1 0]
112 + 86 = 206

iters:2800
Loss:0.47473082870644684
Pred:[0 0 1 1 0 0 0 1]
True:[0 0 1 1 1 0 0 1]
5 + 52 = 49

iters:2900
Loss:0.874855638172857
Pred:[1 1 0 0 1 0 1 1]
True:[1 1 0 1 0 1 0 1]
114 + 99 = 203

iters:3000
Loss:0.8268262677764708
Pred:[0 1 0 0 1 1 0 0]
True:[1 0 0 0 1 1 0 0]
60 + 80 = 76

iters:3100
Loss:0.6526299403435776
Pred:[1 0 1 0 1 0 0 1]
True:[1 0 0 1 1 1 0 1]
67 + 90 = 169

iters:3200
Loss:0.8446296002653312
Pred:[1 0 1 1 0 0 0 1]
True:[1 1 0 0 0 0 0 1]
114 + 79 = 177

iters:3300
Loss:0.915739914338694
Pred:[1 0 0 1 0 0 0 1]
True:[0 1 1 1 0 1 0 1]
111 + 6 = 145

iters:3400
Loss:0.9521146208139456
Pred:[1 1 1 0 1 1 1 0]
True:[1 1 0 1 0 1 0 0]
99 + 113 = 238

iters:3500
Loss:0.6697188826199445
Pred:[1 1 0 1 0 1 0 1]
True:[1 1 0 0 0 1 0 1]
126 + 71 = 213

iters:3600
Loss:0.4801837322101745
Pred:[0 0 1 0 1 1 1 0]
True:[0 0 1 0 0 1 1 0]
26 + 12 = 46

iters:3700
Loss:0.7479089675829357
Pred:[1 1 0 0 1 0 1 0]
True:[1 1 1 1 0 1 1 0]
121 + 125 = 202

iters:3800
Loss:0.6931810886988582
Pred:[0 1 0 1 1 1 1 0]
True:[0 0 1 1 1 1 1 0]
54 + 8 = 94

iters:3900
Loss:0.7787003937547938
Pred:[1 1 0 0 1 1 1 0]
True:[0 1 1 1 1 0 1 0]
17 + 105 = 206

iters:4000
Loss:0.7195657877539869
Pred:[1 0 0 1 0 0 1 1]
True:[1 0 0 0 0 1 1 1]
73 + 62 = 147

iters:4100
Loss:0.5844759516465444
Pred:[1 1 0 1 0 1 0 1]
True:[1 1 1 1 0 0 0 1]
126 + 115 = 213

iters:4200
Loss:0.9167068834703567
Pred:[1 0 1 0 1 1 0 0]
True:[1 0 0 0 0 0 0 0]
54 + 74 = 172

iters:4300
Loss:0.6584861117347192
Pred:[1 1 1 0 1 0 0 1]
True:[1 1 1 0 0 0 0 1]
115 + 110 = 233

iters:4400
Loss:0.6076582342696376
Pred:[1 0 1 1 1 0 1 1]
True:[1 0 0 1 0 0 1 1]
105 + 42 = 187

iters:4500
Loss:0.7099520222103561
Pred:[1 1 1 0 1 0 1 1]
True:[1 0 1 1 1 1 1 1]
89 + 102 = 235

iters:4600
Loss:0.9061440365082576
Pred:[1 1 0 1 1 0 0 1]
True:[1 0 1 1 1 1 0 1]
75 + 114 = 217

iters:4700
Loss:0.6524912707654664
Pred:[0 0 1 1 0 0 1 0]
True:[0 1 0 0 0 0 1 0]
10 + 56 = 50

iters:4800
Loss:0.8222562115380363
Pred:[0 1 0 1 0 1 0 0]
True:[1 0 1 1 0 1 0 0]
120 + 60 = 84

iters:4900
Loss:0.8433673448017093
Pred:[1 1 1 0 0 0 0 1]
True:[1 0 0 0 0 1 0 1]
24 + 109 = 225

iters:5000
Loss:0.5438755727757265
Pred:[0 1 1 0 1 0 1 0]
True:[0 1 1 1 1 1 1 0]
29 + 97 = 106

iters:5100
Loss:0.9597285273124218
Pred:[1 1 0 0 1 0 0 1]
True:[1 0 1 1 1 1 0 1]
87 + 102 = 201

iters:5200
Loss:0.9574398778205091
Pred:[1 0 1 1 0 1 1 1]
True:[1 1 0 1 0 0 1 1]
125 + 86 = 183

iters:5300
Loss:0.7546493508664603
Pred:[1 0 0 1 1 0 0 1]
True:[1 0 0 0 0 1 0 1]
11 + 122 = 153

iters:5400
Loss:0.6432701011902581
Pred:[1 0 0 1 0 0 1 1]
True:[1 0 0 0 1 0 1 1]
64 + 75 = 147

iters:5500
Loss:0.8308850281476157
Pred:[1 0 1 0 1 1 1 1]
True:[1 0 1 1 0 0 1 1]
62 + 117 = 175

iters:5600
Loss:1.0387588165844805
Pred:[1 1 0 1 0 1 0 0]
True:[0 1 1 0 1 1 0 0]
97 + 11 = 212

iters:5700
Loss:0.5559528267142806
Pred:[0 1 0 1 0 1 0 0]
True:[0 1 1 0 0 1 0 0]
45 + 55 = 84

iters:5800
Loss:0.5362626977165481
Pred:[1 0 1 1 0 1 1 1]
True:[1 0 0 0 0 1 1 1]
59 + 76 = 183

iters:5900
Loss:1.0283717908440488
Pred:[1 1 1 0 0 1 0 0]
True:[0 1 0 1 0 1 0 0]
12 + 72 = 228

iters:6000
Loss:0.8434713155994126
Pred:[0 0 1 0 1 1 1 1]
True:[0 1 0 0 1 0 1 1]
37 + 38 = 47

iters:6100
Loss:0.7558873724934484
Pred:[1 1 1 1 0 1 1 0]
True:[1 0 0 1 0 1 1 0]
76 + 74 = 246

iters:6200
Loss:0.4495916778088093
Pred:[0 0 0 1 0 1 1 0]
True:[0 0 0 0 1 1 1 0]
3 + 11 = 22

iters:6300
Loss:0.7905233366862243
Pred:[0 0 0 1 0 1 0 1]
True:[1 0 1 0 0 1 0 1]
122 + 43 = 21

iters:6400
Loss:0.43243981802284087
Pred:[0 1 0 1 1 0 0 1]
True:[0 1 0 0 1 0 0 1]
34 + 39 = 89

iters:6500
Loss:1.495394883024126
Pred:[0 1 1 0 0 0 1 1]
True:[1 1 0 1 0 0 1 1]
88 + 123 = 99

iters:6600
Loss:0.7283969317584513
Pred:[0 1 1 0 0 1 1 0]
True:[0 1 0 0 0 0 1 0]
17 + 49 = 102

iters:6700
Loss:0.8034417231584592
Pred:[1 0 0 0 1 0 1 1]
True:[0 1 1 1 1 0 1 1]
107 + 16 = 139

iters:6800
Loss:1.0125384576761993
Pred:[1 1 1 0 1 0 1 0]
True:[1 0 1 1 0 1 1 0]
88 + 94 = 234

iters:6900
Loss:0.6539157965609765
Pred:[0 1 0 1 1 0 0 0]
True:[0 1 1 1 1 0 0 0]
120 + 0 = 88

iters:7000
Loss:0.8207490037431129
Pred:[1 1 0 1 0 0 1 1]
True:[0 1 1 0 1 0 1 1]
43 + 64 = 211

iters:7100
Loss:0.8063586297379244
Pred:[0 0 0 0 0 1 0 1]
True:[0 0 1 1 0 0 1 1]
22 + 29 = 5

iters:7200
Loss:0.6516776732421684
Pred:[1 1 0 0 0 0 1 0]
True:[1 0 1 0 0 0 1 0]
50 + 112 = 194

iters:7300
Loss:0.8589452495598263
Pred:[0 1 0 1 0 0 1 1]
True:[0 0 1 1 1 0 1 1]
33 + 26 = 83

iters:7400
Loss:0.46020901403225034
Pred:[1 0 1 0 1 1 0 1]
True:[1 0 1 0 0 1 0 1]
39 + 126 = 173

iters:7500
Loss:0.32012358526530105
Pred:[1 0 1 1 0 0 0 1]
True:[1 0 1 0 0 0 0 1]
103 + 58 = 177

iters:7600
Loss:0.8139591009182066
Pred:[1 0 1 1 0 1 0 1]
True:[1 0 0 0 0 1 1 1]
76 + 59 = 181

iters:7700
Loss:0.7798169475366231
Pred:[1 1 1 0 1 1 0 1]
True:[1 1 0 0 1 1 0 1]
92 + 113 = 237

iters:7800
Loss:1.4633157630775486
Pred:[1 1 0 1 0 0 1 1]
True:[1 0 1 1 1 1 0 1]
98 + 91 = 211

iters:7900
Loss:0.37329555762045524
Pred:[1 0 1 0 0 1 0 0]
True:[1 0 1 1 0 1 0 0]
95 + 85 = 164

iters:8000
Loss:0.9607496657789788
Pred:[1 1 0 1 1 0 1 0]
True:[1 0 1 1 1 1 0 0]
113 + 75 = 218

iters:8100
Loss:0.5923169621482619
Pred:[0 1 1 1 1 1 0 1]
True:[0 1 1 1 1 1 0 1]
107 + 18 = 125

iters:8200
Loss:0.9200083546230319
Pred:[0 0 0 1 1 0 0 1]
True:[0 0 1 0 1 1 1 1]
36 + 11 = 25

iters:8300
Loss:0.3844879720872651
Pred:[1 0 0 1 0 0 1 0]
True:[1 0 1 1 0 1 1 0]
77 + 105 = 146

iters:8400
Loss:0.7907169496866977
Pred:[1 0 0 1 0 1 0 0]
True:[0 0 1 0 0 1 0 0]
15 + 21 = 148

iters:8500
Loss:1.1043327381236208
Pred:[0 1 1 0 1 0 0 1]
True:[1 0 0 0 1 1 0 1]
104 + 37 = 105

iters:8600
Loss:0.7738315762436639
Pred:[1 0 1 0 0 0 0 1]
True:[1 0 1 1 0 1 1 1]
58 + 125 = 161

iters:8700
Loss:0.7079459333273881
Pred:[1 1 0 1 1 0 1 1]
True:[1 0 1 1 0 0 1 1]
106 + 73 = 219

iters:8800
Loss:0.9342310062947881
Pred:[1 1 1 0 1 1 0 0]
True:[0 0 1 1 1 1 0 0]
8 + 52 = 236

iters:8900
Loss:0.6660399970508081
Pred:[1 0 1 0 0 1 0 0]
True:[1 0 1 1 0 0 0 0]
123 + 53 = 164

iters:9000
Loss:0.7183588462541307
Pred:[1 0 0 1 0 1 1 0]
True:[1 0 1 0 0 0 1 0]
37 + 125 = 150

iters:9100
Loss:0.9448149702383662
Pred:[0 1 1 0 1 0 1 0]
True:[0 1 1 1 1 1 0 0]
33 + 91 = 106

iters:9200
Loss:0.4548136987100642
Pred:[0 1 0 1 1 0 1 0]
True:[0 1 0 0 0 0 1 0]
41 + 25 = 90

iters:9300
Loss:0.7713648369805215
Pred:[0 0 0 0 0 0 0 1]
True:[0 1 1 1 1 0 0 1]
91 + 30 = 1

iters:9400
Loss:1.0107338513666393
Pred:[1 0 0 0 1 0 1 1]
True:[1 1 0 0 0 1 1 1]
115 + 84 = 139

iters:9500
Loss:0.6041084233126494
Pred:[1 0 1 0 1 0 0 0]
True:[1 0 1 0 0 0 1 0]
79 + 83 = 168

iters:9600
Loss:0.6905380622534354
Pred:[0 0 1 1 0 1 0 1]
True:[0 1 0 0 0 1 0 1]
29 + 40 = 53

iters:9700
Loss:0.7921526330004367
Pred:[1 0 1 1 0 1 0 1]
True:[0 0 1 1 1 0 0 1]
24 + 33 = 181

iters:9800
Loss:0.7119194556905244
Pred:[0 1 1 1 0 1 1 0]
True:[0 1 1 0 0 1 0 0]
85 + 15 = 118

iters:9900
Loss:0.8265829267977466
Pred:[1 1 1 1 0 0 1 0]
True:[1 0 1 1 1 0 1 0]
99 + 87 = 242

2. LSTM

確認テスト
・シグモイド関数を微分した時、入力値が0の時に最大値をとる。その値として正しいものを選択肢から選べ。
0.25

RNNの課題として、時系列を遡れば遡るほど勾配が消失するという問題があり、長い時系列の学習が困難である。
LSTMでは、過去の情報を忘れるか・忘れないかを判断しながら、必要な情報だけ次の時刻に引き継ぐことができる。

CEC
過去のデータを保存するためのユニット。
CECの課題
⇒入力データについて、時間依存度に関係なく重みが一律である。
⇒ニューラルネットワークの学習特性が無いということ。

入力・出力ゲート
入力・出力ゲートを追加することで、それぞれのゲートへの入力値の重みを、重み行列W,Uで可変可能とする。

忘却ゲート
CECの中身をどの程度残すかを調整する。

確認テスト
・以下の文章をLSTMに入力し空欄に当てはまる単語を予測したいとする。文中の「とても」という言葉は空欄の予測においてなくなっても影響を及ぼさないと考えられる。このような場合、どのゲートが作用すると考えられるか。
忘却ゲート

覗き穴結合
CEC自身の値に、重み行列を介して伝播可能にした構造。

3. GRU

従来のLSTMでは、パラメータが多数存在していたため、計算負荷が大きかった。しかし、GRUでは、そのパラメータを大幅に削減し、精度は同等またはそれ以上が望める様になった構造。メリットは、計算負荷が低いこと。
入力ゲートと忘却ゲートが統合され更新ゲート（Update gate）になっている。
記憶セルと出力ゲートがなく、値を0にリセットするリセットゲートがある。

確認テスト
・LSTMとCECが抱える課題について、それぞれ簡潔に述べよ。
LSTM：パラメータ数が多く、計算負荷が高い
CEC：ニューラルネットワークの学習特性がない

確認テスト
・LSTMとGRUの違いを簡潔に述べよ。
LSTMでは、パラメータが多数存在していたため、計算負荷が大きかったが、GRUでは、パラメータを大幅に削減し、精度は同等またはそれ以上が望める様になっており、計算負荷が低い。

4. 双方向RNN

過去の情報だけでなく、未来の情報を加味することで、精度を向上させるためのモデル
実用例：文章の推敲や、機械翻訳等

5. Seq2Seq

Encoder-Decoderモデルの一種

Encoder RNN
Taking：文章を単語等のトークンごとに分割し、さらにトークンごとのIDに分割する。
Embedding：IDからそのトークンを表す分散表現ベクトルに変換する。
Encoder RNN：ベクトルを順にRNNに入力していく。

Encoder RNN処理手順
vec1をRNNに入力し、hidden stateを出力。このhidden stateと次の入力vec2をまたRNNに入力してきたhidden stateを出力という流れを繰り返す。最後のvecを入れたときのhidden stateをfinal stateとしてとっておく。このfinal stateがthought vectorと呼ばれ、入力した文の意味を表すベクトルとなる。

Decoder RNN
システムがアウトプットデータを、単語等のトークンごとに生成する構造。

Decoder RNNの処理
Decoder RNN: Encoder RNNのfinal state (thought vector)から、各token の生成確率を出力していきます。final stateをDecoder RNNのinitial stateととして設定し、Embeddingを入力。
Sampling:生成確率にもとづいてtokenをランダムに選びます。
Embedding:選ばれたtokenをEmbeddingしてDecoder RNNへの次の入力とします。
Detokenize:以上を繰り返し、得られたtokenを文字列に直します。

確認テスト
・下記の選択肢から、seq2seqについて説明しているものを選べ。
（1）時刻に関して順方向と逆方向のRNNを構成し、それら2つの中間層表現を特徴量として利用するものである。（2）RNNを用いたEncoder-Decoderモデルの一種であり、機械翻訳などのモデルに使われる。
（3）構文木などの木構造に対して、隣接単語から表現ベクトル（フレーズ）を作るという演算を再帰的に行い（重みは共通）、文全体の表現ベクトルを得るニューラルネットワークである。
（4）RNNの一種であり、単純なRNNにおいて問題となる勾配消失問題をCECとゲートの概念を導入することで解決したものである。
→(2)

HRED
過去n-1個の発話から次の発話を生成する。
Seq2seqでは、会話の文脈無視で、応答がなされたが、HREDでは、前の単語の流れに即して応答されるため、より人間らしい文章が生成される。
HREDは確率的な多様性が字面にしかなく、会話の「流れ」のような多様性が無い。
→同じコンテキスト（発話リスト）を与えられても、答えの内容が毎回会話の流れとしては同じものしか出せない。
HRED は短く情報量に乏しい答えをしがちである。
→短いよくある答えを学ぶ傾向がある。

VHRED
HREDに、VAEの潜在変数の概念を追加したもの。

確認テスト
・Seq2SeqとHRED、HREDとVHREDの違いを簡潔に述べよ。
Seq2SeqとHRED：Seq2Seqは一問一答しかできないが、HREDは過去n−1個の発話から文脈に応じた回答ができる。
HREDとVHRED：HREDは発話に多様性がなく情報量に乏しいが、VHREDはそれらの課題を解決し、多様性ある発話ができる。

オートエンコーダ
教師なし学習の一つ。そのため学習時の入力データは訓練データのみで教師データは利用しない。
Encoder：入力データから潜在変数zzに変換するニューラルネットワーク
Decoder：潜在変数zから元画像を復元するニューラルネットワーク
出力した画像の潜在変数zが入力画像より小さくなっていれば次元削減とみなすことができる。

VAE
通常のオートエンコーダーの場合、何かしら潜在変数zにデータを押し込めているものの、その構造がどのような状態かわからない。VAEはこの潜在変数zに確率分布z∼N(0,1)を仮定したもの。

確認テスト
・VAEに関する下記の説明文中の空欄に当てはまる言葉を答えよ。
確率分布

6. Word2vec

大規模データの分散表現の学習が、現実的な計算速度とメモリ量で実現可能にした。
×：ボキャブラリ×ボキャブラリだけの重み行列が誕生。
○：ボキャブラリ×任意の単語ベクトル次元で重み行列が誕生。

分散表現では、CBOW（Continuous Bag-of-Words）もしくはSkip-gramというNNが用いられる。
CBOW：前後の単語から対象の単語を予測するNNで、学習に要する時間がSkip-gramより短くなる。
Skip-gram：ある単語から、前後の単語を予測するNN。CBOWよりも学習時間がかかるが、精度が良い。

7. Attention Mechanism

seq2seq の問題は長い文章への対応が難しい
→2単語でも、100単語でも、固定次元ベクトルの中に入力しなければならない。
解決策：文章が長くなるほどそのシーケンスの内部表現の次元も大きくなっていく仕組みが必要
→Attention Mechanism
「入力と出力のどの単語が関連しているのか」の関連度を学習する仕組み。

確認テスト
・RNNとword2vec、seq2seqとAttentionの違いを簡潔に述べよ。
RNNとWord2vec：RNNは時系列データを処理するのに適したNNであり、Word2vecは単語の分散表現ベクトルを得る手法
seq2seqとAttention：Seq2Seqは一つの時系列データから別の時系列データを得るネットワークであり、Attentionは時系列の中身に対して、関連性に重みをつける手法