6【log】Logarithmic Functionc by Machine Learning

深層学習の形式で対数関数のパーセプトロンを考えてみました。

I explored a perceptron using logarithmic functions in a deep learning framework.

# coding=utf-8
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from datetime import datetime
import time
import winsound

# Start
start = time.time()

# Initial values
# Input layer dimensions
x_num = 2
# Output layer dimensions
y_num = 1
# Layer sizes
layer = [x_num, 2, y_num]
# Number of hidden layers
H = len(layer) - 2
# Batch size
batch_size = 100
# Number of training iterations
N = 100000

w = [None for _ in range(H + 1)]
hidden = [None for _ in range(H)]

# Random seeds
np.random.seed(int(datetime.now().strftime('%H%M%S')))
tf.set_random_seed(int(datetime.now().strftime('%H%M%S')))

# Input layer
x = tf.placeholder(tf.complex64, [None, x_num])
# First layer
w[0] = tf.Variable(tf.truncated_normal([x_num, 2]), tf.float64)
hidden[0] = tf.log(tf.log(tf.matmul(x, tf.cast(w[0], tf.complex64))))
# Output layer
w[1] = tf.Variable(tf.zeros([2, y_num]), tf.float64)
p = tf.exp(tf.matmul(hidden[0], tf.cast(w[1], tf.complex64)))

# Backpropagation
t = tf.placeholder(tf.complex64, [None, y_num])
loss = tf.reduce_sum(tf.square(tf.cast(p - t, tf.float64)))
#-tf.reduce_sum(t * tf.log(p))#tf.reduce_sum(tf.square(p - t), name='loss')
train_step = tf.train.AdamOptimizer().minimize(loss)
#train_step = tf.train.GradientDescentOptimizer(0.0001).minimize(loss)

# Initialization
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# Weights
w_save = [None for _ in range(N + 1)]
w_save[0] = sess.run(w)
print(w_save[0])

# Training
for n in range(N):
    train_x = np.random.uniform(np.e, 10.0, (batch_size, x_num))
    train_x = np.array(train_x, dtype=np.complex)
    train_t = np.log(train_x[:, 1]) / np.log(train_x[:, 0])
    train_t = train_t.reshape(batch_size, 1)
    #train_t = np.array(train_t, dtype=np.float64)
    sess.run(train_step, feed_dict={x:train_x, t:train_t})

    w_save[n + 1] = sess.run(w)

# Output
# Values
print(w_save[N])
# Plot
# Number of subplot rows
py = np.amax(layer)
# Number of subplot columns
px = (H + 1) * 2
# Figure size
plt.figure(figsize = (16, 9))
# Horizontal axis values
x = np.arange(0, N + 1, 1) # From 0 to N in steps of 1
# Draw plots
for h in range(H + 1):
    for l in range(layer[h + 1]):
        # Subplot position
        plt.subplot(py, px, px * l + h * 2 + 1)
        for m in range(layer[h]):
            # Line (l and m are transposed)
            plt.plot(x, np.array([w_save[n][h][m][l] for n in range(N + 1)]), label = "w[" + str(h) + "][" + str(l) + "," + str(m) + "]")
        # Grid lines
        plt.grid(True)
        # Legend
        plt.legend(bbox_to_anchor = (1, 1), loc = 'upper left', borderaxespad = 0, fontsize = 10)

# Save the figure
plt.savefig('graph_log_tf.png')
# Display the figure
plt.show()
# End ######################################################################
print (time.time() - start)
print(datetime.now().strftime('%Y%m%d%H%M%S'))
winsound.Beep(500,500)
############################################################################

重み

Weights

w[0]=
\begin{pmatrix}
△ & □\\
▲ & ■
\end{pmatrix},
w[1]=
\begin{pmatrix}
〇\\
●
\end{pmatrix}\\

入力値とw[0]の積

Product of the input and w[0]

\begin{pmatrix}
a & b
\end{pmatrix}
\begin{pmatrix}
△ & ▲\\
□ & ■
\end{pmatrix}
=
\begin{pmatrix}
△a+□b & ▲a+■b
\end{pmatrix}

第1層入力

First layer input

\begin{pmatrix}
log(log(△a+□b) & log(log(▲a+■b)
\end{pmatrix}

第1層出力とw[1]の積

Product of the first layer output and w[1]

\begin{align}
&\begin{pmatrix}
log(log(△a+□b)) & log(log(▲a+■b))
\end{pmatrix}
\begin{pmatrix}
〇\\
●
\end{pmatrix}\\
　\\

=&〇log(log(△a+□b))+●log(log(▲a+■b))\\
=&log(log(△a+□b))^{〇}-log(log(▲a+■b))^{-●}\\
=&log\frac{(log(△a+□b))^{〇}}{(log(▲a+■b))^{-●}}
\end{align}

出力層入力

Output layer input

\begin{pmatrix}
e^{log\frac{(log(△a+□b))^{〇}}{(log(▲a+■b))^{-●}}}
\end{pmatrix}
=\frac{(log(△a+□b))^{〇}}{(log(▲a+■b))^{-●}}
　\\
\left\{
\begin{array}{l}
△=0,□=1,〇=1 \\
▲=1,■=0,●=-1
\end{array}
\right.
\\
=\frac{logb}{loga}\\
=log_ab

最も簡単な場合、上記条件を満たせば$\log_a b$を出力することができます。

In the simplest case, the network can output $\log_a b$ if the conditions above are satisfied.

初期値を乱数で決めてから学習を繰り返すと目標値に収束するか試してみました。

I tested whether repeated training would converge to the target values after random initialization.

目標値

Target values

w[0]=
\begin{pmatrix}
1 & 0\\
0 & 1
\end{pmatrix}
,w[1]=
\begin{pmatrix}
1\\
-1
\end{pmatrix}\\

初期値

Initial values

w[0]=
\begin{pmatrix}
1.655222 & -0.03430884\\
0.36429077 & 0.75053316
\end{pmatrix}
,w[1]=
\begin{pmatrix}
0.0\\
0.0
\end{pmatrix}\\

計算値

Computed values

w[0]=
\begin{pmatrix}
1.0000112e+00 & -4.5090604e-07\\
1.8058329e-07 & 1.0000116e+00
\end{pmatrix}
,w[1]=
\begin{pmatrix}
-1.0000067\\
1.0000072
\end{pmatrix}

自作で組んだpythonのコードではオーバーフローばかり起してうまくいきませんでしたが

My own Python implementation kept running into overflow errors and did not work, but

tensorflowを導入したら、あっさり成功しました。

once I switched to TensorFlow, it worked with little difficulty.