KLダイバージェンスの式と計算例

KLダイバージェンスの式

連続型確率分布

\begin{align}
\mathrm{KL}(p||q) = \int p(x) \log{\frac{p(x)}{q(x)}} dx = \int \left[ p(x) \log{p(x)} - p(x) \log{q(x)} \right] dx
\end{align}

離散型確率分布

\begin{align}
\mathrm{KL}(p||q) = \sum_{i=1}^{n} p(x_{i}) \log{\frac{p(x_{i})}{q(x_{i})}} = \sum_{i=1}^{n} \left[ p(x_{i}) \log{p(x_{i})} - p(x_{i}) \log{q(x_{i})} \right]
\end{align}

KLダイバージェンスの計算例

連続型確率分布

離散型確率分布

import numpy as np
import matplotlib.pyplot as plt


x = np.array([1, 2])

p = np.array([0.8, 0.2])
q_1 = np.array([0.75, 0.25])
q_2 = np.array([0.5, 0.5])

plt.plot(x, p, label="GroundTruth")
plt.plot(x, q_1, label="Prediction1")
plt.plot(x, q_2, label="Prediction2")

plt.legend()
plt.savefig("distribution.png")

KL1 = np.sum(p*np.log(p/q_1))
KL2 = np.sum(p*np.log(p/q_2))

print("KL of prediction1: {:.5f}".format(KL1))
print("KL of prediction2: {:.5f}".format(KL2))

・実行結果

KL of prediction1: 0.00700
KL of prediction2: 0.19274