Deeplearning4jで手書き文字認識の続きです。
去年の記事ですが、【ディープラーニング】ChainerでAutoencoderを試して結果を可視化してみる。がとても面白かったので、同じことをDeeplearning4jでやってみました。
環境
前の記事と同じ。
実装
ソースコード
まずはネットワークを学習する方。完全なソースコードはGithubを参照(中間層のノード数が異なりますが)。
AutoEncoderCreator.java
public static void main(String[] args) throws Exception {
Logger log = LoggerFactory.getLogger(AutoEncoderCreator.class);
final int numRows = 28;
final int numColumns = 28;
final int hiddenNodesNum = 100;
int numSamples = 200; //60000;
int batchSize = 50;
int iterations = 100;
int seed = 123;
int listenerFreq = batchSize / 10;
String binFile = "logs/autoencoder.bin";
String confFile = "logs/autoencoder.json";
log.info("Load data....");
DataSetIterator iter = new MnistDataSetIterator(batchSize, numSamples, true);
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed)
.iterations(iterations)
.list(2)
// // encoding
.layer(0, new DenseLayer.Builder()
.nIn(numRows * numColumns)
.nOut(hiddenNodesNum)
.activation("relu")
.dropOut(0.3)
.build())
// decoding
.layer(1, new OutputLayer.Builder()
.nIn(hiddenNodesNum)
.nOut(numRows * numColumns)
.activation("relu")
.dropOut(0.3)
.lossFunction(LossFunctions.LossFunction.MSE)
.build())
.pretrain(true)
.backprop(true)
.build();
MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();
model.setListeners(new ScoreIterationListener(listenerFreq));
// model.setListeners(new HistogramIterationListener(listenerFreq));
log.info("Train model....");
while (iter.hasNext()) {
DataSet next = iter.next();
model.fit(new DataSet(next.getFeatureMatrix(), next.getFeatureMatrix()));
}
log.info("Save model....");
try (OutputStream fos = new FileOutputStream(binFile);
DataOutputStream dos = new DataOutputStream(fos)) {
Nd4j.write(model.params(), dos);
}
FileUtils.writeStringToFile(new File(confFile), model.getLayerWiseConfigurations().toJson());
log.info("Finish!");
// AutoEncoderReader.main(new String[0]);
}
データは前回と同じMnistです。計算時間の関係上、200サンプルのみを学習に使っています。
入力層(784) --> 中間層(100) --> 出力層(748)
という構成で、答えに入力と同じ値を設定しています。
なお、pretrain(true)を設定すると、バックプロパゲーションの前に以下の処理を行います。(Github上のソースコードとしてはここ)
- RBM(制限付きボルツマンマシン)のcontrastive divergence
- OutputLayerに対して確率的勾配法(SGD)
もしかして純粋なAutoEncoderじゃないのかもしれませんが、pretrain(false)だと思ったような結果にならなかった。。。
なお、今回は最適化アルゴリズムなどはデフォルトのまま変えていませんが、DL4Jとしては他にも色々サポートされているようです(DL4J Featuresを参照)。
そして、読み込む側はこちら。完全なソースコードはGithubを参照。
AutoEncoderReader.java
public static void main(String[] args) throws Exception {
String confFile = "logs/autoencoder.json";
String binFile = "logs/autoencoder.bin";
Logger log = LoggerFactory.getLogger(AutoEncoderReader.class);
log.info("Load stored model ...");
MultiLayerConfiguration confFromJson = MultiLayerConfiguration.fromJson(FileUtils.readFileToString(new File(confFile)));
DataInputStream dis = new DataInputStream(new FileInputStream(new File(binFile)));
INDArray newParams = Nd4j.read(dis);
dis.close();
MultiLayerNetwork model = new MultiLayerNetwork(confFromJson);
model.init();
model.setParams(newParams);
System.out.println(model);
log.info("Decode ...");
List<INDArray> originalArrays = new ArrayList<>();
List<INDArray> decodedArrays = new ArrayList<>();
MnistDataSetIterator testIter = new MnistDataSetIterator(100, 500);
while (testIter.hasNext()) {
DataSet testMnist = testIter.next();
INDArray feature = testMnist.getFeatureMatrix();
INDArray decoded = model.output(feature, Layer.TrainingMode.TEST);
for (int rowIndex=0 ; rowIndex<feature.rows() ; rowIndex++) {
originalArrays.add(feature.getRow(rowIndex));
decodedArrays.add(decoded.getRow(rowIndex));
}
}
log.info("Visualize ...");
MnistViewer.visualize("Original", originalArrays, 2.0);
MnistImgWriter.write("logs/original.png", originalArrays, 2.0);
MnistViewer.visualize("Decoded", decodedArrays, 2.0);
MnistImgWriter.write("logs/decoded.png", decodedArrays, 2.0);
try {
INDArray paramW1 = model.getLayer(0).getParam("W"); // rows=784, columns=1000
List<INDArray> params = new ArrayList<>();
for (int columnIndex=0 ; columnIndex<paramW1.columns() ; columnIndex++) {
params.add(paramW1.getColumn(columnIndex));
}
List<INDArray> normalized = normalizedParamW(params);
MnistViewer.visualize("Params of layer 0" , normalized, 2.0);
MnistImgWriter.write("logs/param0.png", normalized, 2.0);
} catch (Exception e) {
e.printStackTrace();
}
try {
INDArray paramW2 = model.getLayer(1).getParam("W").transposei(); // rows=784, columns=1000
List<INDArray> params = new ArrayList<>();
for (int columnIndex=0 ; columnIndex<paramW2.columns() ; columnIndex++) {
params.add(paramW2.getColumn(columnIndex));
}
List<INDArray> normalized = normalizedParamW(params);
MnistViewer.visualize("Params of layer 1" , normalized, 2.0);
MnistImgWriter.write("logs/param1.png", normalized, 2.0);
} catch (Exception e) {
e.printStackTrace();
}
}
AutoEncoderCreatorで作成したautoencoder.jsonとautoencoder.binを読み込んで、計算結果や重み行列を画像化しています。
ビルドスクリプト
前の記事と同じ。
実行結果
コマンドライン
>set JAVA_HOME="c:\Program Files\Java\jdk1.8.0_45"
>gradle run -Pmain=jp.hashiwa.dl4j.sample.AutoEncoderCreator
:compileJava
:processResources
:classes
:pathingJar
:run
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) Server VM (build 25.45-b02, mixed mode)
15:42:24.144 INFO j.h.d.s.AutoEncoderCreator Load data....
--- snip ---
15:42:26.442 INFO j.h.d.s.AutoEncoderCreator Train model....
15:42:26.496 INFO o.d.n.m.MultiLayerNetwork Training on layer 1 with 50 examples
15:42:26.496 INFO o.d.n.m.MultiLayerNetwork Finetune phase
15:42:26.563 WARN o.d.o.s.BaseOptimizer Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.
15:42:27.001 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 51.221025390625
15:42:27.438 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 28.21199462890625
15:42:27.813 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 22.1663916015625
15:42:28.155 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 18.75435791015625
15:42:28.483 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 16.511124267578126
15:42:28.800 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 14.880093994140625
15:42:29.162 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 13.637303466796874
15:42:29.465 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 12.654112548828126
15:42:29.765 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 11.8371533203125
15:42:30.089 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 11.169140625
15:42:30.373 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 10.600634765625
15:42:30.658 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 10.11435302734375
15:42:31.019 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 9.691630859375
15:42:31.348 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 9.318866577148437
15:42:31.632 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 8.98861083984375
15:42:31.911 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 8.694522094726562
15:42:32.261 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 8.429197387695313
15:42:32.560 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 8.189866943359375
15:42:32.916 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 7.973997802734375
15:42:33.204 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 7.77048583984375
15:42:33.463 WARN o.d.o.s.BaseOptimizer Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.
15:42:34.077 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 45285.7
15:42:34.159 WARN o.d.o.s.BackTrackLineSearch Value is infinite after jump. oldStep=1.0. score=Infinity, score2=45285.7. Scaling back step size...
15:42:34.219 WARN o.d.o.s.BackTrackLineSearch Value is infinite after jump. oldStep=0.2. score=8.17222308817912E28, score2=Infinity. Scaling back step size...
15:42:35.258 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 36.547490234375
15:42:36.560 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 36.1521337890625
15:42:38.120 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 36.12509521484375
15:42:39.490 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 35.94593505859375
15:42:40.597 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 36.380234375
15:42:41.765 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 36.091943359375
15:42:43.018 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 35.09519775390625
15:42:44.504 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 35.52658935546875
15:42:45.411 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 35.97473388671875
15:42:46.742 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 35.5510546875
15:42:48.018 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 35.27791259765625
15:42:49.241 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 35.2272119140625
15:42:50.566 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 35.00723876953125
15:42:51.623 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 35.12083740234375
15:42:52.698 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 34.8168212890625
15:42:54.434 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 34.8677197265625
15:42:55.599 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 34.9637646484375
15:42:57.014 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 34.9985693359375
15:42:58.484 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 35.04102783203125
15:42:59.414 INFO o.d.n.m.MultiLayerNetwork Training on layer 1 with 50 examples
15:42:59.415 INFO o.d.n.m.MultiLayerNetwork Finetune phase
15:42:59.525 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 72.3163720703125
15:42:59.808 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 53.254521484375
15:43:00.103 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 50.3268017578125
15:43:00.372 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 46.2170654296875
15:43:00.683 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 44.2616015625
15:43:00.975 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 43.711962890625
15:43:01.346 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 43.3212255859375
15:43:01.676 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 43.13658203125
15:43:02.132 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 43.058974609375
15:43:02.424 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 42.944951171875
15:43:02.800 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 42.8766845703125
15:43:03.186 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 42.7393359375
15:43:03.505 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 42.6743701171875
15:43:03.837 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 42.635146484375
15:43:04.143 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 42.611337890625
15:43:04.460 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 42.5852685546875
15:43:04.838 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 42.581962890625
15:43:05.211 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 42.5786376953125
15:43:05.489 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 42.570869140625
15:43:05.846 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 42.54314453125
15:43:06.657 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 175042.7
15:43:06.742 WARN o.d.o.s.BackTrackLineSearch Value is infinite after jump. oldStep=1.0. score=Infinity, score2=175042.7. Scaling back step size...
15:43:06.795 WARN o.d.o.s.BackTrackLineSearch Value is infinite after jump. oldStep=0.2. score=2.5810312997188796E33, score2=Infinity. Scaling back step size...
15:43:08.097 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 38.37738525390625
15:43:09.363 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 37.90353759765625
15:43:10.511 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 37.8761767578125
15:43:11.360 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 37.32078369140625
15:43:12.575 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 37.73943115234375
15:43:13.825 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 37.37815673828125
15:43:15.037 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 37.28591552734375
15:43:16.562 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 37.2585595703125
15:43:17.532 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 37.70104248046875
15:43:18.475 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 37.3643505859375
15:43:19.750 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 37.311806640625
15:43:20.953 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 37.43316650390625
15:43:22.187 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 37.20260986328125
15:43:23.141 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 37.48757568359375
15:43:24.391 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 37.1484423828125
15:43:25.440 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 37.3223828125
15:43:26.491 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 37.2163330078125
15:43:27.490 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 37.1728466796875
15:43:28.563 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 37.34130859375
15:43:29.433 INFO o.d.n.m.MultiLayerNetwork Training on layer 1 with 50 examples
15:43:29.434 INFO o.d.n.m.MultiLayerNetwork Finetune phase
15:43:29.543 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 70.12888671875
15:43:29.805 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 64.3011767578125
15:43:30.126 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 64.2672021484375
15:43:30.482 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 64.253359375
15:43:30.828 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 64.2510302734375
15:43:31.192 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 64.2503369140625
15:43:31.556 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 64.250029296875
15:43:32.038 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 64.25
15:43:32.663 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 64.25
15:43:33.227 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 64.25
15:43:33.814 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 64.25
15:43:34.369 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 64.25
15:43:34.954 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 64.25
15:43:35.549 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 64.25
15:43:36.098 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 64.25
15:43:36.698 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 64.25
15:43:37.274 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 64.25
15:43:37.873 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 64.25
15:43:38.467 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 64.25
15:43:39.050 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 64.25
15:43:40.071 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 193770.7
15:43:41.129 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 37.4401220703125
15:43:42.350 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 37.47879150390625
15:43:43.499 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 37.22159423828125
15:43:44.872 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 37.0895556640625
15:43:45.851 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 37.11908203125
15:43:46.849 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 37.032529296875
15:43:48.062 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 37.1989013671875
15:43:49.139 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 37.114775390625
15:43:50.359 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 37.090537109375
15:43:51.439 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 37.13592041015625
15:43:52.415 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 37.26075927734375
15:43:53.648 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 37.06206787109375
15:43:54.864 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 37.16815673828125
15:43:55.981 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 37.215966796875
15:43:57.025 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 37.068134765625
15:43:58.328 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 37.2640478515625
15:43:59.351 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 37.21443603515625
15:44:00.341 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 37.3040576171875
15:44:01.177 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 37.34953125
15:44:01.983 INFO o.d.n.m.MultiLayerNetwork Training on layer 1 with 50 examples
15:44:01.984 INFO o.d.n.m.MultiLayerNetwork Finetune phase
15:44:02.089 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 66.58548828125
15:44:02.461 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 66.228798828125
15:44:02.796 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 66.2111474609375
15:44:03.176 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 66.2100830078125
15:44:03.577 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 66.21
15:44:04.098 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 66.21
15:44:04.602 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 66.21
15:44:05.138 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 66.21
15:44:05.646 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 66.21
15:44:06.150 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 66.21
15:44:06.659 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 66.21
15:44:07.162 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 66.21
15:44:07.659 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 66.21
15:44:08.189 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 66.21
15:44:08.678 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 66.21
15:44:09.174 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 66.21
15:44:09.743 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 66.21
15:44:10.237 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 66.21
15:44:10.741 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 66.21
15:44:11.217 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 66.21
15:44:12.202 INFO o.d.o.l.ScoreIterationListener Score at iteration 0 is 38.26293701171875
15:44:13.420 INFO o.d.o.l.ScoreIterationListener Score at iteration 5 is 36.89760009765625
15:44:14.641 INFO o.d.o.l.ScoreIterationListener Score at iteration 10 is 36.9484423828125
15:44:15.805 INFO o.d.o.l.ScoreIterationListener Score at iteration 15 is 37.3839794921875
15:44:16.685 INFO o.d.o.l.ScoreIterationListener Score at iteration 20 is 36.83066650390625
15:44:18.262 INFO o.d.o.l.ScoreIterationListener Score at iteration 25 is 36.8345751953125
15:44:19.094 INFO o.d.o.l.ScoreIterationListener Score at iteration 30 is 36.43686279296875
15:44:20.732 INFO o.d.o.l.ScoreIterationListener Score at iteration 35 is 36.75513427734375
15:44:22.036 INFO o.d.o.l.ScoreIterationListener Score at iteration 40 is 37.11474609375
15:44:22.936 INFO o.d.o.l.ScoreIterationListener Score at iteration 45 is 36.8320458984375
15:44:23.803 INFO o.d.o.l.ScoreIterationListener Score at iteration 50 is 36.81147216796875
15:44:24.946 INFO o.d.o.l.ScoreIterationListener Score at iteration 55 is 36.8683740234375
15:44:25.900 INFO o.d.o.l.ScoreIterationListener Score at iteration 60 is 36.9046728515625
15:44:26.938 INFO o.d.o.l.ScoreIterationListener Score at iteration 65 is 36.63814697265625
15:44:28.155 INFO o.d.o.l.ScoreIterationListener Score at iteration 70 is 37.0087744140625
15:44:29.273 INFO o.d.o.l.ScoreIterationListener Score at iteration 75 is 36.64892822265625
15:44:30.441 INFO o.d.o.l.ScoreIterationListener Score at iteration 80 is 36.875947265625
15:44:31.542 INFO o.d.o.l.ScoreIterationListener Score at iteration 85 is 36.78516357421875
15:44:32.688 INFO o.d.o.l.ScoreIterationListener Score at iteration 90 is 36.746904296875
15:44:33.958 INFO o.d.o.l.ScoreIterationListener Score at iteration 95 is 36.59603515625
15:44:34.876 INFO j.h.d.s.AutoEncoderCreator Save model....
15:44:37.850 INFO j.h.d.s.AutoEncoderCreator Finish!
BUILD SUCCESSFUL
Total time: 2 mins 26.871 secs
2分半くらいで学習が終わりました。
画像化結果
学習したMnistのデータとデコード後(Mnistのデータを入力とした出力結果)はこちら。
| Mnistのデータ | デコード後 |
|---|---|
 |
 |
ぐちゃぐちゃで何がなんやらといったところですが、層と層の間の重み行列を正規化して画像化してみたのがこちら。
| w0 | w1の転置行列 |
|---|---|
 |
 |
w0は入力層と中間層の間の重み行列(784×100)、w1は中間層と出力層の間の重み行列(100×784)です。特にw1の転置行列の方は文字の痕跡っぽいのが見受けられます(出すぎているものもありますが)。
果たしてこれがAutoEncoderとして正しく実装されているのか怪しいですが、個人的には達成感を感じられたので、とりあえずここまで。
なお、DL4J公式ドキュメントにはDeep AutoEncoderの実装例が載ってました(ここ)。機会があればこちらも実装していじってみたいです。