動作環境

GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 14.04 LTS desktop amd64
TensorFlow v0.11
cuDNN v5.1 for Linux
CUDA v8.0
Python 2.7.6
IPython 5.1.0 -- An enhanced Interactive Python.

v0.1 http://qiita.com/7of9/items/b364d897b95476a30754

http://qiita.com/7of9/items/ce72fea33fc0a2dea479
においてsine curveの学習が進まない。

なぜなのか。

tf.train.shuffle_batch() + eval()の効果を確認することにした。

失敗編

シーケンスデータの作成

shuffleした時の効果が分かるように、入力するcsvファイルを作ることにした。　

1列目: 1から100
2列目: 501から600

これで、１列目と２列目の対応が分かる。

python prep_seq_data.py > input2.csv
を実行する。

Jupyterで確認。

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

data1 = np.loadtxt('input2.csv', delimiter=',')

input1 = data1[:,0]
output1 = data1[:,1]

fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)

#ax1.plot(input1, output1, color='black', linestyle='solid', label='original')
ax1.scatter(input1, output1)

ax1.set_title('loss')
ax1.set_xlabel('step')
ax1.set_ylabel('loss')
ax1.grid(True)
ax1.legend()

fig.show()

tf.train.shuffle_batch() + eval()の効果

test_batch.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import tensorflow as tf
import tensorflow.contrib.slim as slim

filename_queue = tf.train.string_input_producer(["input2.csv"])
reader = tf.TextLineReader()
key, value = reader.read(filename_queue)
input1, output = tf.decode_csv(value, record_defaults=[[0.], [0.]])
inputs = tf.pack([input1])
output = tf.pack([output])

batch_size=4 # [4]
inputs_batch, output_batch = tf.train.shuffle_batch([inputs, output], batch_size, capacity=40, min_after_dequeue=batch_size)

input_ph = tf.placeholder("float", [None,1])
output_ph = tf.placeholder("float",[None,1])

## NN のグラフ生成
hiddens = slim.stack(input_ph, slim.fully_connected, [1,7,7,7], 
  activation_fn=tf.nn.sigmoid, scope="hidden")
prediction = slim.fully_connected(hiddens, 1, activation_fn=tf.nn.sigmoid, scope="output")
loss = tf.contrib.losses.mean_squared_error(prediction, output_ph)

#train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.01))
train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.001))

#def feed_dict(inputs, output):
#    return {input_ph: inputs.eval(), output_ph: output.eval()}

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
  coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(sess=sess, coord=coord)

  try:
    sess.run(init_op)
    for i in range(3): #[30000]
    	print(inputs_batch.eval(), output_batch.eval())

#      _, t_loss = sess.run([train_op, loss], feed_dict={input_ph:inputs_batch.eval(), output_ph:output_batch.eval()})
#      if (i+1) % 100 == 0:
#        print("%d,%f" % (i+1, t_loss))
#        print("%d,%f,#step, loss" % (i+1, t_loss))
  finally:
    coord.request_stop()

  coord.join(threads)

結果

$ python test_batch.py 
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
WARNING:tensorflow:sum_of_squares (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-10-01.
Instructions for updating:
Use mean_squared_error.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: 
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.7715
pciBusID 0000:01:00.0
Total memory: 7.91GiB
Free memory: 7.24GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
(array([[ 27.],
       [ 13.],
       [ 21.],
       [ 40.]], dtype=float32), array([[ 541.],
       [ 531.],
       [ 520.],
       [ 506.]], dtype=float32))
(array([[ 14.],
       [ 44.],
       [ 34.],
       [  4.]], dtype=float32), array([[ 537.],
       [ 519.],
       [ 515.],
       [ 535.]], dtype=float32))
(array([[ 38.],
       [ 11.],
       [ 10.],
       [ 23.]], dtype=float32), array([[ 542.],
       [ 503.],
       [ 546.],
       [ 530.]], dtype=float32))

一列目 {27, 13, 21, 40}に対して、二列目{541, 531, 520, 506}になっている。

つまり、inputs_batch.eval()をした時のインデックスとoutput_batch().eval()した時のインデックスが一致していない。

これではsine curveが学習できるわけがない。

.eval()の使い方を学習しないといけない。

evalの代わり

参考 http://stackoverflow.com/questions/34010987/does-tensorflow-rerun-for-each-eval-call

.eval()を使わず、sess.run()を使う。

#       print(inputs_batch.eval(), output_batch.eval())
      inp, out = sess.run([inputs_batch, output_batch])
      print (inp, out)

test_batch2.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import tensorflow as tf
import tensorflow.contrib.slim as slim

filename_queue = tf.train.string_input_producer(["input2.csv"])
reader = tf.TextLineReader()
key, value = reader.read(filename_queue)
input1, output = tf.decode_csv(value, record_defaults=[[0.], [0.]])
inputs = tf.pack([input1])
output = tf.pack([output])

batch_size=4 # [4]
inputs_batch, output_batch = tf.train.shuffle_batch([inputs, output], batch_size, capacity=40, min_after_dequeue=batch_size)

input_ph = tf.placeholder("float", [None,1])
output_ph = tf.placeholder("float",[None,1])

## NN のグラフ生成
hiddens = slim.stack(input_ph, slim.fully_connected, [1,7,7,7], 
  activation_fn=tf.nn.sigmoid, scope="hidden")
prediction = slim.fully_connected(hiddens, 1, activation_fn=tf.nn.sigmoid, scope="output")
loss = tf.contrib.losses.mean_squared_error(prediction, output_ph)

#train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.01))
train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.001))

#def feed_dict(inputs, output):
#    return {input_ph: inputs.eval(), output_ph: output.eval()}

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
  coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(sess=sess, coord=coord)

  try:
    sess.run(init_op)
    for i in range(3): #[30000]
#    	print(inputs_batch.eval(), output_batch.eval())
  
      inp, out = sess.run([inputs_batch, output_batch])
      print (inp, out)

#      _, t_loss = sess.run([train_op, loss], feed_dict={input_ph:inputs_batch.eval(), output_ph:output_batch.eval()})
#      if (i+1) % 100 == 0:
#        print("%d,%f" % (i+1, t_loss))
#        print("%d,%f,#step, loss" % (i+1, t_loss))
  finally:
    coord.request_stop()

  coord.join(threads)

結果　

$ python test_batch2.py 
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
WARNING:tensorflow:sum_of_squares (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-10-01.
Instructions for updating:
Use mean_squared_error.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: 
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.7715
pciBusID 0000:01:00.0
Total memory: 7.91GiB
Free memory: 7.18GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
(array([[ 36.],
       [ 29.],
       [ 11.],
       [ 37.]], dtype=float32), array([[ 536.],
       [ 529.],
       [ 511.],
       [ 537.]], dtype=float32))
(array([[ 43.],
       [ 31.],
       [ 24.],
       [ 34.]], dtype=float32), array([[ 543.],
       [ 531.],
       [ 524.],
       [ 534.]], dtype=float32))
(array([[  1.],
       [ 21.],
       [ 41.],
       [ 45.]], dtype=float32), array([[ 501.],
       [ 521.],
       [ 541.],
       [ 545.]], dtype=float32))

一列目 {36, 29, 11, 37}に対して、二列目{536, 529, 511, 537}になっている。

成功。