TensorFlow > TFRecords > tf_record_iterator()での読込みを進めるに連れ、読み込み時間が増加するようだ

動作環境

GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 16.04 LTS desktop amd64
TensorFlow v1.1.0
cuDNN v5.1 for Linux
CUDA v8.0
Python 3.5.2
IPython 6.0.0 -- An enhanced Interactive Python.
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

record_iterator = tf.python_io.tf_record_iterator(path=INP_FILE)
...
for record in record_iterator:
    example = tf.train.Example()
    example.ParseFromString(record)

上記でのファイル読み込みを4,113,648レコードのファイルに対して行おうとした。
処理が終わらない。

レコード読込み時間が一定ではないようだ。

calc_mean_std_170819.py

import numpy as np
import tensorflow as tf
import sys
import time

"""
v0.1 Aug. 19, 2017
  - add time profiling
  - change [INP_FILE] to those with _170819 prefix

=== branched from [calc_mean_std_170812.py] ===

v0.1 Aug. 12, 2017
  - calculate [mean], [stddev]

=== branched from [test_readCombined_170722.py] ===
v0.2 Jul. 09, 2017
  - read [mr] and [mi]
v0.1 Jul. 09, 2017
  - read position and Ex, Ey, Ez
     + add get_feature_float32()
"""

# on
#   Ubuntu 16.04 LTS
#   TensorFlow v1.1
#   Python 3.5.2

# codingrule: PEP8


def print_mean_stddev(xs, label):
    print('%s mean:%f std:%f' % (label, xs.mean(), xs.std()))


def get_feature_float32(example, feature_name):
    wrk_raw = (example.features.feature[feature_name]
               .bytes_list
               .value[0])
    wrk_1d = np.fromstring(wrk_raw, dtype=np.float32)
    wrk_org = wrk_1d.reshape([1, -1])
    return wrk_org

INP_FILE = 'combined_IntField-Y_170819.tfrecords'

record_iterator = tf.python_io.tf_record_iterator(path=INP_FILE)

exrs, exis = np.array([]), np.array([])
eyrs, eyis = np.array([]), np.array([])
ezrs, ezis = np.array([]), np.array([])

cnt = 0
start = time.time()
for record in record_iterator:
    if cnt % 10000 == 0:
        print("%d, %.3f" % (cnt, time.time() - start))
        start = time.time()
    example = tf.train.Example()
    example.ParseFromString(record)

    xpos_org = get_feature_float32(example, 'xpos_raw')
    ypos_org = get_feature_float32(example, 'ypos_raw')
    zpos_org = get_feature_float32(example, 'zpos_raw')
    mr_org = get_feature_float32(example, 'mr_raw')
    mi_org = get_feature_float32(example, 'mi_raw')
    exr_org = get_feature_float32(example, 'exr_raw')
    exi_org = get_feature_float32(example, 'exi_raw')
    eyr_org = get_feature_float32(example, 'eyr_raw')
    eyi_org = get_feature_float32(example, 'eyi_raw')
    ezr_org = get_feature_float32(example, 'ezr_raw')
    ezi_org = get_feature_float32(example, 'ezi_raw')

    exrs, exis = np.append(exrs, exr_org), np.append(exis, exi_org)
    eyrs, eyis = np.append(eyrs, eyr_org), np.append(eyis, eyi_org)
    ezrs, ezis = np.append(ezrs, ezr_org), np.append(ezis, ezi_org)

    cnt += 1

print_mean_stddev(exrs, 'exr')
print_mean_stddev(exis, 'exi')

print_mean_stddev(eyrs, 'eyr')
print_mean_stddev(eyis, 'eyi')

print_mean_stddev(ezrs, 'ezr')
print_mean_stddev(ezis, 'ezi')

run

$ python3 calc_mean_std_170819.py 
0, 0.000
10000, 0.843
20000, 0.999
30000, 1.195
40000, 1.410
50000, 1.636
60000, 1.916
70000, 2.128
80000, 2.328
90000, 2.835
100000, 4.844
110000, 5.850
120000, 6.591
130000, 7.376
140000, 8.045
150000, 8.955
160000, 9.681
170000, 10.421
180000, 11.165
190000, 11.878
200000, 12.575
210000, 13.276
220000, 13.959
230000, 14.734
240000, 15.446

右側の数値が一定になることを期待していたが、増加している。
別の処理方法を検討しないと4,113,648レコードのファイルを処理できそうにない。

関連