動作環境

GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 16.04 LTS desktop amd64
TensorFlow v1.1.0
cuDNN v5.1 for Linux
CUDA v8.0
Python 3.5.2
IPython 6.0.0 -- An enhanced Interactive Python.
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

v0.1: http://qiita.com/7of9/items/cd488763913f0435c39a

概要

This article is related to ADDA (light scattering simulator based on the discrete dipole approximation).

ADDAの計算で重要となるのが、X,Y,Z方向の電場の値。ランダムな初期値を用いると計算が遅く、最終の解に近い初期値を用いると計算が早くなることは経験済。

supercomputerで計算した最終解を元にDeep learningで学習を行い、その結果を通常のPCで用いる。そうすることで、通常のPC上での計算を高速化し、Communityとしての計算資源の効率利用を目論んでいる。

X,Y,Z方向の電場の値をTensorFlowで学習させようとしている。

入力パラメータとして検討しているのは以下

x,y,z : dipole position
mr, mi : refractive index
- ADDA実行ごとにパラメータとして与える

今回はmr, miをTFRecordsに追加するようにした。

code

対象ファイル

TFRecords保存 v0.2

toTFRecords_InitFieldY_170709.py

import numpy as np
import tensorflow as tf
import argparse

"""
::runtime parameters::
  - mr: real part of refractive index
  - mi: imaginary part of refractive index
"""

"""
v0.2 Jul. 09, 2017
   - read [mr] and [mi] from runtime parameters
      + add get_runtimeparams_mr_mi()
v0.1 Jul. 09, 2017
   - save as [TFRecords] format
      + add convert_to_raw()
      + add [OUT_FILE]
      + add _int64_feature()
      + add _bytes_feature()
=== branched to [IntFieldY_to_TFRecords_170709.py] ===
v0.1 Apr. 22, 2017 (convertToInputcsv_170422.py)
   - read 'IntField' then output for TensorFlow
"""

# on
#   Ubuntu 16.04 LTS
#   TensorFlow v1.1
#   Python 3.5.2

# codingrule: PEP8


def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))


def _int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


def convert_to_raw(orgval):
    wrk_org = np.array(orgval, dtype=np.float32)
    wrk_raw = wrk_org.tostring()
    return wrk_raw


def get_runtimeparams_mr_mi():
    parser = argparse.ArgumentParser(description="test170709")

    parser.add_argument(
        '--mr',
        dest='mr',
        type=float,
        help='real part of the refractive index',
        required=True)

    parser.add_argument(
        '--mi',
        dest='mi',
        type=float,
        help='imaginary part of the refractive index',
        required=True)

    cmd_args = parser.parse_args()
    return cmd_args.mr, cmd_args.mi


OUT_FILE = 'IntField-Y_170709.tfrecords'
data = np.genfromtxt('IntField-Y', delimiter=' ')
arg_mr, arg_mi = get_runtimeparams_mr_mi()

xpos, ypos, zpos = data[1:, 0], data[1:, 1], data[1:, 2]  # dipole location
E2 = data[1:, 3]
Exr, Exi = data[1:, 4], data[1:, 5]  # real and imaginary parts of Ex
Eyr, Eyi = data[1:, 6], data[1:, 7]  # real and imaginary parts of Ey
Ezr, Ezi = data[1:, 8], data[1:, 9]  # real and imaginary parts of Ez

list = zip(xpos, ypos, zpos, Exr, Exi, Eyr, Eyi, Ezr, Ezi)

with tf.python_io.TFRecordWriter(OUT_FILE) as tf_writer:
    for tpl in list:
        xpos, ypos, zpos = tpl[0:3]
        exr, exi, eyr, eyi, ezr, ezi = tpl[3:9]
        # for debug
        print("%s, %s, %s, %s, %s, %s, %s, %s, %s" % tpl)
        #
        example = tf.train.Example(features=tf.train.Features(feature={
            'xpos_raw': _bytes_feature(convert_to_raw(xpos)),
            'ypos_raw': _bytes_feature(convert_to_raw(ypos)),
            'zpos_raw': _bytes_feature(convert_to_raw(zpos)),
            'mr_raw': _bytes_feature(convert_to_raw(arg_mr)),
            'mi_raw': _bytes_feature(convert_to_raw(arg_mi)),
            'exr_raw': _bytes_feature(convert_to_raw(exr)),
            'exi_raw': _bytes_feature(convert_to_raw(exi)),
            'eyr_raw': _bytes_feature(convert_to_raw(eyr)),
            'eyi_raw': _bytes_feature(convert_to_raw(eyi)),
            'ezr_raw': _bytes_feature(convert_to_raw(ezr)),
            'ezi_raw': _bytes_feature(convert_to_raw(ezi))
            }))
        tf_writer.write(example.SerializeToString())

試し読み v0.2

fromTFRecords_InitFieldY_170709.py

import numpy as np
import tensorflow as tf

"""
v0.2 Jul. 09, 2017
  - read [mr] and [mi]
v0.1 Jul. 09, 2017
  - read position and Ex, Ey, Ez
     + add get_feature_float32()
"""

# on
#   Ubuntu 16.04 LTS
#   TensorFlow v1.1
#   Python 3.5.2

# codingrule: PEP8


def get_feature_float32(example, feature_name):
    wrk_raw = (example.features.feature[feature_name]
               .bytes_list
               .value[0])
    wrk_1d = np.fromstring(wrk_raw, dtype=np.float32)
    wrk_org = wrk_1d.reshape([1, -1])
    return wrk_org

INP_FILE = 'IntField-Y_170709.tfrecords'

record_iterator = tf.python_io.tf_record_iterator(path=INP_FILE)
for record in record_iterator:
    example = tf.train.Example()
    example.ParseFromString(record)

    xpos_org = get_feature_float32(example, 'xpos_raw')
    ypos_org = get_feature_float32(example, 'ypos_raw')
    zpos_org = get_feature_float32(example, 'zpos_raw')
    mr_org = get_feature_float32(example, 'mr_raw')
    mi_org = get_feature_float32(example, 'mi_raw')
    exr_org = get_feature_float32(example, 'exr_raw')
    exi_org = get_feature_float32(example, 'exi_raw')
    eyr_org = get_feature_float32(example, 'eyr_raw')
    eyi_org = get_feature_float32(example, 'eyi_raw')
    ezr_org = get_feature_float32(example, 'ezr_raw')
    ezi_org = get_feature_float32(example, 'ezi_raw')

    list_pos = *xpos_org, *ypos_org, *zpos_org, *mr_org, *mi_org
    list_e = *exr_org, *exi_org, *eyr_org, *eyi_org, *ezr_org, *ezi_org

    print(*list_pos, *list_e)

実行

--mrと--miを指定する。

Pope and Fry (1997)の波長550nmあたりの水粒子の値など。
http://www.philiplaven.com/p20.html

TFRecords格納時の出力は今回は使わないので/dev/nullにする。

$ python3 toTFRecords_InitFieldY_170709.py --mr 1.335 --mi 1e-8 > /dev/null

$ python3 fromTFRecords_InitFieldY_170709.py | head
[-0.23620997] [-1.6534698] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.01643909] [ 0.00506505] [ 0.53640616] [ 0.29324207] [ 0.03475441] [-0.07514133]
[ 0.23620997] [-1.6534698] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [ 0.01643909] [-0.00506505] [ 0.53640616] [ 0.29324207] [ 0.03475441] [-0.07514133]
[-1.18104982] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.06216653] [ 0.02425352] [ 0.61625606] [ 0.27670044] [ 0.02440614] [-0.05486268]
[-0.70862991] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.07402581] [ 0.00639074] [ 0.51409435] [ 0.30612269] [ 0.02715302] [-0.06470039]
[-0.23620997] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.03451369] [-0.00239405] [ 0.52134091] [ 0.35712481] [ 0.03088294] [-0.06684458]
[ 0.23620997] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [ 0.03451369] [ 0.00239405] [ 0.52134091] [ 0.35712481] [ 0.03088294] [-0.06684458]
[ 0.70862991] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [ 0.07402581] [-0.00639074] [ 0.51409435] [ 0.30612269] [ 0.02715302] [-0.06470039]
[ 1.18104982] [-1.18104982] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [ 0.06216653] [-0.02425352] [ 0.61625606] [ 0.27670044] [ 0.02440614] [-0.05486268]
[-1.18104982] [-0.70862991] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.06618973] [ 0.00543136] [ 0.67104828] [ 0.33572644] [ 0.01676365] [-0.03849823]
[-0.70862991] [-0.70862991] [-5.90524912] [ 1.33500004] [  9.99999994e-09] [-0.04588795] [ 0.00480362] [ 0.54361492] [ 0.37286496] [ 0.02019378] [-0.04343719]
Traceback (most recent call last):
  File "fromTFRecords_InitFieldY_170709.py", line 50, in <module>
    print(*list_pos, *list_e)
BrokenPipeError: [Errno 32] Broken pipe

４列目と５列目に入った。