動作環境
GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 16.04 LTS desktop amd64
TensorFlow v1.1.0
cuDNN v5.1 for Linux
CUDA v8.0
Python 3.5.2
IPython 6.0.0 -- An enhanced Interactive Python.
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
v0.1: http://qiita.com/7of9/items/cd488763913f0435c39a
概要
This article is related to ADDA (light scattering simulator based on the discrete dipole approximation).
ADDAの計算で重要となるのが、X,Y,Z方向の電場の値。ランダムな初期値を用いると計算が遅く、最終の解に近い初期値を用いると計算が早くなることは経験済。
supercomputerで計算した最終解を元にDeep learningで学習を行い、その結果を通常のPCで用いる。そうすることで、通常のPC上での計算を高速化し、Communityとしての計算資源の効率利用を目論んでいる。
X,Y,Z方向の電場の値をTensorFlowで学習させようとしている。
入力パラメータとして検討しているのは以下
- x,y,z : dipole position
- mr, mi : refractive index
- ADDA実行ごとにパラメータとして与える
今回はmr, miをTFRecordsに追加するようにした。
code
対象ファイル
TFRecords保存 v0.2
toTFRecords_InitFieldY_170709.py
import numpy as np
import tensorflow as tf
import argparse
"""
::runtime parameters::
- mr: real part of refractive index
- mi: imaginary part of refractive index
"""
"""
v0.2 Jul. 09, 2017
- read [mr] and [mi] from runtime parameters
+ add get_runtimeparams_mr_mi()
v0.1 Jul. 09, 2017
- save as [TFRecords] format
+ add convert_to_raw()
+ add [OUT_FILE]
+ add _int64_feature()
+ add _bytes_feature()
=== branched to [IntFieldY_to_TFRecords_170709.py] ===
v0.1 Apr. 22, 2017 (convertToInputcsv_170422.py)
- read 'IntField' then output for TensorFlow
"""
# on
# Ubuntu 16.04 LTS
# TensorFlow v1.1
# Python 3.5.2
# codingrule: PEP8
def _bytes_feature(value):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
def _int64_feature(value):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
def convert_to_raw(orgval):
wrk_org = np.array(orgval, dtype=np.float32)
wrk_raw = wrk_org.tostring()
return wrk_raw
def get_runtimeparams_mr_mi():
parser = argparse.ArgumentParser(description="test170709")
parser.add_argument(
'--mr',
dest='mr',
type=float,
help='real part of the refractive index',
required=True)
parser.add_argument(
'--mi',
dest='mi',
type=float,
help='imaginary part of the refractive index',
required=True)
cmd_args = parser.parse_args()
return cmd_args.mr, cmd_args.mi
OUT_FILE = 'IntField-Y_170709.tfrecords'
data = np.genfromtxt('IntField-Y', delimiter=' ')
arg_mr, arg_mi = get_runtimeparams_mr_mi()
xpos, ypos, zpos = data[1:, 0], data[1:, 1], data[1:, 2] # dipole location
E2 = data[1:, 3]
Exr, Exi = data[1:, 4], data[1:, 5] # real and imaginary parts of Ex
Eyr, Eyi = data[1:, 6], data[1:, 7] # real and imaginary parts of Ey
Ezr, Ezi = data[1:, 8], data[1:, 9] # real and imaginary parts of Ez
list = zip(xpos, ypos, zpos, Exr, Exi, Eyr, Eyi, Ezr, Ezi)
with tf.python_io.TFRecordWriter(OUT_FILE) as tf_writer:
for tpl in list:
xpos, ypos, zpos = tpl[0:3]
exr, exi, eyr, eyi, ezr, ezi = tpl[3:9]
# for debug
print("%s, %s, %s, %s, %s, %s, %s, %s, %s" % tpl)
#
example = tf.train.Example(features=tf.train.Features(feature={
'xpos_raw': _bytes_feature(convert_to_raw(xpos)),
'ypos_raw': _bytes_feature(convert_to_raw(ypos)),
'zpos_raw': _bytes_feature(convert_to_raw(zpos)),
'mr_raw': _bytes_feature(convert_to_raw(arg_mr)),
'mi_raw': _bytes_feature(convert_to_raw(arg_mi)),
'exr_raw': _bytes_feature(convert_to_raw(exr)),
'exi_raw': _bytes_feature(convert_to_raw(exi)),
'eyr_raw': _bytes_feature(convert_to_raw(eyr)),
'eyi_raw': _bytes_feature(convert_to_raw(eyi)),
'ezr_raw': _bytes_feature(convert_to_raw(ezr)),
'ezi_raw': _bytes_feature(convert_to_raw(ezi))
}))
tf_writer.write(example.SerializeToString())
試し読み v0.2
fromTFRecords_InitFieldY_170709.py
import numpy as np
import tensorflow as tf
"""
v0.2 Jul. 09, 2017
- read [mr] and [mi]
v0.1 Jul. 09, 2017
- read position and Ex, Ey, Ez
+ add get_feature_float32()
"""
# on
# Ubuntu 16.04 LTS
# TensorFlow v1.1
# Python 3.5.2
# codingrule: PEP8
def get_feature_float32(example, feature_name):
wrk_raw = (example.features.feature[feature_name]
.bytes_list
.value[0])
wrk_1d = np.fromstring(wrk_raw, dtype=np.float32)
wrk_org = wrk_1d.reshape([1, -1])
return wrk_org
INP_FILE = 'IntField-Y_170709.tfrecords'
record_iterator = tf.python_io.tf_record_iterator(path=INP_FILE)
for record in record_iterator:
example = tf.train.Example()
example.ParseFromString(record)
xpos_org = get_feature_float32(example, 'xpos_raw')
ypos_org = get_feature_float32(example, 'ypos_raw')
zpos_org = get_feature_float32(example, 'zpos_raw')
mr_org = get_feature_float32(example, 'mr_raw')
mi_org = get_feature_float32(example, 'mi_raw')
exr_org = get_feature_float32(example, 'exr_raw')
exi_org = get_feature_float32(example, 'exi_raw')
eyr_org = get_feature_float32(example, 'eyr_raw')
eyi_org = get_feature_float32(example, 'eyi_raw')
ezr_org = get_feature_float32(example, 'ezr_raw')
ezi_org = get_feature_float32(example, 'ezi_raw')
list_pos = *xpos_org, *ypos_org, *zpos_org, *mr_org, *mi_org
list_e = *exr_org, *exi_org, *eyr_org, *eyi_org, *ezr_org, *ezi_org
print(*list_pos, *list_e)
実行
--mrと--miを指定する。
Pope and Fry (1997)の波長550nmあたりの水粒子の値など。
http://www.philiplaven.com/p20.html
TFRecords格納時の出力は今回は使わないので/dev/nullにする。
$ python3 toTFRecords_InitFieldY_170709.py --mr 1.335 --mi 1e-8 > /dev/null
$ python3 fromTFRecords_InitFieldY_170709.py | head
[-0.23620997] [-1.6534698] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.01643909] [ 0.00506505] [ 0.53640616] [ 0.29324207] [ 0.03475441] [-0.07514133]
[ 0.23620997] [-1.6534698] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [ 0.01643909] [-0.00506505] [ 0.53640616] [ 0.29324207] [ 0.03475441] [-0.07514133]
[-1.18104982] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.06216653] [ 0.02425352] [ 0.61625606] [ 0.27670044] [ 0.02440614] [-0.05486268]
[-0.70862991] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.07402581] [ 0.00639074] [ 0.51409435] [ 0.30612269] [ 0.02715302] [-0.06470039]
[-0.23620997] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.03451369] [-0.00239405] [ 0.52134091] [ 0.35712481] [ 0.03088294] [-0.06684458]
[ 0.23620997] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [ 0.03451369] [ 0.00239405] [ 0.52134091] [ 0.35712481] [ 0.03088294] [-0.06684458]
[ 0.70862991] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [ 0.07402581] [-0.00639074] [ 0.51409435] [ 0.30612269] [ 0.02715302] [-0.06470039]
[ 1.18104982] [-1.18104982] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [ 0.06216653] [-0.02425352] [ 0.61625606] [ 0.27670044] [ 0.02440614] [-0.05486268]
[-1.18104982] [-0.70862991] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.06618973] [ 0.00543136] [ 0.67104828] [ 0.33572644] [ 0.01676365] [-0.03849823]
[-0.70862991] [-0.70862991] [-5.90524912] [ 1.33500004] [ 9.99999994e-09] [-0.04588795] [ 0.00480362] [ 0.54361492] [ 0.37286496] [ 0.02019378] [-0.04343719]
Traceback (most recent call last):
File "fromTFRecords_InitFieldY_170709.py", line 50, in <module>
print(*list_pos, *list_e)
BrokenPipeError: [Errno 32] Broken pipe
４列目と５列目に入った。