Edited at

[TensorFlow] Python ⇔ Protocol Buffers ⇔ GPU/分散コンピューティング

More than 3 years have passed since last update.


はじめに

ディープラーニングのフレームワークTensorFlowは各種処理をProtocol Buffers経由で外だしすることで、PythonとGPU/分散コンピュータとのスイッチングを減らし、計算を効率化しています。

Protocol Buffersは、Googleの分散コンピューティングを支える技術で、言語非依存、プラットフォーム非依存にデータ構造をシリアライズする仕組みです。現在、C++, C#, GO, Java, Pythonがサポートされています。

ProtocolBuffers.png

TensorFlowは、処理をノードとしてグラフを構築し、一気に計算する仕組みになっています。

グラフの例

以下TensorFlowのグラフがProtocol Buffers形式にシリアライズされる様子を見てみたいと思います。


サンプルプログラム

TensorFlowで足し算をしてみます。

初期値0で加算1を3回実行します。


add.py

import tensorflow as tf

state = tf.Variable(0, name="counter")

one = tf.constant(1)
new_value = tf.add(state, one)
update = tf.assign(state, new_value)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
sess.run(init_op)
print(sess.run(state))
for _ in range(3):
sess.run(update)
print(sess.run(state))



TensorBoardのグラフ

TensorBoardでグラフを表示

add.png


Protocol Buffers

以下、TensorFlowを使ったPythonのプログラムが、Protocol Buffersのノードに変換されると、どのようになるのかを見ていきます。


変数の定義


  • Python

state = tf.Variable(0, name="counter")


  • Protocol Buffers:counterという名前の変数

name: "counter"

op: "Variable"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "shape"
value {
shape {
}
}
}
attr {
key: "shared_name"
value {
s: ""
}
}


  • Protocol Buffers:初期値0(定数)の定義

name: "counter/initial_value"

op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 0
}
}
}


  • Protocol Buffers:変数counterに初期値0をアサイン

name: "counter/Assign"

op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}


  • Protocol Buffers:変数counterを読みだす

name: "counter/read"

op: "Identity"
input: "counter"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}


定数(加算値)の定義


  • Python

one = tf.constant(1)


  • Protocol Buffers:定数1の定義

name: "Const"

op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 1
}
}
}


加算オペレーションの定義


  • Python

one = tf.constant(1)


  • Protocol Buffers:加算オペレーションを定義

name: "Add"

op: "Add"
input: "counter/read"
input: "Const"
attr {
key: "T"
value {
type: DT_INT32
}
}


加算結果を変数に値を代入


  • Python

update = tf.assign(state, new_value)


  • Protocol Buffers:加算結果を変数に値を代入

name: "Assign"

op: "Assign"
input: "counter"
input: "Add"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}


おわりに

Pythonプログラムの裏でTensorFlowがProtocol Buffersを利用していることを理解していると、なぜその場所にTensorFlowの処理が差し込まれているのか、分かってくるのではないでしょうか。

TensorFlowには、ベースとなる各種処理が用意されています(API)。


  • グラフ

  • 変数

  • テンソル

  • 数学

  • 文字列

  • 制御構文

  • アサーション

  • イメージ

  • 動画

  • 入出力

  • ニューラルネットワーク

  • オペレーションのサマリー

  • ユニットテスト

TensorFlowをディープラーニングのフレームワークとしてではなく、GPUコンピューティング/分散コンピューティングのフレームワークとして利用してみるのも面白いですね。


      

 

 

 

 

 

 

 

 

 

 

 

 


(参考)


グラフ表示プログラム

graph = tf.get_default_graph()

summary_writer = tf.train.SummaryWriter('log_valiable', graph)
operations = graph.get_operations()
for operation in operations:
print("======================")
print("=== name ===")
print(operation.name)
print("=== type ===")
print(operation.type)
print("=== inputs ===")
for input in operation.inputs:
print(input)
print("=== control_inputs ===")
for control_input in operation.control_inputs:
print(control_input)
print("=== outputs ===")
for output in operation.outputs:
print(output)
print("=== node_def ===")
print(operation.node_def)
print("=== op_def ===")
print(operation.op_def)
print("=== traceback ===")
print(operation.traceback)
print("")


コンソール出力

0

1
2
3
======================
=== name ===
counter/initial_value
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/initial_value"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 0
}
}
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 293, '_init_from_args', 'initial_value, name="initial_value", dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 657, 'convert_to_tensor', 'ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 180, '_constant_tensor_conversion_function', 'return constant(v, dtype=dtype, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter
=== type ===
Variable
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter"
op: "Variable"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "shape"
value {
shape {
}
}
}
attr {
key: "shared_name"
value {
s: ""
}
}

=== op_def ===
name: "Variable"
output_arg {
name: "ref"
type_attr: "dtype"
is_ref: true
}
attr {
name: "shape"
type: "shape"
}
attr {
name: "dtype"
type: "type"
}
attr {
name: "container"
type: "string"
default_value {
s: ""
}
}
attr {
name: "shared_name"
type: "string"
default_value {
s: ""
}
}
is_stateful: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 300, '_init_from_args', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py', 146, 'variable_op', 'container=container, shared_name=shared_name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 490, '_variable', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("counter/Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}

=== op_def ===
name: "Assign"
input_arg {
name: "ref"
type_attr: "T"
is_ref: true
}
input_arg {
name: "value"
type_attr: "T"
}
output_arg {
name: "output_ref"
type_attr: "T"
is_ref: true
}
attr {
name: "T"
type: "type"
}
attr {
name: "validate_shape"
type: "bool"
default_value {
b: true
}
}
attr {
name: "use_locking"
type: "bool"
default_value {
b: true
}
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 317, '_init_from_args', 'validate_shape=validate_shape).op'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/read
=== type ===
Identity
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== control_inputs ===
=== outputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/read"
op: "Identity"
input: "counter"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}

=== op_def ===
name: "Identity"
input_arg {
name: "input"
type_attr: "T"
}
output_arg {
name: "output"
type_attr: "T"
}
attr {
name: "T"
type: "type"
}

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 327, '_init_from_args', 'self._snapshot = array_ops.identity(self._variable, name="read")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py', 1128, 'identity', 'result = _op_def_lib.apply_op("Identity", input=input, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Const
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("Const:0", shape=(), dtype=int32)
=== node_def ===
name: "Const"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 1
}
}
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 7, '<module>', 'one = tf.constant(1)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Add
=== type ===
Add
=== inputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Add:0", shape=(), dtype=int32)
=== node_def ===
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
key: "T"
value {
type: DT_INT32
}
}

=== op_def ===
name: "Add"
input_arg {
name: "x"
type_attr: "T"
}
input_arg {
name: "y"
type_attr: "T"
}
output_arg {
name: "z"
type_attr: "T"
}
attr {
name: "T"
type: "type"
allowed_values {
list {
type: DT_HALF
type: DT_FLOAT
type: DT_DOUBLE
type: DT_UINT8
type: DT_INT8
type: DT_INT16
type: DT_INT32
type: DT_INT64
type: DT_COMPLEX64
type: DT_COMPLEX128
type: DT_STRING
}
}
}

=== traceback ===
[('./valiable.py', 8, '<module>', 'new_value = tf.add(state, one)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py', 71, 'add', 'result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("Add:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}

=== op_def ===
name: "Assign"
input_arg {
name: "ref"
type_attr: "T"
is_ref: true
}
input_arg {
name: "value"
type_attr: "T"
}
output_arg {
name: "output_ref"
type_attr: "T"
is_ref: true
}
attr {
name: "T"
type: "type"
}
attr {
name: "validate_shape"
type: "bool"
default_value {
b: true
}
}
attr {
name: "use_locking"
type: "bool"
default_value {
b: true
}
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 9, '<module>', 'update = tf.assign(state, new_value)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
init
=== type ===
NoOp
=== inputs ===
=== control_inputs ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}

=== outputs ===
=== node_def ===
name: "init"
op: "NoOp"
input: "^counter/Assign"

=== op_def ===
name: "NoOp"

=== traceback ===
[('./valiable.py', 11, '<module>', 'init_op = tf.initialize_all_variables()'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1063, 'initialize_all_variables', 'return initialize_variables(all_variables())'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1051, 'initialize_variables', 'return control_flow_ops.group(*[v.initializer for v in var_list], name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2645, 'group', 'return _GroupControlDeps(dev, deps, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2603, '_GroupControlDeps', 'return no_op(name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py', 184, 'no_op', 'result = _op_def_lib.apply_op("NoOp", name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 756, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]


Protocol Buffersのベンチマーク

Protocol Buffers が本当に遅いのか実際に確かめてみた @2016/8/24