LoginSignup
18
20

More than 5 years have passed since last update.

@hitomatagi

[TensorFlow] Python ⇔ Protocol Buffers ⇔ GPU/分散コンピューティング

Last updated at Posted at 2016-11-05

はじめに

ディープラーニングのフレームワークTensorFlowは各種処理をProtocol Buffers経由で外だしすることで、PythonとGPU／分散コンピュータとのスイッチングを減らし、計算を効率化しています。

Protocol Buffersは、Googleの分散コンピューティングを支える技術で、言語非依存、プラットフォーム非依存にデータ構造をシリアライズする仕組みです。現在、C++, C#, GO, Java, Pythonがサポートされています。

ProtocolBuffers.png

TensorFlowは、処理をノードとしてグラフを構築し、一気に計算する仕組みになっています。
グラフの例

以下TensorFlowのグラフがProtocol Buffers形式にシリアライズされる様子を見てみたいと思います。

サンプルプログラム

TensorFlowで足し算をしてみます。
初期値０で加算１を３回実行します。

add.py
import tensorflow as tf

state = tf.Variable(0, name="counter")

one = tf.constant(1)
new_value = tf.add(state, one)
update = tf.assign(state, new_value)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
  sess.run(init_op)
  print(sess.run(state))
  for _ in range(3):
    sess.run(update)
    print(sess.run(state))

TensorBoardのグラフ

TensorBoardでグラフを表示
add.png

Protocol Buffers

以下、TensorFlowを使ったPythonのプログラムが、Protocol Buffersのノードに変換されると、どのようになるのかを見ていきます。

変数の定義

  • Python
state = tf.Variable(0, name="counter")
  • Protocol Buffers：counterという名前の変数
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}
  • Protocol Buffers：初期値０（定数）の定義
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}
  • Protocol Buffers：変数counterに初期値０をアサイン
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}
  • Protocol Buffers：変数counterを読みだす
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}

定数（加算値）の定義

  • Python
one = tf.constant(1)
  • Protocol Buffers：定数１の定義
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}

加算オペレーションの定義

  • Python
one = tf.constant(1)
  • Protocol Buffers：加算オペレーションを定義
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

加算結果を変数に値を代入

  • Python
update = tf.assign(state, new_value)
  • Protocol Buffers：加算結果を変数に値を代入
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

おわりに

Pythonプログラムの裏でTensorFlowがProtocol Buffersを利用していることを理解していると、なぜその場所にTensorFlowの処理が差し込まれているのか、分かってくるのではないでしょうか。

TensorFlowには、ベースとなる各種処理が用意されています（API）。

  • グラフ
  • 変数
  • テンソル
  • 数学
  • 文字列
  • 制御構文
  • アサーション
  • イメージ
  • 動画
  • 入出力
  • ニューラルネットワーク
  • オペレーションのサマリー
  • ユニットテスト

TensorFlowをディープラーニングのフレームワークとしてではなく、GPUコンピューティング/分散コンピューティングのフレームワークとして利用してみるのも面白いですね。

　　　　　　

　
　
　
　
　
　
　
　
　
　
　
　

（参考）

グラフ表示プログラム

graph = tf.get_default_graph()
summary_writer = tf.train.SummaryWriter('log_valiable', graph)
operations =  graph.get_operations()
for operation in operations:
    print("======================")
    print("=== name ===")
    print(operation.name)
    print("=== type ===")
    print(operation.type)
    print("=== inputs ===")
    for input in operation.inputs:
        print(input)
    print("=== control_inputs ===")
    for control_input in operation.control_inputs:
        print(control_input)
    print("=== outputs ===")
    for output in operation.outputs:
        print(output)
    print("=== node_def ===")
    print(operation.node_def)
    print("=== op_def ===")
    print(operation.op_def)
    print("=== traceback ===")
    print(operation.traceback)
    print("")

コンソール出力

0
1
2
3
======================
=== name ===
counter/initial_value
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 293, '_init_from_args', 'initial_value, name="initial_value", dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 657, 'convert_to_tensor', 'ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 180, '_constant_tensor_conversion_function', 'return constant(v, dtype=dtype, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter
=== type ===
Variable
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}

=== op_def ===
name: "Variable"
output_arg {
  name: "ref"
  type_attr: "dtype"
  is_ref: true
}
attr {
  name: "shape"
  type: "shape"
}
attr {
  name: "dtype"
  type: "type"
}
attr {
  name: "container"
  type: "string"
  default_value {
    s: ""
  }
}
attr {
  name: "shared_name"
  type: "string"
  default_value {
    s: ""
  }
}
is_stateful: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 300, '_init_from_args', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py', 146, 'variable_op', 'container=container, shared_name=shared_name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 490, '_variable', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("counter/Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 317, '_init_from_args', 'validate_shape=validate_shape).op'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/read
=== type ===
Identity
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== control_inputs ===
=== outputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}

=== op_def ===
name: "Identity"
input_arg {
  name: "input"
  type_attr: "T"
}
output_arg {
  name: "output"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
}

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 327, '_init_from_args', 'self._snapshot = array_ops.identity(self._variable, name="read")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py', 1128, 'identity', 'result = _op_def_lib.apply_op("Identity", input=input, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Const
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("Const:0", shape=(), dtype=int32)
=== node_def ===
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 7, '<module>', 'one = tf.constant(1)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Add
=== type ===
Add
=== inputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Add:0", shape=(), dtype=int32)
=== node_def ===
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

=== op_def ===
name: "Add"
input_arg {
  name: "x"
  type_attr: "T"
}
input_arg {
  name: "y"
  type_attr: "T"
}
output_arg {
  name: "z"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
  allowed_values {
    list {
      type: DT_HALF
      type: DT_FLOAT
      type: DT_DOUBLE
      type: DT_UINT8
      type: DT_INT8
      type: DT_INT16
      type: DT_INT32
      type: DT_INT64
      type: DT_COMPLEX64
      type: DT_COMPLEX128
      type: DT_STRING
    }
  }
}

=== traceback ===
[('./valiable.py', 8, '<module>', 'new_value = tf.add(state, one)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py', 71, 'add', 'result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("Add:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 9, '<module>', 'update = tf.assign(state, new_value)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
init
=== type ===
NoOp
=== inputs ===
=== control_inputs ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== outputs ===
=== node_def ===
name: "init"
op: "NoOp"
input: "^counter/Assign"

=== op_def ===
name: "NoOp"

=== traceback ===
[('./valiable.py', 11, '<module>', 'init_op = tf.initialize_all_variables()'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1063, 'initialize_all_variables', 'return initialize_variables(all_variables())'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1051, 'initialize_variables', 'return control_flow_ops.group(*[v.initializer for v in var_list], name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2645, 'group', 'return _GroupControlDeps(dev, deps, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2603, '_GroupControlDeps', 'return no_op(name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py', 184, 'no_op', 'result = _op_def_lib.apply_op("NoOp", name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 756, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

Protocol Buffersのベンチマーク

Protocol Buffers が本当に遅いのか実際に確かめてみた @2016/8/24

18
20
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
18
20