Help us understand the problem. What is going on with this article?

[TensorFlow] Python ⇔ Protocol Buffers ⇔ GPU/分散コンピューティング

More than 3 years have passed since last update.

はじめに

ディープラーニングのフレームワークTensorFlowは各種処理をProtocol Buffers経由で外だしすることで、PythonとGPU/分散コンピュータとのスイッチングを減らし、計算を効率化しています。

Protocol Buffersは、Googleの分散コンピューティングを支える技術で、言語非依存、プラットフォーム非依存にデータ構造をシリアライズする仕組みです。現在、C++, C#, GO, Java, Pythonがサポートされています。

ProtocolBuffers.png

TensorFlowは、処理をノードとしてグラフを構築し、一気に計算する仕組みになっています。
グラフの例

以下TensorFlowのグラフがProtocol Buffers形式にシリアライズされる様子を見てみたいと思います。

サンプルプログラム

TensorFlowで足し算をしてみます。
初期値0で加算1を3回実行します。

add.py
import tensorflow as tf

state = tf.Variable(0, name="counter")

one = tf.constant(1)
new_value = tf.add(state, one)
update = tf.assign(state, new_value)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
  sess.run(init_op)
  print(sess.run(state))
  for _ in range(3):
    sess.run(update)
    print(sess.run(state))

TensorBoardのグラフ

TensorBoardでグラフを表示
add.png

Protocol Buffers

以下、TensorFlowを使ったPythonのプログラムが、Protocol Buffersのノードに変換されると、どのようになるのかを見ていきます。

変数の定義

  • Python
state = tf.Variable(0, name="counter")
  • Protocol Buffers:counterという名前の変数
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}
  • Protocol Buffers:初期値0(定数)の定義
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}
  • Protocol Buffers:変数counterに初期値0をアサイン
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}
  • Protocol Buffers:変数counterを読みだす
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}

定数(加算値)の定義

  • Python
one = tf.constant(1)
  • Protocol Buffers:定数1の定義
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}

加算オペレーションの定義

  • Python
one = tf.constant(1)
  • Protocol Buffers:加算オペレーションを定義
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

加算結果を変数に値を代入

  • Python
update = tf.assign(state, new_value)
  • Protocol Buffers:加算結果を変数に値を代入
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

おわりに

Pythonプログラムの裏でTensorFlowがProtocol Buffersを利用していることを理解していると、なぜその場所にTensorFlowの処理が差し込まれているのか、分かってくるのではないでしょうか。

TensorFlowには、ベースとなる各種処理が用意されています(API)。

  • グラフ
  • 変数
  • テンソル
  • 数学
  • 文字列
  • 制御構文
  • アサーション
  • イメージ
  • 動画
  • 入出力
  • ニューラルネットワーク
  • オペレーションのサマリー
  • ユニットテスト

TensorFlowをディープラーニングのフレームワークとしてではなく、GPUコンピューティング/分散コンピューティングのフレームワークとして利用してみるのも面白いですね。

      

 
 
 
 
 
 
 
 
 
 
 
 

(参考)

グラフ表示プログラム

graph = tf.get_default_graph()
summary_writer = tf.train.SummaryWriter('log_valiable', graph)
operations =  graph.get_operations()
for operation in operations:
    print("======================")
    print("=== name ===")
    print(operation.name)
    print("=== type ===")
    print(operation.type)
    print("=== inputs ===")
    for input in operation.inputs:
        print(input)
    print("=== control_inputs ===")
    for control_input in operation.control_inputs:
        print(control_input)
    print("=== outputs ===")
    for output in operation.outputs:
        print(output)
    print("=== node_def ===")
    print(operation.node_def)
    print("=== op_def ===")
    print(operation.op_def)
    print("=== traceback ===")
    print(operation.traceback)
    print("")

コンソール出力

0
1
2
3
======================
=== name ===
counter/initial_value
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 293, '_init_from_args', 'initial_value, name="initial_value", dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 657, 'convert_to_tensor', 'ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 180, '_constant_tensor_conversion_function', 'return constant(v, dtype=dtype, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter
=== type ===
Variable
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}

=== op_def ===
name: "Variable"
output_arg {
  name: "ref"
  type_attr: "dtype"
  is_ref: true
}
attr {
  name: "shape"
  type: "shape"
}
attr {
  name: "dtype"
  type: "type"
}
attr {
  name: "container"
  type: "string"
  default_value {
    s: ""
  }
}
attr {
  name: "shared_name"
  type: "string"
  default_value {
    s: ""
  }
}
is_stateful: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 300, '_init_from_args', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py', 146, 'variable_op', 'container=container, shared_name=shared_name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 490, '_variable', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("counter/Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 317, '_init_from_args', 'validate_shape=validate_shape).op'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/read
=== type ===
Identity
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== control_inputs ===
=== outputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}

=== op_def ===
name: "Identity"
input_arg {
  name: "input"
  type_attr: "T"
}
output_arg {
  name: "output"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
}

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 327, '_init_from_args', 'self._snapshot = array_ops.identity(self._variable, name="read")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py', 1128, 'identity', 'result = _op_def_lib.apply_op("Identity", input=input, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Const
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("Const:0", shape=(), dtype=int32)
=== node_def ===
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 7, '<module>', 'one = tf.constant(1)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Add
=== type ===
Add
=== inputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Add:0", shape=(), dtype=int32)
=== node_def ===
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

=== op_def ===
name: "Add"
input_arg {
  name: "x"
  type_attr: "T"
}
input_arg {
  name: "y"
  type_attr: "T"
}
output_arg {
  name: "z"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
  allowed_values {
    list {
      type: DT_HALF
      type: DT_FLOAT
      type: DT_DOUBLE
      type: DT_UINT8
      type: DT_INT8
      type: DT_INT16
      type: DT_INT32
      type: DT_INT64
      type: DT_COMPLEX64
      type: DT_COMPLEX128
      type: DT_STRING
    }
  }
}

=== traceback ===
[('./valiable.py', 8, '<module>', 'new_value = tf.add(state, one)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py', 71, 'add', 'result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("Add:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 9, '<module>', 'update = tf.assign(state, new_value)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
init
=== type ===
NoOp
=== inputs ===
=== control_inputs ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== outputs ===
=== node_def ===
name: "init"
op: "NoOp"
input: "^counter/Assign"

=== op_def ===
name: "NoOp"

=== traceback ===
[('./valiable.py', 11, '<module>', 'init_op = tf.initialize_all_variables()'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1063, 'initialize_all_variables', 'return initialize_variables(all_variables())'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1051, 'initialize_variables', 'return control_flow_ops.group(*[v.initializer for v in var_list], name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2645, 'group', 'return _GroupControlDeps(dev, deps, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2603, '_GroupControlDeps', 'return no_op(name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py', 184, 'no_op', 'result = _op_def_lib.apply_op("NoOp", name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 756, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

Protocol Buffersのベンチマーク

Protocol Buffers が本当に遅いのか実際に確かめてみた @2016/8/24

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away