fluentd+zabbix-agent+libratoでサーバのメトリクスを可視化する

fluent-plugin-zabbix-agentを使った、Zabbixサーバレス＋libratoでのメトリクスの可視化について書きます。

用意するもの

• fluentd (v0.12以降)
• fluent-plugin-zabbix-agent (v0.1.4以降)
• fluent-plugin-librato
• fluent-plugin-filter-record-map
• librato
• zabbix-agent (あらかじめ起動しておく）

fluent.conf

以下のようなfluent.confでfluentdを立ち上げると、libratoにメトリクスが送信されます。

<source>
  @type zabbix_agent
  @id zabbix_agent_input
  interval 60
  items_file /path/to/zabbix-items.json
  @label @raw
</source>

<label @raw>
  <filter zabbix.**>
    @type record_map
    @id record_map_filter
    map1 key, source = record["key"].split("/", 2)
    map2 source ||= key
    map3 new_record["key"] = hostname + "." + key
    map4 new_record["source"] = source
    map5 new_record["value"] = record["value"]
  </filter>
  <match zabbix.**>
    @type librato
    @id librato_output
    flush_interval 1s
    email [e-mail]
    apikey ...
  </match>
</label>

zabbix-items.json

収集対象のzabbixのitemは以下の通り

{
  "agent.ping": "agent.ping",
  "net.if.in[eth0,bytes]": "net.if.in.eth0/bytes",
  "net.if.in[eth0,dropped]": "net.if.in.eth0/dropped",
  "net.if.in[eth0,errors]": "net.if.in.eth0/errors",
  "net.if.in[eth0]": "net.if.in.eth0",
  "net.if.in[lo,bytes]": "net.if.in.lo/bytes",
  "net.if.in[lo,dropped]": "net.if.in.lo/dropped",
  "net.if.in[lo,errors]": "net.if.in.lo/errors",
  "net.if.in[lo]": "net.if.in.lo",
  "net.if.out[eth0,bytes]": "net.if.out.eth0/bytes",
  "net.if.out[eth0,dropped]": "net.if.out.eth0/dropped",
  "net.if.out[eth0,errors]": "net.if.out.eth0/errors",
  "net.if.out[lo,bytes]": "net.if.out.lo/bytes",
  "net.if.out[lo,dropped]": "net.if.out.lo/dropped",
  "net.if.out[lo,errors]": "net.if.out.lo/errors",
  "proc.mem[crond]": "proc.mem/crond",
  "proc.mem[klogd]": "proc.mem/klogd",
  "proc.mem[syslogd]": "proc.mem/syslogd",
  "proc.num[,,run]": "proc.num/run",
  "proc.num[crond]": "proc.num/crond",
  "proc.num[klogd]": "proc.num/klogd",
  "proc.num[syslogd]": "proc.num/syslogd",
  "proc.num[]": "proc.num",
  "system.boottime": "system.boottime",
  "system.cpu.intr": "system.cpu.intr",
  "system.cpu.load[,avg15]": "system.cpu.load/avg15",
  "system.cpu.load[,avg1]": "system.cpu.load/avg1",
  "system.cpu.load[,avg5]": "system.cpu.load/avg5",
  "system.cpu.util[,idle,avg1]": "system.cpu.util/idle.avg1",
  "system.cpu.util[,nice,avg1]": "system.cpu.util/nice.avg1",
  "system.cpu.util[,system,avg1]": "system.cpu.util/system.avg1",
  "system.cpu.util[,user,avg1]": "system.cpu.util/user.avg1",
  "system.cpu.util[,iowait,avg1]": "system.cpu.util/wait.avg1",
  "system.swap.size[,free]": "system.swap.size/free",
  "system.swap.size[,pused]": "system.swap.size/pused",
  "system.swap.size[,total]": "system.swap.size/total",
  "system.uptime": "system.uptime",
  "system.users.num": "system.users.num",
  "vfs.dev.read[xvda,operations]": "vfs.dev.operations.xvda/read",
  "vfs.dev.write[xvda,operations]": "vfs.dev.operations.xvda/write",
  "vfs.fs.inode[/,pused]": "vfs.fs.inode.root/pused",
  "vfs.fs.inode[/,total]": "vfs.fs.inode.root/total",
  "vfs.fs.inode[/boot,pused]": "vfs.fs.inode.boot/pused",
  "vfs.fs.inode[/boot,total]": "vfs.fs.inode.boot/total",
  "vfs.fs.inode[/home,pused]": "vfs.fs.inode.home/pused",
  "vfs.fs.inode[/home,total]": "vfs.fs.inode.home/total",
  "vfs.fs.inode[/var,pused]": "vfs.fs.inode.var/pused",
  "vfs.fs.inode[/var,total]": "vfs.fs.inode.var/total",
  "vfs.fs.size[/,free]": "vfs.fs.size.root/free",
  "vfs.fs.size[/,pused]": "vfs.fs.size.root/pused",
  "vfs.fs.size[/,total]": "vfs.fs.size.root/total",
  "vfs.fs.size[/boot,free]": "vfs.fs.size.boot/free",
  "vfs.fs.size[/boot,pused]": "vfs.fs.size.boot/pused",
  "vfs.fs.size[/boot,total]": "vfs.fs.size.boot/total",
  "vfs.fs.size[/home,free]": "vfs.fs.size.home/free",
  "vfs.fs.size[/home,pused]": "vfs.fs.size.home/pused",
  "vfs.fs.size[/home,total]": "vfs.fs.size.home/total",
  "vfs.fs.size[/var,free]": "vfs.fs.size.var/free",
  "vfs.fs.size[/var,pused]": "vfs.fs.size.var/pused",
  "vfs.fs.size[/var,total]": "vfs.fs.size.var/total",
  "vm.memory.size[available]": "vm.memory.size/available",
  "vm.memory.size[buffers]": "vm.memory.size/buffers",
  "vm.memory.size[cached]": "vm.memory.size/cached",
  "vm.memory.size[free]": "vm.memory.size/free",
  "vm.memory.size[total]": "vm.memory.size/total"
}

librato

こんな感じに出力されます。

説明

まず、fluent-plugin-zabbix-agentを使って、zabbix-agentからメトリクスを取得します。

<source>
  @type zabbix_agent
  @id zabbix_agent_input
  interval 60
  items_file /path/to/zabbix-items.json
  @label @raw
</source>

fluentdに流れるデータはこんな感じ。

2015-09-13 03:41:12 +0000 zabbix.item: {"key":"agent.ping","value":1}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.eth0/bytes","value":125242591}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.eth0/dropped","value":0}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.eth0/errors","value":0}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.eth0","value":125242591}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.lo/bytes","value":10815574}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.lo/dropped","value":0}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.lo/errors","value":0}
2015-09-13 03:41:12 +0000 zabbix.item: {"key":"net.if.in.lo","value":10817951}
...

次に、fluent-plugin-record-mapでlibratoに送信できるフォーマットに修正します。

<filter zabbix.**>
  @type record_map
  @id record_map_filter
  map1 key, source = record["key"].split("/", 2)
  map2 source ||= key
  map3 new_record["key"] = hostname + "." + key
  map4 new_record["source"] = source
  map5 new_record["value"] = record["value"]
</filter>

fluentdに流れるデータはこんな感じ。

2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.agent.ping","source":"agent.ping","value":1}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.eth0","source":"bytes","value":125250724}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.eth0","source":"dropped","value":0}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.eth0","source":"errors","value":0}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.eth0","source":"net.if.in.eth0","value":125250724}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.lo","source":"bytes","value":10862158}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.lo","source":"dropped","value":0}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.lo","source":"errors","value":0}
2015-09-13 03:42:39 +0000 zabbix.item: {"key":"ip-10-0-129-12.net.if.in.lo","source":"net.if.in.lo","value":10864535}
...

最後にfluent-plugin-libratoでlibratoにメトリクスを送信します。

<match zabbix.**>
  type librato
  email [e-mail]
  apikey ...
</match>