fluentd handler for Diamond

I've been looking for an alternative of system metric collector that supports forwarding data to fluentd. I came across Brightcove/Diamond. It's python daemon that capable of collecting various metrics and also have a decent list of handlers (Graphite, Librato, so on). Fluentd was not in the list so, I forked it and added one. It was quite trivial because there's already a Fluentd client written in python (fluent-logger).

The new Fluentd handler obviously requires fluent-logger. I don't know how to add such requirement into Diamond build system, so I put it as a note.

Once added, we can use the new handler by specifying these option in diamond.conf:

[server]

# Handlers for published metrics.
handlers = diamond.handler.fluentd.FluentdHandler

[[FluentdHandler]]
host = 192.168.59.103
port = 24224
prefix_tag = diamond

Further, I want to relay those data to datadog. So I added this into td-agent.conf:

# only applicable to diamond metrics
# if you set above prefix_tag, don't forget to adjust 
# the "match" config here

<match diamond.**>
    type copy
    <store>
      type record_reformer
      tag  datadog.diamond
      enable_ruby true
      <record>
        metric ${tag_suffix[1]}
        tag    group:diamond, service:api, instance:${host}, marker:${ENV['TEST_MARKER']}
        host   fugahoge
        value  ${value}
      </record>
    </store>
</match>

<match datadog.**>
  type copy

  <store>
    type dd
    dd_api_key da7ado9da7ado9da7ado9da7ado9
    host fugahoge
  </store>

  <store>
    type stdout
  </store>
</match>

You can test it by running make run. You should see something like this in td-agent.log.

2015-02-27 07:56:45 +0000 datadog.diamond: {"timestamp":1425023805,"type":"gauge","host":"fugahoge","value":"2460704.0","metric":"diskspace.root.inodes_avail","tag":"group:diamond, service:api, instance:ubuntu-build, marker:fugahoge"}

Note: Wait for a few minutes until Diamond start collecting metrics -- it's not immediate!

Pro tips:
The easiest way to monitor server metric is by installing the datadog agent on the server you want to monitor.