apache log x fluentd x datadog

Based on this article, we can setup apache log processor to forward data to datadog.

Mind the format setting, change ltsv => apache2

<source>
    type tail
    format apache2
    #time_format %d/%b/%Y:%H:%M:%S %z
    path /var/log/httpd/access_log
    pos_file /tmp/access_log.pos
    tag apache.access
</source>

Required plugins:

1. fluent-plugin-datacounter
2. fluent-plugin-rename-key
3. fluent-plugin-record-reformer
4. fluent-plugin-forest
5. fluent-plugin-dd

If you are using td-agent, plugin can be installed using: sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-record-reformer

Permission issue may arise, so:
chmod 0645 /var/log/httpd

Shoot, apparently we need to tweak to http status matching as well. Based on this article, we should math the code instead of status key

<match apache.access>
    type datacounter
    tag apache.status
    aggregate tag
    count_key status
    pattern1 2xx ^2\d\d$
    pattern2 3xx ^3\d\d$
    pattern3 4xx ^4\d\d$
    pattern4 5xx ^5\d\d$
</match>

Voila

Now we're talking ;)

PS:
Both fluentd and datadog are new to me