Qiita Teams that are logged in
You are not logged in to any team

Log in to Qiita Team
Community
OrganizationAdvent CalendarQiitadon (β)
Service
Qiita JobsQiita ZineQiita Blog
Help us understand the problem. What is going on with this article?

fluentd理解

More than 3 years have passed since last update.

概要

さくらのナレッジの説明が一番わかりやすかったです。
http://knowledge.sakura.ad.jp/tech/1336/

環境

  • CentOS 6.7
  • fluentd 0.12.20

Install+設定

こんな感じでansibleでinstallしました。
webserverに立てたapachのlogをlogserverで受けています。
logを送る側もlogを受ける側もどちらもtd-agentという同じserviceが動いています。
webserverが複数台になった時にlogserverにlogが集約されるので、あとはElastic searchを何か入れれば良いと思います。

  • td-agentというuserで実行しようとしてpermission deniedが発生していたので、rootで実行するように設定fileを書き換えている。

ansible-galaxyから次のroleを使わせていただきました。

  • geerlingguy.apache
  • williamyeh.fluentd
webserver.yml
# webserver playbook
- name: configure the web server
  hosts: webserver
  vars:
    - apache_listen_port: 10443
  roles:
    - geerlingguy.apache
  tasks:
    - name: copy init files
      copy: src=roles/tomcat8/files/httpd-proxy.conf dest=/etc/httpd/conf/httpd-proxy.conf owner=root group=wheel mode=0644

    - name: deploy setting file
      lineinfile: dest=/etc/httpd/conf/httpd.conf line='Include /etc/httpd/conf/httpd-proxy.conf'

    - name: restart service and auto startup setting
      service: name=httpd state=restarted

- name: configure fluentd
  hosts: webserver
  vars:
    - tdagent_conf_template: "roles/td-agent/templates/web-td-agent.conf.j2"
    - aggregator_ip: "{{ logserver.ip }}"
    - tdagent_port: "{{ logserver.tdagent_port }}"
  roles:
    - williamyeh.fluentd
  tasks:
    - name: change start user to root
      lineinfile: >
        dest=/etc/init.d/td-agent
        regexp='TD_AGENT_USER'
        line='TD_AGENT_USER=root'
      notify: td-agent restart
    - name: change start group to root
      lineinfile: >
        dest=/etc/init.d/td-agent
        regexp='TD_AGENT_GROUP'
        line='TD_AGENT_GROUP=root'
      notify: td-agent restart
  handlers:
    - name: td-agent restart                                                         
      service: name=td-agent state=restarted
logserver.yml
# logserver playbook
- name: configure fluentd
  hosts: logserver
  vars:
    - tdagent_conf_template: "roles/td-agent/templates/log-td-agent.conf.j2"
    - tdagent_port: 24224
  roles:
    - williamyeh.fluentd
roles/td-agent/templates/web-td-agent.conf.j2
####
## Source descriptions:
##

#
# Apache Log
#
## access
<source>
  type tail
  path /var/log/httpd/access_log
  tag apache.access
  pos_file /var/log/td-agent/httpd-access_log.pos
  format apache2
</source>
## error
<source>
  type tail
  path /var/log/httpd/error_log
  tag apache.error
  pos_file /var/log/td-agent/httpd-error_log.pos
  format apache_error
</source>

####
## Output descriptions:
##

#
# Apache Log
#
<match apache.**>
  type forward
  <server>
    host {{ aggregator_ip }}
    port {{ tdagent_port }}
  </server>
</match>
roles/td-agent/templates/log-td-agent.conf.j2
####
## Source descriptions:
##

#
# Recieve Logs
#
<source>
  type forward
  port {{ tdagent_port }}
</source>

####
## Output descriptions:
##
<match apache.access>
  type file
  path /var/log/td-agent/httpd/access.log
  time_slice_format %Y%m%d
  time_slice-wait 10m
  compress gzip
</match>
<match apache.error>
  type file
  path /var/log/td-agent/httpd/error.log
  time_slice_format %Y%m%d
  time_slice-wait 10m
  compress gzip
</match>

構成はこんな感じ

fluentd_structure.png

結果

webserverのapacheのlog

192.168.1.1 - - [07/Jul/2016:19:26:31 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.1.1 - - [07/Jul/2016:19:26:40 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET /icons/apache_pb.gif HTTP/1.1" 304 - "http://164.70.6.213:10443/" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"

logserverに送られたlog

/var/log/td-agent/httpd/access.log
2016-07-07T19:26:31+09:00       apache.access   {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:26:40+09:00       apache.access   {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00       apache.access   {"host":"192.168.2.3","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00       apache.access   {"host":"192.168.2.3","user":null,"method":"GET","path":"/icons/apache_pb.gif","code":304,"size":null,"referer":"http://164.70.6.213:10443/","agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}

便利ツール

clutter
hello world
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away