Help us understand the problem. What is going on with this article?

Nginx + fluentd + elasticsearch + kibana

More than 3 years have passed since last update.

前回の続き。次は、nginx→fluentd→elasticsearch→kibanaで可視化してみる。

Nginx

install

# rpm -ivh http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
# yum -y install nginx
# nginx -V
nginx version: nginx/1.8.0

nginx.conf

  • log_formatをltsv
# vi /etc/nginx/nginx.con
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format ltsv   "time:$time_local"
                      "\thost:$remote_addr"
                      "\tforwardedfor:$http_x_forwarded_for"
                      "\treq:$request"
                      "\tstatus:$status"
                      "\tsize:$body_bytes_sent"
                      "\treferer:$http_referer"
                      "\tua:$http_user_agent"
                      "\treqtime:$request_time"
                      "\tcache:$upstream_http_x_cache"
                      "\truntime:$upstream_http_x_runtime"
                      "\tvhost:$host";

    access_log  /var/log/nginx/access.log  ltsv;

起動

# chkconfig --add nginx
# chkconfig nginx on
# /etc/init.d/nginx start

td-agent

install

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

plugin

fluent-plugin-elasticsearch

td-agent-gem install fluent-plugin-elasticsearch

fluent-plugin-woothee

td-agent-gem install fluent-plugin-woothee

td-agent.conf

# vi /etc/td-agent/td-agent.conf
## File input
<source>
  type tail
  format ltsv
  path /var/log/nginx/access.log
  pos_file /var/log/nginx/access.log.pos
  tag access.nginx
  time_key time
  time_format %d/%b/%Y:%H:%M:%S %z
</source>

## Merged ua
<match access.nginx>
  type woothee
  key_name ua
  add_prefix merged
  merge_agent_info yes
</match>

## Multiple output
<match merged.access.nginx>
  type copy
  <store>
    type elasticsearch
    index_name service_name
    type_name access
    include_tag_key true
    tag_key @log_name
    host 127.0.0.1
    port 9200
    logstash_format true
    logstash_prefix service_name.access
    flush_interval 3s
  </store>
  <store>
    type file
    path /var/log/nginx/merged.access.nginx.log
    time_slice_format %Y%m%d
    time_slice_wait 10m
    time_format %Y%m%dT%H%M%S%z
    compress gzip
  </store>
</match>

起動

# chkconfig --add td-agent
# chkconfig td-agent on
# /etc/init.d/td-agent start

kibana

  • logstash_prefixに指定した「service_name.access-*」

kibana.png

readyfor
想いをつなぎ、叶える未来を、つくる READYFORのOrganizationです
https://tech.readyfor.jp/
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away