目的

Redmineのパフォーマンスダウン、エラーなどを監視する。

前提条件

Redmineサーバ

オンプレ
Redmine 3.4.6
nginx+unicornのリバースプロキシ構成のため、クライアントの解析は他のコンテンツも含めてnginxのログで行う。
logstash-5.6.13

Elasticsearch/Kibana

AmazonES
Elasticsearch のバージョン6.3
アクセスポリシー：「特定のIPからのドメインへのアクセスを許可」として会社のゲートウェイIPを指定

監視対象

/var/log/nginx/access.log
/usr/local/redmine/log/production.log

production.logの解析対象は処理時間、ステータスコードを含む３パターンとした。

  Rendered issues/index.html.erb within layouts/base (54.0ms)
Completed 200 OK in 31ms (Views: 1.2ms | ActiveRecord: 8.1ms)
Completed 302 Found in 487ms (ActiveRecord: 465.0ms)

Logstashの設定

/etc/logstash/conf.d/input.conf

input {
  file {
    path => ["/var/log/nginx/access.log*"]
    start_position => beginning
    type => "nginx"
    exclude => ["*.gz"]
  }

  file {
    path => ["/usr/local/redmine/log/production.log"]
    start_position => beginning
    type => "redmine"
  }
}

/etc/logstash/conf.d/filter.conf

filter {
  if [type] == "nginx" {
    grok {
      match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:forwarder} %{NUMBER:responsetime}"]
      overwrite => [ "message" ]
    }
    date {
      match=>["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
      locale => us
      remove_field => [ "timestamp" ]
    }

    mutate {
      convert => ["response", "integer"]
      convert => ["bytes", "integer"]
      convert => ["responsetime", "float"]
      gsub => ["referrer","\"",""]
      gsub => ["agent","\"",""]
      gsub => ["forwarder","\"",""]
    }

    grok {
      match => { "request" => "^/%{WORD:first_path}/%{GREEDYDATA}$" }
      tag_on_failure => ["_request_parse_failure"]
    }

    if "_request_parse_failure" in [tags] {
      mutate {
        replace => {"first_path" => "/"}
      }
    } else {
      mutate {
        replace => {"first_path" => "/%{first_path}" }
      }
    }

    useragent {
      source => "agent"
    }
  }

  if [type] == "redmine" {
    grok {
      match => {
        "message" => [
          "  Rendered %{GREEDYDATA:access_path} .*\(%{NUMBER:rendering_time}ms\)",
          "Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(Views: %{NUMBER:view_time}ms \| ActiveRecord: %{NUMBER:active_record_time}ms\)",
          "Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(ActiveRecord: %{NUMBER:active_record_time}ms\)"
        ]
      }
    }

    mutate {
      convert => ["rendering_time", "float"]
      convert => ["total_time", "float"]
      convert => ["view_time", "float"]
      convert => ["active_record_time", "float"]
    }
  }
}

/etc/logstash/conf.d/output.conf

output {

  if "_grokparsefailure" not in [tags] {

    if [type] == "nginx" {
      elasticsearch {
        hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
        ssl => true
        index => "nginx-%{+YYYY.MM.dd}"
      }
    }

    if [type] == "redmine" {
      elasticsearch {
        hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
        ssl => true
        index => "redmine-%{+YYYY.MM.dd}"
      }
    }
  }
}

既存の流用のため、AmazonESプラグインは使用しなかった。
ElasticsearchエンドポイントのFQDN＋ポート443、sslの使用を設定する。

Kibanaの設定

Redmine_millisec

Index:redmine-*
Type:Line
Y-Axis
Aggregation:Max
Field:total_time
Y-Axis
Aggregation:Max
Field:active_record_time
Y-Axis
Aggregation:Max
Field:rendering_time
Y-Axis
Aggregation:Max
Field:view_time
X-Axis
Aggregation:Date Histogram
Field: @￰timestamp
Interval:Hourly

Redmine_err_count

※あらかじめ、「response.keyword:/(4|5).*/」をSearchとして登録

Index:redmine-*
Type:Area
Y-Axis
Aggregation:Count
X-Axis
Aggregation:Date Histogram
Field: @￰timestamp
Interval:Hourly
Split Series
Sub Aggregation:Terms
Field:response.keyword
Order By:Term
Order:Ascending
Size:10