Edited at

Redmineのproduction.logを監視する


目的

Redmineのパフォーマンスダウン、エラーなどを監視する。


前提条件


Redmineサーバ


  • オンプレ


    • Redmine 3.4.6

    • nginx+unicornのリバースプロキシ構成のため、クライアントの解析は他のコンテンツも含めてnginxのログで行う。



  • logstash-5.6.13


Elasticsearch/Kibana


  • AmazonES


    • Elasticsearch のバージョン6.3

    • アクセスポリシー:「特定のIPからのドメインへのアクセスを許可」として会社のゲートウェイIPを指定




監視対象

/var/log/nginx/access.log

/usr/local/redmine/log/production.log

production.logの解析対象は処理時間、ステータスコードを含む3パターンとした。

  Rendered issues/index.html.erb within layouts/base (54.0ms)

Completed 200 OK in 31ms (Views: 1.2ms | ActiveRecord: 8.1ms)
Completed 302 Found in 487ms (ActiveRecord: 465.0ms)


Logstashの設定


/etc/logstash/conf.d/input.conf

input {

file {
path => ["/var/log/nginx/access.log*"]
start_position => beginning
type => "nginx"
exclude => ["*.gz"]
}

file {
path => ["/usr/local/redmine/log/production.log"]
start_position => beginning
type => "redmine"
}
}



/etc/logstash/conf.d/filter.conf

filter {

if [type] == "nginx" {
grok {
match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:forwarder} %{NUMBER:responsetime}"]
overwrite => [ "message" ]
}
date {
match=>["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
locale => us
remove_field => [ "timestamp" ]
}

mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
gsub => ["referrer","\"",""]
gsub => ["agent","\"",""]
gsub => ["forwarder","\"",""]
}

grok {
match => { "request" => "^/%{WORD:first_path}/%{GREEDYDATA}$" }
tag_on_failure => ["_request_parse_failure"]
}

if "_request_parse_failure" in [tags] {
mutate {
replace => {"first_path" => "/"}
}
} else {
mutate {
replace => {"first_path" => "/%{first_path}" }
}
}

useragent {
source => "agent"
}
}

if [type] == "redmine" {
grok {
match => {
"message" => [
" Rendered %{GREEDYDATA:access_path} .*\(%{NUMBER:rendering_time}ms\)",
"Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(Views: %{NUMBER:view_time}ms \| ActiveRecord: %{NUMBER:active_record_time}ms\)",
"Completed %{NUMBER:response}.*in %{NUMBER:total_time}ms \(ActiveRecord: %{NUMBER:active_record_time}ms\)"
]
}
}

mutate {
convert => ["rendering_time", "float"]
convert => ["total_time", "float"]
convert => ["view_time", "float"]
convert => ["active_record_time", "float"]
}
}
}



/etc/logstash/conf.d/output.conf

output {

if "_grokparsefailure" not in [tags] {

if [type] == "nginx" {
elasticsearch {
hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
ssl => true
index => "nginx-%{+YYYY.MM.dd}"
}
}

if [type] == "redmine" {
elasticsearch {
hosts => ["search-XXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443"]
ssl => true
index => "redmine-%{+YYYY.MM.dd}"
}
}
}
}


既存の流用のため、AmazonESプラグインは使用しなかった。

ElasticsearchエンドポイントのFQDN+ポート443、sslの使用を設定する。


Kibanaの設定


Redmine_millisec


  • Index:redmine-*

  • Type:Line

  • Y-Axis


    • Aggregation:Max

    • Field:total_time



  • Y-Axis


    • Aggregation:Max

    • Field:active_record_time



  • Y-Axis


    • Aggregation:Max

    • Field:rendering_time



  • Y-Axis


    • Aggregation:Max

    • Field:view_time



  • X-Axis


    • Aggregation:Date Histogram

    • Field: @￰timestamp

    • Interval:Hourly




Redmine_err_count

※あらかじめ、「response.keyword:/(4|5).*/」をSearchとして登録


  • Index:redmine-*

  • Type:Area

  • Y-Axis


    • Aggregation:Count



  • X-Axis


    • Aggregation:Date Histogram

    • Field: @￰timestamp

    • Interval:Hourly



  • Split Series


    • Sub Aggregation:Terms

    • Field:response.keyword

    • Order By:Term

    • Order:Ascending

    • Size:10