はじめに

Apacheログ解析のためにElasticsearchを使ってみることにした
Logstashのリファレンスの手順に従って進めた様子を、ターミナルに返ってきた出力も（割と）細かく残した備忘録にしてみた

まとめ

macOS環境で動かしてみた
Homebrewを使って簡単インストールできた
- Java8、Elasticsearch、Logstash、Kibana、Filebeat
残り容量は大事、少なくてエラーがでた
- Elasticsearchの設定で変えられるみたいだが、今回はそこまでやらなかった

つづきの記事

動作環境

MacBook Air（テスト機）とMac mini（本番機）で動かしてみた【$ uname -a】

MacBook-Air

$ uname -a
Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

Mac-mini

$ uname -a
Darwin 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 21 20:07:39 PDT 2018; root:xnu-3789.73.14~1/RELEASE_X86_64 x86_64

Java8をインストールした

Installing Logstash (6.2)を参照する
Java8が必要、Java9はサポートしてない
Mac miniにはJava8がインストールされていなかったので、Homebrew-Caskを使ってインストールした【$ brew cask install homebrew/cask-versions/java8】
Oracle のウェブサイトからダウンロードすることもできる
- https://www.oracle.com/technetwork/java/javase/downloads/index.html
- Java SE Development Kit 8u181にあるjdk-8u181-macosx-x64.dmgをダウンロードする

$ brew cask install homebrew/cask-versions/java8
==> Tapping homebrew/cask-versions
...
==> Caveats
This Cask makes minor modifications to the JRE to prevent issues with
packaged applications, as discussed here:

  https://bugs.eclipse.org/bugs/show_bug.cgi?id=411361

If your Java application still asks for JRE installation, you might need
to reboot or logout/login.

Installing java8 means you have AGREED to the license at
  https://www.oracle.com/technetwork/java/javase/terms/license/index.html

==> Satisfying dependencies
==> Downloading http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-macosx-x64.dmg
######################################################################## 100.0%
==> Verifying checksum for Cask java8
==> Installing Cask java8
==> Running installer for java8; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are ignored.
Password:  ## <-- sudoのパスワード入力を求められる
installer: Package name is JDK 8 Update 181
installer: Installing at base path /
installer: The install was successful.
🍺  java8 was successfully installed!

$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

ELKをインストールした

Elasticsearch、Logstash、Kibana（の頭文字を取ってELK）をHomebrewを使ってインストールした
- 最新版は6.3.x
- Homebrewにあるのは6.2.4だったが、これを使うことにした
Logstashのチュートリアルを読むとFilebeatも必要だったのでインストールした

Elasticsearch 6.2.4
Logstash 6.2.4
Kibana 6.2.4
Filebeat 6.2.4

$ brew install elasticsearch
$ elasticsearch -V
Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_181

$ brew install logstash
$ logstash -V
logstash 6.2.4

$ brew install kibana
$ kibana -V
6.2.4

$ brew install filebeat
$ filebeat version
filebeat version 6.2.4 (amd64), libbeat 6.2.4

公式ドキュメントはここ

Elasticsearchのトップページからだと、ドキュメントがぱっと探せなかったので、ここにメモしておく

Apacheのログ

ドキュメントに置いてあるサンプルログを使った
- https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz
- ~/Downloads/tmp/に置くことにした
- 100行のログファイル

サンプルログの行数を調べた

$ cd ~/Downloads/tmp/
$ wc -l logstash-tutorial.log
    100 logstash-tutorial.log

Apacheログは、あとで自分のものに置き換えて使う

まずはLogstashを使ってみる

Stashing Your First Event (6.2)を参照した
とりあえず、以下のコマンドを実行し、テストしてみる【$ logstash -e 'input { stdin { } } output { stdout {} }'】

$ logstash -e 'input { stdin { } } output { stdout {} }'
#### logstashオプションの説明 ####
## -e CONFIG_STRING : Use the given string as the configuration data.
## Same syntax as the config file. 
## If no input is specified, then the following is used as the default input:
##     "input { stdin { type => stdin } }"
## and if no output is specified, then the following is used as the default output:
##     "output { stdout { codec => rubydebug } }"
## If you wish to use both defaults, please use the empty string for the '-e' flag.
#### #### #### ####
## 20秒くらい待つ
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:28:31,715][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:28:31,741][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:28:31,869][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/queue"}
[2018-08-20T16:28:31,873][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/dead_letter_queue"}
[2018-08-20T16:28:32,053][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:28:32,108][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"93a06fb0-2047-4bb7-98db-4cb4fb1bf6c3", :path=>"/usr/local/Cellar/logstash/6.2.4/libexec/data/uuid"}
[2018-08-20T16:28:32,804][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:28:33,392][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:28:36,513][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[2018-08-20T16:28:36,725][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5814ebe0 run>"}
[2018-08-20T16:28:36,822][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
## 何か文字列を入力する
はじめてのlogstash　　
{
          "host" => "thibetanus",
    "@timestamp" => 2018-08-20T07:29:15.826Z,
       "message" => "はじめてのlogstash",
      "@version" => "1"
}
## Control-Cで抜け出す
^C
[2018-08-20T16:30:01,454][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2018-08-20T16:30:01,797][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x5814ebe0 run>"}

いよいよログをパースしてみる

Parsing Logs with Logstashの手順に沿って進めた
実際にログをパースする場合、input／filter／outputに分割して考える
以下の流れを書いた設定ファイルを作成する

input : Apache web logs を Filebeat を使って読み込む
filter : web logs をパースする
output : パースしたデータをElasticsearch clusterに書き出す

Filebeatを設定する

FilebeatからLogstashにログを送信する設定をする
設定ファイル名は filebeat.yml なので探す【$ mdfind filebeat.yml】
Logstash pipelineを作成する前に、設定する必要がある【$ emacs filebeat.yml】
LogstashをインストールするとBeats inputプラグインもインストールされる（らしい。実は別のことをいじっていてFilebeatをインストールしたので、よく分からない）

$ mdfind filebeat.yml
/usr/local/etc/filebeat/filebeat.yml

コピーを取って編集する

$ cd /usr/local/etc/filebeat/
$ cp filebeat.yml filebeat.yml.original
$ emacs filebeat.yml

変更箇所と内容は以下の通り

/usr/local/etc/filebeat/filebeat.yml

filebeat.prospectors:
- type: log
  # Change to true to enable this prospector configuration.
  enabled: true    # false -> true に変更
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /Users/shotakaha/Downloads/tmp/logstash-tutorial.log

# -------------------------- Elasticsearch output ------------------------------
# -> コメントアウトした
# output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

# ----------------------------- Logstash output --------------------------------
# -> コメントを外した
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Filebeatを起動

$ cd /usr/local/etc/filebeat/
$ filebeat -e -c filebeat.yml -d "publish"
#### Filebeatのオプションの説明 ####
## -e, --e        : Log to stderr and disable syslog/file output
## -c, --c string : Configuration file, relative to path.config (default "filebeat.yml")
## -d, --d string : Enable certain debug selectors
2018-08-20T16:35:45.542+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T16:35:45.543+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T16:35:45.543+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T16:35:45.546+0900	INFO	pipeline/module.go:76	Beat name: thibetanus
2018-08-20T16:35:45.546+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T16:35:45.547+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T16:35:45.547+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T16:35:45.547+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T16:35:45.548+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T16:35:45.548+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T16:35:45.548+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T16:35:45.548+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T16:35:45.550+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T16:35:45.551+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T16:35:45.551+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T16:35:45.551+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T16:35:45.551+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T16:35:45.552+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T07:35:45.551Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "hostname": "thibetanus",
    "version": "6.2.4",
    "name": "thibetanus"
  }
}
...（続）...
2018-08-20T16:35:47.563+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:35:49.565+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp 127.0.0.1:5044: connect: connection refused
2018-08-20T16:35:53.571+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:36:01.574+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp [::1]:5044: connect: connection refused
2018-08-20T16:36:15.554+0900	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":59,"time":59,"value":59},"user":{"ticks":28,"time":28}},"info":{"ephemeral_id":"c1b0fc97-b700-44ca-893b-1a95caa78444","uptime":{"ms":30036}},"memstats":{"gc_next":4194304,"memory_alloc":2367888,"memory_total":3854200,"rss":15699968}},"filebeat":{"events":{"active":100,"added":101,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":100,"filtered":1,"published":100,"retry":400,"total":101}}},"registrar":{"states":{"current":1,"update":1},"writes":2},"system":{"cpu":{"cores":4},"load":{"1":1.6953,"15":1.9009,"5":1.9126,"norm":{"1":0.4238,"15":0.4752,"5":0.4781}}}}}}
2018-08-20T16:36:17.576+0900	ERROR	pipeline/output.go:74	Failed to connect: dial tcp 127.0.0.1:5044: connect: connection refused

この段階では、Filebeatが5044ポートに接続しようとしているが、Logstashが動いていないので、エラーっぽい表示がされていても問題ない
Filebeatはこのまま動かしたままLogstashの設定を行う

Logstashを設定する

Logstashの設定ファイルを作成する
任意のディレクトリでよいみたいなので、サンプルログと同じ場所（~/Downloads/tmp/）に first-pipeline.conf を作成した

~/Downloads/tmp/first-pipeline.confを編集した

# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
# filter {
# }
output {
    stdout { codec => rubydebug }
}

設定ファイルの内容の確認

--config.test_and_exitオプションをつけて、設定ファイルの内容をチェックする

$ cd ~/Downloads/tmp/
$ logstash -f first-pipeline.conf --config.test_and_exit
####
## -f CONFIG_PATH : Load the logstash config from a specific file or directory.
## If a directory is given, all files in that directory will be concatenated
## in lexicographical order and then parsed as a single config file.
## You can also specify wildcards (globs) and any matched files will
## be loaded in the order described above.
##
## -t, --config.test_and_exit : Check configuration for valid syntax and then exit.
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:44:37,574][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:44:37,748][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:44:38,076][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-08-20T16:44:42,122][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Ignoring the 'pipelines.yml'という警告（WARN）は無視してよい
OKなようなので、--config.reload.automaticオプションに変更して実行

設定ファイル読み込みモードで内容を追記していく

--config.reload.automaticオプションをつけておくと、設定ファイルを変更した際にLogstashを再起動する必要がなくなる

$ logstash -f first-pipeline.conf --config.reload.automatic
####
## -r, --config.reload.automatic : Monitor configuration changes and reload whenever it is changed.
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:46:15,522][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:46:15,543][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:46:15,913][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:46:16,620][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:46:17,081][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:46:20,468][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T16:46:21,223][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T16:46:21,340][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xa65fcfe run>"}
[2018-08-20T16:46:21,453][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T16:46:21,653][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}

## ...（これ以降何も表示されなかった・・・）

チュートリアルによるとpiplineがきちんと設定されていれば、読み込んだログがパースされて表示されるらしいが、何も表示されなかった
/usr/local/var/lib/filebeat/registryが空っぽだったので、Filebeatを停止してからregistryを削除し、そしてまたFilebeatを再起動した

$ rm /usr/local/var/lib/filebeat/registry
$ cd /usr/local/etc/filebeat/
$ filebeat -e -c filebeat.yml -d "publish"
$ cd ~/Downloads/tmp/
$ logstash -f first-pipeline.conf --config.reload.automatic
Sending Logstashs logs to /usr/local/Cellar/logstash/6.2.4/libexec/logs which is now configured via log4j2.properties
[2018-08-20T16:49:00,055][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/fb_apache/configuration"}
[2018-08-20T16:49:00,325][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/Cellar/logstash/6.2.4/libexec/modules/netflow/configuration"}
[2018-08-20T16:49:00,651][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-20T16:49:01,635][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-20T16:49:02,218][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-20T16:49:06,834][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T16:49:07,622][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T16:49:07,757][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xf3cf3c0 run>"}
[2018-08-20T16:49:08,004][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T16:49:08,127][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
        "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
    "prospector" => {
        "type" => "log"
    },
          "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1\" 200 171717 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
        "offset" => 654,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "host" => "thibetanus",
      "@version" => "1",
    "@timestamp" => 2018-08-20T07:48:14.328Z
}
{
        "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
    "prospector" => {
        "type" => "log"
    },
          "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1\" 200 430406 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
        "offset" => 1945,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "host" => "thibetanus",
      "@version" => "1",
    "@timestamp" => 2018-08-20T07:48:14.328Z
}
...（続）...

きちんと表示されたっぽい
Filebeatから送られてくるログをLogstashで読むことができた

`Grok`フィルターを作成する

Apacheログをもっときれいに読み込めるようにフィルタを作成する
Logstash標準フィルタのひとつであるGrok Filterプラグインを使う
実は %{COMBINEDAPACHELOG} という便利なフィルターパターンがある（←当たり前か）

Apacheのログ形式（コンバイン形式）

$ head -n 1 logstash-tutorial.log
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

~/Downloads/tmp/first-pipeline.confのfilterを編集した

# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}

output {
    stdout { codec => rubydebug }
}

設定ファイルを保存するとLogstashが自動で再起動する

first-pipeline.ymlを編集したあとのLogstash出力

## Logstashが自動で再起動する
[2018-08-20T17:02:59,441][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2018-08-20T17:03:00,914][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0xf3cf3c0 run>"}
[2018-08-20T17:03:01,219][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T17:03:01,607][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T17:03:01,624][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x55669250 sleep>"}
[2018-08-20T17:03:01,625][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T17:03:01,653][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
## ここで表示が止まるので、Filebeatの再起動を行う

Filebeatのデータベース（registry）を削除してから、再実行する

Filebeatを再起動したときの出力

## Control-CでFilebeatを停止
^C
2018-08-20T17:06:44.921+0900	INFO	beater/filebeat.go:323	Stopping filebeat
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:109	Stopping Crawler
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:119	Stopping 1 prospectors
2018-08-20T17:06:44.922+0900	INFO	cfgfile/reload.go:222	Dynamic config reloader stopped
2018-08-20T17:06:44.922+0900	INFO	prospector/prospector.go:121	Prospector ticker stopped
2018-08-20T17:06:44.922+0900	INFO	prospector/prospector.go:138	Stopping Prospector: 3853306801411698003
2018-08-20T17:06:44.922+0900	INFO	crawler/crawler.go:135	Crawler stopped
2018-08-20T17:06:44.922+0900	INFO	registrar/registrar.go:239	Stopping Registrar
2018-08-20T17:06:44.922+0900	INFO	registrar/registrar.go:167	Ending Registrar
2018-08-20T17:06:44.923+0900	INFO	instance/beat.go:308	filebeat stopped.
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:132	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":89,"time":89},"total":{"ticks":192,"time":192,"value":192},"user":{"ticks":103,"time":103}},"info":{"ephemeral_id":"71f49981-8388-488d-a685-5411777868d3","uptime":{"ms":1110582}},"memstats":{"gc_next":4194304,"memory_alloc":2011024,"memory_total":27248576,"rss":18726912}},"filebeat":{"events":{"added":102,"done":102},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":100,"batches":1,"total":100},"read":{"bytes":6},"type":"logstash","write":{"bytes":6607}},"pipeline":{"clients":0,"events":{"active":0,"filtered":2,"published":100,"retry":700,"total":102},"queue":{"acked":100}}},"registrar":{"states":{"current":1,"update":102},"writes":5},"system":{"cpu":{"cores":4},"load":{"1":2.3979,"15":2.0972,"5":2.1934,"norm":{"1":0.5995,"15":0.5243,"5":0.5483}}}}}}
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:133	Uptime: 18m30.583259034s
2018-08-20T17:06:44.923+0900	INFO	[monitoring]	log/log.go:110	Stopping metrics logging.

$ rm /usr/local/var/lib/filebeat/registry

$ filebeat -e -c filebeat.yml -d "publish"
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T17:08:06.702+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T17:08:06.703+0900	INFO	pipeline/module.go:76	Beat name: thibetanus
2018-08-20T17:08:06.703+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T17:08:06.703+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T17:08:06.704+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T17:08:06.704+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T17:08:06.704+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T17:08:06.704+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T17:08:06.705+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T17:08:06.705+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T17:08:06.706+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T17:08:06.706+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T17:08:06.707+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T17:08:06.707+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:08:06.707Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "thibetanus",
    "hostname": "thibetanus",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\""
}
...（続）...

Logstashの出力

## ($logstash -f first-pipeline.conf --config.reload.automatic)
{
         "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
     "prospector" => {
        "type" => "log"
    },
           "verb" => "GET",
          "ident" => "-",
        "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "response" => "200",
           "host" => "thibetanus",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
       "@version" => "1",
        "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
           "auth" => "-",
           "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "clientip" => "83.149.9.216",
         "offset" => 325,
          "bytes" => "203023",
       "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
    "httpversion" => "1.1",
      "timestamp" => "04/Jan/2015:05:13:42 +0000",
     "@timestamp" => 2018-08-20T08:08:06.707Z
}
...（続）...

FilebeatとLogstashは別々のターミナルで起動しておいた
さっきと比べて、いろいろと追加されているのを確認した
次はさらにGeoIPフィルターも追加してみる

`GeoIP`フィルターの追加

first-pipeline.confのfilterをさらに編集した

# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
}

設定ファイル（first-pipeline.conf）を編集した
Filebeatを停止した
データベース（registry）を削除した
Filebeatの再起動した
（Logstashはreload.automaticしていたので、再起動せずともきちんと動いた）

Filebeatを再起動したときの出力

$ rm /usr/local/var/lib/filebeat/registry
$ filebeat -e -c filebeat.yml -d "publish"
## ...（省略）...
2018-08-20T17:13:28.651+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:13:28.651Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "thibetanus",
    "hostname": "thibetanus",
    "version": "6.2.4"
  }
}
## ...（続）...

Logstashの出力

## (logstash -f first-pipeline.conf --config.reload.automatic)
## ...（省略）...
{
         "source" => "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
     "prospector" => {
        "type" => "log"
    },
           "verb" => "GET",
          "ident" => "-",
        "message" => "83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] \"GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1\" 200 7697 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "response" => "200",
           "host" => "thibetanus",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
       "@version" => "1",
        "request" => "/presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js",
          "geoip" => {
           "region_name" => "Moscow",
             "longitude" => 37.6184,
         "country_code3" => "RU",
              "timezone" => "Europe/Moscow",
             "city_name" => "Moscow",
          "country_name" => "Russia",
           "postal_code" => "101194",
                    "ip" => "83.149.9.216",
              "latitude" => 55.7485,
           "region_code" => "MOW",
        "continent_code" => "EU",
         "country_code2" => "RU",
              "location" => {
            "lat" => 55.7485,
            "lon" => 37.6184
        }
    },
           "auth" => "-",
           "beat" => {
            "name" => "thibetanus",
         "version" => "6.2.4",
        "hostname" => "thibetanus"
    },
       "clientip" => "83.149.9.216",
         "offset" => 1304,
          "bytes" => "7697",
       "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
    "httpversion" => "1.1",
      "timestamp" => "04/Jan/2015:05:13:44 +0000",
     "@timestamp" => 2018-08-20T08:13:28.651Z
}
...（続）...

GeoIPの項目が、さらにいろいろ追加されていることを確認した

Elasticsearchに書き出す

outputを標準出力からelasticsearchに変更する

~/Downloads/tmp/first-pipeline.confのoutputを編集した

# The # character at the beginning of a line indicates a comment.
# Use comments to describe your configuration.
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that
# it is optional.
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Logstashの出力

## ($ logstash -f first-pipeline.conf --config.reload.automatic)
[2018-08-20T17:19:27,401][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:main}
[2018-08-20T17:19:32,420][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{"other"=>[{"thread_id"=>93, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.13-java/lib/logstash/inputs/beats.rb:198:in `run'"}], ["LogStash::Filters::GeoIP", {"source"=>"clientip", "id"=>"d22d425d24614b7079e7eb5dfcb5b07c15d270877a3ee1bc69819bd0a41ef389"}]=>[{"thread_id"=>89, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>90, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>91, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}, {"thread_id"=>92, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}]}}
[2018-08-20T17:19:32,421][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2018-08-20T17:19:33,847][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x6a659349 run>"}
[2018-08-20T17:19:34,106][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-20T17:19:35,207][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-08-20T17:19:35,226][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:35,558][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:35,585][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-08-20T17:19:35,628][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in `read_template_file'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `get_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:in `install_template'", "/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/output_delegator.rb:42:in `register'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:342:in `register_plugin'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:353:in `register_plugins'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:730:in `maybe_setup_out_plugins'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:363:in `start_workers'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:290:in `run'", "/usr/local/Cellar/logstash/6.2.4/libexec/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"]}
[2018-08-20T17:19:35,653][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-08-20T17:19:35,700][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/local/Cellar/logstash/6.2.4/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-08-20T17:19:35,732][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-20T17:19:35,751][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-20T17:19:35,759][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x62aa0dd4 run>"}
[2018-08-20T17:19:35,766][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-08-20T17:19:40,601][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:40,617][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:45,628][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:45,633][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:50,641][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:50,644][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-08-20T17:19:55,652][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:19:55,656][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

## ...（エラーが続く）...

なんか警告（WARN）が出たので確認した
Ignoring the 'pipeline.yml' ...は無視する
Running health check to see if an Elasticsearch connection is workingは無視してはいけなさそう
Elasticsearchを起動した

elasticsearchを起動

$ elasticsearch
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory

[2018-08-20T17:22:46,943][INFO ][o.e.n.Node               ] [] initializing ...
[2018-08-20T17:22:47,031][INFO ][o.e.e.NodeEnvironment    ] [bOhJS94] using [1] data paths, mounts [[/ (/dev/disk2)]], net usable_space [1.3tb], net total_space [1.9tb], types [hfs]
[2018-08-20T17:22:47,031][INFO ][o.e.e.NodeEnvironment    ] [bOhJS94] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] node name [bOhJS94] derived from node ID [bOhJS94JTEO6dkZgOfbX8g]; set [node.name] to override
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] version[6.2.4], pid[12778], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Mac OS X/10.12.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-08-20T17:22:47,033][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/k8/x_r4b8q52fg6_fv_7z32kj040000gn/T/elasticsearch.WbkW7qkz, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/local/Cellar/elasticsearch/6.2.4/libexec, -Des.path.conf=/usr/local/etc/elasticsearch]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [aggs-matrix-stats]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [analysis-common]
[2018-08-20T17:22:47,877][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [ingest-common]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-expression]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-mustache]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [lang-painless]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [mapper-extras]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [parent-join]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [percolator]
[2018-08-20T17:22:47,878][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [rank-eval]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [reindex]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [repository-url]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [transport-netty4]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] loaded module [tribe]
[2018-08-20T17:22:47,879][INFO ][o.e.p.PluginsService     ] [bOhJS94] no plugins loaded
[2018-08-20T17:22:51,601][INFO ][o.e.d.DiscoveryModule    ] [bOhJS94] using discovery type [zen]
[2018-08-20T17:22:52,210][INFO ][o.e.n.Node               ] initialized
[2018-08-20T17:22:52,211][INFO ][o.e.n.Node               ] [bOhJS94] starting ...
[2018-08-20T17:22:52,389][INFO ][o.e.t.TransportService   ] [bOhJS94] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-08-20T17:22:55,447][INFO ][o.e.c.s.MasterService    ] [bOhJS94] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300}
[2018-08-20T17:22:55,451][INFO ][o.e.c.s.ClusterApplierService] [bOhJS94] new_master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {bOhJS94}{bOhJS94JTEO6dkZgOfbX8g}{ymM_Ll5vQ522PDzmHujc-g}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-08-20T17:22:55,475][INFO ][o.e.h.n.Netty4HttpServerTransport] [bOhJS94] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-08-20T17:22:55,475][INFO ][o.e.n.Node               ] [bOhJS94] started
[2018-08-20T17:22:55,477][INFO ][o.e.g.GatewayService     ] [bOhJS94] recovered [0] indices into cluster_state

## MBAの場合
[2018-08-15T17:04:22,275][INFO ][o.e.g.GatewayService     ] [RYRm5T8] recovered [3] indices into cluster_state
[2018-08-15T17:04:23,160][INFO ][o.e.c.r.a.AllocationService] [RYRm5T8] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2018.08.14][3], [.kibana][0]] ...]).

Logstashに追記された出力

[2018-08-20T17:22:57,233][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-20T17:22:57,420][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-08-20T17:22:57,500][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-08-20T17:22:57,509][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}

さっきの警告（WARN）がなくなった
別の警告（WARN）が出ているけれどとりあえず無視する
Filebeatを再起動する

Filebeatを再起動したときの出力

$ rm /usr/local/var/lib/filebeat/registry
$ filebeat -e -c filebeat.yml -d "publish"
2018-08-20T17:30:02.132+0900	INFO	instance/beat.go:468	Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2018-08-20T17:30:02.133+0900	INFO	instance/beat.go:475	Beat UUID: 9a84ae5b-db57-4f01-a7a7-6abe9547c00c
2018-08-20T17:30:02.133+0900	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-08-20T17:30:02.134+0900	INFO	pipeline/module.go:76	Beat name: tora.kek.jp
2018-08-20T17:30:02.135+0900	ERROR	fileset/modules.go:95	Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2018-08-20T17:30:02.135+0900	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-08-20T17:30:02.136+0900	INFO	instance/beat.go:301	filebeat start running.
2018-08-20T17:30:02.136+0900	INFO	registrar/registrar.go:73	No registry file found under: /usr/local/var/lib/filebeat/registry. Creating a new registry file.
2018-08-20T17:30:02.138+0900	INFO	registrar/registrar.go:110	Loading registrar data from /usr/local/var/lib/filebeat/registry
2018-08-20T17:30:02.138+0900	INFO	registrar/registrar.go:121	States Loaded from registrar: 0
2018-08-20T17:30:02.138+0900	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-20T17:30:02.138+0900	INFO	crawler/crawler.go:48	Loading Prospectors: 1
2018-08-20T17:30:02.141+0900	INFO	log/prospector.go:111	Configured paths: [/Users/shotakaha/Downloads/tmp/logstash-tutorial.log]
2018-08-20T17:30:02.142+0900	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 1
2018-08-20T17:30:02.142+0900	INFO	cfgfile/reload.go:127	Config reloader started
2018-08-20T17:30:02.142+0900	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-08-20T17:30:02.142+0900	INFO	log/harvester.go:216	Harvester started for file: /Users/shotakaha/Downloads/tmp/logstash-tutorial.log
2018-08-20T17:30:02.143+0900	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-20T08:30:02.142Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "hostname": "tora.kek.jp",
    "version": "6.2.4",
    "name": "tora.kek.jp"
  },
  "source": "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
  "offset": 325,
  "message": "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  "prospector": {
    "type": "log"
  }
}
...（続）...

Elasticsearchに追記された出力

[2018-08-20T17:30:03,707][INFO ][o.e.c.m.MetaDataCreateIndexService] [bOhJS94] [logstash-2018.08.20] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-08-20T17:30:04,107][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] create_mapping [doc]
[2018-08-20T17:30:04,127][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] update_mapping [doc]
[2018-08-20T17:30:04,281][INFO ][o.e.c.m.MetaDataMappingService] [bOhJS94] [logstash-2018.08.20/Cr1beaUFT1KDp3uC2YCrMw] update_mapping [doc]

ちゃんと動いているかを確認する

日付を確認する

$ curl -XGET 'localhost:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2018.08.20 Cr1beaUFT1KDp3uC2YCrMw   5   1        100            0    380.9kb        380.9kb

レスポンスを取ってくる

$ curl -XGET 'localhost:9200/logstash-2018.08.20/_search?pretty&q=response=200'
{
  "took" : 24,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 98,
    "max_score" : 2.5306559,
    "hits" : [
      {
        "_index" : "logstash-2018.08.20",
        "_type" : "doc",
        "_id" : "waF0VmUBuPIQbAmZjlw9",
        "_score" : 2.5306559,
        "_source" : {
          "source" : "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
          "prospector" : {
            "type" : "log"
          },
          "verb" : "GET",
          "ident" : "-",
          "message" : "83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] \"GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1\" 200 52878 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "response" : "200",
          "host" : "tora.kek.jp",
          "agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "@version" : "1",
          "request" : "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
          "geoip" : {
            "region_name" : "Moscow",
            "longitude" : 37.6184,
            "country_code3" : "RU",
            "timezone" : "Europe/Moscow",
            "city_name" : "Moscow",
            "country_name" : "Russia",
            "postal_code" : "101194",
            "ip" : "83.149.9.216",
            "latitude" : 55.7485,
            "region_code" : "MOW",
            "continent_code" : "EU",
            "country_code2" : "RU",
            "location" : {
              "lat" : 55.7485,
              "lon" : 37.6184
            }
          },
          "auth" : "-",
          "beat" : {
            "name" : "tora.kek.jp",
            "version" : "6.2.4",
            "hostname" : "tora.kek.jp"
          },
          "clientip" : "83.149.9.216",
          "offset" : 2932,
          "bytes" : "52878",
          "referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
          "httpversion" : "1.1",
          "timestamp" : "04/Jan/2015:05:13:45 +0000",
          "@timestamp" : "2018-08-20T08:30:02.143Z"
        }
      },
      {
        "_index" : "logstash-2018.08.20",
        "_type" : "doc",
        "_id" : "rKF0VmUBuPIQbAmZjlwy",
        "_score" : 0.09646662,
        "_source" : {
          "source" : "/Users/shotakaha/Downloads/tmp/logstash-tutorial.log",
          "prospector" : {
            "type" : "log"
          },
          "verb" : "GET",
          "ident" : "-",
          "message" : "200.49.190.101 - - [04/Jan/2015:05:17:39 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"-\" \"-\"",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "response" : "200",
          "host" : "tora.kek.jp",
          "agent" : "\"-\"",
          "@version" : "1",
          "request" : "/reset.css",
          "geoip" : {
            "region_name" : "Departamento de Huehuetenango",
            "longitude" : -91.3497,
            "country_code3" : "GT",
            "timezone" : "America/Guatemala",
            "country_name" : "Guatemala",
            "ip" : "200.49.190.101",
            "latitude" : 15.4731,
            "region_code" : "HU",
            "continent_code" : "NA",
            "country_code2" : "GT",
            "location" : {
              "lat" : 15.4731,
              "lon" : -91.3497
            }
          },
          "auth" : "-",
          "beat" : {
            "name" : "tora.kek.jp",
            "version" : "6.2.4",
            "hostname" : "tora.kek.jp"
          },
          "clientip" : "200.49.190.101",
          "offset" : 12114,
          "bytes" : "1015",
          "referrer" : "\"-\"",
          "httpversion" : "1.1",
          "timestamp" : "04/Jan/2015:05:17:39 +0000",
          "@timestamp" : "2018-08-20T08:30:02.145Z"
        }
      },
...（続）...

やっていることがよくわからないけれど、なんだかちゃんと動いてそう
ターミナルでcurlコマンドを使ったが、ブラウザでも確認できる

Kibanaで確認した

Kibanaを起動する

Kibanaの出力

$ kibana
  log   [08:40:40.174] [info][status][plugin:kibana@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.224] [info][status][plugin:elasticsearch@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [08:40:40.422] [info][status][plugin:timelion@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.428] [info][status][plugin:console@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.433] [info][status][plugin:metrics@6.2.4] Status changed from uninitialized to green - Ready
  log   [08:40:40.454] [info][listening] Server running at http://localhost:5601
  log   [08:40:40.507] [info][status][plugin:elasticsearch@6.2.4] Status changed from yellow to green - Ready

ここに書いてあるlocalhost:5601をブラウザで開く
とりあえずDiscoverを開く
横軸@timestamp、縦軸countの棒グラフが表示された
- 別の日に開いてみたらNo results foundと表示された
- 期間がLast 15 minutsになっていたため、右上の時計マークのところをクリックしてThis weekに変更した

@timestampが、ログファイルを読み込んだ日時になっているためフィルタ時に変更する

残り容量の少ないPCでエラーがでた

MacBook AirのHD容量が5%を切っていたため、エラーに悩まされた・・・
解決法は見つかったが、試していない。放置状態

エラーがでた

Elasticsearchを起動したターミナルと、Logstashを起動したターミナルの出力にもエラーが出ていた

Elasticsearchのエラー

[2018-08-15T17:04:52,065][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
[2018-08-15T17:05:22,099][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
[2018-08-15T17:05:52,131][WARN ][o.e.c.r.a.DiskThresholdMonitor] [RYRm5T8] flood stage disk watermark [95%] exceeded on [RYRm5T8DQI2Yz1xF7-PdiQ][RYRm5T8][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.7gb[3.3%], all indices on this node will marked read-only
...（続）...

Logstashのエラー

[2018-08-15T17:09:51,873][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-08-15T17:09:51,873][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-08-15T17:09:51,881][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
...（続）...

エラーについて調べてみた

なんだかストレージ不足が原因みたい