5
Help us understand the problem. What are the problem?

posted at

updated at

DockerでELK基盤構築

内容

  • 下記環境を作成した際(ELK基盤の箇所)の備忘録です。
    elk-01.png

  • FESSを使用して運用で使用しているOSSツールを全文検索する。

  • ELKでSyslogサーバのログ分析を行う。

  • portainerでDockerの管理を行う。

環境

  • AlmaLinux release 8.5
  • Docker version 20.10.14

構築手順

ELKコンテナ作成

  • ダウンロード
cd ~
git clone  https://github.com/deviantony/docker-elk.git
  • 起動
docker compose -f /root/docker-elk/docker-compose.yml up -d
  • パスワード取得
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal
docker compose exec elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system
  • envファイル修正
/root/docker-elk/.env
ELASTIC_PASSWORD='**********'
LOGSTASH_INTERNAL_PASSWORD='**********'
KIBANA_SYSTEM_PASSWORD='**********'
  • コンテナrebuild
docker compose up -d logstash kibana
  • kibanaにアクセス確認
    http://xx.xx.xx.xx:5601/
    user: elastic
    password: **********

  • 下記を追加
    既にFESS用にESが動いているためポート番号変更
    /mntの箇所は後でSyslogサーバの/var/logをNFSマウントする

/root/docker-elk/compose-elk.yml
elasticsearch:
  ports:
    - "9201:9200"
    - "9301:9300"

logstash:
  volumes:
    - /mnt:/mnt:ro
  • 有償機能を無効化
/root/docker-elk/elasticsearch/config/elasticsearch.yml
xpack.license.self_generated.type: basic

systemd設定変更

/usr/lib/systemd/system/docker-compose-elk.service
ExecStart=/usr/bin/docker compose -f /root/docker-elk/docker-compose.yml up -d
systemctl daemon-reload
systemctl enable docker-compose-elk.service
systemctl restart docker-compose-elk.service

Syslogサーバ設定

yum -y install nfs-utils
systemctl enable nfs-server
systemctl start nfs-server
/etc/exports
/var/log 192.168.108.0/24(rw,no_root_squash)
systemctl restart nfs-server

Dockerホストサーバ設定

yum -y install nfs-utils
systemctl enable nfs-server
systemctl start nfs-server
mount -t nfs 192.168.108.148:/var/log /mnt
/etc/fstab
192.168.108.148:/var/log /mnt               nfs     defaults        0 0

portainer追加

  • docker systemdの設定変更
/usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:12345
systemctl daemon-reload
systemctl restart docker.service
netstat -lntp | grep docker
  • イメージ取得とテスト起動
docker pull portainer/portainer
docker run -d -p 9000:9000 --name test portainer/portainer
  • Composeから起動設定
/root/portainer/compose/portainer.yaml
services:

  portainer01:
    image: portainer/portainer
    container_name: portainer01
    ports:
      - "9000:9000"
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - data:/data

volumes:
  data:
    driver: local
/usr/lib/systemd/system/docker-compose.service
[Unit]
After=docker.service
Description=Docker-Compose
[Service]
ExecStart=/usr/bin/docker compose -f /root/portainer/compose/portainer.yaml up -d
Type=simple
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl restart docker-compose

Syslog取込設定

各種設定ファイル修正

/root/docker-elk/docker-compose.yml
下記追加
  logstash:
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      - ./logstash/patterns:/usr/share/logstash/patterns:ro,Z
      - /mnt:/mnt:ro
/root/docker-elk/logstash/Dockerfile
下記追加
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD pipeline/ /usr/share/logstash/pipeline/
ADD config/ /usr/share/logstash/config/

/root/docker-elk/logstash/config/pipelines.yml
- pipeline.id: server-syslog
  pipeline.batch.size: 125
  path.config: "/usr/share/logstash/pipeline/server-syslog.conf"
  pipeline.workers: 1
/root/docker-elk/logstash/pipeline/server-syslog.conf
input {
     file {
        path => "/mnt/Server/********.log"
        start_position => "beginning"
     }
}
filter {
    grok {
      patterns_dir => ["/usr/share/logstash/patterns"]
      match => { "message" => "%{SYSLOG_SERVER_PATTERN}" }
    }
}
output {
     elasticsearch {
         hosts => ["elasticsearch:9200"]
         index => "server-syslog-%{+YYYY-MM-dd}"
         user => "logstash_internal"
         password => "${LOGSTASH_INTERNAL_PASSWORD}"
     }
}

/root/docker-elk/logstash/patterns/server-pattern
SYSLOG_PROG [^:]+
SYSLOG_MSG (?<=: ).+

SYSLOG_SERVER_PATTERN %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOG_PROG:syslog_program}: %{SYSLOG_MSG:syslog_messages}

Logstash_internalユーザにロール追加

  • 書き込み権限がなくエラーとなってしまったため追加
  • テストのためsuperuser追加

logstash2.png

コンテナ起動・ログ確認

docker compose -f /root/docker-elk/docker-compose.yml up -d
docker logs docker-elk-logstash-1 -f

kibana設定

  • 「Stack Management」⇒「Data Views」⇒「Create data view」でdata view作成
  • 「Analytics」⇒「Discover」からデータ確認

logstash.png

Register as a new user and use Qiita more conveniently

  1. You can follow users and tags
  2. you can stock useful information
  3. You can make editorial suggestions for articles
What you can do with signing up
5
Help us understand the problem. What are the problem?