AnsibleとDockerを使用して、CentOS 7にELK(Elasticsearch, Logstash, Kibana)を構築する
作成するに至った経緯
- ELKを手動で構築するのが面倒なので、自動化したかった。
- ELKを構築するために必要なコンフィグの編集、コンテナ起動時に様々なオプションを指定などを行うのが面倒なので、自動化したかった。
対象者
- CentOS 7 にて、ELKの構築を簡単に行いたい方
前提条件
- Ansibleはインストール済みであるとする
- config、inventoryの設定は完了済みであるとする
- 鍵生成、鍵交換、疎通確認は完了済みであるとする
- 今回は、proxy環境下ではないものとする
実行環境
# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
# ansible --version
ansible 2.8.5
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /bin/ansible
python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
本playbookで行っていることの概要
- yum updateの実施
- 古いdockerをアンインストール
- 必要なパッケージのインストール
- dockerリポジトリを追加
- docker-ceをインストール
- dockerの自動起動有効化 & dockerの起動
- Elasticsearch, Logstash, Kibana, nginxの取得
- ja-JP.jsonのダウンロード
- logstash.conf, kibana.ymlの設定
- nginx, Elasticsearch, Logstash, Kibanaコンテナの起動
- filebeatのインストール, 自動起動有効化, filebeatの起動
本playbook実行後に得られるもの
- ELKが構築される
- KibanaのGUIが日本語表示になる
- filebeatがインストールされた状態になる
本playbook実行後に得られる環境のバージョン情報
項目 | Version |
---|---|
Docker | 18.06.1-ce |
Elasticsearch | 7.4.0 |
Logstash | 7.4.0 |
Kibana | 7.4.0 |
filebeat | 7.4.0 |
nginx | 1.17.4 |
本playbookの実行方法
# ansible-playbook SetupELK.yml
Elasticsearch, Logstash, Kibanaを構築するplaybook
SetupELK.yml
- name: Setup Elasticsearch, Logstash, Kibana
hosts: localhost
vars:
elk_shared_directory:
- log/nginx
- log/logstash
- elasticsearch
- elasticsearch/data/nodes
- logstash/pipeline
- kibana
- kibana/config
tasks:
- name: Upgrade all packages
yum:
name: '*'
state: latest
- name: Remove the Docker package
yum:
name:
- docker
- docker-client
- docker-client-latest
- docker-common
- docker-latest
- docker-latest-logrotate
- docker-logrotate
- docker-selinux
- docker-engine-selinux
- docker-engine
state: absent
- name: Install a list of packages
yum:
name:
- device-mapper-persistent-data
- lvm2
- yum-utils
- python-pip
state: latest
- name: Add docker repo
shell: "yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo"
args:
chdir: "/etc/yum.repos.d"
creates: docker-ce.repo
- name: Install a list of packages
yum:
name: docker-ce
state: latest
- name: Systemctl enabled docker & systemctl start docker
systemd:
name: docker
state: started
enabled: yes
- name: Install docker python package
pip:
name: docker
- name: Pull an elasticsearch image
docker_image:
name: docker.elastic.co/elasticsearch/elasticsearch
source: pull
tag: 7.4.0
- name: Pull a logstash-docker image
docker_image:
name: docker.elastic.co/logstash/logstash
source: pull
tag: 7.4.0
- name: Pull a kibana image
docker_image:
name: docker.elastic.co/kibana/kibana
source: pull
tag: 7.4.0
- name: Pull a nginx image
docker_image:
name: nginx
source: pull
tag: latest
- name: Ensure group "elk" exists
group:
name: elk
state: present
- name: Add the user 'elk' with a bash shell, appending the group 'elk' to the user's groups
user:
name: elk
shell: /bin/bash
groups: elk
append: yes
- name: Create a shared folder for ELK
file:
path: /usr/share/{{ item }}
state: directory
owner: elk
group: elk
mode: '0777'
with_items: "{{ elk_shared_directory }}"
- name: Download ja-JP.json
get_url:
url: https://github.com/elastic/kibana/raw/master/x-pack/plugins/translations/translations/ja-JP.json
dest: /usr/share/kibana
mode: '0755'
become: yes
become_user: root
- name: Insert/Update kibana.yml in /usr/share/kibana/config/kibana.yml
blockinfile:
path: /usr/share/kibana/config/kibana.yml
create: yes
regexp: '^server.name'
insertafter: '^#server.name'
block: |
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: "ja-JP"
- name: Insert/Update logstash.conf in /usr/share/logstash/pipeline/logstash.conf
blockinfile:
path: /usr/share/logstash/pipeline/logstash.conf
create: yes
regexp: '^input { '
insertafter: '^#input { '
block: |
input {
file {
path => "/var/log/nginx/access.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
remove_field => "message"
}
mutate {
add_field => { "read_timestamp" => "%{@timestamp}" }
}
date {
match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
remove_field => "[nginx][access][time]"
}
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"
}
geoip {
source => "[nginx][access][remote_ip]"
target => "[nginx][access][geoip]"
}
}
output {
elasticsearch {
hosts => [ 'elasticsearch' ]
index => "access_log1"
}
}
- name: Create a nginx container
docker_container:
name: elk_nginx
image: nginx
restart_policy: always
state: started
ports:
- 10080:80
volumes:
- /usr/share/log/nginx:/var/log/nginx
- name: Create an elasticsearch container
docker_container:
name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
restart_policy: always
state: started
ports:
- 9200:9200
- 9300:9300
volumes:
- /usr/share/elasticsearch:/usr/share/elasticsearch/data
env:
discovery.type: "single-node"
- name: Create an logstash container
docker_container:
name: logstash
image: docker.elastic.co/logstash/logstash:7.4.0
restart_policy: always
state: started
ports:
- 5044:5044
- 9600:9600
links:
- elasticsearch:elasticsearch
volumes:
- /usr/share/logstash/pipeline:/usr/share/logstash/pipeline
- /usr/share/log/logstash:/var/log
- name: Create an kibana container
docker_container:
name: kibana
image: docker.elastic.co/kibana/kibana:7.4.0
restart_policy: always
state: started
ports:
- 5601:5601
links:
- elasticsearch:elasticsearch
volumes:
- /usr/share/kibana/ja-JP.json:/usr/share/kibana/node_modules/x-pack/plugins/translations/translations/ja-JP.json
- /usr/share/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
- name: Download filebeat
get_url:
url: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-x86_64.rpm
dest: /usr/local/src
mode: '0755'
become: yes
become_user: root
- name: Install filebeat
yum:
name: /usr/local/src/filebeat-7.4.0-x86_64.rpm
state: present
- name: filebeat modules enable elasticsearch
shell: filebeat modules enable elasticsearch
- name: Waits for port 9200 of localhost to close active connections, don't start checking for 10 seconds
wait_for:
host: localhost
port: 9200
timeout: 300
delay: 10
state: drained
- name: Pause for 3 minutes to build app cache.
pause:
minutes: 3
- name: filebeat setup
shell: filebeat setup
- name: Systemctl enabled filebeat & systemctl start filebeat
systemd:
name: filebeat
state: started
enabled: yes
playbook実行時のログ
# ansible-playbook SetupELK.yml
PLAY [Setup Elasticsearch, Logstash, Kibana] **********************************************************************************************************************************************************************
TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Upgrade all packages] ***************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Remove the Docker package] **********************************************************************************************************************************************************************************
changed: [localhost]
TASK [Install a list of packages] *********************************************************************************************************************************************************************************
ok: [localhost]
TASK [Add docker repo] ********************************************************************************************************************************************************************************************
ok: [localhost]
TASK [Install a list of packages] *********************************************************************************************************************************************************************************
changed: [localhost]
TASK [Systemctl enabled docker & systemctl start docker] **********************************************************************************************************************************************************
changed: [localhost]
TASK [Install docker python package] ******************************************************************************************************************************************************************************
ok: [localhost]
TASK [Pull an elasticsearch image] ********************************************************************************************************************************************************************************
changed: [localhost]
TASK [Pull a logstash-docker image] *******************************************************************************************************************************************************************************
changed: [localhost]
TASK [Pull a kibana image] ****************************************************************************************************************************************************************************************
changed: [localhost]
TASK [Pull a nginx image] *****************************************************************************************************************************************************************************************
changed: [localhost]
TASK [Ensure group "elk" exists] **********************************************************************************************************************************************************************************
ok: [localhost]
TASK [Add the user 'elk' with a bash shell, appending the group 'elk' to the user's groups] ***********************************************************************************************************************
ok: [localhost]
TASK [Create a shared folder for ELK] *****************************************************************************************************************************************************************************
changed: [localhost] => (item=log/nginx)
changed: [localhost] => (item=log/logstash)
changed: [localhost] => (item=elasticsearch)
changed: [localhost] => (item=elasticsearch/data/nodes)
changed: [localhost] => (item=logstash/pipeline)
changed: [localhost] => (item=kibana)
changed: [localhost] => (item=kibana/config)
TASK [Download ja-JP.json] ****************************************************************************************************************************************************************************************
changed: [localhost]
TASK [Insert/Update kibana.yml in /usr/share/kibana/config/kibana.yml] ********************************************************************************************************************************************
changed: [localhost]
TASK [Insert/Update logstash.conf in /usr/share/logstash/pipeline/logstash.conf] **********************************************************************************************************************************
changed: [localhost]
TASK [Create a nginx container] ***********************************************************************************************************************************************************************************
changed: [localhost]
TASK [Create an elasticsearch container] **************************************************************************************************************************************************************************
changed: [localhost]
TASK [Create an logstash container] *******************************************************************************************************************************************************************************
changed: [localhost]
TASK [Create an kibana container] *********************************************************************************************************************************************************************************
changed: [localhost]
TASK [Download filebeat] ******************************************************************************************************************************************************************************************
changed: [localhost]
TASK [Install filebeat] *******************************************************************************************************************************************************************************************
changed: [localhost]
TASK [filebeat modules enable elasticsearch] **********************************************************************************************************************************************************************
changed: [localhost]
TASK [Waits for port 9200 of localhost to close active connections, don't start checking for 10 seconds] **********************************************************************************************************
ok: [localhost]
TASK [Pause for 3 minutes to build app cache.] ********************************************************************************************************************************************************************
Pausing for 180 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]
TASK [filebeat setup] *********************************************************************************************************************************************************************************************
changed: [localhost]
TASK [Systemctl enabled filebeat & systemctl start filebeat] ******************************************************************************************************************************************************
changed: [localhost]
PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost : ok=29 changed=20 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
コンテナの状態とイメージの確認
まずは、コンテナの状態と、イメージを確認してみましょう。
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
67080a477ab5 docker.elastic.co/kibana/kibana:7.4.0 "/usr/local/bin/dumb…" 17 minutes ago Up 17 minutes 0.0.0.0:5601->5601/tcp kibana
424e6b9642a5 docker.elastic.co/logstash/logstash:7.4.0 "/usr/local/bin/dock…" 17 minutes ago Up 17 minutes 0.0.0.0:5044->5044/tcp, 0.0.0.0:9600->9600/tcp logstash
cc6b8a1cd7fc docker.elastic.co/elasticsearch/elasticsearch:7.4.0 "/usr/local/bin/dock…" 17 minutes ago Up 17 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch
fad3c5ec3136 nginx "nginx -g 'daemon of…" 17 minutes ago Up 17 minutes 0.0.0.0:10080->80/tcp elk_nginx
# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 5a9061639d0a 5 days ago 126MB
docker.elastic.co/logstash/logstash 7.4.0 c2c1ac6b995b 3 weeks ago 889MB
docker.elastic.co/kibana/kibana 7.4.0 0328df36f79f 3 weeks ago 1.1GB
docker.elastic.co/elasticsearch/elasticsearch 7.4.0 dd156dd42341 3 weeks ago 859MB
Kibanaへのアクセス確認
- Webブラウザを起動し、[http://localhost:5601/] にアクセスします。
-
下記の順にクリックして、Kibanaのインデックスパターンを開きます。
- [管理]
- [Kibana]
- [インデックスパターン]
- [Kibana]
- [管理]
Elasticsearchへのアクセス確認
- Webブラウザを起動し、[http://localhost:9200/] にアクセスします。
- Elasticsearchのトップ画面が表示されることを確認します。
Nginxへのアクセス確認
- Webブラウザを起動し、[http://localhost:10080/] にアクセスします。
- Nginxのトップ画面が表示されることを確認します。
まとめ
AnsibleとDockerを使用して、CentOS 7にELK(Elasticsearch, Logstash, Kibana)を構築することができました。あとは、ELK関連の記事を見つつ、お試しでダミーデータをcurlコマンドで飛ばして、可視化、ダッシュボードの作成を行って、ELKの使い方と理解を深めていきたいと思います。
参考URL
今回使用したモジュール
- 【docs.ansible.com】yum – Manages packages with the yum package manager
- 【docs.ansible.com】shell – Execute shell commands on targets
- 【docs.ansible.com】systemd – Manage services
- 【docs.ansible.com】pip – Manages Python library dependencies
- 【docs.ansible.com】docker_image – Manage docker images
- 【docs.ansible.com】group – Add or remove groups
- 【docs.ansible.com】user – Manage user accounts
- 【docs.ansible.com】file – Manage files and file properties
- 【docs.ansible.com】get_url – Downloads files from HTTP, HTTPS, or FTP to node
- 【docs.ansible.com】blockinfile – Insert/update/remove a text block surrounded by marker lines
- 【docs.ansible.com】docker_container – manage docker containers
- 【docs.ansible.com】wait_for – Waits for a condition before continuing
- 【docs.ansible.com】pause – Pause playbook execution
Ansible関連
Docker, Elasticsearch, Logstash, Kibana関連
- Docker @ Elastic
- CentOS7.5に最新バージョンのDockerをインストールする
- vagrant上のcentosにansibleでdocker-ceをインストールする
- DockerでELK(Elasticsearch Logstash Kibana) を構築してみました
- Elasticsearch + KibanaをDocker Composeで動かす
- Install Elasticsearch with Dockeredit
- Running Kibana on Dockeredit
- Configuring Kibana on Dockeredit
- Specify data directory permissions for Docker containers like Elasticsearch
- Kibana の 日本語化
- KIBANA 日本語対応
- はじめての Elasticsearch
- Elasticsearch、Logstash 、Kibana、によるログの可視化 ~価値のある情報を導き出す方法のご紹介 ~