LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@rururu_kenken(nao)

docker-composeでElasticsearch/Kibana/ElastAlertを同時に起動して初回起動時にElastAlertのIndexが作成されるようにする

Posted at

対象バージョン

・Elasticsearch 7.6.2
・Kibana 7.6.2
・servercentral/elastalert:latest
　→ ElastAlert Server & ElastAlert 0.2.1

エラーメッセージ出力例

以下のようなエラーメッセージが大量に出るのと、ElastAlertのIndexがElasticsearchに作られないので、ElastAlertの再起動が必要

16:42:28.793Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:42:59.055Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:43:06.877Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.009s]
    
16:43:06.877Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:43:06.867791Z', 'to': '2020-04-27T16:43:06.867819Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
        res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
        "GET", _make_path(index, "_search"), params=params, body=body
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
        timeout=timeout,
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
        status_code, error_message, additional_info
    elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
    
16:43:29.504Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:43:59.813Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:44:06.871Z ERROR elastalert-server:
    ProcessController:  WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.007s]
    
16:44:06.874Z ERROR elastalert-server:
    ProcessController:  ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:44:06.864223Z', 'to': '2020-04-27T16:44:06.864250Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
    Traceback (most recent call last):
      File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
        res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
        "GET", _make_path(index, "_search"), params=params, body=body
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
        timeout=timeout,
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
        self._raise_error(response.status_code, raw_data)
      File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
        status_code, error_message, additional_info
    elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
    
16:44:30.100Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

再起動すると作られるが。。。
最初から作られて欲しいなぁと思う

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

16:46:38.792Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:46:38.794Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
16:46:38.795Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:46:38.804Z  INFO elastalert-server: Router:  Listening for GET request on /.
16:46:38.804Z  INFO elastalert-server: Router:  Listening for GET request on /status.
16:46:38.805Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
16:46:38.805Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
16:46:38.807Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /test.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
16:46:38.808Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for POST request on /config.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for POST request on /download.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
16:46:38.809Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
16:46:38.810Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:46:38.814Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
16:46:38.814Z  INFO elastalert-server: ProcessController:  Creating index
16:46:43.074Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.6.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
16:46:43.075Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
16:46:43.076Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
16:46:43.087Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 36)
16:46:43.091Z  INFO elastalert-server: Server:  Server listening on port 3030
16:46:43.092Z  INFO elastalert-server: Server:  Websocket listening on port 3333
16:46:43.095Z  INFO elastalert-server: Server:  Server started
16:47:06.693Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:47:37.013Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:48:07.292Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

キャプチャ.PNG

対応

Elasticsearchが接続できる状態になってからElastAlertを起動するようにする

|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules
docker-compose.yml
version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:0.2.1
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
es/config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
kibana/config/kibana.yml
server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true
Dockerfiles/Dockerfile-elastalert
FROM servercentral/elastalert:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh && chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
elastalert/bin/elastic_search_status.sh
# !/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
elastalert/bin/elastalert-start.sh
# !/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start
es/config/elasticsearch.yml
elastalert/config/api.config.json
{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "elastalert_status"
}
elastalert/config/elastalert.yml
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
# es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
# use_ssl: True

# Verify TLS certificates
# verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
# es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
# es_username: someusername
# es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

$ chmod 777 es/data
$ chmod 777 elastalert/rules
$ chmod 777 elastalert/rule_templates
$ docker-compose up -d

$ docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

16:53:28.919Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:53:28.927Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
16:53:28.931Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:53:28.942Z  INFO elastalert-server: Router:  Listening for GET request on /.
16:53:28.943Z  INFO elastalert-server: Router:  Listening for GET request on /status.
16:53:28.944Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
16:53:28.944Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
16:53:28.947Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
16:53:28.948Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
16:53:28.949Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
16:53:28.949Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
16:53:28.950Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
16:53:28.951Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
16:53:28.952Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
16:53:28.952Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
16:53:28.953Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
16:53:28.954Z  INFO elastalert-server: Router:  Listening for POST request on /test.
16:53:28.958Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
16:53:28.959Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:53:28.960Z  INFO elastalert-server: Router:  Listening for POST request on /config.
16:53:28.961Z  INFO elastalert-server: Router:  Listening for POST request on /download.
16:53:28.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
16:53:28.963Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
16:53:28.964Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
16:53:28.964Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
16:53:28.965Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
16:53:28.966Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
16:53:28.967Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
16:53:28.967Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
16:53:28.968Z  INFO elastalert-server: Router:  Listening for GET request on /config.
16:53:28.973Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
16:53:28.974Z  INFO elastalert-server: ProcessController:  Creating index
16:53:35.304Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.6.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
16:53:35.304Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
16:53:35.307Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
16:53:35.332Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 241)
16:53:35.352Z  INFO elastalert-server: Server:  Server listening on port 3030
16:53:35.353Z  INFO elastalert-server: Server:  Websocket listening on port 3333
16:53:35.358Z  INFO elastalert-server: Server:  Server started
16:53:45.843Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
16:54:16.146Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

キャプチャ.PNG

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?