dockerで起動したelasticsearchのsnapshotをs3に保存する方法

目次

• dockerfile
• docker-composefile
• aws認証設定
• snapshotrepositoryの登録
• snapshotの作成

dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.6.0
RUN bin/elasticsearch-plugin install --batch repository-s3

docker-composefile

version: '3.8'

services:
  localstack:
    container_name: localstack
    image: localstack/localstack:0.13.3
    environment:
      SERVICES: s3
      EDGE_PORT: 4566
    ports:
      - '4566:4566'

  es01:
    build: ./elasticsearch
    volumes:
      - /usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - '9200:9200'

aws認証設定

コンテナ内部に入る

docker container exec -it container_name bash

アクセスキーとシークレットキーを登録し確認

elasticsearch-keystore add s3.client.default.access_key

elasticsearch-keystore add s3.client.default.secret_key

bin/elasticsearch-keystore list

ノードにあるkeystoreをリロード

curl -X POST -v "localhost:9200/_nodes/reload_secure_settings" 

snapshotrepositoryの登録

まずs3にバケットを作成したのちに、下記のようなリクエストを送る

registerrepository.py

import requests

url = 'http://localhost:9200/_snapshot/snapshotrepository_name'
headers ={'Content-Type': 'application/json'}
payload = {
  "type": "s3",
  "settings": {
    "bucket": "bucket_name",
    "endpoint" : "endpoint_name",
    "protocol": "http",
    "path_style_access": "true"
  }
}
response = requests.put(url,headers=headers,json=payload)
print(response.text)

snapshotの作成

makesnapshot.py

import requests

url = 'http://localhost:9200/_snapshot/repository_name/snapshot_name'
response = requests.put(url)

print(response.text)

備考

keystoreのリロードを行わないと下記のようなエラーになる

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository_name] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[repository_name] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-rQf2zvkRSFmXhH4i52mbcg/master.dat] using a single upload","caused_by":{"type":"sdk_client_exception","reason":"Unable to load credentials from service endpoint","caused_by":{"type":"connect_exception","reason":"Connection refused"}}}},"status":500}