Elasticsearchコードリーディング 番外編: ローカルES起動Tips

コードリーディングの強い味方、デバッグ実行のために見出した小技集

公式ドキュメント: TESTING.asciidoc

起動

./gradlew run

なおデフォルトでセキュリティが有効なのでアクセスには要認証 username: 'elastic-admin', password: 'elastic-password' (gradle/run.grade に記述)

デバッグ実行

IntelliJ IDEAでデバッガを起動しておいてから

./gradlew run --debug-jvm

xpack.security 無効化

デバッグ実行時にsecurityチェック系のステップが邪魔なときに

gradle/run.gradle

 testClusters {
   runTask {
     testDistribution = System.getProperty('run.distribution', 'default')
     if (System.getProperty('run.distribution', 'default') == 'default') {
       String licenseType = System.getProperty("run.license_type", "basic")
       if (licenseType == 'trial') {
         setting 'xpack.ml.enabled', 'true'
         setting 'xpack.graph.enabled', 'true'
         setting 'xpack.watcher.enabled', 'true'
         setting 'xpack.license.self_generated.type', 'trial'
       } else if (licenseType != 'basic') {
         throw new IllegalArgumentException("Unsupported self-generated license type: [" + licenseType + "[basic] or [trial].")
       }
-      setting 'xpack.security.enabled', 'true'
+      setting 'xpack.security.enabled', 'false'
       keystore 'bootstrap.password', 'password'
       user username: 'elastic-admin', password: 'elastic-password', role: 'superuser'
     }
   }
 }

起動時にクラスタのデータをリセットさせない

./gradlew run --data-dir=/tmp/foo

なおデフォルトだと build/testclusters/runTask-0/data

ビルドのたびに消える

複数ノード起動

2020/12/23現在、まだ公式にはサポートされていなさそう https://github.com/elastic/elasticsearch/issues/51837

ひと手間かけると出来た

gradle/run.gradle

 testClusters {
   runTask {
+    numberOfNodes = 2
     testDistribution = System.getProperty('run.distribution', 'default')
     if (System.getProperty('run.distribution', 'default') == 'default') {
       String licenseType = System.getProperty("run.license_type", "basic")
       if (licenseType == 'trial') {
         setting 'xpack.ml.enabled', 'true'
         setting 'xpack.graph.enabled', 'true'
         setting 'xpack.watcher.enabled', 'true'
         setting 'xpack.license.self_generated.type', 'trial'
       } else if (licenseType != 'basic') {
         throw new IllegalArgumentException("Unsupported self-generated license type: [" + licenseType + "[basic] or [trial].")
       }
       setting 'xpack.security.enabled', 'true'
       keystore 'bootstrap.password', 'password'
       user username: 'elastic-admin', password: 'elastic-password', role: 'superuser'
     }
   }
 }

[WARN ][o.e.d.FileBasedSeedHostsProvider] [runTask-1] expected, but did not find, a dynamic hosts list at [echo "127.0.0.1:9300\n127.0.0.1:53398\n" >]
[WARN ][o.e.d.FileBasedSeedHostsProvider] [runTask-0] expected, but did not find, a dynamic hosts list at [/Users/xxxxxx/build/testclusters/runTask-0/config/unicast_hosts.txt]
[WARN ][o.e.c.c.ClusterFormationFailureHelper] [runTask-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [runTask-0, runTask-1] to bootstrap a cluster: have discovered [{runTask-0}{jh5avgLQRySNsv8hlbrYVw}{2OKFKwzfQlmfCZAWvmjNZw}{127.0.0.1}{127.0.0.1:9300}{cdhilmrstw}{ml.machine_memory=17179869184, xpack.installed=true, transform.node=true, testattr=test, ml.max_open_jobs=20}]; discovery will continue using [] from hosts providers and [{runTask-0}{jh5avgLQRySNsv8hlbrYVw}{2OKFKwzfQlmfCZAWvmjNZw}{127.0.0.1}{127.0.0.1:9300}{cdhilmrstw}{ml.machine_memory=17179869184, xpack.installed=true, transform.node=true, testattr=test, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[WARN ][o.e.c.c.ClusterFormationFailureHelper] [runTask-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [runTask-0, runTask-1] to bootstrap a cluster: have discovered [{runTask-1}{wC1-z8tpRTG60jV7r2bxrQ}{ISsboB0QQaux9nzO8m2ogg}{127.0.0.1}{127.0.0.1:53398}{cdhilmrstw}{ml.machine_memory=17179869184, xpack.installed=true, transform.node=true, testattr=test, ml.max_open_jobs=20}]; discovery will continue using [] from hosts providers and [{runTask-1}{wC1-z8tpRTG60jV7r2bxrQ}{ISsboB0QQaux9nzO8m2ogg}{127.0.0.1}{127.0.0.1:53398}{cdhilmrstw}{ml.machine_memory=17179869184, xpack.installed=true, transform.node=true, testattr=test, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

という状態になるので、

echo "127.0.0.1:9300\n127.0.0.1:53398\n" > build/testclusters/runTask-0/config/unicast_hosts.txt
echo "127.0.0.1:9300\n127.0.0.1:53398\n" > build/testclusters/runTask-1/config/unicast_hosts.txt

curl "localhost:9200/_cat/nodes?pretty"

127.0.0.1 39 99 20 2.27   cdhilmrstw - runTask-1
127.0.0.1 38 99 20 2.27   cdhilmrstw * runTask-0

curl "localhost:53400/_cat/nodes?pretty"

127.0.0.1 39 98 20 2.42   cdhilmrstw - runTask-1
127.0.0.1 37 98 20 2.42   cdhilmrstw * runTask-0

複数ノード起動でデバッグ実行

デバッガを複数プロセス起動できるなら不要なはず

IntelliJ IDEAで↑のやり方がわからず、１プロセスだけだとデバッグサーバー接続失敗でエラーになるので、強引にデバッグモードは1プロセスだけに

buildSrc/src/main/java/org/elasticsearch/gradle/testclusters/RunTask.java

     @Override
     public void beforeStart() {
         int debugPort = 5005;
         int httpPort = 9200;
         int transportPort = 9300;
         Map<String, String> additionalSettings = System.getProperties()
             .entrySet()
             .stream()
             .filter(entry -> entry.getKey().toString().startsWith(CUSTOM_SETTINGS_PREFIX))
             .collect(
                 Collectors.toMap(
                     entry -> entry.getKey().toString().substring(CUSTOM_SETTINGS_PREFIX.length()),
                     entry -> entry.getValue().toString()
                 )
             );
         boolean singleNode = getClusters().stream().flatMap(c -> c.getNodes().stream()).count() == 1;
         final Function<ElasticsearchNode, Path> getDataPath;
         if (singleNode) {
             getDataPath = n -> dataDir;
         } else {
             getDataPath = n -> dataDir.resolve(n.getName());
         }

         for (ElasticsearchCluster cluster : getClusters()) {
             cluster.getFirstNode().setHttpPort(String.valueOf(httpPort));
             httpPort++;
             cluster.getFirstNode().setTransportPort(String.valueOf(transportPort));
             transportPort++;
             for (ElasticsearchNode node : cluster.getNodes()) {
                 additionalSettings.forEach(node::setting);
                 if (dataDir != null) {
                     node.setDataPath(getDataPath.apply(node));
                 }
-                if (debug) {
+                if (debug && debugPort == 5005) {
                     logger.lifecycle("Running elasticsearch in debug mode, {} expecting running debug server on port {}", node, debugPort);
                     node.jvmArgs("-agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=" + debugPort);
                     debugPort += 1;
                 }
                 if (keystorePassword.length() > 0) {
                     node.keystorePassword(keystorePassword);
                 }
             }
         }
     }

ログレベル変更

curl -X PUT -H 'content-type: application/json' localhost:9200/_cluster/settings -d '
{
  "transient": {
    "logger.org.elasticsearch.cluster.service.MasterService": "TRACE"
  }
}
'