はじめに
とりとめなく書いてます。Elastic Stack新機能シリーズです。
書いたは良いけど、未リリースものばかりなので、その後の修正で変わっちゃったりすると悲しいですね・・・
ま、それはともかく。
今回はUpgrade Assistantを使った6.x系から7.x系へのアップグレードがお題です。
（2019.03.14 追記あり）
(2019.03.27 さらに追記あり）
Upgrade Assistantとは
5.6系での実装
Upgrade Assistant自体は、古くは5.6系で導入されてます。この時は、設定の確認と指摘、システムが使うIndex（.kibanaとか）の確認とReindexをしてくれてました。
6.6での実装
時は流れて6.6でも、7.0系に向けたUpgrade Assistant機能が実装されました。
ここでも、5.6系と同じように設定に関する部分と、Indexのチェックが行われます。
上の画像で分かる通り、5.6系までに作られたIndexだよ、という指摘はしてくれますが、これらは手動でReindexが必要となります。
5.6以前のElasticsearchって結構導入実績ありますよね。それを全部Reindexか・・・と思うと気が重くなります。
6.7での実装
で、6.7です。
Reindexしてくれる機能が実装されました。
6.6の場合と違って、左側に[Reindex]というボタンが表示されてるのがわかります。
じゃ、これをポチポチ押していけば良いわけですね。早速試してみましょう。
Upgrade Assistantを試す
パターン1
ごく普通のIndexを試してみます。
こんなんを、
PUT /my_index3
{
"mappings": {
"user": {
"_all": {
"enabled": true
}
}
}
}
5.6系で作ってから、6.7系にもっていって、Upgrade Assistantにかけてみると、
「_allは削除されるよ」と。こんな感じに、7.0以降で使えなくなる機能は指摘してくれます。画面のようにチェックを入れて実行すると、
無事に移行が終わります（画面右下に「Done」って出てますよね）。
この程度は問題なさそうです。
パターン2
もちょっと違うものもやってみます。
PUT my_index5
{
"mappings": {
"my_type": {
"properties": {
"yes_or_no": {
"type": "boolean"
}
}
}
}
}
booleanのフィールドがあります。同じように5.6系で作成⇒6.7系でUpgrade Assistantにかけてみます。
_allだけでなく、_sourceにはいるbooleanの値が変わるよ！という指摘を受けます。
これもチェックして進めると、
無事に移行が終わります。
なんか、大丈夫そうですね。
パターン3
さらにやってみます。
PUT my_index2
{
"mappings": {
"my_type": {
"properties": {
"title": {
"type": "text"
},
"content": {
"type": "text"
},
"date": {
"type": "date",
"include_in_all": false
}
}
}
}
}
今度のはinclude_in_allが入っています。これをかけてみると・・・
とりあえず、_allは削除されるよという指摘を受けます。チェックを入れて進めると・・・
おっと、新しいIndexを作成してる時にエラーが出ました。
[mapper_parsing_exception] [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.
6.0.0以降はinclude_in_allを付けたIndexは作れませんよ、と。Upgrade Assistantでは、これ以上進めることはできません。
これを解消するには、元のmy_index2のMappingを修正してinclude_in_allを削除するか、自分でReindexするしかありません。
パターン4
さらにやってみます。
PUT my_index
{
"mappings": {
"my_parent": {},
"my_child": {
"_parent": {
"type": "my_parent"
}
}
}
}
これは明らかにダメなパターンで、そもそもReindexをしてくれません。
青で囲んだところに書いてあります。
This index has more than one mapping type, which is not supported in 7.0. This index must be reindexed into one or more single-type indices. Mapping types in use: [my_child, my_parent]
このパターンのIndexを持っている人は、自分で何とかするしかありません（Elastic 2.xで作ってた人は、結構このパターンあるかな？）。
パターン5
ちょっと変なパターンです。
もととなるIndexはこんな感じ。
PUT my_index4
{
"settings": {
"number_of_shards": 5
}
}
なんの変哲もないですね。
これを、6.7以降でShrinkして、それをUpgradeしようとすると・・・
できそうなんですが・・・
エラーが出ます。
[illegal_argument_exception] unknown setting [index.resize.source.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings :: {"path":"/reindexed-v6-my_index4_shrinked","query":{}
index.resize.souce.nameなんて知らんぞ！
GET my_index4_shrinked/_settings
{
"my_index4_shrinked" : {
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"initial_recovery" : {
"_id" : "C8onVg6uS5Cy9Zd-bvunwA"
}
}
},
"allocation" : {
"max_retries" : "1"
},
"number_of_shards" : "1",
"routing_partition_size" : "1",
"shrink" : {
"source" : {
"name" : "my_index4",
"uuid" : "H5MKwbBUQtCyWd2Kl6mVZA"
}
},
"provided_name" : "my_index4_shrinked",
"resize" : {
"source" : {
"name" : "my_index4",
"uuid" : "H5MKwbBUQtCyWd2Kl6mVZA"
}
},
"creation_date" : "1550458731700",
"number_of_replicas" : "1",
"uuid" : "pbI5MsGnReCGboODcwJ9aQ",
"version" : {
"created" : "5061499",
"upgraded" : "6070099"
}
}
}
}
}
なんか見慣れない設定がいくつかあります。前出のindex.resizeだけでなくindex.shrinkも。
どうも、6.7以降はShrinkするとこいつらがつけられるようです。おまけに消せない。
こうなると、もうUpgrade Assistantは使えないので、自分でReindexするしかありません。
おまけ
その1
Upgrade Assistantで実行した結果は、.kibanaの中でtypeフィールドの値がupgrade-assistant-reindex-operationのものの中に入ってます。
その2
6.0～6.6と違って、7.x系（7.0ブランチ、7.xブランチ）、8.0系（masterブランチ）も変わらずUpgrade Assistantが提供されています。
このまま提供され続けて、機能もアップしてくれると良いですね。
さいごに
これあると、だいぶ移行がしやすいのかともいますが、注意が必要な部分もあります。
移行に当たっては、7.0系のBreaking Changesをよく見て進めていきましょう！
追記
2019.03.14分
こんなんが出るようになりました。
filebeat-6.6.0-2019.02.07-000001
This index has [1127] fields, which exceeds the automatic field
expansion limit of 1024 and does not have [index.query.default_field]
set, which may cause queries which use automatic field expansion, such
as query_string, simple_query_string, and multi_match to fail if fields
are not explicitly specified in the query.
Beats系のIndexは総じてフィールド数が多いんですが、あんまり多いと特定のクエリで失敗する可能性があるよ、と。
2019.03.27分
6.7.0がリリースされたので、さっそく6.7.1をビルド。
あれ？ボタンが追加されちょる。
ポチッとな。
ログ見たら、なんか設定変更してくれてるし。
[2019-03-27T11:54:07,147][INFO ][o.e.c.s.IndexScopedSettings] [iNU-sMH] updating [index.query.default_field] from [["*"]] to [["apache2.access.agent","apache2.access.geoip.city_name","apache2.access.geoip.continent_name","apache2.access.geoip.country_iso_code","apache2.access.geoip.region_iso_code","apache2.access.geoip.region_name","apache2.access.http_version","apache2.access.method","apache2.access.referrer","apache2.access.remote_ip","apache2.access.url","apache2.access.user_agent.device","apache2.access.user_agent.name","apache2.access.user_agent.original","apache2.access.user_agent.os","apache2.access.user_agent.os_name","apache2.access.user_agent.patch","apache2.access.user_name","apache2.error.client","apache2.error.level","apache2.error.message","apache2.error.module","auditd.log.a0","auditd.log.acct","auditd.log.geoip.city_name","auditd.log.geoip.continent_name","auditd.log.geoip.country_iso_code","auditd.log.geoip.region_iso_code","auditd.log.geoip.region_name","auditd.log.item","auditd.log.items","auditd.log.new_auid","auditd.log.new_ses","auditd.log.old_auid","auditd.log.old_ses","auditd.log.pid","auditd.log.ppid","auditd.log.record_type","auditd.log.res","beat.hostname","beat.name","beat.timezone","beat.version","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.region_name","destination.ip","docker.container.id","docker.container.image","docker.container.name","elasticsearch.audit.action","elasticsearch.audit.event_type","elasticsearch.audit.layer","elasticsearch.audit.origin_address","elasticsearch.audit.origin_type","elasticsearch.audit.principal","elasticsearch.audit.request","elasticsearch.audit.request_body","elasticsearch.audit.uri","elasticsearch.gc.phase.name","elasticsearch.gc.tags","elasticsearch.index.id","elasticsearch.index.name","elasticsearch.node.name","elasticsearch.server.component","elasticsearch.shard.id","elasticsearch.slowlog.extra_source","elasticsearch.slowlog.id","elasticsearch.slowlog.logger","elasticsearch.slowlog.routing","elasticsearch.slowlog.search_type","elasticsearch.slowlog.source_query","elasticsearch.slowlog.stats","elasticsearch.slowlog.took","elasticsearch.slowlog.took_millis","elasticsearch.slowlog.total_hits","elasticsearch.slowlog.total_shards","elasticsearch.slowlog.type","elasticsearch.slowlog.types","error.message","error.type","event.dataset","event.type","file.path","fileset.module","fileset.name","haproxy.backend_name","haproxy.bind_name","haproxy.client.ip","haproxy.destination.ip","haproxy.error_message","haproxy.frontend_name","haproxy.geoip.city_name","haproxy.geoip.continent_name","haproxy.geoip.country_iso_code","haproxy.geoip.region_iso_code","haproxy.geoip.region_name","haproxy.http.request.captured_cookie","haproxy.http.request.captured_headers","haproxy.http.request.raw_request_line","haproxy.http.response.captured_cookie","haproxy.http.response.captured_headers","haproxy.mode","haproxy.process_name","haproxy.server_name","haproxy.source","haproxy.termination_state","host.architecture","host.id","host.ip","host.mac","host.name","host.os.family","host.os.platform","host.os.version","http.request.method","icinga.debug.facility","icinga.debug.message","icinga.debug.severity","icinga.main.facility","icinga.main.message","icinga.main.severity","icinga.startup.facility","icinga.startup.message","icinga.startup.severity","iis.access.agent","iis.access.cookie","iis.access.geoip.city_name","iis.access.geoip.continent_name","iis.access.geoip.country_iso_code","iis.access.geoip.region_iso_code","iis.access.geoip.region_name","iis.access.hostname","iis.access.http_version","iis.access.method","iis.access.query_string","iis.access.referrer","iis.access.remote_ip","iis.access.server_ip","iis.access.server_name","iis.access.site_name","iis.access.url","iis.access.user_agent.device","iis.access.user_agent.name","iis.access.user_agent.original","iis.access.user_agent.os","iis.access.user_agent.os_name","iis.access.user_agent.patch","iis.access.user_name","iis.error.geoip.city_name","iis.error.geoip.continent_name","iis.error.geoip.country_iso_code","iis.error.geoip.region_iso_code","iis.error.geoip.region_name","iis.error.http_version","iis.error.method","iis.error.queue_name","iis.error.reason_phrase","iis.error.remote_ip","iis.error.server_ip","iis.error.url","input.type","kafka.log.class","kafka.log.component","kafka.log.level","kafka.log.message","kafka.log.timestamp","kafka.log.trace.class","kafka.log.trace.full","kafka.log.trace.message","kibana.log.state","kibana.log.tags","kubernetes.container.image","kubernetes.container.name","kubernetes.namespace","kubernetes.node.name","kubernetes.pod.name","kubernetes.pod.uid","log.file.path","log.flags","log.level","log.source.address","logstash.log.level","logstash.log.message","logstash.log.module","logstash.log.thread","logstash.slowlog.event","logstash.slowlog.level","logstash.slowlog.message","logstash.slowlog.module","logstash.slowlog.plugin_name","logstash.slowlog.plugin_params","logstash.slowlog.plugin_type","logstash.slowlog.thread","message","meta.cloud.availability_zone","meta.cloud.instance_id","meta.cloud.instance_name","meta.cloud.machine_type","meta.cloud.project_id","meta.cloud.provider","meta.cloud.region","mongodb.log.component","mongodb.log.context","mongodb.log.message","mongodb.log.severity","mysql.error.level","mysql.error.message","mysql.error.timestamp","mysql.slowlog.host","mysql.slowlog.ip","mysql.slowlog.query","mysql.slowlog.user","netflow.application_category_name","netflow.application_description","netflow.application_group_name","netflow.application_name","netflow.application_sub_category_name","netflow.bgp_next_hop_ipv4_address","netflow.bgp_next_hop_ipv6_address","netflow.class_name","netflow.collector_ipv4_address","netflow.collector_ipv6_address","netflow.destination_ipv4_address","netflow.destination_ipv4_prefix","netflow.destination_ipv6_address","netflow.destination_ipv6_prefix","netflow.destination_mac_address","netflow.dot1q_customer_destination_mac_address","netflow.dot1q_customer_source_mac_address","netflow.encrypted_technology","netflow.exporter.address","netflow.exporter_ipv4_address","netflow.exporter_ipv6_address","netflow.http_content_type","netflow.http_message_version","netflow.http_reason_phrase","netflow.http_request_host","netflow.http_request_method","netflow.http_request_target","netflow.http_user_agent","netflow.information_element_description","netflow.information_element_name","netflow.interface_description","netflow.interface_name","netflow.ip_next_hop_ipv4_address","netflow.ip_next_hop_ipv6_address","netflow.ipv4_router_sc","netflow.metro_evc_id","netflow.mib_context_name","netflow.mib_module_name","netflow.mib_object_description","netflow.mib_object_name","netflow.mib_object_syntax","netflow.mib_object_valuei_pa_ddress","netflow.mobile_imsi","netflow.mobile_msisdn","netflow.mpls_top_label_ipv4_address","netflow.mpls_top_label_ipv6_address","netflow.nat_pool_name","netflow.observation_domain_name","netflow.original_exporter_ipv4_address","netflow.original_exporter_ipv6_address","netflow.p2p_technology","netflow.post_destination_mac_address","netflow.post_nadt_estination_ipv4_address","netflow.post_nadt_estination_ipv6_address","netflow.post_nast_ource_ipv4_address","netflow.post_nast_ource_ipv6_address","netflow.post_source_mac_address","netflow.pseudo_wire_destination_ipv4_address","netflow.sampler_name","netflow.selector_name","netflow.source_ipv4_address","netflow.source_ipv4_prefix","netflow.source_ipv6_address","netflow.source_ipv6_prefix","netflow.source_mac_address","netflow.sta_ipv4_address","netflow.sta_mac_address","netflow.tunnel_technology","netflow.type","netflow.user_name","netflow.virtual_station_interface_name","netflow.virtual_station_name","netflow.vr_fname","netflow.wlan_ssid","netflow.wtp_mac_address","nginx.access.agent","nginx.access.geoip.city_name","nginx.access.geoip.continent_name","nginx.access.geoip.country_iso_code","nginx.access.geoip.region_iso_code","nginx.access.geoip.region_name","nginx.access.http_version","nginx.access.method","nginx.access.referrer","nginx.access.remote_ip","nginx.access.url","nginx.access.user_agent.device","nginx.access.user_agent.name","nginx.access.user_agent.original","nginx.access.user_agent.os","nginx.access.user_agent.os_name","nginx.access.user_agent.patch","nginx.access.user_name","nginx.error.level","nginx.error.message","osquery.result.action","osquery.result.calendar_time","osquery.result.host_identifier","osquery.result.name","postgresql.log.database","postgresql.log.level","postgresql.log.message","postgresql.log.query","postgresql.log.timestamp","postgresql.log.timezone","postgresql.log.user","process.program","prospector.type","read_timestamp","redis.log.level","redis.log.message","redis.log.role","redis.slowlog.args","redis.slowlog.cmd","redis.slowlog.key","service.name","source","source_ecs.geo.city_name","source_ecs.geo.continent_name","source_ecs.geo.country_iso_code","source_ecs.geo.region_iso_code","source_ecs.geo.region_name","source_ecs.ip","stream","suricata.eve.alert.action","suricata.eve.alert.category","suricata.eve.alert.signature","suricata.eve.app_proto","suricata.eve.app_proto_expected","suricata.eve.app_proto_orig","suricata.eve.app_proto_tc","suricata.eve.app_proto_ts","suricata.eve.dest_ip","suricata.eve.dns.rcode","suricata.eve.dns.rdata","suricata.eve.dns.rrname","suricata.eve.dns.rrtype","suricata.eve.dns.type","suricata.eve.email.status","suricata.eve.event_type","suricata.eve.fileinfo.filename","suricata.eve.fileinfo.md5","suricata.eve.fileinfo.sha1","suricata.eve.fileinfo.sha256","suricata.eve.fileinfo.state","suricata.eve.flow.reason","suricata.eve.flow.state","suricata.eve.flow_id","suricata.eve.http.hostname","suricata.eve.http.http_content_type","suricata.eve.http.http_method","suricata.eve.http.http_refer","suricata.eve.http.http_user_agent","suricata.eve.http.protocol","suricata.eve.http.redirect","suricata.eve.http.url","sur
長すぎて、ログ切れてるしw