はじめに
Elasticsearchに対する操作をどこまで追えるのか検証する機会があり
有償機能であるSecurityのAudit Loggingを試したので、簡単ですが記事にまとめてみました。
利用環境
| product | version |
|---|---|
| kibana | 6.6.0 |
| Elasticsearch | 6.6.0 |
| X-Pack | 6.6.0 |
| OS(EC2) | Amazon Linux2 |
| AMI ID | ami-04677bdaa3c2b6e24 |
有償機能 Securityとは
現行最新版6.6では、以下のセキュリティ機能を有償機能として提供しています。
※今回の投稿は★の監査ログ(Audit Logging)に関する内容になります。
Elasticsearch
・暗号化通信
・役割ベースのアクセス制御
・監査ログ ★
・IPフィルタリング
・ネイティブ、LDAP、PKI、Active Directory認証
・シングルサインオン認証（SAML、Kerberos)
・属性ベースのアクセス制御
・フィールドレベルおよびドキュメントレベルのセキュリティ
・サードパーティの統合(カスタムレルム)
・認可レルム
・暗号化による保存のサポート
・FIPS 140-2モード
Kibana
・暗号化通信
・役割ベースのアクセス制御(Spacesを含む)
・ネイティブ、LDAP、Active Directory認証
・シングルサインオン認証(SAML)
【参考】
・セキュリティ機能のはじめ方
前提事項
Securityを有効化するとデフォルトで以下2つの機能を設定する必要があります。
① Elasticsearchのパスワード認証設定
② Elasticsearchのノード間TLS暗号化通信設定
6.3からX-Packという文言がFeatureという表現に変わったため、ディレクトリ構造も変わっています。
※本投稿では割愛していますが、以下のサイトを参考に設定してみてください！
6.2以前) /usr/share/elasticsearch/bin/x-pack/setup-passwords
6.3以降) /usr/share/elasticsearch/bin/elasticsearch-setup-passwords
6.2以前) /usr/share/elasticsearch/bin/x-pack/certutil
6.3以降) /usr/share/elasticsearch/bin/elasticsearch-certutil
【参考】
・X-Pack Securityの設定
Audit Logging設定方法
Audit Loggingの設定は以下の通りになります。
※今回は全操作の監査ログをElasticsearchにIndexするように設定
elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: index
xpack.security.audit.index.events.emit_request_body: true
xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted,authentication_success,realm_authentication_failed,connection_granted
kibana.yml
xpack.security.audit.enabled: true
【参考】
・Auditing security events
・Audit event types
実施したこと
セキュリティの監査ログ機能ですので、以下の情報がちゃんと記録されているかを確認してみました。
【検証シナリオ】
① Kibanaへのログイン操作
② Discover画面での検索操作
③ Dashboard画面でのダッシュボード参照操作
④ DevTools画面からGETクエリ操作
⑤ Monitoring画面からクラスタ状態確認の操作
⑥ Management画面から新規ユーザ作成操作
【確認ポイント】
・ 誰が(操作ユーザ)
・ いつ(timestamp)
・ どこから(送信元IP)
・ 何をしたのか(操作内容、実行クエリなど)
・ 実行結果(成功したのか失敗したのか)
結果
Elasticsearchに生成される.security_audit_log-yyyy.MM.ddにElasticsearchのAPI操作ログは記録されます。
172.31.6.143はKibana、172.31.8.191はElasticsearchのIPアドレスになります。
① Kibanaへのログイン操作
elasticというユーザでのログイン成功になります。
Elasticsearch_log_success
{
"_index": ".security_audit_log-2019.02.19",
"_type": "doc",
"_id": "Uot_BGkBVzdfFIt86rgk",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-19T06:44:44.843Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_success",
"principal": "elastic",
"realm": "reserved",
"request_body" : "",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/_authenticate"
},
"fields": {
"@timestamp": [
"2019-02-19T06:44:44.843Z"
]
},
"sort": [
1550558684843
]
}
elastic1というユーザでのログイン失敗になります。
Elasticsearch_log_failed
{
"_index": ".security_audit_log-2019.02.19",
"_type": "doc",
"_id": "1Yt_BGkBVzdfFIt81reZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-19T06:44:40.036Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "realm_authentication_failed",
"principal": "elastic1",
"realm": "reserved",
"request_body" : "",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/_authenticate"
},
"fields": {
"@timestamp": [
"2019-02-19T06:44:40.036Z"
]
},
"sort": [
1550558680036
]
}
{
"_index": ".security_audit_log-2019.02.19",
"_type": "doc",
"_id": "1ot_BGkBVzdfFIt81reZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-19T06:44:40.036Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "realm_authentication_failed",
"principal": "elastic1",
"realm": "default_file",
"request_body" : "",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/_authenticate"
},
"fields": {
"@timestamp": [
"2019-02-19T06:44:40.036Z"
]
},
"sort": [
1550558680036
]
}
{
"_index": ".security_audit_log-2019.02.19",
"_type": "doc",
"_id": "2Yt_BGkBVzdfFIt81reZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-19T06:44:40.040Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_failed",
"principal": "elastic1",
"request_body" : "",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/_authenticate"
},
"fields": {
"@timestamp": [
"2019-02-19T06:44:40.040Z"
]
},
"sort": [
1550558680040
]
}
{
"_index": ".security_audit_log-2019.02.19",
"_type": "doc",
"_id": "2It_BGkBVzdfFIt81reZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-19T06:44:40.040Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "realm_authentication_failed",
"principal": "elastic1",
"realm": "default_native",
"request_body" : "",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/_authenticate"
},
"fields": {
"@timestamp": [
"2019-02-19T06:44:40.040Z"
]
},
"sort": [
1550558680040
]
}
Kibana.ymlで指定したログ出力先にもKibanaのログインログは出力されます。
Kibana_log_success
{"type":"log","@timestamp":"2019-02-15T08:54:11Z","tags":["info","audit","spaces","spaces_authorization_success"],"pid":6358,"eventType":"spaces_authorization_success","username":"elastic","action":"getAll","spaceIds":["default"],"message":"elastic authorized to getAll default spaces"}
Kibana_log_failed
{"type":"log","@timestamp":"2019-02-15T08:57:00Z","tags":["debug","security","basic"],"pid":6358,"message":"Failed to authenticate request via login attempt: [security_exception] unable to authenticate user [elastic1] for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}
② Discover画面での検索操作
search_discover
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "8l8-6mgBKr8WQ3DYrtrV",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:23:22.649Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_success",
"principal": "elastic",
"realm": "reserved",
"request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551861247219}\n{\"version\":true,\"size\":500,\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"_source\":{\"excludes\":[]},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"interval\":\"5m\",\"time_zone\":\"Asia/Tokyo\",\"min_doc_count\":1}}},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"}],\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"elastic\",\"analyze_wildcard\":true,\"default_field\":\"*\"}},{\"range\":{\"@timestamp\":{\"gte\":1551861764755,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"fragment_size\":2147483647},\"timeout\":\"30000ms\"}\n",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
},
"fields": {
"@timestamp": [
"2019-02-14T04:23:22.649Z"
]
},
"sort": [
1550118202649
]
}
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "9F8-6mgBKr8WQ3DYrtrV",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:23:22.649Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "indices:data/read/search",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"indices": [
".security_audit_log-2019.02.14",
".security_audit_log-2019.02.13",
".security_audit_log-2019.02.12",
".security_audit_log-2019.02.11"
],
"request": "SearchRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:23:22.649Z"
]
},
"sort": [
1550118202649
]
}
③ Dashboard画面でのダッシュボード参照操作
view_dashboard
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "mF9E6mgBKr8WQ3DY1fj6",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:30:05.459Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_success",
"principal": "elastic",
"realm": "reserved",
"request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551862232912}\n{\"aggs\":{},\"size\":0,\"_source\":{\"excludes\":[]},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":{\"field\":\"timestamp\",\"format\":\"date_time\"},\"query\":{\"bool\":{\"must\":[{\"match_all\":{}},{\"range\":{\"timestamp\":{\"gte\":1551858325840,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"timeout\":\"30000ms\"}\n",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
},
"fields": {
"@timestamp": [
"2019-02-14T04:30:05.459Z"
]
},
"sort": [
1550118605459
]
}
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "ml9E6mgBKr8WQ3DY1fj6",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:30:05.460Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "indices:data/read/search",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"indices": [
".security_audit_log-2019.02.14",
".security_audit_log-2019.02.13",
".security_audit_log-2019.02.12",
".security_audit_log-2019.02.11"
],
"request": "SearchRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:30:05.460Z"
]
},
"sort": [
1550118605460
]
}
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "mV9E6mgBKr8WQ3DY1fj6",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:30:05.460Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "indices:data/read/msearch",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"request": "MultiSearchRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:30:05.460Z"
]
},
"sort": [
1550118605460
]
}
④ DevTools画面からGETクエリ操作
[DevTools]からGET _cat/indciesを実行した操作ログになります。
getapi_devtools
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "L19F6mgBKr8WQ3DYXvvD",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:30:40.606Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "indices:admin/get",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"indices": [
".triggered_watches",
".monitoring-es-6-2019.02.08",
".monitoring-es-6-2019.02.07",
".monitoring-es-6-2019.02.09",
".security_audit_log-2019.02.14",
".security_audit_log-2019.02.13",
".kibana",
".security_audit_log-2019.02.12",
".security_audit_log-2019.02.11",
".monitoring-alerts-6",
".monitoring-kibana-6-2019.02.10",
".monitoring-kibana-6-2019.02.11",
".monitoring-kibana-6-2019.02.12",
".monitoring-kibana-6-2019.02.13",
".monitoring-kibana-6-2019.02.14",
".watches",
".monitoring-es-6-2019.02.11",
".monitoring-es-6-2019.02.10",
".monitoring-es-6-2019.02.14",
".monitoring-es-6-2019.02.13",
".monitoring-es-6-2019.02.12",
".kibana_1",
".monitoring-kibana-6-2019.02.07",
".security",
".monitoring-kibana-6-2019.02.08",
".monitoring-kibana-6-2019.02.09",
"logstash-2019.02.03",
".security-6"
],
"request": "GetIndexRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:30:40.606Z"
]
},
"sort": [
1550118640606
]
}
⑤ Monitoring画面からクラスタ状態確認の操作
[Monitoring]でElasticsearch Clusterの性能を確認した操作ログになります。
view_monitoring
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "mmBH6mgBKr8WQ3DYVwUZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:32:50.349Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_success",
"principal": "elastic",
"realm": "reserved",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/.monitoring-es-2-*%2C.monitoring-es-6-*/_search?size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source.license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._source.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state%2Chits.hits._source.cluster_settings.cluster.metadata.display_name"
},
"fields": {
"@timestamp": [
"2019-02-14T04:32:50.349Z"
]
},
"sort": [
1550118770349
]
}
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "m2BH6mgBKr8WQ3DYVwUZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:32:50.349Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "indices:data/read/search",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"indices": [
".monitoring-es-6-2019.02.11",
".monitoring-es-6-2019.02.10",
".monitoring-es-6-2019.02.14",
".monitoring-es-6-2019.02.13",
".monitoring-es-6-2019.02.12",
".monitoring-es-6-2019.02.08",
".monitoring-es-6-2019.02.07",
".monitoring-es-6-2019.02.09"
],
"request": "SearchRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:32:50.349Z"
]
},
"sort": [
1550118770349
]
}
⑥ Management画面から新規ユーザ作成操作
test_userという新規ユーザを作成した操作ログになります。
create_user
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "FmBG6mgBKr8WQ3DYxgJ_",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:32:12.465Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "rest",
"event_type": "authentication_success",
"principal": "elastic",
"realm": "reserved",
"request_body" : """{"full_name":"test_user","roles":["kibana_dashboard_only_user"],"email":""}""",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"uri": "/_xpack/security/user/test_user"
},
"fields": {
"@timestamp": [
"2019-02-14T04:32:12.465Z"
]
},
"sort": [
1550118732465
]
}
{
"_index": ".security_audit_log-2019.02.14",
"_type": "doc",
"_id": "H2BG6mgBKr8WQ3DYxgJ_",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-02-14T04:32:12.547Z",
"node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
"node_host_name": "172.31.8.191",
"node_host_address": "172.31.8.191",
"layer": "transport",
"event_type": "access_granted",
"origin_type": "rest",
"origin_address": "172.31.6.143",
"action": "cluster:admin/xpack/security/user/put",
"principal": "elastic",
"realm": "reserved",
"roles": [
"superuser"
],
"request": "PutUserRequest"
},
"fields": {
"@timestamp": [
"2019-02-14T04:32:12.547Z"
]
},
"sort": [
1550118732547
]
}
まとめ
・誰が(操作ユーザ)
"principal": "elastic"でelasticというユーザの操作であることが判断出来ました。
・いつ(timestamp)
@timestampでログの記録時間が追えました。
・どこから(送信元IP)
Kibana操作は全て"origin_address": "172.31.6.143"となり、操作者端末の特定は出来ませんでした。
今のところ、X-forwarded-forをElasticsearchのAudit Logに記録する方法は無いようです。
・何をしたのか(操作内容、実行クエリなど)
xpack.security.audit.index.events.emit_request_body: trueを設定していることで
request_bodyがログに表示されます。こちらを見ることで実行した検索クエリを追うことが出来ました。
・実行結果(成功したのか失敗したのか)
以下のevent_typeで実行結果が判断出来ました。
・ログイン成功: authentication_success
・ログイン失敗: authentication_failed
・実行成功: access_granted
・実行失敗: access_denied