LoginSignup
6
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@hssh2_bin(Hisashi Hibino)

ElasticsearchのAPI監査ログ

Last updated at Posted at 2019-03-03

はじめに

Elasticsearchに対する操作をどこまで追えるのか検証する機会があり
有償機能であるSecurityのAudit Loggingを試したので、簡単ですが記事にまとめてみました。

利用環境

product version
kibana 6.6.0
Elasticsearch 6.6.0
X-Pack 6.6.0
OS(EC2) Amazon Linux2
AMI ID ami-04677bdaa3c2b6e24

有償機能 Securityとは

現行最新版6.6では、以下のセキュリティ機能を有償機能として提供しています。
※今回の投稿は★の**監査ログ(Audit Logging)**に関する内容になります。

Elasticsearch
・暗号化通信
・役割ベースのアクセス制御
・監査ログ　★
・IPフィルタリング
・ネイティブ、LDAP、PKI、Active Directory認証
・シングルサインオン認証（SAML、Kerberos)
・属性ベースのアクセス制御
・フィールドレベルおよびドキュメントレベルのセキュリティ
・サードパーティの統合(カスタムレルム)
・認可レルム
・暗号化による保存のサポート
・FIPS 140-2モード

Kibana
・暗号化通信
・役割ベースのアクセス制御(Spacesを含む)
・ネイティブ、LDAP、Active Directory認証
・シングルサインオン認証(SAML)

【参考】
・セキュリティ機能のはじめ方

前提事項

Securityを有効化するとデフォルトで以下2つの機能を設定する必要があります。
① Elasticsearchのパスワード認証設定
② Elasticsearchのノード間TLS暗号化通信設定

6.3からX-Packという文言がFeatureという表現に変わったため、ディレクトリ構造も変わっています。
※本投稿では割愛していますが、以下のサイトを参考に設定してみてください！

6.2以前) /usr/share/elasticsearch/bin/x-pack/setup-passwords
6.3以降) /usr/share/elasticsearch/bin/elasticsearch-setup-passwords

6.2以前) /usr/share/elasticsearch/bin/x-pack/certutil
6.3以降) /usr/share/elasticsearch/bin/elasticsearch-certutil

【参考】
・X-Pack Securityの設定

Audit Logging設定方法

Audit Loggingの設定は以下の通りになります。
※今回は全操作の監査ログをElasticsearchにIndexするように設定

elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: index
xpack.security.audit.index.events.emit_request_body: true
xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted,authentication_success,realm_authentication_failed,connection_granted
kibana.yml
xpack.security.audit.enabled: true

【参考】
・Auditing security events
・Audit event types

実施したこと

セキュリティの監査ログ機能ですので、以下の情報がちゃんと記録されているかを確認してみました。

【検証シナリオ】
① Kibanaへのログイン操作
② Discover画面での検索操作
③ Dashboard画面でのダッシュボード参照操作
④ DevTools画面からGETクエリ操作
⑤ Monitoring画面からクラスタ状態確認の操作
⑥ Management画面から新規ユーザ作成操作

【確認ポイント】
・ 誰が(操作ユーザ)
・ いつ(timestamp)
・ どこから(送信元IP)
・ 何をしたのか(操作内容、実行クエリなど)
・ 実行結果(成功したのか失敗したのか)

結果

Elasticsearchに生成される.security_audit_log-yyyy.MM.ddにElasticsearchのAPI操作ログは記録されます。
172.31.6.143はKibana、172.31.8.191はElasticsearchのIPアドレスになります。

① Kibanaへのログイン操作

elasticというユーザでのログイン成功になります。

Elasticsearch_log_success
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "Uot_BGkBVzdfFIt86rgk",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:44.843Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:44.843Z"
    ]
  },
  "sort": [
    1550558684843
  ]
}

elastic1というユーザでのログイン失敗になります。

Elasticsearch_log_failed
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "1Yt_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.036Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "reserved",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.036Z"
    ]
  },
  "sort": [
    1550558680036
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "1ot_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.036Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "default_file",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.036Z"
    ]
  },
  "sort": [
    1550558680036
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "2Yt_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.040Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_failed",
    "principal": "elastic1",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.040Z"
    ]
  },
  "sort": [
    1550558680040
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "2It_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.040Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "default_native",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.040Z"
    ]
  },
  "sort": [
    1550558680040
  ]
}

Kibana.ymlで指定したログ出力先にもKibanaのログインログは出力されます。

Kibana_log_success
{"type":"log","@timestamp":"2019-02-15T08:54:11Z","tags":["info","audit","spaces","spaces_authorization_success"],"pid":6358,"eventType":"spaces_authorization_success","username":"elastic","action":"getAll","spaceIds":["default"],"message":"elastic authorized to getAll default spaces"}
Kibana_log_failed
{"type":"log","@timestamp":"2019-02-15T08:57:00Z","tags":["debug","security","basic"],"pid":6358,"message":"Failed to authenticate request via login attempt: [security_exception] unable to authenticate user [elastic1] for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}

② Discover画面での検索操作

search_discover
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "8l8-6mgBKr8WQ3DYrtrV",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:23:22.649Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
　　"request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551861247219}\n{\"version\":true,\"size\":500,\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"_source\":{\"excludes\":[]},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"interval\":\"5m\",\"time_zone\":\"Asia/Tokyo\",\"min_doc_count\":1}}},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"}],\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"elastic\",\"analyze_wildcard\":true,\"default_field\":\"*\"}},{\"range\":{\"@timestamp\":{\"gte\":1551861764755,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"fragment_size\":2147483647},\"timeout\":\"30000ms\"}\n",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:23:22.649Z"
    ]
  },
  "sort": [
    1550118202649
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "9F8-6mgBKr8WQ3DYrtrV",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:23:22.649Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:23:22.649Z"
    ]
  },
  "sort": [
    1550118202649
  ]
}

③ Dashboard画面でのダッシュボード参照操作

view_dashboard
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mF9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.459Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551862232912}\n{\"aggs\":{},\"size\":0,\"_source\":{\"excludes\":[]},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":{\"field\":\"timestamp\",\"format\":\"date_time\"},\"query\":{\"bool\":{\"must\":[{\"match_all\":{}},{\"range\":{\"timestamp\":{\"gte\":1551858325840,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"timeout\":\"30000ms\"}\n",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.459Z"
    ]
  },
  "sort": [
    1550118605459
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "ml9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.460Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.460Z"
    ]
  },
  "sort": [
    1550118605460
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mV9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.460Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/msearch",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "request": "MultiSearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.460Z"
    ]
  },
  "sort": [
    1550118605460
  ]
}

④ DevTools画面からGETクエリ操作

[DevTools]からGET _cat/indciesを実行した操作ログになります。

getapi_devtools
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "L19F6mgBKr8WQ3DYXvvD",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:40.606Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:admin/get",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".triggered_watches",
      ".monitoring-es-6-2019.02.08",
      ".monitoring-es-6-2019.02.07",
      ".monitoring-es-6-2019.02.09",
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".kibana",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11",
      ".monitoring-alerts-6",
      ".monitoring-kibana-6-2019.02.10",
      ".monitoring-kibana-6-2019.02.11",
      ".monitoring-kibana-6-2019.02.12",
      ".monitoring-kibana-6-2019.02.13",
      ".monitoring-kibana-6-2019.02.14",
      ".watches",
      ".monitoring-es-6-2019.02.11",
      ".monitoring-es-6-2019.02.10",
      ".monitoring-es-6-2019.02.14",
      ".monitoring-es-6-2019.02.13",
      ".monitoring-es-6-2019.02.12",
      ".kibana_1",
      ".monitoring-kibana-6-2019.02.07",
      ".security",
      ".monitoring-kibana-6-2019.02.08",
      ".monitoring-kibana-6-2019.02.09",
      "logstash-2019.02.03",
      ".security-6"
    ],
    "request": "GetIndexRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:40.606Z"
    ]
  },
  "sort": [
    1550118640606
  ]
}

⑤ Monitoring画面からクラスタ状態確認の操作

[Monitoring]でElasticsearch Clusterの性能を確認した操作ログになります。

view_monitoring
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mmBH6mgBKr8WQ3DYVwUZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:50.349Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/.monitoring-es-2-*%2C.monitoring-es-6-*/_search?size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source.license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._source.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state%2Chits.hits._source.cluster_settings.cluster.metadata.display_name"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:50.349Z"
    ]
  },
  "sort": [
    1550118770349
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "m2BH6mgBKr8WQ3DYVwUZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:50.349Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".monitoring-es-6-2019.02.11",
      ".monitoring-es-6-2019.02.10",
      ".monitoring-es-6-2019.02.14",
      ".monitoring-es-6-2019.02.13",
      ".monitoring-es-6-2019.02.12",
      ".monitoring-es-6-2019.02.08",
      ".monitoring-es-6-2019.02.07",
      ".monitoring-es-6-2019.02.09"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:50.349Z"
    ]
  },
  "sort": [
    1550118770349
  ]
}

⑥ Management画面から新規ユーザ作成操作

test_userという新規ユーザを作成した操作ログになります。

create_user
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "FmBG6mgBKr8WQ3DYxgJ_",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:12.465Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body" : """{"full_name":"test_user","roles":["kibana_dashboard_only_user"],"email":""}""",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/user/test_user"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:12.465Z"
    ]
  },
  "sort": [
    1550118732465
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "H2BG6mgBKr8WQ3DYxgJ_",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:12.547Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "cluster:admin/xpack/security/user/put",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "request": "PutUserRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:12.547Z"
    ]
  },
  "sort": [
    1550118732547
  ]
}

まとめ

・誰が(操作ユーザ)
"principal": "elastic"でelasticというユーザの操作であることが判断出来ました。

・いつ(timestamp)
@timestampでログの記録時間が追えました。

・どこから(送信元IP)
Kibana操作は全て"origin_address": "172.31.6.143"となり、操作者端末の特定は出来ませんでした。
今のところ、X-forwarded-forをElasticsearchのAudit Logに記録する方法は無いようです。

・何をしたのか(操作内容、実行クエリなど)
xpack.security.audit.index.events.emit_request_body: trueを設定していることで
request_bodyがログに表示されます。こちらを見ることで実行した検索クエリを追うことが出来ました。

・実行結果(成功したのか失敗したのか)
以下のevent_typeで実行結果が判断出来ました。
・ログイン成功: authentication_success
・ログイン失敗: authentication_failed
・実行成功: access_granted
・実行失敗: access_denied

6
5
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
6
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?