transformのretention_policyでは、_(アンダースコア)で始まるfieldは使えない

Field names prefixed with underscores are omitted from latest transformsedit
If you use the latest type of transform and the source index has field names that start with an underscore (_) character, they are assumed to be internal fields. Those fields are omitted from the documents in the destination index.

また、retention_policyを指定できる。

retention_policy
(Optional, object) Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

了解可能な仕様ではあるが、これを組み合わせてretention_policyでアンダースコアのついたフィールドを指定し、dest_indexにインデックスされないフィールドでretentionを制御しようとすると機能しなかった。

アンダースコア付きの"_timestamp"フィールドで試す

PUT test_transform_retention
{
  "mappings": {
    "properties": {
      "_timestamp": {"type": "date"},
      "foo": {"type": "keyword"}
    }
  }
}
POST test_transform_retention/_doc
{
  "_timestamp": "2000-01-02",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2000-01-01",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2022-06-17",
  "foo":"B"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2022-06-18",
  "foo":"B"
}

PUT _transform/transform_retention
{
  "source": {
    "index": "test_transform_retention"
  },
  "dest":{
    "index": "test_transform_retention_dest"
  },
  "latest":{
    "unique_key": "foo",
    "sort": "_timestamp"
  },
  "retention_policy":{
    "time":{
      "field": "_timestamp",
      "max_age": "365d"
    }
  }
}

POST _transform/transform_retention/_start

GET test_transform_retention_dest/_search

"hits" : [
      {
        "_index" : "test_transform_retention_dest",
        "_type" : "_doc",
        "_id" : "QRRx52klPRvG45a5oLgZ95sAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "foo" : "A"
        }
      },
      {
        "_index" : "test_transform_retention_dest",
        "_type" : "_doc",
        "_id" : "Qq7col5MOHvjTNMiAGonnqAAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "foo" : "B"
        }
      }
    ]

2件ともヒットしてしまう。

アンダースコアを排した"timestamp"フィールドで試す

"timestamp"フィールドに変更すると、1件のみインデックスされる。

PUT test_transform_retention2
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "foo": {"type": "keyword"}
    }
  }
}

POST test_transform_retention/_doc
{
  "_timestamp": "2000-01-02",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2000-01-01",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2022-06-17",
  "foo":"B"
}
POST test_transform_retention/_doc
{
  "_timestamp": "2022-06-18",
  "foo":"B"
}

PUT _transform/transform_retention2
{
  "source": {
    "index": "test_transform_retention2"
  },
  "dest":{
    "index": "test_transform_retention_dest2"
  },
  "latest":{
    "unique_key": "foo",
    "sort": "timestamp"
  },
  "retention_policy":{
    "time":{
      "field": "timestamp",
      "max_age": "365d"
    }
  }
}

POST _transform/transform_retention2/_start

GET test_transform_retention_dest2/_search

"hits" : [
      {
        "_index" : "test_transform_retention_dest2",
        "_type" : "_doc",
        "_id" : "Qq7col5MOHvjTNMiAGonnqAAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "foo" : "B",
          "timestamp" : "2022-06-18"
        }
      }
    ]

Runtime Fieldも機能しない

PUT test_transform_retention_on_runtime
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "runtimestamp":{
        "type": "date",
        "script":{
          "source": "emit(doc['timestamp'].value.toEpochMilli());"
        }
      },
      "foo": {"type": "keyword"}
    }
  }
}

POST test_transform_retention/_doc
{
  "timestamp": "2000-01-02",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "timestamp": "2000-01-01",
  "foo":"A"
}
POST test_transform_retention/_doc
{
  "timestamp": "2022-06-17",
  "foo":"B"
}
POST test_transform_retention/_doc
{
  "timestamp": "2022-06-18",
  "foo":"B"
}

GET test_transform_retention_on_runtime/_search
{
  "fields": ["*"]
}

PUT _transform/transform_retention_on_runtime
{
  "source": {
    "index": "test_transform_retention_on_runtime"
  },
  "dest":{
    "index": "test_transform_retention_on_runtime_dest"
  },
  "latest":{
    "unique_key": "foo",
    "sort": "runtimestamp"
  },
  "retention_policy":{
    "time":{
      "field": "runtimestamp",
      "max_age": "10d"
    }
  }
}

POST _transform/transform_retention_on_runtime/_start

GET _transform/transform_retention_on_runtime/_stats
GET test_transform_retention_on_runtime_dest/_search
{
  "fields": ["*"]
}

runtime fieldもretention_policyにおいては機能しない（sortとしては用いることができそう）。

したがって、retention_policyでは、dest indexでも実体のあるフィールドを指定する必要がありそうだ。