[BUG] datastream shrinking policy creates hidden indices ignored by ISM

[shaigbdb]

This is the same as the now-closed #478 . It occurs in 2.17.1 as well.
What is the bug?
We created a shrink action in a policy for our datastreams to reduce the number of shards.

We start out with index .ds-testshrink-000001 which turns into shrunk_.ds-testshrink-000001. The shrunken index is not part of the datastream after shrinking and treated as a hidden index. A retention policy isn't automatically applied for these shrunken indices, because the ISM ignores them:

https://github.com/opensearch-project/index-management/blob/main/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt#L348

How can one reproduce the bug?

Create the datastream index template

PUT _index_template/testshrink
{
  "index_patterns": [
    "testshrink*"
  ],
  "template": {
    "settings": {
      "index": {
        "hidden": false
      },
      "number_of_replicas": 0,
      "number_of_shards": 2
    }
  },
  "data_stream": {
    "timestamp_field": {
      "name": "@timestamp"
    }
  }
}

Create the policy

{
 "policy": {
   "description": "A sample description of the policy",
   "default_state": "rollover",
   "states": [
     {
       "name": "rollover",
       "actions": [
         {
           "rollover": {
             "min_index_age": "1m",
             "copy_alias": false
           }
         }
       ],
       "transitions": [
         {
           "state_name": "shrink"
         }
       ]
     },
     {
       "name": "shrink",
       "actions": [
         {
           "shrink": {
             "num_new_shards": 1,
             "target_index_name_template": {
               "source": "shrunk_{{ctx.index}}",
               "lang": "mustache"
             },
             "aliases": [
               {
                 "shrunk_alias": {
                   "is_write_index": null
                 }
               }
             ],
             "switch_aliases": false,
             "force_unsafe": true
           }
         }
       ],
       "transitions": []
     }
   ],
   "ism_template": [
     {
       "index_patterns": [
         "testshrink*"
       ],
       "priority": 1
     }
   ]
 }
}

Create a policy for the shrunk indices

PUT _plugins/_ism/policies/shrunk
{
  "policy":{
    "description": "",
    "default_state": "hot",
    "states":[
      {"name": "hot",
        "actions" :[]
      
      }
      ],
      "ism_template":[
        {"index_patterns": ["shrunk*"]}
      ]
  }
}

Create a document

POST testshrink/_doc
{
    "@timestamp": "2022-08-16T10:00:00.0",
    "@version": "1",
    "message": "example message 1"
}

Now we wait and after a while we have the following indices:

.ds-testshrink-000002
.ds-testshrink-000001
shrunk_.ds-testshrink-000001

shrunk_.ds-testshrink-000001 is set to index.hidden: "true" and does not appear in _cat/indices.
GET _plugins/_ism/explain/shrunk_.ds-testshrink-000001 shows that no policy is attached to the shrunk index.
templates matching shrunk* also do not apply to the new index.

Current workaround is to:

1. Create a separate policy for shrunk indices
2. Have the policy for the non-shrunk indices delete them after shrinking
3. Manually change the index.hidden setting for the shrunk index - POST shrunk_.../_settings...
4. Manually apply a policy to the shrunk index - POST _plugins/_ism/add/shrunk_... (This step doesn't require step 3)

What is the expected behavior?

At best, I would expect the shrink action to generate an index that replaces the original index in the index lifecycle, meaning lifecycle would proceed from the same state and the index age would be the same as the original index.

As a minimum:
a. I would expect that policies with index patterns matching the generated shrunken index would automatically apply to it.
b. I'm not sure if there was a reason for setting the shrunk index to hidden (this seems to be a behavior shared by the shrink API, though overridable in the API with settings that apply after the index creation), but at the very least I would expect to have some way to override it in ISM as well.

What is your host/environment?

Docker Deskstop 4.3.1
opensearchproject/opensearch:2.17.1

[soosinha]

[Triage - attendees 1 2 3 4]
Thanks for reporting this. We would a welcome a PR for this.