How do you automate index lifecycle management (ISM) for log indices without risking deletion of data still needed for an active incident?

Think about an office building's shredding service that comes weekly to destroy documents past a set retention period. A well-run office doesn't let the shredding truck take anything past the cutoff date automatically and irreversibly — there's a process where anyone who knows a specific box of documents is part of an active legal matter can tag it 'hold, do not shred,' and the shredding service respects that tag even if the box is technically past its normal retention date on paper.

OpenSearch's Index State Management plugin automates exactly this kind of lifecycle for time-series log indices, moving indices through states like hot, actively written; warm, read-only and less resourced; cold; and eventually delete, based on age or size thresholds. It exists because manually rolling over and deleting daily log indices across dozens of services doesn't scale, and disk exhaustion from unbounded log retention is one of the most common self-inflicted OpenSearch outages. But a fully automated deletion pipeline is dangerous without a safety valve, since incident investigations frequently need log data older than the 'normal' retention window, especially for a slow-building issue that took days or weeks to even notice.

A safe ISM setup defines a policy per index pattern, like payments-api-logs-*, with rollover conditions based on index age or size, for example rolling over daily or at 50GB, a warm-phase transition after some days to move less-queried indices to cheaper, denser nodes, and a delete phase after a defined retention, say 30 or 90 days depending on compliance needs. The safety mechanism layered on top: before the delete action actually executes, ISM can be configured to check a metadata flag or a separate protected-indices allowlist, maintained via a small automation script responding to an incident-hold tag applied by on-call engineers, and skip deletion for any index matching that flag, plus send a notification through the policy's notification action a set number of days before deletion actually happens, giving humans a real window to intervene before data is gone.

In production, teams monitor ISM policy execution history — GET _plugins/_ism/explain on an index shows its current state and any errors in transitioning — to catch policies stuck retrying a failed transition, often due to an allocation problem preventing a warm-phase relocation, and track disk usage trends to confirm the retention policy is actually keeping pace with data growth. A policy that's technically running but consistently behind due to failed transitions provides a false sense of safety while disk quietly fills anyway, undetected until it becomes a real incident of its own.

The non-obvious gotcha: teams often set retention purely based on typical operational needs, like 'we only ever look back 14 days,' without accounting for the fact that security or compliance incidents are frequently discovered weeks or months after the fact. By the time someone realizes a suspicious authentication pattern from user-auth-service six weeks ago needs investigating, a 30-day ISM delete policy has already permanently destroyed the evidence. Retention windows need to be set based on the slowest realistic detection time for the kinds of incidents that matter most, not just typical day-to-day debugging needs that rarely look back more than a couple of weeks.