What is the hot-warm-cold architecture in Elasticsearch and how do you implement it for cost-effective log storage?
Quick Answer
Hot nodes use SSDs for recent, frequently queried data. Warm nodes use HDDs for older, less-queried data. Cold nodes use cheapest storage for compliance/archival data. ILM policies automate transitions.
Detailed Answer
Architecture Tiers
Hot tier: Recent data (last 1-7 days), fast SSDs, high CPU/RAM. All writes go here. Handles real-time queries and dashboards.
Warm tier: Older data (7-30 days), HDDs acceptable, moderate resources. Read-only (force merge to 1 segment for efficiency). Still queryable but slower.
Cold tier: Historical data (30-90+ days), cheapest storage. Searchable snapshots (ES 7.13+) allow querying data stored in S3/GCS without local replicas — massive cost savings.
Frozen tier (ES 7.12+): Data stays entirely in object storage. Searched on-demand with caching. Cheapest option for rarely accessed data.
Implementation: Assign node roles via node.roles: [data_hot], create ILM policies with rollover conditions, and let Elasticsearch automatically migrate indices between tiers.
Cost Impact: A typical production deployment sees 60-70% cost reduction compared to keeping all data on hot-tier SSDs.
Code Example
# Node configuration for each tier
# Hot node
node.roles: [data_hot, data_content]
path.data: /mnt/nvme/elasticsearch
# Warm node
node.roles: [data_warm]
path.data: /mnt/hdd/elasticsearch
# ILM policy with hot-warm-cold
PUT _ilm/policy/logs-tiered
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": { "max_size": "50gb", "max_age": "1d" },
"set_priority": { "priority": 100 }
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 },
"allocate": { "require": { "data": "warm" }},
"set_priority": { "priority": 50 }
}
},
"cold": {
"min_age": "30d",
"actions": {
"searchable_snapshot": { "snapshot_repository": "s3-repo" }
}
},
"delete": { "min_age": "90d", "actions": { "delete": {} }}
}
}
}Interview Tip
Mention searchable snapshots — they're a game-changer for cost. Data in S3 costs ~$0.023/GB/month vs ~$0.10/GB/month for EBS SSDs. Also mention the frozen tier for compliance data.