How would you design an Elasticsearch cluster to handle 1 million log events per second with 30-day retention?
Quick Answer
Use dedicated node roles (master/data_hot/data_warm/ingest), multiple ingest pipelines, time-based data streams with ILM, and size for ~86TB of daily data with appropriate shard count.
Detailed Answer
Capacity Planning
- 1M events/sec × 1KB avg = 1GB/sec raw ingest - Daily: ~86TB raw, ~43TB after compression (2:1) - 30 days retention: ~1.3PB compressed storage
Cluster Design
- 3 dedicated master nodes (16GB RAM, no data) - 20 hot data nodes (64GB RAM, NVMe SSDs, 10TB each) - 40 warm data nodes (32GB RAM, HDDs, 40TB each) - 5 ingest nodes (32GB RAM, CPU-optimized for parsing) - 3 coordinating-only nodes for query load balancing
Indexing Strategy
- Data streams with daily rollover - 20 primary shards per index (1 per hot data node) - 1 replica (doubles hot storage need but provides redundancy) - Target shard size: 30-50GB
Performance Tuning
- Bulk indexing with 5-15MB bulk size - Increase index.refresh_interval to 30s (default 1s) - Use index.translog.durability: async for higher throughput (accept risk of losing last 5s of data on crash) - Disable _source if you don't need re-indexing capability
Query Performance
- Warm nodes: force merge to 1 segment per shard - Use index sorting for common query patterns - Pre-built Kibana dashboards with date filters default to last 24h
Code Example
# Data stream template
PUT _index_template/logs-template
{
"index_patterns": ["logs-*"],
"data_stream": {},
"template": {
"settings": {
"number_of_shards": 20,
"number_of_replicas": 1,
"index.refresh_interval": "30s",
"index.translog.durability": "async",
"index.translog.sync_interval": "5s",
"index.lifecycle.name": "logs-tiered"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"message": { "type": "text" },
"level": { "type": "keyword" },
"service": { "type": "keyword" }
}
}
}
}
# Bulk insert optimization
curl -XPOST 'es:9200/logs/_bulk' -H 'Content-Type: application/x-ndjson' --data-binary @bulk.ndjsonInterview Tip
Show the math: 1M events/sec → daily volume → 30-day retention → total storage. Then map to node count and shard strategy. This demonstrates you can design systems, not just configure them.