Loading...
Elasticsearch
Everything for Elasticsearch in one place — pick a section below. 15 reviewed items across 4 content types.
📝Interview Questions6Q&A with detailed answers and code examples→
🏭Use Cases3Real-world production patterns and scenarios→
🔥Production Issues3On-call incidents with root causes and fixes→
⭐Best Practices3Security, scaling, and day-2 operations→
6 / 6
⚡
Quick Answer
Elasticsearch needs a majority of master-eligible nodes for safe cluster coordination, replicas for shard availability, zone awareness to avoid correlated failure, and client routing that does not depend on one node. Resilience is a capacity and topology problem, not just a replica count.
Detailed Answer
A library with duplicate books still fails readers if the only librarian is unavailable. Elasticsearch needs both data redundancy and coordination redundancy.
Elastic production guidance frames resilience at node, zone, and index levels. Each level removes a different single point of failure: process, infrastructure failure domain, and data copy.
The elected master manages cluster state and shard allocation. Data nodes hold primary and replica shards. When a node fails, the cluster detects the fault, promotes or reallocates shards, and rebuilds missing copies when capacity is available.
Operators watch cluster health, unassigned shards, master elections, JVM pressure, disk watermarks, thread pool rejections, and search/index latency. Load balancers or clients should target multiple nodes so one coordinating node does not become a hard dependency.
A green cluster can still be under-resilient after a failure if remaining nodes are overloaded while shards rebuild. The design must reserve enough capacity to serve traffic and recover at the same time.
Code Example
PUT /logs-payments-*/_settings
{
"index.number_of_replicas": 1, # Keeps one extra copy of each shard for node failure tolerance.
"index.routing.allocation.include._tier_preference": "data_hot" # Keeps hot logs on the intended data tier.
}
GET /_cluster/health?pretty # Confirms health, active shards, and unassigned shard count.💡
Interview Tip
A junior engineer typically answers with the feature name and a happy-path command, but for a senior/architect role, the interviewer is actually looking for production judgment. Explain where the Elastic control plane stores intent, how changes are rolled out, what telemetry proves the system is healthy, and what failure mode creates the largest blast radius. Strong answers include rollback behavior, ownership boundaries, permissions, and the exact signal you would page on.
◈ Architecture Diagram
┌──────────┐
│ Signal │
└────┬─────┘
↓
┌──────────┐
│ Elastic │
└────┬─────┘
↓
┌──────────┐
│ Action │
└──────────┘💬 Comments
⚡
Quick Answer
Applications send logs to Logstash (or Beats), which parses and enriches them before indexing into Elasticsearch. Kibana queries Elasticsearch to render dashboards and visualizations.
Detailed Answer
Data Flow
1. Collection: Filebeat/Fluentd tails log files or receives syslog, adding metadata (hostname, service, timestamp) 2. Processing: Logstash receives events, applies grok filters to parse unstructured logs into fields, enriches with GeoIP/DNS lookup, and transforms 3. Indexing: Logstash outputs to Elasticsearch, which indexes documents into time-based indices (logs-2026.06.23) 4. Storage: Elasticsearch distributes data across shards on data nodes. Primary shards handle writes, replicas provide redundancy 5. Querying: Kibana sends queries to Elasticsearch, which searches across shards in parallel and aggregates results
Production at Scale
Use Index Lifecycle Management (ILM) to automatically move indices through hot→warm→cold→delete phases. Hot nodes use SSDs for fast writes, warm nodes use HDDs for cost-effective storage. Set shard size to 30-50GB for optimal performance. Use data streams instead of raw indices for time-series data.
Common Pitfall: Over-sharding. Each shard consumes ~50MB heap. 1000 indices × 5 primary shards × 1 replica = 10,000 shards = 500MB heap overhead before any data.
Code Example
# Logstash pipeline configuration
input {
beats { port => 5044 }
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}" }
}
date {
match => [ "timestamp", "ISO8601" ]
target => "@timestamp"
}
}
output {
elasticsearch {
hosts => ["http://es-cluster:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
}
# ILM policy
PUT _ilm/policy/logs-policy
{
"policy": {
"phases": {
"hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" }}},
"warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 }}},
"delete": { "min_age": "30d", "actions": { "delete": {} }}
}
}
}💡
Interview Tip
Walk through the data flow end-to-end. Mention ILM and shard sizing — these show you've operated ELK at scale, not just set it up once.
💬 Comments
⚡
Quick Answer
Use an odd number of master-eligible nodes (minimum 3) with discovery.zen.minimum_master_nodes set to (N/2)+1. In ES 7+, the cluster coordination layer handles this automatically.
Detailed Answer
Split-Brain Problem
If a network partition divides the cluster, both sides could elect their own master, leading to two independent clusters accepting writes. When the partition heals, data is inconsistent and potentially lost.
Solution (ES 7+)
Elasticsearch 7+ uses a new cluster coordination subsystem based on the Raft consensus algorithm. It automatically calculates the quorum requirement — you just need to configure cluster.initial_master_nodes during bootstrap. The cluster won't elect a master unless a majority of master-eligible nodes agree.
Minimum Configuration
- 3 master-eligible nodes (quorum = 2) - Dedicated master nodes (don't use them as data nodes in production) - Spread across 3 availability zones for network partition resilience
Pre-7.0 (Legacy)
Required manual setting: discovery.zen.minimum_master_nodes: 2 for a 3-node cluster. If set incorrectly (e.g., 1), split-brain is possible.
Production Best Practice: Use 3 dedicated master nodes, separate from data nodes. This keeps cluster management lightweight and prevents resource contention from heavy indexing/searches affecting master stability.
Code Example
# elasticsearch.yml for dedicated master node node.name: master-1 node.roles: [master] cluster.name: production cluster.initial_master_nodes: ["master-1", "master-2", "master-3"] discovery.seed_hosts: ["master-1:9300", "master-2:9300", "master-3:9300"] # Check cluster health GET _cluster/health # Look for: status (green/yellow/red), number_of_nodes, active_shards # Check master election GET _cat/master?v # Shows current master node # Check node roles GET _cat/nodes?v&h=name,node.role,master
💡
Interview Tip
Mention the Raft-based coordination in ES 7+ — it shows you know the modern approach, not just the legacy minimum_master_nodes setting.
💬 Comments
⚡
Quick Answer
Hot nodes use SSDs for recent, frequently queried data. Warm nodes use HDDs for older, less-queried data. Cold nodes use cheapest storage for compliance/archival data. ILM policies automate transitions.
Detailed Answer
Architecture Tiers
Hot tier: Recent data (last 1-7 days), fast SSDs, high CPU/RAM. All writes go here. Handles real-time queries and dashboards.
Warm tier: Older data (7-30 days), HDDs acceptable, moderate resources. Read-only (force merge to 1 segment for efficiency). Still queryable but slower.
Cold tier: Historical data (30-90+ days), cheapest storage. Searchable snapshots (ES 7.13+) allow querying data stored in S3/GCS without local replicas — massive cost savings.
Frozen tier (ES 7.12+): Data stays entirely in object storage. Searched on-demand with caching. Cheapest option for rarely accessed data.
Implementation: Assign node roles via node.roles: [data_hot], create ILM policies with rollover conditions, and let Elasticsearch automatically migrate indices between tiers.
Cost Impact: A typical production deployment sees 60-70% cost reduction compared to keeping all data on hot-tier SSDs.
Code Example
# Node configuration for each tier
# Hot node
node.roles: [data_hot, data_content]
path.data: /mnt/nvme/elasticsearch
# Warm node
node.roles: [data_warm]
path.data: /mnt/hdd/elasticsearch
# ILM policy with hot-warm-cold
PUT _ilm/policy/logs-tiered
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": { "max_size": "50gb", "max_age": "1d" },
"set_priority": { "priority": 100 }
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 },
"allocate": { "require": { "data": "warm" }},
"set_priority": { "priority": 50 }
}
},
"cold": {
"min_age": "30d",
"actions": {
"searchable_snapshot": { "snapshot_repository": "s3-repo" }
}
},
"delete": { "min_age": "90d", "actions": { "delete": {} }}
}
}
}💡
Interview Tip
Mention searchable snapshots — they're a game-changer for cost. Data in S3 costs ~$0.023/GB/month vs ~$0.10/GB/month for EBS SSDs. Also mention the frozen tier for compliance data.
💬 Comments
⚡
Quick Answer
Use dedicated node roles (master/data_hot/data_warm/ingest), multiple ingest pipelines, time-based data streams with ILM, and size for ~86TB of daily data with appropriate shard count.
Detailed Answer
Capacity Planning
- 1M events/sec × 1KB avg = 1GB/sec raw ingest - Daily: ~86TB raw, ~43TB after compression (2:1) - 30 days retention: ~1.3PB compressed storage
Cluster Design
- 3 dedicated master nodes (16GB RAM, no data) - 20 hot data nodes (64GB RAM, NVMe SSDs, 10TB each) - 40 warm data nodes (32GB RAM, HDDs, 40TB each) - 5 ingest nodes (32GB RAM, CPU-optimized for parsing) - 3 coordinating-only nodes for query load balancing
Indexing Strategy
- Data streams with daily rollover - 20 primary shards per index (1 per hot data node) - 1 replica (doubles hot storage need but provides redundancy) - Target shard size: 30-50GB
Performance Tuning
- Bulk indexing with 5-15MB bulk size - Increase index.refresh_interval to 30s (default 1s) - Use index.translog.durability: async for higher throughput (accept risk of losing last 5s of data on crash) - Disable _source if you don't need re-indexing capability
Query Performance
- Warm nodes: force merge to 1 segment per shard - Use index sorting for common query patterns - Pre-built Kibana dashboards with date filters default to last 24h
Code Example
# Data stream template
PUT _index_template/logs-template
{
"index_patterns": ["logs-*"],
"data_stream": {},
"template": {
"settings": {
"number_of_shards": 20,
"number_of_replicas": 1,
"index.refresh_interval": "30s",
"index.translog.durability": "async",
"index.translog.sync_interval": "5s",
"index.lifecycle.name": "logs-tiered"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"message": { "type": "text" },
"level": { "type": "keyword" },
"service": { "type": "keyword" }
}
}
}
}
# Bulk insert optimization
curl -XPOST 'es:9200/logs/_bulk' -H 'Content-Type: application/x-ndjson' --data-binary @bulk.ndjson💡
Interview Tip
Show the math: 1M events/sec → daily volume → 30-day retention → total storage. Then map to node count and shard strategy. This demonstrates you can design systems, not just configure them.
💬 Comments
⚡
Quick Answer
Elasticsearch: full-text search, complex queries, rich ecosystem. Loki: cost-effective, label-based, Grafana-native. ClickHouse: SQL analytics, high compression, columnar storage for structured logs.
Detailed Answer
Elasticsearch
- Full-text indexing of all log content - Rich query language (KQL, Lucene, aggregations) - Mature ecosystem (Beats, Logstash, Kibana) - Best for: Teams needing full-text search across all log fields, complex aggregations, existing ELK investment - Drawback: Expensive — indexes every field, high memory/CPU/storage costs
Grafana Loki
- Only indexes labels (metadata), not log content - Log content is compressed and stored in object storage (S3/GCS) - Uses LogQL (PromQL-like) for querying - Best for: Cost-sensitive environments, Kubernetes-native logging, teams already using Grafana/Prometheus - Drawback: Grep-style search on content is slower than Elasticsearch's inverted index. Not suitable for complex analytics.
ClickHouse
- Columnar storage with extreme compression (10:1 to 50:1) - SQL interface for analytics - Blazing fast aggregation queries - Best for: Structured/semi-structured logs, analytics-heavy use cases, very high volume (billions of rows), teams comfortable with SQL - Drawback: Less mature log ecosystem, requires more custom pipeline work
Cost Comparison (1TB/day, 30-day retention)
- Elasticsearch: ~$15K-25K/month (hot-warm architecture) - Loki: ~$2K-5K/month (object storage + minimal compute) - ClickHouse: ~$3K-8K/month (high compression reduces storage)
Code Example
# Loki LogQL query (equivalent to ES full-text search)
{namespace="production", app="api-gateway"}
|= "error"
| json
| status_code >= 500
| line_format "{{.timestamp}} {{.message}}"
# ClickHouse log table
CREATE TABLE logs (
timestamp DateTime64(3),
service LowCardinality(String),
level LowCardinality(String),
message String,
trace_id String
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (service, timestamp)
TTL timestamp + INTERVAL 30 DAY;
# Elasticsearch equivalent query
GET logs-*/_search
{
"query": {
"bool": {
"must": [
{ "match": { "message": "error" }},
{ "range": { "status_code": { "gte": 500 }}}
]
}
}
}💡
Interview Tip
Lead with the trade-off triangle: search capability vs cost vs query speed. Show you can make a recommendation based on the team's specific needs rather than just listing features.
💬 Comments