Explain Grafana Loki's architecture and how LogQL works. How does Loki achieve cost-effective log storage compared to Elasticsearch?
Quick Answer
Loki only indexes labels (not log content), stores compressed log chunks in object storage (S3/GCS). LogQL uses label matchers for stream selection and filter expressions for content search. This reduces storage costs by 10-50x vs Elasticsearch.
Detailed Answer
Loki Architecture
Write Path
1. Promtail/Grafana Agent scrapes logs, adds labels (pod, namespace, container) 2. Distributor receives log entries, validates, and hashes labels to determine ingester 3. Ingester builds compressed chunks in memory, flushes to object storage (S3/GCS) 4. Index (label→chunk mapping) stored in DynamoDB/Cassandra/BoltDB
Read Path
1. Query frontend splits query by time range 2. Querier finds relevant chunks using label index 3. Chunks are downloaded from object storage, decompressed, and grep-searched 4. Results aggregated and returned
Why it's cheaper than Elasticsearch
- No full-text indexing — labels are tiny compared to full log content - Logs stored as compressed chunks in cheap object storage - Minimal index size (labels only, not every word) - Trade-off: Content search is slower (grep vs inverted index)
LogQL
- Stream selector: {namespace="production", app="api"} (like PromQL label matchers) - Filter: |= "error" (contains), != "debug" (excludes), |~ "err(or)?" (regex) - Parser: | json or | logfmt to extract fields - Aggregation: count_over_time({app="api"} |= "error" [5m])
Code Example
# LogQL: Find errors in production API
{namespace="production", app="api-gateway"}
|= "error"
| json
| status_code >= 500
| line_format "{{.ts}} [{{.level}}] {{.msg}}"
# LogQL: Error rate over time (metric query)
sum(rate(
{namespace="production"} |= "error" [5m]
)) by (app)
# LogQL: Top 5 error messages
topk(5,
sum(count_over_time(
{app="api"} | json | level="error" [1h]
)) by (msg)
)
# Promtail config (scrapes Kubernetes pod logs)
server:
http_listen_port: 3101
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: podInterview Tip
The key insight is the indexing trade-off: Loki indexes labels (tiny), Elasticsearch indexes content (huge). This is WHY Loki is cheaper. Know LogQL syntax — it's increasingly replacing Elasticsearch in modern stacks.