OpenSearch: Zero to Hero

The open-source search and analytics engine (an Elasticsearch fork) that powers log search, observability, and full-text search. You'll run a node locally, index documents, write queries, aggregate data, ship logs into it, and understand the production concerns of a distributed cluster. Hands-on throughout. Work top to bottom, or jump to a part.

The journey:

1. What OpenSearch is and the data model
2. Run a node locally
3. Index and read documents
4. Search — full-text and structured queries
5. Aggregations — analytics over your data
6. Mappings and index templates
7. Ingesting logs (the observability use case)
8. Sharding, replicas, and cluster health
9. Production practices

---

Prerequisites

OpenSearch is a fork of Elasticsearch 7.10 — concepts and much of the API carry over. Everything here uses its REST API on port 9200.

---

Part 1 — What OpenSearch is and the data model

OpenSearch is a distributed document store with a powerful search engine on top. Its data model:

The magic is the inverted index: OpenSearch analyzes text into terms and maps each term to the documents containing it, making full-text search over huge datasets fast. It pairs a search engine with a JSON document API and OpenSearch Dashboards for visualization.

---

Part 2 — Run a node locally

docker run --rm -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Str0ng!Pass" \
  opensearchproject/opensearch:2.15.0

Confirm it's up (self-signed TLS, so -k):

curl -k -u admin:'Str0ng!Pass' https://localhost:9200
curl -k -u admin:'Str0ng!Pass' https://localhost:9200/_cluster/health?pretty

The health response shows status: green|yellow|red — green means all shards allocated. On a single node you'll often see yellow (replica shards can't be placed) — that's expected locally.

---

Part 3 — Index and read documents

Indexing is a PUT/POST of JSON. OpenSearch creates the index and infers a mapping on first write:

# Index a document with id 1
curl -k -u admin:'Str0ng!Pass' -X PUT "https://localhost:9200/logs/_doc/1" \
  -H 'Content-Type: application/json' -d '
{ "level": "error", "service": "api", "message": "timeout calling db", "@timestamp": "2026-07-16T10:00:00Z" }'

# Read it back
curl -k -u admin:'Str0ng!Pass' "https://localhost:9200/logs/_doc/1?pretty"

# Bulk index many at once (faster — one request)
curl -k -u admin:'Str0ng!Pass' -X POST "https://localhost:9200/_bulk" \
  -H 'Content-Type: application/x-ndjson' --data-binary '
{ "index": { "_index": "logs" } }
{ "level": "info", "service": "web", "message": "started" }
{ "index": { "_index": "logs" } }
{ "level": "error", "service": "web", "message": "500 on /pay" }
'

Use the _bulk API for volume — one document per request doesn't scale. Documents are near-real-time: searchable within ~1 second (a refresh interval), not instantly.

---

Part 4 — Search: full-text and structured queries

Query with the Query DSL (JSON). Full-text match analyzes the query; term matches exact keywords; bool combines conditions:

curl -k -u admin:'Str0ng!Pass' -X GET "https://localhost:9200/logs/_search?pretty" \
  -H 'Content-Type: application/json' -d '
{
  "query": {
    "bool": {
      "must":   [ { "match": { "message": "timeout" } } ],
      "filter": [ { "term": { "level": "error" } } ]
    }
  }
}'

The distinction to internalize: must contributes to the relevance score (full-text, ranked), while filter is a yes/no match that's cached and faster (use it for exact terms, ranges, dates). text fields are analyzed for search; keyword fields are exact (for filtering, sorting, aggregating) — most string fields get both.

---

Part 5 — Aggregations: analytics over your data

Aggregations turn OpenSearch into an analytics engine — group, count, and compute over matching documents:

curl -k -u admin:'Str0ng!Pass' -X GET "https://localhost:9200/logs/_search?pretty" \
  -H 'Content-Type: application/json' -d '
{
  "size": 0,
  "aggs": {
    "by_service": {
      "terms": { "field": "service" },
      "aggs": { "errors": { "filter": { "term": { "level": "error" } } } }
    }
  }
}'

size: 0 skips the document hits and returns only the aggregation. Bucket aggregations group documents (terms, date_histogram, range); metric aggregations compute values (avg, sum, percentiles). Nest them — "error count per service per hour" — to build dashboards. This is the engine behind log analytics.

---

Part 6 — Mappings and index templates

A mapping is the schema — field names and types. Auto-mapping is fine for demos but risky in production (a field guessed as text can't be aggregated well). Define it explicitly:

curl -k -u admin:'Str0ng!Pass' -X PUT "https://localhost:9200/events" \
  -H 'Content-Type: application/json' -d '
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "service":    { "type": "keyword" },
      "message":    { "type": "text" },
      "latency_ms": { "type": "integer" }
    }
  }
}'

For time-series data (logs, metrics) use an index template so every new daily index (logs-2026.07.16) gets the right mapping and settings automatically, plus ISM (Index State Management) policies to roll over and delete old indices. Getting mappings right up front avoids painful reindexing later.

---

Part 7 — Ingesting logs (the observability use case)

The most common OpenSearch deployment is centralized logging. The pipeline: apps → a shipper → OpenSearch → Dashboards.

• Data Prepper / Logstash / Fluent Bit collect and parse logs, then bulk-index into OpenSearch.
• Index templates + ISM manage daily indices and retention.
• OpenSearch Dashboards provides search, visualizations, and alerting.

A typical Fluent Bit output:

[OUTPUT]
    Name            opensearch
    Host            opensearch
    Port            9200
    Index           logs
    Suppress_Type_Name On

Structure your logs as JSON at the source (fields like service, level, trace_id) so they're queryable and aggregatable without brittle regex parsing later.

---

Part 8 — Sharding, replicas, and cluster health

OpenSearch scales by splitting an index into primary shards (distributed across nodes) and copying each to replica shards (redundancy + read throughput).

• Primaries determine how data is spread; you set the count at index creation and it's (mostly) fixed.
• Replicas are copies on other nodes — they provide HA and serve reads; adjustable any time.
• Cluster status: green (all shards allocated), yellow (replicas unassigned — fine on one node, a warning in prod), red (a primary is missing — data unavailable).

curl -k -u admin:'Str0ng!Pass' "https://localhost:9200/_cat/shards?v"
curl -k -u admin:'Str0ng!Pass' "https://localhost:9200/_cat/indices?v"

The classic mistake is too many small shards (each has overhead) or one giant shard. Aim for shards in the tens-of-GB range and size shard count to your data and node count.

---

Part 9 — Production practices

• Explicit mappings + index templates — never rely on auto-mapping in production.
• Time-based indices + ISM for logs — roll over by size/age, delete old data on a policy.
• Right-size shards (~10–50GB each); avoid shard explosion.
• Run ≥3 master-eligible nodes for quorum; dedicate roles at scale.
• Set refresh_interval higher for heavy indexing (e.g. 30s) to boost throughput.
• Secure it — TLS, real auth/roles (never expose 9200 openly), and the security plugin.
• Snapshot to object storage for backups; monitor cluster health and JVM heap.
• Use _bulk for indexing volume; watch for rejected bulk requests (queue backpressure).

---

Common mistakes to avoid

• Relying on auto-mapping. A misinferred field type forces a costly reindex. Define mappings.
• Too many shards. Each shard has real overhead; thousands of tiny shards degrade a cluster.
• One document per request at scale. Use _bulk; single writes don't keep up.
• must where filter belongs. Exact-match/date/range conditions should be filter (cached, faster, no scoring).
• Ignoring yellow/red status. Red means missing primaries — data loss risk; investigate immediately.
• Unbounded retention. Logs grow forever without ISM rollover/deletion — the disk fills.
• Exposing 9200 without auth/TLS. A wide-open cluster is a data breach; enable the security plugin.

---

Practice: try it yourself

1. Index + read. Start a node, index three log documents (mix of info/error), and fetch one by id.
2. Structured search. Write a bool query with a match on the message and a filter on level: error — note which part scores.
3. Aggregate. Run a terms aggregation over service with size: 0, then nest an error count per service.
4. Explicit mapping. Create an index with a defined mapping (date, keyword, text, integer) and confirm with _mapping.
5. Cluster view. Inspect _cluster/health, _cat/indices, and _cat/shards — explain why status is yellow on one node.

Self-check:

• What's the difference between a text and a keyword field, and when do you use each?
• must vs filter — which scores, which is cached?
• What do primary vs replica shards each provide?
• What does a yellow vs red cluster status mean?

---

Where to go next

• Visualize and correlate with the Grafana tutorial (OpenSearch as a data source) or OpenSearch Dashboards.
• Feed it from Kubernetes — the Kubernetes tutorial plus Fluent Bit is the standard logging stack.
• Practice with observability interview questions in the question bank.

You now have the loop: index → search → aggregate → map → ingest logs → scale with shards → harden. That's OpenSearch as teams run it for search and observability.