Starter-kit for Elasticsearch operations

Elasticsearch is that kind of product, which is very easy to start, but then you’re getting one hundred and one questions about it on the go. Personally, I spent weeks jumping over documentation (it’s excellent, but it’s massive), querying API, reading blogs — everything just to understand what the hell is going on. Cluster unexpectedly (for me) becomes red, or just rejects new data, or one node going down once a day.

The following article is my side-notes after a year of running Elasticsearch cluster to store a mid-size distributed project logs.

The article does not teach how to install Elasticsearch cluster. Official installation and getting-started posts are pretty good (there is also kubernetes operator).

The article expects that you have already Elasticsearch cluster running and now you have a question about it. Here you can find:

• high-level explanation of Elasticsearch concepts;
• a few clues how to size nodes;
• references to other posts and parts of official documentation, which should be useful in particular context;
• proposed troubleshooting guide.

I wish, I had something like this when I started. Thus, I hope to make someone’s life a bit easier.

© Library of congress

A step back. What is Elasticsearch?

Elasticsearch is distributed search engine. Originally it was built for a full-text search with miscellaneous tokenizers and text-analyzers. Later, turned out that it also can do a good job in storing and querying logs.

Important logs feature:

• In most cases customer is interested in text (log message) and full-text search on top of it;
• Often, logs have attached attributes, to express context of logged event, that are important to be able to filter. For example, event-origin (like host name).
• Mostly, logs are stored for a fixed amount of days and then can be discarded.
• Amount of logs might vary and likely would not fit to a single machine

Elasticsearch can satisfy all of these requirements and it’s very easy to start with (but not that easy to maintain later).

Indices and shards

All the data in Elasticsearch is organized in indices. Index is a namespace, which consists of real unit with data — shards. Every shard is an instance of Lucene.

• every index has at least one primary shard and 0..N replicas;
• a primary shard can technically contain up to Integer.MAX_VALUE-128 documents;
• data structure — Inverted Index.

The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time.

Elasticsearch cluster

Elasticsearch can run as a single instance or in a cluster mode. If Elasticsearch instances form a cluster, they might have different roles:

• master: a node that has node.master configuration property set to true (it’s default value), which makes it eligible to be elected as the master node, which controls the cluster.
• data (hot and warm): a node that has node.dataconfiguration property set to true (it’s default value too). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.
• ingest: a node that has node.ingest configuration property set to true (it’s default value again). Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing.

By default, Elasticsearch instance is Jack of all trades: it can be master, data, and ingest at the same time.

Each Elasticsearch instance has a discovery module, which is responsible for finding nodes to form a cluster, electing a master, send state updates. If you have an issue with forming a cluster from individual nodes — discovery module configuration should be one of your first stops.

Elasticsearch Cluster State

Maybe, the most important part of Elasticsearch cluster is Cluster State:

• stored in memory of every node;
• only master node is allowed to update it;
• includes cluster-level settings, nodes, indices (settings, mappings), shards
• state updates are serializable. when master gets a change request, it sends changeset to followers and waits for acknowledges; only when configured minimum of acknowledges is received, change is applied and sent to all the nodes.

State API supports multiple parameters: official documentation.

Elasticsearch Cluster Health

Cluster Health can be queried with GET _cluster/health, where:

• Green means everything is good (cluster is fully functional)
• Yellow: all data is available but some replicas are not yet allocated (cluster is still fully functional)
• Red: some data is not available for whatever reason (cluster is partially functional, but continues serving search requests from the available shards)

Health API also supports multiple parameters: official documentation.

Very simplified, you can think about cluster health in the following way:

1. cluster health depends on amount of healthy indices;
2. index health depends on shards allocation status;
3. shard allocation depends on available resources of your ES cluster;
4. shard allocation happens when (a) index is created (b) when ES tries to rebalance shards (for example, when node leaves or joins the cluster).

Often, if you suspect that your cluster is not healthy you have to find a shard, which fails to be allocated.

Elasticsearch resource consumption

Obviously, you have to monitor used disk space. Once ES has no disk space left — it rejects any new data. To keep size at more or less same level, old indices could be daily/weekly/monthly deleted from the cluster, which is responsibility of Elasticsearch Curator (separate application) or Index Lifecycle Management (part of Elasticsearch if you have a commercial version).

But, maybe the most important thing to keep in mind: Elasticsearch State is kept in memory of each ES instance. This means, the bigger the state — more memory should be dedicated to each ES instance. If the state is too big — instances might crush with OutOfMemory exceptions (very common, if you are running Elastic in docker containers with set memory limits).

The size of Elasticsearch State is very influenced by the way the data is stored. Here are a few quotes from ES blog about sizing shards:

Small shards result in small segments, which increases overhead. Aim to keep the average shard size between at least a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.

plus:

As the overhead per shard depends on the segment count and size, forcing smaller segments to merge into larger ones through a forcemerge operation can reduce overhead and improve query performance. This should ideally be done once no more data is written to the index. Be aware that this is an expensive operation that should ideally be performed during off-peak hours.

and finally:

A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap

Each ingest-data request also use a subset of memory. Therefore, if the cluster has very little room after storing the entire state file, it can also go out of memory because of too many concurrent requests.

More details about sizing the heap:

• A heap of Trouble
• Sizing the heap
• Why bulk requests are rejected
• How memory pressure affect ES performance

Elasticsearch curator

If you use open-source version of Elasticsearch without any license, Elasticsearch Curator is the easiest way to deal with daily ES routine: delete obsolete indices, optimize existing, etc.

There is no official Elasticsearch curator docker image yet:

For now, your best bet is to build one yourself from the Dockerfile in the github repository. You could publish it yourself, or just host it in your local Docker repository. There are a few more steps before Curator will be published with its own Docker image, so I apologize for its absence.

© from Elastic discussion forum.

Curator is often used at least to delete obsolete indices, but also it could be used to close and forcemerge indices, make snapshots, etc.

For example, here is Curator config to delete indices older than 14 days with index name pattern logs-%Y-%m-%d:

actions:
  1:
    action: delete_indices
    description: "Clean up ES by deleting old indices"
    options:
      timeout_override:
      continue_if_exception: False
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 14
      field:
      stats_result:
      epoch:
      exclude: False

Elasticsearch troubleshooting

Most of the issues with Elasticsearch can be investigated through two sources:

• ES instance logs (written to stdout by default)
• ES REST API (Kibana Dev Tools, or connect to any ES instance)

A lot of GET API support query parameters, for example:

• ?v to include headers;
• ?s=column_name to sort by a column.

Full list of common query option you can find at official docs. Also, filtering by index names supports a kind of math, which might be useful as during ingestion time, also during querying API.

Useful APIs:

- GET /_cat/health?v to get cluster health state
- GET /_nodes To get cluster nodes information (also has heap/cpu usage stats)
- GET /_nodes/stats Stats details explained
- GET /_cluster/settings?include_defaults=true
- GET /_cat/thread_pool?v&s=name returns amount of current thread pools (for example, might help to identify how many bulk requests each node process at the moment)
- GET /_cat/indices?v to list all the indices (includes index health, size, number of documents)
- GET /_cat/shards?v&s=index&s=state&human to list all the shards (for example, can help to find unallocated shards)
- GET /_template gets all the index templates
- GET /{index_name} to get index detailed description
- GET /{index_name}/_recovery?human to get details about shard stages and statuses of these stages (if index state is not healthy, the api gives more clues where to dive in further).
- GET /{index_name}/_mapping to get mapping of exact index (might be useful if some data is rejected by ES because of mapping conflicts)
- GET /_cluster/allocation/explain?pretty with request body {“index": “logs-2020-09-18","shard”:3, “primary”:false}— to garner more information about shard allocation issues. More details at allocation explained article.
- POST /_cluster/reroute?retry_failed=true forces allocation retry. Elasticsearch tries to allocate a shard several times and if all tries fails — the shard stays in failed state. For example, at some point of time, Elasticsearch did not have enough disk space; some time later, one you cleaned up disk or added more nodes, you can retry shard allocation.

The most common troubleshooting scenario:

• check cluster health GET /_cat/health?v
• check nodes load GET /_nodes
• get list of indexes with state of each index GET /_cat/indices?v
• get list of shard with allocation state: GET /_cat/shards?v&s=index&s=state&human
• based on the responses, identify the issue.

If API is not available or any instance is disconnected from the cluster — check instance logs.

Elasticsearch further reading

• five most common problems explained by Datadog.
• Short video tips for ES operations team
• How to setup monitoring of Elasticsearch
• Keeping the cluster Healthy
• Anatomy of Elasticsearch Cluster: part 1, part 2, part 3

Questions?

If you have further questions, you are welcome to contact me in twitter or here in comments ;)