build highly available ELK stack

Hi Techrunnr Readers, this document deals with how to build highly available ELK stack.
Here I will show you how to setup High availability for Elasticsearch.

Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License.

ElasticSeach 1: 192.168.10.5
ElasticSeach 2: 192.168.10.6
ElasticSeach 3: 192.168.10.7
HAproxy for Elasticsearch: 192.168.10.8

1. Log in to each elasticsearch server, download the elasticsearch

cd /opt
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.zip

2. Extract

unzip elasticsearch-6.3.2.zip

3. configure Elasticsearch

cd /opt/elasticsearch-6.3.2/conf

vi elasticsearch.yml


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: production
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.10.5
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["192.168.10.5", "192.168.10.6", "192.168.10.7"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

Change as per your environment
Cluster name: It should be unique in all the servers.

discovery.zen.ping.unicast.hosts: add all your host’s address

Do this for all the servers.

Once it is done start the elasticsearch servers.

cd /opt/elasticsearch-6.3.2/bin

./elasticseach &

Check cluster status

curl -XGET 'http://192.168.10.5:9200/_cluster/state?pretty'

The above command gives you the details about the cluster and the number of nodes present in the cluster.

Now Configure HAProxy for load balancing and failover.

Now login to Elasticsearch Haproxy server (192.168.10.8).
1. Install HAproxy

apt-get install haproxy

2. Remove the default haproxy configuration

rm /etc/haproxy/haproxy.cfg

3. Now Create the configuration file and add the following lines.

vi /etc/haproxy/haproxy.cfg

global
log 127.0.0.1 alert 
log 127.0.0.1 alert debug
maxconn 4096

defaults
log global
mode http
option httplog
option dontlognull
option redispatch
retries 3
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000

####################################
#
# loadbalancer
# 10.0.16.222:8555
# / \
# webServerA webServerB
# 10.0.5.91:8181 10.0.5.92:8181
# (active) (passive)
#
####################################

listen elasticsearch
bind 192.168.10.8:9200
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
option httpclose
option forwardfor
option httplog
option httpchk GET _cluster/health 
server elastic1 192.168.10.5:9200 check inter 5000 downinter 500 # active node
server elastic2 192.168.10.6:9200 check inter 5000 downinter 500 # active node
server elastic3 192.168.10.7:9200 check inter 5000 backup # passive node

Here I have added 3 server which acts as load balancing between elastic1 and elastic2 servers. And Failover server as elastic3 when elastic1 and elastic2 fails.

Now you have completed the HA setup for Elasticsearch

Check this document for Kibana HA.

© 2018, Techrunnr. All rights reserved.

#1
#2
#3
Questions Answered
Articles Written
Overall Points

Prabhin Prabharkaran

He is Technical professional. He is a person who loves to share tricks and tips on the Internet. He Posts what he does!!

Leave a Reply

Please wait...

Subscribe to our newsletter

Want to be notified when our article is published? Enter your email address and name below to be the first to know.