build highly available ELK stack

Hi Techrunnr Readers, this document deals with how to build highly available ELK stack. Here I will show you how to setup High availability for Logstash

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.)

Logstash 1: 192.168.10.2
Logstash 2: 192.168.10.3
HAproxy for Logstash: 192.168.10.4

1. Log in to each logstash server and download the logstash from the following link.

cd /opt/
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz

2. Extract downloaded file

tar -xvzf logstash-6.3.2.tar.gz

3. Configure Logstash.conf

cd logstash-6.3.2/bin

Create a configuration file like this, this may change as per your environment.

vi logstash.conf

input {
tcp {
port => 3200
codec => json_lines
}
}

filter {
mutate {
remove_field => [ "@version", "@metdata", "logstash" ]
}
}

output {
if [stack_trace] {
elasticsearch {
hosts => [ "192.168.10.8" ]
index => "access-log%{+YYYY.MM.dd}"
}
} else {
elasticsearch {
hosts => [ "192.168.10.8" ]
index => "error-log-%{+YYYY.MM.dd}"
}
}
}

In hosts, I’m entering 192.168.10.8 which is my elasticsearch HA proxy server IP.
It means logs received by each logstash server will push to elasticsearch haproxy server after transformation.
Elasticseach Haproxy server will decide which elasticseach server should serve the request.

Now you can start Logstash in each server by executing the following command.

cd /opt/logstash-6.3.2/bin
./logstash -f logstash.conf &

Here is the configuration for haproxy for logstash server.

Java application send the logs to Logstash HAproxy server, then Logstash HAproxy server decides which Logstash server should serve the request as per the HAproxy configuration whether its load balancing or failover.

That is, App will send logs to 192.168.10.4 on port TCP 3200

Now login to Logstash Haproxy server (192.168.10.4).
1. Install HAproxy

apt-get install haproxy

2. Remove the default haproxy configuration

rm /etc/haproxy/haproxy.cfg

3. Now Create the configuration file and add the following lines.

vi /etc/haproxy/haproxy.cfg

global
log 127.0.0.1 alert 
log 127.0.0.1 alert debug
maxconn 4096

defaults
log global
mode http
option httplog
option dontlognull
option redispatch
retries 3
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000

####################################
#
# loadbalancer
# 10.0.16.222:8555
# / \
# webServerA webServerB
# 10.0.5.91:8181 10.0.5.92:8181
# (active) (passive)
#
####################################

listen logstash
bind 192.168.10.4:3200
mode tcp
stats enable
stats auth someuser:somepassword
balance leastconn
server logstash1 192.168.10.2:3200 check inter 2000 downinter 500 # active node
server logstash2 192.168.10.3:3200 check inter 2000 backup # passive node

Here I have configured failover using Haproxy for both logstash server.
By Default logstash1 is the master if it fails haproxy server automatically route all the packets to logstash2.
Once logstash1 becomes available haproxy push all the logs to logstash1 since it the master server. Logstash2 is the backup server.

if you want load balancing between the server edit your configuration file like this.

server logstash1 192.168.10.2:3200 check
server logstash2 192.168.10.3:3200 check

You can balance both server with the following ways,
either round robin, or leastconn
Round-robin – sends the packet to each server one after another.
leastconn – sends the packet to the server which servers fewer requests.

Now you can start haproxy service by executing the following command.

 

/etc/init.d/haproxy start

Now your Logstash high availability is ready. You can test it by stopping any of the logstash servers and check another logstash server is serving the request by using any packet tracer tools such as tcpdump.

Check this article for Elasticseach HA-

© 2018, Techrunnr. All rights reserved.

#1
#2
#3
Questions Answered
Articles Written
Overall Points

Prabhin Prabharkaran

He is Technical professional. He is a person who loves to share tricks and tips on the Internet. He Posts what he does!!

Leave a Reply

Please wait...

Subscribe to our newsletter

Want to be notified when our article is published? Enter your email address and name below to be the first to know.