build highly available ELK stack2 min read
build highly available ELK stack
Hi Techrunnr Readers, this document deals with how to build highly available ELK stack.
Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License.
Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.)
Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.
Here we are using 10 Servers for setting high availability for each of the components.
3 Elastic Search servers are clustered and load balancer (with failover) using HAProxy.
2 Logstash servers failover is configured with HAProxy.
2 Kibana viewer servers are configured with failover using Apache web server proxy pass
Here is the topology diagram which I’m are trying to achieve.
I have a simple java application which sends the logs to Logstash server using TCP 3200. Logstash server sends the received logs after filtering to the elastic search server for indexing. Kibana server takes the logs from the elastic search server and provides a visualization to the system admin to see the logs in the form of charts, graphs etc.
Here I’m using an HAProxy server for failover configuration for Logstash and Elastic Search.
HAProxy is free, open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers. It is written in C and has a reputation for being fast and efficient.
Here is the IP assigned for each server in My environment.
Logstash 1: 192.168.10.2
Logstash 2: 192.168.10.3
HAproxy for Logstash: 192.168.10.4
ElasticSeach 1: 192.168.10.5
ElasticSeach 2: 192.168.10.6
ElasticSeach 3: 192.168.10.7
HAproxy for Elasticsearch: 192.168.10.8
Kibana 1: 192.168.10.9
kibana 2: 192.168.10.10
Apache 3: 192.168.10.11
See part 2 for setting Logstash server High availability- https://www.techrunnr.com/build-highly-available-elk-stack-part2/