close and open elasticsearch indexes

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java. In this document, we will show how to open and close the indexes. Here is one of the uses where you might need to open and close the indexes. let’s assume that you are restoring elasticsearch snapshots from the repository using a curator. If you don’t know how to backup and restore elasticsearch data, then check out this…

Read More

how to install clickhouse

Hi All, this document deals with how to install click house ClickHouse is an open-source column-oriented DBMS for online analytical processing. ClickHouse was developed by the Russian IT company Yandex for the Yandex. Metrica web analytics service. ClickHouse allows analysis of data that is updated in real time. The system is marketed for high performance Step 1: check if your system is compatible for the installation. Check if your system allows SSE(Streaming SIMD Extensions). if the output is SSE 4.2 supported then proceed with the further installations. Step 2: Install…

Read More

how ELK works?

Hi All, this document gives you an overview of how ELK works. ELK stands for Elastic Search, Logstash and Kibana. How does it work? Logs from the server can be shipped to the logstash server using file beat or any other alternative software. Or you can use a logstash individual server to do this same operation. Once logs are received to logstash it goes through the input and filter section. In the filter section of the logstash configuration file, you can map values and filter unwanted the string using the…

Read More

how to increase the number of nodes in AWS EMR

Hi All, this document deals with how to increase the number of nodes in AWS EMR. Prerequisites Working AWS EMR cluster, if you don’t have to check this link. Configuration Step 1: Login to AWS console and select EMR from the services. Step 2: Choose your cluster. Step 3: In the hardware section there is an option to resize, choose that. Step 4: Now you can see the current worker node, instance count. Step 5: Increase the instance count as per your requirement, In my case, I’m increasing to 3.…

Read More

how to setup AWS EMR

Hi All, This document deals with how to setup AWS EMR. Amazon Elastic MapReduce (EMR) is an Amazon Web Services (AWS) tool for big data processing and analysis. Amazon EMR offers the expandable low-configuration service as an easier alternative to running in-house cluster computing. Here will show how to set up EMR. Step 1: Login to AWS console and select EMR from the services list. Step 2: Now Click on Create cluster from EMR Dashboard. Step 3: Configure as follows Enter the Cluster name Enable Logging if you want, if…

Read More

build highly available ELK stack-part4

build highly available ELK stack Hi Techrunnr Readers, this document deals with how to build highly available ELK stack. Here I will show you how to setup High availability for kibana with apache. Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data. Kibana 1: 192.168.10.9 kibana 2: 192.168.10.10 Apache 3: 192.168.10.11 1. Login to each…

Read More

build highly available ELK stack-part3

build highly available ELK stack Hi Techrunnr Readers, this document deals with how to build highly available ELK stack. Here I will show you how to setup High availability for Elasticsearch. Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. ElasticSeach 1: 192.168.10.5 ElasticSeach 2: 192.168.10.6 ElasticSeach 3: 192.168.10.7 HAproxy for Elasticsearch: 192.168.10.8 1. Log in…

Read More

build highly available ELK stack – part2

build highly available ELK stack Hi Techrunnr Readers, this document deals with how to build highly available ELK stack. Here I will show you how to setup High availability for Logstash Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.) Logstash 1: 192.168.10.2 Logstash 2: 192.168.10.3 HAproxy for Logstash: 192.168.10.4 1. Log in to each logstash server and download the logstash from the following link. 2. Extract…

Read More

build highly available ELK stack

build highly available ELK stack Hi Techrunnr Readers, this document deals with how to build highly available ELK stack. Elasticsearch Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Logstash Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously transforms it, and then sends it to your…

Read More

authentication for kibana using nginx

Kibana authentication using nginx Hello Techrunnr Readers, This document deals with enabling authentication for kibana using nginx webserver. Kibana is opensource visulalization and analytics tools which works with Logstash and Elasticsearch. By default, kibana doesn’t have any authentication by default. Running kibana in the local machine without authentication doesn’t make security threat, but when you are setting up kibana publically it’s a major threat. Here we are going to achieve this by using create an htaccess user and configuring the proxy pass for kibana. This provides one more layer of…

Read More