Enable elasticsearch slow log3 min read

We need your support!!
Other Amount:
techrunnr.com:
24-Hour Flash Sale. Courses from just ₹ 490.
Prabhin Prabharkaran Administrator
DevOps Engineer

He is a Technical professional. He is a person who loves to share tricks and tips on the Internet. He Posts what he does!

follow me

As a DevOps engineer, you might see some users complaining about the elastic search that searching the logs is very slow. And in some scenarios, this situation is cannot be reproducible. So it will be better if we log a slow log of search queries.

you might have seen the same kind of application performance issue while using Mysql/Mariadb. During this kind of scenario, DBA ( database administrator ) enables the slow query log which helps developers to understand which log is taking more time and can fix.

Luckily elastic search supports slow log, where the admin can enable slow log based on the time taking to fetch the data.

Check this link to understand what is the difference between query and fetch in elastic search.

How to enable?

Login to Kibana ->  dev tools

To enable for all the index

 

PUT /_all/_settings
{
"index.search.slowlog.threshold.query.warn": "15s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "5s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms",
"index.search.slowlog.level": "info"
}

make sure you update the necessary field time.

If you want only a warning alert for query time of more than 15s, you can ignore info, debug and trace.

Once you enabled the log you will get the below response

{
acknowedge: true
}

You can verify your setting has been applied by running the below API call

 

GET /_all/_settings

 

          "level" : "info",
            "threshold" : {
              "fetch" : {
                "warn" : "5s",
                "trace" : "800ms",
                "debug" : "500ms",
                "info" : "200"
              },
              "query" : {
                "warn" : "15s",
                "trace" : "500ms",
                "debug" : "2s",
                "info" : "5s"
              }
            }
          }

 

To enable slow log only for particular index.

PUT /my-index-000001/_settings

{
"index.search.slowlog.threshold.query.warn": "15s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "5s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms",
"index.search.slowlog.level": "info"
}

You can verify your setting has been applied by running the below API call

 

GET /my-index-000001/_settings

 

You can see the output of the slow in the stdout of the elastic search log.

If you use Kubernetes and fluentd for pushing the logs to elk stack you can easily filter the logs over a period of time.

Sample log will be like this,

[2030-08-30T11:59:37,786][WARN ][i.s.s.query ] [node-0] [index6][0] took[78.4micros], took_millis[0], total_hits[0 hits], stats[], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"query":{"match_all":{"boost":1.0}}}], id[MY_USER_ID],

 

We need your support!!
Other Amount:
techrunnr.com:
#1
#2
#3
Questions Answered
Articles Written
Overall Points

Prabhin Prabharkaran

He is Technical professional. He is a person who loves to share tricks and tips on the Internet. He Posts what he does!!

You may also like...

Leave a Reply