Elastic Stack: Truth or Dare?

Elastic Stack is a group of open source products, which provides a distributed, multi-tenant capable search engine that allows you to search, analyze and visualize data in real-time. Previously known as ELK Stack, it has recently been rebranded as Elastic Stack.

We named this article Truth or Dare since the truth is that using Elastic Stack for log management can become quite a challenge. Let’s see five scenarios of erroneously using Elastic Stack and how to best avoid them.

For the purpose of this article, we've divided the scenarios into two main categories, with scenarios in which:

1. Logs are hard to find
2. Logs are lost

Category 1. Logs are hard to find

SCENARIO I

Given:

• lots of microservices communicating via REST
• usage of a field named correlation_id or any other business_key to find log entries from different services

What we (actually) find in Kibana:

• we will find the following columns in Kibana (among other columns):

- correlationId

- correlation_id

- crl_id

- …

Erroneous (field names in Elastic):

Explanation:

• Elastic Stack does not impose specific names to fields

Solution:

• use a common format (for all microservices)
• example: GELF (Graylog Extended Log Format)

host

  timestamp

version

long_message

short_message

+custom fields:

application_name

correlation_id

order_id

…

Correct (field names in Elastic):

SCENARIO 2

Given:

• lots of microservices
• we search for a certain date / hour / minute in fields with the same name, belonging to different microservices

What we (actually) find in Kibana:

• we will find:

Date and time: 03 - 03 - 2022  hour 17:30:00 RO

Timestamp (unix time): 1646321400

Timestamp in milliseconds: 1646321400000

Text date: 202203031530

Stelar date: 99768.75

Erroneous (values in Elastic):

Explanation:

• Elastic Stack does not impose a specific date format

Solution:

• use the same format in different microservices for fields having the same name

Correct (values in Elastic):

SCENARIO III

Given:

• lots of microservices communicating via REST
• we are looking for a generic text

Example: I search for: “checkout-service”

I can also add a condition (in the search field): “app_name:checkout-service”

What we (actually) find in Kibana:

• we will find lots of (irrelevant results)
• the query will last long

Example:

Results (when searching for checkout-service):

- checkout-service

- billing-service

- payment-service

- stock-service

etc.

Explanation:

• Elastic Stack uses full text search (it ignores the “-” sign and search for both words in all fields in all entry logs)

Solution:

• use filters

{"bool":{
"Must":[{
"Match_phrase":{
"App_name":{
"Query":"checkout-service"
}
}
}
]
}
}

Example:

Results (when searching for checkout-service with filter):

- checkout-service

Only one result returned (the correct one)

Category 2. Logs are lost

SCENARIO IV

Given:

• 2 microservices: order_service and checkout_service
• “trouble_field” is of type number in order_service
• “trouble_field” is of type string in checkout_service
• the first entry in the logs of today is from order_service
• “trouble_field” is created with type: NUMBER

All the logs from today coming from the checkout_service with “trouble_field” of type STRING will be lost.

What we (actually) find in Kibana:

• we will observe that some logs cannot be found at all in Elastic

Explanation:

• if the first entry in the logs of today is from order_service
• then “trouble_field” is made of NUMBER type
• all the logs coming from checkout_service that have “trouble_field” STRING will not be stored in Elastic

WHY? Because the type of the fields is determined at the first ingestion of that field

Solution:

• consistency of types for fields with the same name in different microservices

SCENARIO V

Given:

• lots of microservices communicating via REST
• logs using markers

Correct example:

logger.info(Markers.append("smallObjectId", smallObject.getId()), "[getSmallObject]");
=> "smallObjectId":1,

Don’t do this example:

logger.info(Markers.appendEntries(loggingMap), "[logMapDontDoThisAtWork]");
=> lots of key-value entries (lots of fields)

What we (actually) find in Kibana:

• we will observe that some logs cannot be found at all in Elastic

Explanation:

• a new field is created in Elastic for each marker
• if we add N markers → N new fields in Elastic
• if we reach the default limit of field number (1000), all the log entries containing new fields will be rejected (will be lost)

Solution:

• the solution does not mean increasing the default limit
• the real solution is to control the number of fields

CONCLUSION

In conclusion, we should use, as much as possible, the following best practices when working with Elastic Stack:

- consistency in naming

- consistency in content

- filters

- consistency in data types

- limited number of fields

About the Authors:

As a Software Engineer and Technical Team Lead, Oana Brezai designed and implemented solutions for clients in various industries such as retail, banking, insurance, automotive, public administration, and telecom. She is passionate about Information Retrieval and works mostly with open source tools, Elastic Stack being one of them. In her free time, she is actively involved in a public speaking club.

As a Software Engineer, Marius Craus implemented multiple monitoring and tracking tools for clients in the retail business. He is passionate about IoT and IoT software-related projects. In his spare time, he is involved in family life and DIY projects.