Technical
Elastic Stack is a group of open source products, which provides a distributed, multi-tenant capable search engine that allows you to search, analyze and visualize data in real-time. Previously known as ELK Stack, it has recently been rebranded as Elastic Stack.
We named this article Truth or Dare since the truth is that using Elastic Stack for log management can become quite a challenge. Let’s see five scenarios of erroneously using Elastic Stack and how to best avoid them.
For the purpose of this article, we've divided the scenarios into two main categories, with scenarios in which:
Category 1. Logs are hard to find
Given:
What we (actually) find in Kibana:
- correlationId
- correlation_id
- crl_id
- …
Erroneous (field names in Elastic):
orderId |
correlationId |
correlation_id |
crl_id |
appName |
1 |
12345 |
|
|
commandService |
1 |
|
12345 |
|
orderService |
1 |
|
|
12345 |
paymentService |
Explanation:
Solution:
host
timestamp
version
long_message
short_message
+custom fields:
application_name
correlation_id
order_id
…
Correct (field names in Elastic):
order_id |
correlation_id |
long_message |
short_message |
application_name |
1 |
12345 |
|
|
commandService |
1 |
12345 |
|
|
orderService |
1 |
12345 |
|
|
paymentService |
Given:
What we (actually) find in Kibana:
Date and time: 03 - 03 - 2022 hour 17:30:00 RO
Timestamp (unix time): 1646321400
Timestamp in milliseconds: 1646321400000
Text date: 202203031530
Stelar date: 99768.75
Erroneous (values in Elastic):
order_id |
order_date |
application_name |
1 |
03 - 03 - 2022 ora 17:30:00 RO |
commandService |
1 |
1646321400 |
orderService |
1 |
1646321400000 |
paymentService |
1 |
202203031530 |
billingService |
1 |
99768.75 |
stockService |
Explanation:
Solution:
Correct (values in Elastic):
order_id |
order_date |
application_name |
1 |
03 - 03 - 2022 ora 17:30:00 RO |
commandService |
1 |
03 - 03 - 2022 ora 17:30:20 RO |
orderService |
1 |
03 - 03 - 2022 ora 17:30:30 RO |
paymentService |
1 |
03 - 03 - 2022 ora 17:30:40 RO |
billingService |
1 |
03 - 03 - 2022 ora 17:30:50 RO |
stockService |
Given:
Example: I search for: “checkout-service”
I can also add a condition (in the search field): “app_name:checkout-service”
What we (actually) find in Kibana:
Example:
Results (when searching for checkout-service):
- checkout-service
- billing-service
- payment-service
- stock-service
etc.
Explanation:
Solution:
{"bool":{ "Must":[{ "Match_phrase":{ "App_name":{ "Query":"checkout-service" } } } ] } }
Example:
Results (when searching for checkout-service with filter):
- checkout-service
Only one result returned (the correct one)
Category 2. Logs are lost
Given:
All the logs from today coming from the checkout_service with “trouble_field” of type STRING will be lost.
What we (actually) find in Kibana:
Explanation:
WHY? Because the type of the fields is determined at the first ingestion of that field
Solution:
Given:
Correct example:
logger.info(Markers.append("smallObjectId", smallObject.getId()), "[getSmallObject]"); => "smallObjectId":1,
Don’t do this example:
logger.info(Markers.appendEntries(loggingMap), "[logMapDontDoThisAtWork]"); => lots of key-value entries (lots of fields)
What we (actually) find in Kibana:
Explanation:
Solution:
CONCLUSION
In conclusion, we should use, as much as possible, the following best practices when working with Elastic Stack:
- consistency in naming
- consistency in content
- filters
- consistency in data types
- limited number of fields
About the Authors:
As a Software Engineer and Technical Team Lead, Oana Brezai designed and implemented solutions for clients in various industries such as retail, banking, insurance, automotive, public administration, and telecom. She is passionate about Information Retrieval and works mostly with open source tools, Elastic Stack being one of them. In her free time, she is actively involved in a public speaking club.
As a Software Engineer, Marius Craus implemented multiple monitoring and tracking tools for clients in the retail business. He is passionate about IoT and IoT software-related projects. In his spare time, he is involved in family life and DIY projects.
USEFUL LINKS