These days a typical deployment consists of at least a couple of servers, behind a load balancer and a database. As your application complexity grows, you continue to add more components. Then you start collecting tons of messages from logs of all these applications. And the problem starts there! It is not easy to keep track of logs from all these components and detect patterns or anomalies which could trigger further action. Logstash helps solve this problem.
What is Logstash?
In simple terms, Logstash is a data processing pipeline, that ingests data from one or more disparate sources. It transforms it as per your configuration and then sends it to the destination.
The source could be a web server log, application logs, metrics, time series data or even unstructured data. Logstash can handle it all. The possible transformations are deriving structure from unstructured data using Grok, separate out IP addresses, anonymize personally identifiable information (PII). This is achieved using any of the filters available from this list.
Ecosystem
Logstash forms part of the ELK stack that is promoted by Elastic. ELK is a stack comprising of Elasticsearch, Logstash and Kibana. All three products have their open source versions and are very popular amongst the community.
Reference Links
Related Keywords
Elasticsearch, Kibana, Kubernetes, Prometheus