Logit is an ISO 27001 certified centralised logging and metrics management company. Our platform delivers you with a fully customised log and metrics solution based on Elasticsearch, Logstash and Kibana which is scalable, secure and compliant.

Build a hot-warm-cold Elasticsearch cluster for log analytics: data tier node roles, an ILM policy that moves logs between tiers, and Filebeat and Logstash to ship them.

The Elastic toolset for monitoring and observability with the Beats tools (heartbeat, filebeat etc) and more recently an integrated OTel compliant agent have been attractive because of the strength of the analysis capabilities of Elasticsearch, Logstash and Kibana. All the products are available as open source and can be extended (although with 3rd party constraints as a result of licensing) and with enterprise extensions.

You can use Filebeat Docker images on Kubernetes to retrieve and ship container logs. You deploy Filebeat as a DaemonSet to ensure there’s a running instance...

Vulnerability-Lookup - Fast vulnerability lookup correlation from different sources.

We benchmarked vlagent, Vector, Fluent Bit, Filebeat, Fluentd, Promtail, Grafana Alloy, and OpenTelemetry Collector on throughput, resource usage, and delivery correctness - and found correctness issu...

Ever wondered how real-world security teams monitor and analyze logs across systems? Let’s build a...

devops stacks in 2025 podman cockpit filebeat kafka redis logstash elastic kibana zookeeper

In today's fast-paced digital landscape, efficient log management is not just a convenience—it's a necessity. Logs are fundamental to diagnosing errors, tracking performance, and ensuring the overall health of infrastructure and applications. One of the most effective and scalable solutions for centralized logging is the combination of **Filebeat** , **Logstash** , and **Elasticsearch** , commonly referred to as part of the ELK stack. When used together, they provide a flexible pipeline for shipping, transforming, and storing logs, which can then be analyzed and visualized using **Kibana**. This comprehensive guide walks you through a practical setup of Filebeat and Logstash integration, with step-by-step instructions, configuration samples, troubleshooting tips, and best practices to help you get the most out of your logging infrastructure. ## Brief Overview of Technologies Let’s start by understanding the roles each tool plays in the pipeline: 1. **Filebeat** : A lightweight data shipper designed to forward and centralize log data. It reads logs from files, tailing them in near-real-time, and forwards them to Logstash or Elasticsearch. 2. **Logstash** : A flexible data processing pipeline capable of ingesting data from multiple sources. It transforms and enriches the data using a powerful plugin system before sending it to Elasticsearch. 3. **Elasticsearch** : A distributed search and analytics engine. It indexes the incoming structured data and makes it queryable with high performance. 4. **Kibana** : A web-based frontend that connects to Elasticsearch and provides visualization, dashboards, and powerful querying tools. ## Setting Up the Environment ### Pre-requisites To follow along with this guide, make sure you have the following components: * Two servers or virtual machines: * **Server A** : Hosts the ELK stack (Elasticsearch, Logstash, Kibana). * **Server B** : Acts as the log source and hosts Filebeat. * Filebeat installed on Server B. * Logstash, Elasticsearch, and Kibana installed and running on Server A. * Basic familiarity with YAML, Linux CLI, and service management (systemctl). * Open TCP port 5044 on Server A (for Logstash to receive Filebeat logs). > **Pro tip:** Consider using Docker or Docker Compose to quickly spin up the ELK stack for testing purposes. ### Configuration Directory Structure It’s essential to know where configuration files live on your system: * **Logstash** configs: `/etc/logstash/conf.d/` * **Filebeat** config: `/etc/filebeat/filebeat.yml` * **Log files** : Typically in `/var/log/` ## Configuring Logstash to Receive Logs We’ll begin by setting up Logstash to listen for incoming data from Filebeat. 1. Navigate to Logstash’s configuration directory: cd /etc/logstash/conf.d 1. Create a new configuration file, e.g., `filebeat-input.conf`: nano filebeat-input.conf 1. Insert the following configuration to define input and output: input { beats { port => 5044 } } filter { # Optional: add filters here } output { elasticsearch { hosts => ["http://localhost:9200"] index => "filebeat-%{+YYYY.MM.dd}" } } 1. Restart Logstash: systemctl restart logstash 1. Confirm that Logstash is listening on port 5044: netstat -tulnp | grep 5044 ## Configuring Filebeat to Ship Logs 1. Edit Filebeat's main configuration file: nano /etc/filebeat/filebeat.yml 1. Comment out the Elasticsearch output to avoid direct shipping: #output.elasticsearch: # hosts: ["localhost:9200"] 1. Enable the Logstash output and specify the Logstash server: output.logstash: hosts: ["<LOGSTASH-IP>:5044"] 1. Optionally define inputs explicitly: filebeat.inputs: - type: log enabled: true paths: - /var/log/syslog - /var/log/auth.log 1. Restart Filebeat to apply the configuration: systemctl restart filebeat 1. Check Filebeat logs for errors: tail -f /var/log/filebeat/filebeat.log ## Sending Sample Logs to Validate the Setup You can simulate logs manually: logger "This is a test log from Filebeat" Then navigate to Kibana: 1. Open Kibana in your browser: http://<KIBANA-IP>:5601 1. Go to "Stack Management > Index Patterns" and create a new pattern: filebeat-* 1. Head to "Discover" and search for your sample log. ## Parsing and Transforming Logs with Logstash Filters Logstash allows you to extract fields and format logs for improved querying. ### Example: Parsing Nginx Logs filter { grok { match => { "message" => "%{IPORHOST:client} %{USER:ident} %{USER:id} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } } ### Example: JSON Logs filter { json { source => "message" } } > **Tip:** Always test your filters using the Grok Debugger in Kibana or online. ## Working with Filebeat Modules Filebeat includes modules for common applications like Nginx, Apache, MySQL, and System logs. 1. Enable a module: filebeat modules enable nginx 1. List enabled modules: filebeat modules list 1. Test configuration: filebeat test config 1. Run Filebeat setup: filebeat setup --dashboards This automatically configures dashboards and parsers for supported logs. ## Monitoring and Troubleshooting ### Key Log Locations * Filebeat logs: `/var/log/filebeat/filebeat.log` * Logstash logs: `/var/log/logstash/logstash-plain.log` * Elasticsearch logs: `/var/log/elasticsearch/` Use `tail -f` to monitor logs in real time: tail -f /var/log/logstash/logstash-plain.log ### Common Issues * **Connection Refused** : Check if Logstash is listening on the right port. * **Permission Denied** : Ensure Filebeat has access to log files. * **Pipeline Errors** : Validate Logstash config with `--config.test_and_exit` ## Creating Dashboards in Kibana 1. Navigate to **Dashboard > Create New Dashboard** 2. Add visualizations like: * Line chart of logs over time * Pie chart of log sources * Table of error messages ## Best Practices * **Use structured logs** : JSON logs are easier to parse and index. * **Limit fields** : Reduce unnecessary fields to improve indexing performance. * **Secure communication** : * Use TLS for Filebeat to Logstash * Enable Elasticsearch authentication * Use API keys or users with limited permissions ## Conclusion Integrating Filebeat with Logstash and Elasticsearch provides a robust, scalable logging solution. Whether you're debugging a failed deployment or analyzing traffic spikes, a centralized logging pipeline helps you act fast and make informed decisions. With modules, filters, and dashboards, you can tailor the solution to any infrastructure. Invest the time to monitor, fine-tune, and secure your pipeline—it will pay off in visibility and system reliability. ## FAQs 1. **What benefits does this integration provide?** Centralization, consistency, and query power. 2. **Can it handle high-volume logs?** Yes, with proper tuning, buffering, and horizontal scaling. 3. **How can I enrich logs with metadata?** Use the `add_fields` processor in Filebeat or enrichments in Logstash. 4. **Is there a way to archive old logs?** Use Elasticsearch ILM (Index Lifecycle Management). 5. **How to make this setup production-ready?** Add monitoring (Metricbeat), enable security features, and use backups.

**Standard Operating Procedure (SOP): Installation of ELK Stack with Filebeat on AWS Servers** ### **1. Purpose** This SOP outlines the step-by-step process for installing and configuring the ELK (Elasticsearch, Logstash, and Kibana) stack with Filebeat on AWS servers. The document includes prerequisites, system requirements, cluster setup, and AWS-specific configurations. ### **2. Prerequisites** * **AWS Account** : Access to an active AWS account with appropriate IAM permissions. * **Networking** : * VPC created for deployment. * Subnets for public/private resources. * Security groups with the required ports opened (refer to the Ports section). * **System Access** : * SSH key pair for accessing EC2 instances. * Admin/root privileges on servers. * **Software Tools** : * AWS CLI installed and configured locally. * Terraform/CloudFormation (optional for automation). * Remote terminal such as PuTTY or a compatible SSH client. ### **3. System Requirements** #### **Compute Requirements** Component | Instance Type | Minimum vCPU | Minimum Memory ---|---|---|--- Elasticsearch | t3.medium or higher | 2 | 8 GB Logstash | t3.medium or higher | 2 | 4 GB Kibana | t3.small or higher | 1 | 2 GB Filebeat Agents | t2.micro or higher | 1 | 1 GB #### **Storage Requirements** Component | Disk Type | Minimum Storage ---|---|--- Elasticsearch | SSD (gp3) | 50 GB Logstash | SSD (gp3) | 10 GB Kibana | General HDD | 10 GB #### **Ports to Open** Service | Protocol | Port ---|---|--- Elasticsearch | HTTP/HTTPS | 9200 Kibana | HTTP | 5601 Logstash | TCP/UDP | 5044 Filebeat | Outbound HTTP | 9200 ### **4. Cluster Creation** #### **AWS Perspective** 1. **VPC Setup** : * Create a VPC and enable DNS hostnames. * Set up two subnets (public and private) for different components. * Create an Internet Gateway (IGW) and attach it to the VPC. 2. **Security Groups** : * Allow inbound traffic for required ports in the security group associated with EC2 instances. * Restrict access to trusted IP ranges for Kibana and Elasticsearch. 3. **EC2 Instances** : * Launch EC2 instances for Elasticsearch, Logstash, Kibana, and Filebeat. * Use Amazon Linux 2 or Ubuntu 20.04 for compatibility. * Allocate Elastic IPs for external access if required. 4. **IAM Role** : * Attach an IAM role with S3 and CloudWatch permissions for backup and monitoring. #### **Landing Zone Perspective** 1. Use AWS Landing Zone (if applicable) to standardize account setup and ensure governance. 2. Configure logging and monitoring through AWS CloudTrail and AWS CloudWatch. 3. Ensure tagging standards are applied for resource identification. ### **5. Installation Steps** #### **Elasticsearch** 1. **Install Elasticsearch** : sudo apt update wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - sudo apt install apt-transport-https echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list sudo apt update && sudo apt install elasticsearch 1. **Configure Elasticsearch** : * Update `/etc/elasticsearch/elasticsearch.yml` with the following: network.host: 0.0.0.0 cluster.name: elk-cluster * Restart the service: sudo systemctl enable elasticsearch sudo systemctl start elasticsearch #### **Logstash** 1. **Install Logstash** : sudo apt install logstash 1. **Configure Logstash** : * Create a configuration file at `/etc/logstash/conf.d/logstash.conf`: input { beats { port => 5044 } } output { elasticsearch { hosts => ["http://<elasticsearch-IP>:9200"] index => "logs-%{+YYYY.MM.dd}" } } * Start the service: sudo systemctl enable logstash sudo systemctl start logstash #### **Kibana** 1. **Install Kibana** : sudo apt install kibana 1. **Configure Kibana** : * Update `/etc/kibana/kibana.yml`: server.host: "0.0.0.0" elasticsearch.hosts: ["http://<elasticsearch-IP>:9200"] * Restart the service: sudo systemctl enable kibana sudo systemctl start kibana #### **Filebeat** 1. **Install Filebeat** : sudo apt install filebeat 1. **Configure Filebeat** : * Update `/etc/filebeat/filebeat.yml`: filebeat.inputs: - type: log paths: - /var/log/*.log output.logstash: hosts: ["<logstash-IP>:5044"] * Start Filebeat: sudo systemctl enable filebeat sudo systemctl start filebeat ### **6. Post-Installation Verification** 1. **Elasticsearch** : * Verify connectivity: curl http://<elasticsearch-IP>:9200 2. **Logstash** : * Check for log ingestion in Elasticsearch. 3. **Kibana** : * Access the Kibana UI at `http://<kibana-IP>:5601` and configure an index pattern. 4. **Filebeat** : * Confirm logs are being sent to Logstash and indexed in Elasticsearch. ### **7. Maintenance and Monitoring** 1. **Backup** : * Configure snapshot backups to an S3 bucket. 2. **Monitoring** : * Use Elastic Stack’s monitoring features or integrate with AWS CloudWatch. 3. **Scaling** : * Use AWS Auto Scaling Groups for horizontal scaling of Elasticsearch nodes. ### **8. Troubleshooting** * **Logs** : * Check service logs for Elasticsearch (`/var/log/elasticsearch`), Logstash (`/var/log/logstash`), and Kibana (`/var/log/kibana`). * **Network Issues** : * Verify VPC routing tables and security group configurations. * **Resource Bottlenecks** : * Monitor instance metrics and upgrade instance types or increase storage as needed. This SOP ensures a standardized deployment of the ELK stack with Filebeat on AWS, facilitating efficient log management and monitoring.