Logging With the Elastic Stack

gowebing

Applications generate logs because they serve as a mirror to their state and health.

With the voluminous amount of generated logs, it becomes imperative to have a system that can analyze these logs and present a singular view of the application. When the application is deployed in a distributed environment, maintaining and retrieving the logs can be challenging.

Searching for an error across several servers and through a large number of log files is extremely difficult.

The Open-Source Elastic Stack

Centralized logging provided by the Elastic Stack is a step in this direction. It allows searching through all logs at a central place. The Elastic Stack is a versatile collection of open-source software tools that are implemented based on a distributed log collector approach that makes gathering insights from data easier. It is also referred to as the ELK stack (Elasticsearch, Logstash, Kibana).

Elastic stack overview

Logical Architecture

The below image indicates logical architecture of the Elastic Stack.

Logical Architecture

Different kinds of logs are generated like system log files, database log files, logs generated by message queues, and other middlewares. These logs are collected by Filebeat installed on all servers producing logs. Filebeat sends the logs to Redis for store and forward. Redis serves as a temporary storage area for logs coming from beats. It further supplies these logs to Logstash at a fixed interval.

Logstash transforms the logs and processes them as configured. Elasticsearch will then index and store these logs. Finally, Kibana provides a visual interface for searching and analyzing the logs.

Deployment Architecture

A suggested deployment of the Elastic Stack is given below. It ensures high availability and load balancing.

Deployment Architecture

Logstash Overview

Logstash is typically configured as given below.

Logstash configuration

Below is a sample Filebeat configuration:

filebeat.prospectors:
- type: log
  # Change to true to enable this prospector onfiguration.
 enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - d:\ElasticStack\Logs\logfile.log
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Below is the Logstash configuration:

input {
    beats {
        port => "5044"
    }
}
 filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Elasticsearch Overview

Elasticsearch is a distributed, RESTful search and analytics engine. It allows us to store, search, and analyze big volumes of data quickly and in near real time.

A quick review of some of the basic concepts:

To summarize, each index can be split into multiple shards. An index can also be replicated zero (meaning no replicas) or more times. Once replicated, each index will have primary shards (the original shards that were replicated from) and replica shards (the copies of the primary shards).

Below is the Elasticsearch configuration:

cluster.name: elasticsearch                   # Cluster name
node.name: "Node 1“                           # Node name
node.master: false                            # Allow this node to be eligible as a master node
node.data: true                               # Allow this node to store data
index.number_of_shards: 5                     # Set  the number of shards (splits) of an index
index.number_of_replicas: 1                   # Set the number of replicas (additional copies) of an index
path.conf: /elasticsearch-1.4.4/config/conf   # Path to directory containing configuration
path.data: /elasticsearch-1.4.4/data          # Path to directory where to store index data allocated for this node.
path.work: /elasticsearch-1.4.4/work          # Path to temporary files
network.bind_host: 192.168.0.1                # Binds itself to this address, and listens on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node communication.
http.port: 9200                               # Set a custom port to listen for HTTP traffic

Kibana Overview

Kibana provides data visualization capability. It has a browser-based UI. It is the analytics and visualization components of the Elastic Stack. Kibana provides various kinds of charts like histograms, line graphs, and pie charts. It integrates easily with Elasticsearch. It has easy-to-share reports as PDFs, CSVs, embed links, etc.

Below is Kibana configuration:

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
server.host: "localhost"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: http://localhost:9200
elasticsearch.username: "user“
elasticsearch.password: "pass"

To generate Kibana reports, first, ensure that data is loaded in Elasticsearch and then define an index pattern and write a query to discover data within the indexes meeting the pattern criteria. To visualize data, choose from a variety of diagrams available and finally glue your preferred diagrams on a customizable dashboard.

 

 

 

 

Top