使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践

使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践

前言

Elasticsearch Logstash Kibana(ELK)是一套开源的日志管理方案,分析网站的访问情况时我们一般会借助Google/百度/CNZZ等方式嵌入JS做数据统计,但是当网站访问异常或者被攻击时我们需要在后台分析如Nginx的具体日志,而Nginx日志分割/GoAccess/Awstats都是相对简单的单节点解决方案,针对分布式集群或者数据量级较大时会显得心有余而力不足,而ELK的出现可以使我们从容面对新的挑战。

  • Logstash:负责日志的收集,处理和储存
  • Elasticsearch:负责日志检索和分析
  • Kibana:负责日志的可视化

ELK(Elasticsearch Logstash Kibana)


更新记录

2015年08月31日 – 初稿

阅读原文 – http://wsgzao.github.io/post/elk/

扩展阅读

CentOS 7.x安装ELK(Elasticsearch Logstash Kibana) – http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/
Centos 6.5 安装nginx日志分析系统 elasticsearch logstash redis kibana –http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
logstash-forwarder and grok examples – https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/
三斗室 – http://chenlinux.com/
elastic – https://www.elastic.co/guide
LTMP索引 – http://wsgzao.github.io/index/#LTMP


组件预览

JDK – http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch – https://www.elastic.co/downloads/elasticsearch
Logstash – https://www.elastic.co/downloads/logstash
Kibana – https://www.elastic.co/downloads/kibana
redis – http://redis.io/download

设置FQDN

创建SSL证书的时候需要配置FQDN

#修改hostname
cat /etc/hostname
elk

#修改hosts
cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 10-10-87-19
10.10.87.19 elk.ooxx.com elk

#刷新环境
hostname -F /etc/hostname

#复查结果
hostname -f
elk.ooxx.com

hostname
elk

服务端

Java

cat /etc/redhat-release
CentOS release 6.5 (Final)

yum install java-1.7.0-openjdk
java -version

java version "1.7.0_85"
OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)

Elasticsearch

#下载安装
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm
yum localinstall elasticsearch-1.7.1.noarch.rpm 

#启动相关服务
service elasticsearch start
service elasticsearch status

#查看Elasticsearch的配置文件
rpm -qc elasticsearch

/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf

#查看端口使用情况
netstat -nltp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd  

#测试访问
curl -X GET http://localhost:9200/

Kibana

#下载tar包
wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
#解压
tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv kibana-4.1.1-linux-x64 kibana

#创建kibana服务
vi /etc/rc.d/init.d/kibana

#!/bin/bash
### BEGIN INIT INFO
# Provides:          kibana
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO

# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"

# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin

# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME

# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log

# Begin Script
RETVAL=0

if [ `id -u` -ne 0 ]; then
        echo "You need root privileges to run this script"
        exit 1
fi

# Function library
. /etc/init.d/functions
 
start() {
        echo -n "Starting $DESC : "

pid=`pidofproc -p $PID_FILE kibana`
        if [ -n "$pid" ] ; then
                echo "Already running."
                exit 0
        else
        # Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
                        mkdir $PID_FOLDER
                fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
                sleep 2
                pidofproc node > $PID_FILE
                RETVAL=$?
                [[ $? -eq 0 ]] && success || failure
echo
                [ $RETVAL = 0 ] && touch $LOCK_FILE
                return $RETVAL
        fi
}

reload()
{
    echo "Reload command is not implemented for this service."
    return $RETVAL
}

stop() {
        echo -n "Stopping $DESC : "
        killproc -p $PID_FILE $DAEMON
        RETVAL=$?
echo
        [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
 
case "$1" in
  start)
        start
;;
  stop)
        stop
        ;;
  status)
        status -p $PID_FILE $DAEMON
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  reload)
reload
;;
  *)
# Invalid Arguments, print the following message.
        echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
        ;;
esac

#修改启动权限
chmod  x /etc/rc.d/init.d/kibana

#启动kibana服务
service kibana start
service kibana status

#查看端口
netstat -nltp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           
tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd

Logstash

#下载rpm包
wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
#安装
yum localinstall logstash-1.5.4-1.noarch.rpm 

#设置ssl,之前设置的FQDN是elk.ooxx.com
cd /etc/pki/tls
#openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com
openssl req -subj \'/CN=elk.ooxx.com/\' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
 

#创建一个01-logstash-initial.conf文件
cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF input { lumberjack { port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
EOF

#启动logstash服务
service logstash start
service logstash status

#查看5000端口
netstat -nltp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java           
tcp        0      0 0.0.0.0:9301                0.0.0.0:*                   LISTEN      2309/java           
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd           
tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node           
tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      2309/java           
tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd  


#启动服务
service logstash-forwarder start
service logstash-forwarder status

#访问Kibana,Time-field name 选择 @timestamp
http://localhost:5601/

#增加节点和客户端配置一样,注意同步证书
/etc/pki/tls/certs/logstash-forwarder.crt

客户端

Logstash Forwarder

#登陆到客户端,安装Logstash Forwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm

#查看logstash-forwarder的配置文件位置
rpm -qc logstash-forwarder
/etc/logstash-forwarder.conf

#备份配置文件
cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save

#编辑 /etc/logstash-forwarder.conf,需要根据实际情况进行修改

cat > /etc/logstash-forwarder.conf << EOF
{
  "network": {
    "servers": [ "elk.ooxx.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
      ],

      "fields": { "type": "syslog" }
    }
  ]
}
EOF

配置Nginx日志策略

#修改客户端配置
vi /etc/logstash-forwarder.conf

{
  "network": {
    "servers": [ "elk.ooxx.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
      ],
      "fields": { "type": "syslog" }
    }, {
      "paths": [
        "/app/local/nginx/logs/access.log"
      ],
      "fields": { "type": "nginx" }
    }
  ]
}

#服务端增加patterns
mkdir /opt/logstash/patterns
vi /opt/logstash/patterns/nginx

NGUSERNAME [a-zA-Z\.\@\-\ _%] 
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

#官网pattern的debug在线工具
https://grokdebug.herokuapp.com/

#修改logstash权限
chown -R logstash:logstash /opt/logstash/patterns

#修改服务端配置
vi /etc/logstash/conf.d/01-logstash-initial.conf 

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
  if [type] == "nginx" {
    grok {
       match => { "message" => "%{NGINXACCESS}" }
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

其它注意事项

修改kibana端口

#编辑kibana.yaml
vi /usr/local/kibana/config/kibana.yml


# Kibana is served by a back end server. This controls which port to use.
#port: 5601
port: 80

# The host to bind the server to.
host: "0.0.0.0"

# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "http://localhost:9200"

# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
# then the host you use to connect to *this* Kibana instance will be sent.
elasticsearch_preserve_host: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations
# and dashboards. It will create a new index if it doesn\'t already exist.
kibana_index: ".kibana"

# If your Elasticsearch is protected with basic auth, this is the user credentials
# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied thorugh
# the Kibana server)
# kibana_elasticsearch_username: user
# kibana_elasticsearch_password: pass

# If your Elasticsearch requires client certificate and key
# kibana_elasticsearch_client_crt: /path/to/your/client.crt
# kibana_elasticsearch_client_key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsarech instance, put
# the path of the pem file here.
# ca: /path/to/your/CA.pem

# The default application to load.
default_app_id: "discover"

# Time in milliseconds to wait for elasticsearch to respond to pings, defaults to
# request_timeout setting
# ping_timeout: 1500

# Time in milliseconds to wait for responses from the back end or elasticsearch.
# This must be > 0
request_timeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards.
# Set to 0 to disable.
shard_timeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
# startup_timeout: 5000

# Set to false to have a complete disregard for the validity of the SSL
# certificate.
verify_ssl: true

# SSL for outgoing requests from the Kibana Server (PEM formatted)
# ssl_key_file: /path/to/your/server.key
# ssl_cert_file: /path/to/your/server.crt

# Set the path to where you would like the process id file to be created.
# pid_file: /var/run/kibana.pid

# If you would like to send the log output to a file you can set the path below.
# This will also turn off the STDOUT log output.
# log_file: ./kibana.log

# Plugins that are included in the build, and no longer found in the plugins/ folder
bundled_plugin_ids:
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

JVM调优

#修改elasticsearch.in.sh
vi /usr/share/elasticsearch/bin/elasticsearch.in.sh

if [ "x$ES_MIN_MEM" = "x" ]; then
    ES_MIN_MEM=1g
fi
if [ "x$ES_MAX_MEM" = "x" ]; then
    ES_MAX_MEM=1g

via: http://wsgzao.github.io/

ElasticSearch+LogStash+Kibana+Redis日志服务的高可用方案

1. 高可用方案的架构

在上一篇文章使用ElasticSearch LogStash Kibana Redis搭建日志管理服务中介绍了日志服务的整体框架以及各组件的搭建部署,本篇文章主要讨论一下日志服务框架的高可用方案,主要从以下三个方面考虑:

  • 作为broker的Redis,可以使用redis cluster或者主备结构代替单实例,提高broker组件的可用性;
  • 作为indexer的LogStash,可以部署多个LogStash实例,协作处理日志信息,提高indexer组件的可用性;
  • 作为search&storage的ElasticSearch,采用ElasticSearch Cluster,提高search&storage组件的性能和可用性;

日志服务的高可用方案的示意图为:

下面分别予以介绍。

2. Redis Cluster和主备结构

2.1 Redis Cluster

首先需要部署一个redis cluster,为了方便,我在本机上部署了一个三主三从的cluster,端口分别为:7000, 7001, 7002, 7003, 7004, 7005,以端口7000为例,配置文件为:

include ../redis.conf
daemonize yes
pidfile /var/run/redis_7000.pid
port 7000
logfile /opt/logs/redis/7000.log
appendonly yes
cluster-enabled yes
cluster-config-file node-7000.conf

 

对于redis来说,远程Logstash和中心LogStash都是redis cluster的client,因此只需要与cluster中的任何一个节点连接即可;远程LogStash和中心LogStash的redis配置部分为:

shipper.conf:

output {
    redis {
        host => "20.8.40.49:7000"
        data_type => "list"
        key => "key_count"
    }   
}

 

central.conf:

input {
    redis {
        host => "20.8.40.49"
        port => 7000
        type => "redis-cluster-input"
        data_type => "list"
        key => "key_count"
    }
}

 

使用redis cluster的优劣分析:

  • 可以提高broker组件的可用性:当每一个master节点都有slave节点的时候,任何一个节点挂掉,都不影响集群的正常工作;如果启用cluster-require-full-coverage no,则有效节点构成的子集群仍然可用。
  • 当作为broker组件的redis成为瓶颈的时候,redis cluster提供良好的扩展性。但是redis cluster有一个比较头疼的问题,就是在伸缩(增删节点)时,需要手动做sharding,官方提供了redis-trib.rb工具,我自己实现了一个java版本,可以作为参考redis-toolkit
  • 当前的redis cluster还是RC1版,稳定版还需要等待一段时间。

2.2 redis主备结构

注意,主备不是主从,备用的redis实例只是作为冗余节点,当主节点挂掉时,备用的节点会顶上,任何时刻仅有一个节点在提供服务。无论是远程LogStash还是中心LogStash,都需要明确配置所有主备redis节点的信息,LogStash会轮询节点列表,选择一个可用的节点。比如,配置两个redis实例,6379作为主,6380作为从,则远程LogStash和中心LogStash的配置分别为:

shipper.conf:

output {
    redis {
        host => ["20.8.40.49:6379", "20.8.40.49:6380"]
        data_type => "list"
        key => "key_count"
    }   
}

 

central.conf:

input {
    redis {
        host => "20.8.40.49"
        port => 6379
        type => "redis-cluster-input"
        data_type => "list"
        key => "key_count"
    }

    redis {
        host => "20.8.40.49"
        port => 6380
        type => "redis-cluster-input"
        data_type => "list"
        key => "key_count"
    }
}

 

redis主备结构的优劣分析:

  • 可以提高broker组件的可用性:只要主备节点中有一个节点可用,broker组件服务就是可用的;
  • 主备结构无法解决redis成为瓶颈的情况;

3. 部署多个中心LogStash

当日志信息的量很大时,作为indexer的LogStash很可能成为瓶颈,此时,可以部署多个中心LogStash,它们之间的关系是对等的,共同从broker处提取消息,中心LogStash节点之间是相互独立的。每一个中心LogStash节点的配置是完全一样的,比如当broker是redis cluster时,中心LogStash的配置为:

central.conf:

input {
    redis {
        host => "20.8.40.49"
        port => 7000
        type => "redis-cluster-input"
        data_type => "list"
        key => "key_count"
    }
}

 

部署多个中心LogStash的优劣分析:

  • 可以提高indexer组件的可用性:多个中心LogStash节点之间是相互独立的,任何一个节点的失效不会影响其它节点,更不会影响整个indexer组件;当broker是redis时,各个中心LogStash都是redis的client,它们都是执行BLPOP命令从redis中提取消息,而redis的任何单个命令都是原子的,因此多中心LogStash不仅可以提高indexer组件的可用性,也可以提高indexer组件的处理能力和效率;
  • 多中心LogStash节点的部署和维护成本,可以借助配置管理工具如PuppetSaltStack

4. ElasticSearch Cluster

ElasticSearch原生支持Cluster模式,节点之间通过单播或多播进行通信;ElasticSearch Cluster能自动检测节点的增加、失效和恢复,并重新组织索引。

比如,我们启动两个ElasticSearch实例构成集群,使用默认配置即可,如:

$ bin/elasticsearch -d
$ bin/elasticsearch -d

 

使用默认配置,两个实例的HTTP监听端口分别为9200和9201,它们的节点间通信端口分别为9300和9301,默认使用多播构成集群,集群的名称为elasticsearch;

中心LogStash只需要配置ElasticSearch的cluster name即可(如果不能发现ES集群,可以指定host: host => "20.8.40.49"),如:

output {
    elasticsearch {
        cluster => "elasticsearch"
        codec => "json"
        protocol => "http"
    }   
}

 

使用ElasticSearch Cluster的优劣分析:

  • 提高组件的可用性:cluster中任何一个节点挂掉,索引及副本都会自动重新分配;
  • 极佳的水平扩展性:向cluster中增加新的节点即可,索引会自动重新组织;

5. 后续工作

关于ELKR日志服务,下一步的工作方向有:

  • 学习grok正则表达式,匹配自定义的日志输出格式;
  • 研究ElasticSearch的查询功能及原理;
  • 熟悉Kibana丰富的图标展示功能;

Ok,高可用方案的介绍告一段落,如果用到实际的生产环境中,应该会遇到很多意想不到的问题,后续会继续总结和分享。

参考:

via: http://nkcoder.github.io/

使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务

1. 日志平台的结构示意图

使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务

说明:

  • 多个独立的agent(Shipper)负责收集不同来源的数据,一个中心agent(Indexer)负责汇总和分析数据,在中心agent前的Broker(使用redis实现)作为缓冲区,中心agent后的ElasticSearch用于存储和搜索数据,前端的Kibana提供丰富的图表展示。
  • Shipper表示日志收集,使用LogStash收集各种来源的日志数据,可以是系统日志、文件、redis、mq等等;
  • Broker作为远程agent与中心agent之间的缓冲区,使用redis实现,一是可以提高系统的性能,二是可以提高系统的可靠性,当中心agent提取数据失败时,数据保存在redis中,而不至于丢失;
  • 中心agent也是LogStash,从Broker中提取数据,可以执行相关的分析和处理(Filter);
  • ElasticSearch用于存储最终的数据,并提供搜索功能;
  • Kibana提供一个简单、丰富的web界面,数据来自于ElasticSearch,支持各种查询、统计和展示;

2. 搭建部署

环境:

  • 本机(20.8.40.49)上部署:redis, 中心agent(LogStash), ElasticSearch以及Kibana
  • 远程测试机(20.20.79.75)上部署:独立agent(LogStash)
  • Redis版本:3.0.0-rc1
  • LogStash版本;logstash-1.4.2
  • ElasticSearch版本:elasticsearch-1.3.4
  • Kibana版本:kibana-3.1.1

2.1 部署redis

部署一个redis单机实例:

$ wget https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz
$ tar zxvf 3.0.0-rc1.tar.gz -C /usr/local

 

redis.conf配置文件为:

include ../redis.conf
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
logfile /opt/logs/redis/6379.log
appendonly yes

 

启动:

$ redis.server redis.conf

ip为10.7.40.40, 端口为6379

2.2 部署中心LogStash

下载并解压:

$ wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
$ tar zxvf logstash-1.4.2.tar.gz -C /usr/local/
$ cd /usr/local/logstash-1.4.2
$ mkdir conf logs

配置文件conf/central.conf:

input {
    redis {
        host => "127.0.0.1"
        port => 6379
        type => "redis-input"
        data_type => "list"
        key => "key_count"
    }   
}

output {
    stdout {}
    elasticsearch {
        cluster => "elasticsearch"
        codec => "json"
        protocol => "http"
    }   
}

 

启动:

$ bin/logstash agent --verbose --config conf/central.conf --log logs/stdout.log

配置文件表示输入来自于redis,使用redis的list类型存储数据,key为”key_count”;输出到elasticsearch,cluster的名称为”elasticsearch”;

2.3 部署ElasticSearch

下载并解压:

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.tar.gz
$ tar zxvf elasticsearch-1.3.4.tar.gz -C /usr/local

elasticsearch使用默认配置即可,默认的cluster name为:elasticsearch;

启动:

$ bin/elasticsearch -d

2.4 部署远程LogStash

与部署中心LogStash的步骤是类似的,只是配置文件不一样,使用新的配置文件启动即可;

配置文件conf/shipper.conf的内容为:

input {
    file {
        type => "type_count"
        path => ["/data/logs/count/stdout.log", "/data/logs/count/stderr.log"]
        exclude => ["*.gz", "access.log"]
    }   
}

output {
    stdout {}
    redis {
        host => "20.8.40.49"
        port => 6379
        data_type => "list"
        key => "key_count"
    }   
}

配置文件表示输入来自于目录/data/logs/count/下的stdout.log和stderr.log两个文件,且排除该目录下所有.gz文件和access.log;(这里因为path没有使用通配符,所以exclude是没有效果的);输出表示将监听到的event发送到redis服务器,使用redis的list保存,key为”key_count”,这里的data_type属性和key属性应该与中心agent的配置一致;

2.5 部署Kibana

下载并安装:

$ wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.1.tar.gz
$ tar zxvf kibana-3.1.1.tar.gz

 

修改配置文件config.js,仅需要配置elasticsearch的地址即可:

elasticsearch: "http://20.8.40.49:9200"

 

将目录kibana-3.1.1拷贝到jetty的webapp目录下,并启动jetty:

$ mv kibana-3.1.1 /usr/local/jetty-distribution-9.2.1.v20140609/webapps/
$ bin/jetty start

 

浏览器访问:

http://20.8.40.49:8080/kibana-3.1.1/

 

3. 简单测试

打开LogStash的远程agent和中心agent的日志:

$ tail -f logs/stdout.log

 

远程agent的数据是以rpush操作将event推送到redis的list中,中心agent通过blpop命令从redis的list中提取数据,因此,测试时由于数据量小,通过命令llen key_count的返回结果很可能为空,因此为了观察redis中数据流的变化,可以使用monitor命令:

$ redis-cli -p 6379 monitor

 

现在,我们向/data/logs/count目录下的stdout.log和stderr.log各发送一条数据:

$ echo "stdout: just a test message" >> stdout.log
$ echo "stderr: just a test message" >> stderr.log

 

远程agent和中心agent都会收到event消息,如远程agent的日志为:

{:timestamp=>"2014-10-31T09:30:40.323000 0800", :message=>"Received line", :path=>"/data/logs/count/stdout.log", :text=>"stdout: just a test message", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"134"}
{:timestamp=>"2014-10-31T09:30:40.325000 0800", :message=>"writing sincedb (delta since last write = 52)", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"177"}
......
{:timestamp=>"2014-10-31T09:30:49.350000 0800", :message=>"Received line", :path=>"/data/logs/count/stderr.log", :text=>"stderr: just a test message", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"134"}
{:timestamp=>"2014-10-31T09:30:49.352000 0800", :message=>"output received", :event=>{"message"=>"stderr: just a test message", "@version"=>"1", "@timestamp"=>"2014-10-31T01:30:49.350Z", "type"=>"type_count", "host"=>"dn1", "path"=>"/data/logs/count/stderr.log"}, :level=>:debug, :file=>"(eval)", :line=>"19"}

 

我们可以观察到redis的输出:

1414714174.936642 [0 20.20.79.75:54010] "rpush" "key_count" "{\"message\":\"stdout: just a test message\",\"@version\":\"1\",\"@timestamp\":\"2014-10-31T00:10:04.530Z\",\"type\":\"type_count\",\"host\":\"dn1\",\"path\":\"/data/logs/count/stdout.log\"}"
1414714174.939517 [0 127.0.0.1:56094] "blpop" "key_count" "0"
1414714198.991452 [0 20.20.79.75:54010] "rpush" "key_count" "{\"message\":\"stderr: just a test message\",\"@version\":\"1\",\"@timestamp\":\"2014-10-31T00:10:28.586Z\",\"type\":\"type_count\",\"host\":\"dn1\",\"path\":\"/data/logs/count/stderr.log\"}"
1414714198.993590 [0 127.0.0.1:56094] "blpop" "key_count" "0"

 

从elasticsearch中执行如下的简单查询:

$ curl \'localhost:9200/_search?q=type:type_count&pretty\'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 0.5945348,
    "hits" : [ {
      "_index" : "logstash-2014.10.31",
      "_type" : "type_count",
      "_id" : "w87bRn8MToaYm_kfnygGGw",
      "_score" : 0.5945348,
      "_source":{"message":"stdout: just a test message","@version":"1","@timestamp":"2014-10-31T08:10:04.530 08:00","type":"type_count","host":"dn1","path":"/data/logs/count/stdout.log"}
    }, {
      "_index" : "logstash-2014.10.31",
      "_type" : "type_count",
      "_id" : "wwmA2BD6SAGeNsuYz5ax-Q",
      "_score" : 0.5945348,
      "_source":{"message":"stderr: just a test message","@version":"1","@timestamp":"2014-10-31T08:10:28.586 08:00","type":"type_count","host":"dn1","path":"/data/logs/count/stderr.log"}
    } ]
  }
}

 

再切换到Kibana的web界面:http://20.8.40.49:8080/kibana-3.1.1

使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务

4. 后续工作


  • 使用LogStash的Filter对日志数据进行过滤和分析;
  • 使用Redis的Cluster模式替换单机模式;
  • 在elasticsearch中对数据进行高级的分析和查询;
  • 熟悉Kibana的展示组件以及查询语法;

参考:

via: http://nkcoder.github.io