从ELK 到 EFK 的演进：日志平台构建实践
共 7259字，需浏览 15分钟
·
2021-05-19 06:57
整体架构
Filebeat: 6.2.4Kafka: 2.11-1Logstash: 6.2.4Elasticsearch: 6.2.4Kibana: 6.2.4相应的版本最好下载对应的插件
具体实践
我们就以比较常见的 Nginx 日志来举例说明下，日志内容是 JSON 格式
{"@timestamp":"2017-12-27T16:38:17+08:00","host":"192.168.56.11","clientip":"192.168.56.11","size":26,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.56.11","url":"/nginxweb/index.html","domain":"192.168.56.11","xff":"-","referer":"-","status":"200"}{"@timestamp":"2017-12-27T16:38:17+08:00","host":"192.168.56.11","clientip":"192.168.56.11","size":26,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.56.11","url":"/nginxweb/index.html","domain":"192.168.56.11","xff":"-","referer":"-","status":"200"}{"@timestamp":"2017-12-27T16:38:17+08:00","host":"192.168.56.11","clientip":"192.168.56.11","size":26,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.56.11","url":"/nginxweb/index.html","domain":"192.168.56.11","xff":"-","referer":"-","status":"200"}{"@timestamp":"2017-12-27T16:38:17+08:00","host":"192.168.56.11","clientip":"192.168.56.11","size":26,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.56.11","url":"/nginxweb/index.html","domain":"192.168.56.11","xff":"-","referer":"-","status":"200"}{"@timestamp":"2017-12-27T16:38:17+08:00","host":"192.168.56.11","clientip":"192.168.56.11","size":26,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.56.11","url":"/nginxweb/index.html","domain":"192.168.56.11","xff":"-","referer":"-","status":"200"}
Filebeat
为什么用 Filebeat ，而不用原来的 Logstash 呢？
$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-darwin-x86_64.tar.gz
解压
tar -zxvf filebeat-6.2.4-darwin-x86_64.tar.gzmv filebeat-6.2.4-darwin-x86_64 filebeatcd filebeat
修改配置
修改 Filebeat 配置，支持收集本地目录日志，并输出日志到 Kafka 集群中
$ vim fileat.ymlfilebeat.prospectors:- input_type: logpaths:- /opt/logs/server/nginx.logjson.keys_under_root: truejson.add_error_key: truejson.message_key: logoutput.kafka:hosts: ["192.168.0.1:9092，192.168.0.2:9092，192.168.0.3:9092"]topic: 'nginx'
Filebeat 6.0 之后一些配置参数变动比较大，比如 document_type 就不支持，需要用 fields 来代替等等。
启动
$ ./filebeat -e -c filebeat.yml
Kafka
生产环境中 Kafka 集群中节点数量建议为（2N + 1 ）个，这边就以 3 个节点举例
下载
直接到官网下载 Kafka
$ wget http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
解压
tar -zxvf kafka_2.11-1.0.0.tgzmv kafka_2.11-1.0.0 kafkacd kafka
修改 Zookeeper 配置
修改 Zookeeper 配置，搭建 Zookeeper 集群，数量 ( 2N + 1 ) 个
ZK 集群建议采用 Kafka 自带，减少网络相关的因素干扰
$ vim zookeeper.propertiestickTime=2000dataDir=/opt/zookeeperclientPort=2181maxClientCnxns=50initLimit=10syncLimit=5server.1=192.168.0.1:2888:3888server.2=192.168.0.2:2888:3888server.3=192.168.0.3:2888:3888
Zookeeper data 目录下面添加 myid 文件，内容为代表 Zooekeeper 节点 id （1，2，3），并保证不重复。
$ vim /opt/zookeeper/myid1
启动 Zookeeper 节点
分别启动 3 台 Zookeeper 节点，保证集群的高可用
$ ./zookeeper-server-start.sh -daemon ./config/zookeeper.properties
修改 Kafka 配置
kafka 集群这边搭建为 3 台，可以逐个修改 Kafka 配置，需要注意其中 broker.id 分别 （1，2，3）
$ vim ./config/server.propertiesbroker.id=1port=9092host.name=192.168.0.1num.replica.fetchers=1log.dirs=/opt/kafka_logsnum.partitions=3zookeeper.connect=192.168.0.1: 192.168.0.2: 192.168.0.3:2181zookeeper.connection.timeout.ms=6000zookeeper.sync.time.ms=2000num.io.threads=8num.network.threads=8queued.max.requests=16fetch.purgatory.purge.interval.requests=100producer.purgatory.purge.interval.requests=100delete.topic.enable=true
启动 Kafka 集群
分别启动 3 台 Kafka 节点，保证集群的高可用
$ ./bin/kafka-server-start.sh -daemon ./config/server.properties
查看 topic 是否创建成功
$ bin/kafka-topics.sh --list --zookeeper localhost:2181nginx
监控 Kafka Manager
Kafka-manager 是 Yahoo 公司开源的集群管理工具。
可以在 Github 上下载安装：https://github.com/yahoo/kafka-manager
如果遇到 Kafka 消费不及时的话，可以通过到具体 cluster 页面上，增加 partition。Kafka 通过 partition 分区来提高并发消费速度。
Logstash
Logstash 提供三大功能
INPUT 进入
FILTER 过滤功能
OUTPUT 出去
如果使用 Filter 功能的话，强烈推荐大家使用 Grok debugger 来预先解析日志格式。
下载
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz
解压重命名
$ tar -zxvf logstash-6.2.4.tar.gz$ mv logstash-6.2.4 logstash
$ vim nginx.confinput {kafka {type => "kafka"bootstrap_servers => "192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181"topics => "nginx"group_id => "logstash"consumer_threads => 2}}output {elasticsearch {host => ["192.168.0.1","192.168.0.2"，"192.168.0.3"]port => "9300"index => "nginx-%{+YYYY.MM.dd}"}}
启动 Logstash
$ ./bin/logstash -f nginx.conf
Elasticsearch
下载
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
解压
$ tar -zxvf elasticsearch-6.2.4.tar.gz$ mv elasticsearch-6.2.4.tar.gz elasticsearch
修改配置
$ vim config/elasticsearch.ymlcluster.name: esnode.name: es-node1network.host: 192.168.0.1discovery.zen.ping.unicast.hosts: ["192.168.0.1"]discovery.zen.minimum_master_nodes: 1
$ ./bin/elasticsearch -d
打开网页 http://192.168.0.1:9200/, 如果出现下面信息说明配置成功
{name: "es-node1",cluster_name: "es",cluster_uuid: "XvoyA_NYTSSV8pJg0Xb23A",version: {number: "6.2.4",build_hash: "ccec39f",build_date: "2018-04-12T20:37:28.497551Z",build_snapshot: false,lucene_version: "7.2.1",minimum_wire_compatibility_version: "5.6.0",minimum_index_compatibility_version: "5.0.0"},tagline: "You Know, for Search"}
控制台Cerebro 这个名字大家可能觉得很陌生，其实过去它的名字叫 kopf ！因为 Elasticsearch 5.0 不再支持 site plugin，所以 kopf 作者放弃了原项目，另起炉灶搞了 cerebro，以独立的单页应用形式，继续支持新版本下 Elasticsearch 的管理工作。
注意点
Master 与 Data 节点分离，当 Data 节点大于 3 个的时候，建议责任分离，减轻压力
Data Node 内存不超过 32G ，建议设置成 31 G ，具体原因可以看上一篇文章
discovery.zen.minimum_master_nodes 设置成 （ total / 2 + 1 ），避免脑裂情况
最重要的一点，不要将 ES 暴露在公网中，建议都安装 X-PACK ，来加强其安全性
kibana
下载
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-darwin-x86_64.tar.gz
解压
$ tar -zxvf kibana-6.2.4-darwin-x86_64.tar.gz$ mv kibana-6.2.4-darwin-x86_64.tar.gz kibana
修改配置
$ vim config/kibana.ymlserver.port: 5601server.host: "192.168.0.1"elasticsearch.url: "http://192.168.0.1:9200"
启动 Kibana
$ nohup ./bin/kibana &
界面展示
创建索引页面需要到 Management -> Index Patterns 中通过前缀来指定
最终效果展示
总结
来源：https://blog.51cto.com/13527416/2117141
文章转载：高效运维
（版权归原作者所有，侵删）
点击下方“阅读原文”查看更多