Vladimir日志采集工具
Vladimir 采集系统
日志采集工具，支持正则 热加载日志文件，发送状态等信息都会实时上报 ETCD
TODO
- 将日志文件一行一行读取出来发送到自定义的 Sink | 100%
- 支持自动重试（fib）
- 心跳检测发送到自定义的 Topic[health] | 100%
- 日志启动点(创建 CheckPoint) | 100%
- Offset 同步到 Etcd | 100%
- Web 监控心跳以及创建新的采集任务 | 100%
- 热启动插件 | 10%
Logs - 采集工具
./Logs_Pulse install -- 安装服务
./Logs_Pulse start -- 启动服务
./Logs_Pulse stop -- 停止服务
./Logs_Pulse restart -- 重启服务
Owner
- 邮箱
echo emhhbmdqaWFueGlubmV0QGdtYWlsLmNvbQo= | base64 --decode
命令行客户端使用教程
(base) ➜ cli ./cli
NAME:
pulse client - a tool for driving pulse cli
USAGE:
cli [global options] command [command options] [arguments...]
VERSION:
v1 darwin/amd64 go1.11.4
AUTHOR:
Zhangjianxin <zhangjianxinnet@gmail.com>
COMMANDS:
list pulse on etcd config
services pulse on services
task addtask on services
clear delete root path / data
host host list
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help
--version, -v print the version
(base) ➜ cli ./cli host
|---------------|
| Host |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
(base) ➜ cli ./cli list
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Topic | Log_path | Service | Send_rate | IP | Type |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log | sentry-test-service | 1000 | 192.168.2.150 | kafka |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log | sentry-test-service | 1000 | 192.168.2.150 | http |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | sentry-test-service | 1000 | 192.168.2.150 | http |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
(base) ➜ cli ./cli services
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Offset | FilePatch | Time | Topic | TPS | QPS | IP |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 39 | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log | 2019-04-27 03:38:32 | test | 0 | 0 | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 63 | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log | 2019-04-27 03:45:17 | test | 0 | 0 | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 87 | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | 2019-04-27 03:49:32 | test | 0 | 0 | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
(base) ➜ cli
./cli task -host 192.168.43.223 -t buried-point-mall -f /var/log/system.log -s buried-point-mall-service-test -p http
./cli task -re "" -host 192.168.2.150 -t buried-point-mall -f /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/ -s buried-point-mall-service-test -p http
./cli task -re "" -host 172.17.193.23 -t sentry-test -f /var/log/messages -s service-test -p http
./cli task -re "" -host 172.19.184.193 -t sentry-test -f /home/soft/Octopoda/release/logs/pulse.log -s system-pulse -p http
./cli task -re "" -host 172.19.184.193 -t buried-point-mall -f /data/sa/ -s sa-pulse -p http
./cli task -re "" -host 172.17.193.23 -t sentry-test -f /home/Octopoda/release/logs/ -s service-test2 -p http
./cli task -re "" -host 192.168.131.249 -t buried-point-sma-r2p10 -f /home/admin/app/logs/ -s service-test4 -p http
./cli task -re "" -host 192.168.131.249 -t buried-point-sma-r2p10 -f /home/admin/app/logs/behaviorSave.log -s service-test4 -p http
./cli task -re "" -host 192.168.131.249 -t sentry-test -f /home/Octopoda/release/logs/pulse.log -s system-pulse -p http
提交作业
./cli task
-host 192.168.2.150 #采集某个节点
-t test #发送到topic
-f /var/log/system.log # 采集日志
-s sentry-test-service # 服务名字定义
-p http #发送方式
# create topic
kafka-topics --create --zookeeper cdh-m1.sxkj.online:2181 --replication-factor 1 --partitions 1 --topic kdc_pdp
kerberos KDC Proxy
https://medium.com/@Santii/how-to-setup-kerberos-behind-a-reverse-proxy-nginx-d692f609191c
评论