go-stash高效的 Logstash 替代方案-技术圈

go-stash高效的 Logstash 替代方案

联合创作 · 2023-10-01 07:24

go-stash 是一个高效的从 Kafka 获取，根据配置的规则进行处理，然后发送到 ElasticSearch 集群的工具。

go-stash 有大概 logstash 5 倍的吞吐性能，并且部署简单，一个可执行文件即可。

安装

cd stash && go build stash.go

Quick Start

可执行文件方式

./stash -f etc/config.yaml

docker 方式，确保配置文件路径正确

docker run -d -v `pwd`/etc:/app/etc kevinwan/go-stash

config.yaml示例如下:

Clusters:
- Input:
    Kafka:
      Name: go-stash
      Log:
        Mode: file
      Brokers:
      - "172.16.48.41:9092"
      - "172.16.48.42:9092"
      - "172.16.48.43:9092"
      Topic: ngapplog
      Group: stash
      Conns: 3
      Consumers: 10
      Processors: 60
      MinBytes: 1048576
      MaxBytes: 10485760
      Offset: first
  Filters:
  - Action: drop
    Conditions:
      - Key: status
        Value: 503
        Type: contains
      - Key: type
        Value: "app"
        Type: match
        Op: and
  - Action: remove_field
    Fields:
    - message
    - source
    - beat
    - fields
    - input_type
    - offset
    - "@version"
    - _score
    - _type
    - clientip
    - http_host
    - request_time
  Output:
    ElasticSearch:
      Hosts:
      - "http://172.16.188.73:9200"
      - "http://172.16.188.74:9200"
      - "http://172.16.188.75:9200"
      Index: "go-stash-{{yyyy.MM.dd}}"
      MaxChunkBytes: 5242880
      GracePeriod: 10s
      Compress: false
      TimeZone: UTC

ES性能写入测试

测试环境

stash服务器：3台 4核 8G
es服务器： 15台 16核 64G

关键配置

- Input:
      Conns: 3
      Consumers: 10
      Processors: 60
      MinBytes: 1048576
      MaxBytes: 10485760
  Filters:
  - Action: remove_field
    Fields:
    - message
    - source
    - beat
    - fields
    - input_type
    - offset
    - request_time
  Output:
      Index: "nginx_pro-{{yyyy.MM.d}}"
      Compress: false
      MaxChunkBytes: 5242880
      TimeZone: UTC

写入速度平均在15W/S以上

编辑分享