go-stash高效的 Logstash 替代方案-技术圈

go-stash高效的 Logstash 替代方案

联合创作 · 2023-10-01 07:24

go-stash 是一个高效的从 Kafka 获取，根据配置的规则进行处理，然后发送到 ElasticSearch 集群的工具。

go-stash 有大概 logstash 5 倍的吞吐性能，并且部署简单，一个可执行文件即可。

安装



cd stash && go build stash.go

Quick Start

可执行文件方式



./stash -f etc/config.yaml

docker 方式，确保配置文件路径正确



docker run -d -v `pwd`/etc:/app/etc kevinwan/go-stash

config.yaml示例如下:



Clusters:

- Input:

    Kafka:

      Name: go-stash

      Log:

        Mode: file

      Brokers:

      - "172.16.48.41:9092"

      - "172.16.48.42:9092"

      - "172.16.48.43:9092"

      Topic: ngapplog

      Group: stash

      Conns: 3

      Consumers: 10

      Processors: 60

      MinBytes: 1048576

      MaxBytes: 10485760

      Offset: first

  Filters:

  - Action: drop

    Conditions:

      - Key: status

        Value: 503

        Type: contains

      - Key: type

        Value: "app"

        Type: match

        Op: and

  - Action: remove_field

    Fields:

    - message

    - source

    - beat

    - fields

    - input_type

    - offset

    - "@version"

    - _score

    - _type

    - clientip

    - http_host

    - request_time

  Output:

    ElasticSearch:

      Hosts:

      - "http://172.16.188.73:9200"

      - "http://172.16.188.74:9200"

      - "http://172.16.188.75:9200"

      Index: "go-stash-{{yyyy.MM.dd}}"

      MaxChunkBytes: 5242880

      GracePeriod: 10s

      Compress: false

      TimeZone: UTC

ES性能写入测试

测试环境

stash服务器：3台 4核 8G

es服务器： 15台 16核 64G

关键配置



- Input:

      Conns: 3

      Consumers: 10

      Processors: 60

      MinBytes: 1048576

      MaxBytes: 10485760

  Filters:

  - Action: remove_field

    Fields:

    - message

    - source

    - beat

    - fields

    - input_type

    - offset

    - request_time

  Output:

      Index: "nginx_pro-{{yyyy.MM.d}}"

      Compress: false

      MaxChunkBytes: 5242880

      TimeZone: UTC

写入速度平均在15W/S以上

编辑分享