redis-auxredis 辅助工具包-技术圈

这篇文章主要是介绍功能点，先看看这个工具包有什么可以用的，目前主要有两个模块——布隆过滤器、基于注解限流。基于redisTemplate

用法

这里用maven作为工具管理包演示，添加jitpack源、添加下面的依赖

    <repositories>
        <repository>
            <id>jitpack.io</id>
            <url>https://jitpack.io</url>
        </repository>
     </repositories>
<dependency>
	    <groupId>com.github.97lele</groupId>
	    <artifactId>redis-aux</artifactId>
	    <version>0.3.8</version>
	</dependency>

布隆过滤器演示

在启动类上添加启用布隆过滤器的注解

@EnableBloomFilter(bloomFilterPath = "com.example.demo")

2个属性，分别为

1.需要支持lambda表达式添加的实体路径

2.是否开启支持@Trancational注解，需要和数据库事务配合使用

配置好redis

spring:
 redis:
   port: 6379
   host: 127.0.0.1

添加方法

只有两种，一种是通过构建操作对象来添加，一种是通过解析lambda表达，获取其字段上的注解信息来添加

若要调用SFunction为参数的方法需要在EnableBloomFilter配置扫描路径

主要是exceptedInsertions,fpp，timeOut,local 这里四个参数,分别为预计插入的个数，允许的错误率，过期时间，是否为本地

演示，添加主要分为普通添加

@Test
   public void simpleTest() {
        boolean isLocal=true;
        String key = "testAdd";
        //默认local为false
        AddCondition addCondition = AddCondition.create().keyName(key).local(isLocal);
        BaseCondition baseCondition = addCondition.toBaseCondition();
        bloomFilter.add(addCondition, "hello");
        System.out.println("contain he:"+bloomFilter.mightContain(baseCondition,"he"));
        System.out.println("contain hello:"+bloomFilter.mightContain(baseCondition,"hello"));
        //多值操作
        bloomFilter.addAll(addCondition,Arrays.asList("h","a","c"));
        System.out.println("before reset："+bloomFilter.mightContains(baseCondition,Arrays.asList("a","b","c")));
        //重置
        bloomFilter.reset(baseCondition);
        System.out.println("after reset："+bloomFilter.mightContains(baseCondition,Arrays.asList("a","hello","qq")));
        System.out.println("before delete："+bloomFilter.containKey(baseCondition));
        //删除
        bloomFilter.remove(baseCondition);
        System.out.println("after delete："+bloomFilter.containKey(baseCondition));
    }

结果

lambda演示

需要实体类实现getter,并且添加上前缀名，否则默认为类名,需要操作的属性上面添加BloomFilterProperty注解，该注解可填充属性有以下，key如果不填按字段名处理，另外要在enableBloomFilter的注解里填写扫描路径

double fpp() default 0.03;

    long exceptionInsert() default 1000;

    String key() default "";

    long timeout() default -1L;

    TimeUnit timeUnit() default TimeUnit.SECONDS;
    
    boolean local() default false;

@BloomFilterPrefix
public class TestEntity {
    @BloomFilterProperty(enableGrow = true,exceptionInsert = 5,timeout = 30)
    private String name;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

基于lambda的测试代码

@Test
    public void lambdaTest() throws InterruptedException {
      bloomFilter.addAll(TestEntity::getName, Arrays.asList(13, 14, 15, 16));
        System.out.println(bloomFilter.mightContain(TestEntity::getName, 15));
        System.out.println(bloomFilter.mightContains(TestEntity::getName, Arrays.asList(13, 200)));
    }

结果

键过期测试

 @Test
    void timeOutTest() {
          boolean isLocal=true;
        bloomFilter.add(AddCondition.create().keyName("a1").timeout(30L).timeUnit(TimeUnit.SECONDS).local(isLocal), 1);
        bloomFilter.addAll(AddCondition.create().keyName("a4").timeUnit(TimeUnit.SECONDS).timeout(10L).local(isLocal), Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9));
        bloomFilter.add(AddCondition.create().keyName("a2").timeout(11L).timeUnit(TimeUnit.SECONDS).local(isLocal), 1);
        bloomFilter.add(AddCondition.create().keyName("a3").timeout(22L).timeUnit(TimeUnit.SECONDS).local(isLocal), 1);
        System.out.println(bloomFilter.mightContain(BaseCondition.create().keyName("a1"), 1));
        try {
            TimeUnit.SECONDS.sleep(35L);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println(bloomFilter.containKey(BaseCondition.create().keyName("a1")));
    }

结果

开启支持事务

清空redis的键

service代码，一个有错，一个无错

package com.example.demo.service;

import com.example.demo.dao.UserTicketMapper;
import com.example.demo.entity.UserTicket;
import com.trendy.util.redis.aux.bloomfilter.autoconfigure.RedisBloomFilter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.time.LocalDateTime;

/**
 * @author: lele
 * @date: 2020/1/4 下午11:51
 */
@Service
public class TestService {

    @Autowired
    private UserTicketMapper userTicketMapper;

    @Autowired
    private RedisBloomFilter redisBloomFilter;

    @Transactional(rollbackFor = Exception.class)
    public void wrong() {
        AddCondition addCondition = AddCondition.create().keyPrefix("news").keyName("user2").exceptionInsert(500L).fpp(0.001);
        redisBloomFilter.add(addCondition,"推送1");
        userTicketMapper.insert(new UserTicket().setCreateTime(LocalDateTime.now()).setTicketId(1L).setUserId(2L));
        int i = 1 / 0;

    }

    @Transactional(rollbackFor = Exception.class)
    public void right() {
        AddCondition addCondition = AddCondition.create().keyPrefix("news").keyName("user1").exceptionInsert(500L).fpp(0.001);
        redisBloomFilter.add(addCondition,"推送3");
        userTicketMapper.insert(new UserTicket().setCreateTime(LocalDateTime.now()).setTicketId(1L).setUserId(2L));

    }
}

访问两个接口查看redis、mysql的结果

确实只有一个成功的，即user1有值（EnableBloomFilter的事务默认不开启）

限流功能

目前支持三种限流模式——滑动窗口限流、漏斗限流、令牌桶限流,这三种模式配置时，要添加fallback方法，否则会抛异常

并在后来加多一个限流组的功能来支持动态配置，但底层核心还是上面三种模式

使用方式

在接口上添加注解@EnableLimiter，会加载对应的类，aop会进行拦截并做相应的处理，通过@Import加载注册类，enableGroup开启限流组

相关注解

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
@LimiterType(mode = LimiterConstants.FUNNEL_LIMITER)
public @interface FunnelLimiter {

    /**
     * 漏斗容量
     *
     * @return
     */
    double capacity();

    /**
     * 每秒漏出的速率
     *
     * @return
     */
    double funnelRate() ;

    /**
     * 时间单位
     *
     * @return
     */
    TimeUnit funnelRateUnit() default TimeUnit.SECONDS;

    /**
     * 每次请求所需加的水量
     *
     * @return
     */
    double requestNeed() default 1;

    String fallback() default "";

    boolean passArgs() default false;

}

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
@LimiterType(mode = LimiterConstants.TOKEN_LIMITER)
public @interface TokenLimiter {

    /**
     * 令牌桶容量
     *
     * @return
     */
    double capacity();

    /**
     * 令牌生成速率
     *
     * @return
     */
    double tokenRate();

    /**
     * 速率时间单位，默认秒
     *
     * @return
     */
    TimeUnit tokenRateUnit() default TimeUnit.SECONDS;

    /**
     * 每次请求所需要的令牌数
     *
     * @return
     */
    double requestNeed() default 1;

    double initToken() default 0;


    String fallback() default "";

    boolean passArgs() default false;

}

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
@LimiterType(mode = LimiterConstants.WINDOW_LIMITER)
public @interface WindowLimiter {
    /**
     * 持续时间，窗口间隔
     *
     * @return
     */
    int during() default 1;

    TimeUnit duringUnit() default TimeUnit.SECONDS;

    /**
     * 通过的请求数
     *
     * @return
     */
    long passCount();

    String fallback() default "";

    boolean passArgs() default false;


}

fallback则是定义于本类的其他public的方法，可以设置是否传参，参数则是被拦截的方法的参数，可以在返回的方法使用这些参数

demo

@RestController
public class TestController2 {
 
    @GetMapping("ha")
    @WindowLimiter(during = 10,passCount = 5)
    public String test() {
        return "hihi1";
    }
    //每秒通过0.5个请求
    @GetMapping("ha2/{userName}")
    @FunnelLimiter(capacity = 5,funnelRate = 0.5,requestNeed = 1,fallback = "test",passArgs = true)
    public Result<String> test2(@PathVariable("userName")String userName) throws NoSuchMethodException {
        return Result.success("ok");
    }
    //默认为秒，该配置为每秒生成0.5个令牌
    @GetMapping("ha3")
    @TokenLimiter(capacity = 5,tokenRate = 0.5,requestNeed = 1)
    public String test3() {
        return "hihi3";
    }
    //一分钟生产一个令牌
    @GetMapping("ha4")
    @TokenLimiter(capacity = 5,tokenRate = 1,tokenRateUnit = TimeUnit.MINUTES,initToken = 5)
    public String test4() {
        return "hihi4";
    }


    public Result<String> test(String userName){
        return Result.success("对不起:"+userName+",挤不进去太多人了");
    }


}

这里给出Resul类

@Data
@AllArgsConstructor
public class Result<T> {
    private T data;
    private String msg;
    private Integer code;

    public static Result success(Object data){
        return new Result(data,"ok",0);
    }
}

下面到动态配置的功能介绍,需要在注解@EnableLimiter上配置 enableGroup=true,默认不开启

首先要定义一个限流器，下面这个demo几乎把所有配置都列出来了，其中id是最重要的，用来标记该限流器，做好配置后，需要添加相关的拦截器，本身有四个拦截器，url前缀拦截器,ip黑/白名单,和本身的限流器，按权重执行从大到小执行，可以调用order方法来更改他的权限大小，默认执行顺序为ip-url-限流，也可以自己实现相关的拦截器，但权重要做相关的调整

@Configuration
public class RateLimitConfig implements InitializingBean {
    @Autowired
    private LimiterGroupService limiterGroupService;

    @Override
    public void afterPropertiesSet() {
        //清除原来的配置
        limiterGroupService.clear("1");
        //新建
        LimiteGroupConfig config = LimiteGroupConfig.of().id("1")
                .remark("this.application").tokenConfig(
                        //令牌桶配置，下面表示令牌桶容量为5，初始桶为3，每1s生产3个令牌，每个请求消耗1个令牌
                        TokenRateConfig.of()
                                .capacity(5.0)
                                .initToken(3.0)
                                .requestNeed(1.0)
                                .tokenRate(3.0)
                                .tokenRateUnit(TimeUnit.SECONDS)
                                .build()
                ).
                        windowConfig(
                                //滑动窗口配置,下面表示10s内只允许5个通过
                                WindowRateConfig.of()
                                        .passCount(5L)
                                        .during(10L)
                                        .duringUnit(TimeUnit.SECONDS)
                                        .build()).currentMode(LimiterConstants.TOKEN_LIMITER)
                //漏斗配置，容纳量为10，每次请求容纳量-1，每3秒增加1个容纳量
                .funnelConfig(FunnelRateConfig.of()
                        .capacity(10.0)
                        .funnelRate(3.0)
                        .funnelRateUnit(TimeUnit.SECONDS)
                        .requestNeed(1.0)
                        .build())
                //黑白名单，网段 xxx.xxx.xxx./24,类似 192.168.0.0-192.168.2.1 以及 192* 分号分隔
                /*.blackRule("127.0.0.1")
                .enableBlackList(true)
                .enableWhiteList(true).
                whiteRule("192.168.0.*")
                */
                .blackRuleFallback("ip")
                //当前限流模式
                .currentMode(LimiterConstants.TOKEN_LIMITER)
                //开启统计，是统计复用该配置下的请求数
                .enableCount(true)
                //统计时间范围,如果没有则从第一次请求开始统计
                .countDuring(1L).countDuringUnit(TimeUnit.MINUTES)
                //url配置,;号分割
                .unableURLPrefix("/user;/qq")
                .enableURLPrefix("/test")
                //url匹配失败后的执行方法
                .urlFallBack("userBlack")
                .build();
        //保存到redis,也可以保存到本地
        limiterGroupService.save(config, true, false);
        //读取redis上的配置
//        limiterGroupService.reload("1");
        //添加对应的拦截器,不然切面中不会执行对应的逻辑，这里也可以实现自己的拦截器并添加上去
        limiterGroupService.addHandler(GroupHandlerFactory.limiteHandler())
                .addHandler(GroupHandlerFactory.ipBlackHandler())
                .addHandler(GroupHandlerFactory.urlPrefixHandler());
        ;
    }
}

内置了一个controller用于动态更改配置，目前主要是ip、url、限流模式、限流器的配置，更改模式时可以选择是否删掉其他限流器在redis上的缓存，url拦截器通过前面urlhanlder配置

@RestController
public class ActuatorController {

    @Autowired
    private LimiterGroupService limiterGroupService;

    @GetMapping("/redis-aux/getIp")
    public String getIp(HttpServletRequest request) {
        return IpCheckUtil.getIpAddr(request);
    }

    //更改ip规则
    @PostMapping("/redis-aux/changeIpRule")
    public LimiteGroupConfig changeRule(@RequestParam("groupId") String groupId,
                                        @RequestParam(value = "rule", required = false) String rule,
                                        @RequestParam(value = "enable", required = false) Boolean enable,
                                        @RequestParam(value = "white", required = false) Boolean white) {
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        if (white) {
            limiter.setWhiteRule(rule);
            limiter.setEnableWhiteList(enable);
        } else {
            limiter.setBlackRule(rule);
            limiter.setEnableBlackList(enable);
        }
        limiterGroupService.save(limiter, true, false);
        return limiter;
    }

    //更改url匹配规则
    @PostMapping("/redis-aux/changeUrlRule")
    public LimiteGroupConfig changeUrlRule(@RequestParam("groupId") String groupId,
                                           @RequestParam("enableUrl") String enableUrl,
                                           @RequestParam("unableUrl") String unableUrl
                                           ) {
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        if (enableUrl != null) {
            limiter.setEnableURLPrefix(enableUrl);
        }
        if (unableUrl != null) {
            limiter.setUnableURLPrefix(unableUrl);
        }
        limiterGroupService.save(limiter, true, false);
        return limiter;
    }


    //更改模式
    @PostMapping("/redis-aux/changeLimitMode")
    public LimiteGroupConfig changeMode(@RequestParam("groupId") String groupId, @RequestParam("mode") Integer mode
            , @RequestParam("removeOther") Boolean removeOther
    ) {
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        if (mode < 4 && mode > 0) {
            limiter.setCurrentMode(mode);
        }
        limiterGroupService.save(limiter, true, removeOther);
        return limiter;
    }

    //更改限流规则
    @PostMapping("/redis-aux/changeFunnelConfig")
    public LimiteGroupConfig changeFunnelConfig(@RequestParam("groupId") String groupId,
                                                @RequestParam(value = "requestNeed", required = false) Double requestNeed,
                                                @RequestParam("capacity") Double capacity,
                                                @RequestParam("funnelRate") Double funnelRate,
                                                @RequestParam(value = "funnelRateUnit", required = false) Integer funnelRateUnit
    ) {

        FunnelRateConfig config = FunnelRateConfig.of().capacity(capacity)
                .funnelRate(funnelRate).requestNeed(requestNeed)
                .funnelRateUnit(TimeUnitEnum.getTimeUnit(funnelRateUnit)).build();
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        limiter.setFunnelRateConfig(config);
        limiterGroupService.save(limiter, true, false);
        return limiter;
    }

    @PostMapping("/redis-aux/changeWindowConfig")
    public LimiteGroupConfig changeWindowConfig(@RequestParam("groupId") String groupId,
                                                @RequestParam("passCount") Long passCount,
                                                @RequestParam(value = "during", required = false) Long during,
                                                @RequestParam(value = "duringUnit", required = false) Integer mode
    ) {
        WindowRateConfig config = WindowRateConfig.of().passCount(passCount).during(during).duringUnit(TimeUnitEnum.getTimeUnit(mode)).build();
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        limiter.setWindowRateConfig(config);
        limiterGroupService.save(limiter, true, false);
        return limiter;
    }

    @PostMapping("/redis-aux/changeTokenConfig")
    public LimiteGroupConfig changeWindowConfig(@RequestParam("groupId") String groupId,
                                                @RequestParam("capacity") Double capacity,
                                                @RequestParam(value = "initToken", required = false) Double initToken,
                                                @RequestParam("tokenRate") Double tokenRate,
                                                @RequestParam(value = "requestNeed", required = false) Double requestNeed,
                                                @RequestParam(value = "duringUnit", required = false) Integer mode
    ) {
        TokenRateConfig config = TokenRateConfig.of().capacity(capacity).initToken(initToken).tokenRate(tokenRate)
                .requestNeed(requestNeed).tokenRateUnit(TimeUnitEnum.getTimeUnit(mode)).build();
        LimiteGroupConfig limiter = limiterGroupService.getLimiterConfig(groupId);
        limiter.setTokenRateConfig(config);
        limiterGroupService.save(limiter, true, false);
        return limiter;
    }

    @GetMapping("/redis-aux/getCount/{groupId}")
    public Map<String, String> changeCountConfig(@PathVariable("groupId") String groupId
    ) {
        return limiterGroupService.getCount(groupId);
    }

}

然后到使用方式，配置好后，可以在类或者方法上使用，如果在类中使用

@RestController
@LimiteGroup(groupId = "1", fallback = "test")
public class TestController {

    @GetMapping("/ok")
    public String ok() {
        return "ok";
    }

    @GetMapping("/user")
    public String user() {
        return "user";
    }
 @GetMapping("/user/t")
    @LimiteExclude
    public String usert() {
        return "usert";
    }

    public String userBlack() {
        return "非法前缀访问";
    }
    public String ip() {
        return "ip错误";
    }
    public String test() {
        return "too much request";
    }
}

可以访问ActuatorController 的接口进行相关的配置，下面的统计功能采用滑动窗口+分桶计算该段时间的qps，是对单个应用的数据统计