Redis 集群搭建-技术圈

Redis 是一个 K-V 存储系统，是一个高性能的 K-V 数据库，是目前大规模使用的缓存中间件。单节点的Redis 已经就达到了很高的性能，为了提高可用性我们可以使用 Redis 集群。

集群简介

Redis 集群是一个可以在多个 Redis 节点之间进行数据共享的设施（installation）。

Redis 集群不支持那些需要同时处理多个键的 Redis 命令，因为执行这些命令需要在多个 Redis 节点之间移动数据，并且在高负载的情况下，这些命令将降低 Redis 集群的性能，并导致不可预测的行为。

Redis 集群通过分区（partition）来提供一定程度的可用性（availability）：即使集群中有一部分节点失效或者无法进行通讯，集群也可以继续处理命令请求。

Redis 集群提供了以下两个好处：

将数据自动切分（split）到多个节点的能力
当集群中的一部分节点失效或者无法进行通讯时，仍然可以继续处理命令请求的能力

集群说明

集群数据共享

Redis 集群使用数据分片（sharding）而非一致性哈希（consistency hashing）来实现：一个 Redis 集群包含 16384 个哈希槽（hash slot），数据库中的每个键都属于这 16384 个哈希槽的其中一个，集群使用公式 CRC16(key) % 16384 来计算键 key 属于哪个槽，其中 CRC16(key) 语句用于计算键 key 的 CRC16 校验和。

集群中的每个节点负责处理一部分哈希槽

一个集群可以有三个哈希槽，其中：

节点 A 负责处理 0 号至 5500 号哈希槽
节点 B 负责处理 5501 号至 11000 号哈希槽
节点 C 负责处理 11001 号至 16384 号哈希槽

集群中的主从复制

为了使得集群在一部分节点下线或者无法与集群的大多数（majority）节点进行通讯的情况下，仍然可以正常运作，Redis 集群对节点使用了主从复制功能：集群中的每个节点都有 1 个至 N 个复制品（replica），其中一个复制品为主节点（master），而其余的 N-1 个复制品为从节点（slave）

集群 `TCP` 端口

每个 Redis Cluster 节点都需要 2 个 TCP 连接打开，正常的 Redis TCP 端口被服务于客户端，例如：6379，加上通过增加 10000 到数据端口获取的端口，例如：16379。

第二个高端口被用来 Redis Cluster Bus，也就是使用二进制协议的一个节点到节点的通信信道。Redis Cluster Bus 被节点用于故障检测，配置更新，故障转移授权等等。客户端不应该尝试与 Redis Cluster Bus 端口进行通信，而是始终使用正常的 Redis 命令端口，然而要确保在防火墙中打开两个端口，否则 Redis Cluster 节点将无法通信。

命令端口和 Redis Cluster Bus 端口偏移是固定的，始终为 10000。

注意，为了使 Redis Cluster 正常工作，你需要为每个节点：

用于客户端进行通信的普通客户端通信端口（通常使用 6379）对所有需要到达集群的客户端，以及所有其它集群节点（使用客户端端口进行密钥迁移）都是开放的
Redis Cluster Bus 端口（客户端端口 +10000）必须可从所有其它集群节点访问

如果不打开两个 TCP 端口，集群将无法按预期工作

Redis Cluster Bus 使用不同的二进制协议进行节占到节点的数据交换，这更适合于使用少量的带宽和处理时间交换节点之间的信息

安装依赖

$ sudo gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3$ sudo curl -sSL https://get.rvm.io | bash -s stable$ sudo source ~/.rvm/scripts/rvm$ sudo echo "ruby_url=https://cache.ruby-china.org/pub/ruby" > ~/.rvm/user/db$ rvm -v$ sudo rvm install 2.4.1$ sudo gem update$ sudo gem install redis

集群配置

安装 Redis

$ cd ~$ wget http://download.redis.io/releases/redis-4.0.9.tar.gz$ tar xzf redis-4.0.9.tar.gz$ cd redis-4.0.9$ sudo make

创建配置目录

$ sudo mkdir -pv /usr/local/redis-cluster/{bin,etc,log,run,var}

创建配置文件

$ sudo cp ~/redis-4.0.9/redis.conf /usr/local/redis-cluster/etc/redis-7001.conf

复制命令文件

$ cd ~/redis-4.0.9/src/$ sudo cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-sentinel redis-cli redis-server /usr/local/redis-cluster/bin

修改配置文件

$ sudo vim /usr/local/redis-cluster/etc/redis-7001.confbind 192.168.33.110port 7001daemonize yes
pidfile /user/local/redis-cluster/run/7001.pidlogfile /usr/local/redis-cluster/log/7001.logdbfilename dump-7001.rdb
dir /usr/local/redis-cluster/var
appendonly yesappendfilename "appendonly-7001.aof"
cluster-enabled yescluster-config-file nodes-7001.confcluster-node-timeout 15000

复制配置文件

将复制后的文件中的 7001 换成相应的名称

$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7002.conf$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7003.conf$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7004.conf$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7005.conf$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7006.conf

集群部署

启动节点

// 启动配置的所有节点$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7001.conf$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7002.conf$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7003.conf$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7004.conf$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7005.conf$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7006.conf

检查服务

$ ps -ef | grep redisroot     20077     1  0 May18 ?        00:01:04 /usr/local/bin/redis-server 192.168.33.110:7001 [cluster]root     20079     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7002 [cluster]root     20081     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7003 [cluster]root     20089     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7004 [cluster]root     20091     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7005 [cluster]root     20096     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7006 [cluster]

停止节点

$ sudo /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001 shutdown

节点加入集群

$ sudo cp ~/redis-4.0.9/src/redis-trib.rb /usr/local/redis-cluster/bin/redis-trib$ sudo chmod a+x /usr/local/redis-cluster/bin/redis-trib$ sudo /usr/local/redis-cluster/bin/redis-trib create --replicas 1 \> 192.168.33.110:7001 \> 192.168.33.110:7002 \> 192.168.33.110:7003 \> 192.168.33.110:7004 \> 192.168.33.110:7005 \> 192.168.33.110:7006 \>>> Creating cluster>>> Performing hash slots allocation on 6 nodes...Using 3 masters:127.0.0.1:7001127.0.0.1:7002127.0.0.1:7003Adding replica 127.0.0.1:7005 to 127.0.0.1:7001Adding replica 127.0.0.1:7006 to 127.0.0.1:7002Adding replica 127.0.0.1:7004 to 127.0.0.1:7003>>> Trying to optimize slaves allocation for anti-affinity[WARNING] Some slaves are in the same host as their masterM: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001   slots:0-5460 (5461 slots) masterM: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002   slots:5461-10922 (5462 slots) masterM: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003   slots:10923-16383 (5461 slots) masterS: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0eS: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9Can I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join..>>> Performing Cluster Check (using node 127.0.0.1:7001)M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001   slots:0-5460 (5461 slots) master   1 additional replica(s)S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004   slots: (0 slots) slave   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0eS: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006   slots: (0 slots) slave   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005   slots: (0 slots) slave   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002   slots:5461-10922 (5462 slots) master   1 additional replica(s)M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003   slots:10923-16383 (5461 slots) master   1 additional replica(s)[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

启动脚本

$ sudo vim /usr/local/redis-cluster/bin/start.sh#!/usr/bin/env bash
RedisServerPath=/usr/local/redis-cluster/binRedisConfPath=/usr/local/redis-cluster/etc
for i in 1 2 3 4 5 6; do    $RedisServerPath/redis-server $RedisConfPath/redis-700$i.confdone
/usr/local/redis-cluster/bin/redis-trib.rb create --replicas 1 \192.168.33.110:7001 \192.168.33.110:7002 \192.168.33.110:7003 \192.168.33.110:7004 \192.168.33.110:7005 \192.168.33.110:7006 \$ sudo chmod a+x /usr/local/redis-cluster/bin/start.sh

停止脚本

$ sudo vim /usr/local/redis-cluster/bin/stop.sh#!/usr/bin/env bash
RedisClusterPath=/usr/local/redis-clusterRedisCliPath=/usr/local/redis-cluster/binip=192.168.33.110
for i in 1 2 3 4 5 6; do    $RedisCliPath/redis-cli -c -h $ip -p 700$i shutdowndone
rm -rf $RedisClusterPath/log/*rm -rf $RedisClusterPath/run/*rm -rf $RedisClusterPath/var/*$ sudo chmod a+x /usr/local/redis-cluster/bin/stop.sh

查看信息

$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001192.168.33.110:7001> cluster nodes # 查看节点信息5521b06a46910c873e7ee8a274f2cb9fc41ffed1 192.168.33.110:7003@17003 master - 0 1526681994008 3 connected 10923-163838153a1b35e0fab1e48f0171653f6cffd82cbc5a6 192.168.33.110:7004@17004 slave d7e0395152496f41fe73fbbdabc8499c79f6632e 0 1526681996033 4 connecteda2ff29661185f6ad17c99232d0490b3814ba0164 192.168.33.110:7006@17006 slave affc237bf177fd4ac591314736a1358e16545d0c 0 1526681995020 6 connectedd7e0395152496f41fe73fbbdabc8499c79f6632e 192.168.33.110:7002@17002 master - 0 1526681993000 2 connected 5461-109228b613e4b5abbdf3c2486dfcb8bc1e9dde1897820 192.168.33.110:7005@17005 slave 5521b06a46910c873e7ee8a274f2cb9fc41ffed1 0 1526681993000 5 connectedaffc237bf177fd4ac591314736a1358e16545d0c 192.168.33.110:7001@17001 myself,master - 0 1526681994000 1 connected 0-5460192.168.33.110:7001> cluster info # 查看集群信息cluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:1cluster_stats_messages_ping_sent:59501cluster_stats_messages_pong_sent:61755cluster_stats_messages_sent:121256cluster_stats_messages_ping_received:61750cluster_stats_messages_pong_received:59501cluster_stats_messages_meet_received:5cluster_stats_messages_received:121256

实例操作

客户端操作

// 从某个节点设置值后，去不同的节点获取设置的值$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001192.168.33.110:7001> set foo bar-> Redirected to slot [12182] located at 192.168.33.110:7003OK$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001192.168.33.110:7001> get foo-> Redirected to slot [12182] located at 192.168.33.110:7003"bar"$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7002192.168.33.110:7002> get foo-> Redirected to slot [12182] located at 192.168.33.110:7003"bar"$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7003192.168.33.110:7003> get foo"bar"$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7004192.168.33.110:7004> get foo-> Redirected to slot [12182] located at 192.168.33.110:7003"bar"$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7005192.168.33.110:7005> get foo-> Redirected to slot [12182] located at 192.168.33.110:7003"bar"$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7006192.168.33.110:7006> get foo-> Redirected to slot [12182] located at 192.168.33.110:7003"bar"

程序操作

下载安装

Flexible and feature-complete Redis client for PHP and HHVM
Quick tour

$ cd ~$ mkdir test$ cd test$ composer init$ composer install$ composer require predis/predis

测试代码


require __DIR__ . '/vendor/autoload.php';
for ($i = 0; $i < 6; $i++) {    $parameters[] = [        'host'               => '192.168.33.110',        'port'               => '700' . $i,        'password'           => null,        'database'           => 0,        'timeout'            => 5,        'read_write_timeout' => 5,    ];}$options = ['cluster' => 'redis'];
try {    $cluster = new \Predis\Client($parameters, $options);} catch (\Predis\PredisException $e) {    echo $e->getMessage();    exit();}
for ($i = 1; $i < 100; $i++) {    $format = 'No.%s is a RedisCluster test.';    $cluster->set("$i", sprintf($format, $i));    echo $cluster->get("$i") . PHP_EOL;}

测试结果

$ php cluster.phpNo.1 is a Redis cluster test.No.2 is a Redis cluster test.No.3 is a Redis cluster test....No.99 is a Redis cluster test.
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001192.168.33.110:7001> get 11"No.11 is a RedisCluster test."192.168.33.110:7001> get 23-> Redirected to slot [9671] located at 192.168.33.110:7002"No.23 is a RedisCluster test."192.168.33.110:7002> get 33"No.33 is a RedisCluster test."192.168.33.110:7002> get 67"No.67 is a RedisCluster test."192.168.33.110:7002> get 88-> Redirected to slot [15207] located at 192.168.33.110:7003"No.88 is a RedisCluster test."192.168.33.110:7003> get 99-> Redirected to slot [6263] located at 192.168.33.110:7002"No.99 is a RedisCluster test."192.168.33.110:7002> get 100-> Redirected to slot [339] located at 192.168.33.110:7001(nil)192.168.33.110:7001> get 17-> Redirected to slot [12304] located at 192.168.33.110:7003"No.17 is a RedisCluster test."192.168.33.110:7003> get 42-> Redirected to slot [8000] located at 192.168.33.110:7002"No.42 is a RedisCluster test."192.168.33.110:7002> get 56-> Redirected to slot [11509] located at 192.168.33.110:7003"No.56 is a RedisCluster test."192.168.33.110:7003> get 61-> Redirected to slot [2369] located at 192.168.33.110:7001"No.61 is a RedisCluster test."

Redis 集群搭建