用Keepalived实现PostgreSQL高可用

Keepalived可提供VRRP以及health-check功能，可以只用它提供双机浮动的vip（VRRP虚拟路由功能），这样可以简单实现一个双机热备高可用功能。

Keepalived是以VRRP虚拟路由冗余协议为基础实现高可用的，可以认为是实现路由器高可用的协议，即将N台提供相同功能的路由器组成一个路由器组，这个组里面有一个master和多个backup，master上面有一个对外提供服务的vip（该路由器所在局域网内其他机器的默认路由为该vip），master会发组播，当backup收不到VRRP包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。

下载地址：（复制链接至浏览器中浏览）

https://www.keepalived.org/download.html
https://github.com/acassen/keepalived

Keepalived工作在TCP/IP参考模型的三层、四层、五层（物理层，链路层）：

2.1 主机规划

2.2 架构图

3.1 关闭防火墙和selinux

[root@localhost ~]# systemctl stop firewalld
[root@localhost keepalived]# vi /etc/selinux/config
SELINUX=disabled
[root@localhost keepalived]# getenforce
Disabled

3.2 安装Keepalived

[root@localhost ~]# yum install gcc openssl openssl‐devel libnl libnl‐devel ipvsadm -y
[root@localhost ~]# cd /usr/local/src/
[root@localhost src]# tar -zxf keepalived-2.2.4.tar.gz 
[root@localhost src]# cd keepalived-2.2.4
[root@localhost keepalived-2.2.4] ./configure --prefix=/usr/local/keepalived
[root@localhost keepalived-2.2.4] make && make install
[root@localhost keepalived]#  cd /usr/local/keepalived
[root@localhost keepalived]# 
ln -s /usr/local/src/keepalived-2.2.4/keepalived/etc/init.d/keepalived /etc/init.d/
[root@localhost keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@localhost keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /usr/local/sbin/
[root@localhost keepalived]# mkdir /etc/keepalived/
[root@localhost keepalived]# ln -s  /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@localhost ~]# cat  /etc/keepalived/keepalived.conf 

[root@localhost keepalived-2.2.4]# systemctl start keepalived
[root@localhost keepalived-2.2.4]# systemctl status keepalived

3.3 PostgreSQL数据库异步主从流复制搭建

192.168.254.128服务器上配置:

1、创建流复制用户

postgres=# create user replia replication superuser password ‘123qwert’;

给superuser权限是为了后面做pg_rewind操作。

2、数据目录下配置pg_hba.conf文件添加（192.168.254.129也需要添加）

     host    replication     all             192.168.254.1/24      md5

3、配置环境变量

[postgres@localhost pg_data]$ cat ~/.bash_profile 
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then 	. ~/.bashrc; fi
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export PGHOME=/usr/local/pgsql
export LD_LIBRARY_PATH=$PGHOME/lib:/lib64:/usr/lib64:/usr/local/lib64:/lib:/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH
export DATE=`date +"%Y%m%d%H%M"`
export PATH=$PGHOME/bin:$PATH:.
export MANPATH=$PGHOME/share/man:$MANPATH
export PGUSER=postgres
export PGDATA=/data/pg_data
export PGDATABASE=postgres
alias rm='rm -i'
alias ll='ls -lh'
export LANG="zh_CN.UTF8"

在192.168.254.129上进行：

4、~/.bash_profile 和上面输入一样的内容

5、制作基础备份

 [postgres@localhost ~]$ pg_basebackup -h192.168.254.128 -Ureplia -R -Fp -P --verbose  -c fast -D /data/pg_data

6、启动从库

 [postgres@localhost ~]$ pg_ctl -D /data/pg_data/ start

192.168.254.128上查看主库信息：

7、查看主库信息

 postgres=# select * from pg_stat_replication ;

3.4 PostgreSQL数据库配置

创建数据库Keepalived，并且创建表探测表sr_delay，后续Keepalived探测，刷新sr_delay表的last_alive字段为当前探测时间。这张表用来判断主备延迟情况，数据库故障切换时会用到这张表。

数据库配置：

postgres=# create user keepalived password '123qwert' CONNECTION LIMIT 4 ;
CREATE ROLE
postgres=# create database keepalived owner keepalived;
CREATE DATABASE
postgres=# \c keepalived keepalived
You are now connected to database "keepalived" as user "keepalived".
keepalived=> create table sr_delay(id int4, last_alive timestamp(0) without time zone);
CREATE TABLE

表sr_delay只允许写入一条记录，并且不允许删除此表数据，通过触发器实现。创建触发器函数，如下所示：

CREATE FUNCTION cannt_delete () 
RETURNS trigger 
LANGUAGE plpgsql AS $$ 
BEGIN 
RAISE EXCEPTION 'Table sr_delay can not delete !';
 END; 
$$;

创建触发器：

CREATE TRIGGER trigger_sr_delay_del 
BEFORE DELETE ON sr_delay 
FOR EACH ROW EXECUTE PROCEDURE cannt_delete() ; 
CREATE TRIGGER trigger_sr_delay_tru BEFORE TRUNCATE ON sr_delay FOR STATEMENT EXECUTE PROCEDURE cannt_delete() ;

插入数据：

 INSERT INTO sr_delay VALUES(1,now()) ;

参数部分：

hot_standby = on
wal_log_hints = on 
archive_mode=on
archive_command='test ! -f /data/pg_archive/%f && cp %p /data/pg_archive/%f'

后续Keepalived会每隔指定时间探测PostgreSQL数据库存活， 并且以Keepalived用户登录Keepalived数据库刷新这张表， 配置主备库pg_hba.conf，增加如下内容：

host    keepalived    keepalived        192.168.254.128/32      md5
host    keepalived    keepalived        192.168.254.129/32      md5

随后pg_ctl reload 操作使配置生效。

3.5 Keepalived 配置

[root@localhost ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   # 邮件通知信息
   notification_email {
     # 定义收件人
    394106346@qq.com 
   }
   # 定义发件人
   notification_email_from Alexandre.Cassen@firewall.loc
   # SMTP服务器地址
   smtp_server 192.168.254.128
   smtp_connect_timeout 30
   # 路由器标识，一般不用改，也可以写成每个主机自己的主机名
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

# 定义用于实例执行的脚本内容，比如可以在线降低优先级，用于强制切换
vrrp_script check_pg_alived { 
script "/data/scripts/pg_monitor.sh" 
interval 10 
fall 3 
}

# 一个vrrp_instance就是定义一个虚拟路由器的，实例名称
vrrp_instance VI_1 {
    # 定义初始状态，可以是MASTER或者BACKUP
    state MASTER
    #非抢占模式
    nopreempt
    # 工作接口，通告选举使用哪个接口进行
    interface ens33
    # 虚拟路由ID，如果是一组虚拟路由就定义一个ID，如果是多组就要定义多个，而且这个虚拟
    # ID还是虚拟MAC最后一段地址的信息，取值范围0-255
    virtual_router_id 51
    #权重 如果你上面定义了MASTER,这里的优先级就需要定义的比其他的高
    priority 100
    #通告频率 单位s
    advert_int 1
    # 通信认证机制，这里是明文认证还有一种是加密认证
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 设置虚拟VIP地址
    virtual_ipaddress {
        192.168.254.11
    }
    # 追踪脚本，通常用于去执行上面的vrrp_script定义的脚本内容
    track_script {
      check_pg_alived
    }
   # 如果主机状态变成Master|Backup|Fault之后会去执行的通知脚本，脚本要自己写
    notify_master "/data/scripts/failover.sh"
    notify_fault  "/data/scripts/fault.sh"
}

以上是Keepalived主节点的配置，Keepalived备节点的priority参数改成90 ，state改为BACKUP，以及smtp_server 改为192.168.254.129，其余参数配置一样。

3.6 数据库监控脚本

[postgres@localhost scripts]$ cat /data/scripts/pg_monitor.sh 
#!/bin/bash 
# 配置环境变量 
pgport=5432 
pguser=keepalived 
pgdb=keepalived 
pgpwd='123qwert'
LANG=en_US.utf8 
PGHOME=/usr/local/pgsql/ 
LD_LIBRARY_PATH=$PGHOME/lib:/lib64:/usr/lib64:/usr/local/lib64:/lib:/usr/lib:/usr/local/lib 
PATH=$PGHOME/bin:$PATH:.
MONITOR_LOG="/data/scripts/pg_monitor.log" 
SQL1="UPDATE sr_delay SET last_alive= now();" 
SQL2='SELECT 1;' 
keeplognums=30000

#pg_port_status=`lsof ‐i :$pgport | grep LISTEN | wc ‐l` 
#pg_port_status=`ps ‐ef | grep LISTEN | wc ‐l` 
#if [ $pg_port_status ‐lt 1 ];then 
# echo ‐e `date +"%F %T"` "Error: The postgreSQL is not running,please chec k the postgreSQL server status!" >> $LOGFILE 
# exit 1 
#fi 

#此脚本不检查备库存活状态，如果是各库则退出 

standby_flg=`psql -p$pgport -Upostgres -At -c "SELECT pg_is_in_recovery();"` 
if [ ${standby_flg} == 't' ]; 
then 
 echo ‐e "`date +%F\ %T`: This is a standby database, exit!\n" > $MONITOR_LOG 
 exit 0 
fi
export PGPASSWORD=$pgpwd
#主库更新sr_delay 表 
echo $SQL1 | psql -At -p $pgport -U $pguser -d $pgdb >> $MONITOR_LOG 
#判断主库是否可用
echo $SQL2 | psql -At -p $pgport -U $pguser -d $pgdb 
if [ $? -eq 0 ] ;
then 
 echo -e "`date +%F\ %T`：Primary db is health." >> $MONITOR_LOG 
 exit 0 
else 
 echo -e "`date +%F\ %T`:Attention: Primary db is not health!" >> $MONITOR_LOG 
 exit 1 
fi

#日志保留 keeplognums 行
if [ ! -f ${MONITOR_LOG} ] ;then touch ${MONITOR_LOG};fi 
lognums=`cat ${MONITOR_LOG} |wc -l`
catnum=$((${lognums} -${keeplognums}))
if [[ $lognums -gt ${keeplognums} ]] ; then sed -i "1,${catnum}d" ${MONITOR_LOG}; fi

此脚本每隔10秒执行一次，执行频率由keepalived.conf配置文件中interval参数设置， 脚本主要作用为：

• 检测主库是否存活。

• 更新sr_delay表last_alive字段为当前探测时间。

[postgres@localhost scripts]$ cat /data/scripts/failover.sh 
#!/bin/bash 
export PGPORT=5432 
export PGUSER=keepalived 
export PG_OS_USER=postgres 
export PGDBNAME=keepalived 
export LANG=en_US.UTF-8 
export PGPATH=/usr/local/pgsql/bin/ 
export PATH=$PATH:$PGPATH 
export PGMIP=127.0.0.1 
LOGFILE='/data/scripts/failover.log'
# 主备数据库同步时延，单位为秒 
sr_allowed_delay_time=100

SQL1='select pg_is_in_recovery from pg_is_in_recovery();' 
SQL2="select count(1) as delay_time from sr_delay where now()<(last_alive + interval '${sr_allowed_delay_time} seconds');"
#SQL2="select count(1) as delay_time from sr_delay where now()<(last_alive + interval '100 seconds');" 
sleep $sr_allowed_delay_time
db_role=`echo $SQL1 | psql -At -p $PGPORT -U $PGUSER -d $PGDBNAME -w` 
db_sr_delaytime=`echo $SQL2 | psql -p $PGPORT -d $PGDBNAME -U $PGUSER -At -w` 
SWITCH_COMMAND='pg_ctl promote -D /data/pg_data/' 
# 如果为备库，且延迟大于指定时间则切换为主库

if [ ${db_role} == 'f' ];
then 
	echo -e `date +"%F %T"` "Attention:The current postgreSQL DB is master database,cannot switched!" >> $LOGFILE 
	exit 0 
fi

if [ $db_sr_delaytime -gt 0 ];
then 
	echo -e `date +"%F %T"` "Attention:The current master database is health,the standby DB cannot switched!" >> $LOGFILE 
	exit 0 
fi

if [ !$db_sr_delaytime ];
then 
	echo -e `date +"%F %T"` "Attention:The current database is statndby,ready to switch master database!" >> $LOGFILE 
	su - $PG_OS_USER -c "$SWITCH_COMMAND" 
elif [ $? -eq 0 ];
then
	echo -e `date +"%F %T"` "success:The current standby database successed to switched the primary PG database !" >> $LOGFILE 
	exit 0 
else 
	echo -e `date +"%F %T"` "Error: the standby database failed to switch the primary PG database !,pelease checked it!" >> $LOGFILE 
	exit 1
fi

[postgres@localhost scripts]$ cat /data/scripts/fault.sh
#!/bin/bash
GFILE=/data/scripts/pg_db_fault.log 
PGPORT=5432 
PGMIP=192.168.254.128
echo -e `date +"%F %T"` "Error:Because of the priamry DB happend some unknown problem,So turn off the PostgreSQL Database!" >> $LOGFILE 

PGPID="`head -n1 /data/pg_data/postmaster.pid`" 

systemctl stop keepalived 

kill -9 $PGPID 
if [ $? -eq 0 ] ;
then 
	echo -e `date +"%F %T"` "Error:Because of the priamry DB happend some unknown problem,So turn off the PostgreSQL Database!" >> $LOGFILE
	systemctl stop keepalived 
	exit 1 
fi

[postgres@localhost scripts]$ chmod 755 /data/scripts/pg_monitor.sh /data/scripts/failover.sh /data/scripts/fault.sh

4.1 在192.168.254.128上停掉数据库

[root@localhost ~]# su - postgres
[postgres@localhost ~]$ pg_ctl -D /data/pg_data/ stop

查看192.168.254.129上的数据库状态：

[postgres@localhost ~]$ psql
psql (13.3)
Type "help" for help.
postgres=# insert into t5 (name) values ('b002');
INSERT 0 1

修复192.168.254.128。

使用pg_rewind 同步新主库的数据到原主库：

pg_rewind -R --target-pgdata '/data/pg_data' --source-server 'host=192.168.254.129 port=5432 user=replia password=123qwert dbname=postgres'

如果这个地方咱们的wal日志已经被覆盖了了很多，那么就需要将我们的备份日志/data/pg_archive/给scp过来。

[postgres@localhost ~]$ scp postgres@192.168.254.129:/data/pg_archive/00000002*  /data/pg_archive/

(此时需要注意新主和备上面TimeLineID的差异。并且这种情况需要在配置文件中加上：

restore_command='cp /data/pg_archive/%f %p'

）

启动数据库，然后检查主备wal sender和receiver是否正常：

[postgres@localhost ~]$ pg_ctl -D /data/pg_data/ start
[postgres@localhost ~]$ ps axu |grep walreceiver
postgres  53474  0.0  0.2 342008  2136 ?        Ss   14:45   0:01 postgres: walreceiver streaming 1/4A036EF0

192.168.254.128上启动Keepalived，查看Keepalived状态：

[root@localhost ~]# systemctl  start keepalived
[root@localhost ~]# systemctl  status keepalived

4.2 再次切换：

在做切换前，先要检查192.168.254.128和192.168.254.129上面的Keepalived正常以及vip和主从正常，在192.168.254.129上停掉数据库:

[postgres@localhost ~]$ pg_ctl -D /data/pg_data/ stop -mf

查看192.168.254.128上的数据库状态：

[postgres@localhost pg_data]$ psql
psql (13.3)
Type "help" for help.
postgres=# insert into t5 (name) values ('b005');
INSERT 0 1

修复192.168.254.129。

使用pg_rewind 同步新主库的数据到原主库：

pg_rewind -R --target-pgdata '/data/pg_data' --source-server 'host=192.168.254.128 port=5432 user=replia password=123qwert dbname=postgres'

如果这个地方咱们的wal日志已经被覆盖了了很多，那么就需要将我们的备份日志/data/pg_archive/给scp过来。

[postgres@localhost ~]$ scp postgres@192.168.254.128:/data/pg_archive/00000003*  /data/pg_archive/  

(此时需要注意新主和备上面TimeLineID的差异。并且这种情况需要在配置文件中加上：

restore_command='cp /data/pg_archive/%f %p'

）

启动数据库，然后检查主备wal sender和receiver是否正常：

[postgres@localhost ~]$ pg_ctl -D /data/pg_data/ start
[postgres@localhost ~]$ ps axu |grep walreceiver

192.168.254.129上启动Keepalived，查看Keepalived状态：

[root@localhost ~]# systemctl  start keepalived
[root@localhost ~]# systemctl  status keepalived

1、可以完善failover时自己的延迟切换逻辑。

2、可以完善pg_rewind的实现，用脚本代替手动方式。

3、Keepalived较为灵活，能够在脚本上添加更多校验和自己的规则。

墨天轮原文链接：https://www.modb.pro/db/113653?sjhy（复制到浏览器或者点击“阅读原文”立即查看）

关于作者

向前龙  ，云和恩墨PG技术顾问，拥有OGCA、PGCE证书。

END

推荐阅读：

Redis 哨兵模式

Dubbo源码分析：小白入门篇

一周学完MyBatis源码，万字总结

欢迎关注微信公众号：互联网全栈架构，收取更多有价值的信息。