keepalived实现haproxy高可用详解

一，keepalived介绍

keepalived是一个可以实现某些资源高可用的开源软件，其主要的组件包括core，check，vrrp，libipfwc，libipvs，这里说下各个组件的功能。

core：keepalived的核心组件，负责主进程的启动和维护以及加载解析配置文件等。

check：负责healthchecker,负责各种健康检查方式，和对应的配置解析以及LVS的配置解析。

vrrp：vrrpd的子进程。

libipfwc：结合iptables的ipchains库来使用。

libipvs：结合LVS使用。

keepalived启动后会生成3个进程，master主进程，VRRP子进程，healthchecker子进程。

VRRP协议是实现keepalived高可用的一个基础，下面说一下VRRP的实现原理：

VRRP虚拟路由(VRRP router)，VRRP是一个“选举”协议，它能够动态地将一个虚拟路由器的责任指定至同一个VRRP组中的其它路由器上，VRRP的优势：

冗余：可以使用多个路由器设备作为LAN客户端的默认网关，大大降低了默认网关成为单点故障的可能性；

负载共享：允许来自LAN客户端的流量由多个路由器设备所共享；

多VRRP组：在一个路由器物理接口上可配置多达255个VRRP组；

多IP地址：基于接口别名在同一个物理接口上配置多个IP地址，从而支持在同一个物理接口上接入多个子网；

抢占：在master故障时允许优先级更高的backup成为master；

通告协议：使用IANA所指定的组播地址224.0.0.18进行VRRP通告；

VRRP追踪：基于接口状态来改变其VRRP优先级来确定最佳的VRRP路由器成为master；

二，实验环境：

192.168.30.116 OS：Centos 6.4 x86_64

192.168.30.117 OS：Centos 6.4 x86_64

keepalived版本：keepalived-1.2.7

首先，分别在2台服务器上，安装keepalived,haproxy，由于系统版本是Centos 6.4，这2个软件已经被整合在内部了，所以使用yum来安装

[root@master ~]# yum -y install keepalived haproxy[root@backup ~]# yum -y install keepalived haproxy

接下来，先配置master与backup服务器的优先级，virtual_router_id（同一个实例2台服务器必须相同），编辑配置文件:

[root@master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak[root@backup ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak[root@master ~]# vi /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {notification_email {acassen@firewall.locfailover@firewall.locsysadmin@firewall.loc}notification_email_from Alexandre.Cassen@firewall.locsmtp_server 192.168.200.1smtp_connect_timeout 30router_id LVS_DEVEL}vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.30.230}}[root@backup ~]# vi /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {notification_email {acassen@firewall.locfailover@firewall.locsysadmin@firewall.loc}notification_email_from Alexandre.Cassen@firewall.locsmtp_server 192.168.200.1smtp_connect_timeout 30router_id LVS_DEVEL}vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 51priority 99advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.30.230}}

global_defs：全局配置标识破；

notification_email：表示告警时发送的邮件地址；

notification_email_from：表示发送邮件的源地址；

smtp_server：发送邮件的smtp服务器地址；

router_id：机器标识；

vrrp_instance：定义一个vrrp实例；

state：state指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定，里如果这里设置为master，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为master；

interface：实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的；

virtual router id：这里设置VRID，这里非常重要，相同的VRID为一个组，他将决定多播的MAC地址；

priority 100：设置本节点的优先级，优先级高的为master；

advert int：检查间隔，默认为1秒；

virtual ipaddress：这里设置的就是VIP，也就是虚拟IP地址，他随着state的变化而增加删除，当state为master的时候就添加，当state为backup的时候删除，这里主要是有优先级来决定的，和state设置的值没有多大关系，这里可以设置多个IP地址；

authentication：这里设置认证；

auth type：认证方式，可以是PASS或AH两种认证方式；

auth pass：认证密码；

启动俩台服务器的keepalived

[root@master ~]# service keepalived startStarting keepalived: [ OK ][root@backup ~]# service keepalived startStarting keepalived: [ OK ][root@master ~]# tail -f /var/log/messagesJan 10 11:40:56 localhost Keepalived_healthcheckers[19368]: Using LinkWatch kernel netlink reflector...Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Opening file '/etc/keepalived/keepalived.conf'.Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Configuration is using : 63019 BytesJan 10 11:40:56 localhost Keepalived_vrrp[19369]: Using LinkWatch kernel netlink reflector...Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]Jan 10 11:40:57 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Entering MASTER STATEJan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) setting protocol VIPs.Jan 10 11:40:58 localhost Keepalived_healthcheckers[19368]: Netlink reflector reports IP 192.168.30.230 addedJan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 11:41:03 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionJan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230[root@master ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWNlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ffinet 192.168.30.116/24 brd 192.168.30.255 scope global eth0inet 192.168.30.230/32 scope global eth0inet6 fe80::20c:29ff:fef3:fcba/64 scope linkvalid_lft forever preferred_lft forever

发现刚才定义的virtual ipaddress在master服务器上，因为优先级较高

停止master服务器的keepalived服务器，看IP会不会转移到backup

[root@master ~]# service keepalived stopStopping keepalived: [ OK ][root@backup ~]# tail -f /var/log/messagesJan 10 12:12:46 localhost Keepalived_vrrp[18581]: Opening file '/etc/keepalived/keepalived.conf'.Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Configuration is using : 63017 BytesJan 10 12:12:46 localhost Keepalived_vrrp[18581]: Using LinkWatch kernel netlink reflector...Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Opening file '/etc/keepalived/keepalived.conf'.Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Configuration is using : 7324 BytesJan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Using LinkWatch kernel netlink reflector...Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advertJan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATEJan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATEJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 addedJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230[root@backup ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWNlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ffinet 192.168.30.117/24 brd 192.168.30.255 scope global eth0inet 192.168.30.230/32 scope global eth0inet6 fe80::20c:29ff:fe5b:50f9/64 scope linkvalid_lft forever preferred_lft forever

发现IP已经转移到backup服务器

下面重新启动master的keepalived

[root@master ~]# service keepalived startStarting keepalived: [ OK ][root@backup ~]# tail -f /var/log/messagesJan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advertJan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATEJan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATEJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 addedJan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advertJan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATEJan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) removing protocol VIPs.Jan 10 12:18:20 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 removed[root@master ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWNlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ffinet 192.168.30.116/24 brd 192.168.30.255 scope global eth0inet 192.168.30.230/32 scope global eth0inet6 fe80::20c:29ff:fef3:fcba/64 scope linkvalid_lft forever preferred_lft forever

发现IP已经重新转移到master服务器

现在编写haproxy状态检测脚本，来实现haproxy的健康检测：

[root@master ~]# cat haproxy_pid.sh#!/bin/bashwhile :dohaproxypid=`ps -C haproxy --no-header | wc -l`if [ $haproxypid -eq 0 ];thenservice haproxy startsleep 5haproxypid=`ps -C haproxy --no-header | wc -l`echo $haproxypidif [ $haproxypid -eq 0 ];then/etc/init.d/keepalived stopfifisleep 5done# 启动backup的haproxy[root@backup ~]# service haproxy startStarting haproxy: [ OK ]

模拟故障，先让httpd进程开启，修改haproxy监听端口为80，使得haproxy进程无法启动，看资源会不会转移到backup服务器

[root@master ~]# vi /etc/haproxy/haproxy.cfgfrontend main *:5000 改为frontend main *:80# 启动httpd进程[root@master ~]# scp -pr /etc/haproxy/haproxy.cfg root@192.168.30.117:/etc/haproxy/[root@master ~]# service httpd startStarting httpd: httpd: apr_sockaddr_info_get() failed for httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName[ OK ][root@master ~]# netstat -antpl | grep :80tcp 00 :::80 :::* LISTEN19965/httpd#运行状态检测脚本[root@master ~]# nohup /root/haproxy_pid.sh &[root@master ~]# scp -pr haproxy_pid.sh root@192.168.30.117:/root/[root@bakcup ~]# nohup /root/haproxy_pid.sh &[root@master ~]# tail -f /var/log/messagesJan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Transition to MASTER STATEJan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Entering MASTER STATEJan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) setting protocol VIPs.Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 12:02:30 localhost Keepalived_healthcheckers[19848]: Netlink reflector reports IP 192.168.30.230 addedJan 10 12:02:35 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230Jan 10 12:14:49 localhost Keepalived[19847]: Stopping Keepalived v1.2.7 (02/21,)Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) sending 0 priorityJan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) removing protocol VIPs.[root@master ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWNlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ffinet 192.168.30.116/24 brd 192.168.30.255 scope global eth0inet6 fe80::20c:29ff:fef3:fcba/64 scope linkvalid_lft forever preferred_lft forever[root@backup ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWNlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ffinet 192.168.30.117/24 brd 192.168.30.255 scope global eth0inet 192.168.30.230/32 scope global eth0inet6 fe80::20c:29ff:fe5b:50f9/64 scope linkvalid_lft forever preferred_lft forever[root@backup ~]# ps aux | grep haproxyhaproxy 19054 0.0 0.0 18688 1280 ? Ss 12:47 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pidroot19097 0.0 0.0 103248 828 pts/0 S+ 12:58 0:00 grep haproxy

发现IP已经转移到backup服务器，实现了keepalived对于haproxy故障的高可用。

keepalived常见的启动报错：

5913 May 16 15:26:04 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.755914 May 16 15:26:04 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert5915 May 16 15:26:04 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!5916 May 16 15:26:04 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment...5917 May 16 15:26:05 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.755918 May 16 15:26:05 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert5919 May 16 15:26:05 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!5920 May 16 15:26:05 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment.

解决方法：

在同一网段内virtual_router_id 值不能相同，如果相同会在messages中收到VRRP错误包 ，所以需要更改 virual_router_id。