This blog post describes a set-up of a high-available web service with just two running servers. Quite a few examples on the internet make use of a third server as master, but it may create an unnecessary single-point-of-failure.

keepalived is normally used to provide a fail-over feature for load balancers, such as HAProxy. In this case, HAProxy does load balancing on your application servers and the setup requires at least 4 servers to make sense.

In this post, we skip the HAProxy layer as the setup is for our service, which has load-balancing built-in. What we need is a simple but robust fail-over for a high-availability service, no need to deploy more servers.


Enigma Bridge PKI - Create Your PKI in 18 Minutes

Start using your own PKI within minutes. Use of secure hardware signing ensures high-level of security while you get all the benefits of the cloud.

Enigma Bridge brings you a fully featured and simple PKI system with a certification authority and an OCSP responder supported with FIPS140-2 Level 3 hardware-protected keys. Includes an out-of-the-box HTTPS with a browser trusted certificate.


Network Setup

Server A is master - running the web service, server B is backup. If the master goes down, the backup will take over the web service for the master.

Let’s assume that our web service has a DNS A record 93.1.1.1.

This is the floating IP address the current master server maps to itself. Once the master server A goes down, server B takes over and re-maps floating IP to itself. Besides that each server has its own public IP address on the en1 interface. The keep-alive mechanism works over the cross cable connected en0 interfaces of both servers, with static IP addresses. (If you don’t have this additional connection, you can easily use en1 wherever your read en0.


Both up

Once the master server goes down situation looks like this:


Master down

The servers communicate with each other via en0 direct cable connection - using Virtual Router Redundancy Protocol (VRRP). It sends each second a keep-alive packet with advertisements. If there are 3 missing keep-alive packets, backup server will take over the floating IP address and designates itself as master.

In our setup, once the master server is back online, it starts sending advertisements to the backup server and re-takes the floating IP address back as it has higher priority.

This keep-alive mechanism is quite robust. Whenever the master server is not able to send keep-alives (e.g., power outage, network problem, OS problem, deadlock) the web service can be impaired as well so it makes sense to switch to the backup server. Floating IP address reassignment is done automatically by keepalived - by sending gratious ARP packet to the network. In this setup both servers must be in the same network segment.

Moreover Keepalived can check another services on the host with scripts. E.g., if the web service server is not running it can switch itself to fault state so backup server takes over the floating IP. More on this below or in the keepalived User Guide.

Installation

Keepalived is in the standard repositories, install it with yum / apt-get.

yum install keepalived

Server A configuration (Master)

The configuration file: /etc/keepalived/keepalived.conf

The constants in the configuration (IPs, interfaces) match the diagrams above.

! Configuration File for keepalived

global_defs {
   notification_email {
     support@enigmabridge.com
   }
   notification_email_from server_a@enigmabridge.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER   # Server A is master
    interface en0  # VRRP is running on en0 interface
    virtual_router_id 151
    priority 101
    advert_int 1
    dont_track_primary

    unicast_src_ip 10.0.0.2 # Server A IP
    unicast_peer {
        10.0.0.3  # Server B IP
    }

    authentication {
        auth_type PASS
        auth_pass secret_password
    }

    virtual_ipaddress {
	    93.1.1.1/24 dev en1
    }
}

Server B configuration (Backup)

The changed directives here are:

  • state - by default, this server is the backup one.
  • priority - lower the priority number so after original master is back up again it takes the floating IP back
  • unicast_peer_ip - the IP address of the server B on the en0 interface
  • unicast_peer - ip address(es) of other servers, here we have only the server A (if not set, VRRP will do broadcasts)


! Configuration File for keepalived

global_defs {
   notification_email {
     support@enigmabridge.com
   }
   notification_email_from server_b@enigmabridge.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state BACKUP  # Server B is backup by default
    interface en0
    virtual_router_id 151
    priority 100
    advert_int 1
    dont_track_primary

    unicast_src_ip 10.0.0.3 # server B IP
    unicast_peer {
        10.0.0.2  # server A IP
    }

    authentication {
        auth_type PASS
        auth_pass secret_password
    }

    virtual_ipaddress {
        93.1.1.1/24 dev en1
    }
}

Enable keepalived after start

CentOS 7:

systemctl enable keepalived.service

CentOS 6:

chkconfig keepalived on

Start keepalived

CentOS 7:

systemctl start keepalived.service

CentOS 6:

/etc/init.d/keepalived start

dont_track_primary

This configuration option is handy if two Keepalived instances are directly connected with a cross cable without any networking component.

If one server goes down, the en0 will be in the disconnected state on the other server - it’s like pulling out the cable from it. This confuses the other running server and may switch it to Fault state - not quite the thing we want as neither of the servers claims the floating address and the service becomes unavailable.

In logs in may look like this:

lis 21 13:53:36 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down
lis 21 13:53:36 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Transition to MASTER STATE
lis 21 13:53:36 parrot NetworkManager[908]: <info>  (en0): link disconnected
lis 21 13:53:37 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 DOWN
lis 21 13:53:37 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Entering FAULT STATE
lis 21 13:53:37 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Now in FAULT state
lis 21 13:53:39 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
lis 21 13:53:39 parrot NetworkManager[908]: <info>  (en0): link connected
lis 21 13:53:41 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down
lis 21 13:53:41 parrot NetworkManager[908]: <info>  (en0): link disconnected
lis 21 13:53:44 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Up 10 Mbps Full Duplex, Flow Control: RX/TX
lis 21 13:53:44 parrot NetworkManager[908]: <info>  (en0): link connected
lis 21 13:53:44 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 UP
lis 21 13:53:48 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Transition to MASTER STATE
lis 21 13:53:48 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down
lis 21 13:53:48 parrot NetworkManager[908]: <info>  (en0): link disconnected
lis 21 13:53:49 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 DOWN
lis 21 13:53:49 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Entering FAULT STATE
lis 21 13:53:49 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Now in FAULT state

The dont_track_primary option will solve this Fault state problem on the peer down event.

Scripts monitoring (optional)

As mentioned above, keepalived can perform regular checks of the services and switch to fault state if not. Keepalived has embedded some checks already - HTTP_GET, SSL_GET or you can use your own check. If script returns 0 as the return value it means everything is OK. If it returns 1 the test fails and keepalived switches itself to fault state.

If your host provides some API the check script can check if API endpoint is operational.

vrrp_script chk_myscript {
  script       "/usr/local/bin/check_api.py"
  interval 2   # check every 2 seconds
  fall 2       # require 2 failures for KO
  rise 2       # require 2 successes for OK
  timeout 10   # 10 second before failing due to timeout
}

To enable this script checking, add the following code under the virtual_ipaddress block, inside the vrrp_instance directive.

track_script {
    chk_myscript
}

Script example

The check script can for example look like the following one. It checks our API endpoint which is supposed to return JSON response with status field in it. If any parsing exception occurs, timeout 15 seconds, status field is missing or any other error is detected, script returns 1. As you guessed, this value means the check failed and the backup server should take over.

#!/usr/bin/env python
import argparse
import sys
import requests
import logging, coloredlogs

logger = logging.getLogger(__name__)
coloredlogs.install(level=logging.INFO)

CHECK_HOST = "127.0.0.1"
CHECK_PORT = 11180
CHECK_TIMEOUT = 15


def main():
    parser = argparse.ArgumentParser(description='EnigmaBridge keepalived test')
    parser.add_argument('--host', dest='host', default=CHECK_HOST,
                        help='Host to check')
    parser.add_argument('--port', dest='port', default=CHECK_PORT, type=int,
                        help='port to check')
    parser.add_argument('--timeout', dest='timeout', default=CHECK_TIMEOUT, type=float,
                        help='request timeout')
    args = parser.parse_args()

    host = args.host
    port = int(args.port)
    timeout = float(args.timeout)

    try:
        r = requests.get('https://%s:%d' % (host, port), timeout=timeout)
        if r.status_code != 200:
            raise ValueError('Status code error: %s' % r.status_code)

        js = r.json()
        if js is None:
            raise ValueError('Json response is empty')

        if 'status' not in js:
            raise ValueError('Status not in JSON')

        # Everything OK.
        sys.exit(0)

    except Exception as e:
        logger.info('Exception: %s' % e)
        sys.exit(1)


if __name__ == '__main__':
    main()

Server A en0 static IP setup

It’s good to tell network manager not to mess with the names of network interfaces by adding NM_CONTROLLED=no to the networking scripts.

The networking script for server A en0 /etc/sysconfig/network-scripts/ifcfg-en0:

TYPE=Ethernet
BOOTPROTO=static
HWADDR=d1:51:88:19:62:09
IPADDR=10.0.0.2
NETMASK=255.255.255.0
IPV4_FAILURE_FATAL=no
IPV6INIT=no
DEVICE=en0
ONBOOT=yes
NM_CONTROLLED=no

Server B en0 static IP setup

The networking script for server B en0 /etc/sysconfig/network-scripts/ifcfg-en0:

TYPE=Ethernet
BOOTPROTO=static
HWADDR=d1:51:88:19:60:09
IPADDR=10.0.0.3
NETMASK=255.255.255.0
IPV4_FAILURE_FATAL=no
IPV6INIT=no
DEVICE=en0
ONBOOT=yes
NM_CONTROLLED=no

Fixed interface ordering

You may also want to fix the network interface ordering by its MAC address so it does not get changed on some system update.

In CentOS we do this by editing /etc/udev/rules.d/70-persistent-net.rules:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="d1:51:88:19:62:09", ATTR{type}=="1", KERNEL=="eth*", NAME="en0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="d1:51:88:19:62:10", ATTR{type}=="1", KERNEL=="eth*", NAME="en1"

Note: we decided not to use ethX as new names for the interfaces because there is some kind of race condition in the udev in CentOS operating system. If we change the order of more eth interfaces it may complain one is already taken. Choosing en is the safe option to avoid such races.

Keepalived: Invalid IP address, skipping…

It may happen keepalived logs something like this:

Nov 21 10:08:43 parrot Keepalived_vrrp[24465]: VRRP parsed invalid IP  93.1.1.1. skipping IP...

The IP looks good so why is keepalived complaining?

If you see this, make sure you have no weird white space characters in the configuration file. It may happen when you copy-paste a configuration snippet from a webpage, a notes application, or a text editor which added these white-space characters unrecognized by keepalived configuration file parser. In that case just re-write the configuration block virtual_ipaddress with vim on the server and restart. It solves this problem.

Configuration reload - service restart

After updating the keepalived configuration and restarting it with systemctl restart keepalived.service you may experience weird behavior of the service, for instance this can be found in logs:

lis 23 11:38:49 panda Keepalived_vrrp[8581]: receive an invalid ip number count associated with VRID!
lis 23 11:38:49 panda Keepalived_vrrp[8581]: bogus VRRP packet received on en0 !!!
lis 23 11:38:49 panda Keepalived_vrrp[8581]: VRRP_Instance(VI_1) Dropping received VRRP packet...

In this case we added a new IP address to the virtual_ipaddress block, restarted both services but it still kept sending these bogus packets.

The problem is restart with systemctl command does not work well for keepalived all the time and sometimes it may fail to reload configuration changes. Then one instances sends advertisements with 2 IP addresses and the other only with one which results in the described behaviour (the same problem occurs if the virtual IP is changed).

The solution is to stop keepalived instances on both servers, wait a few seconds and start them again. This solves the problem.

systemctl stop keepalived.service && sleep 2 && systemctl start keepalived.service

Scripts not working - fault state

It may happen the vrrp_script is not working properly. In that case Keepalived automatically fails to FAULT state without logging a proper error (we used Keepalived 1.2.13). Unfortunately there is also no log in /var/log/audit/audit.log. You can help yourself by attaching a strace to the running daemon to see what is going on:

strace -f -p 11222
[pid 12213] execve("/opt/enigmabridge/eb-keepalived/eb-controller-test.py", ["/opt/enigmabridge/eb-keepalived/"..., "--host", "panda.enigmabridge.com"], [/* 6 vars */]) = -1 EACCES (Permission denied)

The culprit is really a SELinux - at least in our case. Namely the executable check script has to have a type keepalived_unconfined_script_exec_t. If you dump the current SELinux file context you get:

semanage fcontext -l | grep -i keepali
/var/run/keepalived.*                              regular file       system_u:object_r:keepalived_var_run_t:s0
/usr/libexec/keepalived(/.*)?                      all files          system_u:object_r:keepalived_unconfined_script_exec_t:s0
/usr/lib/systemd/system/keepalived.*               regular file       system_u:object_r:keepalived_unit_file_t:s0
/usr/sbin/keepalived                               regular file       system_u:object_r:keepalived_exec_t:s0

So you can either put the check script to the /usr/libexec/keepalived folder or add the tag to the check script. In our case the following worked:

semanage fcontext -a -t keepalived_unconfined_script_exec_t /opt/enigmabridge/eb-keepalived/eb-controller-test.py
restorecon -Rv /opt/enigmabridge/eb-keepalived/eb-controller-test.py
ls -lasZ /opt/enigmabridge/eb-keepalived/eb-controller-test.py
-rwxr-xr-x. keepalived keepalived unconfined_u:object_r:keepalived_unconfined_script_exec_t:s0 /opt/enigmabridge/eb-keepalived/eb-controller-test.py

Extended SELinux policy

If your script also needs to write somewhere (e.g., /tmp folder) this may get a bit more complicated. The recommended approach is to inspect the audit log and create a new policy from that. There is a audit2allow tool that helps with generating SElinux policies from the audit logs (the name of the package is really the full path):

yum install /usr/bin/audit2allow

Then just inspect logs and create a policy files:

grep keepalived_t /var/log/audit/audit.log | audit2allow -M keepalived_t
semodule -i keepalived_t.pp

Manual SELinux policy edit

You can also update policy file by hand keepalived_t.te:

module keepalived_t 1.0;
require {
	type tmp_t;
	type keepalived_t;
	class dir { write add_name };
	class file { write create open };
}

#============= keepalived_t ==============
allow keepalived_t tmp_t:dir { write add_name };
allow keepalived_t tmp_t:file { write create open };

And then rebuild the policy module and load it:

checkmodule -M -m -o keepalived_t.mod keepalived_t.te
semodule_package -o keepalived_t.pp -m keepalived_t.mod
semodule -i keepalived_t.pp

Testing

In order to test the configuration you can either stop the keepalived service or reboot the master server. Here I use the first method:

systemctl stop keepalived.service

Then server B log - switching to the master state and taking over the floating IP:

lis 22 14:51:07 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Transition to MASTER STATE
lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Entering MASTER STATE
lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) setting protocol VIPs.
lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1
lis 22 14:51:08 parrot Keepalived_healthcheckers[3307]: Netlink reflector reports IP 93.1.1.1 added
lis 22 14:51:13 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1

The interface en1 now has assigned IP 93.1.1.1.

When we re-start the keepalived service on the server A:

systemctl start keepalived.service

Server A will start keepalived and takes over the floating IP:

lis 22 14:52:29 panda Keepalived[25936]: Starting VRRP child process, pid=25938
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Netlink reflector reports IP 10.0.0.2 added
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Netlink reflector reports IP 93.1.1.2 added
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering Kernel netlink reflector
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering Kernel netlink command channel
lis 22 14:52:29 panda Keepalived_healthcheckers[25937]: Configuration is using : 7915 Bytes
lis 22 14:52:29 panda systemd[1]: Started LVS and VRRP High Availability Monitor.
lis 22 14:52:29 panda Keepalived_healthcheckers[25937]: Using LinkWatch kernel netlink reflector...
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering gratuitous ARP shared channel
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Opening file '/etc/keepalived/keepalived.conf'.
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Truncating auth_pass to 8 characters
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Configuration is using : 64580 Bytes
lis 22 14:52:29 panda Keepalived_vrrp[25938]: Using LinkWatch kernel netlink reflector...
lis 22 14:52:29 panda Keepalived_vrrp[25938]: VRRP sockpool: [ifindex(4), proto(112), unicast(1), fd(10,11)]
lis 22 14:52:30 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Transition to MASTER STATE
lis 22 14:52:30 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Entering MASTER STATE
lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) setting protocol VIPs.
lis 22 14:52:31 panda Keepalived_healthcheckers[25937]: Netlink reflector reports IP 93.1.1.1 added
lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1
lis 22 14:52:36 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1

Server B switching back to backup state:

lis 22 14:52:30 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Received higher prio advert
lis 22 14:52:30 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Entering BACKUP STATE
lis 22 14:52:30 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) removing protocol VIPs.
lis 22 14:52:30 parrot Keepalived_healthcheckers[3307]: Netlink reflector reports IP 93.1.1.1 removed