2009/06/23

Heartbeat and two-nodes clusters

Heartbeat (from http://www.linux-ha.org/heartbeat/) is a nice software to manage linux high availability clusters. When you are in need to use a two-nodes cluster you want to prevent a situation known as "split-brain": when a node does not communicate with the other but neither two know which is the failing node.

This can lead to having the cluster drop all the services or having both nodes with the services on (in case of shared storage, this will lead to corruption unless the filesystem is clustered and supports multiple writes from multiple nodes). To make a long story short, split-brain is bad.

To counter this effect, you can use a ping host. Ping hosts are fake cluster members used by heartbeat to use the network connectivity as a mean to break ties in cluster node weight computations: if a node pings a ping host and the other doesn't, the hosts which pings is the winner. To make this configuration successful, you need an additional link between the nodes. This additional link could be serial or another ethernet.

To use ping hosts, this is the bare minimum configuration:

ping 10.10.10.10 # insert the ip of the ping host.
respawn hacluster /usr/lib64/heartbeat/ipfail # use /usr/lib on 32 bit hosts
apiauth ipfail gid=haclient uid=hacluster # this is for process security


But which ping host to chose? Use a reliable ping host; this can be the default gateway (if it's a router which has a very high uptime) or a local layer 3 switch. Don't use a server o a workstation!

Use hearbeat reload to configure the ping host without stopping the cluster and lose resources.

No comments:

Post a Comment