Extend MySQL Master HA (MHA) capabilities with MHA Helper

Ovais Tariq -

I have used many tools starting with MMM to be able to manage MySQL replication clusters. Some of the tools need more tools and complex HA solutions such as Pacemaker and Corosync, or Zookeeper. While other tools do not do the failover well which leaves the slaves in an inconsistent state, MMM would be an example.
And I must say that of all the tools I love MySQL Master HA (MHA) the most. MHA is a great tool to manage MySQL replication clusters for the purpose of HA. The most important thing about MHA is that it tries to take all the necessary steps to do a MySQL master failover in a way that provides as much data consistency as possible. The slave promotion also tends to be very quick, on average I have seen it take 10 to 15 seconds. It is also very easy to deploy unlike some of the other complex HA solutions.

I would highly recommend reading about the architecture of MHA on its wiki: https://code.google.com/p/mysql-master-ha/wiki/Architecture

Why MHA Helper?

MHA does one job and it does it well. It handles slave promotion in the best possible way. However, slave promotion in case of a master failure or in case of planned maintenance is only a step in the process. Typically a lot more is involved after a slave promotion has happened. For example, the application needs to be notified that the master has changed and that it needs to write to a new master. Then there may be some other operational tasks that need to be performed, such as notifying the monitoring service that the master has changed, or notifying the configuration management service so that it writes the correct configuration for a master. There can be a host of other operations that may need to be performed, I have just given a few examples that are most likely to be needed in majority of the cases.
That’s where MHA Helper comes in. MHA Helper provides a pluggable interface that extends MHA such that additional tasks may be performed in case of a MySQL failover.

Virtual IP management using MHA Helper

Currently MHA Helper provides virtual IP management for MySQL replication clusters. Virtual IP based HA solution is the most common implementation I have seen across a large number of MySQL users. Below is a simple illustration that shows a Virtual IP being used by the app to connect to MySQL.

Now when a failover happens the Virtual IP gets moved to the promoted master and the apps disconnect and reconnect using the same Virtual IP.

Now where does MHA Helper fit in here? MHA Helper acts as an external plugin that MHA invokes during various stages of the failover and MHA Helper then takes care of the necessary pre and post failover steps depending on the type of the failover.

  1. MySQL read_only flag handling. This also includes supporting the super_read_only flag that is available in Percona Server and other variants of MySQL
  2. Handling of Virtual IP failover
  3. Handling MySQL connections termination to make failover fast

Pluggable and Extensible Architecture of MHA Helper

MHA Helper has been designed to be pluggable and extensible. It is configured through ini-style configuration files, with one file per MySQL replication cluster, meaning that different replication clusters can have totally different configuration setup without effecting each other. A complete list of configuration options with examples can be seen on the Github page https://github.com/ovaistariq/mha-helper#configuration

MHA Helper currently only has the Virtual IP based failover implementation, however it supports adding different implementations, for example a different implementation for MySQL instances running in AWS, or other types of clouds. Because of the pluggable architecture of MHA Helper, it is easy to add more integrations to it. For example, currently Datadog and Chef integrations are in the works. The idea with Datadog integration is to send an event to Datadog whenever failover is performed as well as set appropriate tags on the new master to keep the metrics collection up to date.

If you have any more integration ideas, I would highly suggest that you file a bug https://github.com/ovaistariq/mha-helper/issues

Installation

The installation of MHA Helper is extremely easy. All you need to do is install the yum repository

curl -s https://packagecloud.io/install/repositories/twindb/main/script.rpm.sh | sudo bash

And install the packages

yum install mha4mysql-manager mha4mysql-node python-mha_helper

And that’s it!

For detailed instructions on configuration and installation visit the repository page on Github https://github.com/ovaistariq/mha-helper And don’t forget to file bugs or feature requests.