Apache ActiveMQ 5.7, 5.9 and Master-Slave

With my ActiveMQ friends (especially Dejan and Claus), I’m working on ActiveMQ 5.9 next release.

Today, I focus on the HA with ActiveMQ, and especially Master-Slave configuration.

Update of the documentation

The first thing that I noticed is that the documentation is not really up to date.

If you do a search on the ActiveMQ website about Master-Slave, you will probably find these two links:

• http://activemq.apache.org/kahadb-master-slave.html
• http://activemq.apache.org/shared-file-system-master-slave.html

On the first link (about KahaDB), we can see a note “This is under review – and not currently supported”. It’s confusing for the users as this mechanism is the prefered one !
On the other hand, the second link should be flagged as deprecated as this mechanism is no more maintained.

I sent a message on the dev mailing list to updated these pages.

Lease Database Locker to avoid “dual masters”

In my test cases, I used a JDBC database backend (MySQL) for HA (instead of using KahaDB).

I have two brokers, that use the following configuration:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" />
  </persistenceAdapter>


Broker1 starts, connects to MySQL, and acquires the lock. Broker1 is the master.

Broker2 starts, connects to MySQL, and waits for the lock (as the lock is hold by Broker1). Broker2 is a slave.

Now, I stop MySQL, for instance to do a cold backup. My backup is very fast, and I start MySQL server again, very quickly.

The lock is available in the database, so Broker2 get the lock, whereas Broker1 didn’t yet release it. So I’m in a bad situation where I have two “masters”.

ActiveMQ 5.7.0 introduced the change on locking strategies for shared storage master/slave topologies. Previously storage locking (and thus master election) was hard-coded directly in the particular store. So KahaDB has only the option to use shared file lock, while JDBC was using database lock.

Now, the storage locking is separated from the store, so you can implement your own locking strategies if necessary (or tune existing ones). Of course, every store has its own default locker.

In our previsou use case, to solve the “dual master” issue, we can use a new locker: the lease database locker.

To use it, we update the configuration of each locker like this:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="5000">
      <locker>
        <lease-database-locker lockAcquireSleepInterval="10000"/>
      </locker>
    </jdbcPersistenceAdapter>
  </persistenceAdapter>


Lease database locker solves master/slave problem of the default database locker. Master acquires a lock only for a certain period and must extend it’s lease from time to time. Slave also checks periodically to see if the lease has expired. The lease can survive a db replica failover.

The lease based lock is acquired by blocking at start and retained by the keepAlivePeriod. To retain, the lease is extended by the lockAcquireSleepInterval, so in theory the master is always (lockAcquireSleepInterval-lockKeepAlivePeriod) ahead of the slave w.r.t the lease. It is imperative that lockAcquireSleepInterval > lockKeepAlivePeriod, to ensure the lease is always current.

In the simplest case, the clocks between master and slave must be in sync for this solution to work properly. If the clocks cannot be in sync, the locker can use the system time from the database CURRENT TIME and adjust the timeouts in accordance with their local variance from the db system time. If maxAllowableDiffFromDBTime is > 0 the local periods will be adjusted by any delta that exceeds maxAllowableDiffFromDBTime.

How to know who is the master ?

The “new” mechanism for Master/Slave is great and very easy to set up. You don’t really define who is the master, and who are the slaves. The first broker which get the lock will be the master.

So, a fair question is: how can I know which broker is the master ?

Actually, you already have the response on the JMX layer.

If you connect a JMX client (for instance jconsole) on the broker, and you take a look on the org.apache.activemq:BrokerName=Broker2,Type=Broker MBean, you can see the Slave attribute.

If Slave is true, it means that this broker is a slave. If Slave is false, it’s the master.

Another way to get this information is to use directly the activemq command with bstat argument (instead of JMX):


bin/activemq bstat
...
Connecting to pid: 563
BrokerVersion = 5.9-SNAPSHOT
TempLimit = 53687091200
Persistent = true
MemoryLimit = 67108864
TempPercentUsage = 0
SslURL =
StorePercentUsage = 0
TransportConnectors = {openwire=tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600}
Type = Broker
StompSslURL =
OpenWireURL = tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
Uptime = 3 minutes
DataDirectory = /home/jbonofre/broker2/data
StoreLimit = 107374182400
BrokerName = broker2
VMURL = vm://broker2
StompURL =
MemoryPercentUsage = 0
Slave = true


You can see the Slave attribute there.

If you want to “script” this and get only the Slave attribute, you can use the query argument:


bin/activemq query --objname Type=Broker --view Slave
...
Slave = true