Skip to content

Master Slave

Jump to bottom
Yair Ogen edited this page Jul 2, 2017 · 33 revisions

Table of Contents

Intro

In a cloud environment runtime components are usually expected to be active active and support as many instances as needed. However, there are cases where there is a need to support a Master/Slave (A.K.A active/passive) model in which for a known given set of functionalities only one instance should be active.

This library aims to support this need in a cloud environment and multi Data Center environments.

The library was written with the following requirements in mind:

  • Support callbacks - The library and solution should enable callback to registered listeners and notify upon change of master or slave states (i.e. goMaster() and goSlave() methods). The callbacks will enable the host application to react based on their internal requirements to start and stop internal scheduling etc.
  • Don't use heartbeats - The library and solution should NOT use an internal heartbeat mechanism that involves direct communication between the application nodes. This is mainly to avoid split brain problems.
  • Identical Installation - The library and solution should support a mode of installation where all instances on all nodes are installed and configured the same way. I.e. the solution shouldn't rely on a "preferred master" parameter or similar. This solution doesn't restrict how many instances participate in the cluster, and will guarantee only one instance is the master.
  • Multi DC support - The library and solution should support multiple DCs where only one DC is assumed active and all instances in the other DCs will never become masters (until their host DC becomes Active).
  • Support Active/Inactive Mode - The library and solution should support 2 or more clusters where 1 cluster is active and the other is inactive until the second cluster transitions to active state. This use case materializes itself in an upgrade scenario where we have version 1.x running with let's say 3 nodes and one is master. We then install 3 new instances with version 2.x and we don't want any of them to become master until this version is considered to be "active". These 2 clusters are assumed to be in the same DC.

API

The public API is composed of the following main actors:

  • MasterSlaveListener - Applications implement this interface to react on master and slave state changes. It is expected of the application to start background work or application communication listeners (http, tcp etc.) when onMaster is invoked and to do the revers (stop work) when onSlave is invoked.
  • MasterSlaveRegistry - Applications add or remove their listeners using the registry singleton. This registry exposes a logical name that enables applications to register multiple unit of master/slave work within the same component. This has both the benefit of having different life-cycles to different units of work as well as enables different units to run in different instances of the component.
  • MongoClient - In general you setup the mongo data in configuration (see below), but specifically if hosting application is talking to a secured mongo instance then the application must call MongoClient.INSTANCE.enableAuthentication prior to any other call (MasterSlaveRegistry.INSTANCE.addMasterSlaveListener included).

Configuration

The library supports out of the box 2 implementation: Mongo and Consul.

By default consul is the implementation of choice.

You choose an implementation by specifying the masterSlave.impl key. Possible values are:

  • consul
  • consul-openstack (differs in active key and active DC keys only)
  • mongo
  • Fully Qualified class name of a class that implements the MastershipElector interface.

Consul Configuration

The library uses the OSS Http Client library. As such it uses a client with the api-name: "consulClient". So, any http client configuration is visible in the log file. Specifically, you change the host and port by adding/changing this in config.properties file:

consulClient.1.host=127.0.0.1
consulClient.1.port=8500

Setting Active Version

When the below mentioned singleAcrossVersion flag is set to true, the way to identify if you're the active version of not is dependent on 2 parameters.

  • The internal _ARTIFACT_VERSION environment variable (which is assumed any component already has).
  • A consul Key named '-version'.

If one of the above is missing - we assume we're active.

If they both exist and singleAcrossVersion = true, we are active only if both values of the above keys are equal.

Console (Openstack) Configuration

A preregister for each instance is to have the following ENV variables:

  • STACK_NAME
  • STACK_VERSION
  • DATA_CENTER

The above are compared with the following consul KV to determine whether I am in a current version and current DC:

  • activated/{STACK_NAME} = {STACK_VERSION}
  • primaryDatacenter

If the instance identifies it is "in the game" - i.e. is in the right SC and has the right version it then attempts to acquire a lock using mongo Session API. Consul verifies only one instance can acquire a look (which makes it a master) and as long as the lock is not released all other instances will be slaves.

A release is in the following cases:

  • An instance fails
  • An instance is stuck and can't renew the session (in which case if it gets back on track it will see it doesn't have the lock and will become master)

Mongo DB Configuration

Servers Configuration

The mongo server can be defined by 3 ways

Library Configuration

You specify an array of all the servers:

mongodb.1.host=mongo-1
mongodb.1.port=27017
mongodb.2.host=mongo-2
mongodb.2.port=27017

####Host Application Configuration Assuming your host application already has its own mongo configuration, they want to reuse its own configuration and not ontroduce another set of mongo server config as above.

For this you can call this API before you register your listener:

MasterSlaveConfigurationUtil.setMongodbServerConfigPrefix("my-server-prefix");

It is assumed that the above prefix support the same notion as the library config which is:

<custom-prefix>.1.host=mongo-1
<custom-prefix>.1.port=27017
<custom-prefix>.2.host=mongo-2
<custom-prefix>.2.port=27017

Set Mongo servers by code

The last option is to set the server diretly in code:

List<Pair<String,Integer>> servers = new ArrayList<>();
servers.add(Pair.of("localhost",27017));
MasterSlaveConfigurationUtil.setMongodbServers("my-server-prefix");

In this approach the library will not look for the server in configuration at all and use those supplied by code regardless of their origin.

DB Name

  • mongodb.db.name - Optional. Set a db name for the master slave cluster. Default is: 'cluster-db'.

General Configuration

Job Specific Configuration

  • <logical-name>.masterSlave.leaseTime - This parameter basically determins how often does a background thread, unique to the given job, will awake and check for mastership. Specify the lease time in seconds. This is the period in which the master-slave lib tries to own the lock implemented in the mongodb. If an instance a master and it fails to update the lease within this lease period another instance will take mastership. 'logical-name' correlates to the name you send to the registry when you register a new listener. Note: If you change this configuration parameter you must set it to the same value as all other instances of your component. default: 30 seconds

  • <logical-name>.masterSlave.mastership.multiplicity - an enumeration that can contain either 'multi' or 'single'. Multi means you can have multiple masters and not only one. default: single

  • <logical-name>.masterSlave.mastership.singleAcrossMDC - when set to false it means your master is not data-center aware and any running node can be master on any data-center. When set to true, the "am i in the active DC" logic is running and a master can be elected only within the "active data-center". default: true.

  • <logical-name>.masterSlave.mastership.singleAcrossVersion - when set to false it means your master is not version aware and any running node can be master regardless of its version. When set to true, the "am I the active version" logic is running and a master can only be elected if its current version is the active one. If there's no "active-version" info enabled then the mastership ignores versions. default: true.

Instance Identification

The library identifies a component id by reading from the following sources:

'_RPM_SOFTWARE_NAME' Environment Variable

'_ARTIFACT_NAME' Environment Variable

'_RPM_SOFTWARE_NAME' System property

Note: If all of the above are missing the application will not start. For development environments, please use the system property. In you Linux installations the '_RPM_SOFTWARE_NAME' Environment Variable will usually be set.

The Full Instance Id is constructed from the following:

<fully-qualified-domain-name>-<component-id>-<component-version>-<install-directory>

API Usage Examples

Register Listener

MasterSlaveRegistry.INSTANCE.addMasterSlaveListener("<logical-name>", new MasterSlaveListener() {
            @Override
            public void goMaster() {
                //start background processing, custom listeners etc.
            }

            @Override
            public void goSlave() {
                //stop background processing, custom listeners etc.
            }
        });

Remove Listener

This API effectively stops this instance in participating in the master/slave cluster.

MasterSlaveRegistry.INSTANCE.removeMasterSlaveListener("<logical-name>");

Enable Mongo Authentication

Only relevant if 'masterSlave.impl' is set to 'mongo'.

If relevant this call should be first before any other calls to api's in this lib

MongoClient.INSTANCE.enableAuthentication(dbUser, dbPassword);
Clone this wiki locally