Cluster Manager (DRCM) guide | DataReel Software Development

New for 1.46: the DRCM Administrator Guide (plain text) covers node standby and timed standby, split brain protection, file replication over the internet, the TCP + TLS transport, the security model, witness quorum, AWS floating IPs and the setup tools.

Contents

Overview
Using this Document
Prerequisites
RPM Install
Building and Installing From Source Code
DRCM Auth Tokens
Building CM Configuration Files
Testing CM Configuration Files
CM Node Hostname and IP Settings
CM Global Configuration Settings
Setting Up Global Cluster Resources
Linking Global Cluster Resources to Nodes
Floating IP Address Interface Configuration
Running the DRCM Server
DRCM Resource Scripts
Running Applications
Cluster Monitor and Reports
2-way file replication
2-way file replication over the internet or a tunnel
Setting up cluster alerts
Remaining Work on This Project
Support and Bug Tracking

Overview

The DataReel cluster manager (DRCM) software is used to build high availability Linux server clusters. High availability clusters manage system resources with automatic fail over in event of one or more server failures.

The DRCM project was designed to support multiple cluster nodes in a non-hierarchical arrangement. Cluster resources are distributed between the nodes with backup rolls assigned. The DRCM project requirements were based on the need to replace existing cluster software with multiple dependency issues and cluster software that did not scale well more than 2 cluster nodes.

A Linux server cluster is typically 2 servers with equal capacity, where one server provides primary services and the other server provides backup services. In the event of a primary server failure the secondary server automatically assumes all cluster resources. In data centers utilizing virtualization technology, services are distributed between many virtual machines (VMs). In a VM environment clustering multiple VM nodes running on multiple bare metal servers is a very effective way to manage services in the event of a bare metal server failure.

The DRCM multiple node concept can scale between nodes in one or more data centers in the same or different geographical locations.

Using this Document

All commands prefixed with the $ prompt indicate the command is executed from a user account or from a service account.
All commands prefixed with the # prompt indicate the command is executed as root.
In a configuration file example a ... symbol indicates omitted content for brevity.

Prerequisites

DRCM resources consist of crontabs, floating IP addresses, services, applications, and file systems. When you plan your cluster setup, consider the following:

Number of nodes per cluster
Assigning crons based on CPU, memory, disk space and bandwidth requirements
Floating IP address requirements, where a service is accessed via one IP address
System services: http, mysql, postgresql, etc. Which node(s) needs to run services
File systems: NFS, cifs, etc. Which node(s) require file system mounts
Applications: Programs not part of the OS with user programs and scripting
SSH keys: keyed authentication for file synchronization and automation

RPM Install

Prebuilt RPMs for RHEL 8, 9 and 10 are on the DRCM downloads page:

$ sudo dnf -y install ./drcm_server-<version>-1.el9.x86_64.x86_64.rpm

System administrators, follow the instructions below to build an RPM for distribution:

$ cd $HOME; mkdir -pv git; cd $HOME/git
$ wget https://datareel.com/downloads/drcm/ha-cluster-manager-<version>.tar.gz
$ tar xzf ha-cluster-manager-<version>.tar.gz
$ mv ha-cluster-manager-<version> ha-cluster-manager
$ cd ${HOME}/git/ha-cluster-manager/rpm_builder

The build needs gcc-c++, make, rpm-build and openssl-devel.

RHEL 10:

$ ./make_rhel10_rpm.sh
$ sudo dnf -y install ${HOME}/rpmbuild/RPMS/x86_64/drcm_server-<version>-1.el10.x86_64.x86_64.rpm

RHEL 9:

$ ./make_rhel9_rpm.sh
$ sudo dnf -y install ${HOME}/rpmbuild/RPMS/x86_64/drcm_server-<version>-1.el9.x86_64.x86_64.rpm

RHEL 8:

$ ./make_rhel8_rpm.sh
$ sudo dnf -y install ${HOME}/rpmbuild/RPMS/x86_64/drcm_server-<version>-1.el8.x86_64.x86_64.rpm

RHEL 6 and 7 are no longer supported; 1.37 was the last release for them.

When upgrading a cluster to 1.42 from 1.41 or older, upgrade all nodes together. 1.42 nodes do not talk to older nodes; see CHANGELOG.md.

Useful RPM command to verify package contents:

List package: rpm -qf /usr/sbin/drcm_server
List files: rpm -ql drcm_server
List configs: rpm -qc drcm_server
List docs: rpm -qd drcm_server

To remove package:

$ sudo su root -c 'yum -y remove drcm_server'

Building and Installing From Source Code

Developers and system admins, follow the instructions below to build and install the DRCM server from the latest source code distro:

$ ssh username@cmnode1
$ mkdir -pv ~/git
$ cd ~/git
$ wget https://datareel.com/downloads/drcm/ha-cluster-manager-<version>.tar.gz
$ tar xzf ha-cluster-manager-<version>.tar.gz
$ mv ha-cluster-manager-<version> ha-cluster-manager
$ cd ~/git/ha-cluster-manager/env
$ source gnu_env.sh
$ cd ~/git/ha-cluster-manager/drcm_server
$ make
$ sudo su root -c 'make install_root'

The default configuration directory is: /etc/drcm

NOTE: All nodes in the same cluster must have identical "/etc/drcm" directories. It's recommended to design your initial configuration on a node you designate as a cluster entry point. Then push all "/etc/drcm" updates to all the nodes in the cluster.

DRCM Auth Tokens

All nodes in a cluster must have the same auth tokens. To generate auth tokens for the first time or to replace existing tokens:

# mkdir -pv /etc/drcm/.auth
# chmod 700 /etc/drcm/.auth
# dd if=/dev/urandom bs=1024 count=2 > /etc/drcm/.auth/authkey
# sha1sum /etc/drcm/.auth/authkey | cut -b1-40 > /etc/drcm/.auth/authkey.sha1
# chmod 600 /etc/drcm/.auth/authkey /etc/drcm/.auth/authkey.sha1

Building CM Configuration Files

The default CM configuration is: /etc/drcm/cm.cfg
To build a CM configuration template run the following command:

# /usr/sbin/drcm_server --check-config

To build a CM configuration template in another location:

# /usr/sbin/drcm_server --config-file=/tmp/test1.cfg --check-config

Testing CM Configuration Files

After editing the CM configuration file you must check your CM configuration files for errors before starting the DRCM server:

# /usr/sbin/drcm_server --check-config | grep -i -E 'error|warn|fail|info'

If you have any errors or warnings, review and edit your configuration file:

# /usr/sbin/drcm_server --check-config

CM Node Hostname and IP Settings

After generating the cluster configuration file you must edit the file and set up your global cluster configuration and node configuration. For nodes you include in a cluster you must have the following information for each node:

Fully qualified hostname matching: uname -n
IP address used for cluster keep alive and CM message handling.

# vi /etc/drcm/cm.cfg

[CM_NODE]
nodename = cmnode1  # This can be any name you will recognize this node as
hostname = cmnode1.example.com # uname -n
keep_alive_ip = 192.168.122.183   
...

[CM_NODE]
nodename = cmnode2
hostname = cmnode2.example.com
keep_alive_ip = 192.168.122.184
...

CM Global Configuration Settings

The CM_SERVER configuration setting sets values used by the DRCM service and all cluster nodes. Before starting the DRCM service, set the CM port number and host-based firewall rules:

# Global cluser configuration
[CM_SERVER]
udpport = 53897
tcpport = 53897
bind_to_keep_alive_ip = 1
...

By default the DRCM server will only listen for cluster messages on each node's keep alive IP address. To listen on all interfaces change the bind_to_keep_alive_ip setting to 0.

Each cluster node must have host-based firewall rules allowing the DRCM server access to the TCP and UDP ports set in CM global configuration.

Use firewalld for the host-based firewall rules. Direct iptables commands are legacy and are no longer documented here; on RHEL 8 and later firewalld manages the nftables rules for you.

Review your current zones:

# firewall-cmd --get-active-zones
# firewall-cmd --list-all

The commands below apply to the default zone. If the cluster interface is in a different zone, add --zone=<zone> to each command.

Allow the DRCM UDP and TCP ports from each of the other cluster nodes:

# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.183/32" port port="53897" protocol="udp" accept'
# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.184/32" port port="53897" protocol="udp" accept'
# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.183/32" port port="53897" protocol="tcp" accept'
# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.184/32" port port="53897" protocol="tcp" accept'
# firewall-cmd --reload

The cluster nodes must also be able to ping and SSH to each other. firewalld allows ping by default. If ping has been blocked, check for an echo-request ICMP block and remove it:

# firewall-cmd --list-icmp-blocks
# firewall-cmd --permanent --remove-icmp-block=echo-request
# firewall-cmd --reload

To allow SSH only from the cluster node IPs:

# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.183/32" service name="ssh" accept'
# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.184/32" service name="ssh" accept'
# firewall-cmd --reload

Verify the rules:

# firewall-cmd --list-rich-rules

Setting Up Global Cluster Resources

The latest cluster resource scripts are in the etc/resources folder of the source code release:

https://datareel.com/downloads/drcm/

All nodes in a cluster has the ability to run a global cluster resource as a primary function or as a backup function. DRCM has 5 global resource categories:

CM_CRONTABS: crons to run on a node with fail over to secondary nodes CM_IPADDRS: floating IP addresses to run on a node with fail over to secondary nodes CM_SERVICES: System services to run on all nodes or single nodes CM_APPLICATIONS: Applications to run on all nodes or single nodes CM_FILESYSTEMS: File system to mount on all nodes or single nodes

Under each resource section you must provide a CSV list for each entry. All resources subsections start with a nickname. A nickname can be any unique name you will recognize the resource as. Nicknames are used by the cluster nodes to identify the global resource.

[CM_CRONTABS]
# CSV format: nickname, template_file, install_location, resource_script
system, /etc/drcm/crontabs/system_crons, /etc/cron.d, /etc/drcm/resources/crontab.sh
user, /etc/drcm/crontabs/user_crons, /etc/cron.d, /etc/drcm/resources/crontab.sh
apps1, /etc/drcm/crontabs/apps_package1, /etc/cron.d, /etc/drcm/resources/crontab.sh
apps2, /etc/drcm/crontabs/apps_package2, /etc/cron.d, /etc/drcm/resources/crontab.sh

[CM_IPADDRS]
# CSV format: nickname, Floating IP, netmask, Ethernet interface, ip takeover script
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_nmcli.sh
# IPv6 floating IPs use ipv6addr_nmcli.sh and a prefix length, see below
##cmnodef6, 2001:db8:122::180, 64, ens192, /etc/drcm/resources/ipv6addr_nmcli.sh

[CM_SERVICES]
# CSV format: nickname, service_name, resource_script
web, httpd, /etc/drcm/resources/service.sh

[CM_APPLICATIONS]
# CSV format: nickname, user:group, start_program, stop_program
# CSV format: nickname, user:group, start_program, stop_program, ensure_script
ldm, ldm:users, ~/util/ldm_ensure.sh, ldmadmin stop, ~/util/ldm_ensure.sh

[CM_FILESYSTEMS]
# CSV format: nickname, source, target, resource_script
data, 192.168.122.1:/data, /data, /etc/drcm/resources/nfs.sh
archive, 192.168.122.1:/archive, /archive, /etc/drcm/resources/nfs.sh
webshare, /192.168.122.225/users/web, /home/users/web, /etc/drcm/resources/cifs.sh

Linking Global Cluster Resources to Nodes

The global cluster resource for a node is set in each CM_NODE section. Below is a 2 node cluster example of multiple crontabs with primary and secondary rolls. A single floating IP address for cluster users to access this cluster. In this example system services, file systems, and applications will be ran on both nodes.

[CM_NODE]
nodename = cmnode1
...
node_crontabs = user,system,apps1
node_backup_crontabs = cmnode2:apps2
node_floating_ip_addrs = cmnodef
node_services = web
node_filesystems = data,arhive,webshare
node_applications = ldm

[CM_NODE]
nodename = cmnode2
...
node_crontabs = apps2
node_backup_crontabs = cmnode1:user,cmnode1:system
node_backup_floating_ip_addrs = cmnode1:cmnodef
node_services = web
node_filesystems = data,arhive,webshare
node_applications = ldm

Floating IP Address Interface Configuration

Floating IP Address Interface Configuration (NetworkManager)

Floating IP addresses are managed through NetworkManager (nmcli). This is the supported method on RHEL 8, RHEL 9, RHEL 10 and later. RHEL 9 and later no longer ship the legacy network-scripts, RHEL 8 systems should all be converted to NetworkManager, and the ifcfg sub-interface method (eth0:1) is kept for legacy systems only, see "Legacy Floating IP Address Configuration (network-scripts)" below.

The netmask can be given as a CIDR prefix (24) or a dotted netmask (255.255.255.0); the nmcli resource scripts convert it for nmcli.

Two NetworkManager resource scripts are provided for IPv4:

/etc/drcm/resources/ipv4addr_nmcli.sh
/etc/drcm/resources/ipv4addr_copy_nmcli.sh

For IPv6 floating IPs see "IPv6 Floating IP Addresses" below.

The ipv4addr_nmcli.sh resource script is the default, for simple configurations. It adds the floating IP as an extra address on the existing connection, so the floating IP does not need to be the primary address. No ifcfg sub-interface or template file is needed.

The ipv4addr_copy_nmcli.sh resource script is for complex configurations where you manually edit the network manager config file creating a start and stop interface file.

Both resource scripts assume the NMCLI connection name and the device name are the same. Network manager connection commands update the connection profile on disk but do not apply the configuration to the interface. The network manager device commands read the connection profile from disk and apply the change to the interface. The NMCLI resource script uses both connection and device commands which is why the connection name and the device name must be the same.

To use the ipv4addr_nmcli.sh for simple configurations setup your cluster config to use the IPv4 nmcli config resource file:

# vi /etc/drcm/cm.cfg

[CM_IPADDRS]
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_nmcli.sh

NOTE: If you are clustering VM and bare metal server use the following syntax:

[CM_IPADDRS]
cmnodef, 192.168.122.180, 255.255.255.0, 'ens192|eno8303', /etc/drcm/resources/ipv4addr_nmcli.sh

On the primary cluster node copy the /etc/drcm directory to all the backup nodes:

# rsync -av --force --stats --progress -n /etc/drcm 192.168.122.184:/etc/.

Verify the rsync output from the dry run looks correct and re-run without the dry run option:

# rsync -av --force --stats --progress /etc/drcm 192.168.122.184:/etc/.

To use the ipv4addr_copy_nmcli.sh for complex configurations you must set up template files on all your cluster nodes. For example on a VMWARE VM with 1 interface:

# export ETHINT=ens192
# mkdir -p /root/backups
# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /root/backups/${ETHINT}.nmconnection

On the primary node create a start and stop template:

# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /etc/drcm/my_cluster_conf/$(hostname -s)_start_backup_ip_${ETHINT}

# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /etc/drcm/my_cluster_conf/$(hostname -s)_stop_backup_ip_${ETHINT}

The stop template is for cluster node when it is not the primary holding the floating IP so it does need to be edited.

The start template must be edited to add the floating IP as the primary interface:

# vi /etc/drcm/my_cluster_conf/$(hostname -s)_start_backup_ip_${ETHINT}

[connection]
id=ens192
uuid=(This will be your devices UUID for this host)
type=ethernet
autoconnect-priority=-999
interface-name=ens192
timestamp=1707173943

[ethernet]

[ipv4]
address1=192.168.122.180/24,192.168.122.1
address1=192.168.122.182/24
dns=192.168.122.1;
method=manual

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]

Save changes exit VI.

Log into the secondary node and create the start and stop interfaces. To create the interface templates on the backup node run the following commands as root:

# export ETHINT=ens192
# mkdir -p /root/backups
# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /root/backups/${ETHINT}.nmconnection
# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /etc/drcm/my_cluster_conf/$(hostname -s)_start_backup_ip_${ETHINT}
# cp -av /etc/NetworkManager/system-connections/${ETHINT}.nmconnection /etc/drcm/my_cluster_conf/$(hostname -s)_stop_backup_ip_${ETHINT}

# vi /etc/drcm/my_cluster_conf/$(hostname -s)_start_backup_ip_${ETHINT}

[connection]
id=ens192
uuid=(This will be your devices UUID for this host)
type=ethernet
autoconnect-priority=-999
interface-name=ens192
timestamp=1707173943

[ethernet]

[ipv4]
address1=192.168.122.180/24,192.168.122.1
address1=192.168.122.183/24
dns=192.168.122.1;
method=manual

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]

Save changes exit VI.

Log out the backup node and log back into the primary node. On the primary cluster node SFTP to the backup nodes and copy the template files you created on the backup nodes:

# cd /etc/drcm/my_cluster_conf
# sftp 192.168.122.184
sftp> cd /etc/drcm/my_cluster_conf
sftp> mget *_backup_ip_*
sftp> exit

Setup your cluster config to use the IPv4 copy config resource file:

# vi /etc/drcm/cm.cfg

[CM_IPADDRS]
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_copy_nmcli.sh

NOTE: If you are clustering VM and bare metal server use the following syntax:

[CM_IPADDRS]
cmnodef, 192.168.122.180, 255.255.255.0, 'ens192|eno8303', /etc/drcm/resources/ipv4addr_copy_nmcli.sh

On the primary cluster node copy the /etc/drcm directory to all the backup nodes:

# rsync -av --force --stats --progress -n /etc/drcm 192.168.122.184:/etc/.

Verify the rsync output from the dry run looks correct and re-run without the dry run option:

# rsync -av --force --stats --progress /etc/drcm 192.168.122.184:/etc/.

On each cluster node test the floating IP:

# /etc/drcm/resources/ipv4addr_copy_nmcli.sh 192.168.122.180 255.255.255.0 ens192 start
# ip a

You should see the floating IP as the primary interface and should be able SSH to the floating IP from another node. Once verified stop the sub-interface:

# /etc/drcm/resources/ipv4addr_copy_nmcli.sh 192.168.122.180 255.255.255.0 ens192 stop

Repeat this test on all the cluster nodes.

IPv6 Floating IP Addresses (NetworkManager)

Floating IP addresses can be IPv6 as well as IPv4. The cluster nodes still talk to each other over IPv4 (keep alive IPs, keep alive messages and cluster messages), so every node needs an IPv4 address for the cluster; only the floating IPs that users connect to can be IPv6. A cluster can have both IPv4 and IPv6 floating IPs.

Use the ipv6addr_nmcli.sh resource script and give a prefix length (for example 64) in the netmask column:

# vi /etc/drcm/cm.cfg

[CM_IPADDRS]
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_nmcli.sh
cmnodef6, 2001:db8:122::180, 64, ens192, /etc/drcm/resources/ipv6addr_nmcli.sh

And add the nickname to the nodes the same way as an IPv4 address:

node_floating_ip_addrs = cmnodef,cmnodef6
node_backup_floating_ip_addrs = cmnode1:cmnodef,cmnode1:cmnodef6

Like ipv4addr_nmcli.sh, ipv6addr_nmcli.sh needs the NetworkManager connection name to be the same as the device name, and it accepts a | separated list such as 'ens192|eno8303'. IPv6 must be enabled on the connection; check it on each node with:

# nmcli -g ipv6.method connection show ens192

If it prints "disabled" or "ignore", enable it (auto keeps the router assigned addresses and adds the floating IP next to them):

# nmcli con mod ens192 ipv6.method auto
# nmcli device reapply ens192

On each cluster node test the IPv6 floating IP:

# /etc/drcm/resources/ipv6addr_nmcli.sh 2001:db8:122::180 64 ens192 start
# ip -6 addr show dev ens192
# /etc/drcm/resources/ipv6addr_nmcli.sh 2001:db8:122::180 64 ens192 stop

For complex configurations, ipv4addr_copy_nmcli.sh also works for IPv6: put the floating IPv6 address in the [ipv6] section of the start template (for example address1=2001:db8:122::180/64) and list it with its IPv6 address in CM_IPADDRS.

Clients reaching IPv6 floating IPs need the service ports open for IPv6 in firewalld. Service and port rules without a family cover both IPv4 and IPv6, for example:

# firewall-cmd --permanent --add-service=http
# firewall-cmd --reload

Rich rules with a source address are per family; use family="ipv6" with an IPv6 source:

# firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source address="2001:db8:122::/64" service name="http" accept'
# firewall-cmd --reload

Legacy Floating IP Address Configuration (network-scripts)

LEGACY ONLY: The methods in this section use the network-scripts package (ifcfg files, ifup and "systemctl restart network"). Use them only on older systems that do not run NetworkManager (RHEL 6 and RHEL 7). RHEL 8 and later systems must use the NetworkManager method above; the RHEL 8 network-scripts method is kept only as a reference for converting old clusters. The legacy resource scripts are ipv4addr.sh, ipv4addr_multi.sh and ipv4addr_copy_config.sh, and the ifcfg-eth0_N templates in /etc/drcm/system_templates.

Legacy: RHEL 6 and RHEL 7 sub-interfaces

If your cluster has one or more floating IP addresses you must configure a sub-interface for each floating IP. A floating IP address is set in the global resource configuration CM_IPADDRS section, for example:

[CM_IPADDRS]
cmnodef, 192.168.122.180, 255.255.255.0, eth0:1, /etc/drcm/resources/ipv4addr.sh

In the example above, we need to setup a sub-interface for eth0:

# cd /etc/sysconfig/network-scripts
# vi ifcfg-eth0:1

DEVICE=eth0:1
BOOTPROTO=none
ONBOOT=yes

Save changes and exit VI.

IPADDR and PREFIX or NETMASK will be set by the DRCM server when the node
resource is activated.

# ifup eth0:1

Legacy: RHEL 8 with network-scripts (reference only)

NOTE: All RHEL 8 systems should now use the NetworkManager method above. For RHEL 8 the following method has been tested to fix the sub-interface issue with the legacy network-scripts and network manager.

On all cluster nodes install the network-scripts:

# yum -y install network-scripts
# systemctl enable network
# reboot

On all your cluster nodes setup all the ifcfg-name file using sub-interfaces as follows, for example on a VMWARE VM with 1 interface:

# export ETHINT=ens192
# mkdir -p /root/backups
# cp -av /etc/sysconfig/network-scripts/ifcfg-${ETHINT} /root/backups/ifcfg-${ETHINT}
# vi /etc/sysconfig/network-scripts/ifcfg-${ETHINT}

NAME="System ens192"
TYPE=Ethernet
DEVICE=ens192
## UUID= (This will be your devices UUID for this host)
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV4_FAILURE_FATAL=no
DEFROUTE=yes
DNS1=192.168.122.1
DOMAIN=example.com

#
# Use the subinterace if this is the primary server
##IPADDR0=192.168.122.180
PREFIX0=24
GATEWAY0=192.168.122.1
#
IPADDR1=192.168.122.183
PREFIX1=24
GATEWAY1=192.168.122.1

Save changes and test the config:

# systemctl restart network

Once all your cluster node have the the interface files with the sub-interface setup run the following DRCM resource script:

# export ETHINT=ens192
# cp /etc/sysconfig/network-scripts/ifcfg-${ETHINT} /etc/drcm/my_cluster_conf/$(hostname -s)_start_backup_ip_${ETHINT}
# cp /etc/sysconfig/network-scripts/ifcfg-${ETHINT} /etc/drcm/my_cluster_conf/$(hostname -s)_stop_backup_ip_${ETHINT}

On the primary cluster node SFTP to the backup nodes and copy the template files you created on the backup nodes:

# cd /etc/drcm/my_cluster_conf
# sftp 192.168.122.184
sftp> cd /etc/drcm/my_cluster_conf
sftp> mget *_backup_ip_*
sftp> exit

Edit all the *_start_backup_ip_* templates change uncomment the sub-interface IP:

#
# Use the subinterace if this is the primary server
IPADDR0=192.168.122.180
PREFIX0=24
GATEWAY0=192.168.122.1

So all the start templates have the sub-interface IP enable and all the stop templates have the sub-interface commented out.

Setup your cluster config to use the IPv4 copy config resource file:

# vi /etc/drcm/cm.cfg

[CM_IPADDRS]
cmnodef, 192.168.122.180, 255.255.255.0, ens192, /etc/drcm/resources/ipv4addr_copy_config.sh

NOTE: If you are clustering VM and bare metal server use the following syntax:

[CM_IPADDRS]
cmnodef, 192.168.122.180, 255.255.255.0, 'ens192|eno8303', /etc/drcm/resources/ipv4addr_copy_config.sh

On the primary cluster node copy the /etc/drcm directory to all the backup nodes:

# rsync -av --force --stats --progress -n /etc/drcm 192.168.122.184:/etc/.

Verify the rsync output from the dry run looks correct and re-run with out the dry run option:

# rsync -av --force --stats --progress /etc/drcm 192.168.122.184:/etc/.

On each cluster node test the floating IP:

# /etc/drcm/resources/ipv4addr_copy_config.sh 192.168.122.180 255.255.255.0 ens192 start
# ip a

You should see the floating IP as the primary interface and should be able SSH to the floating IP from another node. Once verified stop the sub-interface:

# /etc/drcm/resources/ipv4addr_copy_config.sh 192.168.122.180 255.255.255.0 ens192 stop

Repeat this test on all the cluster nodes.

Running the DRCM Server

After setting up your cluster configuration and mirroring the "/etc/drcm" directory to all nodes in the cluster, start the DRCM server on each node.

RHEL 7 and later (systemd):

# systemctl start drcm_server
# systemctl status drcm_server

Legacy RHEL 6/CENTOS 6:

# service drcm_server start
# service drcm_server status

View the cluster status monitor:

# drcm_server --client --command=cm_stat

View the cluster logs:

# tail -f /var/drcm/drcm.log

To test resource fail over:

# systemctl stop drcm_server
# drcm_server --client --command=cm_stat

To test resource fail back:

# systemctl start drcm_server
# drcm_server --client --command=cm_stat

To move a node's resources to its backup nodes while DRCM keeps running, for maintenance or application failover tests (1.43 and later), run on that node:

# drcm_server --client --command=standby
# drcm_server --client --command=cm_stat
# drcm_server --client --command=unstandby

Add --time=2h (or 55m, 90s, 1h30m, up to 7d) to standby and the node returns to service by itself when the time is up (1.46 and later).

standby refuses if no peer can take over (--force overrides). The node stays in standby across restarts until unstandby, and unstandby starts each resource only after the peer has stopped its copy. See "Standby" in section 11 of the administrator guide.

After testing make the DRCM service persistent:

RHEL 7 and later (systemd):

# systemctl enable drcm_server

Legacy RHEL 6/CENTOS 6:

# chkconfig drcm_server on

DRCM Resource Scripts

The default DRCM resource script location is: /etc/drcm/resources

The default installation includes system resource script to manage crontabs, floating IP addresses, system services, NFS and CIFS files systems. The NFS and CIFS file system resource script will check for stale mounts and automatically remount if a mount is stale.

Which resource scripts used is set in the global resource configuration. Cluster administrators can use custom resource scripts, provided custom scripts accept the same input arguments as the resource scripts included in the distribution.

Running Applications

Applications refer to programs that are not part of the operating system (OS) or integrated into the OS framework. Managing user applications in a cluster can be difficult. The DRCM server does not manage user applications from resource scripts. This allows flexibility to run applications from a user or service account and use programs and scripting unique to each application. Applications are defined in the global CM_APPLICATIONS section:

    
[CM_APPLICATIONS]

There are 2 formats to define a user application:

# CSV format: nickname, user:group, start_program, stop_program
# CSV format: nickname, user:group, start_program, stop_program, ensure_script

The user and group must be set to the username that will run the application and the group associated with application.

The start_program is the program used to start the application, with or without input arguments. This can be multiple entries separated by semicolons ;

The stop_program is the program used to start the application, with or without input arguments. This can be multiple entries separated by semicolons ;

The ensure_script is a script or program that checks to ensure the application is running. If an ensure script is provided the DRCM server will use the "ensure_script" to watch the application.

NOTE: The user environment will be sourced so you do not have to use absolute paths to programs if the program path is defined in the user environment.

Some examples:

ldm, ldm:users, ldmadmin start, ldmadmin stop

ldm, ldm:users, ldmadmin clean; ldmadmin mkqueue; ldmadmin start, ldmadmin stop; ldmadmin delqueue

ldm, ldm:users, ldmadmin start, ldmadmin stop, if [ -f ~/util/ldm_ensure.sh ]; then ~/util/ldm_ensure.sh; fi

Cluster Monitor and Reports

DRCM provides a formatted reporting tool and raw stats you can use to build custom reports. To view the detailed cluster report, from any cluster node:

/usr/sbin/drcm_server --client --command=cm_stat

To create custom reports use the rstats command from any cluster node, for example:

for CMNODE in cmnode1 cmnode2 cmnode3 cmnode4
> do
> echo $CMNODE
> /usr/sbin/drcm_server --client --command=rstats | sed -n "/$CMNODE/,/$CMNODE/p" | grep -v $CMNODE
> echo ""
> done

2-way file replication

The DRCM distribution includes several utilities to synchronize file systems between a primary and secondary cluster node. This is typically used when a primary node is acting as a NFS server for other servers and/or workstations. Or for primary nodes replicating database directories to a cold spare.

NOTE: File replication requires public SSH key authentication for the backup user account that will run the replication scripts from the user's crontab. NOTE: If you are using "root" to run system backups you can limit access to key SSH authentication to run specific commands. Typically the back user is a special user account used just for running file system backups.

NOTE: Your backup user must have read/write access to the following directory:

/etc/drcm/my_cluster_conf

Following the DRCM installation, the file system utilities are located in the following directory:

/etc/drcm/file_system_sync

To configure a designated primary node and a designated backup node you must set you node configuration in the following file:

/etc/drcm/my_cluster_conf/my_cluster_info.sh

In the "my_cluster_info.sh" file set the following variables:

# Set a simple cluster name, any name you will recognize
# this cluster as, no spaces.
export CLUSTER_NAME="NFS-CLUSTER1"

# Set IP or hostname of online storage server for shared backups
# This is only need if you plan to backup the NFS or database
# directories to a backup storage severs.
export BACKUP_STORAGE_SERVER="192.168.122.1"

# Set the destination directory on the storage server
export BACKUP_STORAGE_LOCATION="/backups"

# Set the user name that runs you backups.
# NOTE: This user name must have keyed SSH authentication access
# NOTE: to the cluster nodes and backup storage server
export BACKUP_USER="root"

#
# Set the following for 2-way file system replication
#
# IP address and Ethernet Interface of the cluster node with primary
# file systems. File systems on this node will be replicated to the
# backup file system in the event of a fail over to the backup node.
export PRIMARY_FILESYSTEMS_IP="192.168.122.111"
export PRIMARY_FILESYSTEMS_ETH="ens192"
#
# IP address and Ethernet Interface of the backup for this cluster node,
# where the primary file systems are mirrored too. When the primary
# node is down this node will host the file systems.
export BACKUP_FILESYSTEMS_IP="192.168.122.112"
export BACKUP_FILESYSTEMS_ETH="ens192"

# Allow RSYNC on a diffent IP/Interface if high bandwidth required to sync file systems
# These settings are used to move high bandwidth consumption to another Ethernet interface
# and/or network.
## export PRIMARY_FILESYSTEMS_RSYNC_IP=""
## export BACKUP_FILESYSTEMS_RSYNC_IP=""

Once you setup your configuration, edit the following file to run a test between the primary and backup node. As the backup user on the primary node:

$ vi /etc/drcm/my_cluster_conf/testfs_sync_list.sh

RsyncMirror /testfs

NOTE: Mirroring replicates an exact copy of the /testfs file system. This means files will be copied and deleted as new files are added or removed. If you want to keep copies of files removed on the backup system use the following line

RsyncCopy /testfs

NOTE: Prior to testing you must create the /testfs directory on both cluster nodes and make sure your backup user has read/write access to the /testfs directory.

Make sure to copy all changes in the /etc/drcm directory to the backup node before you do any testing. To test, as the backup user on the primary node, execute the following script:

$ /etc/drcm/file_system_sync/sync_testfs.sh

This will replicate the contents of the /testfs directory to the backup node. As the backup user on the backup node, execute the same script:

$ /etc/drcm/file_system_sync/sync_testfs.sh

You should see the following output on the backup node:

Checking to see if failback sync flag exists We have no sync flag for testfs files No file sync to primary cluster node will be performed

If the primary node is down the backup node will update the /testfs directory until the primary node is back online. Once the primary node is back online, all files on the backup will be replicated back to the primary node. Once the back node has finished replication, the primary node replicates to the back node. To test a primary to backup fail over and fail back, execute the following script on the primary node, as root or an admin account with sudo rights:

# /etc/drcm/utils/manual_fs_failover.sh

To fail back, start the drcm_server on the primary node.

After verifying 2-way file replication is work between the primary and backup node, add the following line to the backup users crontab. On the primary node as the backup user:

$ crontab -e

*/5 * * * *  /etc/drcm/file_system_sync/sync_testfs.sh > /dev/null 2>&1

On the backup node:

$ crontab -e

* * * * * /etc/drcm/file_system_sync/sync_testfs.sh > /dev/null 2>&1

This will replicate /testfs to the backup node every 5 minutes. The backup node will watch for primary node outages every minute. In the event of a fail back, the backup node will replicate /testfs back to primary within one minte.

The DRCM distribution includes pre-built file replication scripts you can add to the back user’s crontab:

On primary node:

*/5 * * * *  /etc/drcm/file_system_sync/sync_mysql.sh > /dev/null 2>&1
*/5 * * * *  /etc/drcm/file_system_sync/sync_pgsql.sh > /dev/null 2>&1
*/6 * * * *  /etc/drcm/file_system_sync/sync_cifs.sh > /dev/null 2>&1
*/7 * * * *  /etc/drcm/file_system_sync/sync_nfs.sh > /dev/null 2>&1
*/8 * * * *  /etc/drcm/file_system_sync/sync_www.sh > /dev/null 2>&1
*/11 * * * * /etc/drcm/file_system_sync/sync_other.sh > /dev/null 2>&1

On backup node:

* * * * * /etc/drcm/file_system_sync/sync_mysql.sh > /dev/null 2>&1
* * * * * /etc/drcm/file_system_sync/sync_pgsql.sh > /dev/null 2>&1
* * * * * /etc/drcm/file_system_sync/sync_cifs.sh > /dev/null 2>&1
* * * * * /etc/drcm/file_system_sync/sync_nfs.sh > /dev/null 2>&1
* * * * * /etc/drcm/file_system_sync/sync_www.sh > /dev/null 2>&1
* * * * * /etc/drcm/file_system_sync/sync_other.sh > /dev/null 2>&1

All the configuration files will be autogened in the following:

/etc/drcm/my_cluster_conf

NOTE: Your backup user must have write access to the "my_cluster_conf" directory.

Edit the "/etc/drcm/my_cluster_conf/*sync_list.sh" config files to setup the specific directories to be replicated.

2-way file replication over the internet or a tunnel

When the cluster nodes do not share a LAN, for example a node at home and a node in a cloud, the replication addresses are not on a local interface and ICMP ping does not tell if the other node is up. Set the following in "my_cluster_info.sh" on both nodes:

export PRIMARY_FILESYSTEMS_HOST="node1"     # hostname -s of the primary
export BACKUP_FILESYSTEMS_HOST="node2"      # hostname -s of the backup
export PRIMARY_FILESYSTEMS_IP="drcm-node1"  # ssh host alias
export BACKUP_FILESYSTEMS_IP="drcm-node2"   # ssh host alias
export PRIMARY_FILESYSTEMS_KEEPALIVE_IP="127.0.0.1"
export BACKUP_FILESYSTEMS_KEEPALIVE_IP="127.0.0.2"
export NODE_UP_CHECK="ssh"

The _HOST VARs replace the _ETH interface check. The _IP VARs are used for ssh, scp and rsync, so each node's backup user needs a ~/.ssh/config Host entry for both aliases (host name, port and key). The _KEEPALIVE_IP VARs are the keep_alive_ip of each node in cm.cfg. NODE_UP_CHECK=ssh checks the other node with "ssh <alias> true".

The backup user runs "drcm_server --client" to read the cluster status. The client runs for root or for members of the drcm group, which also needs read access to the auth key and TLS files. Create the drcm group for the backup user:

# groupadd -r drcm
# usermod -a -G drcm <backup user>
# chgrp -R drcm /etc/drcm/.auth /etc/drcm/tls
# chmod 750 /etc/drcm/.auth /etc/drcm/tls
# chmod 640 /etc/drcm/.auth/* /etc/drcm/tls/*
# chgrp drcm /var/log/drcm /var/lib/drcm
# chmod 2775 /var/log/drcm /var/lib/drcm

The failback sync flags are kept in /var/lib/drcm, so a reboot of the backup during a fail over does not lose them. Before a normal sync the primary also checks the backup for a failback flag, and waits while there is one. This covers a primary that was down for the whole fail over and never received the flag.

A sync list can exclude files from the RsyncMirror and RsyncCopy lines that follow:

RsyncExclude "/logs/"
RsyncExclude "*.lock"
RsyncMirror /home/user/app_data

Excluded files are not copied, and a mirror does not delete them on the other node.

3-way sync: a third node, for example a web hosting account, can keep an emergency copy. It needs ssh and rsync, not DRCM:

export TERTIARY_FILESYSTEMS_IP="drcm-third"          # ssh host alias
export TERTIARY_FILESYSTEMS_DIR="/home/user/drcm_backup"
export TERTIARY_SYNC_MINUTES="15"

The active node sends the copy: the primary in normal operation, even when the backup node is down, and the backup while the cluster has failed over. Nothing is sent while a failback sync is pending. Each directory lands at TERTIARY_FILESYSTEMS_DIR plus its own path, with the same RsyncMirror, RsyncCopy and RsyncExclude lines. Restoring from the third node is done by hand with rsync.

Jobs that use the replicated data can be installed by DRCM with the crontab_fs_guard.sh resource instead of crontab.sh:

[CM_CRONTABS]
app, /etc/drcm/crontabs/app_crons, /etc/cron.d, /etc/drcm/resources/crontab_fs_guard.sh

On the primary file system node it refuses to install the crontab while a failback flag exists on either node, so the jobs do not start on old data after a fail back. DRCM retries every crontabs_check seconds, and the crontab is installed once the backup has synced its changes back. If the backup cannot be reached the crontab is installed.

Setting up cluster alerts

To setup an hourly cluster monitor with email and/or text message alerts, add the following line to root’s crontab:

00 * * * * /etc/drcm/utils/watch_cluster.sh > /dev/null 2>&1

The watch_cluster.sh script will automatically generate the following an alert list configuration file for you:

/etc/drcm/my_cluster_conf/cm_alert_list.sh

NOTE: If you run the watch_cluster.sh script as another user, the user must have read access to the /etc/drcm/.auth directory and auth keys.

In your "cm_alert_list.sh" file set up the email accounts and/or SMS accounts:

# vi /etc/drcm/my_cluster_conf/cm_alert_list.sh

# Set your comma delimited alert lists here
export EMAILlist="root,name@example.com"
export TEXTlist="555-555-5555@mobile_domain.com"

NOTE: Each mobile carrier has a domain for sending an email as a text message, for example to send SMS messages to a Verizon number:

555-555-5555@vtext.com

By default the DRCM alerts are limited to send messages every 4 hours, once an the first message is sent. This is done to avoid a flood email and/or text messages during periods of server outages or server maintenance.

If your clusters nodes do not have postfix mail configured, you need to determine which Linux server in your data center is setup to relay messages. To setup postfix mail to connect to relay host:

# vi /etc/postfix/main.cf

relayhost = 192.168.122.1

Restart the postfix service and watch the mail log:

# tail -f /var/log/maillog

To test the alerting function:

$ vi ~/test_alert.sh 

#!/bin/bash

export ALERTTIMESPANHOURS="0"
SUBJECT="[!ACTION!] Cluster Alert Test Message"
BODY="This is an cluster manager alert test message"
source /etc/drcm/utils/send_alert.sh
send_alert "${SUBJECT}" "${BODY}"

Save file and exit VI.

$ chmod 755 ~/test_alert.sh
$ ~/test_alert.sh

NOTE: The DRCM alert functions can be use by other system admin functions by writing scripts similar to the above example.

If you do not have a Linux system setup up as a relay host, you can setup postfix on one cluster node as a smart host and allow other nodes to relay messages. For more information on setting up postfix mail as a smart host, search online for: "postfix smarthost setup"

Remaining Work on This Project

IPv6 cluster traffic between nodes, as an opt-in option (IPv4 stays the default). TCP + TLS clusters for WAN links were added in 1.42.

Support and Bug Tracking

https://datareel.com/contact.php