DataReel Cluster Manager

Overview and architecture

The DataReel Cluster Manager (DRCM) builds high availability Linux server clusters. It manages system resources (crontabs, floating IP addresses, system services, applications and file systems) and fails them over automatically when one or more servers fail.

DRCM supports many cluster nodes in a non-hierarchical arrangement. Resources are spread across the nodes, and each resource can have backup nodes. A typical cluster is two servers of equal capacity, one primary and one backup, but DRCM scales past two nodes and across data centers in the same or different locations. In virtualized data centers, clustering VMs that run on different bare metal hosts protects services from the loss of a host.

How it is built

DRCM is a single C++ multithreaded daemon, drcm_server, built on the DataReel class library (threads, sockets, strings, logging). The bundled copy of the library in include/ and src/ is trimmed to the 16 components DRCM uses. Each node runs the same binary with the same /etc/drcm directory.

Conventions in this guide: commands with a $ prompt run as a user or service account, commands with # run as root, and ... in a config example means content left out.

Platform support

RHEL compatible rebuilds (Rocky Linux, AlmaLinux, Oracle Linux) of the same major versions follow the same rules.

Install and build

Plan the cluster before installing: the number of nodes, which node runs which crons, which floating IPs and services are needed, which file systems each node mounts, which applications run where, and SSH keys between nodes for file sync and automation.

RPM install

Prebuilt RPMs for RHEL 8, 9 and 10 are on the downloads page, datareel.com/downloads/drcm.

$ sudo dnf -y install ./drcm_server-1.42-1.el9.x86_64.x86_64.rpm
$ rpm -ql drcm_server          # files
$ rpm -qc drcm_server          # configs
$ sudo dnf -y remove drcm_server

Build your own RPM

The build needs gcc-c++, make, rpm-build and openssl-devel.

$ cd $HOME; mkdir -pv git; cd $HOME/git
$ wget https://datareel.com/downloads/drcm/ha-cluster-manager-1.42.tar.gz
$ tar xzf ha-cluster-manager-1.42.tar.gz
$ mv ha-cluster-manager-1.42 ha-cluster-manager
$ cd $HOME/git/ha-cluster-manager/rpm_builder
$ ./make_rhel10_rpm.sh         # or make_rhel9_rpm.sh, make_rhel8_rpm.sh
$ sudo dnf -y install $HOME/rpmbuild/RPMS/x86_64/drcm_server-1.42-1.el10.x86_64.x86_64.rpm

Build from source

$ sudo dnf -y install gcc-c++ make openssl-devel
$ cd ~/git/ha-cluster-manager/env
$ source gnu_env.sh
$ cd ~/git/ha-cluster-manager/drcm_server
$ make                          # builds ../lib/libdatareel.a, then make -f makedrcm.mak
$ sudo su root -c 'make install_root'

• make in drcm_server builds the bundled DataReel library once (makelib.mak) and then the server with makedrcm.mak.
• The server links OpenSSL through -lssl -lcrypto (set in env/linux.env) for HMAC-SHA256 and TLS.
• env/gnu_env.sh exports hardening flags: -fstack-protector-strong -D_FORTIFY_SOURCE=2 -fPIE, linked with -pie -Wl,-z,relro,-z,now (PIE and full RELRO, the modern RHEL defaults).
• make install_root installs to /usr/sbin/drcm_server and the configuration tree to /etc/drcm.

systemd unit

All supported targets use the native unit systemd_init/drcm_server.service, installed to /usr/lib/systemd/system:

[Service]
Type=simple
Environment=CFGFILE=/etc/drcm/cm.cfg
EnvironmentFile=-/etc/sysconfig/drcm_server
ExecStart=/usr/sbin/drcm_server --config-file=${CFGFILE} $ARGS
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=180
Restart=on-failure

DRCM stops its own resources on SIGTERM, so only the server process is signaled, resource stop scripts get up to 180 seconds, and logs go to the journal as well as the DRCM log. Set CFGFILE or ARGS in /etc/sysconfig/drcm_server.

Cluster auth keys

Every node in a cluster uses the same 2048-byte random authkey. It keys the HMAC-SHA256 on every cluster message (see Security model). To create the key, or to replace it:

# mkdir -pv /etc/drcm/.auth
# chmod 700 /etc/drcm/.auth
# dd if=/dev/urandom bs=1024 count=2 > /etc/drcm/.auth/authkey
# sha1sum /etc/drcm/.auth/authkey | cut -b1-40 > /etc/drcm/.auth/authkey.sha1
# chmod 600 /etc/drcm/.auth/authkey /etc/drcm/.auth/authkey.sha1

• authkey.sha1 only fingerprints the key file at rest. The server checks that it holds a 40 character hex hash. It is not used to authenticate messages.
• The server warns if either file can be read by other users. Keep them mode 600, owned by the cluster user.
• Copy the .auth directory to every node over SSH. A node with a different key cannot talk to the cluster.
• Scripts that query the cluster as another user (for example watch_cluster.sh) need read access to the key.

cm.cfg configuration reference

The default configuration file is /etc/drcm/cm.cfg. If it does not exist, --check-config writes a template you can edit:

# /usr/sbin/drcm_server --check-config
# /usr/sbin/drcm_server --config-file=/tmp/test1.cfg --check-config
# /usr/sbin/drcm_server --check-config | grep -i -E 'error|warn|fail|info'

Always check the file after editing and before starting the server. --check-config warns about unknown settings, nodes missing nodename, hostname or keep_alive_ip, and duplicate node names. In TCP mode it also checks that the TLS files exist and that a witness is set when quorum is required.

The file has a [CM_SERVER] section, five resource sections and one [CM_NODE] section per node. Settings are matched by their full name and # starts a comment.

[CM_SERVER] general settings

[CM_SERVER] transport, TLS, anti-replay and quorum (added in 1.42)

Every key has a safe default, so a config without them runs exactly as before: UDP transport, no quorum.

[CM_NODE] settings

Managed resources and resource scripts

Every node can run a global resource as primary or as backup. Each resource line in a global section is a CSV list that starts with a unique nickname.

[CM_CRONTABS]
# nickname, template_file, install_location, resource_script
system, /etc/drcm/crontabs/system_crons, /etc/cron.d, /etc/drcm/resources/crontab.sh
apps2, /etc/drcm/crontabs/apps_package2, /etc/cron.d, /etc/drcm/resources/crontab.sh

[CM_IPADDRS]
# nickname, floating IP, netmask, interface, resource_script
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_nmcli.sh

[CM_SERVICES]
# nickname, service_name, resource_script
web, httpd, /etc/drcm/resources/service.sh

[CM_APPLICATIONS]
# nickname, user:group, start_program, stop_program[, ensure_script]
ldm, ldm:users, ~/util/ldm_ensure.sh, ldmadmin stop, ~/util/ldm_ensure.sh

[CM_FILESYSTEMS]
# nickname, source, target, resource_script
data, 192.168.122.1:/data, /data, /etc/drcm/resources/nfs.sh
webshare, /192.168.122.225/users/web, /home/users/web, /etc/drcm/resources/cifs.sh

Resource scripts

Scripts live in /etc/drcm/resources and take start, stop and watch actions. Custom scripts work if they accept the same arguments.

Linking resources to nodes

[CM_NODE]
nodename = cmnode1
...
node_crontabs = user,system,apps1
node_backup_crontabs = cmnode2:apps2
node_floating_ip_addrs = cmnodef
node_services = web
node_filesystems = data,archive,webshare
node_applications = ldm

[CM_NODE]
nodename = cmnode2
...
node_crontabs = apps2
node_backup_crontabs = cmnode1:user,cmnode1:system
node_backup_floating_ip_addrs = cmnode1:cmnodef
node_services = web
node_filesystems = data,archive,webshare
node_applications = ldm

Applications

Applications are programs outside the OS. DRCM runs them as the given user:group with that user's environment sourced, so programs on the user's PATH need no absolute path. Start and stop programs can be several commands separated by ;. With an ensure script, DRCM uses it to watch the application.

ldm, ldm:users, ldmadmin start, ldmadmin stop
ldm, ldm:users, ldmadmin clean; ldmadmin mkqueue; ldmadmin start, ldmadmin stop; ldmadmin delqueue
ldm, ldm:users, ldmadmin start, ldmadmin stop, if [ -f ~/util/ldm_ensure.sh ]; then ~/util/ldm_ensure.sh; fi

Transports: UDP and TCP with TLS

The ping, rstats, cm_stat and keep alive clients all use one transport-aware exchange, so every command works the same way in both modes. In TCP mode the UDP thread still starts (for local clients); peers are reached over TLS.

Open the matching ports in firewalld on every node, from every other node:

# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.184/32" port port="53897" protocol="udp" accept'
# firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.122.184/32" port port="53897" protocol="tcp" accept'
# firewall-cmd --reload
# firewall-cmd --list-rich-rules

Nodes must also be able to ping and SSH each other. firewalld allows ping by default; if it is blocked, remove the echo-request ICMP block. Direct iptables rules are legacy and not documented.

Security model

Authenticated version 2 frames

• Every cluster message is a version 2 frame: a 76-byte header (magic, version, command, flags, payload length, timestamp, sequence number, 16-byte nonce) plus the payload, with an HMAC-SHA256 over the whole frame keyed with the 2048-byte cluster authkey.
• The HMAC is checked in constant time before any field is trusted, so a forged or changed command is rejected. This replaces the static SHA-1 hash older releases sent in the clear.
• Payloads are capped at 65,507 bytes and stats replies at 64 KB, which bounds every allocation made from a length field.

Anti-replay

• Each sender adds a monotonic sequence number and a random 16-byte nonce to every frame.
• The receiver checks the nonce cache first: the last 256 nonces per peer are remembered, and an exact replay is rejected. This needs no clock.
• On a TCP connection the sequence number must also increase strictly.
• Optional: replay_window_secs rejects frames whose timestamp is outside the window. Use it only with synchronized clocks.

TLS (TCP transport)

• TLS 1.2 is the floor (tls_min_version = 1.3 raises it).
• Mutual authentication: both ends present a certificate signed by the cluster CA and verify the peer against tls_ca_file. The handshake fails closed when verification fails.
• Optional pinning with tls_peer_fingerprint rejects any certificate other than the pinned one.
• Tested against a live server: plain TCP, no client certificate and a certificate from another CA are refused at the handshake, and a client with a valid cluster certificate still cannot forge a command because the HMAC check fails.

Hardening

• Built with stack protector, _FORTIFY_SOURCE=2, PIE and full RELRO.
• Signals are handled in a dedicated sigwait() thread, not in a signal handler.
• Commands run through su or sudo are shell quoted.
• Scripts use StrictHostKeyChecking=accept-new (override with DRCM_SSH_OPTS), mode 644 lock and time stamp files, and no world writable mount points.
• The UDP protocol was fuzzed with 2000 malformed packets under AddressSanitizer and UBSan without a fault.

TCP and TLS setup

The default is a private, self-signed cluster CA that signs one certificate per node. Externally issued certificates, such as Let's Encrypt via acme.sh, also work: DRCM only needs a CA file, a certificate and a key in PEM format.

1. Create the cluster CA (once, on the entry point node)

# mkdir -pv /etc/drcm/tls && chmod 700 /etc/drcm/tls && cd /etc/drcm/tls
# openssl req -x509 -newkey rsa:3072 -sha256 -days 3650 -nodes \
    -keyout ca-key.pem -out ca.pem -subj "/CN=DRCM Cluster CA"

Keep ca-key.pem offline or on one admin host. Nodes only need ca.pem.

2. Issue a certificate for each node

# NODE=cmnode1.example.com
# openssl req -newkey rsa:3072 -nodes -keyout ${NODE}-key.pem \
    -out ${NODE}.csr -subj "/CN=${NODE}"
# printf "subjectAltName=DNS:${NODE}\nextendedKeyUsage=serverAuth,clientAuth\n" > ${NODE}.ext
# openssl x509 -req -in ${NODE}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
    -days 825 -sha256 -extfile ${NODE}.ext -out ${NODE}.pem

Each node uses its certificate for both roles (it serves and connects), so include both serverAuth and clientAuth. Install the files as /etc/drcm/tls/node.pem and /etc/drcm/tls/node-key.pem on that node, mode 600.

3. Configure the cluster

[CM_SERVER]
...
cluster_transport = tcp
tcpport = 53897
tls_ca_file = /etc/drcm/tls/ca.pem
tls_cert_file = /etc/drcm/tls/node.pem
tls_key_file = /etc/drcm/tls/node-key.pem
tls_min_version = 1.2

Because /etc/drcm is identical on all nodes, the paths are the same everywhere; only the file contents differ per node. Exclude /etc/drcm/tls from the rsync that copies /etc/drcm to the other nodes, or each node will get the entry point's certificate.

4. Optional: pin node certificates

# openssl x509 -in cmnode2.example.com.pem -noout -fingerprint -sha256
sha256 Fingerprint=4A:1F:...:9C

[CM_NODE]
nodename = cmnode2
...
tls_peer_fingerprint = 4A:1F:...:9C

Remember to update the pin when a node certificate is renewed.

Externally issued certificates

With acme.sh or another ACME client, point tls_cert_file at the full chain, tls_key_file at the key and tls_ca_file at the issuing root. Any certificate that root signs will pass verification, so pin node fingerprints (and update the pins on renewal) when you use a public CA. Restart DRCM after a renewal so it loads the new files.

Witness quorum

In a two-node cluster a broken link between the nodes looks the same as a dead peer. Without help, both nodes could take the resources at once (split brain). A witness breaks the tie.

[CM_SERVER]
...
quorum_required = 1
witness_endpoint = 10.0.0.9

• Before taking over a dead peer's resources, the node pings the witness with an authenticated v2 frame over the configured transport.
• Witness reachable: this node is in the majority (itself plus the witness) and fails over.
• Witness unreachable: this node is probably the isolated one and stands down. An attacker who blocks the peer link cannot force a false failover.
• Quorum required with no witness set fails closed (no takeover), and --check-config reports it.

The witness is any reachable DRCM endpoint that answers the authenticated protocol: a small third VM running drcm_server with the same authkey (and, in TCP mode, a certificate from the cluster CA). Place it so that it shares a failure domain with neither node, for example a third availability zone or site.

Running, monitoring, stats and logs

After the configuration checks clean and /etc/drcm is mirrored to every node, start the server on each node:

# systemctl start drcm_server
# systemctl status drcm_server
# systemctl enable drcm_server

Client commands

Per-node report from raw stats:

for CMNODE in cmnode1 cmnode2 cmnode3 cmnode4
do
  echo $CMNODE
  /usr/sbin/drcm_server --client --command=rstats | sed -n "/$CMNODE/,/$CMNODE/p" | grep -v $CMNODE
  echo ""
done

Logs

# tail -f /var/log/drcm/drcm.log
# tail -f /var/log/drcm/ipv4addr_resource.log     # floating IP scripts (IPv4 and IPv6)
# journalctl -u drcm_server -f

Testing failover and failback

# systemctl stop drcm_server       # on the primary: resources move to the backup
# drcm_server --client --command=cm_stat    # from the backup
# systemctl start drcm_server      # failback
# drcm_server --client --command=cm_stat

Other server options: --version, --debug, --debug-level=1..5, --verbose, --log-file=, --log-level=, --log-file-clear, --log-disable.

Floating IP failover per platform

On premises and VMs: NetworkManager

NetworkManager (nmcli) is the method on RHEL 8, 9 and 10. The NetworkManager connection name must be the same as the device name, because the scripts update the connection and then reapply it to the device.

[CM_IPADDRS]
cmnodef, 192.168.122.180, 24, ens192, /etc/drcm/resources/ipv4addr_nmcli.sh
# VM and bare metal in one cluster: list both interface names
cmnodef, 192.168.122.180, 255.255.255.0, 'ens192|eno8303', /etc/drcm/resources/ipv4addr_nmcli.sh

Test on each node:

# /etc/drcm/resources/ipv4addr_nmcli.sh 192.168.122.180 24 ens192 start
# ip a
# /etc/drcm/resources/ipv4addr_nmcli.sh 192.168.122.180 24 ens192 stop

For complex layouts, ipv4addr_copy_nmcli.sh swaps per-node <host>_start_backup_ip_<dev> and <host>_stop_backup_ip_<dev> connection files kept in /etc/drcm/my_cluster_conf. The start file lists the floating IP; the stop file is the node's normal connection.

IPv6 floating IPs

cmnodef6, 2001:db8:122::180, 64, ens192, /etc/drcm/resources/ipv6addr_nmcli.sh
# nmcli -g ipv6.method connection show ens192    # must not be "disabled" or "ignore"
# nmcli con mod ens192 ipv6.method auto && nmcli device reapply ens192

Cluster traffic between nodes stays IPv4; only the floating IPs users connect to can be IPv6.

AWS: Elastic IP and route-table VIP

In AWS an interface alias does not move between instances. aws_ipv4addr.sh takes the same arguments as the other IPv4 scripts and moves the address through the EC2 API, using the AWS CLI v2 and instance metadata through IMDSv2 (falling back to IMDSv1).

web, eipalloc-0abc1234, 32, eth0, /etc/drcm/resources/aws_ipv4addr.sh

web, 10.0.12.50, 32, eth0, /etc/drcm/resources/aws_ipv4addr.sh

Set the variables in the cluster user's bash_rc_file. The instance role needs ec2:AssociateAddress, ec2:DisassociateAddress, ec2:DescribeAddresses, ec2:ReplaceRoute, ec2:CreateRoute and ec2:DescribeRouteTables. Clusters that span AWS and another site should use cluster_transport = tcp.

Legacy: network-scripts

ipv4addr.sh, ipv4addr_multi.sh, ipv4addr_copy_config.sh and the ifcfg-eth0_N templates are for systems without NetworkManager only. See README.txt for the conversion notes.

Setup and migration tools

tools/migrate_cfg.py

Migrates a pre-1.42 cm.cfg. It is additive: the file is copied through unchanged and the new keys are inserted at the end of [CM_SERVER]. It is idempotent: a file that already has cluster_transport is left alone (apart from --tcp switching the transport). By default the cluster stays on UDP.

$ python3 tools/migrate_cfg.py /etc/drcm/cm.cfg -o /tmp/cm.cfg.new
$ python3 tools/migrate_cfg.py /etc/drcm/cm.cfg -o /tmp/cm.cfg.new \
      --tcp --quorum --witness 10.0.0.9 --tls-min 1.3
$ diff /etc/drcm/cm.cfg /tmp/cm.cfg.new
# drcm_server --config-file=/tmp/cm.cfg.new --check-config

tools/drcm_gui.py

A tkinter GUI (Python standard library only) with two tabs: New cluster builds a cm.cfg from cluster settings, nodes, services, transport, TLS and quorum; Migrate runs migrate_cfg.py on an existing file. The config generator is a pure function, so it can be checked without a display:

$ sudo dnf -y install python3-tkinter
$ python3 tools/drcm_gui.py              # GUI
$ python3 tools/drcm_gui.py --selftest   # headless check

Always run --check-config on a generated file before you use it.

Two-node cluster walkthrough

Based on etc/cm_2_node_example.cfg: cmnode1 (192.168.122.111) is primary for the web and ldm floating IPs, httpd and the data NFS mount; cmnode2 (192.168.122.112) backs them up and is primary for archive.

1. Install DRCM on both nodes (RPM or source build).
2. Auth key: create /etc/drcm/.auth/authkey on cmnode1 as shown in Cluster auth keys.
3. Config: copy the example and edit host names, IPs and resources.

   # cp /etc/drcm/cm_2_node_example.cfg /etc/drcm/cm.cfg
   # vi /etc/drcm/cm.cfg

4. Turn off debug for production: the example sets debug = 1 and debug_level = 5.
5. Check it:

   # drcm_server --check-config | grep -i -E 'error|warn|fail|info'

6. Firewall: on each node allow UDP and TCP 53897 and SSH from the other node.
7. Copy /etc/drcm to cmnode2, dry run first:

   # rsync -av --force --stats --progress -n /etc/drcm 192.168.122.112:/etc/.
   # rsync -av --force --stats --progress /etc/drcm 192.168.122.112:/etc/.

8. Test the floating IPs on each node with the resource script start and stop actions.
9. Start drcm_server on both nodes and check cm_stat.
10. Fail over: stop DRCM on cmnode1 and watch cmnode2 take the web IP, httpd and the backup resources. Start cmnode1 again to fail back.
11. Optional hardening for nodes in different sites: run migrate_cfg.py --tcp --quorum --witness HOST, install TLS files, and add a witness VM.
12. Enable the service on both nodes: systemctl enable drcm_server.

File replication and alerts

Two-way file replication

/etc/drcm/file_system_sync holds rsync over SSH scripts that mirror file systems from a primary to a backup node, and back again after a failback. Configure the nodes in /etc/drcm/my_cluster_conf/my_cluster_info.sh (CLUSTER_NAME, BACKUP_USER, PRIMARY_FILESYSTEMS_IP, BACKUP_FILESYSTEMS_IP and the matching *_ETH settings), list directories in the *sync_list.sh files with RsyncMirror (exact mirror) or RsyncCopy (keep deleted files), and run the sync scripts from the backup user's crontab: every 5 minutes on the primary, every minute on the backup. The backup user needs SSH key access between the nodes and write access to my_cluster_conf. /etc/drcm/utils/manual_fs_failover.sh tests a failover.

$ crontab -e      # primary
*/5 * * * *  /etc/drcm/file_system_sync/sync_testfs.sh > /dev/null 2>&1
$ crontab -e      # backup
* * * * * /etc/drcm/file_system_sync/sync_testfs.sh > /dev/null 2>&1

Cluster alerts

00 * * * * /etc/drcm/utils/watch_cluster.sh > /dev/null 2>&1     # root crontab
# vi /etc/drcm/my_cluster_conf/cm_alert_list.sh
export EMAILlist="root,name@example.com"
export TEXTlist="555-555-5555@vtext.com"

Alerts are limited to one every 4 hours after the first message. Nodes need a working postfix relay (relayhost in /etc/postfix/main.cf). Other scripts can send alerts by sourcing /etc/drcm/utils/send_alert.sh and calling send_alert "subject" "body".

Troubleshooting and support

Support

• Product page and online docs: datareel.com/info_drcm.php
• Source code and downloads: datareel.com/downloads/drcm
• Questions and bug reports: datareel.com/contact.php
• Release notes: CHANGELOG.md in the source distribution.