Master PBX High Availability in 2023 with VitalPBX: Unlock Uninterrupted Communication

1. Introduction

High availability is a characteristic of a system which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period.

Make a high-availability cluster out of any pair of VitalPBX servers. VitalPBX can detect a range of failures on one VitalPBX server and automatically transfer control to the other server, resulting in a telephony environment with minimal down time.

PBX High Availability

2. Requirements to install VitalPBX in High Availability

Setting up High Availability (HA) with DRBD (Distributed Replicated Block Device) involves ensuring critical systems and services are available with minimal downtime in case of failures. DRBD enables real-time data replication between nodes to ensure data availability and integrity. Here are some general requirements to set up High Availability with DRBD:

1.- Physical Infrastructure and Networking:

Two or more identical or very similar nodes (servers) to implement redundancy.

Reliable and low-latency network connection between the nodes. This can be a dedicated replication network (preferred) or a shared network if it’s of high quality.

2.- Operating System – VitalPBX version:

Nodes should run the same operating system, preferably the same version and the same version of VitalPBX.

3.- Disk Partitioning:

The storage device to be replicated should be partitioned and accessible on both nodes.

Each node should have sufficient storage space to accommodate replication and data.

4.- Network Configuration:

Each node should have static IP addresses and resolve correctly in the local DNS system or in the /etc/hosts file of the other node.

Hostnames should be consistent across nodes.

5.- DRBD:

Install and configure DRBD on both nodes.

Configure DRBD resources that define which devices will be replicated and how replication will be established.

At the time of installation leave the largest amount of space on the hard drive to store the variable data on both servers.

Define node roles: primary and secondary nodes.

3. Installation

We are going to start by installing VitalPBX on two servers

a.- When you get to the partitions part you must select “Guide – use entire disk”:

b.- Select:

All file in one partition (recommended for new user)

c.- Select primary:

#1 primary

d.- Delete partition

e.- Select pri/log

f.- Create New Partition

g.- Change the capacity to:

New partition size: 20GB

We need enough space for the operating system and its applications in the future; then <continue>

Select Primary

Select Beginning

Select Done setting up the partition

h.- Finish

Pres Yes to confirm

And continue with the installation.

4. Configurations

4.1 IP and Hostname Configuration.

We will configure in each server the IP address and the host name.

Name

Master

Slave

Hostname

vitalpbx-master.local

vitalpbx-slave.local

IP Address

192.168.10.31

192.168.10.32

Netmask

255.255.254.0

255.255.254.0

Gateway

192.168.10.1

192.168.10.1

Primary DNS

8.8.8.8

8.8.8.8

Secondary DNS

8.8.4.4

8.8.4.4

4.1.1 Remote access with the root user.

As a security measure, the root user is not allowed to remote access the server. If you wish to unblock it, remember to set a complex password, and perform the following steps.

Enter the Command Line Console with the root user directly on your server with the password you set up earlier.

Edit the following file using nano on both servers, /etc/ssh/sshd_config.

				
					nano /etc/ssh/sshd_config
				
			

Change the following line

				
					#PermitRootLogin prohibit-password
				
			

With

				
					PermitRootLogIn yes
				
			

Save, Exit and restart the sshd service.

				
					systemctl restart sshd
				
			

4.1.2 Changing your IP Address to a static address.

By default, our system’s IP address in Debian is obtained through DHCP, however, you can modify it and assign it a static IP address using the following steps:

Edit the following file with nano on both servers, /etc/network/interfaces.

				
					nano /etc/network/interfaces
				
			

Change

				
					#The primary network interface
allow-hotplug eth0
iface eth0 inet dchp

				
			

With the following. Server Master.

				
					#The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.10.31
netmask 255.255.254.0
gateway 192.168.10.1

				
			

With the following. Server Slave.

				
					#The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.10.32
netmask 255.255.254.0
gateway 192.168.10.1

				
			

Lastly, reboot both servers and you can now log in via ssh.

4.2 Install Dependencies

Install dependencies on both servers

				
					apt -y install drbd-utils corosync pacemaker pcs chrony xfsprogs
				
			

4.3 Hostname

Go to the web interface to:

ADMIN>System Settings>Network Settings


First, change the Hostname in both servers, and remember to press the Save button.

 

Master

Slave

Now we connect through ssh to each of the servers and we configure the hostname of each server in the /etc/hosts file, so that both servers see each other with the hostname.

 

Server Master

				
					hostname vitalpbx-master.local
				
			

Server Slave

				
					hostname vitalpbx-slave.local
				
			

Server Master and Slave

				
					nano /etc/hosts
				
			

Add the following lines on both servers

				
					192.168.10.31 vitalpbx-master.local
192.168.10.32 vitalpbx-slave.local
				
			

4.4.- Create the partition on both servers

Initialize the partition to allocate the available space on the hard disk. Do these on both servers.

				
					fdisk /dev/sda
				
			
				
					Command (m for help): n
Partition type:
  p   primary (3 primary, 0 extended, 1 free)
  e   extended
Select (default e): p
Selected partition 3 (take note of the assigned partition number as we will need it later)
First sector (35155968-266338303, default 35155968): [Enter]
Last sector, +sectors or +size{K,M,G} (35155968-266338303, default 266338303): [Enter]
Using default value 266338303
Partition 4 of type Linux and of size 110.2 GiB is set
Command (m for help): t
Partition number (1-4, default 4): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): w
				
			

Then, restart the servers so that the new table is available.

				
					reboot
				
			

4.5.- Using Script

We have two ways to configure the Cluster. Using the following Script or following this manual step by step. If you decide to use the following Script, the next step you should follow in this manual is 5 if you consider it necessary. Otherwise continue with step 4.6.

 

Create authorization key for the Access between the two servers without credentials.

Create key in Server Master

				
					ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' >/dev/null
				
			
				
					ssh-copy-id root@192.168.10.32
				
			
				
					Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
root@192.168.10.62's password: (remote server root’s password)

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.32'"
and check to make sure that only the key(s) you wanted were added. 

root@vitalpbx-master:~#
				
			

Create key in Server Slave

				
					ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' >/dev/null
				
			
				
					ssh-copy-id root@192.168.10.31
				
			
				
					Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
root@192.168.10.61's password: (remote server root’s password)

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.31'"
and check to make sure that only the key(s) you wanted were added. 

root@vitalpbx-slave:~#
				
			

Now copy and run the following script in Master Server.

				
					mkdir /usr/share/vitalpbx/ha
				
			
				
					cd /usr/share/vitalpbx/ha
				
			
				
					wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/vpbxha.sh
				
			
				
					chmod +x vpbxha.sh
				
			
				
					./vpbxha.sh

				
			

Now we enter all the information requested.

				
					************************************************************
*  Welcome to the VitalPBX high availability installation  *
*                All options are mandatory                 *
************************************************************
IP Server1............... > 192.168.10.31
IP Server2............... > 192.168.10.32
Floating IP.............. > 192.168.10.30
Floating IP Mask (SIDR).. > 24
Disk (sdax).............. > sda3
hacluster password....... > MyPassword
************************************************************
*                   Check Information                      *
*        Make sure you have internet on both servers       *
************************************************************
Are you sure to continue with this settings? (yes,no) > yes

				
			

CONGRATULATIONS, you have installed high availability in VitalPBX 4

4.6 Firewall

Configure the Firewall (Both Servers)

Port

Description

TCP 2224

Required on all nodes (needed by the pcsd Web UI and required for node-to-node communication)

It is crucial to open port 2224 in such a way that pcs from any node can talk to all nodes in the cluster, including itself. When using the Booth cluster ticket manager or a quorum device you must open port 2224 on all related hosts, such as Booth arbiters or the quorum device host.

TCP 3121

Required on all nodes if the cluster has any Pacemaker Remote nodes

Pacemaker’s crmd daemon on the full cluster nodes will contact the pacemaker_remoted daemon on Pacemaker Remote nodes at port 3121. If a separate interface is used for cluster communication, the port only needs to be open on that interface. At a minimum, the port should open on Pacemaker Remote nodes to full cluster nodes. Because users may convert a host between a full node and a remote node, or run a remote node inside a container using the host’s network, it can be useful to open the port to all nodes. It is not necessary to open the port to any hosts other than nodes.

TCP 5403

Required on the quorum device host when using a quorum device with corosync-qnetd. The default value can be changed with the -p option of the corosync-qnetd command.

UDP 5404

Required on corosync nodes if corosync is configured for multicast UDP.

 

UDP 5405

Required on all corosync nodes (needed by corosync)

TCP 21064

Required on all nodes if the cluster contains any resources requiring DLM (such as clvm or GFS2)

TCP 9929, UDP 9929

Required to be open on all cluster nodes and booth arbitrator nodes to connections from any of those same nodes when the Booth ticket manager is used to establish a multi-site cluster.

TCP 7789

Required by DRBD to synchronize information.

Add in both Servers the Service and Rule with this Service.

Add the Service in ADMIN/Firewall/Services Add Service

We add the Rule in /ADMIN/Firewall/Rules Add Rules

And we apply changes

4.7 Format the partition

Now, we will proceed to format the new partition in both servers with the following command:

				
					mke2fs -j /dev/sda3
				
			
				
					dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync
				
			

4.8 Configuring DRBD

Load the module and enable the service on both nodes, using the follow command:

				
					modprobe drbd
				
			
				
					systemctl enable drbd.service
				
			

Create a new global_common.conf file on both servers with the following contents:

				
					mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.orig
				
			
				
					nano /etc/drbd.d/global_common.conf


				
			
				
					global {
  usage-count no;
}
  common {
net {
  protocol C;
  }
}
				
			

Next, we will need to create a new configuration file called /etc/drbd.d/drbd0.res for the new resource named drbd0, with the following contents on both servers:

				
					nano /etc/drbd.d/drbd0.res
				
			
				
					resource drbd0 {
startup {
        wfc-timeout  5;
	outdated-wfc-timeout 3;
        degr-wfc-timeout 3;
	outdated-wfc-timeout 2;
}
syncer {
	rate 10M;
 	verify-alg md5;
}
net {
    after-sb-0pri discard-older-primary;
    after-sb-1pri discard-secondary;
    after-sb-2pri call-pri-lost-after-sb;
}
handlers {
    pri-lost-after-sb "/sbin/reboot";
}
on vitalpbx-master.locall {
	device /dev/drbd0;
   	disk /dev/sda3;
   	address 192.168.10.31:7789;
	meta-disk internal;
}
on vitalpbx-slave.local {
	device /dev/drbd0;
   	disk /dev/sda3;
   	address 192.168.10.32:7789;
	meta-disk internal;
   	}
}
				
			

Note:

Although the access interfaces can be used, which in this case is ETH0. It is recommended to use an interface (ETH1) for synchronization, this interface must be directly connected between both servers.

Initialize the meta data storage on each nodes by executing the following command on both servers

				
					drbdadm create-md drbd0
				
			
				
					Writing meta data...
New drbd meta data block successfully created.
				
			

Let’s define the DRBD Primary node as first node “vitalpbx-master”

				
					drbdadm up drbd0
				
			
				
					drbdadm primary drbd0 --force

				
			

On the Secondary node “vitalpbx-slave” run the following command to start the drbd0

				
					drbdadm up drbd0
				
			

You can check the current status of the synchronization while it’s being performed. The cat /proc/drbd command displays the creation and synchronization progress of the resource, as shown here:

4.8.1.- Formatting and Test DRBD Disk

Create the directory where we are going to mount the volume with all the information to be replicated on both servers.

				
					mkdir /vpbx_data
				
			

In order to test the DRBD functionality we need to Create a file system, mount the volume and write some data on primary server “vitalpbx-master” and finally switch the primary node to “vitalpbx-slave”

Run the following command on the primary server to create an xfs filesystem on /dev/drbd0 and mount it to the vpbx_data directory, using the following commands in vitalpbx-master.

				
					mkfs.xfs /dev/drbd0
				
			
				
					mount /dev/drbd0 /vpbx_data
				
			

Create some data using the following command on vitalpbx-master:

				
					touch /vpbx_data/file{1..5}
				
			
				
					ls -l /vpbx_data
				
			
				
					-rw-r--r-- 1 root root 0 Nov 17 11:28 file1
-rw-r--r-- 1 root root 0 Nov 17 11:28 file2
-rw-r--r-- 1 root root 0 Nov 17 11:28 file3
-rw-r--r-- 1 root root 0 Nov 17 11:28 file4
-rw-r--r-- 1 root root 0 Nov 17 11:28 file5
				
			

Let’s now switch primary mode “vitalpbx-server” to second node “vitalpbx-slave” to check the data replication works or not.

First, we have to unmount the volume drbd0 on the first drbd cluster node “vitalpbx-master” and change the primary node to secondary node on the first drbd cluster node “vitalpbx-master”

				
					umount /vpbx_data
				
			
				
					drbdadm secondary drbd0
				
			

Change the secondary node to primary node on the second drbd cluster node “vitalpbx-slave”

				
					drbdadm primary drbd0 --force
				
			

Mount the volume and check the data available or not.

				
					mount /dev/drbd0 /vpbx_data
				
			
				
					ls -l  /vpbx_data
				
			
				
					total 0
-rw-r--r-- 1 root root 0 Nov 17 11:28 file1
-rw-r--r-- 1 root root 0 Nov 17 11:28 file2
-rw-r--r-- 1 root root 0 Nov 17 11:28 file3
-rw-r--r-- 1 root root 0 Nov 17 11:28 file4
-rw-r--r-- 1 root root 0 Nov 17 11:28 file5
				
			

Normalize Server-Slave

				
					umount /vpbx_data
				
			
				
					drbdadm secondary drbd0
				
			

Normalize Server-Master

				
					drbdadm primary drbd0
				
			
				
					mount /dev/drbd0 /vpbx_data

				
			

4.9 - Configure Cluster

Create the password of the hacluster user on both nodes

				
					echo hacluster:Mypassword | chpasswd
				
			

Start PCS on both servers

				
					systemctl start pcsd 
				
			

Configure the start of services on both nodes

				
					systemctl enable pcsd.service 
				
			
				
					systemctl enable corosync.service 
				
			
				
					systemctl enable pacemaker.service
				
			

Server Authenticate in Master

On vitalpbx-master, use pcs cluster auth to authenticate as the hacluster user:

				
					pcs cluster destroy
				
			
				
					pcs host auth vitalpbx-master.local vitalpbx-slave.local -u hacluster -p Mypassword
				
			
				
					vitalpbx-master.local: Authorized
vitalpbx-slave.local: Authorized
				
			

Next, use pcs cluster setup on the vitalpbx-master to generate and synchronize the corosync configuration.

				
					pcs cluster setup cluster_vitalpbx vitalpbx-master.local vitalpbx-slave.local --force
				
			

Starting Cluster in Master

				
					pcs cluster start --all
				
			
				
					pcs cluster enable --all
				
			
				
					pcs property set stonith-enabled=false
				
			
				
					pcs property set no-quorum-policy=ignore
				
			

Prevent Resources from Moving after Recovery
In most circumstances, it is highly desirable to prevent healthy resources from being moved around the cluster. Moving resources almost always requires a period of downtime. For complex services such as databases, this period can be quite long.

				
					pcs resource defaults update resource-stickiness=INFINITY
				
			

Create resource for the use of Floating IP

				
					pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.10.30 cidr_netmask=24 op monitor interval=30s on-fail=restart
				
			
				
					pcs cluster cib drbd_cfg
				
			
				
					pcs cluster cib-push drbd_cfg --config
				
			

Create resource for the use of DRBD

				
					pcs -f drbd_cfg resource create DrbdData ocf:linbit:drbd drbd_resource=drbd0 op monitor interval=60s

				
			
				
					pcs -f drbd_cfg resource promotable DrbdData promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true
				
			
				
					pcs cluster cib-push drbd_cfg --config
				
			

Create FILESYSTEM resource for the automated mount point

				
					pcs cluster cib fs_cfg
				
			
				
					pcs -f fs_cfg resource create DrbdFS Filesystem device="/dev/drbd0" directory="/vpbx_data" fstype="xfs" 
				
			
				
					pcs -f fs_cfg constraint colocation add DrbdFS with DrbdData-clone INFINITY with-rsc-role=Master 
				
			
				
					pcs -f fs_cfg constraint order promote DrbdData-clone then start DrbdFS
				
			
				
					pcs -f fs_cfg constraint colocation add DrbdFS with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order DrbdData-clone then DrbdFS
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

Stop and disable all services in both servers

				
					systemctl stop mariadb
				
			
				
					systemctl disable mariadb
				
			
				
					systemctl stop fail2ban
				
			
				
					systemctl disable fail2ban
				
			
				
					systemctl stop asterisk
				
			
				
					systemctl disable asterisk
				
			
				
					systemctl stop vpbx-monitor
				
			
				
					systemctl disable vpbx-monitor
				
			

Create resource for the use of MariaDB in Master

				
					mkdir /vpbx_data/mysql 
				
			
				
					mkdir /vpbx_data/mysql/data 
				
			
				
					cp -aR /var/lib/mysql/* /vpbx_data/mysql/data
				
			
				
					chown -R mysql:mysql /vpbx_data/mysql
				
			
				
					sed -i 's/var\/lib\/mysql/vpbx_data\/mysql\/data/g' /etc/mysql/mariadb.conf.d/50-server.cnf
				
			
				
					pcs resource create mariadb service:mariadb op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add mariadb with ClusterIP INFINITY

				
			
				
					pcs -f fs_cfg constraint order DrbdFS then mariadb
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

Change MariaDB Path on vitalpbx-slave

				
					sed -i 's/var\/lib\/mysql/vpbx_data\/mysql\/data/g' /etc/mysql/mariadb.conf.d/50-server.cnf
				
			

Path Asterisk service in both servers

				
					sed -i 's/RestartSec=10/RestartSec=1/g'  /usr/lib/systemd/system/asterisk.service
				
			
				
					sed -i 's/Wants=mariadb.service/#Wants=mariadb.service/g'  /usr/lib/systemd/system/asterisk.service
				
			
				
					sed -i 's/After=mariadb.service/#After=mariadb.service/g'  /usr/lib/systemd/system/asterisk.service
				
			

Create resource for Asterisk

				
					pcs resource create asterisk service:asterisk op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add asterisk with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order mariadb then asterisk
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs resource update asterisk op stop timeout=120s
				
			
				
					pcs resource update asterisk op start timeout=120s
				
			
				
					pcs resource update asterisk op restart timeout=120s
				
			

Copy folders and files the DRBD partition on the server1

				
					tar -zcvf var-asterisk.tgz /var/log/asterisk 
				
			
				
					tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk
				
			
				
					tar -zcvf var-lib-vitalpbx.tgz /var/lib/vitalpbx
				
			
				
					tar -zcvf etc-vitalpbx.tgz /etc/vitalpbx
				
			
				
					tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk
				
			
				
					tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk
				
			
				
					tar -zcvf etc-asterisk.tgz /etc/asterisk
				
			
				
					tar xvfz var-asterisk.tgz -C /vpbx_data
				
			
				
					tar xvfz var-lib-asterisk.tgz -C /vpbx_data
				
			
				
					tar xvfz var-lib-vitalpbx.tgz -C /vpbx_data
				
			
				
					tar xvfz etc-vitalpbx.tgz -C /vpbx_data
				
			
				
					tar xvfz usr-lib-asterisk.tgz -C /vpbx_data
				
			
				
					tar xvfz var-spool-asterisk.tgz -C /vpbx_data
				
			
				
					tar xvfz etc-asterisk.tgz -C /vpbx_data
				
			
				
					rm -rf /var/log/asterisk 
				
			
				
					rm -rf /var/lib/asterisk
				
			
				
					rm -rf /var/lib/vitalpbx 
				
			
				
					rm -rf /etc/vitalpbx 
				
			
				
					rm -rf /usr/lib/asterisk
				
			
				
					rm -rf /var/spool/asterisk
				
			
				
					rm -rf /etc/asterisk 
				
			
				
					ln -s /vpbx_data/var/log/asterisk /var/log/asterisk 
				
			
				
					ln -s /vpbx_data/var/lib/asterisk /var/lib/asterisk
				
			
				
					ln -s /vpbx_data/var/lib/vitalpbx /var/lib/vitalpbx 
				
			
				
					ln -s /vpbx_data/etc/vitalpbx /etc/vitalpbx 
				
			
				
					ln -s /vpbx_data/usr/lib/asterisk /usr/lib/asterisk 
				
			
				
					ln -s /vpbx_data/var/spool/asterisk /var/spool/asterisk 
				
			
				
					ln -s /vpbx_data/etc/asterisk /etc/asterisk
				
			
				
					rm -rf var-asterisk.tgz
				
			
				
					rm -rf var-lib-asterisk.tgz
				
			
				
					rm -rf var-lib-vitalpbx.tgz
				
			
				
					rm -rf etc-vitalpbx.tgz
				
			
				
					rm -rf usr-lib-asterisk.tgz
				
			
				
					rm -rf var-spool-asterisk.tgz
				
			
				
					rm -rf etc-asterisk.tgz
				
			

Configure symbolic links on the server-slave

				
					rm -rf /var/log/asterisk
				
			
				
					rm -rf /var/lib/asterisk
				
			
				
					rm -rf /var/lib/vitalpbx
				
			
				
					rm -rf /etc/vitalpbx
				
			
				
					rm -rf /usr/lib/asterisk
				
			
				
					rm -rf /var/spool/asterisk
				
			
				
					rm -rf /etc/asterisk
				
			
				
					ln -s /vpbx_data/var/log/asterisk /var/log/asterisk
				
			
				
					ln -s /vpbx_data/var/lib/asterisk /var/lib/asterisk
				
			
				
					ln -s /vpbx_data/var/lib/vitalpbx /var/lib/vitalpbx 
				
			
				
					ln -s /vpbx_data/etc/vitalpbx /etc/vitalpbx 
				
			
				
					ln -s /vpbx_data/usr/lib/asterisk /usr/lib/asterisk 
				
			
				
					ln -s /vpbx_data/var/spool/asterisk /var/spool/asterisk 
				
			
				
					ln -s /vpbx_data/etc/asterisk /etc/asterisk
				
			

Create VitalPBX Service

				
					pcs resource create vpbx-monitor service:vpbx-monitor op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add vpbx-monitor with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order asterisk then vpbx-monitor
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

Create fail2ban Service

				
					pcs resource create fail2ban service:fail2ban op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add fail2ban with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order asterisk then fail2ban
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

Initialize the services of corosync and pacemaker in server-slave

				
					systemctl restart corosync.service
				
			
				
					systemctl restart pacemaker.service

				
			

Note:

All configuration is stored in the file: /var/lib/pacemaker/cib/cib.xml

Show the Cluster Status

				
					pcs status resources
				
			
				
					  * ClusterIP  (ocf::heartbeat:IPaddr2):        Started vitalpbx-master.local
  * Clone Set: DrbdData-clone [DrbdData] (promotable):
    * Masters: [ vitalpbx-master.local ]
    * Slaves: [ vitalpbx-slave.local ]
  * DrbdFS      (ocf::heartbeat:Filesystem):     Started vitalpbx-master.local
  * mysql       (ocf::heartbeat:mysql):  Started vitalpbx-master.local
  * asterisk    (service:asterisk):      Started vitalpbx-master.local
  * vpbx-monitor        (service:vpbx-monitor):  Started vitalpbx-master.local
  * fail2ban    (service:fail2ban):      Started vitalpbx-master.local

				
			

Note:

Before doing any high availability testing, make sure that the data has finished syncing. To do this, use the cat/proc/drbd command.

4.10 - Bind Address

Managing the bind address also is critical if you use multple IP addresses on the same NIC [as, for example, when using a floating IP address in an HA cluster]. In that circumstance, asterisk has a rather nasty habit of listening for SIP/IAX on the virtual IP address but replying on the base address of the NIC causing phones/trunks to fail to register

In the Master server go to SETTINGS/PJSIP Settings and configure the Floating IP that we are going to use in “Bind” and “TLS Bind”. Also do it in SETTINGS/SIP Settings Tab NETWORK fields “TCP Bind Address” and “TLS Bind Address”.

4.11 - Create “bascul” command in both servers

The bascul command permanently moves services from one server to another. If you want to return the services to the main server you must execute the command again.

Download file

				
					wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/bascul
				
			

Or create file

				
					nano bascul
				
			
				
					#!/bin/bash
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 1-Agu-2023
# Change the status of the servers, the Master goes to Stanby and the Standby goes to Master.
#funtion for draw a progress bar
#You must pass as argument the amount of secconds that the progress bar will run
#progress-bar 10 --> it will generate a progress bar that will run per 10 seconds
set -e
progress-bar() {
        local duration=${1}
        already_done() { for ((done=0; done<$elapsed; done++)); do printf ">"; done }
        remaining() { for ((remain=$elapsed; remain<$duration; remain++)); do printf " "; done }
        percentage() { printf "| %s%%" $(( (($elapsed)*100)/($duration)*100/100 )); }
        clean_line() { printf "\r"; }
        for (( elapsed=1; elapsed<=$duration; elapsed++ )); do
                already_done; remaining; percentage
                sleep 1
                clean_line
        done
        clean_line
}
server_a=`pcs status | awk 'NR==11 {print $4}'`
server_b=`pcs status | awk 'NR==11 {print $5}'`
server_master=`pcs status resources | awk 'NR==1 {print $5}'`
#Perform some validations
if [ "${server_a}" = "" ] || [ "${server_b}" = "" ]
then
    echo -e "\e[41m There are problems with high availability, please check with the command *pcs status* (we recommend applying the command *pcs cluster unstandby* in both servers) \e[0m"
    exit;
fi
if [[ "${server_master}" = "${server_a}" ]]; then
        host_master=$server_a
        host_standby=$server_b
else
        host_master=$server_b
        host_standby=$server_a
fi
arg=$1
if [ "$arg" = 'yes' ] ;then
        perform_bascul='yes'
fi
# Print a warning message and ask to the user if he wants to continue
echo -e "************************************************************"
echo -e "*     Change the roles of servers in high availability     *"
echo -e "*\e[41m WARNING-WARNING-WARNING-WARNING-WARNING-WARNING-WARNING  \e[0m*"
echo -e "*All calls in progress will be lost and the system will be *"
echo -e "*     be in an unavailable state for a few seconds.        *"
echo -e "************************************************************"
#Perform a loop until the users confirm if wants to proceed or not
while [[ $perform_bascul != yes && $perform_bascul != no ]]; do
        read -p "Are you sure to switch from $host_master to $host_standby? (yes,no) > " perform_bascul
done
if [[ "${perform_bascul}" = "yes" ]]; then
        #Unstandby both nodes
        pcs node unstandby $host_master
        pcs node unstandby $host_standby
        #Do a loop per resource
        pcs status resources | awk '{print $2}' | while read -r resource ; do
                #Skip moving the ClusterIP resource, it will be moved at the end
                if [[ "${resource}" != "ClusterIP" ]] && [[ "${resource}" != "Clone" ]] && [[ "${resource}" != "Masters:" ]] && [[ "${resource}" != "Slaves:" ]]; then
                        echo "Moving ${resource} from ${host_master} to ${host_standby}"
                        pcs resource move ${resource} ${host_standby}
                fi
        done
        sleep 5 && pcs node standby $host_master & #Standby current Master node after five seconds
        sleep 20 && pcs node unstandby $host_master & #Automatically Unstandby current Master node after$
        #Move the ClusterIP resource to standby node
        echo "Moving ClusterIP from ${host_master} to ${host_standby}"
        pcs resource move ClusterIP ${host_standby}
        #End the script
        echo "Becoming ${host_standby} to Master"
        progress-bar 10
        echo "Done"
else
        echo "Nothing to do, bye, bye"
fi
sleep 15
role

				
			

Add permissions and move to folder /usr/local/bin

				
					chmod +x bascul
				
			
				
					mv bascul /usr/local/bin
				
			

4.12 - Create “role” command in both servers

Download file

				
					wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/role
				
			

Or create file

				
					nano role
				
			
				
					#!/bin/bash
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 10-oct-2022
# Show the Role of Server.
#Bash Colour Codes
green="\033[00;32m"
txtrst="\033[00;0m"
linux_ver=`cat /etc/os-release | grep -e PRETTY_NAME | awk -F '=' '{print $2}' | xargs`
vpbx_version=`aptitude versions vitalpbx | awk '{ print $2 }'`
server_master=`pcs status resources | awk 'NR==1 {print $5}'`
host=`hostname`
if [[ "${server_master}" = "${host}" ]]; then
        server_mode="Master"
else
        server_mode="Standby"
fi
logo='
 _    _ _           _ ______ ______ _    _
| |  | (_)_        | (_____ (____  \ \  / /
| |  | |_| |_  ____| |_____) )___)  ) \/ /
 \ \/ /| |  _)/ _  | |  ____/  __  ( )  (
  \  / | | |_( ( | | | |    | |__)  ) /\ \\
   \/  |_|\___)_||_|_|_|    |______/_/  \_\\
'
echo -e "
${green}
${logo}
${txtrst}
 Role           : $server_mode
 Version        : ${vpbx_version//[[:space:]]}
 Asterisk       : `asterisk -rx "core show version" 2>/dev/null| grep -ohe 'Asterisk [0-9.]*'`
 Linux Version  : ${linux_ver}
 Welcome to     : `hostname`
 Uptime         : `uptime | grep -ohe 'up .*' | sed 's/up //g' | awk -F "," '{print $1}'`
 Load           : `uptime | grep -ohe 'load average[s:][: ].*' | awk '{ print "Last Minute: " $3" Last 5 Minutes: "$4" Last 15 Minutes "$5 }'`
 Users          : `uptime | grep -ohe '[0-9.*] user[s,]'`
 IP Address     : ${green}`ip addr | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p' | xargs `${txtrst}
 Clock          :`timedatectl | sed -n '/Local time/ s/^[ \t]*Local time:\(.*$\)/\1/p'`
 NTP Sync.      :`timedatectl |awk -F: '/NTP service/ {print $2}'`
"
echo -e ""
echo -e "************************************************************"
echo -e "*                  Servers Status                          *"
echo -e "************************************************************"
echo -e "Master"
pcs status resources
echo -e ""
echo -e "Servers Status"
pcs cluster pcsd-status

				
			

Add permissions and copy to folder /etc/profile.d/

				
					cp -rf role /etc/profile.d/vitalwelcome.sh
				
			
				
					chmod 755 /etc/profile.d/vitalwelcome.sh
				
			

Now add permissions and move to folder /usr/local/bin

				
					chmod +x role
				
			
				
					mv role /usr/local/bin
				
			

4.13 - Create “drbdsplit” command in both servers

Split brain can be caused by intervention by cluster management software or human error during a period of failure for network links between cluster nodes, causing both nodes to switch to the primary role while disconnected.

Split brain occurs when both High Availability nodes switch into the primary role while disconnected. This behavior can allow data to be modified on either node without being replicated on the peer, leading to two diverging sets of data on each node, which can be difficult to merge.

Download file

				
					wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/drbdsplit
				
			

Or create file

				
					nano drbdsplit
				
			
				
					#!/bin/bash
set -e
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 1-Agu-2023
# DRBD split-brain recovery
#
drbdadm secondary drbd0
drbdadm disconnect drbd0
drbdadm -- --discard-my-data connect drbd0
echo "Disk Status"
drbdadm status

				
			

Add permissions and move to folder /usr/local/bin

				
					chmod +x drbdsplit
				
			
				
					mv drbdsplit /usr/local/bin
				
			

5 - Add New Services

5.1 - Add Sonata Switchboard

If we decide to install Sonata Switchboard, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install Sonata Switchboard

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install Sonata Switchboard

6.- Execute the following command in Master console

				
					bascul
				
			

5.2 - Add Sonata Stats

If we decide to install Sonata Stats, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install Sonata Stats

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install Sonata Stats

6.- Execute the following command in Master console

				
					bascul
				
			

7.- We added the Sonata Stats automation service.

On both servers stop and disable the sonata-stats service

				
					systemctl stop sonata-stats
				
			
				
					systemctl disable sonata-stats

				
			

Now in the server-master create the service

				
					pcs resource create sonata-stats service:sonata-stats op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg 
				
			
				
					pcs cluster cib-push fs_cfg --config


				
			
				
					pcs -f fs_cfg constraint colocation add sonata-stats with ClusterIP INFINITY 
				
			
				
					pcs -f fs_cfg constraint order vpbx-monitor then sonata-stats 
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

5.3.- Add Sonata Recording

If we decide to install Sonata Recording, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install Sonata Recording

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install Sonata Recording

6.- Execute the following command in Master console

				
					bascul
				
			

7.- We added the Sonata Recordings automation service.

On both servers stop and disable the recordings service

				
					systemctl stop recordings
				
			
				
					systemctl disable recordings
				
			

Now in the server-master create the service

				
					pcs resource create sonata-recordings service:recordings op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add recordings with ClusterIP INFINITY 
				
			
				
					pcs -f fs_cfg constraint order vpbx-monitor then recordings
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

5.4.- Add Sonata Billing

If we decide to install Sonata Billing, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install Sonata Billing

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install Sonata Billing

6.- Execute the following command in Master console

				
					bascul
				
			

5.5.- Add Sonata Dialer

If we decide to install Sonata Dialer, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install Sonata Dialer

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install Sonata Dialer

6.- Execute the following command in Master console

				
					bascul
				
			

7.- We added the Sonata Dialer automation service.

On both servers stop and disable the sonata-dialer service

				
					systemctl stop sonata-dialer
				
			
				
					systemctl disable sonata-dialer
				
			

Now in the server-master create the service

				
					pcs resource create sonata-dialer service:sonata-dialer op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add sonata-dialer with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order vpbx-monitor then sonata-dialer
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

5.6.- Add VitXi

If we decide to install VitXi, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install VitXi

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install VitXi

6.- Execute the following command in Master console

				
					bascul
				
			

7.- We added the VitXi automation service.

On both servers stop and disable the vitxi service

				
					systemctl stop vitxi
				
			
				
					systemctl disable vitxi
				
			

On the server-master change the location of the VitXi files and create a virtual link

				
					tar -zcvf vitxi-storage.tgz /usr/share/vitxi/backend/storage 
				
			
				
					tar xvfz vitxi-storage.tgz -C /vpbx_data
				
			
				
					rm -rf /usr/share/vitxi/backend/storage
				
			
				
					ln -s /vpbx_data/usr/share/vitxi/backend/storage  /usr/share/vitxi/backend/storage
				
			
				
					rm -rf vitxi-storage.tgz
				
			

On the server-slave create a virtual link

				
					rm -rf /usr/share/vitxi/backend/storage
				
			
				
					ln -s /vpbx_data/usr/share/vitxi/backend/storage  /usr/share/vitxi/backend/storage
				
			

Now in the server-master create the service

				
					pcs resource create vitxi service:vitxi op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add vitxi with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order vpbx-monitor then vitxi
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

5.7.- Add OpenVPN

If we decide to install OpenVPN, then we show the procedure.

1.- From your browser, go to ip 192.168.10.30

2.- Install OpenVPN

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Install OpenVPN

6.- Execute the following command in Master console

				
					bascul
				
			

7.- We added the OpenVPN automation service.

On both servers stop and disable the openvpn service

				
					systemctl stop openvpn
				
			
				
					systemctl disable openvpn
				
			

Now in the server-master create the service

				
					pcs resource create openvpn service:openvpn op monitor interval=30s
				
			
				
					pcs cluster cib fs_cfg
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			
				
					pcs -f fs_cfg constraint colocation add openvpn with ClusterIP INFINITY
				
			
				
					pcs -f fs_cfg constraint order vpbx-monitor then openvpn
				
			
				
					pcs cluster cib-push fs_cfg --config
				
			

6.- Test

To execute the process of changing the role, we recommend using the following command:

				
					bascul
				
			
				
					************************************************************
*     Change the roles of servers in high availability     *
* WARNING-WARNING-WARNING-WARNING-WARNING-WARNING-WARNING  *
*All calls in progress will be lost and the system will be *
*     be in an unavailable state for a few seconds.        *
************************************************************
Are you sure to switch from vitalpbx-master.local to vitalpbx-slave.local? (yes,no) >
				
			

This action convert the vitalpbx-master.local to Slave and vitalpbx-slave.local to Master. If you want to return to default do the same again.

If you want to know the current state of High Availability from a phone, please create the following context.

				
					nano /etc/asterisk/vitalpbx/extensions__61-ha-test.conf
				
			
				
					[cos-all](+)
exten => *777,1,NoOp(Say Asterisk IP Address)
 same => n,Answer()
 same => n,Set(ASTERISK_IP=${SHELL(/usr/bin/hostname -I | | awk -F " " '{print $1}'`)})
 same => n,NoOp(IP Address: ${ASTERISK_IP})
 same => n,SayDigits(${CUT(ASTERISK_IP,.,1)})
 same => n,Playback(letters/dot)
 same => n,SayDigits(${CUT(ASTERISK_IP,.,2)})
 same => n,Playback(letters/dot)
 same => n,SayDigits(${CUT(ASTERISK_IP,.,3)})
 same => n,Playback(letters/dot)
 same => n,SayDigits(${CUT(ASTERISK_IP,.,4)})
 same => n,PlayBack(silence/1&vm-goodbye)
 same => n,Hangup()
				
			

Now for the context to be used we must restart the Asterisk dial plan.

				
					asterisk -rx"dialplan reload"
				
			

To try just dial *777

7.- Turn off and turn on

Warning Note ⚠

When you must turn off the servers, when you turn it on always start with the Master, wait for the Master to start and then turn on the Slave.

8.- Update VitalPBX or Add-Ons

To update VitalPBX to the latest version just follow the following steps:

1.- From your browser, go to ip 192.168.10.30

2.- Update VitalPBX from the interface

3.- Execute the following command in Master console

				
					bascul
				
			

4.- From your browser, go to ip 192.168.10.30 again

5.- Update VitalPBX from the interface

6.- Execute the following command in Master console

				
					bascul
				
			

CONGRATULATIONS, you have installed and tested the high availability in VitalPBX

9.- Some useful commands

  • bascul, is used to change roles between high availability servers. If all is well, a confirmation question should appear if we wish to execute the action. The bascul command permanently moves services from one server to another. If you want to return the services to the main server you must execute the command again.
  • role, shows the status of the current server. If all is well you should return Masters or Slaves.
  • pcs resource refresh –full, to poll all resources even if the status is unknown, enter the following command.
  • pcs cluster unstandby host, in some cases the bascul command does not finish tilting, which causes one of the servers to be in standby (stop), with this command the state is restored to normal.
  • pcs resource delete, removes the resource so it can be created.
  • pcs resource create, create the resource.
  • corosync-cfgtool -s, to check whether cluster communication is happy.
  • ps axf, confirmed that Corosync is functional, we can check the rest of the stack. Pacemaker has already been started, so verify the necessary processes are running.
  • pcs status, check the pcs status output.
  • crm_verify -L -V, check the validity of the configuration.
  • drbdadm status, shows the integrity status of the disks that are being shared between both servers in high availability. If for some reason the status of Connecting or Standalone returns to us, wait a while and if the state remains it is because there are synchronization problems between both servers, and you should execute the drbdsplit command.
  • drbdsplit, solves DRBD split brain recovery.
  • cat /proc/drbd, the state of your device is kept in /proc/drbd.
  • drbdadm role drbd0, another way to check the role of the block device.
  • drbdadm primary drbd0, switch the DRBD block device to Primary using drbdadm.
  • drbdadm secondary drbd0, switch the DRBD block device to Secondary using drbdadm.
  • /usr/share/vitalpbx/ha/ ./vpbxha.sh, create the cluster automatically.
  • /usr/share/vitalpbx/ha/ ./destroy.sh, completely destroy the cluster, leaving the DRBD intact.
  • /usr/share/vitalpbx/ha/ ./rebuild.sh, recreates the cluster starting from the fact that the DRBD

10.- Credits

10.1 Sources of Information

Our Latest Post