1. Introduction
High availability is a characteristic of a system which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period.
Make a high-availability cluster out of any pair of VitalPBX servers. VitalPBX can detect a range of failures on one VitalPBX server and automatically transfer control to the other server, resulting in a telephony environment with minimal down time.
2. Requirements to install VitalPBX in High Availability
Setting up High Availability (HA) with DRBD (Distributed Replicated Block Device) involves ensuring critical systems and services are available with minimal downtime in case of failures. DRBD enables real-time data replication between nodes to ensure data availability and integrity. Here are some general requirements to set up High Availability with DRBD:
1.- Physical Infrastructure and Networking:
Two or more identical or very similar nodes (servers) to implement redundancy.
Reliable and low-latency network connection between the nodes. This can be a dedicated replication network (preferred) or a shared network if it’s of high quality.
2.- Operating System – VitalPBX version:
Nodes should run the same operating system, preferably the same version and the same version of VitalPBX.
3.- Disk Partitioning:
The storage device to be replicated should be partitioned and accessible on both nodes.
Each node should have sufficient storage space to accommodate replication and data.
4.- Network Configuration:
Each node should have static IP addresses and resolve correctly in the local DNS system or in the /etc/hosts file of the other node.
Hostnames should be consistent across nodes.
5.- DRBD:
Install and configure DRBD on both nodes.
Configure DRBD resources that define which devices will be replicated and how replication will be established.
At the time of installation leave the largest amount of space on the hard drive to store the variable data on both servers.
Define node roles: primary and secondary nodes.
3. Installation
We are going to start by installing VitalPBX on two servers
a.- When you get to the partitions part you must select “Guide – use entire disk”:
b.- Select:
All file in one partition (recommended for new user)
c.- Select primary:
#1 primary
d.- Delete partition
e.- Select pri/log
f.- Create New Partition
g.- Change the capacity to:
New partition size: 20GB
We need enough space for the operating system and its applications in the future; then <continue>
Select Primary
Select Beginning
Select Done setting up the partition
h.- Finish
Pres Yes to confirm
And continue with the installation.
4. Configurations
4.1 IP and Hostname Configuration.
We will configure in each server the IP address and the host name.
Name | Master | Slave |
Hostname | vitalpbx-master.local | vitalpbx-slave.local |
IP Address | 192.168.10.31 | 192.168.10.32 |
Netmask | 255.255.254.0 | 255.255.254.0 |
Gateway | 192.168.10.1 | 192.168.10.1 |
Primary DNS | 8.8.8.8 | 8.8.8.8 |
Secondary DNS | 8.8.4.4 | 8.8.4.4 |
4.1.1 Remote access with the root user.
As a security measure, the root user is not allowed to remote access the server. If you wish to unblock it, remember to set a complex password, and perform the following steps.
Enter the Command Line Console with the root user directly on your server with the password you set up earlier.
Edit the following file using nano on both servers, /etc/ssh/sshd_config.
nano /etc/ssh/sshd_config
Change the following line
#PermitRootLogin prohibit-password
With
PermitRootLogIn yes
Save, Exit and restart the sshd service.
systemctl restart sshd
4.1.2 Changing your IP Address to a static address.
By default, our system’s IP address in Debian is obtained through DHCP, however, you can modify it and assign it a static IP address using the following steps:
Edit the following file with nano on both servers, /etc/network/interfaces.
nano /etc/network/interfaces
Change
#The primary network interface
allow-hotplug eth0
iface eth0 inet dchp
With the following. Server Master.
#The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.10.31
netmask 255.255.254.0
gateway 192.168.10.1
With the following. Server Slave.
#The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.10.32
netmask 255.255.254.0
gateway 192.168.10.1
Lastly, reboot both servers and you can now log in via ssh.
4.2 Install Dependencies
Install dependencies on both servers
apt -y install drbd-utils corosync pacemaker pcs chrony xfsprogs
4.3 Hostname
Go to the web interface to:
ADMIN>System Settings>Network Settings
First, change the Hostname in both servers, and remember to press the Save button.
Master
Slave
Now we connect through ssh to each of the servers and we configure the hostname of each server in the /etc/hosts file, so that both servers see each other with the hostname.
Server Master
hostname vitalpbx-master.local
Server Slave
hostname vitalpbx-slave.local
Server Master and Slave
nano /etc/hosts
Add the following lines on both servers
192.168.10.31 vitalpbx-master.local
192.168.10.32 vitalpbx-slave.local
4.4.- Create the partition on both servers
Initialize the partition to allocate the available space on the hard disk. Do these on both servers.
fdisk /dev/sda
Command (m for help): n
Partition type:
p primary (3 primary, 0 extended, 1 free)
e extended
Select (default e): p
Selected partition 3 (take note of the assigned partition number as we will need it later)
First sector (35155968-266338303, default 35155968): [Enter]
Last sector, +sectors or +size{K,M,G} (35155968-266338303, default 266338303): [Enter]
Using default value 266338303
Partition 4 of type Linux and of size 110.2 GiB is set
Command (m for help): t
Partition number (1-4, default 4): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): w
Then, restart the servers so that the new table is available.
reboot
4.5.- Using Script
We have two ways to configure the Cluster. Using the following Script or following this manual step by step. If you decide to use the following Script, the next step you should follow in this manual is 5 if you consider it necessary. Otherwise continue with step 4.6.
Create authorization key for the Access between the two servers without credentials.
Create key in Server Master
ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' >/dev/null
ssh-copy-id root@192.168.10.32
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
root@192.168.10.62's password: (remote server root’s password)
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@192.168.10.32'"
and check to make sure that only the key(s) you wanted were added.
root@vitalpbx-master:~#
Create key in Server Slave
ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' >/dev/null
ssh-copy-id root@192.168.10.31
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
root@192.168.10.61's password: (remote server root’s password)
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@192.168.10.31'"
and check to make sure that only the key(s) you wanted were added.
root@vitalpbx-slave:~#
Now copy and run the following script in Master Server.
mkdir /usr/share/vitalpbx/ha
cd /usr/share/vitalpbx/ha
wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/vpbxha.sh
chmod +x vpbxha.sh
./vpbxha.sh
Now we enter all the information requested.
************************************************************
* Welcome to the VitalPBX high availability installation *
* All options are mandatory *
************************************************************
IP Server1............... > 192.168.10.31
IP Server2............... > 192.168.10.32
Floating IP.............. > 192.168.10.30
Floating IP Mask (SIDR).. > 24
Disk (sdax).............. > sda3
hacluster password....... > MyPassword
************************************************************
* Check Information *
* Make sure you have internet on both servers *
************************************************************
Are you sure to continue with this settings? (yes,no) > yes
CONGRATULATIONS, you have installed high availability in VitalPBX 4
4.6 Firewall
Configure the Firewall (Both Servers)
Port | Description |
TCP 2224 | Required on all nodes (needed by the pcsd Web UI and required for node-to-node communication) It is crucial to open port 2224 in such a way that pcs from any node can talk to all nodes in the cluster, including itself. When using the Booth cluster ticket manager or a quorum device you must open port 2224 on all related hosts, such as Booth arbiters or the quorum device host. |
TCP 3121 | Required on all nodes if the cluster has any Pacemaker Remote nodes Pacemaker’s crmd daemon on the full cluster nodes will contact the pacemaker_remoted daemon on Pacemaker Remote nodes at port 3121. If a separate interface is used for cluster communication, the port only needs to be open on that interface. At a minimum, the port should open on Pacemaker Remote nodes to full cluster nodes. Because users may convert a host between a full node and a remote node, or run a remote node inside a container using the host’s network, it can be useful to open the port to all nodes. It is not necessary to open the port to any hosts other than nodes. |
TCP 5403 | Required on the quorum device host when using a quorum device with corosync-qnetd. The default value can be changed with the -p option of the corosync-qnetd command. |
UDP 5404 | Required on corosync nodes if corosync is configured for multicast UDP.
|
UDP 5405 | Required on all corosync nodes (needed by corosync) |
TCP 21064 | Required on all nodes if the cluster contains any resources requiring DLM (such as clvm or GFS2) |
TCP 9929, UDP 9929 | Required to be open on all cluster nodes and booth arbitrator nodes to connections from any of those same nodes when the Booth ticket manager is used to establish a multi-site cluster. |
TCP 7789 | Required by DRBD to synchronize information. |
Add in both Servers the Service and Rule with this Service.
Add the Service in ADMIN/Firewall/Services Add Service
We add the Rule in /ADMIN/Firewall/Rules Add Rules
And we apply changes
4.7 Format the partition
Now, we will proceed to format the new partition in both servers with the following command:
mke2fs -j /dev/sda3
dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync
4.8 Configuring DRBD
Load the module and enable the service on both nodes, using the follow command:
modprobe drbd
systemctl enable drbd.service
Create a new global_common.conf file on both servers with the following contents:
mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.orig
nano /etc/drbd.d/global_common.conf
global {
usage-count no;
}
common {
net {
protocol C;
}
}
Next, we will need to create a new configuration file called /etc/drbd.d/drbd0.res for the new resource named drbd0, with the following contents on both servers:
nano /etc/drbd.d/drbd0.res
resource drbd0 {
startup {
wfc-timeout 5;
outdated-wfc-timeout 3;
degr-wfc-timeout 3;
outdated-wfc-timeout 2;
}
syncer {
rate 10M;
verify-alg md5;
}
net {
after-sb-0pri discard-older-primary;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
}
handlers {
pri-lost-after-sb "/sbin/reboot";
}
on vitalpbx-master.locall {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.10.31:7789;
meta-disk internal;
}
on vitalpbx-slave.local {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.10.32:7789;
meta-disk internal;
}
}
Note:
Although the access interfaces can be used, which in this case is ETH0. It is recommended to use an interface (ETH1) for synchronization, this interface must be directly connected between both servers.
Initialize the meta data storage on each nodes by executing the following command on both servers
drbdadm create-md drbd0
Writing meta data...
New drbd meta data block successfully created.
Let’s define the DRBD Primary node as first node “vitalpbx-master”
drbdadm up drbd0
drbdadm primary drbd0 --force
On the Secondary node “vitalpbx-slave” run the following command to start the drbd0
drbdadm up drbd0
You can check the current status of the synchronization while it’s being performed. The cat /proc/drbd command displays the creation and synchronization progress of the resource, as shown here:
4.8.1.- Formatting and Test DRBD Disk
Create the directory where we are going to mount the volume with all the information to be replicated on both servers.
mkdir /vpbx_data
In order to test the DRBD functionality we need to Create a file system, mount the volume and write some data on primary server “vitalpbx-master” and finally switch the primary node to “vitalpbx-slave”
Run the following command on the primary server to create an xfs filesystem on /dev/drbd0 and mount it to the vpbx_data directory, using the following commands in vitalpbx-master.
mkfs.xfs /dev/drbd0
mount /dev/drbd0 /vpbx_data
Create some data using the following command on vitalpbx-master:
touch /vpbx_data/file{1..5}
ls -l /vpbx_data
-rw-r--r-- 1 root root 0 Nov 17 11:28 file1
-rw-r--r-- 1 root root 0 Nov 17 11:28 file2
-rw-r--r-- 1 root root 0 Nov 17 11:28 file3
-rw-r--r-- 1 root root 0 Nov 17 11:28 file4
-rw-r--r-- 1 root root 0 Nov 17 11:28 file5
Let’s now switch primary mode “vitalpbx-server” to second node “vitalpbx-slave” to check the data replication works or not.
First, we have to unmount the volume drbd0 on the first drbd cluster node “vitalpbx-master” and change the primary node to secondary node on the first drbd cluster node “vitalpbx-master”
umount /vpbx_data
drbdadm secondary drbd0
Change the secondary node to primary node on the second drbd cluster node “vitalpbx-slave”
drbdadm primary drbd0 --force
Mount the volume and check the data available or not.
mount /dev/drbd0 /vpbx_data
ls -l /vpbx_data
total 0
-rw-r--r-- 1 root root 0 Nov 17 11:28 file1
-rw-r--r-- 1 root root 0 Nov 17 11:28 file2
-rw-r--r-- 1 root root 0 Nov 17 11:28 file3
-rw-r--r-- 1 root root 0 Nov 17 11:28 file4
-rw-r--r-- 1 root root 0 Nov 17 11:28 file5
Normalize Server-Slave
umount /vpbx_data
drbdadm secondary drbd0
Normalize Server-Master
drbdadm primary drbd0
mount /dev/drbd0 /vpbx_data
4.9 - Configure Cluster
Create the password of the hacluster user on both nodes
echo hacluster:Mypassword | chpasswd
Start PCS on both servers
systemctl start pcsd
Configure the start of services on both nodes
systemctl enable pcsd.service
systemctl enable corosync.service
systemctl enable pacemaker.service
Server Authenticate in Master
On vitalpbx-master, use pcs cluster auth to authenticate as the hacluster user:
pcs cluster destroy
pcs host auth vitalpbx-master.local vitalpbx-slave.local -u hacluster -p Mypassword
vitalpbx-master.local: Authorized
vitalpbx-slave.local: Authorized
Next, use pcs cluster setup on the vitalpbx-master to generate and synchronize the corosync configuration.
pcs cluster setup cluster_vitalpbx vitalpbx-master.local vitalpbx-slave.local --force
Starting Cluster in Master
pcs cluster start --all
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
Prevent Resources from Moving after Recovery
In most circumstances, it is highly desirable to prevent healthy resources from being moved around the cluster. Moving resources almost always requires a period of downtime. For complex services such as databases, this period can be quite long.
pcs resource defaults update resource-stickiness=INFINITY
Create resource for the use of Floating IP
pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.10.30 cidr_netmask=24 op monitor interval=30s on-fail=restart
pcs cluster cib drbd_cfg
pcs cluster cib-push drbd_cfg --config
Create resource for the use of DRBD
pcs -f drbd_cfg resource create DrbdData ocf:linbit:drbd drbd_resource=drbd0 op monitor interval=60s
pcs -f drbd_cfg resource promotable DrbdData promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs cluster cib-push drbd_cfg --config
Create FILESYSTEM resource for the automated mount point
pcs cluster cib fs_cfg
pcs -f fs_cfg resource create DrbdFS Filesystem device="/dev/drbd0" directory="/vpbx_data" fstype="xfs"
pcs -f fs_cfg constraint colocation add DrbdFS with DrbdData-clone INFINITY with-rsc-role=Master
pcs -f fs_cfg constraint order promote DrbdData-clone then start DrbdFS
pcs -f fs_cfg constraint colocation add DrbdFS with ClusterIP INFINITY
pcs -f fs_cfg constraint order DrbdData-clone then DrbdFS
pcs cluster cib-push fs_cfg --config
Stop and disable all services in both servers
systemctl stop mariadb
systemctl disable mariadb
systemctl stop fail2ban
systemctl disable fail2ban
systemctl stop asterisk
systemctl disable asterisk
systemctl stop vpbx-monitor
systemctl disable vpbx-monitor
Create resource for the use of MariaDB in Master
mkdir /vpbx_data/mysql
mkdir /vpbx_data/mysql/data
cp -aR /var/lib/mysql/* /vpbx_data/mysql/data
chown -R mysql:mysql /vpbx_data/mysql
sed -i 's/var\/lib\/mysql/vpbx_data\/mysql\/data/g' /etc/mysql/mariadb.conf.d/50-server.cnf
pcs resource create mariadb service:mariadb op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add mariadb with ClusterIP INFINITY
pcs -f fs_cfg constraint order DrbdFS then mariadb
pcs cluster cib-push fs_cfg --config
Change MariaDB Path on vitalpbx-slave
sed -i 's/var\/lib\/mysql/vpbx_data\/mysql\/data/g' /etc/mysql/mariadb.conf.d/50-server.cnf
Path Asterisk service in both servers
sed -i 's/RestartSec=10/RestartSec=1/g' /usr/lib/systemd/system/asterisk.service
sed -i 's/Wants=mariadb.service/#Wants=mariadb.service/g' /usr/lib/systemd/system/asterisk.service
sed -i 's/After=mariadb.service/#After=mariadb.service/g' /usr/lib/systemd/system/asterisk.service
Create resource for Asterisk
pcs resource create asterisk service:asterisk op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add asterisk with ClusterIP INFINITY
pcs -f fs_cfg constraint order mariadb then asterisk
pcs cluster cib-push fs_cfg --config
pcs resource update asterisk op stop timeout=120s
pcs resource update asterisk op start timeout=120s
pcs resource update asterisk op restart timeout=120s
Copy folders and files the DRBD partition on the server1
tar -zcvf var-asterisk.tgz /var/log/asterisk
tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk
tar -zcvf var-lib-vitalpbx.tgz /var/lib/vitalpbx
tar -zcvf etc-vitalpbx.tgz /etc/vitalpbx
tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk
tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk
tar -zcvf etc-asterisk.tgz /etc/asterisk
tar xvfz var-asterisk.tgz -C /vpbx_data
tar xvfz var-lib-asterisk.tgz -C /vpbx_data
tar xvfz var-lib-vitalpbx.tgz -C /vpbx_data
tar xvfz etc-vitalpbx.tgz -C /vpbx_data
tar xvfz usr-lib-asterisk.tgz -C /vpbx_data
tar xvfz var-spool-asterisk.tgz -C /vpbx_data
tar xvfz etc-asterisk.tgz -C /vpbx_data
rm -rf /var/log/asterisk
rm -rf /var/lib/asterisk
rm -rf /var/lib/vitalpbx
rm -rf /etc/vitalpbx
rm -rf /usr/lib/asterisk
rm -rf /var/spool/asterisk
rm -rf /etc/asterisk
ln -s /vpbx_data/var/log/asterisk /var/log/asterisk
ln -s /vpbx_data/var/lib/asterisk /var/lib/asterisk
ln -s /vpbx_data/var/lib/vitalpbx /var/lib/vitalpbx
ln -s /vpbx_data/etc/vitalpbx /etc/vitalpbx
ln -s /vpbx_data/usr/lib/asterisk /usr/lib/asterisk
ln -s /vpbx_data/var/spool/asterisk /var/spool/asterisk
ln -s /vpbx_data/etc/asterisk /etc/asterisk
rm -rf var-asterisk.tgz
rm -rf var-lib-asterisk.tgz
rm -rf var-lib-vitalpbx.tgz
rm -rf etc-vitalpbx.tgz
rm -rf usr-lib-asterisk.tgz
rm -rf var-spool-asterisk.tgz
rm -rf etc-asterisk.tgz
Configure symbolic links on the server-slave
rm -rf /var/log/asterisk
rm -rf /var/lib/asterisk
rm -rf /var/lib/vitalpbx
rm -rf /etc/vitalpbx
rm -rf /usr/lib/asterisk
rm -rf /var/spool/asterisk
rm -rf /etc/asterisk
ln -s /vpbx_data/var/log/asterisk /var/log/asterisk
ln -s /vpbx_data/var/lib/asterisk /var/lib/asterisk
ln -s /vpbx_data/var/lib/vitalpbx /var/lib/vitalpbx
ln -s /vpbx_data/etc/vitalpbx /etc/vitalpbx
ln -s /vpbx_data/usr/lib/asterisk /usr/lib/asterisk
ln -s /vpbx_data/var/spool/asterisk /var/spool/asterisk
ln -s /vpbx_data/etc/asterisk /etc/asterisk
Create VitalPBX Service
pcs resource create vpbx-monitor service:vpbx-monitor op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add vpbx-monitor with ClusterIP INFINITY
pcs -f fs_cfg constraint order asterisk then vpbx-monitor
pcs cluster cib-push fs_cfg --config
Create fail2ban Service
pcs resource create fail2ban service:fail2ban op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add fail2ban with ClusterIP INFINITY
pcs -f fs_cfg constraint order asterisk then fail2ban
pcs cluster cib-push fs_cfg --config
Initialize the services of corosync and pacemaker in server-slave
systemctl restart corosync.service
systemctl restart pacemaker.service
Note:
All configuration is stored in the file: /var/lib/pacemaker/cib/cib.xml
Show the Cluster Status
pcs status resources
* ClusterIP (ocf::heartbeat:IPaddr2): Started vitalpbx-master.local
* Clone Set: DrbdData-clone [DrbdData] (promotable):
* Masters: [ vitalpbx-master.local ]
* Slaves: [ vitalpbx-slave.local ]
* DrbdFS (ocf::heartbeat:Filesystem): Started vitalpbx-master.local
* mysql (ocf::heartbeat:mysql): Started vitalpbx-master.local
* asterisk (service:asterisk): Started vitalpbx-master.local
* vpbx-monitor (service:vpbx-monitor): Started vitalpbx-master.local
* fail2ban (service:fail2ban): Started vitalpbx-master.local
Note:
Before doing any high availability testing, make sure that the data has finished syncing. To do this, use the cat/proc/drbd command.
4.10 - Bind Address
Managing the bind address also is critical if you use multple IP addresses on the same NIC [as, for example, when using a floating IP address in an HA cluster]. In that circumstance, asterisk has a rather nasty habit of listening for SIP/IAX on the virtual IP address but replying on the base address of the NIC causing phones/trunks to fail to register
In the Master server go to SETTINGS/PJSIP Settings and configure the Floating IP that we are going to use in “Bind” and “TLS Bind”. Also do it in SETTINGS/SIP Settings Tab NETWORK fields “TCP Bind Address” and “TLS Bind Address”.
4.11 - Create “bascul” command in both servers
The bascul command permanently moves services from one server to another. If you want to return the services to the main server you must execute the command again.
Download file
wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/bascul
Or create file
nano bascul
#!/bin/bash
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 1-Agu-2023
# Change the status of the servers, the Master goes to Stanby and the Standby goes to Master.
#funtion for draw a progress bar
#You must pass as argument the amount of secconds that the progress bar will run
#progress-bar 10 --> it will generate a progress bar that will run per 10 seconds
set -e
progress-bar() {
local duration=${1}
already_done() { for ((done=0; done<$elapsed; done++)); do printf ">"; done }
remaining() { for ((remain=$elapsed; remain<$duration; remain++)); do printf " "; done }
percentage() { printf "| %s%%" $(( (($elapsed)*100)/($duration)*100/100 )); }
clean_line() { printf "\r"; }
for (( elapsed=1; elapsed<=$duration; elapsed++ )); do
already_done; remaining; percentage
sleep 1
clean_line
done
clean_line
}
server_a=`pcs status | awk 'NR==11 {print $4}'`
server_b=`pcs status | awk 'NR==11 {print $5}'`
server_master=`pcs status resources | awk 'NR==1 {print $5}'`
#Perform some validations
if [ "${server_a}" = "" ] || [ "${server_b}" = "" ]
then
echo -e "\e[41m There are problems with high availability, please check with the command *pcs status* (we recommend applying the command *pcs cluster unstandby* in both servers) \e[0m"
exit;
fi
if [[ "${server_master}" = "${server_a}" ]]; then
host_master=$server_a
host_standby=$server_b
else
host_master=$server_b
host_standby=$server_a
fi
arg=$1
if [ "$arg" = 'yes' ] ;then
perform_bascul='yes'
fi
# Print a warning message and ask to the user if he wants to continue
echo -e "************************************************************"
echo -e "* Change the roles of servers in high availability *"
echo -e "*\e[41m WARNING-WARNING-WARNING-WARNING-WARNING-WARNING-WARNING \e[0m*"
echo -e "*All calls in progress will be lost and the system will be *"
echo -e "* be in an unavailable state for a few seconds. *"
echo -e "************************************************************"
#Perform a loop until the users confirm if wants to proceed or not
while [[ $perform_bascul != yes && $perform_bascul != no ]]; do
read -p "Are you sure to switch from $host_master to $host_standby? (yes,no) > " perform_bascul
done
if [[ "${perform_bascul}" = "yes" ]]; then
#Unstandby both nodes
pcs node unstandby $host_master
pcs node unstandby $host_standby
#Do a loop per resource
pcs status resources | awk '{print $2}' | while read -r resource ; do
#Skip moving the ClusterIP resource, it will be moved at the end
if [[ "${resource}" != "ClusterIP" ]] && [[ "${resource}" != "Clone" ]] && [[ "${resource}" != "Masters:" ]] && [[ "${resource}" != "Slaves:" ]]; then
echo "Moving ${resource} from ${host_master} to ${host_standby}"
pcs resource move ${resource} ${host_standby}
fi
done
sleep 5 && pcs node standby $host_master & #Standby current Master node after five seconds
sleep 20 && pcs node unstandby $host_master & #Automatically Unstandby current Master node after$
#Move the ClusterIP resource to standby node
echo "Moving ClusterIP from ${host_master} to ${host_standby}"
pcs resource move ClusterIP ${host_standby}
#End the script
echo "Becoming ${host_standby} to Master"
progress-bar 10
echo "Done"
else
echo "Nothing to do, bye, bye"
fi
sleep 15
role
Add permissions and move to folder /usr/local/bin
chmod +x bascul
mv bascul /usr/local/bin
4.12 - Create “role” command in both servers
Download file
wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/role
Or create file
nano role
#!/bin/bash
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 10-oct-2022
# Show the Role of Server.
#Bash Colour Codes
green="\033[00;32m"
txtrst="\033[00;0m"
linux_ver=`cat /etc/os-release | grep -e PRETTY_NAME | awk -F '=' '{print $2}' | xargs`
vpbx_version=`aptitude versions vitalpbx | awk '{ print $2 }'`
server_master=`pcs status resources | awk 'NR==1 {print $5}'`
host=`hostname`
if [[ "${server_master}" = "${host}" ]]; then
server_mode="Master"
else
server_mode="Standby"
fi
logo='
_ _ _ _ ______ ______ _ _
| | | (_)_ | (_____ (____ \ \ / /
| | | |_| |_ ____| |_____) )___) ) \/ /
\ \/ /| | _)/ _ | | ____/ __ ( ) (
\ / | | |_( ( | | | | | |__) ) /\ \\
\/ |_|\___)_||_|_|_| |______/_/ \_\\
'
echo -e "
${green}
${logo}
${txtrst}
Role : $server_mode
Version : ${vpbx_version//[[:space:]]}
Asterisk : `asterisk -rx "core show version" 2>/dev/null| grep -ohe 'Asterisk [0-9.]*'`
Linux Version : ${linux_ver}
Welcome to : `hostname`
Uptime : `uptime | grep -ohe 'up .*' | sed 's/up //g' | awk -F "," '{print $1}'`
Load : `uptime | grep -ohe 'load average[s:][: ].*' | awk '{ print "Last Minute: " $3" Last 5 Minutes: "$4" Last 15 Minutes "$5 }'`
Users : `uptime | grep -ohe '[0-9.*] user[s,]'`
IP Address : ${green}`ip addr | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p' | xargs `${txtrst}
Clock :`timedatectl | sed -n '/Local time/ s/^[ \t]*Local time:\(.*$\)/\1/p'`
NTP Sync. :`timedatectl |awk -F: '/NTP service/ {print $2}'`
"
echo -e ""
echo -e "************************************************************"
echo -e "* Servers Status *"
echo -e "************************************************************"
echo -e "Master"
pcs status resources
echo -e ""
echo -e "Servers Status"
pcs cluster pcsd-status
Add permissions and copy to folder /etc/profile.d/
cp -rf role /etc/profile.d/vitalwelcome.sh
chmod 755 /etc/profile.d/vitalwelcome.sh
Now add permissions and move to folder /usr/local/bin
chmod +x role
mv role /usr/local/bin
4.13 - Create “drbdsplit” command in both servers
Split brain can be caused by intervention by cluster management software or human error during a period of failure for network links between cluster nodes, causing both nodes to switch to the primary role while disconnected.
Split brain occurs when both High Availability nodes switch into the primary role while disconnected. This behavior can allow data to be modified on either node without being replicated on the peer, leading to two diverging sets of data on each node, which can be difficult to merge.
Download file
wget https://raw.githubusercontent.com/VitalPBX/vitalpbx4_drbd_ha/main/drbdsplit
Or create file
nano drbdsplit
#!/bin/bash
set -e
# This code is the property of VitalPBX LLC Company
# License: Proprietary
# Date: 1-Agu-2023
# DRBD split-brain recovery
#
drbdadm secondary drbd0
drbdadm disconnect drbd0
drbdadm -- --discard-my-data connect drbd0
echo "Disk Status"
drbdadm status
Add permissions and move to folder /usr/local/bin
chmod +x drbdsplit
mv drbdsplit /usr/local/bin
5 - Add New Services
5.1 - Add Sonata Switchboard
If we decide to install Sonata Switchboard, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install Sonata Switchboard
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install Sonata Switchboard
6.- Execute the following command in Master console
bascul
5.2 - Add Sonata Stats
If we decide to install Sonata Stats, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install Sonata Stats
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install Sonata Stats
6.- Execute the following command in Master console
bascul
7.- We added the Sonata Stats automation service.
On both servers stop and disable the sonata-stats service
systemctl stop sonata-stats
systemctl disable sonata-stats
Now in the server-master create the service
pcs resource create sonata-stats service:sonata-stats op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add sonata-stats with ClusterIP INFINITY
pcs -f fs_cfg constraint order vpbx-monitor then sonata-stats
pcs cluster cib-push fs_cfg --config
5.3.- Add Sonata Recording
If we decide to install Sonata Recording, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install Sonata Recording
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install Sonata Recording
6.- Execute the following command in Master console
bascul
7.- We added the Sonata Recordings automation service.
On both servers stop and disable the recordings service
systemctl stop recordings
systemctl disable recordings
Now in the server-master create the service
pcs resource create sonata-recordings service:recordings op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add recordings with ClusterIP INFINITY
pcs -f fs_cfg constraint order vpbx-monitor then recordings
pcs cluster cib-push fs_cfg --config
5.4.- Add Sonata Billing
If we decide to install Sonata Billing, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install Sonata Billing
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install Sonata Billing
6.- Execute the following command in Master console
bascul
5.5.- Add Sonata Dialer
If we decide to install Sonata Dialer, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install Sonata Dialer
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install Sonata Dialer
6.- Execute the following command in Master console
bascul
7.- We added the Sonata Dialer automation service.
On both servers stop and disable the sonata-dialer service
systemctl stop sonata-dialer
systemctl disable sonata-dialer
Now in the server-master create the service
pcs resource create sonata-dialer service:sonata-dialer op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add sonata-dialer with ClusterIP INFINITY
pcs -f fs_cfg constraint order vpbx-monitor then sonata-dialer
pcs cluster cib-push fs_cfg --config
5.6.- Add VitXi
If we decide to install VitXi, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install VitXi
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install VitXi
6.- Execute the following command in Master console
bascul
7.- We added the VitXi automation service.
On both servers stop and disable the vitxi service
systemctl stop vitxi
systemctl disable vitxi
On the server-master change the location of the VitXi files and create a virtual link
tar -zcvf vitxi-storage.tgz /usr/share/vitxi/backend/storage
tar xvfz vitxi-storage.tgz -C /vpbx_data
rm -rf /usr/share/vitxi/backend/storage
ln -s /vpbx_data/usr/share/vitxi/backend/storage /usr/share/vitxi/backend/storage
rm -rf vitxi-storage.tgz
On the server-slave create a virtual link
rm -rf /usr/share/vitxi/backend/storage
ln -s /vpbx_data/usr/share/vitxi/backend/storage /usr/share/vitxi/backend/storage
Now in the server-master create the service
pcs resource create vitxi service:vitxi op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add vitxi with ClusterIP INFINITY
pcs -f fs_cfg constraint order vpbx-monitor then vitxi
pcs cluster cib-push fs_cfg --config
5.7.- Add OpenVPN
If we decide to install OpenVPN, then we show the procedure.
1.- From your browser, go to ip 192.168.10.30
2.- Install OpenVPN
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Install OpenVPN
6.- Execute the following command in Master console
bascul
7.- We added the OpenVPN automation service.
On both servers stop and disable the openvpn service
systemctl stop openvpn
systemctl disable openvpn
Now in the server-master create the service
pcs resource create openvpn service:openvpn op monitor interval=30s
pcs cluster cib fs_cfg
pcs cluster cib-push fs_cfg --config
pcs -f fs_cfg constraint colocation add openvpn with ClusterIP INFINITY
pcs -f fs_cfg constraint order vpbx-monitor then openvpn
pcs cluster cib-push fs_cfg --config
6.- Test
To execute the process of changing the role, we recommend using the following command:
bascul
************************************************************
* Change the roles of servers in high availability *
* WARNING-WARNING-WARNING-WARNING-WARNING-WARNING-WARNING *
*All calls in progress will be lost and the system will be *
* be in an unavailable state for a few seconds. *
************************************************************
Are you sure to switch from vitalpbx-master.local to vitalpbx-slave.local? (yes,no) >
This action convert the vitalpbx-master.local to Slave and vitalpbx-slave.local to Master. If you want to return to default do the same again.
If you want to know the current state of High Availability from a phone, please create the following context.
nano /etc/asterisk/vitalpbx/extensions__61-ha-test.conf
[cos-all](+)
exten => *777,1,NoOp(Say Asterisk IP Address)
same => n,Answer()
same => n,Set(ASTERISK_IP=${SHELL(/usr/bin/hostname -I | | awk -F " " '{print $1}'`)})
same => n,NoOp(IP Address: ${ASTERISK_IP})
same => n,SayDigits(${CUT(ASTERISK_IP,.,1)})
same => n,Playback(letters/dot)
same => n,SayDigits(${CUT(ASTERISK_IP,.,2)})
same => n,Playback(letters/dot)
same => n,SayDigits(${CUT(ASTERISK_IP,.,3)})
same => n,Playback(letters/dot)
same => n,SayDigits(${CUT(ASTERISK_IP,.,4)})
same => n,PlayBack(silence/1&vm-goodbye)
same => n,Hangup()
Now for the context to be used we must restart the Asterisk dial plan.
asterisk -rx"dialplan reload"
To try just dial *777
7.- Turn off and turn on
Warning Note ⚠
When you must turn off the servers, when you turn it on always start with the Master, wait for the Master to start and then turn on the Slave.
8.- Update VitalPBX or Add-Ons
To update VitalPBX to the latest version just follow the following steps:
1.- From your browser, go to ip 192.168.10.30
2.- Update VitalPBX from the interface
3.- Execute the following command in Master console
bascul
4.- From your browser, go to ip 192.168.10.30 again
5.- Update VitalPBX from the interface
6.- Execute the following command in Master console
bascul
CONGRATULATIONS, you have installed and tested the high availability in VitalPBX
9.- Some useful commands
- bascul, is used to change roles between high availability servers. If all is well, a confirmation question should appear if we wish to execute the action. The bascul command permanently moves services from one server to another. If you want to return the services to the main server you must execute the command again.
- role, shows the status of the current server. If all is well you should return Masters or Slaves.
- pcs resource refresh –full, to poll all resources even if the status is unknown, enter the following command.
- pcs cluster unstandby host, in some cases the bascul command does not finish tilting, which causes one of the servers to be in standby (stop), with this command the state is restored to normal.
- pcs resource delete, removes the resource so it can be created.
- pcs resource create, create the resource.
- corosync-cfgtool -s, to check whether cluster communication is happy.
- ps axf, confirmed that Corosync is functional, we can check the rest of the stack. Pacemaker has already been started, so verify the necessary processes are running.
- pcs status, check the pcs status output.
- crm_verify -L -V, check the validity of the configuration.
- drbdadm status, shows the integrity status of the disks that are being shared between both servers in high availability. If for some reason the status of Connecting or Standalone returns to us, wait a while and if the state remains it is because there are synchronization problems between both servers, and you should execute the drbdsplit command.
- drbdsplit, solves DRBD split brain recovery.
- cat /proc/drbd, the state of your device is kept in /proc/drbd.
- drbdadm role drbd0, another way to check the role of the block device.
- drbdadm primary drbd0, switch the DRBD block device to Primary using drbdadm.
- drbdadm secondary drbd0, switch the DRBD block device to Secondary using drbdadm.
- /usr/share/vitalpbx/ha/ ./vpbxha.sh, create the cluster automatically.
- /usr/share/vitalpbx/ha/ ./destroy.sh, completely destroy the cluster, leaving the DRBD intact.
- /usr/share/vitalpbx/ha/ ./rebuild.sh, recreates the cluster starting from the fact that the DRBD
10.- Credits
10.1 Sources of Information
- voip-info.org
- asterisk.org
- DRBD Website (https://www.linbit.com/en/)
- Pacemaker Website (https://clusterlabs.org/pacemaker/)
- https://github.com/VitalPBX/vitalpbx4_drbd_ha