Is Asterisk PBX Multicore? Test with more than 4,500 concurrent calls

Introduction

Yes, Asterisk PBX is multicore capable. Asterisk, being an open-source telecommunications software, can utilize multiple cores or processors in a system. Starting from version 1.8, Asterisk introduced significant improvements in its multicore support. These improvements allow for more efficient distribution of workload across multiple CPU cores, resulting in enhanced performance and processing capacity.

With each subsequent version, Asterisk has continued to enhance its multicore capabilities. However, it’s important to note that the extent to which Asterisk effectively utilizes multicore resources can vary depending on factors such as system configuration, workload, and specific optimization for the deployment environment. Additionally, not all Asterisk modules and components may be specifically designed to fully leverage multiple cores. Some tasks, such as audio transcoding or processing certain protocols, may still be more dependent on a single core.

In general, Asterisk can utilize multiple cores to distribute workload and improve performance in appropriately configured environments. However, the actual effectiveness of multicore capabilities will depend on the aforementioned factors and how the system is configured and optimized for multicore usage.

The number of concurrent calls that Asterisk can support depends on various factors, including hardware capabilities, system configuration, network conditions, and the specific configuration of the Asterisk server.

Asterisk is known for its scalability and ability to handle many concurrent calls. With proper hardware and configuration, Asterisk can handle hundreds or even thousands of concurrent calls on a single server.

The specific number of concurrent calls that Asterisk can support will vary based on factors such as:

  • Hardware Resources: The processing power (CPU), memory, and network capacity of the server running Asterisk can impact the number of concurrent calls it can handle. A server with more powerful hardware will generally be able to handle a larger number of calls.
  • Network Conditions: The available bandwidth and network stability play a role in determining how many concurrent calls Asterisk can handle. Insufficient bandwidth or network congestion can limit the number of concurrent calls.
  • Codec Selection: The choice of codecs used for the calls can also affect the number of concurrent calls. Some codecs require more processing power than others, so selecting efficient codecs can help increase the number of concurrent calls.
  • System Configuration: The configuration of Asterisk, including settings related to media handling, call routing, and call processing, can impact the capacity for concurrent calls. Proper optimization and tuning of Asterisk can improve its performance and scalability.
  • Call Recording: Recording calls can impact the capacity of concurrent calls in Asterisk. When call recording is enabled, Asterisk needs to allocate additional resources to capture and store the audio data for each recorded call. This can consume CPU, memory, and disk I/O resources, which may reduce the overall capacity for concurrent calls.

It’s worth noting that for high-volume or enterprise-grade deployments, it’s common to use load balancing techniques, distributed architectures, and multiple Asterisk servers to handle many concurrent calls efficiently.

In summary, the exact number of concurrent calls that Asterisk can support is highly dependent on various factors, including hardware resources, network conditions, codec selection, and system configuration. With proper hardware and optimization, Asterisk can handle a significant number of concurrent calls on a single server.

Test

To carry out the tests of how many concurrent calls Asterisk supports depending on the hardware, we set ourselves the task of making a script and with it carry out the tests.

Next, we describe how the script works:

  • Servers, two servers are needed, one where the test is performed and another where the calls for the test are sent. The Server where the test is carried out is our source server and the one where the calls are sent is our target server. For the test, the destination server must have more or equal hardware capacity than the source.
  • Local IP, because a trunk connection is made between both servers, and it is authenticated by IP, it is necessary to know the IP of the source server.
  • Remote IP, since you must access the destination server from the source server to create the IP trunk, you need to know the IP. When the test starts for the first time you will be prompted for the root password of the target server.
  • SSH Remote Port (Default is 22), ssh port, it is requested since in some servers this port can be different for security reasons.
  • Network Interface name (ej: eth0), It is the network interface through which both servers will interconnect. This data is very important since the consumption of bandwidth is also measured. To know the name of the interface you can use the command “ip a”.
  • Protocol (1.-SIP, 2.-IAX, 3.-PJSIP), Although the protocol selection is not going to seriously impact the test result since the greatest load is on the RPT packets when the call is already established, we put it here as an option for you to carry out your own tests.
  • Codec (1.-None, 2.-G79, 3.- GSM), codec to use for testing. Currently, due to the improvement of internet connections, it is no longer necessary to use compression codecs, this is because of these codecs.
  • Recording Calls (yes, no), if you want the calls to be recorded or not. Call recording seriously decreases the capacity of concurrent calls on the server demands a lot of CPU processing and seriously decreases the processing capacity of concurrent calls on the processor.
  • Max CPU Load (Recommended 75%), is the maximum allowable load on the processor at the time of testing. It is recommended that this value is always less than 75%.
  • Call Interval, there are two parameters that are configured:
    • Calls Steps, number of calls to send in a certain time, which is configured in seconds between each step.
    • Seconds between each step, is the number of seconds in which the calls will be sent, that is, if we have 50 calls in the Calls Steps parameter and in this, we configure 5 seconds, it means that a call will be sent every 0.1 second (every 100 ms) (5/50). After sending the 50 calls, a second is waited to send the next packet of 50.

Test Result

These are the descriptions of each column in the result:

  • Step, Number of tests that are done every N seconds.
  • Calls, Number of calls generated by the test.
  • Asterisk Calls, number of calls that Asterisk currently has in progress.
  • CPU Load, percentage of CPU used, if the CPU has more than one Core, this percentage is already considered.
  • Load, cpu load average in the last minute. If this number is greater than the number of cores that the cpu has, it means that processes are remaining in the queue, which seriously affects calls.
  • Memory, percentage of memory being used.
  • BW TX kb/s, upload bandwidth in kb/s on the call.
  • BW RX kb/s, download bandwidth in kb/s on the call.

After carrying out a series of tests on vultr.com servers, we reached the following conclusions:

  • Asterisk is Multicore and takes advantage of Threads in a single process, which makes it very efficient.
  • We conducted a test with a 48-core Bare Metal Server and reached a maximum of 4,542 concurrent calls with 46% CPU occupancy. This limit is due to Asterisk not allowing more concurrent calls; however, you can modify the source code to remove this limit. The error we got was the following:

“FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object”

If we go to the Asterisk source code and edit the files, we could have more concurrent calls.

				
					# file main/astobj2.c
EXCESSIVE_REF_COUNT 900000
# file include/asterisk/taskprocessor.h 
AST_TASKPROCESSOR_HIGH_WATER_LEVEL 20000
				
			

Although we do not recommend this practice since we believe that it is more than enough for a server to have 4,500 concurrent calls.

  • If we want to keep calls with good audio quality in our PBX and we want to avoid an overload that affects the entire system, we recommend using the Asterisk parameter called maxload. When this parameter is configured, Asterisk does not allow more calls when it is exceeded. To better understand this parameter, the following must be considered:
    • All servers have a Load record, the load average refers to the average number of processes in a runnable (or waiting to be executed) state over a specific period. It provides an indication of the system’s workload or the number of processes that are demanding CPU time. Asterisk takes the Load Average of the last minute, when this Load Average exceeds the maxload, Asterisk stops processing calls.
    • Consider that the permissible Load Average value is directly related to the number of Cores (CPUs), in a system that has 4 CPU Cores it can reach a maximum of 4 without affecting the processing capacity of the system. If the value exceeds 4 it means that there are processes that are waiting, for data this is not a problem to wait a bit, but for voice it is a problem, since voice cases begin between cuts or loss of voice packets.
    • Therefore, we recommend that the value of maxload be the number of Cores (CPUs) – 1 (Systems with 4 to 12 CPUs). To make sure that everything works well.
    • Considering that this Load Average is the average load in the last minute, it is not real time. So, we could have audio problems even if we apply Cores (CPUs) – 1 (Systems with 4 to 12 CPUs).
    • The file where this parameter is configured is:

                  

				
					nano /etc/Asterisk/vitalpbx/asterisk__20-options.conf
				
			
				
					[options](+)
Maxload=3 

				
			

Then you must restart Asterisk

				
					systemctl restart asterisk
				
			

To perform the tests, we recommend you go to the following link:

https://github.com/VitalPBX/VitalPBX-Stress-Test

Test results on Bare Metal Servers:

asterisk pbx stress call test 1

Test results on Dedicated vCPU Virtual Servers:

asterisk pbx stress call test 2

As we can see the results when using Bare Metal were much better than with Virtualization.

We do not recommend using shared CPU virtualization in production, since the results will depend a lot on the environment (neighbors) where the server resides.

To carry out your own tests and have a gift of US$ 100.00 at vultr.com we recommend the following link:

https://www.vultr.com/?ref=9472294-8H

 

We also recommend you watch the following video, where the concurrent call capability is shown manually on a Raspberry PI 3 and PI 4.

 

Below we show a detail of the result of one of the tests:

Test Result with 4 cores

				
					Virtual
Model name: AMD EPYC-Rome Processor
CPU MHz: 2000
CPU(s): 4
Thread(s) per core: 2
Core(s) per socket: 2
Mem: 8Gb
Disk: 50GB NVMe
Cost: US$ 80.00/month
				
			

Protocol: SIP, Recording: No

Protocol: SIP, Recording: Yes

I have seen that the load of Asterisk calls is distributed equally across the 4 cores, in which cases is it possible for a single core to be overloaded?

While the load of Asterisk calls may be distributed evenly across 4 cores in many scenarios, there are situations where one core might become overloaded:

  1. Non-Asterisk Processes: Other processes and applications on the system might be consuming significant resources on a specific core. While the operating system should efficiently distribute the load, there could be situations where this doesn’t occur optimally.

  2. CPU Affinity or Pinning: On some systems, it’s possible to set a process’s “affinity” to run on a specific core (known as “CPU pinning”). If Asterisk or one of its subprocesses is accidentally (or deliberately) set to run only on a specific core, that core might become overloaded.

  3. Tasks That Aren’t Inherently Parallelizable: While Asterisk is multi-threaded, not all tasks can be efficiently parallelized. If there’s a particular operation that requires a specific sequence or can’t be broken down into smaller tasks, this operation might end up running on a single core.

  4. Older Versions or Configuration: Older versions of Asterisk might not have the optimizations present in newer releases to fully take advantage of multi-core systems. Additionally, certain configurations or modules might impact how Asterisk uses cores.

  5. Operating Systems and Scheduling: How the operating system handles the scheduling of processes and threads on cores can influence how the load is distributed. In most cases, modern systems are good at distributing, but there are specific situations or configurations that could lead to imbalance.

  6. Interrupts and Hardware Handling: System interrupts, especially those related to hardware (like network cards or telephony cards), might be handled preferentially by a specific core. If there’s a high volume of interrupts, this could lead to a disproportionate load on one core.

To diagnose and address issues of overload on a single core, it’s essential to:

  • Use monitoring tools like top, htop, or mpstat to observe the load distribution.
  • Review the Asterisk and system configuration to ensure there are no evident issues.
  • Consider upgrading to a more recent version of Asterisk if not using one.
  • Check system interrupts and how they’re distributed among cores.
  • Ensure there aren’t other processes or applications on the system monopolizing resources.

 

If a single core is overloaded, it can lead to audio problems in Asterisk. Here’s why:

  1. Real-time Processing: Audio handling in a PBX system like Asterisk is a real-time task. This means that audio packets must be processed and transmitted within a specific timeframe to ensure continuous audio flow. If a core is overloaded and cannot process these packets in time, you could experience audio disruptions.

  2. Dropped Packets: Overloading can cause the system to drop RTP (Real-time Transport Protocol) packets. RTP is the protocol Asterisk uses to transmit audio. Dropped packets result in gaps or jitter in the audio stream.

  3. Increased Latency: An overloaded core can introduce delays in processing, leading to increased audio latency. In voice communications, even small delays can be noticeable and disruptive.

  4. Transcoding Delays: If Asterisk is configured to transcode audio (convert from one codec to another), this process requires CPU resources. An overloaded core can struggle with real-time transcoding, causing audio issues.

  5. Effects on Other Processes: Asterisk performs numerous tasks other than audio processing, such as call signaling, call routing, handling of various modules, and more. If a core is overloaded, the performance of these tasks can also be affected, indirectly influencing audio quality. For instance, if the core is too busy to handle signaling properly, it might not set up, modify, or tear down calls efficiently.

  6. Echo Cancellation: Some tasks, like echo cancellation, are CPU-intensive. An overloaded CPU might not perform these tasks efficiently, leading to poor audio quality.

To mitigate audio problems due to CPU overload:

  1. Monitor System Performance: Regularly monitor your system’s performance using tools like top, htop, and mpstat. Keep an eye on CPU usage, load averages, and context switches.

  2. Optimize Configuration: Ensure that Asterisk’s configuration is optimized. This includes using appropriate codecs, minimizing unnecessary transcoding, and optimizing dialplan scripts.

  3. Consider Hardware Upgrades: If you’re consistently facing performance issues, it might be worth upgrading to a more powerful server or adding additional servers and distributing the load.

  4. Load Balancing: If you have a high call volume, consider implementing a load balancer to distribute the SIP traffic across multiple Asterisk servers.

  5. Avoid Running Other Intensive Applications: Ensure that other CPU-intensive applications are not running on the same server as Asterisk, especially if you’re facing performance issues.

In summary, yes, an overloaded core can lead to audio problems in Asterisk because of the real-time nature of audio processing. Regular monitoring and optimization are key to ensuring smooth and clear voice communication.

Conclusions

After having carried out stress tests on VitalPBX-Asterisk for several days, we reached the following conclusions:

  • Asterisk does not allow more than 4,500 concurrent calls; this is limited in the source code.
  • We do not recommend more than 5,000 extensions or 100 tenants on a single server, since it must be considered that other processes are running in addition to the calls, such as:
    • System Administration.
    • Call Recording.
    • Obtaining statistical data (Sonata Stat).
    • VitXi sessions, which loads the system.
    • Maintenance processes.
    • Fail2ban.
    • Supervision of Extensions, queues of calls, etc. With Sonata Switchboard.
    • And many more processes typical of the operating system.
  • Efficient codecs: use efficient codecs with low computational load. Some codecs require less CPU resources than others, allowing a higher number of concurrent calls to be handled. Evaluate and select codecs that balance voice quality and resource efficiency.
  • Efficient storage: Use fast and efficient storage systems for call recording, such as solid-state drives (SSDs), to minimize the impact on server performance.
  • When using a virtualized environment with shared CPUs, multiple virtual machines share the resources of the physical host, including CPU cores. This means that the performance of each virtual machine can vary depending on the workload of other virtual machines on the same host.
  • Do not run processes with high demand for consumption at times of high call traffic. For example, never schedule a Backup CRON at times of high call traffic.
  • Remember that having a good bandwidth alone is not sufficient to effectively transmit VoIP (Voice over Internet Protocol). While bandwidth is important, there are other factors that come into play to ensure optimal voice quality and a satisfactory calling experience. Some of these factors include:
    • Latency: Latency refers to the time it takes for a data packet to travel from the source to the destination. In VoIP calls, high latency can cause delays and difficulties in communication. Therefore, it’s important to have low latency for smooth voice transmission.
    • Jitter: Jitter refers to variations in packet latency during transmission. If there is high jitter, it can lead to synchronization issues and fluctuations in voice quality. Minimizing jitter is important to maintain clear communication.
    • Packet Loss: Data packets can be lost or discarded during transmission due to network congestion or other issues. Excessive packet loss can negatively affect voice quality. Minimizing packet loss is necessary for proper voice transmission.
    • Quality of Service (QoS): Implementing QoS in the network can prioritize voice packets over other types of traffic to ensure smooth and uninterrupted transmission.
    • Hardware Performance: In addition to bandwidth, the performance of routers, switches, and other network components can also influence voice quality. It’s important to ensure that the hardware being used can handle the VoIP traffic load.

 

Therefore, while bandwidth is important, factors such as latency, jitter, packet loss, QoS implementation, and hardware performance need to be considered to ensure quality VoIP transmission. A comprehensive approach to addressing these aspects will contribute to a better calling experience.

Our Latest Post