StarTrinity.com

VoIP software

VoIP community

VoIP marketplace


logged in as
log out

Performance and Stress Testing of SIP Servers, Clients and IP Networks

Introduction

The emergence of Voice over IP (VoIP) technology offered numerous advantages for end users and providers, but simultaneously introduced security threats, vulnerabilities and attacks which hadn't previously encountered in networks with a closed architecture like the Public Switch Telephone Network (PSTN). If audio data is transferred across public internet, there is no guarantee in constant traffic flow or reliable transport. Also, publically accessible SIP endpoints are subject to denial of service (DoS) attacks which can cause severe degradation of voice quality and call setup success rate. Even in a closed ethernet environment VoIP audio quality is still vulnerable to delays in RTP stream processing caused by network, client or server. These delays can be caused by overloads of CPU and network stack or file system operations. That's why it is necessary to know minimum hardware system requirements to handle certain call load. An integrator must test IP network and SIP servers before launching and make sure that there are no overloads in the system.

On this page we present methodology and results of SIP performance evaluation (benchmarking) for IP networks, servers and clients using a software tool StarTrinity SIP Tester. Our results provide insight into resources' usage by SIP servers and clients. Our measurements show that there is no straightforward relationship between call quality and simulated call load. Moreover, we show that CPU load peaks and big RTP delays (which result in audio gaps) can be caused by some other (non-SIP) processes in device's operating system.

SIP call quality measurement

The process of VoIP system assessment implies making many simultaneous SIP calls to device(s) under test (DUT) and measurement of call quality parameters. Basic parameters are:

  • Maximum RTP jitter per call
  • Global maximum RTP jitter (for all calls)
  • Call answer delay (time between INVITE and 200 OK)
  • Global maximum call answer delay
  • Average CPU load
VoIP call quality depends on IP network, type of server hardware and software, and parameters of processed calls which are generated in test.

Archive of performance and stability reports

Our company is continuously making research in field of SIP call quality for various software, hardware and networks. The research is made by running stressing tests with thousands and millions of VoIP calls and analysing call quality indicators. Some of our reports are published here.

2015-01-19 StarTrinity Softswitch x64 on Windows Server 2012 R2 - 800+800 G729 passthru channels, 2.22 million calls during 3 days
Report for UAC (SIP call generator)
StarTrinity Softswitch was configured as B2BUA: it accepted calls from UAC SIP Tester #1 (UAC) and connected them with SIP tester #2 (UAS):

<transfer value="sip:111@devpc2:5090"/>
<exit />

StarTrinity Softswitch passed 2.22 million SIP calls successfully, but with few 4-second peaks of answer delay (see image below). It is not critical. RTP jitter was low: avg=9.27ms, max=42.76ms, 99.9-percentile=13.87ms
StarTrinity test - answer delay history chart

StarTrinity test - lost packets history chart

StarTrinity test - max RTP delta history chart

StarTrinity test - max RFC3550 jitter history chart

2015-01-16 StarTrinity Softswitch x64 on Windows Server 2012 R2 - 150..1650 G711 channels
We simulated variable call load from 150 to 1750 G.711 channels from SIP Tester to server with StarTrinity Softswitch via Cisco router: Windows Server 2012 R2, Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was 90 seconds. StarTrinity Softswitch was configured to accept calls and play audio file:

<accept />
<playaudio value="music.wav" maxtime="90s" />
<exit >

Configuration was following:

EnableSignalDetector = 0
MediaThreadsCount = 64

StarTrinity Softswitch performance chart - jitter vs number of G.711 channels
2014-08-26 Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - 100 G.711 channels, 6 days, 1.5M calls
We simulated variable call load from 5 to 110 G.711 channels from SIP Tester to server with Asterisk via Cisco router: Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was random from 30 to 40 seconds. We installed Asterisk PBX using Ubuntu software center, and used default configuration. Extension #500 by default accepts calls and plays demo IVR message.
RTP jitter was low, stability was great. Asterisk passed 1.5M SIP calls without crash (see detailed report). There were few calls with lost packets though (see the chart below).
Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - RTP jitter history chart
Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - lost packets history chart
Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - -24dB delay history chart

2014-08-18 Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - 5..110 G.711 channels
We simulated variable call load from 5 to 110 G.711 channels from SIP Tester to server with Asterisk via Cisco router: Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was random from 30 to 40 seconds. We installed Asterisk PBX using Ubuntu software center, and used default configuration. Extension #500 by default accepts calls and plays demo IVR message.
Asterisk performance chart - jitter vs number of G.711 channels. Crash with 110 G.711 channels
RTP jitter was fairly low, but Asterisk crashed with 110 G.711 channels (see the chart). Stability of FreeSWITCH is much better.
Crash of Asterisk 11.7.0 on Ubuntu 14.04 LTS x64

2014-07-27 802.11n 72Mbps wireless LAN - 2..80 G.711 channels, distance = 3m, 1 wall
We built call capacity chart for 802.11n 72Mbps wireless LAN between a server connected to WiFi adapter (Upvel UR-314AN) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). RTP jitter increases with number of channels, but there is no straightforward linear dependency. The jitter greatly depends on some other variable. Max call load capacity is hard to estimate because of sporadic RTP jitter.
Wireless LAN VoIP call load capacity

2014-07-25 802.11g 54Mbps wireless LAN - 2..24 G.711 channels, distance = 3m, 1 wall
We built call capacity chart for 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). RTP jitter increases with number of channels, but there is no straightforward linear dependency. The jitter greatly depends on some other variable, probably obstacles between adapters. Max call load capacity is 24 G.711 channels = 2.1Mbps, it is much less than theoretical 54MBps (3.9%)
Wireless LAN VoIP call load capacity

2014-07-22 FreeSWITCH 1.5.13 x64 on Windows Server 2012 R2 - 150..2250 G711 channels
We simulated variable call load from 150 to 2250 G.711 channels from SIP Tester to server with FreeSWITCH via Cisco router: Windows Server 2012 R2, Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was 90 seconds. FreeSWITCH was configured to accept calls and play audio file:

<action application="answer"/>
<action application="playback" data="ponce-preludio-in-e-major.wav"/>

Configuration in conf/autoload_configs/switch.conf.xml was following:

<param name="max-sessions" value="10000" />
<param name="sessions-per-second" value="1000" />

FreeSWITCH performance chart - jitter vs number of G.711 channels
FreeSWITCH showed good performance with high call load. RTP jitter was fairly low (see the chart). However it crashed in some other long-running tests with millions of calls. We are going to continue tests with FreeSWITCH installed on Linux, it can make a difference

2014-07-21 FreeSWITCH 1.5.13 x64 on Windows Server 2012 R2 - 250+250 G711 passthru channels, 2.89 million calls during 4 days
Report for UAC (SIP call generator)
FreeSWITCH was configured as B2BUA: it accepted calls from UAC SIP Tester #1 (UAC) and connected them with SIP tester #2 (UAS):

<action application="answer"/>
<action application="bridge" data="sofia/internal/freeswitch2siptester@192.168.4.4:5070" />

FreeSWITCH passed 2.89 million SIP calls successfully, but with few large peaks of answer delay (see image below). After that, on 5th day of testing, it crashed with error code 0xC0000005. RTP jitter was low: avg=10.34ms, max=130.55ms, 99.99-percentile=16.63ms
FreeSWITCH test - answer delay history chart. 5-20 second peaks of answer delay and post dial delay

2014-07-27 RTP round-trip delay over 802.11n 72Mbps wireless LAN (distance 3m, 1 wall), single G.711 channel
We simulated 5713 SIP calls via 802.11n 72Mbps wireless LAN between a server connected to WiFi adapter (Upvel UR-314AN) and a laptop (Lenovo G700 - Broadcom). Test duration: 10 hours. RTT: avg=12.82ms, max=2122ms, 99.9-percentile=106ms. 0% RTP packets were lost. The results for 802.11n adapters are worse than previous results for 802.11g adapters.
Detailed report for UAC (SIP call generator)
RTP VoIP packets over 802.11n wireless LAN - round-trip delay + SIP response delay chart

2014-07-24 RTP round-trip delay over 802.11g 54Mbps wireless LAN (distance 3m, 1 wall), single G.711 channel
We simulated 8343 SIP calls via 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). Test duration: 14 hours. RTT: avg=12.25ms, max=73ms. Quality for 100% of calls was perfect, 0% RTP packets were lost. It looks strange because the results are better for larger distance (3m > 1m). It means that there are some unknown variables which affect quality of WLAN.
Detailed report for UAC (SIP call generator)
The charts show that round-trip delay varies with time, and it depends on obstacles between wifi adapters
RTP VoIP packets over wireless LAN - round-trip delay time chart

RTP VoIP packets over wireless LAN - SIP response delay chart

2014-07-23 RTP round-trip delay over 802.11g 54Mbps wireless LAN (distance 1m), single G.711 channel
We simulated 12593 SIP calls via 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). Test duration: 21 hours. RTT: avg=12.34ms, max=393ms, 99.9-percentile=93ms. We observed few peaks of high RTT and packet loss (see the chart). Quality for 99.9% of calls was perfect.
Detailed report for UAC (SIP call generator)
RTP VoIP packets over wireless LAN - round-trip delay time chart

2014-07-22 RTP round-trip delay over 1GBit LAN, single G.711 channel
We simulated 6684 SIP calls via 1GBit Cisco switch between 2 servers. VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). RTT did not go over RTP packet time (20ms). No RTP packet losses were detected.
Detailed report for UAC (SIP call generator)
RTP VoIP packets over 1GBit LAN - round-trip delay time chart

2014-07-21 RTP round-trip delay over internet, single G.711 channel
We simulated 12509 SIP calls via cable internet from office in Russia, Kazan to a VPS server in US, NJ, Secausus. VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). We observed 4 peaks of RTT with maximum of 2779ms. Average RTT: 189ms. 99.9-percentile: 374ms.
Average RTP packet loss: 0.01%. Max RTP packet loss: 9.24%. 99.5-percentile RTP packet loss: 0.33%

Detailed report for UAC (SIP call generator)
RTP VoIP packets over internet - round-trip delay time chart

2014-03-14 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels
Call generator (UAC) nodeLaptop CPU Pentium 2020M 2.4GHz 2 cores detailed report | jitter chart
Call receiver (UAS) nodeLaptop CPU AMD A8 3520M 1600MHz 4 cores detailed report | jitter chart
IP NetworkWireless 54MBit, ASUS USB-N53 USB Adapter at UAS
Number of G711 channels5Number of calls attempted7155
RTP jitter at UAS node avg: 80ms 99-percentile: 577ms max: 10635ms
RTP jitter at UAC node avg: 53ms 99-percentile: 88ms max: 5696ms
Answer delay at UAC node (INVITE...200 OK) avg: 25ms 99-percentile: 51ms max: 715ms
-24dB delay at UAC node (INVITE...RTP audio signal) avg: 97ms 99-percentile: 123ms max: 783ms

2014-03-12 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels.
Call generator (UAC) nodeLaptop CPU Pentium Dual-Core T3100 1.9GHz 2 cores view detailed report
Call receiver (UAS) nodeLaptop CPU AMD A8 3520M 1600MHz 4 cores view detailed report
IP NetworkWireless 54MBit, ASUS USB-N53 USB Adapter at UAS
Number of G711 channels5Number of calls attempted2235
RTP jitter at UAS node avg: 80ms 99-percentile: 256ms max: 738ms
Media threads delay at UAC node avg: 41ms 99-percentile: 155ms max: 570ms
RTP jitter at UAC node avg: 30ms 99-percentile: 63ms max: 255ms
Answer delay at UAC node avg: 36ms 99-percentile: 195ms max: 337ms

2014-03-11 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels.
Call generator (UAC) nodeLaptop CPU AMD A8 3520M 1600MHz 4 cores view detailed report
Call receiver (UAS) nodeDesktop PC CPU i7 3400MHz 8 cores view detailed report
IP NetworkWireless 54MBit, ASUS USB-N53 USB Adapter at UAS
Number of G711 channels5Number of calls attempted2178
RTP delta at UAS node avg: 162ms 99-percentile: 3577ms max: 4621ms
RTP delta at UAC node avg: 38ms 99-percentile: 162ms max: 2018ms
Answer delay at UAC node avg: 154ms 99-percentile: 7507ms max: 15528ms

2014-03-09 ASUS USB-N53 Wireless USB Adapter, 20 G711 channels.
Report for UAC node (SIP call generator - laptop Samsung NP305E7A)
Report for UAS node (SIP call responder - desktop PC i7 3400MHz 8 cores)
The test was performed using laptop and desktop PC (UAC and UAS) connected via USB WiFi adapter (installed on UAS node). RTP audio quality was measured using winpcap RTP/SIP sniffer which is integrated into SIP Tester on both UAC and UAS nodes. For 20075 simulated G711a SIP calls max RTP jitter was 3106ms on UAC node, 11799ms on UAS node. For 99% of calls it was less than 619ms on UAC node, 1099ms on UAS node. RTP packet loss reached 56% for 1 minute long SIP call. Conclusion - tested hardware is not acceptable for 20 G711 channels. We are going to continue tests with less number of parallel calls.
2014-03-08 ASUS USB-N53 Wireless USB Adapter, 10 G711 channels.
Report for UAC node (SIP call generator - laptop Samsung NP305E7A)
Report for UAS node (SIP call responder - laptop Lenovo G700)
The test was performed using 2 laptops (UAC and UAS) connected via USB WiFi adapter (installed on UAS node). RTP audio quality was measured using winpcap RTP/SIP sniffer which is integrated into SIP Tester. For 8350 simulated G711a SIP calls max RTP jitter was 456ms on UAC node. For 99% of calls it was less than 135ms (UAC node). Conclusion - tested hardware is good for non-critical VoIP applications
2014-01-27 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 2 G711 channels
A test with minimal call load (2 channels). Max jitter is 154ms which is good, but max answer delay is not acceptable (7.1 seconds). In worst case the server ignores few INVITE messages. 4751 calls have been attempted. See comments in report file.
2014-01-24 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 4 G711 channels
Max jitter is 381ms, but for 99.99% of calls it is less than 106ms. 32989 calls have been attempted. See comments in report file.
2014-01-23 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 4 G711 channels
PESQ_MOS >= 1.25. 9928 calls have been attempted, part of them have successfully passed through PBX back to SIP Tester. See comments in report file.
2014-01-17 3CX Phone System 12.0.33463.0 with RTP audio going over 54Mbps wireless LAN without router, 4 G711 channels
Test is successful, quality of audio passing through 3CX PBX is good enough (PESQ_MOS >= 2.64). Wireless LAN is not a problem. 2188 calls have been attempted.
2014-01-16 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet via router, 4 G711 channels | PESQ_MOS=3.03 PESQ_MOS=3.28
Test is successful, quality of audio passing through 3CX PBX is good enough. 2008 calls have been attempted.
2014-01-15 3CX Phone System 12.0.33463.0 with RTP audio going over 54Mbps wifi via router, 4 G711 channels |
PESQ_MOS=1.06 PESQ_MOS=1.79 PESQ_MOS=2.59 PESQ_MOS=4.54
To configure 3CX we have set Extension options -> 'pbx delivers audio' to 'true'.
NetGear router WGR614v9 is not suitable for VoIP traffic if wifi is used

Elastix 2.4 (Asterisk 1.8.2) IP PBX running on a laptop

The server (DUT) is configured to answer call (send 200 OK to INVITE), activate RTP media flow and make another call to SIP Tester (bridged redirection). Asterisk performance stress test

Asterisk SIP test setup
  • SIP server software: Asterisk 1.8.20
  • Operating system: CentOS 5.9
  • Processor: Intel Celeron M 1.6GHz
  • RAM: 512MB DDR2 400MHz
  • Network: Ethernet 100Mbps
  • Router: NetGear WGR614v9
  • HDD: 120GB Samsung 4800rpm
  • Call duration: 1 minute
  • Audio codec: G.711
Test results

We have measured call quality parameters for different numbers of parallel calls. VoIP server call capacity is 20 calls if one needs to achieve mean jitter level below 100ms.

Elastix - Max jitter per call vs number of simultaneous calls Elastix - Max jitter per call vs number of simultaneous calls Elastix - CPU load vs number of simultaneous calls Elastix - System report
  • Mean jitter and average CPU load increase with number of simultaneous calls
  • Low average CPU load does not mean low jitter
  • There are bursts in CPU load
  • Max jitter is much higher than mean jitter
  • Max jitter depends on number of calls observed
  • There is no straightforward linear relationship between max jitter and number of simultaneous calls
  • Delays in server are random and do not increase linearly with number of calls. They are absolutely random and we guess that they can be independent from RTP media processing.
  • CPU load peaks occur even without any calls
  • RTP processing is sensitive to execution of other tasks performed on Linux server
Example asterisk configuration files for SIP Tester
/etc/asterisk/sip_custom.conf

[1001]
secret=
dtmfmode=rfc2833
canreinvite=no
context=from-internal
host=dynamic
trustrpid=yes
sendrpid=no
type=friend
nat=no
port=5060
qualify=yes
qualifyfreq=60
transport=udp
encryption=no
callgroup=
pickupgroup=
dial=SIP/1001
mailbox=1001@device
permit=0.0.0.0/0.0.0.0
callerid=1001 <1001>
callcounter=yes
faxdetect=no

/etc/asterisk/extensions_custom.conf

[ext-local-custom]
exten => 1001,1,dial(SIP/1001)
exten => 1001,hint,SIP/1001

Android SIP Client (3CXPhone) with WiFi connection

  • SIP client software: 3CXPhone 2.0.5
  • Operating system: Anfroid 4.0.4
  • Mobile device: ZTE V880E
  • Network: WiFi 802.11bg
  • Router: NetGear WGR614v9
  • Call duration: 10 minutes
  • Audio codec: G.711

We have made SIP calls from Android device to computer with SIP tester and measured call quality. We have found out that RTP jitter does not depend on WiFi signal strength if it is greater than -78 dBm. Big delays in RTP flow happen when one launches some other application during the test call.

  • Max jitter without interference from other applications: 60..70ms
  • Max jitter with interference from other applications: 110..130ms

Internet connection between SIPP at Raspberry Pi 2 and StarTrinity SIP Tester at windows server

We have installed SIPP (open-source SIP Tester) at Raspberry PI 2, configured it to make 1-channel test calls to our server. The SIPP client played an audio file to StarTrinity SIP Tester, the audio quality was verified with PESQ algorithm. view detailed report

Results

PESQ score drops below 4.5 for 0.05% of calls because of packet loss in internet, probability of the packet loss changes with time: March 24 .. 28 - there is packet loss, March 29 .. April 7 - no packet losses. This means that one needs to run tests during at least 2 weeks to check stability of internet connection. One more finding: packet losses are assymetric: download and upload packet drops occur at different time

PESQ MOS chart

SIPP - StarTrinity PESQ MOS chart

Called to caller packet loss chart

SIPP - StarTrinity packet loss chart

Caller to called packet loss chart

SIPP - StarTrinity packet loss chart

Caller to called RTP jitter chart

SIPP - StarTrinity jitter chart

Leave a comment
Copyright 2011-2017 StarTrinity.com | Blog | Contact lead developer via LinkedIn | Get support via TeamViewer