Voice Quality Benchmark Report

Introduction

Voice Quality is a key metric for end-user experience and many Mobile Network Operators and Regulators alike see it as a measure of service of excellence. It is true that nowadays assessment of Audio MOS quality or voice experience evaluation is part of all regulator's assessments across the world.

With the various transitions of technology from circuit switch 2G and 3G voice services to packet switch services with the introduction of Voice over Long-Term Evolution (VoLTE) and soon Voice over New Radio (VoNR) is important to assess the voice experience on all these technologies. The usage of Over the top (OTT) voice Apps services like WhatsApp and Viber is also stronger than ever posing an additional requirement for Regulators and Mobile Network Operators or communication service providers (CSPs) to assess the voice MOS scope of these technologies as well.

At SmartViser we have run an extensive testing campaign using our test automation product viSer together with our audio MOS solution using POLQA algorithm to assess the voice experience and speech quality MOS using three devices: iPhone 14 Pro, Pixel 7 Pro, and Samsung Galaxy S22 Ultra.

Testing Configuration

Testing has been performed under the SFR network in Rennes France in February 2023. The testing environment was static and the following cellular and OTT technologies and voice services were tested:

• 3G

• VoLTE

• VoWiFi

• WhatsApp

• Viber

• Microsoft Teams

15 calls were performed in each technology for each manufacturer.

The OTT tests were conducted on a cellular network using auto network selection. The devices were not forced to select a particular codec or technology in order to emulate a realistic user experience.

viSer test automation was used to ensure all the KPIs are collected, reliability, and repeatability of the testing. Together with ViSer test automation the add-on viSer Audio MOS solution using the POLQA algorithm was also used for the accurate measurement of the audio quality.

POLQA™ MOS scores:

• Score 1: means that despite great effort, it is impossible to understand what is being said during the call (the call repeatedly cuts in and out)

• Score 2: means the quality is not good and a lot of effort is needed to understand

• Score 3: indicates an acceptable level of quality and fairly understandable conversation

• Score 4: is a good quality level, understandable with minor interference noise

• Score 5: is an excellent quality level, understandable discussion without interference noise

Results

3G	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	3.08	3.36	3.39
Min MOS	2.69	2.72	2.85
Max MOS	3.8	3.59	3.88

At 3G both Pixel 7 Pro and Samsung S22 Ultra had a similar performance whilst the iPhone 14 Pro had an average score of 10% less. A score of around 3 indicates an acceptable level of quality and fairly understandable conversation. When the score drops below 3 it impacts the end-user experience negatively as a lot of effort is needed to understand what is being spoken.

---

VoLTE	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	3.27	4.17	3.58
Min MOS	3.13	3.99	3.49
Max MOS	3.37	4.31	3.65

For the VoLTE part, it was surprising that both iPhone 14 Pro and Samsung S22 Ultra were below 4 on average. An average of 3.8 and above is normally expected for VoLTE calls but this can vary depending on devices and mobile network operators. Pixel 7 Pro performed very well on VoLTE calls.

---

VoWiFi	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	3.23	3.86	3.41
Min MOS	2.67	3.45	2.8
Max MOS	3.44	4.12	3.62

On The VoWiFi we have seen similar values to VoLTE with the Pixel 7 Pro consistently performing better than the iPhone 14 Pro and the Samsung S22 Ultra.

---

WhatsApp	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	4.06	4.25	3.94
Min MOS	3.89	4.16	3.22
Max MOS	4.16	4.3	4.26

For the WhatsApp assessment, we have seen excellent results from all three manufacturers. An average close to 4 is a very good result and offers a really good Quality of Experience.

---

Viber	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	3.08	3.35	3.48
Min MOS	3.04	3.3	3.42
Max MOS	3.16	3.42	3.54

Viber had slightly lower average scores with some minimum scores reaching 3 which means the quality is not good and a lot of effort is needed to understand. Again both Pixel 7 Pro and Samsung S22 Ultra had similar results with the iPhone 14 Pro falling slightly behind.

---

Microsoft Teams	iPhone 14 Pro	Pixel 7 Pro	Samsung S22 Ultra
Average MOS	3.70	4.01	4.04
Min MOS	3.51	3.95	3.91
Max MOS	3.83	4.07	4.13

For Microsoft Teams we see a similar average across all manufacturers to 4

indicates a good quality level.

---

Conclusion

For the majority of the testing, there were no big surprises. Both Pixel 7 and the Samsung S22 Ultra had really good performance all around with the Apple 14 Pro slightly behind.

WhatsApp is offering the best quality of experience on Voice Calls across the three manufacturers tested with average scores close to 4, offering a good QoE.

This testing provides a short snapshot of the quality at a certain time with one Mobile Network Operator. Further testing will be planned with additional Mobile Network Operators and smartphone devices and results will be shared in the coming months.

While our benchmark results demonstrate the capabilities of our test automation solution, viSer, it's important to note that determining which smartphone offers a better voice quality experience is not an absolute conclusion.

---

To find out more about SmartViser and schedule a demo of the products and solutions you get in touch now to arrange.

Get in Touch

---

To arrange a free trial please click below. Once the request is received a team member will get in touch to set up a call to discuss your main challenges that need to be addressed with the trial.

Free Trial