When dealing with network performance, it is often the case that we get lost in the technical details - how fast can I push data through this network? What are the delays like? How many packets are being lost? Those details are relatively easy to understand and measure, yet they don't give a complete picture of how the network performs. The reason for this is that the ultimate measure of network performance lies with the user and is heavily dependent on the applications they use.
The way in which users are aware of network Quality of Service (QoS) is by its influence on how they experience the applications they are using. So, for example, while say, a 30% decrease in one-way delay can be a significant improvement in terms of QoS, it might not make any difference to a user doing video streaming. To really understand the performance of a network from the user's point of view — and ultimately, this is what matters most — we need to consider more than just network QoS indicators, and start thinking in terms of Quality of Experience (QoE).
QoE is "the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user's personality and current state." The ways in which network performance, in terms of QoS, relates to QoE are varied, and depend heavily on the application considered.
As an example, the QoE of Voice over IP (VoIP) applications is typically sensitive to several factors that influence it. Some of those factors are intrinsic to the application being used, e.g. the codec in use, the amount of error correction used, etc. Other factors are very much related to the network, such as the rate and temporal distribution of lost packets, the end-to-end delay and jitter. There are ways to improve VoIP, though!
7SIGNAL's system implements a VoIP listening quality estimation algorithm developed by VTT Technical Research Centre of Finland, based on Pseudo-Subjective Quality Assessment (PSQA). This enables the monitoring system to, based on current network measurements, predict how a user will perceive the listening quality of a voice call.
PSQA is an approach to developing QoE models for different applications by correlating a set of parameters expected to affect quality with actual quality assessments done by users. This is typically represented by a Mean Opinion Score (MOS) on a five-point scale where 1 represents bad quality and 5 represents excellent quality. A typical work-flow for creating a PSQA-based model is as follows.
In the case of 7SIGNAL's Sapphire VoIP Listening Quality estimation, that trained neural net resembles something like the one in the figure accompanying this article.
Quality prediction models based on PSQA have been shown to be very accurate for a variety of applications ranging from VoIP to video streaming, and are very lightweight. Within the Sapphire system, they are used as part of an active measurement suite, so that the network's performance, in terms of QoE, can be assessed at any time, whether or not actual VoIP traffic can be found in the network.
*Special 7SIGNAL thanks to guest blogger Martin Varela, Principal Scientist at VTT Technical Research Centre of Finland.