Mobile WiMAX field trial test through multimedia performance evaluation

Abstract

IEEE 802.16e is a mobile version of Worldwide Interoperability for Microwave Access (WiMAX) that plays an important role in the evolution towards 4G. In this study, we focus on multimedia performance measurement for the purpose of a more realistic mobile WiMAX network test. This study aims to make a contribution in better understanding the mobile WiMAX performance for multimedia applications. For that purpose, we employ Voice over Internet Protocol (VoIP) and video streaming to test the network performance, where two distinct evaluation systems are used, professional and user-friendly. Our test results show that the mobile WiMAX network can support well the bandwidth-intense and delay-sensitive multimedia application. We find that the VoIP quality at the cell center is perfect, where the value of Perceptual Evaluation of Speech Quality exceeds 4. At the cell edge, the quality is degraded but still adequate. We also observe that the downlink of mobile WiMAX network can support video streaming up to 4 Mbps with the Mean Opinion Score (MOS) value of 4.5. On the uplink, the bitrate of 1 Mbps is supported with MOS 4.5 at the cell center and with MOS 3.2 at the cell edge, respectively. Our experiments further indicate that a smooth playback of YouTube 480P video is consistently provided. Finally, the handover case has very limited impact to the overall quality degradation of both VoIP and video streaming.

1. Introduction

With the increasing popularity of networked applications, multimedia traffics are expected to account for a large portion in the next-generation mobile communication systems. Many technologies are being developed to support broadband wireless communication, among which Worldwide Interoperability for Microwave Access (WiMAX) and Long-Term Evolution (LTE) are prominent on the aspects of high-data rate and long-range coverage. Both WiMAX and LTE are playing an important role in the evolution towards 4G. As a mobile version of WiMAX, mobile WiMAX uses similar technologies and has comparable performance to LTE. Since the standardization of mobile WiMAX is a little earlier than that of LTE, most of pilots are based on the WiMAX technology. Besides the standardizations of radio interfaces, many projects are launched to enhance the performance of mobile communication systems. Wireless Initiative New Radio (WINNER) [1] was a research project funded by the European Union 6th Framework. The objective of WINNER is to develop a ubiquitous radio interface for Beyond 3rd Generation.

Despite the significant interests in next generation technologies of mobile communication, there are very few publicly reported measurements on field trials, because of the limited deployments and the proprietary nature of these deployments [2, 3]. Most of research studies were conducted through system simulation or numerical analysis. Consequently, there is a need to bridge the gap between the performance perception and the actual performance limitations of WiMAX. To this end, the French project POSEIDON [4] deployed a mobile WiMAX testbed in both rural and urban areas. This study is an empirical investigation of multimedia performance in the mobile WiMAX field trials.

Throughput, latency, jitter, and packet loss are widely recognized as major metrics for network performance measurement, where application-independent traffics are usually generated for the purpose of test. However, these metrics just overviews the general network performance, which cannot provide a thorough analysis for the specific performance of application over the network. Unlike traditional data applications, multimedia applications not only generate heavy traffics, but also have more stringent Quality of Service (QoS) requirements. Moreover, multimedia data such as voice and video are usually error-tolerant but delay-sensitive. So, even some errors introduced, the original information may still be reconstructed with tolerable distortion. The influence can be further lightened by the mechanism of Error Concealment (EC) at the side of decoder. However, a tiny delay may significantly degrade the quality of experience due to the real-time characteristic of multimedia. Since delay and jitter are randomly affected by a variety of complicated factors, it is very difficult to simulate them accurately by numerical analysis. Therefore, in this article, we have investigated the performance of mobile WiMAX network in practical conditions, which aims at achieving better understanding the network performance for multimedia applications.

In this article, we employ multimedia traffics, specifically Voice over Internet Protocol (VoIP) and video streaming. Unlike the past research, two experimental systems are set up, namely dedicated one and general one. The former is to make use of a dedicated evaluation framework, which is professional but technology oriented. The latter is to utilize popular applications, which is simple but user friendly. Moreover, the test results from the latter may serve as a benchmark for future use. Apart from the widely used metrics like throughput and delay, we design comprehensive scenarios to evaluate multimedia performance over the real mobile WiMAX testbed. In order to show the characteristics of adaptive modulation and handover, we have conducted all tests at the cases of the cell edge, cell center as well as handover. Since there is no equivalent mobile network deployed commercially, we further compare the test results with those of well-known networks such as Ethernet or Asymmetrical Digital Subscriber Loop (ADSL). We find that the VoIP performance on the downlink is perfect, which can be even comparable with that of 100M Ethernet. On the uplink, the quality is degraded but still adequate and better than ADSL. We also observe that the downlink of mobile WiMAX network can support video streaming up to 4 Mbps with the Mean Opinion Score (MOS) of 4.5. On the uplink, it is 1 Mbps with MOS 4.5 at the cell center and with MOS 3.2 at the cell edge. Our experiments further indicate that a smooth playback of YouTube 480P video is consistently provided, even though the startup latency is obviously bigger than those over Ethernet.

The rest of this article is organized as follows. Section 2 reviews the background and related study. In Section 3, we describe the experimental environment in network layer and application layer, respectively. Section 4 presents the test results and gives a sufficient analysis. Section 5 concludes the article and points out the future research.

2. Related study

IEEE 802.16 is a family of standards for broadband wireless metropolitan networks, which have recently been consolidated as 802.16e-2005 [5]. These standards define the Physical (PHY) and Medium Access Control (MAC) layers of the air interface. The physical layer of 802.16e defines the Orthogonal Frequency Division Multiple Access (OFDMA) as the digital modulation scheme. The physical layer supports Adaptive Modulation and Coding (AMC), which is used to achieve the highest data rate for a given link quality. The modulation schemes can be adjusted at very short time intervals (e.g. 5 ms) to provide robust transmission links and high system capacity. Considering the feature, we conduct all tests in two cases, cell center with good signal, and cell edge with poor signal. Received Signal Strength Indicator (RSSI) is a measurement of the power present in a received radio signal. Carrier to Interference-plus-Noise Ratio (CINR) is a measurement of signal effectiveness, which provides information on how strong the desired signal is compared to the interference plus noise. In the MAC layer of 802.16e, the QoS features enable operators to optimize network performance depending on the service type (e.g., voice, video) and the user's service level. 802.16e defines five QoS classes, Unsolicited Grant Service (UGS), Extended Real-time Polling Service (ertPS), Real-time Polling Service (rtPS), Non-real-time Polling Service (nrtPS), and Best Effort (BE).

Prior to mobile WiMAX 802.16e, the fixed version 802.16d was standardized in 2004. Thus, most of the existing works studied the network performance for the fixed WiMAX networks. There are relatively few experimental results available for mobile WiMAX networks. Grondalen et al. [6] presented the measurement of throughput and physical parameters. They reported that their WiMAX system can deliver 9.6 Mb/s to a single flow in the downlink even at a distance of 5 km from the BS. Pentikousis et al. [3] conducted an experimental investigation of the network performance over a fixed WiMAX testbed. They employed multiple competing traffic sources over a point-to-multipoint topology and measure the network capacity. Although the multimedia services including VoIP and video streaming were applied in their experiment to generate traffic, they ignored the QoS from the viewpoint of end-users. Halepovic et al. [2] used experimental measurement to study the performance of VoIP and video streaming over a commercial fixed WiMAX network. However, they considered only the single-user scenario. In addition, their results, especially for the video streaming case, were more based on subjective evaluation, which was neither comparable nor applicable. Coelho et al. [7] reported a measurement campaign in a sub-urban area. Their study focused on signal coverage. Lin et al. [8] investigated the performance of a WiMAX-based VoIP established under a field trial program. The most related study was presented by Kim et al. [9]. They conducted measurements over Wireless Broadband, a Korean version of mobile WiMAX system, for both system performance and single-user performance.

To the best of authors' knowledge, none of the past research conducts both VoIP and video streaming experiments in a real mobile WiMAX network. Moreover, we set up two kinds of experimental system for professional assessment and user-center tests. Note that the MAC protocol and scheduling policy are either proprietary to vendors or non-public to subscribers. Thus, different from most simulation-based researches, we treat the WiMAX card as a black box in order to make experiments more realistic.

3. Experimental environment

In this section, we briefly describe the experimental environment from the aspects of network layer and application layer.

3.1. Network environment

Mobile WiMAX is not just the last mile wireless network as the case of fixed WiMAX, but it requires a WiMAX Core Network (WCN) behind the Radio Access Network (RAN) in order to manage QoS, mobility and security, etc. Typically, a mobile WiMAX system comprises four basic elements: User Equipment (UE), WiMAX Base Station (WBS), WiMAX Access Control (WAC), and Operation & Maintenance Center (OMC). Our mobile WiMAX testbed was deployed at the campus of Institut Telecom SudParis as a part of the urban scenario in the POSEIDON project. The campus is covered by two WBS with an overlap area allowing handover between them. RAN and WCN are deployed in two distant sites linked by an IPsec tunnel. Figure 1 presents the network architecture of our mobile WiMAX testbed.

Figure 1

The architecture of mobile WiMAX testbed.

Our project partner Alcatel-Lucent provides all the network equipments. Alcatel-Lucent 9715 L-WBS is a lightweight WiMAX-based station, which is one step further in the All-in-One-Box concept. It implements the physical and MAC layers. Table 1 shows the physical characteristics of the L-WBS.

Table 1 Alcatel-Lucent 9715 L-WBS characteristics

The Alcatel-Lucent 9740 works as WAC, which ensures session control and data transport functions. All 9740 WAC traffics are handled by IP protocol. The Alcatel-Lucent 9753 works as OMC, which provides centralized management function for all the elements belongings to the WiMAX access network. At the same time, OMC hosts servers such as Dynamic Host Configuration Protocol (DHCP), Domain Name System (DNS), and Network Time Protocol (NTP). As far as Authentication, Authorization, and Accounting (AAA) are concerned, beside its traditional operations, it attributes the appropriate service flow to the authenticated users depending on the service they are using and their subscription. UEs can be Mobile Subscription Station (MSS) or Costumer Premise Equipment (CPE). In the experiments, we have three kinds of UE: an Alcatel-Lucent PCMCIA card, a Sequans USB dongle, and a Zyxel CPE.

In this study, we select three experimental locations: cell center, cell edge, and handover. The cell center with the Line-of-Sight (LOS) link is around 100 m to BS in distance, while the cell edge under the Non-Line-of-Sight (NLOS) link is about 800 m. Table 2 presents the mean value of CINR, the mean value of RSSI as well as the adopted modulation schemes respectively.

Table 2 Signal measurement

IEEE 802.16e has implemented a full mobility support of handover. The Hard Handover (HHO) is the only one mandatory specified in IEEE 802.16e and supported by our testbed. HHO is easy for implementation, but it increases the end-to-end delay that is critical for the delay-sensitive services such as VoIP. In the configuration of our testbed, MSS starts the neighbor BS scanning process at 14 dB of CINR. To investigate the impact of HHO, an MSS moves from one BS to the other during the VoIP session or video streaming. Regarding QoS, five classes mentioned above have already been implemented by the Alcatel-Lucent equipment WAC. Unfortunately, our testbed was configured to support only BE in this stage. As a result, all experiments in this study are conducted under the QoS class of BE, even though BE is not originally designed for multimedia services. According to the QoS settings, the maximum data rate is limited to 4 Mbps on the downlink and 800 kbps on the uplink. To be noted, the reference networks in this article are the Ethernet-based campus network of Institut Telecom SudParis and the commercial ADSL network operated by France Telecom.

3.2. Application environment

In this article, we propose to use two kinds of test systems, the dedicated systems for professional measurement and the user-center systems for user-friendly assessment. The specific application environment will be described in the following subsections.

3.2.1. VoIP

We focus on two aspects of VoIP performance. First, we evaluate the perceived voice quality. The International Telecommunication Union (ITU) recommends the Perceptual Evaluation of Speech Quality (PESQ) method standardized as ITU-T P.862 [10]. PESQ requires the sent audio wav-file and the received wav-file as input and returns as result a value ranging from -0.5 (worst) to 4.5 (best). The degradation of voice quality has different causes such as codec and network, etc. In order to eliminate interference, we set PESQ over Ethernet as reference to other networks. Second, we show the network conditions in terms of delay, loss, and jitter.

We select the softphone Phoner as the dedicated VoIP system as shown in Figure 2. The version of Phoner is v2.5.2 at the time of experiment. The voice codec used is G.711 A-Law (64 kbps). Similar to many other softphone solutions, Phoner uses Session Initiation Protocol (SIP) for signaling and Real-time Transport Protocol (RTP) for media transmission. Specifically, two clients of Phoner are installed on two laptops connected with WiMAX and Ethernet, respectively. We capture protocol logs by Wireshark at both sides. Wireshark can provide an advanced analysis of Telephony. The speech media data extracted from logs are further compared to evaluate the voice quality. Since we extract audio samples from RTP payloads at both the sender and the receiver, the key factor which impacts the quality degradation is the network.

Figure 2

Phoner test system.

As one of the most dominant VoIP applications today, Skype is selected to enable user-friendly VoIP experiment. We install two Skype clients in two laptops. One laptop is connected with a WiMAX modem as callee or caller for the tests of uplink and downlink, respectively. The other laptop is interfaced with Ethernet. In order to evaluate the voice quality, we play a speech sample at the callee and record the speech at the caller. The tool Pamela is used for this purpose. This Skype add-on is integrated with an auto answer machine and a voice recorder. The VoIP quality is evaluated by analyzing the input wav-sample and the recorded wav-sample. The call conditions including delay, loss, and jitter are reported by the Skype build-in menu named "Call technical information". Figure 3 presents the Skype test system. Although, Skype is characterized by its peer-to-peer structure and the proprietary protocol, these values can still work as comparable results in various networks. All results in terms of voice quality and network conditions are further compared with those of getting from Ethernet-to-Ethernet. The degradation of Skype voice quality has different causes such as codec and network. In order to eliminate the interference, we set PESQ over Ethernet as reference to other networks.

Figure 3

Skype test system.

3.2.2. Video streaming

There are two widely accepted protocols for video streaming, RTP over User Data Protocol (UDP) and Hyper Text Transport Protocol (HTTP) over Transmission Control Protocol (TCP). RTP/UDP is used extensively in communication and entertainment systems. The recent measurement studies indicate that a significant fraction of Internet streaming media is currently delivered over HTTP/TCP [11].

In this article, the professional measurements are conducted with EvalVid [12] over RTP/UDP, as shown in Figure 4. Specifically, EvalVid is a framework for evaluating the quality of video transmitted over a real or simulated network. It is targeted for researchers who want to evaluate their network designs or setups in terms of user-perceived video quality. Video quality is measured by calculating the average Peak Signal-to-Noise Ratio (PSNR) over all the decoded frames. However, the metric of PSNR does not directly correspond to the user-perceived quality. Subsequently, the subjective quality is calculated on the heuristic conversion from PSNR to MOS as shown in Table 3 [12]. In video transmission systems, not only the actual loss is important for the perceived video quality, but also the delay of frames and the variation of the delay. The network parameters including loss rate, delay, and jitter can be measured by the trace analyzing tool of EvalVid.

Figure 4

EvalVid test system [12].

Table 3 Possible PSNR to MOS conversion

We use YouTube for user-friendly video streaming experiment. YouTube is one of the most popular web sites for video sharing and streaming. In fact, the High Definition (HD) video 1080P is supported by YouTube. However, the HD version is accompanied with the bitrate up to 5 Mbps, which puts a big challenge to the access networks. YouTube uses HTTP/TCP to buffer video data to the flash player. The most critical issue is buffer-under-run, which results in video freezing. It substantially degrades the user experience very much. Therefore, we first evaluate the streaming performance by subjective assessment of buffer-under-run. We further measure and analyze the network parameters, throughput and delay, in different network environments. In this part of study, we use Firefox v3.6.6 where Adobe Flash Player 10 is integrated.

4. Performance studies

In this section, we analyze the test results. If no otherwise specified, the test results in above-mentioned cases, cell center, cell edge, and handover.

4.1. VoIP

4.1.1. Phoner

We use Phoner v2.5.2 at the time of experiment. Please note that the SIP-based softphone works on the mode of point-to-point, more specifically WiMAX-to-Ethernet, between which there is no proxy or server. Therefore, below test results reveal the network performance distinctly. Figure 5 illustrates the VoIP performances over downlink and uplink, respectively. Since we extract audio samples from RTP payloads at both the sender and the receiver, there is no quality degradation incurred by the codec. As we can see, PESQ over both uplink and downlink is perfect even at the cell edge. This result complies with the measurement reported in [8]. But the jitter increases sharply, which may be caused by the retransmission mechanism due to the bad radio condition. The voice quality during the transition of handover is pretty good. For a normal user, the quality degradation can be ignorable. After analyzing the Wireshark log, the loss incurred by HHO is only one RTP packet. Considering the sample frequency of 8000 Hz, we can deduce that the duration of two RTP packets is 40 ms. It implies that HHO is less than 40 ms. However, the jitter is seriously affected by HHO as shown in Figure 6. After about 1 s, the jitter goes back to normal. We further find that when the radio link gets extremely worse at the boundary if without Handover (e.g., 16 dB and -82 dBm for CINR and RSSI, respectively), the voice quality over uplink becomes annoying (PESQ 1.7). At that case, the packet loss can even reach 35% and the mean jitter is about 20 ms. It demonstrates that the mobility of handover is crucial to the VoIP quality.

Figure 5

Phoner VoIP quality.

Figure 6

Phoner jitter.

4.1.2. Skype

We use the latest version of Skype (v4.2) available at the time of experiment. The voice samples from Signalogic are wav-files with 8 kHz sample rate and 16-bit encoding. The average PESQ values are shown in Figure 7. As we can observe, the VoIP performance over the downlink is perfect. They are almost similar to that of the 100M Ethernet. The experiment results indicate that the mobile WiMAX network supports a good network performance. It is noted that the degradation of voice quality has different causes such as codec and network, etc. In this experiment, we ignore the quality degradation incurred by the speech codec of Skype. The reference sample is the original one before being encoded, whereas the degraded sample is the one being decoded afterward. As a result, even at the very good network condition, the PESQ value is lower than 4.5 (e.g., 4.2 for Ethernet). In this study, we set the PESQ value over the Ethernet network as the reference for the measurements.

Figure 7

Skype VoIP quality.

Skype measures and reports the call technical information including packet loss, Round Trip Time (RTT), jitter, etc. We observe that there is no packet loss in all test cases, even during HHO. It is reasonable because of the lower bandwidth usage and the reliable TCP used for speech media data in our tests. Considering the different mechanism of underlying protocols, we may find the quality degradation of Skype is different from that of Phoner. The latter is mainly affected by packet loss due to the unreliability of UDP. Whereas, jitter and delay are the main factors to the voice quality of Skype. The tests show that the jitter has more dynamic variation compared to RTT. This result confirms the conclusion in [13] that the jitter relative to delay has a significant impact. According to Figure 7, the voice quality over downlink is different from that of uplink. This could be due to the duplexing mode of TDD. The asymmetrical TDD ratio differentiates the jitters over downlink and uplink, and the differences of quality degradations between the cell edge and the cell center can be further explained that the worse radio conditions increase the retransmission of TCP at the cell edge.

The mean values of jitter are presented in Figure 8. Obviously, HHO increase jitter. The jitter values during HHO are nearly twice of normal cases. Compared with Phoner, Skype is more affected by HHO, and the influence by HHO last much longer time than that of Phoner. To be noted that due to the different calculating method, the jitter value for Phoner is different from that of Skype.

Figure 8

Skype mean jitter.

4.2. Video streaming

4.2.1. EvalVid

Since H.264/MPEG-4 AVC is widely used, in this study we select ×264 as the video codec. For the codec settings, the Group of Picture (GoP) size is set as IPPP at 30 frames, and the frame rate is 30 Hz. Moreover, in order to focus on only the network performance, we do not activate any EC mechanism in the codec. The video sample Highway in the format of Common Intermediate Format (CIF) is encoded with constant bitrates in 400, 600, 1000, and 2000 kbps. The video sample City in the format of 4CIF is encoded in 3000, 4000, 5000, and 6000 kbps. These two samples are used for the tests of uplink and downlink, respectively. After being packetized, the video sample is streamed from the sender to the receiver over RTP/UDP.

Packet losses are usually calculated on the basis of packet identifiers. In the context of video transmission, it is interesting to figure out how many packets gets lost, and which types of frame these lost packets constitute. Thus, the frame loss is counted on after analyzing lost packets further. Figure 9 shows the packet loss and frame loss in all test cases. Obviously, mobile WiMAX suffers from a growing loss along with the increment of bitrate.

Figure 9

Video streaming data loss. (a) Packet loss. (b) Frame loss.

Accompanying with packet loss and frame loss, the video quality MOS is shown in Figure 10. At the cell center, the video quality over the downlink is good (MOS 3.8) with the bitrate of 5 Mbps, while at the cell edge the value of MOS is 3.6. However, the bitrate of acceptable quality at the cell center is 1 Mbps over the uplink, while at the cell edge it is 0.4 Mbps.

Figure 10

Video streaming MOS.

To figure out the impact of handover on the video quality, we also conducted handover tests. The Highway sequence encoded with 600 kbps is selected in order to minimize the packet loss excluding HHO. The overall video quality is perfect at the case of handover (MOS 4.9). However, the quality degradation during handover is very annoying as shown in Figure 11. We further observe that the number of lost packets directly incurred by HHO is about 10, which generally affects 15 sequential frames (0.5 s in case of 30 fps).

Figure 11

HHO test. (a) Original image. (b) Affected image.

4.2.2. YouTube

We take the YouTube video link of "Test Speed" as the test sample. This test video can display technical information (e.g., the downloading speed, the video fps and video bitrate, etc.) in real time. Figure 12 presents the HTTP throughputs together with standard deviation. At the edge of cell, the average HTTP throughput is around 1.5 Mbps much lower than the bitrate of 720P, which results in unsmooth playback. We observe buffer-under-run in 65% of the measurement time. It greatly degrades the user experience. At the center of cell, the 720 version is played much more smoothly, where we experience no picture freezing. The HTTP throughput reported by YouTube is over 2.8 Mbps in average. We further observe that in the cell edge the throughput varies drastically. The peak can even reach 4.3 Mbps. Our results are consistent with the research reported in [11] that TCP streaming generally offers good performance when the available network bandwidth is twice the media bitrate. Considering the short duration of HHO (less than 50 ms) and the underlying protocol of TCP, we find that handover has almost no effect on the HTTP throughput.

Figure 12

HTTP throughput.

5. Conclusions

In this article, we conduct comprehensive experiments of multimedia performance evaluation in a real mobile WiMAX testbed. Our test cases focus on the asymmetrical link, AMC scheme, and mobility of WiMAX. Specially, we employ multimedia traffics of VoIP and video streaming via uplink and downlink at the cell edge and the cell center, as well as handover. In general, the multimedia performance over WiMAX is good. We find that the VoIP quality at the cell center is perfect, where the value of PESQ exceeds 4. At the cell edge, the quality is degraded but still adequate. The performance of video streaming is consistently good with the bitrate as high as 5 Mbps through the downlink, and a smooth playback of YouTube 480P video is consistently provided. In spite of packet loss due to handover, the overall quality degradation is negligible. In the future, we will extend this study by enabling other QoS classes in the testbed. Furthermore, we will introduce real cross-layer optimizations to enhance the multimedia performance at the dynamic network condition of mobile WiMAX.