Mobile WiMAX field trial test through multimedia performance evaluation
© Zhang et al; licensee Springer. 2012
Received: 15 July 2011
Accepted: 20 February 2012
Published: 20 February 2012
IEEE 802.16e is a mobile version of Worldwide Interoperability for Microwave Access (WiMAX) that plays an important role in the evolution towards 4G. In this study, we focus on multimedia performance measurement for the purpose of a more realistic mobile WiMAX network test. This study aims to make a contribution in better understanding the mobile WiMAX performance for multimedia applications. For that purpose, we employ Voice over Internet Protocol (VoIP) and video streaming to test the network performance, where two distinct evaluation systems are used, professional and user-friendly. Our test results show that the mobile WiMAX network can support well the bandwidth-intense and delay-sensitive multimedia application. We find that the VoIP quality at the cell center is perfect, where the value of Perceptual Evaluation of Speech Quality exceeds 4. At the cell edge, the quality is degraded but still adequate. We also observe that the downlink of mobile WiMAX network can support video streaming up to 4 Mbps with the Mean Opinion Score (MOS) value of 4.5. On the uplink, the bitrate of 1 Mbps is supported with MOS 4.5 at the cell center and with MOS 3.2 at the cell edge, respectively. Our experiments further indicate that a smooth playback of YouTube 480P video is consistently provided. Finally, the handover case has very limited impact to the overall quality degradation of both VoIP and video streaming.
With the increasing popularity of networked applications, multimedia traffics are expected to account for a large portion in the next-generation mobile communication systems. Many technologies are being developed to support broadband wireless communication, among which Worldwide Interoperability for Microwave Access (WiMAX) and Long-Term Evolution (LTE) are prominent on the aspects of high-data rate and long-range coverage. Both WiMAX and LTE are playing an important role in the evolution towards 4G. As a mobile version of WiMAX, mobile WiMAX uses similar technologies and has comparable performance to LTE. Since the standardization of mobile WiMAX is a little earlier than that of LTE, most of pilots are based on the WiMAX technology. Besides the standardizations of radio interfaces, many projects are launched to enhance the performance of mobile communication systems. Wireless Initiative New Radio (WINNER)  was a research project funded by the European Union 6th Framework. The objective of WINNER is to develop a ubiquitous radio interface for Beyond 3rd Generation.
Despite the significant interests in next generation technologies of mobile communication, there are very few publicly reported measurements on field trials, because of the limited deployments and the proprietary nature of these deployments [2, 3]. Most of research studies were conducted through system simulation or numerical analysis. Consequently, there is a need to bridge the gap between the performance perception and the actual performance limitations of WiMAX. To this end, the French project POSEIDON  deployed a mobile WiMAX testbed in both rural and urban areas. This study is an empirical investigation of multimedia performance in the mobile WiMAX field trials.
Throughput, latency, jitter, and packet loss are widely recognized as major metrics for network performance measurement, where application-independent traffics are usually generated for the purpose of test. However, these metrics just overviews the general network performance, which cannot provide a thorough analysis for the specific performance of application over the network. Unlike traditional data applications, multimedia applications not only generate heavy traffics, but also have more stringent Quality of Service (QoS) requirements. Moreover, multimedia data such as voice and video are usually error-tolerant but delay-sensitive. So, even some errors introduced, the original information may still be reconstructed with tolerable distortion. The influence can be further lightened by the mechanism of Error Concealment (EC) at the side of decoder. However, a tiny delay may significantly degrade the quality of experience due to the real-time characteristic of multimedia. Since delay and jitter are randomly affected by a variety of complicated factors, it is very difficult to simulate them accurately by numerical analysis. Therefore, in this article, we have investigated the performance of mobile WiMAX network in practical conditions, which aims at achieving better understanding the network performance for multimedia applications.
In this article, we employ multimedia traffics, specifically Voice over Internet Protocol (VoIP) and video streaming. Unlike the past research, two experimental systems are set up, namely dedicated one and general one. The former is to make use of a dedicated evaluation framework, which is professional but technology oriented. The latter is to utilize popular applications, which is simple but user friendly. Moreover, the test results from the latter may serve as a benchmark for future use. Apart from the widely used metrics like throughput and delay, we design comprehensive scenarios to evaluate multimedia performance over the real mobile WiMAX testbed. In order to show the characteristics of adaptive modulation and handover, we have conducted all tests at the cases of the cell edge, cell center as well as handover. Since there is no equivalent mobile network deployed commercially, we further compare the test results with those of well-known networks such as Ethernet or Asymmetrical Digital Subscriber Loop (ADSL). We find that the VoIP performance on the downlink is perfect, which can be even comparable with that of 100M Ethernet. On the uplink, the quality is degraded but still adequate and better than ADSL. We also observe that the downlink of mobile WiMAX network can support video streaming up to 4 Mbps with the Mean Opinion Score (MOS) of 4.5. On the uplink, it is 1 Mbps with MOS 4.5 at the cell center and with MOS 3.2 at the cell edge. Our experiments further indicate that a smooth playback of YouTube 480P video is consistently provided, even though the startup latency is obviously bigger than those over Ethernet.
The rest of this article is organized as follows. Section 2 reviews the background and related study. In Section 3, we describe the experimental environment in network layer and application layer, respectively. Section 4 presents the test results and gives a sufficient analysis. Section 5 concludes the article and points out the future research.
2. Related study
IEEE 802.16 is a family of standards for broadband wireless metropolitan networks, which have recently been consolidated as 802.16e-2005 . These standards define the Physical (PHY) and Medium Access Control (MAC) layers of the air interface. The physical layer of 802.16e defines the Orthogonal Frequency Division Multiple Access (OFDMA) as the digital modulation scheme. The physical layer supports Adaptive Modulation and Coding (AMC), which is used to achieve the highest data rate for a given link quality. The modulation schemes can be adjusted at very short time intervals (e.g. 5 ms) to provide robust transmission links and high system capacity. Considering the feature, we conduct all tests in two cases, cell center with good signal, and cell edge with poor signal. Received Signal Strength Indicator (RSSI) is a measurement of the power present in a received radio signal. Carrier to Interference-plus-Noise Ratio (CINR) is a measurement of signal effectiveness, which provides information on how strong the desired signal is compared to the interference plus noise. In the MAC layer of 802.16e, the QoS features enable operators to optimize network performance depending on the service type (e.g., voice, video) and the user's service level. 802.16e defines five QoS classes, Unsolicited Grant Service (UGS), Extended Real-time Polling Service (ertPS), Real-time Polling Service (rtPS), Non-real-time Polling Service (nrtPS), and Best Effort (BE).
Prior to mobile WiMAX 802.16e, the fixed version 802.16d was standardized in 2004. Thus, most of the existing works studied the network performance for the fixed WiMAX networks. There are relatively few experimental results available for mobile WiMAX networks. Grondalen et al.  presented the measurement of throughput and physical parameters. They reported that their WiMAX system can deliver 9.6 Mb/s to a single flow in the downlink even at a distance of 5 km from the BS. Pentikousis et al.  conducted an experimental investigation of the network performance over a fixed WiMAX testbed. They employed multiple competing traffic sources over a point-to-multipoint topology and measure the network capacity. Although the multimedia services including VoIP and video streaming were applied in their experiment to generate traffic, they ignored the QoS from the viewpoint of end-users. Halepovic et al.  used experimental measurement to study the performance of VoIP and video streaming over a commercial fixed WiMAX network. However, they considered only the single-user scenario. In addition, their results, especially for the video streaming case, were more based on subjective evaluation, which was neither comparable nor applicable. Coelho et al.  reported a measurement campaign in a sub-urban area. Their study focused on signal coverage. Lin et al.  investigated the performance of a WiMAX-based VoIP established under a field trial program. The most related study was presented by Kim et al. . They conducted measurements over Wireless Broadband, a Korean version of mobile WiMAX system, for both system performance and single-user performance.
To the best of authors' knowledge, none of the past research conducts both VoIP and video streaming experiments in a real mobile WiMAX network. Moreover, we set up two kinds of experimental system for professional assessment and user-center tests. Note that the MAC protocol and scheduling policy are either proprietary to vendors or non-public to subscribers. Thus, different from most simulation-based researches, we treat the WiMAX card as a black box in order to make experiments more realistic.
3. Experimental environment
In this section, we briefly describe the experimental environment from the aspects of network layer and application layer.
3.1. Network environment
Alcatel-Lucent 9715 L-WBS characteristics
2.57 and 2.59 GHz
Fast Fourier transform (FFT) size
QPSK, 16-QAM, 64-QAM
Convolutional turbo code (CTC)
Multiple access method
Time division duplex (TDD)
5 ms, UL/DL = 1/2
Automatic Repeat reQuest (ARQ) and Hybrid ARQ
The Alcatel-Lucent 9740 works as WAC, which ensures session control and data transport functions. All 9740 WAC traffics are handled by IP protocol. The Alcatel-Lucent 9753 works as OMC, which provides centralized management function for all the elements belongings to the WiMAX access network. At the same time, OMC hosts servers such as Dynamic Host Configuration Protocol (DHCP), Domain Name System (DNS), and Network Time Protocol (NTP). As far as Authentication, Authorization, and Accounting (AAA) are concerned, beside its traditional operations, it attributes the appropriate service flow to the authenticated users depending on the service they are using and their subscription. UEs can be Mobile Subscription Station (MSS) or Costumer Premise Equipment (CPE). In the experiments, we have three kinds of UE: an Alcatel-Lucent PCMCIA card, a Sequans USB dongle, and a Zyxel CPE.
64-QAM over downlink
16-QAM over uplink
16-QAM over downlink
QPSK over uplink
IEEE 802.16e has implemented a full mobility support of handover. The Hard Handover (HHO) is the only one mandatory specified in IEEE 802.16e and supported by our testbed. HHO is easy for implementation, but it increases the end-to-end delay that is critical for the delay-sensitive services such as VoIP. In the configuration of our testbed, MSS starts the neighbor BS scanning process at 14 dB of CINR. To investigate the impact of HHO, an MSS moves from one BS to the other during the VoIP session or video streaming. Regarding QoS, five classes mentioned above have already been implemented by the Alcatel-Lucent equipment WAC. Unfortunately, our testbed was configured to support only BE in this stage. As a result, all experiments in this study are conducted under the QoS class of BE, even though BE is not originally designed for multimedia services. According to the QoS settings, the maximum data rate is limited to 4 Mbps on the downlink and 800 kbps on the uplink. To be noted, the reference networks in this article are the Ethernet-based campus network of Institut Telecom SudParis and the commercial ADSL network operated by France Telecom.
3.2. Application environment
In this article, we propose to use two kinds of test systems, the dedicated systems for professional measurement and the user-center systems for user-friendly assessment. The specific application environment will be described in the following subsections.
We focus on two aspects of VoIP performance. First, we evaluate the perceived voice quality. The International Telecommunication Union (ITU) recommends the Perceptual Evaluation of Speech Quality (PESQ) method standardized as ITU-T P.862 . PESQ requires the sent audio wav-file and the received wav-file as input and returns as result a value ranging from -0.5 (worst) to 4.5 (best). The degradation of voice quality has different causes such as codec and network, etc. In order to eliminate interference, we set PESQ over Ethernet as reference to other networks. Second, we show the network conditions in terms of delay, loss, and jitter.
3.2.2. Video streaming
There are two widely accepted protocols for video streaming, RTP over User Data Protocol (UDP) and Hyper Text Transport Protocol (HTTP) over Transmission Control Protocol (TCP). RTP/UDP is used extensively in communication and entertainment systems. The recent measurement studies indicate that a significant fraction of Internet streaming media is currently delivered over HTTP/TCP .
Possible PSNR to MOS conversion
We use YouTube for user-friendly video streaming experiment. YouTube is one of the most popular web sites for video sharing and streaming. In fact, the High Definition (HD) video 1080P is supported by YouTube. However, the HD version is accompanied with the bitrate up to 5 Mbps, which puts a big challenge to the access networks. YouTube uses HTTP/TCP to buffer video data to the flash player. The most critical issue is buffer-under-run, which results in video freezing. It substantially degrades the user experience very much. Therefore, we first evaluate the streaming performance by subjective assessment of buffer-under-run. We further measure and analyze the network parameters, throughput and delay, in different network environments. In this part of study, we use Firefox v3.6.6 where Adobe Flash Player 10 is integrated.
4. Performance studies
In this section, we analyze the test results. If no otherwise specified, the test results in above-mentioned cases, cell center, cell edge, and handover.
Skype measures and reports the call technical information including packet loss, Round Trip Time (RTT), jitter, etc. We observe that there is no packet loss in all test cases, even during HHO. It is reasonable because of the lower bandwidth usage and the reliable TCP used for speech media data in our tests. Considering the different mechanism of underlying protocols, we may find the quality degradation of Skype is different from that of Phoner. The latter is mainly affected by packet loss due to the unreliability of UDP. Whereas, jitter and delay are the main factors to the voice quality of Skype. The tests show that the jitter has more dynamic variation compared to RTT. This result confirms the conclusion in  that the jitter relative to delay has a significant impact. According to Figure 7, the voice quality over downlink is different from that of uplink. This could be due to the duplexing mode of TDD. The asymmetrical TDD ratio differentiates the jitters over downlink and uplink, and the differences of quality degradations between the cell edge and the cell center can be further explained that the worse radio conditions increase the retransmission of TCP at the cell edge.
4.2. Video streaming
Since H.264/MPEG-4 AVC is widely used, in this study we select ×264 as the video codec. For the codec settings, the Group of Picture (GoP) size is set as IPPP at 30 frames, and the frame rate is 30 Hz. Moreover, in order to focus on only the network performance, we do not activate any EC mechanism in the codec. The video sample Highway in the format of Common Intermediate Format (CIF) is encoded with constant bitrates in 400, 600, 1000, and 2000 kbps. The video sample City in the format of 4CIF is encoded in 3000, 4000, 5000, and 6000 kbps. These two samples are used for the tests of uplink and downlink, respectively. After being packetized, the video sample is streamed from the sender to the receiver over RTP/UDP.
In this article, we conduct comprehensive experiments of multimedia performance evaluation in a real mobile WiMAX testbed. Our test cases focus on the asymmetrical link, AMC scheme, and mobility of WiMAX. Specially, we employ multimedia traffics of VoIP and video streaming via uplink and downlink at the cell edge and the cell center, as well as handover. In general, the multimedia performance over WiMAX is good. We find that the VoIP quality at the cell center is perfect, where the value of PESQ exceeds 4. At the cell edge, the quality is degraded but still adequate. The performance of video streaming is consistently good with the bitrate as high as 5 Mbps through the downlink, and a smooth playback of YouTube 480P video is consistently provided. In spite of packet loss due to handover, the overall quality degradation is negligible. In the future, we will extend this study by enabling other QoS classes in the testbed. Furthermore, we will introduce real cross-layer optimizations to enhance the multimedia performance at the dynamic network condition of mobile WiMAX.
This study was sponsored by the POSEIDON project, the Knowledge Innovation Program of CAS (Grant Nos. Y022081131, Y022091131, and Y022101131), and Science and Technology Commission of Shanghai Municipality (Grant No. 09DZ1502100). We would like to thank all partners for their contributions. And also thank Mr. Robin Carnino and Mr. Franck Gillet for the valuable discussions.
- WINNER Project[http://www.ist-winner.org]
- Halepovic E, Ghaderi M, Williamson C: Multimedia application performance on a WiMAX network. The International Society for Optical Engineering 2009.Google Scholar
- Pentikousis K, Pinola J, Piri E, Fitzek F: An experimental investigation of VoIP and video streaming over fixed WiMAX. The 6th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks and Workshops, Berlin 2008, 8-15.Google Scholar
- POSEIDON project[http://poseidon.mag-projects.homeip.net/index.php/Accueil]
- IEEE Std. 802.16e-2005: IEEE Standard for Local and Metropolitan Area Networks--Part 16: Air Interface for Fixed Broadband Wireless Access Systems--Amendment 2: Physical and Media Access Control Layers for Combined Fixed and Mobile Operation in Licensed Bands. 2005.Google Scholar
- Grondalen O, Gronsund P, Breivik T, Engelstad P: Fixed WiMAX field trial measurements and analyses. The 16th IST Mobile and Wireless Communications Submit 2007.Google Scholar
- Coelho N, Cabral N, Pereira A, Rocha A, Navarro A: Mobile WiMAX field trials in a sub-urban area. IEEE International Symposium on Broadband Multimedia Systems and Broadcasting 2009.Google Scholar
- Lin Y-B, Sung Y-C, Rashvand HF, Liu C-L, Liao Y-J: Mobile-Taiwan experience in voice over IP-worldwide interoperability for microwave access. IET Commun 2011., 5(15):Google Scholar
- Kim D, Cai H, Na M, Choi S: Performance measurement over mobile WiMAX/IEEE 802.16e network. The International Symposium on a World of Wireless, Mobile and Multimedia Networks 2008.Google Scholar
- ITU-T Recommendation P.862: Perceptual Evaluation of Speech Quality (PESQ): an objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs. 2001.Google Scholar
- Wang B, Kurose J, Shenoy P, Towsley D: Multimedia streaming via TCP: an analytic performance study. The 12th ACM International Conference on Multimedia, New York 2004, 908-915.Google Scholar
- Klaue J, Rathke B, Wolisz A: EvalVid--a framework for video transmission and quality evaluation. The 13th International Conference on Modelling Techniques and Tools for Computer Performance Evaluation 2003, 255-272.View ArticleGoogle Scholar
- Chen K, Huang C, Huang P, Lei C: Quantifying skype user satisfaction. The Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications 2006.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.