Multi-domain SDN controller federation in hybrid FiWi-MANET networks

Abstract

Traditionally, mobile ad hoc networks (MANET) and hybrid optical-wireless networks, also known as fiber-wireless (FiWi), have been considered disjointly and independently, with no synergic management solutions. The former has primarily focused on dispatching packets among mobile nodes in infrastructure-less and very dynamic environments, the latter on offering high-bandwidth and low-latency access to cellular-equipped mobile nodes. The recent advancements and penetration of software-defined networking (SDN) techniques have stimulated the adoption of SDN-based flexible monitoring and control also for the MANET and FiWi domains. In this perspective, the paper originally proposes a model and an architecture that loosely federate MANET and FiWi domains based on the interaction of MANET and FiWi controllers: the primary idea is that our MANET and FiWi federated controllers can seldom exchange monitoring data and control hints the one with the other, thus mutually enhancing their capability of packet management over hybrid FiWi-MANET networks. Our model is applied to several relevant use cases to practically point out the benefits of the proposal in terms of both load balancing and fairness improvements.

1 Introduction

To increase the quality of service (QoS) perceived by mobile users, many industrial and academic research activities are pushing for two primary goals: high performance and dynamic management. In particular, mobile and wireless networking is achieving relevant results towards two directions, each one considering specific goals: on the one hand, large bandwidth and wide coverage by providing Internet access via infrastructure-based hybrid optical-wireless networks; on the other hand, dynamic and flexible connectivity by supporting peer-to-peer services via (partially) infrastructure-less multi-hop mobile networks, e.g., mobile ad hoc networks (MANETs) and vehicular ad hoc networks (VANETs), also based on spontaneous sharing of computing/storage/networking resources provided by mass market off-the-shelf smartphones/devices.

The two directions are typically considered as separated domains and managed in a silos-based fashion. Hybrid optical-wireless networks, also known as fiber-wireless (FiWi) networks, aim at maximizing the offered bandwidth to cellular-equipped mobile devices, considering as a block-box anything beyond the identity of connected mobile devices, e.g., without any awareness of the applications running on mobile devices or even of the presence of other mobile devices that may potentially interact, also in a multi-hop spontaneous way. On the contrary, spontaneous MANETs [1] aim at supporting multi-hop dispatching of packets among devices that join/leave the network very dynamically, as well as the flexible offering/invoking of peer-to-peer services, by considering as a black-box the access network that provides them with Internet connectivity. In fact, mobile devices can traditionally gather very little information about the access network status, with no possibility to directly configure/tune it.

This silos-based approach has facilitated the development and management of the associated networks. In addition, since FiWi and MANET domains are typically managed in a different way (via traffic engineering teams of network operators the former, in a completely decentralized and individual fashion the latter), considering one another as a black-box was a sound and simplifying assumption for integrated network management. However, nowadays, this sharp separation does not allow to fully exploit the potential of some novel scenarios demanding for both performance and flexibility, such as the following:

• Edge networks in Internet of things (IoT), where several (usually fixed) devices with sensing and actuation capabilities are inter-connected, by forming a multi-hop local network to dispatch and refine raw data as well as to deliver commands among nodes. In addition, some nodes are cellular-equipped to dispatch packets back and forth to cloud-based software components via the Internet. In this scenario, FiWi-MANET federated management may allow to dynamically reroute traffic among MANET nodes based also on the current load of the employed FiWi access network(s);

• Sparse Smart Cities [2], characterized by users willing to collaborate by allowing new applications to be deployed on their smartphones to remotely monitor and control fixed/mobile devices. For instance, notable contributions in the recent literature exploit node collaboration to support better content delivery [3] and better distribution of computational tasks [4]. In this case sparse nodes collaborate with one another while moving; users access different IP subnets, by increasing the probability of spreading packets towards their destinations. In this perspective, MANET and FiWi domains can benefit from coordinating one another to minimize performance degradations generated by frequent mobile node movements (and related disconnections/reconnections).

In addition, let us note that in such scenarios, there is the need to consider end-to-end capabilities and requirements in line with recent 5G guidelines. According to the latest specifications published by 3GPP about the service requirements for the 5G system [5], the user equipment (UE) should be able to support one or both of the following connectivity models: traditional direct network connection and/or indirect network connection based on other UEs used as relays. Involved UEs can be anything from legacy mobile devices to various types of smart devices and sensors that operate in the context of IoT. The relay devices are expected to be able to access the network over 3GPP, through a wireless local area network (WLAN) or via personal area network (PAN). Under the indirect network connection scheme, a UE should support discovery of relays in proximity, forming that way a MANET. Moreover, the specific connectivity model needs to allow connections in a visited public land mobile network (VPLMN), when remote and relay UEs are subscribed to different PLMNs with a roaming agreement. Further related considerations that have to be made include battery optimization and subscriber identity hiding of the relay UE. Hence, it becomes evident that the 5G vision dictates holistic management of all involved network domains, from the core optical and the wireless access segments to the interconnected MANETs formed by end-users in an ad hoc manner.

The recent embracing of the software-defined networking (SDN) approach has vividly changed how the two domains are managed. For example, the availability of SDN controllers for FiWi networks [6] enables management mechanisms that can be used even from outside the access network itself, with proper techniques for ensuring privacy of mobile users and correct management of network capabilities. At the same time, the availability of an SDN controller in spontaneous networks allows to achieve a centralized management point of view of the decentralized multi-hop network as well as to more easily control it by adopting some mechanisms typical of traffic engineering [7].

By originally considering SDN techniques plus hybrid FiWi-MANET networks, this paper proposes a novel model to maximizing the QoS perceived by final users in terms of load balancing and fairness, by considering the state and requirements of both FiWi and MANET domains and by managing/controlling them accordingly. In particular, we claim that to take full advantage of both networks, there is the need to foster the interaction of the two domains by supporting the loosely federation of SDN controllers. In fact, while we agree that the two domains should be managed by different SDN controllers with typically differentiated and specific optimizations, we believe that enabling the exchange of information and control data between them can relevantly improve their management effectiveness, while keeping low the needed growth of complexity.

In particular, the proposed model is based on distributed and different MANET and FiWi SDN controllers. The former appropriately reroutes packets among mobile nodes by considering the current MANET state, application requirements, and the access network load advertised by the FiWi SDN controller. The latter detects eventually overloaded cellular base stations (BSs) and reroutes traffic towards less loaded ones, by also informing the MANET SDN controller to appropriately manage the MANET data plane to dispatch packets to the destination mobile node.

In addition, also to clarify the exact positioning of the proposal if compared with the existing literature in the field, the paper provides an original contribution in terms of a novel taxonomy of state-of-the-art SDN approaches, by classifying them on the basis of their possible distribution, controller integration, management openness, and domain heterogeneity. This taxonomy also allows to clarify the need for further research activities in the area of multi-domain SDN controller federation in hybrid FiWi-MANET networks.

The remainder of the paper is structured as follows. Section 2 describes our novel taxonomy for SDN controllers, while Section 3 sketches the background needed for the full and easy understanding of our proposed model for federated SDN controllers for hybrid FiWi-MANET networks (Section 4). Application use cases and traffic management modeling show how the proposed model and architecture can apply to some practical scenarios of growing relevance. Some practical implementation insights and challenges, as well as conclusive remarks and directions of ongoing work end the paper.

2 A novel taxonomy for state-of-the-art SDN controllers

At first, SDN has emerged primarily to manage switches of closed and geographically centralized environments such as datacenters and department networks via the OpenFlow protocol [8], the de facto standard supported by networking industrial-grade devices. Readers interested in a general presentation of the SDN architecture, together with a thorough survey of SDN solutions applied to optical networks, are encouraged to refer to [9].

In particular, the centralized nature of the SDN approach makes it the natural choice for managing networks of small-to-medium size related to a single organization. However, the adoption of the SDN approach has quickly proven its benefits also in different scenarios with more relaxed requirements in terms of closeness and geographical centralization. For instance, SDN is exploited in wide area networks to efficiently interconnect different datacenters [10, 11], eventually based on a multi-controller SDN architecture [12]. In addition, the SDN approach has been proposed also in scenarios differing from traditional datacenters, such as MANETs [13, 14], vehicular networks [15], naval systems [16], and access/transport networks [6, 10, 17].

Based on these considerations and with the main purpose of providing a wider point of view of the state-of-the-art, we propose to model the literature along four primary directions: location distribution, controller integration, management openness, and domain heterogeneity.

Location distribution refers to the geographical spreading of managed devices, deployed either in a relatively small area such as a single datacenter or scattered in multiple location such as different and geographically distant departments of the same organizations.

Controller integration focuses on how multiple SDN controllers, e.g., each one specialized for a different location, interact on each other to take proper decisions. For instance, SDN controllers can be federated in a tight fashion with a hierarchical architecture or loosely coupled in a peer-to-peer fashion.

Management openness identifies if there is only one organization in charge of managing the target environment (typically delving into close scenarios), such as a single university, or if multiple operators collaborate to provide the service, for instance multiple datacenters of different vendors (with higher degrees of openness). Note that in the latter case, it is required that each organization provides some sort of controllability of their own environments to other organizations.

Domain heterogeneity refers to the environments that are managed within the same SDN-based solution that can be of the same or different types. For instance, in case of multiple datacenters federated via SDN, there is no domain heterogeneity, while in case of an access network and a datacenter jointly managed by the same operator, there is domain heterogeneity.

Based on the four directions above, it is possible to identify some primary research activity directions (see Table 1):

• Single datacenter: this is the simplest case where a single datacenter is managed in an SDN-based approach to optimize the intra-datacenter traffic, either by adopting a single centralized SDN controller or multiple SDN controllers;

• Multiple datacenters with a single centralized controller: in this case, there are multiple datacenters in different locations, typically managed by a single cloud provider, and a single SDN controller monitoring and optimizing the whole inter-datacenter traffic. Note that there could be multiple intra-datacenter SDN controllers, each one managing the traffic of a single datacenter, but the traffic among datacenters is engineered by only one SDN controller;

• Multiple datacenters with distributed controllers: similarly to the previous case, there are multiple datacenters in different locations managed by the same cloud provider. However, each datacenter has its own SDN controller managing the inter-datacenter traffic. SDN controllers are tightly integrated, coordinating one each other to optimize inter-datacenter traffic, e.g., either in a peer-to-peer fashion or adopting a hierarchical architecture, in the latter case with an SDN controller taking final control decisions;

• Federated datacenters: multiple cloud providers manage their own datacenters in an independent fashion, but the traffic among datacenters of different cloud providers is engineered by adopting an SDN approach, typically by loosely coupling SDN controllers in a peer-to-peer fashion. In this case, there is the need of opening the borders of datacenters by allowing external cloud vendors to control (or at least to influence) how incoming and outgoing traffic is managed;

• Inter-domain operator: only one operator, e.g., a telco company, is in charge of managing heterogeneous domains, e.g., both a datacenter and an access network. Typically, multiple SDN controllers (each one dedicated to a single domain) are tightly integrated and interact to optimize the end-to-end QoS;

• Federated domains: multiple actors are in charge of managing heterogeneous domains with different loosely integrated SDN controllers interacting to optimize the end-to-end QoS. For instance, telco operators, service providers, and users manage datacenters, access networks, MANETs/VANETs respectively with different SDN controllers interacting in a peer-to-peer fashion.

Table 1 Primary clusters of the SDN literature based on the proposed model

The traditional adoption of SDN in datacenters is clearly evidenced by the many solutions already available in the literature. For instance, considering the network optimization of single datacenters, Cziva et al. [18] provide an SDN-based solution optimizing intra-datacenter virtual machine (VM) migration by jointly considering server and network resources. Son et al. [19] adopt the SDN approach to optimize the energy consumption of a datacenter by dynamically consolidating traffic to fully exploit only part of network resources. Lu and Zhu [20] exploit SDN in datacenters to optimize TCP based on information provided by OpenFlow-enabled switches. Xie et al. [21] consider a more articulated scenario characterized by multiple SDN controllers in charge of virtualizing a single datacenter, by addressing the three primary issues related to minimal coverage, minimal fault-tolerant coverage, and minimal communication overhead among controllers.

Considering the adoption of SDN to improve the QoS of inter-datacenter traffic, some solutions adopt a centralized architecture with a unique controller. For instance, software-driven WAN (SWAN) [22] improves inter-datacenter network exploitation by centrally tuning when and how much traffic services generate and re-configuring the data plane in relation to actual traffic demand. Wu et al. [11] address the issue of managing bulk data transfers among geo-distributed datacenters hosted by a single cloud provider. To this purpose, they propose that a central SDN controller gathers information from distributed gateways to optimally schedule inter-datacenter transfers based on per-chunk routing choices. However, many solutions adopt multiple SDN controllers to manage inter-datacenter traffic, such as multiple datacenters with distributed controllers. To this purpose, Ahmed and Boutaba [12] present multiple possible architectures, e.g., with either horizontal or vertical multi-controller architecture and based on either in-band or out-of-band communication.

The federation of datacenters based on distributed SDN controllers deployed and managed by different operators is considered by Levin et al. [23], identifying as main requirements supporting the capability of inter-operating among heterogeneous SDN controllers, VM migration among clouds in a seamless fashion, and cross-cloud virtual networks that allow tenants to transparently exploit multiple datacenters. CHIEF [24] supports the interoperability of community networks based on a controller farm managing the federation of SDN controllers, with the goal of enabling very large horizontal scalability, tenant isolation, and additional services such as billing. It is also interesting to note that the research activity in the SDN field is also pushing for the deployment and joint management of large testbeds, eventually distributed in multiple locations [25]. Finally, Risdianto et al. [26] present how the adoption of open-source software can leverage the federation of SDN and cloud environments under separate administrative domains.

More recently, the state-of-the art literature is moving towards the adoption of SDN in other domains. Alvizu et al. [17] present a survey modeling the state-of-the-art literature about SDN-based solutions managing heterogeneous transport networks based on monolithic, hierarchical, and flat or mesh control plane architectures. Considering general purpose wireless networks, Abolhasan et al. [27] propose to extend the SDN approach towards a centralized/distributed mixed architecture, with a centralized SDN controller gathering and pre-processing information and several distributed nodes (typically BSs providing connectivity) performing decision-making and configuring the data plane of mobile nodes. Focusing on wireless sensor networks, Zhou et al. [28] exploit the SDN approach to efficiently manage cooperative communication and task execution among nodes. Some solutions in the wireless sensor network domain not only adopt the SDN approach, but also exploit the OpenFlow-like protocols. For instance, SDN-WISE [29] extends OpenFlow to optimize the communication among sensor nodes and the controller and to program nodes as finite state machines. Anadiotis et al. [30] adopt SDN-WISE to optimize the deployment of MapReduce tasks among nodes, with the controller in charge of actuating the data plane of nodes to better route traffic from mappers to reducers. Luo et al. [31] propose an extension (but backward compatible) of OpenFlow to improve its flexibility, making it more appropriate for the inherent dynamicity of wireless networks. Lai et al. [32] and Fontes et al. [15] exploit SDN to optimize vehicular networks, e.g., to deliver multimedia streams.

Even the SDN-based federation of heterogeneous domains is recently emerging, but usually in specific environments. For instance, Yu et al. [10] consider both datacenter and optical network domains, by adopting a multi-controller collaboration framework in charge of managing not only network devices but also cloud-based storage and computing resources. Similarly, Zhao et al. [33] jointly manage datacenter resources, optical networks, and IP-based networks in a unified control system providing available capabilities based on a unified resource description model. Finally, let us stress that we envision that SDN-based federation of heterogeneous domains will gain additional attention, based on the easier management of heterogeneous resources the SDN approach leverages.

3 Background on FiWi and MANET

As already stated, this paper originally focuses on the challenging issues raised by the integration of hybrid FiWi and MANETs through federated SDN controllers. In particular, we believe that providing (sub-)optimal QoS management by jointly considering the capabilities of access networks and MANET nodes represents the primary aspect that has to be addressed, while we intend to integrate the management of federated datacenters in our future research activity. Before presenting our proposed framework in Section 4 and to facilitate its full and easy understanding, the section outlines the main characteristics of the two target FiWi and MANET domains.

3.1 Hybrid optical-wireless access networks

The integration of optical and wireless networks provides a cost-effective and flexible access network, which combines the huge bandwidth potential of the optical domain, in the backhaul, and the advantageous characteristics of the wireless networks, in the fronthaul, such as mobility, reachability, roaming, and mobile services provisioning. In essence, the integration of optical and wireless domains in a single access network defines a FiWi access network, which is divided into two main categories based on the level of integration, namely Radio over Fiber (RoF) and Radio and Fiber (R&F). While the RoF concept has low practical value since it entails complex PHY operations such as converged modulation, coding, and transmission, R&F seems to be much more functional. R&F paradigms allow flexible architectures without imposing serious modifications in each domain. As a result, efficient and cost-effective topologies are feasible allowing an effective way of converging multiple types of optical with various wireless or cellular technologies [34].

The R&F architecture comes with two main paradigms in the literature: optical-wireless mesh networking and optical-wireless (broadband) access networking (or hybrid optical-wireless access networking). In the former case, several wireless routers and a number of gateways are connected to an optical unit, e.g., to the optical network unit (ONU) in the case that a passive optical network (PON) is used as the main technology for the backhaul of the network, and thus to the network backbone and the Internet. It is worth mentioning that this kind of hybrid network introduces a routing sub-network at the edges, where multiple wireless nodes (smartphones, sensors, IoT devices, vehicles, and anything that is considered mobile and is distinguished with an IP address) are indirectly connected to the optical backhaul through multiple gateways and relay wireless links. In the latter case, optical-wireless access networks employ multiple users and nodes that are connected to a hybrid BS which is equipped with two interfaces, the optical interface that terminates the optical fiber and the wireless interface that provides a 4/5G radio interface (cell, macrocell, or picocell). Details about the components of a hybrid optical-wireless FiWi architecture are provided in [35].

In the context of this paper, the former paradigm is adopted where multiple mobile users (or nodes) are connected with each other in an ad hoc basis. The optical domain in the fronthaul is a PON infrastructure where various PON technologies could be used, i.e., Ethernet PONs, Gigabit PONs, or multi-wavelength PONs. In the central office (CO) premises is deployed the optical line terminal (OLT), the main decision-making component of the optical domain. Next, the OLT is connected directly with the passive splitter/combiner via optical fiber, thus, single or multiple wavelength light-paths are created between the CO and the edge of the optical network. As a result, a cost-effective topology, mostly a tree topology, is realized that engages several merits such as low maintenance, protocol transparency, and low operation cost. At the edge of the optical network, the conventional ONU, which is used in pure PONs, is replaced by the enhanced ONU-BS which consists of two interfaces, namely an optical interface that interconnects the ONU-BS with the OLT through optical fiber and a radio interface, e.g., a long-term evolution (LTE) radio access network. For instance, the architecture proposed in [6] introduces an ONU-eNB, where the optical interface supports an XG-PON system, while the radio interface supports an LTE network. The evolved packet core (EPC), as part of the LTE radio technology, is located at the CO. Its architecture separates the user data (user plane) and the signaling (control plane) to make the scaling independent. Thus, telecom providers and operators could handle the channel and (cellular) network configurations easily. In this way, two directions are defined, the downstream direction supporting 9.95328 Gbps from the CO to the ONU-eNB and the upstream direction which supports either 2.48832 Gbps or even 9.95328 Gbps in a symmetrical fashion.

Effective traffic engineering in FiWi networks is crucial for the provision of advanced quality of experience (QoE) to users of hybrid next-generation networks. A lot of interest was recently attracted to resource allocation techniques both at the optical as well as at the wireless domains. Most of the related research endeavors adopt the dynamic bandwidth allocation (DBA) approach to improve QoS and energy efficiency in FiWi networks [36]. Another relevant factor, which has a high impact on traffic engineering, is resource allocation fairness, which is the focus of the work in [37], while network performance is maintained at high levels. Balancing fairness in bandwidth distribution with network efficiency is also the primary aim of the DBA scheme proposed in [38], which targets XG-PONs. At the wireless access domain, a number of techniques of issues for the provision of QoS in heterogeneous wireless networks are presented in [39]. Game theory has arisen as a promising approach for fair resource allocation to mobile stations, as shown in [40] which introduced a related algorithm. Energy efficiency is apparently of high importance for the autonomy of mobile devices and a major consideration of modern bandwidth distribution schemes, such as the medium access control protocol proposed in [41].

Finally, the integration of traffic engineering techniques traversing across the optical core and the wireless access domains renders as an important challenge, which is currently being addressed by the research community. Authors in [42] have devised a holistic resource allocation solution in optical-wireless networks, focusing on balancing fairness and efficiency across WiMAX and 10-EPON sectors. A key aspect of this work is mapping service classes between optical and wireless domains in an effort to provide end-to-end QoS support. In more detail, unsolicited grant service (UGS), real-time polling service (rtPS), and best effort (BE) traffic services of WiMAX are mapped to expedited forwarding (EF), assured forwarding (AF), and BE classes of 10G-EPON, respectively. The bandwidth distribution process between the OLT and ONU-BSs is realized via the multi-point control protocol (MPCP), which involves the GATE and the REPORT control messages. The former is used by the OLT to assign transmission opportunities to ONU-BSs, whereas the latter is used by ONU-BSs to inform the OLT about its buffers’ size and to ask for bandwidth allocation in the following frame. The algorithm efficiently balances fairness and performance through an optimization scheme.

3.2 Spontaneous MANETs

The most relevant and specific property of multi-hop spontaneous networks is that they originate from the willingness of social interactions of people via impromptu interconnections of the personal devices they carry, e.g., smartphones, tablets, and laptops [43]. In spontaneous networks, devices discover and interact with one another opportunistically and without any prior mutual knowledge, by exploiting all supported connectivity opportunities, e.g., Wi-Fi or Bluetooth ad hoc links and LTE infrastructure-based ones [1, 44]. In particular, group-related behavior and the ever-increasing willingness to share rich user-generated contents, also pertaining to the personal sphere, calls for a user-centric communication paradigm shift, where the ad-hoc interconnection of portable devices plays a central role. On the one hand, the user-centric nature of spontaneous networking partially relaxes the constraint of having infrastructure-based communication support (e.g., anywhere cellular coverage, which is often expensive). On the other hand, it naturally yields to very heterogeneous, uncoordinated, and dynamic networking environments where, for instance, any node can create its self-administered layer2 links. In addition, spontaneous networking nodes are expected to be able to take advantage of simultaneous exploitation of different communication interfaces to join/create multiple IP networks (via either ad hoc or infrastructure connectivity); these networks are autonomously created, configured, and destroyed by collaborating users in a completely decentralized way.

It is worth noting that, traditionally, spontaneous network nodes take management decisions based on their limited scope visibility and without a global knowledge of network topology/conditions, by typically reacting to modifications in local resource availability. In fact, also because of its general purpose and collaborative nature, spontaneous networking has usually focused on simplifying the dispatching of packets at multi-hop distance, eventually aiming at improving the QoS with a per-application view [45]. From a wider point of view, the efficient application and optimization of traffic engineering techniques have not been a primary topic in the MANET research area. While state-of-the-art literature recognizes the importance of improving QoS in MANET scenarios, traditional traffic engineering solutions based on strictly enforced resource allocation can be hard to adopt, in particular because of the general-purpose nature and the limited resources available over MANET nodes. In other words, MANET QoS has been mainly addressed so far by only considering localized visibility and decisions, e.g., based on link and path performance [46]. Pease et al. [47] proposed a middleware solution to support timely MANET communications based on an adaptive approach where their solution reacts to dynamic network conditions by switching the employed channels to ensure their optimal and robust exploitation. Abuashour and Kadoch [48] explored VANET QoS issues: in particular, they aim at increasing path stability and throughput, while reducing delay, by selecting cluster heads based on vehicles’ lifetime. Li and Shen [49] focused on hybrid networks (MANET nodes plus a wireless infrastructure), by exploiting anycast communication and by modeling packet routing issues as resource scheduling problems.

Differently from what already available in state-of-the-art literature, we claim that the SDN approach very well fits the dynamic and heterogeneous nature of spontaneous networking. On the one hand, since spontaneous networking nodes interact to offer and access services in a collaborative manner, there is no a priori knowledge of service availability. Thus, it is suitable to have a centralized point of view with full visibility, able to take proper control decisions. On the other hand, spontaneous networking nodes are willing to further cooperate to improve QoS by better exploiting the currently available networking opportunities. In fact, based on their limited visibility of the network, competing applications/nodes may exploit the same (apparently best) multi-hop path, while erroneously neglecting alternative paths that could be preferred because of more limited load. In other words, we claim that the adoption of the SDN approach in spontaneous networking allows to gain deeper knowledge of the available topology and of its state, as well as to consider application requirements and to adapt packet dispatching mechanisms accordingly. Additional information on our mechanisms and strategies for SDN-based management of spontaneous networks can be found at [7].

4 Proposed methodology

As already anticipated, our target environment consists of a FiWi domain (based on fixed hybrid optical-wireless access networks) and a MANET domain (based on multi-hop spontaneous networks). The former provides Internet wireless access via cellular technology while ensuring huge bandwidth, whereas the latter enables infrastructure-less multi-hop communication among mobile nodes via, e.g., Wi-Fi Direct and Bluetooth. Mobile nodes exploit the multi-hop MANET to interact with one another by offering and invoking services in a peer-to-peer fashion without any need of Internet connectivity, e.g., to stream multimedia content among MANET participants. In addition, some of the mobile nodes have Internet connectivity via cellular technology and can share it with other neighbors, e.g., without cellular capabilities. The generated traffic can be either complete within the MANET, i.e., involving only mobile nodes and not reaching the FiWi access network, or traversing the FiWi, i.e., generated/received by a mobile node but also traversing the fixed FiWi access network.

We claim that, to the purpose of effective QoS management, there are significant benefits in deploying two different but cooperating/federated SDN controllers, i.e., the FiWi SDN controller and the MANET SDN controller, each one in charge of managing only a specific domain. Each SDN controller, by default, separately manages traffic either within the MANET or in the FiWi network. In this manner, it is possible to ensure QoS while limiting the associated complexity/overhead, since each SDN controller is in charge of collecting information and managing nodes of only a subset of the whole network. Thus, usually SDN controllers manage their own domain in an uncoordinated and independent fashion, each one with silos-based visibility. On the one hand, the FiWi SDN controller evaluates uplink-downlink traffic in an aggregated fashion, without distinguishing among packets related to different mobile nodes or different flows of the same mobile node, with full compliance with 5G communication standards. On the other hand, the MANET SDN controller manages traffic in its multi-hop wireless domain by taking advantage of the greater flexibility that the MANET overlay provides, e.g., by allowing to fully control how mobile nodes dispatch packets/flows to their neighbors and also by easily permitting to modify/upgrade the adopted management mechanisms and policies.

4.1 Federated FiWi-MANET architecture

Figure 1 presents an overview of our proposed FiWi-MANET architecture. The FiWi domain is divided into three parts: the central office (CO), the backhaul/fronthaul domain, and the radio access domain. Note that the exact structure of the backhaul/fronthaul domain depends on the location where the backhaul data are decoded, so it is technology-specific. At the edge of the network, mobile nodes form a multi-hop spontaneous network (denoted as MANET) that can be regarded as an independent domain, locally able to dispatch packets without exploiting the cellular infrastructure.

Fig. 1

FiWi and MANET domains combined into our federated architecture

The FiWi domain consists of the OLT (located at the CO), the powerless passive splitter/combiner, and hybrid optical-wireless units at the fronthaul. Hybrid optical-wireless units are the evolution of the integration of the traditional ONU, which terminates the PON by providing optical interface to the final users (realizing the Fiber To The x—FTTx paradigm), and the BS which can be a traditional 4G BS (a WiMAX BS or an LTE evolved NodeB) or a 5G cloud radio access network (C-RAN) BS with remote radio head (RRH). In the former case, various integration methods of the ONU and BS can be applied such as independent, hybrid, combined, unified, and microwave over fiber (MoF) architecture [34].

In the context of this work, a simple integration method, such as the independent architecture, is adopted to form an integrated framework that allows ONUs and BSs to be connected independently, e.g., by using a single (bridging) cable. For example, assuming that both devices apply a common protocol such as Ethernet as the common interface protocol, no special requirements are needed and no additional complex hardware is necessary. Moreover, the recent proliferation of the C-RAN concept [50, 51], which was first introduced by IBM [52] and is based on the concept of distributed wireless communication systems, established the separation of the BS in a radio unit, called RRH, and a signal processing unit, called baseband unit (BBU). While RRH offers the interface to the fiber and performs digital processing, e.g., digital to analog conversion and vice versa, the BBU equipment can be placed in a more flexible way, meaning that it is not required to be co-located with the BS. As a result, a cost-effective solution is provided where rental and maintenance costs are reduced. In this way, traditional ONU-BS devices are replaced by modern RRH and BBU units towards the integration of the optical backhaul network with latest 5G standards. In addition, traditional C-RAN with RRHs is further evolved by upgrading the concept of BBU deployment. While RRH is placed together with the antenna at the fronthaul of the FiWi network, BBUs that serve many RRHs are co-located in a pool. This pool can be located far away from the RRH location such as the CO premises. As a result, energy and cost savings are permitted while the control and data planes can be smoothly separated. In the light of the aforementioned aspects, Fig. 1 shows how traditional (e.g., 4G LTE) and the evolved (e.g., 5G C-RAN) technologies can be co-located in the same federated network.

4.2 Application use cases and primary technical challenges for FiWi-MANET federation

This section presents how the joint management of FiWi and MANET domains based on federated SDN controllers can improve the overall QoS by achieving better load balancing and higher fairness.

4.2.1 Suitability of federation for load balancing purposes

Considering load balancing, Fig. 2 outlines the case of the detection and management of a traffic flow coming from the Internet and directed to a mobile node. At first, the traffic flow is dispatched to the destination node towards the shortest path but involving the already heavily loaded RRH unit on the center. After a while, the FiWi SDN controller detects it and informs the MANET SDN controller that the traffic flow will be shortly redirected towards the bottom RRH unit. Then, the MANET SDN controller selects an LTE-equipped mobile node connected to the bottom RRH, e.g., based on current traffic and CPU load, appropriately manages the MANET data plane from the selected mobile node to the destination one, and replies to the FiWi SDN controller with the network address of the LTE-equipped mobile node connected to the bottom RRH unit in charge of receiving the flow from the access network. Finally, the FiWi SDN controller reroutes the traffic flow towards the longer but less loaded bottom path.

Fig. 2

Load balancing scenario

In this example, it is possible to note that FiWi and MANET SDN controllers interact and collaborate by sharing some control plane information and (indirectly) triggering data plane modifications in each other’s control plane. In fact, the former selects the less loaded RRH unit based on information only available in the FiWi domain. Then, the latter selects the less loaded LTE-equipped mobile node (among the ones connected to the given RRH unit) based on control plane information only available in the MANET domain. Finally, the MANET SDN controller appropriately manages its own data plane and notifies the FiWi SDN controller about the LTE-equipped mobile node it has to reroute the traffic flow to.

Delving into finer details, we identify two primary use-cases where the federation and collaboration of MANET and FiWi SDN controllers can relevantly increase the QoS by achieving load balancing: (i) an application on-top-of a mobile node makes available its application-specific requirements and (ii) an unusually huge traffic flow traverses the FiWi network. These cases can be addressed with either a proactive application-driven or a reactive access network-driven approach (Figs. 3 and 4).

Fig. 3

Proactive application-driven sequence diagram

Fig. 4

Reactive network-driven sequence diagram

In case of proactive application-driven inter-domain management, an application running on top of a mobile node proactively declares that it is going to generate a traffic flow, i.e., before packets are actually transmitted. To this purpose, the application:

• 1. Informs the local control agent (CA) about the type of traffic it is going to generate, e.g., multimedia stream with a given bitrate or file transfer of a given size;

• 2. The local CA informs the MANET SDN controller;

• 3. The MANET SDN controller informs the FiWi SDN controller, the latter replying with information about the best (e.g., since less loaded) RRH (ONU-BS) unit to exploit;

• 4. Then, the MANET SDN controller manages the MANET data plane, e.g., to route the forthcoming traffic flow towards an LTE equipped, not currently overloaded mobile node currently served by the indicated RRH (ONU-BS) unit;

• 5. At the same time, the FiWi SDN controller manages the backhaul to maximize the QoS of the forthcoming traffic flow, e.g., by making the appropriate bandwidth slots reservations in the PON, which interconnects the gateway RRH (ONU-BS) to the central OLT, as well as efficiently allocating OFDMA (orthogonal frequency division multiple access) resources at the corresponding LTE eNB.

It is worth noting that, since applications explicitly indicate their own requirements, MANET and FiWi SDN controllers are able to finely tune the data plane. In fact, by specifying application requirements of single flows, the management of forthcoming packets can be optimized. For instance, in case of long lasting multimedia streams with limited bitrate it is better to exploit low latency and reliable paths. On the contrary, to transfer huge files it would be better to exploit paths with broad bandwidth, even if available only for a relatively small time period [53].

In case of reactive access network-driven inter-domain management, a traffic flow is reactively detected by the FiWi SDN controller, i.e., the traffic is not advertised by any mobile node. In this case,

• 1. The FiWi SDN controller detects that some notable traffic with a given flow id has just started traversing the access network towards the MANET;

• 2. The FiWi SDN controller informs the MANET SDN controller about the traffic flow also specifying the destination mobile node;

• 3. Then, the MANET SDN controller notifies the FiWi controller about one or more LTE equipped mobile nodes that could efficiently receive the traffic and dispatch it towards the destination mobile node;

• 4. Finally, the FiWi SDN controller reroute the traffic towards one of the LTE-equipped mobile nodes suggested by the MANET SDN controller.

Let us stress that in this approach the destination mobile node is completely unaware of the data plane management, since every traffic engineering procedure is done by MANET and FiWi SDN controllers in a completely transparent fashion. However, since in this case there is no a priori awareness of application requirements, it is possible to manage traffic only in a reactive fashion.

4.2.2 Suitability of federation for fairness purposes

The scenario in Fig. 5 tries to highlight the benefits of our proposed model and architecture to the purpose of fairness. Assuming that each RAN defines an area of mobile and ad hoc nodes, an interesting challenge emerges in case it is required to fairly distribute bandwidth among the different areas, given that the allocated bandwidth is controlled by the backhaul (optical) network. This challenge is also in line with the latest recommendations of the Telecommunication standardization sector of the International Telecommunication Union (ITU-T) regarding the next-generation PONs (NG-PONs), where the provisioning of fairness in allocating bandwidth to the various ONUs is of paramount importance. In other words, the provisioning of fairness in the different MANET areas, as marked in Fig. 1, highly depends on the bandwidth allocation decisions of the OLT. Recent advances in that direction [42]indicate that the fairness provisioning is strongly influenced by the number of mobile devices that are connected in each cell. Hence, considering that MANET nodes are either directly or indirectly connected to cells, the fair bandwidth distribution in MANET areas is highly affected by the bandwidth allocated to each cell. As a result, SDN controllers at MANETs could monitor the traffic received and inform corresponding FiWi SDN controllers for keeping a fair bandwidth distribution among MANET nodes (see Section 5 for details about our MANET SDN controller architecture).

Fig. 5

Fair bandwidth allocation scenario among various MANET areas

Cooperation between RRHs (ONU-BSs) and MANET nodes is also needed to support QoS in terms of high-throughput, low latency, and dynamic traffic scheduling. Figure 6 depicts a case where the uplink-downlink ratio of the RRH (ONU-BS) devices is configurable, allowing traffic engineering capabilities in the underlying FiWi-MANET federated network. Considering that both WiMAX and LTE technologies allow the dynamic configuration of the uplink-to-downlink ratio, dynamic decisions on adjusting the ratio in RRH (ONU-BS) devices enable better resource allocation in cells and MANET areas based on the ongoing data traffic conditions. For example, [6] introduces an optical-wireless access architecture where SDN controllers monitor the traffic conditions in both optical and wireless domains to dynamically adjust the uplink-to-downlink configuration of the LTE framing in ONU-eNB. This is beneficial for the whole network, since it can determine the most appropriate uplink-to-downlink ratio, e.g., by using a machine learning algorithm. The specific concept is also beneficial for the MANET network in the introduced FiWi-MANET architecture since traffic conditions in MANET nodes could be also recorded using SDN controllers. Therefore, the MANET traffic condition information can be also used in RRHs as a feedback towards adjusting the uplink-to-downlink ratio, in a different way in each RRH based on the ongoing downlink and uplink traffic loads.

Fig. 6

Traffic engineering paradigm by adjusting the uplink-to-downlink ratio in the fronthaul network to support traffic engineering in the MANET domain

5 Results on modeling and managing traversing traffic in our architecture

To enable the proper management of the traffic related to different nodes/applications, packets related to the same content, e.g., different packets carrying parts of the same file or of the same audio/video stream, are grouped into a given traffic flow. MANET and FiWi data planes are able to detect that different packets belong to the same application flow by exploiting a flow identifier. To this purpose, packets dispatched by the MANET overlay network are managed by the Real Ad-hoc Multi-hop Peer-to-peer (RAMP) middleware and tagged with specific flow ids at the application layer, i.e., exploiting a RAMP packet header field (additional details in [7]). Instead, the FiWi data plane exploits the IPv6 Flow Label header field to identify packets related to the same flow. Mobile nodes equipped with cellular capabilities managing traversing traffic between FiWi and MANET domains are in charge of modifying packets to maintain flow id information, i.e., forging IPv6 packets with the appropriate Flow Label in case of traffic from the MANET to the FiWi and specifying the appropriate application layer flow id in RAMP packets for traffic from the FiWi to the MANET domain.

By specifically focusing on the traffic traversing the access network, we adopt differentiated management for regular and special traffic. The former is managed in a best-effort but very dynamic way, e.g., by adopting machine learning to improve network utilization (like the DIANA solution [6]). In this case, traffic from/to mobile nodes is monitored in an aggregated fashion and the FiWi SDN controller does not need to differentiate among flows to/from different nodes. In fact, in this case, since applications have not indicated any specific requirement, the goal is to correctly dispatch packets without any differentiated management. The latter is managed in a quality-aware way, since either an application has specified requirements that the forthcoming traffic flow should fulfill or the traffic flow greatly differs from the regular pattern and thus should be appropriately managed to reduce its (potentially) negative impact on the remainder of the traffic. The guideline is to manage only “elephant flows,” with the associated overhead, but not “mice flows.” In particular, elephant flows consist of relatively huge packets and/or last for a long-time period, thus can impose a relevant load to the network and their appropriate management can improve overall QoS in a very significant way. On the contrary, mice flows are composed by small packets and are usually short lived; their impact on network performance is singularly limited and, moreover, the overhead due to control message exchange can impose an overhead greater than the traffic flow itself.

Let us stress that even if FiWi and MANET SDN controllers are logically centralized for each domain (abstracted as a single point of control in an inter-domain perspective), their implementation may be centralized or distributed (see Fig. 7). On the one hand, the FiWi SDN controller typically consists of multiple controllers, i.e., a centralized OLT SDN controller and multiple ONU-BS SDN controllers. However, the former is in charge of taking final traffic engineering decisions, while the latter are in charge of collecting information, sending them to the OLT SDN controller, and receive/apply decisions of the former. On the other hand, multiple distributed SDN controllers may be employed in MANET as well. In fact, the dynamic nature of spontaneous networks (if compared with traditional datacenters) prevents from the adoption of a unique truly centralized MANET SDN controller in charge of monitoring and managing hundreds or thousands of nodes. Each distributed MANET SDN controller has the primarily goal of enforcing QoS in its spontaneous island, i.e., a subset of nodes residing at relatively small multi-hop distance and sharing common interests (thus, with a relevant difference if compared with Open Network Operating System—ONOS [54], where a logically centralized SDN controller is distributed mainly to increase scalability and fault tolerance). Moreover, distributed SDN controllers of neighbor spontaneous islands eventually coordinate one each other to also support (as a secondary goal) QoS of inter-island traffic flows, generated by sender/receiver nodes residing in different spontaneous islands (additional details in [7]). For instance, Fig. 7 outlines three MANET SDN controllers, one for each ONU-BS, deployed on-top-of an LTE-equipped mobile node.

Fig. 7

Distributed SDN deployment and implementation in the FiWi-MANET architecture

6 Discussion on paving the way towards implementation: practical considerations and technical challenges

Enrolling SDN in integrated hybrid domains, such as the FiWi-MANET scenario, is still an early stage research field, hence, it is necessary to thoroughly consider several practical aspects and to discuss some open technical challenges related to efficient implementation. In fact, towards the full implementation and experimentation of our proposal over deployment environments of industrial interest, as Table 2 points out, several open issues call for further investigation:

• OpenFlow compatibility. OpenFlow has been developed to demonstrate the feasibility of shaping a unified control and management framework. Currently, many OpenFlow functionalities are available, e.g., load monitoring, finding the shortest way in a routing network, and providing per-flow control in a network. Nevertheless, new extensions of OpenFlow are needed to enable advanced capabilities, as discussed in Section 4, for example advanced bandwidth allocation strategies based on traffic prediction schemes;

• Global vision. SDN controllers should have a global vision of the MANET topology (lightweight coordination among MANET islands). Also, they should be able to monitor traffic flows in the MANET domain in real-time;

• SDN-based traffic monitoring in the MANET domain. Smartphones and tablets consume 40% of the data traffic and generate 99% of the signaling traffic [55]. Such high levels of traffic entail efficient bandwidth allocation in the CO. Hence, SDN capabilities have to be enhanced in order to support real-time traffic monitoring in MANET networks so as to provide a global and accurate traffic status in the backhaul domain (optical network);

• RAN topology information. SDN controllers should be able to acquire a global network view in terms of topology, uplink/downlink traffic flows and requirements, granted and required traffic per domain, and throughput and latency monitoring. In fact, in this manner, they would be able to acquire full knowledge about the current network status as well as about the forthcoming traffic generated by applications, thus allowing to take proper traffic management decisions. In [56], a fine-grained network resource allocation framework is provided; however, this framework is limited to LTE cellular networks. A wider cooperation of SDN controllers is needed to ensure that a global view in terms of bandwidth allocation is feasible for the whole network. By achieving this, fairness could be ensured by monitoring and adjusting fairness indices either per cell or per MANET domain;

• Additional resources. MANET devices and EPC servers are likely to need more resources to store the required information about the networking state under its domain of control. These resources may include flow entries, data traffic indices, traffic history, and bandwidth allocation matrices. Consequently, appropriate memory and CPU capabilities are required to store such information and to calculate the adequate handling for each session [57];

• Hardware abstractions. To support advanced PHY modifications based on information acquired by SDN controllers, new hardware abstractions are needed. Devices that should be adjusted to provide sophisticated capabilities, e.g., dynamic bandwidth allocation and uplink-to-downlink modifications, have to be enhanced with additional hardware abstractions that SDN controllers may use and exploit.

Table 2 Open implementation issues for federated FiWi-MANET deployments

The realization of the FiWi-MANET federated architecture requires advanced collaboration between the respective SDN controllers. Figure 8 presents an architectural block diagram of the FiWi-MANET SDN inter-connection network: all SDN controllers are interconnected through an inter-controller interface; the manager can abstract all technological domains in the underlying FiWi-MANET network and can configure network elements by using flow tables and advanced OpenFlow extensions for providing inter-controller policies. In finer detail, network management takes place centrally and is performed by the network operator, such as a telecommunications company or an Internet service provider (ISP), which has control over the whole infrastructure, including the optical core and the wireless access domain. Representative involved technologies are variants of PONs (e.g., XG-PON, NG-PON, 10G-EPON, 25G-EPON) as well as variants of cell standards (e.g., LTE, LTE-A, LTE-A Pro). In the MANET side, different versions of ad hoc mesh networking technologies can be used, such as schemes based on IEEE 802.11 or IEEE 802.15. Depending on the tenants’ real-time requirements and/or predefined service-level agreements (SLAs), the manager applies end-to-end policies over the FiWi-MANET architecture. These policies are mainly enforced through targeted resource allocations for traffic engineering in various parts of the involved network domains, enabling the formation of agile data paths. The establishment, maintenance, and adaptation of routes are realized through SDN commands in SDN-capable networking devices, issued by the FiWi SDN controller in collaboration with the MANET SDN controller.

Fig. 8

An SDN controller management network. A management policy is applied to the SDN network through a management computer located at the CO

The use of OpenFlow extensions for the implementation of the introduced inter-domain federated SDN control is the most dominant approach, due to the growing interest attracted by the OpenFlow standard. In the following, we present three related solutions. NETCONF [58] is a promising protocol for network management defined by the Internet Engineering Task Force (IETF). It relies on remote procedure calls (RPCs) to allow the configuration of various networking devices, such as routers, switches, and firewalls. It has been shown to produce less control overhead, but also to achieve lower bandwidth utilization, if compared to OpenFlow [59]. The Open vSwitch Database Management Protocol (OVSDB) is a network configuration solution based on OpenFlow designed for managing Open vSwitch [60]. The latter constitutes a virtual switch that allows the automation of networking processes through management protocols, such as NetFlow [61]. According to OVSBD, each SDN controller can manage multiple virtual switches, while each one of the latter can be managed by multiple controllers. The OpenFlow Configuration and Management Protocol (OF-CONFIG) has been lately defined by the Open Networking Foundation (ONF) as the fundamental mechanism for configuring vSwitches through OpenFlow SDN controllers [62]. A detailed review of SDN management approaches can be found in [63].

Here, we present some implementation fundamentals of the functions demonstrated in Section 5 in the context of specific scenarios. Specifically, the flow charts in Figs. 9, 10, and 11 show the course of tasks that need to be implemented to provide load balancing, fairness, and traffic engineering respectively, as they were illustrated in the corresponding scenarios. Specifically, Fig. 9 corresponds to the load balancing scenario depicted in Fig. 2 and presents the course of actions to be followed to avoid collided paths and to select less loaded ones from the CO to the MANET through the FiWi domain. Figure 10 presents the involved flow of actions for fairly allocating bandwidth among multiple RRHs, in line with the scenario illustrated in Fig. 5. Lastly, the traffic engineering scenario presented in Fig. 6 can be realized through the execution of the sequence of tasks presented in Fig. 11, allowing the adaptation of the uplink-to-downlink sub-frames ratio at the RRH, according to the respective traffic requirements.

Fig. 9

Federated tasks performed by FiWi and MANET SDN controllers (SCs) to provide load balancing

Fig. 10

Federated tasks performed by FiWi and MANET SDN controllers (SCs) to provide Fairness

Fig. 11

Federated tasks performed by FiWi and MANET SDN controllers (SCs) to conduct Traffic Engineering

7 Conclusions

The paper presents a novel architecture allowing to improve end-to-end QoS in hybrid MANET-FiWi environments by loosely integrating SDN controllers in charge of monitoring and controlling each domain. In particular, by allowing to share monitoring information among MANET and FiWi SDN controllers, we have shown the potential to take better control decisions, e.g., to achieve better load balancing and/or fair exploitation of resources based on requirements specified by applications running on top of mobile nodes. The encouraging preliminary results already achieved are stimulating our ongoing research work. We are mainly working on the development of a proof-of-concept prototype to apply the proposed architecture in ETSI mobile edge computing (MEC) networks, in particular to allow the efficient “elephant-oriented” dispatching of multimedia streams generated by surveillance cameras.