In order to deploy 5G private network, key enabling technology components as well as optimization and planning methodologies must be discussed or investigated for industrial non-public 5G networks. Before the deployment, stakeholders should consider spectrum selection and license but also channel measurement in order to fully understand the propagation characteristic of the environment and to set up end-to-end system parameters. During the deployment, a monitoring tool is necessary to validate the deployment and to make sure that the end-to-end system meets the target KPI. Finally, some optimization can be made individually for service placement, network slicing, network orchestration or jointly at RAN, MEC or core network level.
Spectrum allocation and channel models
Private networks constitute a paradigm shift for design and operation of mobile radio networks which originate from standards created for global deployments of public mobile network infrastructure to connect mobile phones everywhere where people live, travel and work. 5G setting the foundation for purpose targeted private networks in particular in an industrial context addressing non-functional and functional requirements and KPIs as well as spectrum and regulation, thus providing the framework for non-public deployments of 5G technology in Mission Critical Communication (MCC) in factory environments.
Spectrum allocation models for private networks
The electromagnetic spectrum is, for most parts, regulated by governments and harmonization across regions and continents was key for global success of standardized radio technologies. Some parts of the spectrum are allocated to general purposes, e.g., Industrial, Scientific, Medical (ISM), and are available unlicensed under strict usage rules. Other parts of the spectrum are licensed, which means that only the license holder can deploy and operate radio equipment and services in this particular spectrum. Spectrum that is designated for terrestrial mobile telecommunication services, needed to operate 4G, 5G and beyond, is usually divided into sub-bands auctioned off to MNO at high prices for a state or country wide license. Such mechanism became an obstacle for availability of designated spectrum to be licensed to professional users for localized private networks which had so far to share unlicensed spectrum potentially with other users and equipment operated in the same spectrum and the same location. This puts feasibility limits for MCC because it is impossible to guarantee quality of service or latency. As a consequence locally available licensed spectrum becomes a prerequisite for the success of Industry 4.0, while unlicensed spectrum provides additional capacity for non-MCC supplementary services. Fortunately, governments have started the process of opening allocated spectrum as licensed shared spectrum or dynamic spectrum sharing for localized use in specific bands to enable the deployment and operation of private 5G networks.
Licensed Shared Operation: Several countries, including Germany, UK, and Taiwan, have started the process of allocating parts of the 5G spectrum for local private use instead of for nationwide coverage. Non- internet service providers can apply for a license for up to 100 MHz of spectrum in the range of 3.7 to 4.9 GHz, depending on the country. For a small (yearly) fee, companies can then use those frequencies exclusively on their premises to deploy a private network.
Dynamic Spectrum Access: In the USA, the 3.5 GHz frequency band was recently opened up for commercial use by the US Federal Communications Commission. This band is known as the citizen broadband radio service and does not require spectrum licenses. Access and operation is governed by a dynamic spectrum access system, but the users are required to take care not to interfere with others already using nearby bands.
Measurement campaign
Before any large-scale system deployment, the propagation characteristics of the environment must be fully understood and parameterized for system level simulations. The aspect of interference coordination is especially important for private networks. Although a first version of the 3GPP TR 38.901 channel model supporting Indoor-Industrial scenarios has already been released, further research on the indoor industrial channel [33, 34] is necessary. Due to the highly reflective nature of the environment, caused by shop floors usually densely packed with metallic machinery, the indoor industrial channel shows effects such as diffuse or dense multi-path propagation that are not common in other types of scenarios. During the 5G CONNI project, the consortium conducted an extensive measurement campaign in an industrial environment at 3.7, 28 and 300 GHz. The results of this measurement campaign will be used to contribute to standardization and to enhance existing channel models.
The measurements were conducted in a machine hall with a dimension of approximately 100 by 45 m and a height of 4 m. On the ceiling, metallic air ducts cover most of the hall. Figure 14 shows a stylized floor plan of the machine hall. Several measurements were conducted in two general areas of the hall, displayed in Fig. 14 as Measurement Area 1 in blue and Measurement Area 2 in green. While Measurement Area 1 is sparsely packed with industrial machines and serves as a research, work and storage area, Measurement Area 2 is densely packed with industrial machinery and robots.
In Measurement Area 1, two transmitter locations at a height of 2.7 m above ground were chosen, while the receiver was placed at 15 different locations throughout the area. At both 3.7 and 28 GHz, the measurements were conducted using a Virtual Uniform Circular Array [35] antenna in order to estimate the angles of arrival. Additionally, at 9 points, measurements were conducted at 300 GHz [36]. In Measurement Area 2, a single transmitter location was chosen and measurements were conducted at 30 points throughout the area at 3.7 and 28 GHz. At 4 measurement points, the scenario was also characterized at 300 GHz.
In addition to the angle resolved measurements, the large-scale parameters at 3.7 and 28 GHz were also evaluated along a trajectory around the machine hall with a moving receiver. The trajectory is displayed in red in Fig. 14. For these measurements, the transmitter was placed in Measurement Area 1, the receiver was moved at a constant speed of 0.5 m per second along the trajectory, and the channel impulse responses were recorded every 8.1 cm. Finally, Indoor to Outdoor measurements were made at 3.7 and 28 GHz.
The results of the measurement campaign will be used both in the 5G CONNI project for connectivity maps and cell planning, and outside of the project for the standardization and refinement of indoor industrial channel models. Initial large-scale parameter evaluations of the measurements at 3.7 and 28 GHz have recently been published in [37].
Monitoring
A 5G end-to-end system including 5G RAN, MEC, CN and related application will be deployed into factories to provide demonstration and service for specific industrial application in 5G CONNI project. The target is not only to build up the 5G enabled communication infrastructure but also to make sure the operation and maintenance of specific industrial application will meet the KPI. To fulfill this, an OAM and KPI monitoring system will be designed into the system as well. Its implementation includes the management of fault, configuration, account, performance and security. In 5G CONNI project, except the account management, other management requirement is going to build up in this 5G end-to-end system. The specific KPI will be implemented with two requirements. One is from 3GPP specification. The other is from the use cases proposed in the project (e.g., end-to-end latency, service bit rate, time synchronization and secure remote access). With the implementation of OAM and associated KPIs in the end-to-end 5G system, users will be able to monitor, configure the system and improve operational efficiency accordingly.
To better describe the implementation, Fig. 15 shows the setup of the end to end 5G system. The management server will configure, e.g., Configuration Management, the RAN that includes the Radio Unit (RU) and Distributed Unite (DU)/ Central Unit (CU) via NetConf protocol while the RAN is setting up. If unexpected scenarios occur, the RAN generates an alarm, e.g., fault management, to the management server. During operation, the RAN generates counters or KPIs to management server. Some KPIs are calculated in the management server to form the KPIs.
Service placement
In the MEC paradigm, resource-intensive and delay-sensitive applications are handled directly in the edge cloud, avoiding as much as possible the access to cloud computing available in possibly distant data centers to prevent high service delays, which are not tolerable in the Industry 4.0 scenario. Of course the edge cloud offers less storage and computational resources than the large data centers. To overcome this issue, virtualization techniques such as virtual machines and containers help in creating an open edge computing environment, in which storage and computational resources are distributed across the edge cloud when and where needed. To properly exploit this possibility, it is necessary to optimize resource placement and scheduling, in particular, it is necessary to choose the nodes that are the most suitable to store the data and to run the applications offloaded from the querying sensors. In performing this selection, we advocate the use of a joint strategy that considers computational and storage resources jointly. Only few works have considered together communication, computation and caching resource. In [38], the authors solve the problem of service placement and request scheduling aiming to maximize the number of serving requests, allowing direct access only to one server per user. The work in [39] considers overlapping coverage areas for edge nodes with the aim to minimize the request routing to the core cloud (i.e., to maximize the assignment at the edge). Here, we propose a method to minimize the total delay spent on satisfying all service requests, jointly solving the problem where to store services and where to run the requested applications. We consider a wireless edge network consisting of a set \({\mathcal {N}}\) of edge nodes, each one endowed with a MEC server to enable virtualizing network services/applications (VNFs), i.e., equipped with storage, computation and communication capabilities, and endowed with a wireless access point, covering local areas, possibly overlapping. There is a set \({\mathcal {S}}\) of network services, usually consisting of virtual machines and/or containers, able to run such sophisticated applications. Services can be stored on the edge clouds in order to satisfy a set \({\mathcal {K}}\) of sensor devices, each demanding for network services, if there is an active connection link. The aim of our work is to minimize the total system delay occurring for transmitting data from each sensor to an edge server and for processing them. Thus, latency depends on the amount of information data needed to run the desired application, apart from the radio capacity of the transmission link and the computation capacity at the edge cloud. Sensors can offload their data if connected to an access point, then, if none of the neighbor edge nodes cached the demanded service, the device can be routed to a data center, generically indicated as a core cloud \({\mathcal {C}}\), storing the entire set S of services but placed at a much longer distance. The same occurs if none of the edge server has enough capacity to serve the request. To tackle the problem to solve an Integer Linear Program, we leverage an approximation algorithm based on a randomized rounding [40], with provably guarantees to satisfy the imposed communication and computation constraints on expectation.
To prove the effectiveness of the proposed algorithm, we compare the results with a matching algorithm, which aims to route each device to a node with the best SNR on the communication link, an admission occurs if the node has enough storage and computation capabilities to cache and run the requested service. In our simulations, we consider three types of virtual machines [41] micro, small and extra large, with CPU cycles from 500 to 2000 MHz and RAM from 0.6 to 3.7 GB, as processing and storage minimum resource necessary to run the application. The arrivals from sensors are generated with a Poisson distribution with mean arrival rate uniformly distributed \(\in [0.4, 1.1]\) Mbps. There are \({\mathcal {K}} = 100\) sensors, uniformly distributed in an area 100 m × 100 m, which offload their data to three possible edge nodes and request services following a popularity profile derived from the Zipf distribution with shape parameter 0.8. The channel model is based on [42]. As we can see from Fig. 16, the matching algorithm experiences a bigger delay due to a frequent request routing to the core cloud, especially when the bound on the storage capacity is very tight.
Our algorithm performs better because the service placement allows the system to amortize the cost of storing a lot of data, reusing the same virtual machine for more users. Of course this can be done for shareable resource as for analytic data cached at the server, otherwise computation and communication resource are dedicated to the specific querying device. The more storage capacity is available at each server, the less requests are routed to the core cloud, and then the system experiences less delay in delivering all the applications. For delay-sensitive services, it is very important to have access at the edge, and the proposed algorithm satisfies the request for such applications in a very fast and efficient way.
Key enabling technologies
MEC
The implementation of Multi-access Edge Computing (MEC) can be based on User Plane Function (UPF) [43] or bump-in-the-wire MEC, which can meet the requirements of high bandwidth and low latency service, and enhance security. In 5G-CONNI project, the UPF for the European testbed is designed according to 3GPP standard, and the MEC platform leverages the 5G network architecture and performs the traffic routing and steering function in the UPF. An UL classifier of the UPF can be used to steer the UP traffic matching the filters controlled by the Session Management Function (SMF) to the local data network, where it can be consumed by the MEC application. The Policy Control Function (PCF) and the SMF can set the policy to influence such traffic routing in the UPF. Also, the application function can influence the traffic routing and steering via the PCF. Therefore, MEC in 5G is able to influence the UPF through the standardized Cloud Provider (CP) interface in the SMF. Furthermore, the MEC platform is completely virtualized environment in order to enable seamless application lifecycle management paired with seamless platform management. On the other hand, the bump-in-the-wire MEC for the Taiwanese testbed is implemented according to ETSI standard, where MEC cloud includes an NFV infrastructure and data plane functions. NFV infrastructure is named ECoreCloud. The data plane functions comprise Software-Defined Networking (SDN) switch and Mobile Edge Enabler (MEE) VNF developed, where SDN switch is used to route and mirror the traffic, and MEE VNF provides a traffic steering function so that selected data traffic can be offloaded locally. The MEE VNF is divided into two modules, including the Control Plane Analyzer module and Data Plane Processor. The Control Plane Analyzer module takes charging in decoding and correlating signals, and the Data Plane Processor is to process data plane traffic and steering. The bump-in-the-wire MEC standalone prototype is deployed between 5G New Radio and 5G standalone core network, which is a convenient deployment because it does not need the additional configurations for the core and RAN network. Furthermore, the MEC must handle N2 interface according to 3GPP TS38.413 [44] and process the GPRS tunneling protocol user plane extension header packet for N3 interface based on 3GPP TS29.281 [45] so that the GPRS tunneling protocol user plane can be handled properly.
Network slicing and orchestration
Network slicing is a key feature of 5G networks to support diverse requirements on a single physical infrastructure through multiple logical virtual network functions. The virtual network functions are managed by NFV technology, which flexibly allocates virtual resources and provides modular architecture. In addition, SDN enables the communication between virtual network functions. By programmable network routing and separation planes, SDN technology achieves resource isolation for each network slice. With the benefit of NFV and SDN, network slicing allows operators to fast create on-demand network services for 5G vertical industries (e.g., smart factories, remote robotic surgery, and autonomous driving) based on slice configuration. Our goal is to realize a NFV-like lightweight 5G core in 5G CONNI. First, we implement a set of VNFs to run a 4G core and analyze the total completion time including instantiation time and configuration time as our performance metric. Based on our experimental result, the completion time is less than 12 minutes for deploying a total of 20 instances. The deployment efficiency is acceptable for operators to fast create different services. As a result, we will extend the lightweight framework with the same architecture to run a 5G core prototype including AMF, SMF, AUSF, UDM, and UPF.
Network orchestration is another key feature of 5G networks. The orchestrator needs to efficiently manage the 5G network in order to meet diverse and/or extreme QoS requirements and AI can play a key role in providing sub-optimal approaches. Because of the diversity and variation over time of QoS requirements, a versatile and adaptable network is needed to configure network parameters in dynamic environments and contexts while maintaining performance. Network management must also elastically orchestrate RAN, MEC, transport and core networks simultaneously, by exploiting scalable and flexible infrastructure.
In 5G CONNI, we propose an original methodology to design an end-to-end orchestrator taking into account the heterogeneity and coexistence of services, the dynamic evolution of needs (e.g., traffic, number of users, QoS) and the changing environment. The AI-based end-to-end orchestrator measures, predicts the network performance, dynamically modifies network parameters and elastically combines diversity to face to a multitude of (un)predictable impairments. Its design dynamically, elastically and efficiently deals with the complex ecosystem of tenants, network slices with a diversity of service requirements and efficient usage of resources. Figure 17 illustrates the orchestrator framework.
Before evaluating the AI-based orchestrator, we have modified a 5G NR network simulator based on NS-3 to multiplex low latency mechanisms (e.g., frame design) and reliability enhancing mechanisms (e.g., multiple antenna, redundancy and adaptive Modulation and Coding Scheme (MCS) exploiting code/time/space diversity). This simulator is used to evaluate the impact of the combination of mechanisms exploiting a diversity subset on the end-to-end Ultra-Reliably Low-Latency Communications (URLLC) performance at RAN level cooperating with EPC/LTE core network. Figure 18 compares the performance of the Adaptive Modulation and Coding (AMC) mechanism with the robust MCS5 and the high throughput MCS17 in a fast fading channel for indoor office scenario. The results provide insights into the behavior of different combinations with respect to URLLC performance and are a first step for further research using AI.
Core network
In the Taiwanese testbed, the 5G CN is designed based on service-based architecture and follows 3GPP Release 15+ as a standalone solution. The III-5G CN containerizes all core network functions with Control/User split architecture, enabling the enterprise to distribute these functions wherever and whenever needed. All the modules can be deployed on virtual machines on top of a large number of virtualization environments, and managed as a Kubernetes platform. The AMF establishes the UE context and Packet Data Unit (PDU) resource allocation via Single Network Slice Selection Assistance Information (S-NSSAI) provided by the UE. The S-NSSAI is set up per PDU session for the policy management in the PDU session level. The SMF controls the user plane function and therefore directs and redirects the service flows as required for the applications. We are testing the core network basic functions like UE registration, PDU session establishment, service request and Xn & N2 handover procedure via Spirent Landslide emulator. Furthermore, for supporting the industrial applications, the development especially focuses on data plane efficiency and system reliability. Thus, we are developing both software and hardware accelerating solutions for data plane to enhance packet processing and load monitoring.
Joint optimization of enabling technologies
In private industrial networks, devising computation offloading strategies to enable complex processing of data collected by mobile inspectors/sensors/machines is necessary in order to guarantee continuous monitoring and anomaly detection and control decisions during industrial processes. Thus, we now present a strategy for dynamic resource allocation for computation offloading of machine learning tasks at the edge, in the new framework of Edge Machine Learning [46, 47]. The goal is to allocate radio (e.g., transmit power, quantization) and computation resources (e.g., CPU scheduling at the edge server), to explore the trade-off between network energy consumption, end-to-end (E2E) delay and accuracy of the learning task. Here, the E2E delay is intended as the time elapsed from the generation of a new data unit by a sensor/mobile device, until its computation is performed at the Edge Server (ES) to run the online learning task. Indeed, as clear from Fig. 19, data units experience a local queueing delay (red queue), a transmission delay for data uploading to the ES, a remote queueing delay at the ES (blue queue), and a computing time to be elaborated. The output of the computation performed by the ES is an estimation/prediction/classification result (up right part of the figure).
In [48], the authors present a distributed learning algorithm at the edge, where end devices, helped by an ES, minimize an empirical loss function in a collaborative fashion. The work in [49] proposes an strategy to maximize learning accuracy under latency constraints. Elgabli et al. [50] proposes a decentralized machine learning algorithm that dynamically optimizes a stochastic quantization method, with applications to regression and image classification, and with a communication-efficient perspective. A stochastic gradient descent distributed machine learning algorithm at the edge is presented in [51]. In particular, the authors consider the trade-off between local update and global aggregation. In [52], the authors present a data compression algorithm to reduce the communication burden and energy consumption of an IoT network, to enable machine learning with a desired target accuracy. Edge machine learning is a research topic at its infancy, so that the research community just started investigating the possible directions. Our goal is to jointly optimize radio and computation resources to minimize the network energy consumption, under constraints on the E2E delay and the accuracy of the learning tasks, which in this case is an estimation task based on least mean squares. The power of the method is that it does not require any prior knowledge of the statistics of data arrivals, radio channels, and data distributions. It should be noted that both the transmit energy consumption of the sensors and the learning accuracy are affected by the number of quantization bits used to encode the data. More bits lead to better accuracy but higher energy consumption, due to the longer payloads to be transmitted. As a measure for the accuracy, we use the Mean Squared Deviation (MSD) between the true parameter and the estimation performed on offloaded data. More technical details can be found in [46].
We now illustrate the performance of our solution in terms of trade-off between energy, delay, and learning accuracy. In particular, we want to highlight how, given a certain E2E delay constraint (set to 30 ms), the accuracy affects the performance in term of energy consumption. To this aim, we consider 5 sensors at the same distance from the AP, with the same arrival rate, the same average E2E delay constraint, but different constraints on the MSD (i.e., the accuracy). In particular, we assume that two sensors represent two benchmarks: (i) minimum energy, obtained by the device always transmitting with 3 quantization bits (i.e., the minimum number of bits), for all t; (ii) best accuracy, obtained by the device always transmitting data with 8 quantization bits (i.e., the maximum number of bits). The other three devices have different intermediate requirements for the MSD. In Fig. 20a, we show the average E2E delay as a function of the sensor energy consumption, obtained by tuning a trade-off parameter from Lyapunov optimization [46]. In particular, the parameter increases from right to left, as shown in the figure. From Fig. 20a, we can notice how the energy consumption decreases while the E2E delay increases. However, this trade-off is different among the different devices due to the different accuracy constraints. In particular, let us first comment on the results for the two benchmarks. The green curve (squared marker) shows the best accuracy case, which indeed achieves the highest minimum energy among all devices, given the E2E delay (the legend of all figures is shown in Fig. 20a). At the same time, from Fig. 20b, which shows the MSD vs. the time index, we can notice how this device achieves the minimum MSD. These first two results are due to the maximum number of quantization bits shown in Fig. 20c, which reaches the highest value for this device. Note that the average number of quantization bits is shown as a function of the Lyapunov tuning parameter, using the same values as for Fig. 20a. On the other hand, the blue curve (triangle marker) represents the minimum energy case, and it achieves the worst accuracy due to the minimum number of quantization bits adopted, as can be noticed from Fig. 20a–c. The other curves represent intermediate energy cases obtained fixing a target MSD constraint, and can be interpreted via similar analysis over Fig. 20a–c. Finally, the energy consumption of the ES decreases as the Lyapunov parameter increases, until reaching a floor as the device energy consumption (Fig. 20d). In summary, the take-home message of Fig. 20 is twofold: (i) Our method is able to obtain a low system energy solution with accuracy and E2E delay guarantees; (ii) by relaxing the accuracy constraint, lower energy can be achieved due to the lower number of quantization bits, which translates into a lower average data rate over the wireless interface.
The accuracy required is highly dependent on the particular use case. It is not always desirable to obtain the best possible accuracy, if it results in higher energy cost, given an E2E delay. With our method, by setting a target accuracy, it is possible to achieve lower energy consumption without degrading the application performance below the target threshold.